LCA 2016 Day 4

An overcast, damp day in Ballarat. It’s like being at home!

Keynote - Community 3.0

Jono Bacon

“The only way we can build strong communities is to do it together.”

“When I was 18 or 19 I got interesting in Linux” via his brother putting Slackware on his computer. It gave access to a whole new world - now he was no longer limited to the scope of his local community; this was “Community 1.0” - online, building things together.

This worked observationally: Linus, for example, did not think conciously about becoming a community manager. It was ad-hoc, trial and error, organic. It could be difficult to find interesting to find projects; it was also highly and specifically technical; if you didn’t know C you couldn’t really contrbute or participate.

Then we get to Community 2.0: Wikipedia, OpenStack, RedHat. Wikipedia was a concious effort to create a community; Red Hat have become the first billion dollar free software company; GitHub has become the de-facto place to collaborate.

The Community Management rennaisance: people were now documenting how this all worked, allowing people to gain a head start; also the mergence of self-organising groups like the Free Desktop group, the GNOME and KDE teams, and so on and so forth.

It’s also the emergence of more concern for diversity: diversity of who we are (gender, race, etc) but also diversity of skills (documentation, advocacy, and non core programming technical skills).

And finally: company engagement. Companies wanting to join in with communities, which has been challenging.

So what does community 3.0 look like? And who cares?

We should care because thoughtful communities make humanity better: as a species we aren’t reaching our potential, because we aren’t doing communities as well as we could. If we put some effort into doing communities better, humanity will be better.

So where are we today: computing power has escalated, as has access (mobile). Power and connectivity have escalated massively and are becoming ubiquitous, and we’re gaining new capabilites: 3d printing, drones, cloud, open source, crowdfunding. New ways of doing things, new tools, and new sources of funding.

We have hundreds of millions of new minds from all around the world gaining new opportunities.

We need to empower diversity: in terms of who people are, and in terms of the skills we empower: artists should be as important as C programmers.

We need bold and audacious goals: “How do we document the planet?” was an audacious goal. “How do we get to Mars?” is an ambitious goal.

Openness is a key part of this: Arduino, Rasberry Pi, and so on. “Kids are growing up expecting openness”, and this is something Jono is trying to teach companies. There are python-programmable drones, for example. Similarly, AI should be open source: companies shouldn’t control it.

We need to build a bridge between atoms and bits: “Open source is where society innovates and collaborates”. The nut to crack, though, is predicatble collaboration - communities shouldn’t be dependent on having the right expert to make things happen. It’s about making collaborative communities reliable and autonomous.

We can do better. Much better. This is why Jono is at GitHub, because he thinks they can make a difference. This should be infromation that is useful to everybody, not just people using GitHub.

This is not just about technology. When you remove all of the stuff - the laptops and lights and banners - we’re just humans, with out human emotions. Everyone wants to live a life of dignity, to have a sense of self-respect and self-worth. A feeling that we can contribute is part of that, and to contribute is to have access.

Think about tangible and intangible value: tangible value is very useful, but it’s not the hard problem: it’s the intangible values. “I don’t really like spreadsheets” and the world as seen through the eyes of economists. They work on the assumption - completely false - that humans are rational. Humans are not rational. We make irrational choices. We’re animals. We’re predictable in irrational ways, though.

This study, the study of predictable irrationality, is Behavioural Economics. We have a (chocolate) brain, with types of thinking; one suggested way of thinking about these: “System 1” thinking; instinctive, automatic. What’s 1+1? That’s System 1 thinking. System 2 thinking is based around analysis: “I have a budget for a car, what do I buy?”.

If we can understand System 1 thinking, if we can understand irrational behaviour, we can build better communities. The main problem with Behavioural Economics is that while it’s very interesting, it’s not necessarily practical and applicable. Probably the most fascinating and applicable theory is the “SCARF model”:

SCARF

• Status: clarity in relative importance.
• Certainty: creating security and predictability. This can be very problematic where there is information asymmetery - an anxiety gap.
• Autonomy: creting intentional choices. Sometimes we will structure choices with no meaningful difference in outcome.
• Relatedness: define clear social groupings. “Developers”, “Documentation writers”, and so on. It’s comforting to be part of a small sub-culture in a broader community.
• Fairness: reducing unfair opportunity and rewards.

Initiative and Opportunity

Two golden rules:

• We accomplish our goals indirectly. Consider Boeing: when they moved from “making great aeroplanes” they delivered great value. When they moved to “deliver shareholder value” and stopped making great planes, and made less money. When they moved to “make great aeroplanes” again, they started making more money as a side-effect.
• We influence behaviour with small actions. Consider one of New Zealand’s most effective anti-smoking campaigns: “Do you smoke? Do you want help stopping?”

There are many small choices that have an effect on how our community works; in aggregate, the workflow of how you do things is a series of small choices that add up to an overall experience. That in and of itself is both very important but also quite poorly understood; there are many different models, but they usually have developed ad-hoc rather than thoughtfully.

Mapping out our workflows and understanding them and refining them could be very helpful, but it’s not that simple, because workflows can have very different experiences for different people in the same workflow: the experience of a new developer traversing the workflow may be very different to an experience community member traversing the same workflow.

System 1 & 2 thinking + behavioural patters + workflow + experiences + packaged guidance = Community 3.0.

When we can package this up so it’s accessible, reproducible, and fair to people, we have Community 3.0. And the most important thing we can give people is a sense of belonging. You will join, you will stick around for years.

Q&A

• Do you have any ideas for improving the signal:noise ration in communities? The challenge is that a lot of the people who are generating noise are often coming from a good place, not trolls. Mentorship can help - helping people understand how to be effective. Another problem is reputation - how do we have a notion of what great contrinution and good reputation looks like so we can communicate what that looks like.
• We see sites like StackOverflow bridge technical and non-technical communities, but also very destructive communities and behaviours; how do we bridge technical expertise to better behaviours? Leaders in communities need to set a great example. There are people who don’t like that - who just don’t want to have to give fucks. We shouldn’t ask people to not be themselves - but we shouldn’t accept bad, destructive behaviours. Tools can’t really solve this problem, but they can help humans make those decisions.
• How do we work with leaders like Linus who didn’t set out to be leaders, and don’t want to modify their behaviours? Well, leadership is about setting a good example - and people who don’t want that responsibility shouldn’t be leaders, but it’s important not conflate “direct communication” with “abusive behaviour”.

Open Source Tools for Distributed Systems Administration

Elizabeth K Joseph

How most projects to infrastructure

• Team or company manage it - or code hosting is used.
• Mailing lists, bug reports, or ticketing are used for submitting (infrastructure) changes to the infrastructre team.
• This means the priority of requests are determined by the infrastructure team, not the project team.

The OpenStack team thought there ought to be a better way - after all, they’re an infrastructure project!

OpenStack Infrastructure Team

• The job is to make sure the devs can do their job.
• If you find github repos, they’re unofficial mirrors.
• The system configurations are open source and tracked in git.
• Anyone can propose infrastructure changes.
• Everyone works remotely: US Russia, Australia, Spain.
• There is a fleet of 800 machines for running tests.

OpenStack CI

• Many projects (over 800).
• All projects must work together.
• Changes can’t break master.
• Code should be clean.
• Testing must be automated, and you can’t check code in until the tests run successfully.
• Tools include:
  • Launchpad (some day: openstackid).
  • Git.
  • Gerrit.
  • Zuul (the gatekeeper, feeds requests from gerrit into Jenkins).
  • Gearman (distributed Jenkins).
  • Jenkins (using the jenkins-job-builder, which lets you spec Jenkins jobs in YAML).
  • Nodepool manages the test server fleet.

Why do you care? The infrastructure team use it too!

Automated Testing for Infrastructure

• flake8 for checking the python scripts.
• puppet parser validate and puppet lint to check puppet code and ensure it’s standardised.
• Puppet application tests (e.g. rspec).
• XML checkers.
• Alphabetized files - humans are terrible at the alphabet, it turns out, but humans are quite good at it.
• Check IRC channel permissions for IRC bots (on freenode).

Peer review means

• Multiple eyes on changes prior to merging.
• Good infrastructure for developing new solutons.
• No special process for commit access.
• Trains us to be collaborators by default. It’s critical for a distributed team, and it can’t just be “dump it in pastebin”.
• Since anyone can contribute, anyone can devote resources. This is important - for example, when some people wanted an Asterix server, the core team didn’t really know how to do it; instead, people volunteered to set it up and were able to.

Can you really manage an infrastructure via git commits?

Yes you can. There are some tools to help people who are anxious about this.

They use cacti to monitor the servers - for example disk full events, or what have you.

They don’t do any active monitoring. Developers will tell them if anything is broken (they would like help in this space, anyway).

PuppetBoard is good for letting you watch your changes get applied. Or fail. PuppetBoard, by default, will give you searching and infodumps that shouldn’t be publicly exposed, so they had to turn options off to run publicly.

Good documentation is important; their important is written in Markdown and links to the actual artifacts.

Not All Servers

• Automation is not perfect, and sometimes you need to log in - MySQL interaction, certain log files, legacy systems that haven’t been migrated to a full automation stack.
• Complicated migrations may not work through git.
• Initial persistent server depoyment still has manual components.

Maintenence on IRC+Etherpad

• Use IRC to co-ordinate maintenence, but also Etherpad to documentation activites.
• There are seperate IRC channels for incidents, sprints, weekly meetings, and so on.
• Each serves a different purpose; keeping incidents out of the general chat is invaluable for focus; sprints are part of the regular decision-making process; weekly meetings are part of the general team catch-up/bonding.
• Everything is logged.
• And pastebin is actually used for chucking log fragments and the like around.
• In-person meetings are still crucial - it’s not a “nice to have”, it’s a “need to have”. Critical for team cohesion, even though they only happen a few times a year.
• No voice or video calls.

But…

• Timezones are a problem. They improve coverage, but the first/early members in a region can feel isolated.
• More reluctant to land changes into production.
• Makes for slower on-boarding for key tasks.
• Only solved by increasing coverage in the zone.

In Closing

• It’s mostly really great!
• The team always needs help!
• Helping is a path to learning heaps about OpenStack. Learning heaps about OpenStack can be a great way to get a job.

Q&A

• What don’t you like about voice and video? There’s no recording/searchability/accountability.
• The culture of taking time out (holidays and so on) requires good documentation - you can’t have a healthy culture of people taking care of themselves.
• How do you allocate work? If you’re working, you’re on IRC, so allocating incident work is co-ordinated there. Sprint and weekly meetings are used for allocating out more long-term/periodic tasks.

Edlib - because one more editor is never enough

“How many guitars are enough,” I asked my son, “Just one more” he said.

“I love emacs, I have been using it for 30 years. I love vim, too! Just not as much.”

emacs is good, but it can’t be perfect. After all, there’s a hundred more editors listed on Wikipedia. So let’s break down how emacs works in an MVC kind of way.

Seriously, there are some shortcomings; things like hexl-mode purports to be a hex editor, but it’s not very effective for large objects. The mail reader isn’t a “list of email” it’s a “text buffer snapshot of the list of email”. The directory or buffer lists are likewise “screenshots of the list of things”.

Views in emacs are very limited - they can do some nice things, but they’re aren’t very programmable; no display hooks, for example.

Neil’s breaking point with emacs was when he attempted to do reporting with a LaTeX-driven spreadsheet. After discovering that what he thought should be a fairly simple flow required re-applying hooks and repainting the screen constantly, he decided there must be a better way.

“elisp. You know, really? In this day and age. Python or lua or rust or C or anything.”

edlib - scratching an itch

Everything is pluggable: the language bindings, the document format backends, multiple renderers, multiple display managers, and loadable libraries for extending it.

Everything is based around panes: a pain is an area of interaction; they may have children, and send and recieve all messages. There may be a pane that performs the keybindings, a pane that renders to the display, a pane that provides for four windows on the document.

Documents and marks - marks are places in the document that you might want to perform work or comparisons on. It doesn’t fit into the abstration quite as well as Neil would like, so it’s a bit of a work in progress.

Commands have a fixed set of arguments as part of their calling conventions, and can act on panes, regions in panes, or call another commands. One consequence of the structure is you can easily have commands in different languages calling one another.

Attributes are named strings that apply to characters.

cygwin build notes:

-fPIC is not for cygwin; make CFLAGS= is an interim work-around. –disable-new-dtags for the linker is only supported on GNU ld.

Playing to Lose - Making Sensible Security Decisions by Assuming the Worst

Tom Eastman

What if you found out you were hacked? Six months ago? Hysteria?

People are often nervous when asked that thought experiment. But there is a correct answer: “Well, it depends.”

Imagining how you’ll get hacked is how you protect yourself from getting hacked. “It’s going to be a litany of despair.”

Who are your threats?

Opportunistic Script Kiddies: metasploit, readily accessible tools are dangerous because they have time, because they’re in it for the lulz.

Organised criminals: they have time and money to come after you seriously. They have a goal; their goal is money. But that’s easy - if you can derail their business plan, they’ll give up.

Disguntled former employees: these people are hard. You can be the nicest company in the world, but that doesn’t mean someone isn’t angry anyway.

Hacktivists: wildly varying means and motivation.

Nation-state actors: “My advice probably isn’t good enough for you.”

These are “Security personas”. If you work in Agile, you already make personas for people who want to use your site legitimately; add in personas for people who are trying to harm you.

And then I ran out of battery.