Sydney may have a lot of things going for it, but the climate is definitely amongst of them. Still, in spite of the hot, sticky weather, I’m enjoying wending my way through the streets of Redfern every morning; with it’s tiny old row houses and narrow alleys it would seem to be more what you’d find in the heart of an old European city like York; then I exit onto some of the stunningly designed newer buildings provides a sharp and delightful contrast.
I have no idea how anyone affords to live here, though.
Today’s lucky winner was here first time, which makes a nice change. A reminder that this conf’s charity is Code Club Australia.
Wandering Through the Commons
Hugh Blemings
Hugh notes that he’s largely self-taught, both with hardware and software. His only early formal training was a paper during university; he moved from playing bands to working in electronics and embedded software design, and after some twists and turns he’s ended up working at the OpenPOWER Foundation, although he is not representing them here.
Hugh ran through a higstory of LCA, the high point of which for me was the story about Van Jacobson pitching in to crimp Cat5 cables back in the early days.
Foss in Australia & New Zealand
“Between Australia and New Zelaand we have one of the most wonderfully vibrant FOSS communities in the world.” Hugh shows a word cloud of individuals and groups
- Back in the day John Lions wrote the famous Lions Book, 6th edition, which was one of the great teaching tools for operationing systems in general, and Unix in particular.
- Pia Andrews (née Waugh) has had a huge influence on open source and open data in government in Australia and now New Zealand.
- Vik Olliver is perhaps less well-known than he ought to be, for his work on the RepRap, one of the blue-sky visionary 3D printers.
- Hugh also mentions Claire Curran for her enthusiasm for open source in New Zealand governments.
Working in FOSS and Open Hardware
Getting ready:
- Be visible in projects of relevance.
- Be yourself - but be business friendly.
- No matter what you might think of it, Linkedin is a thing. You need a basic presence.
Finding a new job:
- Your people networks.
- Local user groups.
- Conferences.
- The projects you work on.
Applications & Negotiations:
- Be professional and courteous.
- Do your homeowrk on their culture.
- Talk to people who work there - it can be very different on the inside from the facade.
- Spend time on interview prep. Know your stuff. Be honest.
- Think about your salary expecations and stick to them - Val Aurora’s page on this is excellent.
- “People are more likely to ask for too little, not too much.”
- Ask to keep copyright on your code.
In the Job
- Don’t sweat getting into the groove in a new job.
- Get out every now and then, particularly if you’re working from home.
- Be mindful of work/life balance.
- Know when to jump.
- If you manage people, bring your best or don’t bother.
Looking after you:
- Ours is, in the main, a sedentary and solitary pursuit (aside: I have to disagree with the second assertion. Programming is a team sport, even if not always face to face).
- Sitting or standing is a real thing.
- Depression is a real thing for many of is (aside: please don’t recommend exercise for depression.)
- Beyond Blue and Blue Hackers are great.
- Eat more vegetables. Beer is not a vegetable.
- Exercise.
Stay current:
- Do something that’s not part of your day job.
- Stay up to take with security blogs and the like.
- Take the lid off/tinker with hardware.
Make hay while the sun shines:
- Save!
- Keep your networks going.
You’re fired… now what?
- Don’t panic. Going out in a tweetstorm won’t help.
- It’s not personal (well… about that).
- Is it legal? If you think it’s unfair, seek further advice before responding.
- It’s normal to feel a bit rubbish.
- Doubt can creep in.
- Beware imposter syndrome.
- Try to keep 2 - 3 opportunities in the pipeline.
- Don’t assume people will remember you - keep in touch with people.
- Keep your search narrow for a few weeks, then expand widely.
- Balance finding “something” and “the perfect thing”.
Dream Jobs
Hugh passed around some POWER9 cores for the audience to look at, so that was nice.
- 8 x 10^9 transistors.
- 25 km of 14 nm wires.
- Carrying up to 150 Amps at times.
- 3900 off-chip pins.
- No management engine (plug plug).
The OpenPOWER Foundation was formed in 2013 by Google, IBM, Mellanox, NVIDIA, and Tyan, and now sits with over 300 members, including Red Hat and Ubuntu. The foundation’s goals are to build hardware specs that can be turned into competitive systems. Hugh notes that they have an entirely open source software stack:
- Boot loaders.
- Management code.
- Firmware.
- Hypervisor.
- Linux kernel.
Spectre and Meltdown Panel
Jessie Frazelle, Benno Rice, Kees Cook, Katie McLaughlin, Andrew ‘bunnie’ Huang, Jonathan Corbet
“This panel was organised the way the response was organised, the embargo only broke on the panel this morning … this is more about how the community responded to it, rather than the technical details of Spectre variant 2.”
Jessie: There was a lot of confusion and a lack of knowledge around how containers work had people asking if containers would stop the flaw. The embargo was a “shitshow”.
Benno: FreeBSD weren’t told about the flaw until 11 days before the embargo was going to lift.
Kees: Within Google the knowledge was very constrained, so trying to speak to people, even within Google, was difficult.
Katie: Only heard about it on twitter. Trying to work out how to respond as a platform provider is really hard.
bunnie: I feel for the engineers who work on the chips; people demand more and more performance all the time, and this is a side effect of people trying to meet that demand. bunnie is concerned this will cause a push back on openness.
Jessie noted that the class system of the emargo is very concerning - many providers have been effectively crippled by being in the “tier 2” of disclosure. Katie described the scramble to work around the effects at her provider, with only 6 days notice. Things were locked down, code repos were made read-only, No-one had a good idea on how fix this problem at the moment.
(I would add this is really the tip of the iceberg - banks, stock exchanges, hospitals, and so on are all in this bind.)
Benno notes that these people are Intel’s customers, they pay Intel for hardware, and they haven’t been told their hardware is faulty.
bunnie: “Any time you fake performance you will have side channel attacks … open hardware can’t work around the laws of physics … there are whole classes of bugs waiting to be found with register aliasing, embedded code [ and other scary things ]”
Jon: “Can reproducible build techniques help here?”
bunnie: Checking that a chip is built as intended is really hard. You would have to decap the ship; some processors don’t even have the source available.
bunnie: We build chips with gates a few atoms wide. This is really, really hard. We can’t build our own tools (fabs), like software developers do. We can’t really apply the same techniques.
Fraser Tweedie (question): People have been warning against timing attacks, including with speculative execution, but were ignored.
Jon: People say that about using C, too. At some point we have to take risk.
bunnie: As a hardware engineer I think it’s insane you think you can run secret and non-secret code on the same hardware.
From Twitter: How were people selected for notification?
Jon: Existing disclosure processes weren’t used, which is frustrating.
Kees: This is a new scenario - previous mechanisms were oriented to software, not hardware. This is a work in progress.
Benno: I’ve gone to my Project Zero contacts and said “if you see something that makes you want to patch Linux, let us know.” The point of the emargo is to let a subset of the world get ready for the disclosure, and it didn’t work this time - hardware disclosure shouldn’t really be that different.
Jon: Did containers help?
Jessie: It should make it easier to move from one vulnerable platform to another, and make kernel upgrades easier. But she’s not aware how that panned out in practise.
Katie: We were able to move some workloads quickly from vulnerable hardware to patched hardware, but this ultimately relied on patched Amazon regions being available.
Question: Is this an opening slavo of 20 years of hell?
Kees: The page table protection mechanisms are intended to be generic enough to shut down a lot of timing attacks.
Benno: People have been showing leaking since hyperthreads. All these optimisations will be under scrutiny.
bunnie: The hardware can be made more secure, adding noise to the predictors, for example, but all these techniques will degrade performance. Will customers agree to buy more security for less performance?
Question: How do we explain things like these to a broader audience?
Katie: Fancy logos [ laughter ] no seriously, people will recognise fancy logos. People like to make fun of vulnerability logos, but they raise awareness. It works.
Jon: Creating a greater security awareness has been a work in progress, but even kernel developers get their laptops compromised.
Question: Have people looked at older CPUs?
Jon: This has been a feature since the Pentium. I think it’s a technique that’s with us until we get rid of the memory bandwidth problem.
bunnie: Who are you actually trying to stop with this disclosure process? Script kiddies? Nation state actors.
Jon: Script kiddies are less a problem than commercial interests. There are rumours of these exploits being on the market before the knowledge.
bunnie: State level actors are already listening to your conversations about this problem. This just prevents us responding quickly to the problem.
Closing remarks:
Katie: patch your stuff. No, you actually need to do it. You need to be up to date.
Kees: Run a up to date kernel. This is not a bad year! The badness was last year! The fix is this year!
Benno: We need to do disclosure better.
Jessie: Containers will not save you, but they will make upgrades better.
Jon: there is still too much secrecy. We can’t work out how to do better.
Making Technology More Inclusive Through Papercraft and Sound
Andrew ‘bunnie’ Huang
Why Inclusiveness?
Open source derives power from inclusiveness. The more people who can contribute, the better. In general.
Some execptions: legal threats led to a dev junking his module. People loading poisoned payloads to NPM or rubygems. bunnie was sad by the xscreensaver timebomb. He uses the last as an example of how few people are fully empowered by open source and asks, “what if this kind of time bomb triggered on politicians’ phones.”
So what if someone decided that programmers, should, like locksmiths, be bonded. Ridiculous, right? Well, Brexit, Trump, Australian broadband all happened.
bunnie invites us to “sober up”, presenting a large, multinational survery of technology skills: in most countries fewer than 50% of people have better than ‘poor’ skills, meaning can’t do more than type a search into Google. You should be scared.
Because open source is politically powerless without inclusiveness.
- < 1% of people are empowered by open source, even though 100% of people rely on it.
- But pestering people to learn to use apt-get is not the answer.
As an engineer bunnie wants to look for obvious issues, the hot spots. So what’s an obvious problem - well, look around the conference? Women massively underparticipate in computer science generally and open source in particular. Computer science is vastly more exclusionary that maths or physics, for example.
One observation bunnie makes: this is different in China. The tech gender balance is not a fundamental issue, it’s a cultural issue that is keeping women out. In fact, CMU managed to go from 10% to 40% enrollment in 5 years. Harvey Mudd got graduation from 12% to 45% in 5 years by changing their program.
CS is unique because it assumes you are a programmer before you start. Law school doesn’t assume you’ve been in court. This functions as a hostile environment to women, who have been told they shouldn’t be interested in technology or engineering since they were young.
Paper Electronics
So bunnie has been working on different ways to teach electronics - for example, mixing electronics and origami via paper electronics. It lets you make the electronics more easily and interesting - and bunnie notes, the paper electronics is actually closer to real electronics problems than breadboard - for example, it teaches routing.
(Paper electronics uses copper tape and paper as the base, with components added on.)
Barriers to mass adoption
Combine this with a flex PCB, making paper-compatible stickers to give more re-usable circut stickers.
The “accidental start-up”. Now we need to work out what we do:
- We are not pinked-out or watered-down tech.
- We are rigorous tech that is seriously fun, and recognise technology and design are equals.
Interestingly enough the channel they’ve sold through affects the gender of the purchasers - for example, Amazon skews male, but a Chinese vendor skews female. Users who identify as crafters, artists, or educators are majority women; users who identify as makers skew more male.
“Engineering finds meaning with design.”
- Balance.
- Familiarity.
- Simplicity.
Diverse thinking is inclusive and balances the axes of achievement:
- Measure achievement on multiple axes.
- Not just how well one can code.
- Not just how well one can code.
Paper is an expressive material; paper is a technical material. It’s also a familiar material.
Simlicity: it is hard to get code into microcontroller. This is a problem; there are a lot of conventions that are often hostile to developers. And the conventions - connectors are constantly changing.
So bunnie uses sound as a carrier for code, rather than USB or serial or SD or microSD or thunderbolt. You play sound from a browser-based compiler which plays a sound that the controller listens to and loads code.
The big challenge is that developing hardware is hard. It’s taken a long time to work out what the right form for the microcontroller; in the end it’s a clip-on.
All of which needs to be supported by classroom material to walk people through the ideas. The circuts are crafted into the book. The “Learn to Code” book is free to download, as well as teh code itself.
As well as the native vode for ChibiOS, there’s support for Microsoft MakeCode, which gives a more Scratch-like experience.
To sum up:
- Just pushing to git is not inclusive.
- Reaching across the technology divide is really hard.
- We need to empower society so they can understand its value.
Mirror, mirror, on the wall: testing Conway’s Law in open source communities
Lindsay Holmwood
The quote referred to as Conway’s Law comes from a paper called_How Committees Innovate_. Cited in The Mythical Man-Month and popularised.
There are two seperate research traditions how have studied this area in parallel; computer science, sure, but also in Management theory, both in organisation design and product design. There are, in fact, over 250 papers in the management theory area. This culminated in a metstudy called The mirroring hypothesis.
What is mirroring?
In any effort to collaborate, there are two seperate networks in play: an organisational network and a technical structure. The former is how people are organised, the latter is what is build and how components inter-related.
So why do we mirror? Because it’s an effective and efficient problem-solving technique. It reduces cognitive overhead. With a 1:1 mapping between systems and people, it is easy to see who owns what.
This makes it an economical way to maintain systems in an organisation.
Reviewing the review
Galbraith’s Organisation Design: An Information Processing View is a seminal work that is oft-cited in the field. One key note is that uncertainty increases, the amount of information that must be processed by decision makers increases. So one can either increase the capacity to process inforation, or decrease need for processing.
Creation of Lateral Relations (increasing capacity to process information) is congruous with DevOps ideas: direct contact liason roles, task forces, teams, integrated roles, a matrix organisation. A matrix organisation is a way of organisating what we now call cross-functional teams.
The level of uncertainty influences the decisions made.
D.L. Parnas: On the Criteria To Be Used in Decomposing Systems in Modules; a massively referenced paper. One key idea: Information hiding - and interface or definition to reveal as little as possible about its inner workings. The full context of a component is obscured so that we don’t require everyone in an org to understand everything about how it works..
Henderson & Clark: Architectural Innovation: was a 1990, 3 year field study of how the photolithographic adjustment industry worked. It’s a high rate of change industry. There were 4 waves of innovation across 45 years. Each wave lead to a new leader in the market; effectively every decade. Each time, the incumbent failed to adjust to change, even though they invested massively to try to take advantage of the new technology. The paper noted the failure was because the orgs structured their new product teams like their old organisational structure.
The analysis framework was based on observations from the aviation industry with two major strategies identified:
- Problem solving: Channels (A reports to B); Filters (deciding what is and isn’t important); stretegies (how we use these channels and filters to solve problems).
- Innovation (a 2x2 grid around which organisations move about).
In your organisation, don’t mirror if you’re looking to take advantage of new opportunities. If you do want to take advantage of existing opportunities, do mirror. Partial mirroring - say a guild structure - can be a useful way of achieving this.
If you are going to innovate, you need to break the mirror. Even if you expose the same service to the rest of the organisation, you can change the internal structure to take advantage of the new opportunities.
Open source projects that mirror do not perform as well
This is a surprising conclusion but there are three dynamics
The core-periphery organisation
The core are the team working on the big picture, maintaining coherance: project leaders and core developers, with a periphery of occasional developers.
The periphery are solving specific problems in a very limited scope.
If you see a steady shift from the core to the periphery, your project is in trouble; if you see an oscillation between the core and periphery it indicates a period of innovation. If you see no movement, it’s in a stable state. If there is no influx of would-be core developers, the project has no long-term viability.
Analysing 1200 projects with design system matrices grouped projects into hierarchial, multi-core, or core-periphery model projects. The overwhelming majority or open source projects were core-peripheral or borderline Open Source projects must have smaller cores than proprietary systems in order to reduce the cognitive load for making changes.
Modular technical systems
Mozilla is an example of a project that started as a hierarchial system, showing its closed system roots, and steadily evolved to a core-periphery model over a series of releases. Simlarly Terraform made a concious decision to split out the providers (specific to clouds) from the core, driving a similar change.
Transient groups of problem solvers who are involved for short periods of time
“Temporary organisational ties can quickly created at low cost to support highly interdependent collaboration.” Per Conway “organisational flexibility is important to innovation in design.”
Lessons
- Invest heavily in modularity
- Have good documentation
- Participation is the identified technical architecture, and the result of the organisational structure.
Burning Down the Castle
Daniel Vetter
“This is going to be a personal talk - a story starting with seeing kernel maintainers as my heroes, building this awesome and inspiring operating system; I started hacking in the graphics subsystem, and I got hired then became a kernel maintainer. I started to realise the way kernel maintainers work made me unhappy, and makes people unhappy, and I want to share my story. How I learned how broken it is, and how I don’t think it’s going to get better.”
So what’s broken? Let’s start with the Code of Conflict which asserted that for “the good of the kernel” you should expect to get “shredded” if you want to contribute. People say it’s gotten better, focusing on the violent language, but that ignores the behaviour.
Interlude: Daniel read Why does he do that? Inside the Minds of Angry and Controlling Men by Lundy Bancroft. Daniel found the patterns abusers behave in to keep in power. The core takeaways Daniel has:
- Abuse = power + controlling behaviour.
- Kernel maintainers have huge power. Maintainers have absolute power over their subsystem.
- Abuse isn’t necessarily physical violence. Violence is not necessarily abuse (consider martial arts). And control isn’t necessarily violent: consider someone who controls when you go out, who your friends are, or other patterns in domestic abuse. Maintainers can exert control over what people can and can’t do.
Consider: “only technical topics are in scope”; contributers may only contribute or talk about technical topics, but maintainers will feel free to have the full range of emotions and behaviours - from shouting at people, to demanding constant emotional support, to picking away relentlessly at contributers work. This is consistent with abusive patterns of behaviour. And by ruling discussions about these behviours out of scope, it becomes impossible to address and modify the behaviours and cultures.
It also creates a negative space for leadership, which is devalued by the idea that only technical output matters; after all leadership is far more about people.
By making maintainers themselves wholly responsible for their subsystem outputs, you create a fear in maintainers that if anything goes wrong they will suffer. It’s anxious.
It also leads to cults of personality: people dig into an area, sometimes for decades. Some of them are toxic, but their power is perpetuated by the idea that only they have the knowledged necessary to have that power. Power is not shared: things are not documented, the knowledge is kept in the head of a person. Testing is rejected, never mind more sophisticated concepts like CI/CD are rejected so as to keep the knowledge internalised to select individuals or groups.
Reviews: reviews are hierarchical for the most part, only 25% of reviews are between peers. The rest are from maintainers commenting on contributers, who are effectively subordinate to the maintainer. Reviews can and are used to reinforce power relationships.
Interlude: We can do better; consider edunham’s Life is better with Rust’s community automation and VM Brasseur’s Have it your way: maximising drive-thru contributions, which Daniel summarises as the idea of interactive documentation.
Daniel was maintining a tremendously successful subsystem, in the sense it was growing, but he was becoming uneasy. He didn’t feel the model was the right way to manage people. He went to his first kernel summit, hoping to meet some allies to work on improving aspects of culture. He met people, talking with them. Some people made the right noises, but in reality, their behaviours were as unhelpful as ever.
“Which brings us to the topic of who is not an ally.” There are many people who have been around for a long time who will talk about the problems, but ultimately will be apologists, excusing the way things are as “it is how it is.”
Daniel talked to lost contributers, the people who gave up. He found recurring names showing up as problem people. People who would complain about culture problems that affected them, but were perpetuating them. Some of them have fone completely; some are burned out maintainers. Neither, unfortunately, can help fix the problem.
The graphics community have been trying to improve their own corner of the world; moving to a multi-commiter model, rather than one or two maintainers. They’ve adopted a code of conduct, and so far it works. Something particularly striking is how many toxic people suddenly are able to behave reasonably when it might cost them their place in the community.
Mostly, though, the graphics folks are not wanting to try to tackle the problems of the kernel overall. It’s too big and hard.
Daniel talks to the idea of the sandiwched maintainers; the people who get abuse from on high, as well as dealing with unhappy contributers.
The fact that lkml is a spectator sport leads to it all being treated as a form of “abusive performance art” that reinforces the abusive power the maintainers can wield. If you chose to challenge anything your humiliation will be on the Internet forever, cheered on by the peanut gallery.
People are scarred. People are scared. “I can’t afford to do this, I can’t afford to risk it. What if there’s a regression?” They don’t want the pain.
It can blow out their job security. This is exacerbated by the Linux Foundation; they employ the top maintainers. This creates a cycle of self-interest between those maintainers and the LF, which supports the hierarchy. The risk is that the LF becomes like the old X Foundation, whose conflicts of interest ran it into the ground. Their code decisions became influenced by their commercial interested.
So what’s the status quo? The successful maintainers benefit from the status quo. They do not want change, so what are the levers for change?
- Contributers: do not get stuck. DO NOT GET STUCK. Have your exit plan. No matter how good kernel development is for your career - and it is - one day it will stop being good, and you need an exit plan.
- Maintainers: you could start a revolution and burn it all down, but that is probably not good overall. In fact, maintainers need to effect change before a revolution happens and does significant damage to Linux:
- Drop your powers. Share your commit.
- Document.
- Most importantly, care about people.
BE A STEWARD, NOT A LORD.
Q&A
- How did it get this way? Daniel doesn’t know, but he thinks it may simply be that we didn’t know how to do that 25 years ago, and by the time we did, the behaviour is entrenched.
- Should ex-maintainers get re-involved to make things better? I have a hard time recommending anyone get involved. I don’t think the problem is well understood enough to even start fixing it. Perhaps, like graphics, if you can create a critical group who can defend one another, another subsystem can do what the graphics subsystem has done.