Continuing on from the theme of “visiting parts of Australia I probably wouldn’t otherwise see”, this year LCA has come to Hobart the second time Hobart has hosted an LCA.
Opening Reception
Chris Neugebauer
Moving away from the generally fairly boozy Professional networking sessions of most years, the conference opens by revisiting Canberra’s effort to have a more civilised breakfast session, which seems like a good idea.
Breakfast was well-catered and delicious. Bonus points to Mark for his HPE sponsors speech, was was flattering to the audeience and, more importantly, brief.
In stark contrast to Kiwicon the opening music for the conference introduction kicked off with C&C Music Factory and continued in the same vein while people came in.
Some reminders on the “fill from the centre out” was requried on the seating. Such is conference life.
Chris opened by acknowledging the traditional custodians of the land.
Chris noted buses will be going between the university accomodation and the CBD in the evenings, to make it easier to enjoy Hobart and get home, which is a splendid idea - it isn’t always easy to have a look about the cities at a conference, so this is a great idea.
Karen Sandler came up to speak briefly about the work Outreachy is doing to try and improve the field for women and nonbinary people; Karen described entering the field as someone who believed that “women just needed to be awesome enough and then their would be no problems” which she now sees as being rather naive; she provided the example of code contributions: when women identify as men, or submit code anonymously, their code is avaluated more positively than when submitted as a women.
Outreachy is trying to change that via mechanisms such as paid FOSS internships, which are generally producing good results in terms of former interns staying in free software, coding, speaking, or otherwise participating.
Chris explained that, dovetailing with the theme of The Future of Open Source, this year’s raffle will be aimed at providing an Outreachy sponsorship.
There were 421 talk proposals, with 80 selected. That’s a record - almost one proposal per attendee, and over 100 more than the next highest year.
Human-Driven Development
Aurynn Shaw @aurynn
- Started out as a dev, but has moved into the devops world a few years back.
- Devops is more important shift than headlines about automation and working together.
Tradtionally there’s a big split between ops and dev. This is a result of people’s priorities:
- Ops care about stability and availability.
- Dev cares about shipping new things.
- This has one of two failure modes: “ops says no”, or “yolo into production.”
Auryn spoke about her introduction to devops at Weta Digital, where she first encountered Puppet, throwaway infrastructure. This was all good, but it lead to her first realisation: devops is about helping people: helping people to deploy, test, support better.
The more Auryn learned about builduing better experiences for users, the more it became clear that one of the biggest underlying problems was that tech has a culture of exclusion, often rooted in an unhappy childhood; computers became a place to feel control and power over a life. The problem is that power is addictive; we want to keep it, so we make it exclusive.
This is reflected in our language and anecdotes: people are wizards, who perform magic (or, for sysadmins, black magic). We’ve evolved the language: rockstars, ninjas, 10x engineers. But exclusion is still the oucome (and, arguably aim).
Developer culture encourages us to look down on ops people. Developers are the elite; ops teams are the grunts. This is part of a culture where status is gained by demonstrating contempt for others; “you used what language?” “My OS is better.”
Newcomers want that power - so you end up with a community of “that guy”; who is that guy? Someone whose poor behaviours are enabled by an unwillingness to call him out because of supposed technical skills.
We all know “that guy”; you may be doing it now. You could go and ask them for guidance and help. Or not - because it’s a relief not to.
Auryn has a confession: she has been “that guy”; being rude about languages, text editors, you name it. Because we want to belong; because she wanted to belong, and this is how you belong.
This behaviour wrecks teams. People drift away from communities, quit their jobs; but we don’t challenge “that guy” because we don’t want to risk losing their skills. But we lose their skills anyway - because people are afraid to ask foir help, because people leave; they’re a net loss for the community.
It’s hard to unlearn this behaviour: to learn to listen; to adopt an attitude of service and helpfulness. To be interested in helping people achieve their goals.
Agile is supposed to help with these things: we’re supposed to be focused on the user story. But it doesn’t work out that way; because dominion-focused thinking - thinking that says “I know enough” - we don’t always listen.
Auryn offers some examples: when Apple shipped Applea Health, it has no facilities of rtracking reproductive health issues. Google,s image tagging didn’t work with black people. There are countless examples: and they all come back to not engaging outside the group to understand what people need. “We know enough.”
(This sounds like everything wrong with GNOME 3 to me…)
Aurynn mentioned Etsy’s Kiwicon talk about disassembling contempt culture (which was an awesome talk).
Accepting that we do this is hard; we don’t want to admit that even to ourselves. But we have to ask these hard questions; we need to be honest about not knowing everything, about being willing to ask questions and be vulnerable in our lack of knowledge, because that’s the only way we can learn about hat other people need and want.
- Ask questions.
- Hire for diversity, so you can broaden your knowledge base.
- Listen to what people tell you, whether they’re your colleagues, your users.
This is human-driven development. And it’s important because we’re building the future. And if we keep being that guy, we will be treated as damage and routed around; people will not want to be our users, if we treat our users with contempt.
And this is DevOps: building bridges, building a service culture. By asking who we missed, and how we can help. That is also human-driven development: by moving away from a culture of dominion to a culture of service.
The Sound of Silencing
Julien Goodwin
- Level 0: No silencing.
- Humans learn to ignore events because they expect to see alerts because of maintenance events. “I expect to see an alert and for it to be safe to ignore.”
- This inevitably leads to people learning to ignore alerts they should pay attention to.
- At scale it is impossible for humans to know all the alerts they should ignore at scale.
- Level 1: Supress all alerts.
- Maybe OK if you are a single-site alert.
- But probably not even then.
- Level 2: Silence a single site.
- Works better - but if the SAN fails while you’re working on a router, wouldn’t you like to know?
- Level 3: Turn of groups of alerts.
- Works much better.
- But only when people remember to switch off the alerts, and get the scope right.
- Level 4: Change management integration.
- Link the silence generator to the change management system.
- This makes it more reliable.
- But the scope of change record may not marry up with your supression.
- Level 5: Inhibiting Alerts
- Use service level indications to mask expect failures: if a router is down, don’t alert on the servers behind it.
- Fire “go nowehere”.
- The failure mode is over-silencing.
- Level 6: Global monitoring and preventing over-silencing
- Monitor global live capacity, alert if too much as down, regardless of what the silencing is down.
- Page on over-aggresive silencing.
Getting there from here is pretty hard.
- Incrementally - don’t try to change the world.
- If you see something, fix something. If an alert was rubbish, fix it.
Q&A
- What’s your favourite system? If I were using a FOSS system, I’d use Prometheus because it’s like Borg.
Running Production Workloads in a Programmable Infrastructure
Alejandro Tesch
The tooling is an OpenStack centric approach. It’s live demos all the way down.
Orchestration in a Two-Tier Environment
- Scripted calls out to Heat to automate the environment instantiation.
- Load balancer + Tomcat + Oracle.
- Heat will create e.g. the Cinder volume.
The live demos are not kind.
LBaaS
OpenStack’s monitoring can trigger auto-scale load balancers via Heat.
FWaaS
IPtables on the Neutron routers. May not be considered a replacement for an “enterprise firewall”.
Available as an OpenStack plug-in.
Data Cloning as a service
Uses Cinder snapshots to move data.
The DB has to be in backup mode, and there are no smarts around transforming the data for e.g. data sanitisation.
Managing Performance Parameters through systemd
Sander van Vugt
“In the begnning I considered [systemd] quite shocking, but I dove into it and now I think I understand it.'
“I’m going to focus on CPU shares.” But the principles are the same.
“You’ll be able to achieve performance control comparable to containers or even VMs.”
Demo is CentOS 7.3.
- Three slices: system slices, machines slices, and user slices.
- The unit file is the systemd building block.
- The oneshot type can let you run any command once during startup.
- For the demo there’s a “stress1.unit”.
- Using the CPUShares=1024 to weight the stress job.
- In and of itself it’s meaningless. It’s relative to the other shares on the system. Work is weighted based on the sum of shares in the system.
- The numbers are irrelevant, the elative amounts are important.
- CPUAccounting is on. This is a pre-req for using CPU weighting.
- User slice: for users; the system slice is for the OS; the machine slice is reserved for VMs.
- Each slice defaults to 1024 shares.
- The demo is two system jobs, and one user job (started from the user shell).
- You might expect that the split would be half to the user job, half to the system jobs to share between them.
- This isn’t what happens.
- Why not?
- This is because it only kicks in when you’re constrained.
- So if we switch off a CPU by
/sys/devices/system/cpu/$cpunum/onlinesetting to 0 we can force the contraints to kick in. - But the user job is getting 66% of the CPU time. Why is that?
- Because of the system vs user vs machine weightings. These can be reconfigured from the global systemd config in
/usr/lib/systemdby tweaking the system, machine, and user slice weightings to prevent a user logging into a system and using up to 50% of the CPU. - There are handy analytic tools to help you see what is going on with the policies.
systemd-cgtopwill show you the service and slice heirarchy on a top-style interactive dispaly.cglswill show you the scopes that everythibng runs in.systemctl show $SERVICEwill show you all the parameters set for a given service, including the performance parameters, including the other performance parameters liike memory limits, IO limits, OOM adjustments, for example.man systemd.resource-controlhas a really good level of detail around what is available.- Memory limits are pretty dangerous if they aren’t set correctly. You can stop things even being able to start.
- BlockIO is very flexible and powerful.
“I like it that everything ias in the unit file - it was a lot more complicated to use cgroups in pre-systemd Linux.”
This was an excellent talk full of knowledge bombs - what I think of as the best sort of Sysasmin Miniconf talk with solid examples of things you can do.
Q&A
- Do sub-processes inherit their parent settings? Yes, they do.
- Are the numbers in
systemctl showthe defaults? Yes. - Can kernel processes be allocated into cgroups? That is an interesting question, but I don’t know. If you have experience in this, please share.
- When running
tophow could you identify which slice they were coming from? Well,topdoesn’t tell you; you needcglsandcgtop.
Mamas Don’t Let Your Babies Grow Up to Be Rock Star Developers
Rikki Endsley
“I like older music because I’m older; I’ve been doing this for a while.” “I have been working in behind the scenes.” Rikki has been working in tech journalism, community management, and community evangelism and produced a thesis around diversity in tech.
Twenty years into her career: technology has changed a lot. But diversity hasn’t. People are still recruiting for “rock stars.” Many women - and many men - simply won’t apply for a role with that demand.
An inspiration for this talk was a Pycon keynote by Jacob Kaplan-Moss in 2015; he talks about how he gets credit he doesn’t deserve and treated like a rockstar, which he hates. Rikki recommends you watch it.
- Calling people rockstars demotivates the whole team - and these things are a team sport. Teams are bigger than one programmer, and they’re bigger than the programming team.
- Worse yet, the programmer might believe it.
- Good programmers solve problems. Good teams solve problems.
Rock stars: coming to work drunk, weird hours and contractual demands, biting the heads off bats. Maybe you should be a little more specific about what you actually want people to do.
Rikki offers, instead, a description of Willie Nelson: poetic and logical, and talks about his history: a songwriter for others, a DJ, a collaborator; he’s been on TV, movies, practise martial arts, tried many different genres of music.
Don’t we need more Willie Nelson developers, asks Rikki?
He’s also accessible: Rikki’s seen him play county fairs, barns with dirt floors, through to high-end theatres that sell for $200 a set. Developers should be able to work in many different contexts.
Developers should use many different tools: he uses an old $50 classical guitar, which isn’t the norm for country music.
Willie Nelson also collaborates with a wide range of people: other country artists, rappers, reggae artists. He co-invented outlaw country as a result of hanging out with a different set of people around that time.
Q&A
- Can I do a search and replace on the presentation with Johnny Cash? Sure, great idea.
- I’ve hired a few rockstars, people who create code only they can fix. Do you have any ideas about what to do with these people? There’s a whole talk in that! One thing is making sure that people who aren’t self-promoters, which rockstars usually are, get recognition for the work they do and contributions they make.
- It sounds like you think this is a personality type? I don’t think that’s so. People may be trained into it, be encouraged into it by the context.
- Follow-up: so how do we encourage the Willies and not the Van Halens? I believe recognition is critical. Help people learn to stand up for and promote themselves; promote their work.
- How would you help rockstars into rehab? What are the positive aspects they can give us? They can be strong leaders; but you need to learn to lead from behind.
- Sometimes the problem is that management have brought into the rockstar and support them personally - how do you help people with that? Had a boss suffering from unconcious sexism, to the extend of assigning credit for her work to a male colleague. In this context she had to sit him down and talk him through the problem.