LCA 2019 Day 4

The tar syntax has been corrected today: while true; do tar zcf lecture_theatres.gzip /dev/{c1,c2,c3,a1,a2,a3} ; done.

We have the first fail on the Chromebook draw; the second person drawn is here though. There have been 564 tickets sold for the raffle.

Tonight’s Penguin Dinner is at the air force museum at Wigram, which sounds pretty cool.

Keynote: Personal Branding for the Security Concious

Shannon Morse @snubs snubsie.com

Shannon has been working in infosec for 10-12 years. Like many people, she suffers from imposter syndrome; one of the ways she deals with this is to say yes to things that can advance your career and life, all the while surrounding yourself with positive people. Shannon has made 1,500 educational and tutorial videos over the last 10 years; some of her material is used in universities as part of the course material.

Like many people Shannon is self-taught; she doesn’t come from an orthodox tech background: most of her interests as a teen were theatre, although she also learned to build her first computer at eight or nine. While she trained for hospo management at university, she decided she didn’t really want to do that for a career, rather, she wanted to combine her childhood interests: building computers and theatre. Which was the genesis of Hak5: after the launch of Youtube in February 2005, Hak5 in Autumn of 2005 - a true early adopter. She did camera work on Hak5 while working in a bank, and ended up hostin by 2008.

One thing Shannon hated was here early experience of trying to learn, because there were a lot of egos and elitism, particularly in infosec or open source. There was no place for beginners, particularly if those beginners were women. So she decided to provide that resource.

Starting your career

Regardless of what you want to do, the starting points are the same:

• What do companies need? What are they hiring for?
  • Are there certifications?
  • Do you need them?
  • Are there more holders than jhobs, or is it the other way around?
  • Of course, you don’t need certs: Shannon doesn’t have them.
• How has the industry changed over the years?
  • How is the industry changing - infosec is changing and growing.
  • For infosec, there is massive growth.
• What are the diversity numbers?
  • The world is 50% women, yet only 11% of people employed in infosec are women. We need to do better.
  • Shannon hoipes that she will influence women to understand they can do this.
• How can you provide a positive influence?
  • What do you want to learn and get good at?
  • How do you best learn?
    • Certs?
    • Read books? Go to classes? Watch videos?
  • Compile a list of things you care about.

Who influences you? Work out who influences you.

• Follow their blogs, social media, etc.
• Learn from your role models.
• You may end up being a thought leader in their eyes one day!

Don’t just follow people directly in your career path; learn from people in many areas.

Always keep learning:

• Block out time every day to learn something on your list.
• Do it every day until it becomes a habit, a routine.
• Never stop learning.

Building your business

• Your resume should be positive: you didn’t “drop out”, you “went on hiatus”.
• Create a one-pager.
• Networking.
• Business cards are still important.

Build your platform

• Be an expert opinion on the topic.
• What do you want to do with your platform?
• Do speaking gigs, write articles, publish videos, answer forum questions, offer advice.
• Use your platform for goos: offer complimentary sessions for under represented groups or students.
• Don’t undersell yourself: take what you think you’re worth. And double it.

Building your brand

Personal branding leads to:

• Interviews.
• Job advances or positions.
• Partnerships.
• Sponsorships.
• Opportunities!

Develop the building blocks of your brand:

• Think about your values and beliefs.
  • How do you remain consistent with them?
• What do you want to do in the near and far future?
• Surround yourself with people who help with this.
• Live by your vision statement.
• Self-reflect.
  • Showcase your best self.
  • Understand your strengths and weaknesses.
  • Build a “tribe” who share the same values.

This will establish your credibility; demonstrate your value through your actions.

You want to do this, because if you don’t control and manage your personal brand, other people will do it for you. And that can go bad if they don’t like you or what you have to say.

Think about where you draw the line on privacy: your income might be private; Shannon shares her Patreon money, but not her sponsorship income. She uses privacy resources - DeleteMe.com, 2FA, password manager, VPNs, guest vs home networks. KonMarie your digital life! Clean up your old social media so it sparks joy. Be social, but hide sensitive information. And have a plan of action if your security is breached.

Be ready to deal with targeted harrassment:

• When you attract attention, some of it will be bad.
• Keep evidence of any bad interaction.
• Know who to go to - social media reps, police, whoever is appropriate for a particular scenario.
• Remember: people will be jealous because of your success.
  • People will tell you “you don’t belong here!”
  • So what. “My accomplishments speak for themselves.” Nobody can take that away from you.

Don’t job you don’t like - do it because you enjoy it, and work out how to make a living from it.

A Long Day’s Journey Into Backups

Rachel Kelly @wholemilk

This feels like the talk for responsible adults: partly because it’s abnout backsup, and partly because I recognise quite a few people in the audience as such. I may have wandered in here by mistake.

Rachel wants to emphasise this is a descriptive talk, not a prescriptive talk; she really wants to know how the audience does backups - if you use an “enterprise” solution, come talk about it! This talk has a beautiful structure: a three act structure with a glossary in the introduction.

Act 1: Shared Definitions

What are backups?

• A complete copy of the filesystem?
• System and logs?
• A diff of the previous day’s logs?
• The best CYA is?

Rachel: A backup is a lump of data you take in order to restore a previous state.

The primary compliance concern for healthcare in the US, where Rachel works, is HIPPA; note that the P stands for Portability, not Privacy, but does talk about Act 2 has a lot about patient privacy; she also has to attend to PCI-DSS, the GDPR, and HITRUST. They outsource payment processing, so PCI-DSS is not so much of a concern. While GDPR doesn’t directly apply in the US, having customers who are Eurpean citizens requires compliance anyway.

Act 2: Hitor/icity

Try 1

The state of backups in August 2017: they had backups using Duply/Duplicity. Daily diffing, but no full backups had never been taken. Which meant they could not, in fact, restore their backups. Which wasn’t great. But they were running, at least!

Try 2

This was a compliance nightmare! They needed a real solution. So they dropped Duply/Duplicity because they couldn’t get it working. So, in crisis mode, they fell back on full tarballs of the filesystems to S3 buckets. It worked, and they could restore, but it wasn’t terribly reliable, and required a lot of manual intervention, which was not great. Not ideal, but a huge advance!

Try 3

One issue was running out of spool space for the backups on the server disk; so rather than using the local VM disk, they used a shared EFS disk for all the servers, tarring to the EFS system. This was significantly more reliable, albeit at the cost of $5,000 a month! They were entirely reliable, they became part of the server builds, they were able to be restored, and they were in complete compliance.

Try 4: “How to save $30,000 in one year”

Tarball to S3, but properly integrated into the infrastructure and build process. This gave the reliability they required, but at a fraction of the cost of the same storage on EFS. What fraction? Less than a tenth of EFS! While also being reliable, no manual intervention required. Moreover, as part of the review of the backup process, they winnowed out the system parts of the build, limiting themselves to only the unique data.

(Also, catastrophically from a HIPPA point of view, neither EFS storage nor the EFS network transport are encrypted.)

So what does it look like:

• A cron job calls a bash script.
• The back script sets up safeties with set -euo pipefail; note that “u” stops the script if there are uninitialised variables.
• tar with exclusions.
• Send the tarball to S3 with the AWS CLI.
• After 56 days it’s sent to glacier.
• Restore script to get a file.

Act 3: What’s Next

• Make it easier to restore to a new instance.
• Make it easier to restore from anywhere - get any file and restore anywhere.
• “Microservice backups” - we don’t have a solid backup process for these servers is different.
• Better CLI tool.
• A better way of pulling data out of Glacier.
  • There needs to be a retention policy.
  • There needs to be a way of querying and restoring data.

Faucet SND Tutorial

Brad Cowie

Faucet is an OpenFlow based SDN controller that integrates well with a variety of other open source tools such as Prometheus and Grafana; much of the development has been driven from New Zealand.

This session was a facilitated journey through the material at that’s published with the Faucet docs; it was a solid tutorial, but was definitely trying to cover a lot of ground in the time available.

The Tragedy of systemd

Benno Rice

Benno leads in with concepts that have fed into the talk, highlighting Aurynn’s [contempt] culture.

Act 1: The ancestry of systemd

Unix was a happy accident; a coming together of time, people, and place. It emphasised simplicity above all as a reaction against the time of its birth. Even in 1979 the man page for init still referred to typewriters!

Network servers were handled by inetd for many years; there were no daemons as such, rather a superserver that would manage forking server processes to handle each connection. The problem with this was that fork()-ing was expensive, so people started writing standalone services - which init doesn’t really understand. It can’t understand the state of the service.

So now we are conflating many things together: system startup, mounting filesystems, and service management. And doing it badly.

A brief segue into comparison: from day one Windows NT had a very strong notion of a service model from day one, allowing for the service management from day one. Another example is OS X.

Act 2: The idea of systemd

Let’s look at launchd in OS X. It arrived in OS X Tiger, and quickly expanded to take over pretty much every service management task in the system: long running daemons, ad-hoc services, you name it. Lazy loading reduced boot time, resource consumption, and security by running only what is needed when it is needed.

Five years after launchd this guy who no-one’s heard of decided to improve the state of Linux by borrowing a lot of ideas for systemd. His foundational document is “Rethinking PID 1”; a modern operating system lives in a much more dynamic environment - hotplug hardware or virtual resources, amongst other things - and Linux needs to cope with this better.

Many people think of a system as having two things: the userspace and a kernel. But the reality is that it isn’t that simple any more. We no longer build a kernel which is specific to our hardware config, for example. Benno likes to describe a glue later, the system layer: this provides a reliable interface between the userland and the complexity of a modern kernel and hardware (or virtual layer). systemd is one way of doing this, but stops us re-inventing the same things over and over.

Act ?: systemd in Reality

systemd was adopted fairly quickly - most distros adopted it in a couple of years, with Debian and Ubuntu trailing behind a bit. Debian was an exception; it turned into a very bitter conversation. So let’s look at some of the reasons people were unhappy.

• “It violates UNIX philosophy!” As a BSD person, Benno is pretty comfortable with many programs in the same repo.
• “It’s bloated and monolithic.” Not really.
• “It’s buggy!” It’s software. C’mon, you don’t have to be bug free to replace an incumbent. Moreover, systemd has a pretty acceptable failure mode - not taking the system down, for example.
  • Maybe don’t use C.
  • More to the point, distinguish between good ideas and their implementation.
• “I can’t stand Lennart Poettering!” While Benno isn’t defending Lennart’s interaction with the community, he admires his determination.
• “It’s not portable!” Yes, systemd is aggresively Linux-specific. Yes, this is a risk for the other platforms, which Benno as a FreeBSD guy cares about.
  • You know what? UNIX is dead. UNIX was about maintaining perfect compatibility across an absurd range of combiunation of hardware and software. That’s where POSIX came from. These days we’ve got Linux, and some rounding errors. And Benno says that as a FreeBSD guy.
  • Yes, it’s a monoculture. Maybe that’s a negative - but it doesn’t have to be. It can be tremendously freeing for everyone.
  • cgroups are fantastic! There’s a great example of that freedom.
  • User-level units are another thing systemd gets write. Daemons running as a user are trivial, which was not the case, historically.

Act 4: The tragedy of systemd

People have a complicated relationship with change. We love change! Change is awesome! WHen we’re the one who are imposing it on others. We don’t like it when it’s being imposed on use by others.

Just because we’re comfortable with a maze of shell scripts doesn’t make it good: maybe it’s like our worn-out old blanket. Maybe it’s time to accept that change, and it doesn’t help that Lennart appears to have limited ability to sympathise with the people he’s imposing change on.

But you know, it gets worse. No-one needs to be sending death threats over pieces of software. That’s not cool.

Neither’s contempt. The FreeBSD community, for example, treat systemd with contempt - and, as a result is refusing to learn and grow and steal ideas.

If you don’t like systemd, you should understand why it emerged - because that way if you don’t like it, you can write your own. And you’ll need to, by the way. Because the next generation don’t think about things the way previous ones do.

Act 5: The Promise of systemd

• Messaging transports: “I’m not a big fan of dbus, but I am a big fan of messaging.” If Benno were to write one, it would be in the kernel spafe, and have a lot more robustness in auth and security.
  • Then you can have an RPC framwework.
  • You can abstract away whether a given interface is handled by a userspace daemon or a kernel service.
• Service Lifecycle: Windows had this from day one. Unix never really did. You don’t have to install all sorts of bolt-ons.
• Automation via API: once you can setup and configure by API, instead of a gaggle of shell scripts, you’ve got everything an appliance vendor needs out of the box.
• Containers are great: it’s a wonderful way of encapsulating thing.

A system layer is a great way of doing these things!

Consistent naming of devices is great! Logging, auditing, and event handling that are properly structured, and append only, is a great thing.

You now have a new model of an application; instead of having to cram everything into a single binary, you can wrap a group of binaries into a container and manage it is a single unit.

The world is changing around us: we can either try the change, evaluate it calmly and logically, rather than rejecting it blindly.

Q&A

• Could launchd have been ported across? Probably not; launchd is as OS X specific as systemd is Linux specific.
• Aren’t containers DLL hell? No. That’s when applications see too many versions of a library and applications get the wrong one. It’s more like Go static builds.
• Would you be willing to come to the Linux Plumbers’ Conference and run a systemd session? I’ll take that on notice.
• Do you feel there’s been a chilling effect on people putting good ideas forward because of the hostile reaction against systemd? It’s possible; whenever you touch things like this, you’ll get angst. The code is hard, the people management is hard.
• Where’s FreeBSD at with next generation init? There’s OpenRC, but they’re a long way from any idea of a system layer. This is one of the things Benno finds very frustrating - it’s more than just starting processes.
• systemd has taken on a bunch of services; do all of them make sense? Here’s an interesting thought experiment: if systemd had focused less on implementing these things, but having a well-defined API, perhaps they could have relied on a bigger community of projects and people? Maybe they don’t need to be there in the long term.
• What are the advantages of systemd on embedded devices rather than desktop or server cases? If embedded devices have any dynanism you no longer have to write code to manage it.
• How well did this talk go down with the FreeBSD audience? The only real pushback was against the specific tool: the idea is not considered unreasonable.
• What needs to change betweem the broader community and the systemd team to make things go better in the future? Watch Adam Harvey’s talk on major version changes; you have to give people good reasons to want to change, not just beat them around the head.
• Are there any areas systemd go further in? The messaging and RPC should be better thought about and more pervasive. It would allow a lot more interesting control and security features.

Penguin Dinner

A great night at the Wigram Air Force museam - a fantastic venue, a very nice dinner, and of course, great company.