linux.conf.au 2015 Day 1

A shorter hike to LCA this year, being located in Auckland. As always, the jaunt up from Wellington takes less time than the taxi from the airport takes to get into the city; the University of Auckland halls of residence are pleasant, but as always staying in a tall building just feels wrong what with normally living in an earthquake zone.

Cloud Herding: Delivering Services Across Multiple Environments

• Evolutionary patters: New technologies are adopted, proliferate, and then are reined in to solve the problems of ad-hoc development.
• Often those management tools have been proprietary frameworks.
• The cycles are getting faster, which means one-company solutions are harder and harder pressed to keep up with the floods of new tools.
• John is hoping the open sourcing of ManageIQ will be a counter to this.
• Open source is the "Revenge of the Customer", putting customers on a level footing with vendors.
• Interest in hybrid cloud computing: 70% yes, 23% maybe, only 7% no. But that's from, you know, Gartner.
  • Most people are not going to be 100% external for the forseeable future.
  • Private clouds have a common set of problems with external clouds.
  • Not all problems are technical.
• Abstracting the differences between self-service models are important.
• Lifecycle management is a problem.
• A bit too much of a "here is the problem, and we have a product for that" talk. But that may reflect the fact that I've spent a lot of time trying to work through myself.
• Emphasis on this being a proper open source toolset, with commercial being a subset of the open tooling, not an "open core" play.
• "Anand" release in Spetember 2014.
• "Botvinik" is the next major release.
  • Migrating from SOAP to REST.
  • CloudFormations, Heat, Foreman.
• This could have used more demos of what ManageIQ actually does. I know because I've used it, but I suspect a lot of the people in the audience are suffering from a case of "why do I care".

LXD: The Hypervisor that Isn't

• Pron "lex-d".
• Being called a "container hypervisor."
  • Based on LXC, namespaces, cgroups, and AppArmour.
  • Not actually a hypervisor.
  • I remember how much pain this caused people with Solaris Zones.
• REST API (does anyone not have one of these now?)
• It doesn't do:
  • Network management.
  • No storage management.
• "We see Docker, Rocket, and LXD as orthogonal."
  • LXD is trying to give full Linux environments rather than the service provisioning model provided by Docker/Rocket.
• The suggestion that making config items relative to the host rather than absolute is a feature does not convince me.
  • Because "reconfigure when you migrate" is not the answer I was looking for.
• LXD supports "stateful snapshots", serialising the RAM as well as the disk state, which is neat.
• Injection: pushing files in and out of the environment via REST.
• The live migration scheme described sounds the same as, well, every other system available. But over websockets!
• 0.1 released in January as an MVP - container management only.
• 0.2 to be released in February with images, experimental migration.
• 0.3 "Summer 2015". Specification release.
  • Which fucking summer? Summer is now. Learn2hemisphere.

Rocket

Laptop battery fail prevented me from taking comprehensive notes, but I was impressed by Rocket. In particular:

• It's a traditional *ix binary and can be run how you like.
• It doesn't mandate a lot about the host or guests (e.g. it can work with upstart or runit).
• It allows running multiple services inside a container.
• It has a simple container format (an optionally-signed tarball with some JSON).
• It can fetch over HTTPS, will verify containers via PGP if they're signed.
• The container internals are simple.
• It's not an open core - there are no third-party dependencies.

The overall impression I got was that the people working on Rocket had thought carefully about the problems they wanted to solve and had done so. It's working for reals.

On a slightly snarky note, the contrast between LXD and Rocket was very signficant. Rocket works now, while LXD is still basically promiseware.

Containers and PCP

• PCP has been around for 20 years, originally proprietary and now open source. "A hidden gem."
• System level analysis, live and historical, extensible, distributed.
• Focus on performance, can be used as an aide for capacity planning (for example).
• Clean separation between collecting, gathering, and reporting on data.

Metrics

• Every metric is represented and indexed by names (e.g. disk.dev.read).
• pminfo will let you --fetch metrics by name and index.
• There is rich metadata: cumulative vs intant, units (kB, users, etc), data type (int), and a help text.

Kernel Instrumentation for Containers

• Containers are a user-space concept, not a kernel concept.
  • The kernel understands namespaces and cgroups.
  • These are used to create a container.
  • The group accounting metrics are the core of understanding the performance of a container.
  • The common kernel stats will vary inside and outside the container; e.g. cat /proc/net/dev inside a container will show the container-specific information.

Goals

• Allow PCP to focus on specific containers.
• e.g. giving pminfo a --container option so it can understand container vs global numbers.
• Zero installation inside the container: PCP need only be on the host, and it will understand the containers.
• Device automapping.
• Allow for filtering by container to reduce the data load.

Implementation

• A good start. The wire protocol has been extended for the extra container identifiers.
• This code will drop in a few weeks.
• There's still refinement required.

The Challenge of Containerising Your Datacentre

• Application are stateless.
• Deployed in Docker.
• Placed by Mesos and Zookeeper.
• Marathon for keeping daemons running.
• Chronos is a replacement for cron.

Challenges

• Docker security is... not optimal. It's improving, but control your images. Building and running random images will hurt you.
  • Don't run as root inside the container.
  • Keep current on the kernel.
  • Use selinux on the host.
  • Slim images.
• Discovery can be a problem. You have lots of stuff, and it gets moved around at random.
  • Sure, Marathon knows, but you don't want to do a Marathon lookup before every single API call.
  • Marathon can push your marathon state to an HAProxy config.
  • HAProxy runs on every slave, so all lookups are local, and HAProxy to route correctly.
• Docker doesn't believe in logging.
  • rsyslog logs locally, which is really HAProxy, which then forwards it to a central environment.
  • Mount /dev/log into the container, but use systemd for logging, then forward from systemd to rsyslog to HAProxy to aaargh.
• Debugging is painful. You can't easily find your docker image or get into it.
  • Lookup via Marathon, login via docker exec.
  • Most tools can work from the ourside.

Cloud, Containers, and Orchestration Panel

This was an entertaining panel, with members from CoreOS, Red Hat, and Canonical all weighing in.

• Which will be the dominant container technology in 5 years?
  • Answers: more specialisation, more technolgies, not fewer.
  • If I could pick 5 years out in the technology industry, I wouldn't be here, I'd be cleaning up in the sharemarket.
  • "It's Docker's to lose, and they're doing a good job of losing it."
  • "They need to get a sense of what the community needs, not what is easiest to monetise."
• Do we have a chance of standardising on container formats?
  • Standardisation at an early stage is a great way to kill it. Wait three or four years.
  • The kernel presents a de-facto standard, but if use cases vary radically, let them.
  • CoreOS guy says: the image format should be a standard, even if the mechanism isn't. It hurts that there's no way of moving a container image from one place to another. But it's too early for the run-time.
• How do we deal with kernel API skew with the rate of development in the kernel.
  • Containers really only work well with newer kernels, so use them.
  • Furious agreement.
• Kubernetes is under heavy development - can you really rely on it.
  • This is a problem on the OpenShift community team.
  • A common problem to relying on community projects.
  • Some people are using it in production. Depends on how comfortable you are.
  • "We're using swagger to autogenerate API server and clients".
• Big machines - can containers control NUMA and other similar tooling?
  • Not yet, but they shouldn't.
  • You shouldn't docker everything. VMs still provide better mechanisms for some use cases.
• Storage, networking, host migration are painful and containers are pinned to hosts. How do you mitigate that?
  • LXD won't, it will rely on pre-existing projects.
  • CoreOS: networking is evolving. Docker started with the idea of port-based linking. Kubernetes hands out real IPs which makes everything much easier, including migration. Volumes and migration is too hard.
  • "These are the pain points everyone is hitting."
  • My observation: if your app relies on persistant resources, that would tend to indicate it's a bad fit for containers.
• "There are two sorts of container - lxc/LXD-style running a full system from init down, vs containers that are just running a single containerised application."
  • KVM have released the NoVM driver.
  • libvirt-lxc is a different code base to lxc.
  • Containers are a collection of APIs exposed by the kernel. There's no "container" API in the kernel.
• Much of this isn't new technology - why do you think Docker is growing like crazy? Why have containers taken off?
  • The tooling used to be a bunch of obscure little custom scripts. There's now a critical mass of making it easy.
  • Docker have built enough standardisation to make it usable.
• VMWare are moving to make money on management tooling. Will containers go the same way?
  • Canonical: "The magic comes from Juju".
  • Red Hat: Agreed. The money is higher in the stack. "It's going to be interesting times." * * Manfully resisted the temptation to sell ManageIQ.
• What is the future for container formats?
  • It's an ongoing problem. Consider the many OpenSSL binaries running in container images. We should fix that and sign them while we're at it. It's a big problem and a WIP.
  • OpenShift is based around stacks from a base image and that seems like a better solution than Docker's unique images.
  • It's not a perfect approach - it makes it hard to have special snowflake versions of software.
  • "Most people don't seem to care about it, look at mobile phones. It's a real problem."
  • "App stores have an advantage in terms of deploying fixes and banning bad deployments. There's nothing like that in the container world yet."
• What's the value in software defined networking? Why not just put everything in one subnet?
  • You can do your topology in software.
  • Migrations are a good example - move the IP address/range to a new host with zero interruption.
  • My observation:
    • Supports the trend to use network containerisation for simplifying/enhancing security.
    • Supports complex multi-tier network technolgies - red/yellow/green zone topologies.
• "Does the word container mean anything?"
  • It's the distinction between using hardware virt and kernel facilities.
  • Me: virtualisation on one kernel vs virtualisation on many kernels.