The first day of the Summit proper. I am glad I registered yesterday - there are long queues of desperate people waiting for their tags as the keynote is about to kick off, which is definitely a rookie error.

The Moscone convention centre is vast - it sprawls across multiple city blocks, and the room used for the keynote would comfortably house whole conventions in Australia or New Zealand; 5,000 people didn’t even touch the sides.

In the corporate extravagence stakes there is a harp on the stage, presumably for some sort of performance.

(There is a store to buy corporate merchandise. Who pays for stuff like this?)

Keynote

The keynote does, in fact, open with a small orchestra, playing along to an abstract animation in the background. Very tasteful, and quite clever.

(Why oh why did I get stuck behind the guy with the GoPro on a stick?)

Jim Whitehurst - Red Hat

• “Participation and innovation are tightly linked” and the buzzwords of the conference.
• “The computer paradox” - “We see computers everywhere except in productivity statistics”. This is apparently a long-standing problem of management and economics; after an initial growth spurt we are apparently spending a lot (on computers and programmers) to stand still.
• “Much of that is because technologies have become so complex we can’t solve problems on our own.”
• Jim posits that we are on the cusp of the “fourth industrial revolution” - machine learning, additive manufacturing, and so on - which looks like having an impact comparable to the second industrial revolution - mass manufacturing in the 19th century.
  • from the first industrial revolution onwards we have been doubling of productivity on a regular cadence until the 1980s, when improvements ground to a halt.
  • This was not merely a business and manufacturing revolution; the industrial revolutions have driven massive, broad social changes; the first and second industrial revolutions drove a move from family-owned and operating manufacturing close to homes, to mass employment and workplaces remote from suburbs.
  • Prolitical revolutions accompanying social upheaval: social democracy, communism, etc.
  • Radical new models of how people are organised: systems of management; strong hierarchies, statistical process control, Taylorism (Jim is not a fan).
• These models are changing, especially management of people.
  • But jobs are going away, as more an more can be automated.
  • Remaining jobs are those which require judgement. There is no future for doing a thing over and over again, and there is no future in organisational structures tuned to optimising human repetition.
  • Innovation at pace is not possible when using the mechanisms developed in the earlier industrial revolutions.
  • This is a common problem Jim sees in customers around the world - solving the problems associated with it it is a focus everywhere.
• It’s very hard to break old habits; the things that made people sucessful and celebrated are now hinderances; the control that ensured success now impedes it.
• Organisations must recognise that innovation comes from the edge, close to the customer, not from the boardroom.
• Jim argues that individual organisations are not enough. Success requires diversity: of skills, of backgrounds, and so on. It’s almost impossible to have in one organisation.
• “I’m a history buff. I need to do a history lesson at the summit.”
  • History gives you a frame for learning for today.
  • But there aren’t a lot of great examples from the industrial revolutions, because they were mostly about command and control. If anything, they’re a list of things not to do.
• So shall we pay a game of what-if?
  • Consider Michael Faraday: Victorian scientist. A different background: self-taught, working class. An avid eperimentalist and networker - because Faraday wasn’t that flash at maths (due to his lack of formal education) he collaborated with Maxwell, who was a briliiant mathematician.
  • His showmanship lead to him developing a great following of people interested in magnetism and electricity.
  • He started the Christmas lectures, open to the public, which still run today.
  • Unlike, say, Edison, he gave away his inventions, allowing others to build upon them. Imagine if he’d held onto those inventions and research? Would we be 20, 30, 40, even 50 years behind in everything that requires electricity?
• “Building the capability for communities to innovate beyond the sum of their individual members is the leadership challenge of our time.”
• “The leadership challenge is to get people to create things that wouldn’t exist as a result of people working on their own.”
  • Contrasted with traditional industrial processes which is about making more of the same thing, not new things.
• Consider the current state of healthcare: healthcare is something done to people, not with people.
  • What if doctors assumed collaboration with patients was the norm?
  • What if research and health products shared as a norn, not siloed away?
  • Pharma exec: “we know we have data from studies locked away, that could be useful to others, but we don’t have a way to sharethem .”
  • Contrast that with Google and Facebook sharing some parts of their technology stacks and innovation to improve the commons.
• “Are we a participant or a leader?”; Jim suggests Red Hat aspire to be neither: “We’re a catalyst.” Creating and driving change, without using a traditional top-down leadership/management role.

Elwin Loomis - Target

• “What about hacking culture and human systems?”
• Buzzword bingo is in full flight - “Lean in! Be sustainable! Disrupt! 10x! Rare unicorn!” The presentation was worth persisting with, but I almost zoned out at this point.
• Infrastructure used to be the barrier to entry: massive investments in physical and financial infrastructure. In retail, for example, you had to have a supply chain, infrastructure, and so on.
  • Those barriers to entry have been worn away. You can’t rely on them easily.
• “10x developers are rare, but you can make 10x teams fairly easily.” (I would add that one of the challenged of the cult of the 10x developer is that it acts as an enabler to behaviours that are net losses to teams.)
• Big companies are great at building tactical systems, and care and feeding of legacy systems.
• “I hate the word ’transformation’.” Me too, Elwin. Not least dure to bitter experience.
• “Target wants to thrive in markets that are faster and smarter than we are. This is a challenge.”
• Elwin has spent the last three years working on this problem for Target.
  • What does talent want? To work for a cause, beyond “make more money”; to work with talented people, to solve problems.
  • To be part of a company with a cuture that knows how to look after people.
• Target from 1993 to 2005:
  • A small cog is a broader conglomorate; the suits and t-shirts division.
  • Test and Learn Culture.
  • Guest and neighbour focused.
  • Gave 5% back to the community.
  • Elwin recounted sitting on one side of a table with “the suits” from head office and thinking “I’m going to eat your lunch.*
• Target in 2013:
  • Everything had changed.
  • The group owning Target had come to focus on Target, to adopt Target’s branding as the group identity.
  • Unfortunately with that focus and success came sclerosis. The group did not become more like Target, but rather forced target to become like the group.
  • Definers, not doers.
  • Suits everywhere.
  • Monolithic culture.
  • After stints at various start-ups, Elwin returned to Target and found himself on the suit side of the table, looking at guys in jeans and t-shirts, who he could see thinking, “I’m going to eat your lunch.” It wasn’t a great moment.
• What prevents change? Fear of the unknown. The innovators dilemma.
• How to change back to a healthy place?
  • Create a place people want to work.
  • “We create the machine, we are not cogs in the machine.”
  • Don’t build a code factory.
  • Find the doers. Join forces. Show me, don’t tell me.
  • People may not work for you, but they must want to work with you.
  • “I minted a coin. I created an alliance. Everyone gets a coin and can give a coin. But there are rules about who can get one. You must be a doer.”
  • No-one kows who all the Target Alliance members are, because one of the rights of Alliance members is to give coins out to create new alliance members.
  • Small teams, small pods. Say to the business “we can do your ideas. But you have to come with us.”
  • “Create a model that infects the orgaisation.”
  • “Be the tip of the spear.”

Secure Your Enterprise Software Supply Chain with Containers

• Zohaib Kahn (Red Hat)
• Randy Kilmon (Black Duck Software)
• Scott McCarty (Red Hat)
• Curtis Yanko (Sonatype)

Adoption and Concerns

• 30 - 40 % of Black Duck Customers are using containers in production; lifecycle management and security are the main sources of anxiety.
• Sonatype see the same anxienty: fear that developers can push to production with little governance/control.
• Red Hat see that security teams being anxious can shutdown initiatives if the aren’t brought onboard.
  • People are blasé about what’s in the container, which RH insist is wrong.
  • In fact, th quality of the kernel (in the container host) and the quality of the container components matter.

Security

What are the main security concerns?

• The quality of the contents of containers are the same whether they’re in the containter or not. A faulty OpenSSL library is a problem no matter where it is.
• There are a number of ways that an attacker can ruin your day:
  • How easy is it to pivot from an in-container exploit to a system-wide exploit? There’s a common kernel, guarded by the namespaces.
  • Exploits of the namespace isolation (network, filesystem, etc) in the kernel.
  • Kernel quality is critical.
  • Don’t make it easy for the container root to access the global root context.
  • As well, there can be exploits in the management tools like the Docker daemon.
• Many of the problems are just the same old same old. Keeping current versions of software, monitoring for exploits, and so on.
• Velocity and volume of change is the critical difference; the surface area increases dramatically as a function of the number of containers, which is a function of the number of applications and services. You need to bake security concerns into the lifecycle from day one: rolling forwards and backwards at scale.
• Think of it as three layers: Core (platform - kernetes, OpenShift, kernel), middleware (SpringBoot, PHP, etc), application (your code).

How do you get it right from day one?

• Treat it as a supply chain that can create repeatable, reliable results.
• Start with a secure/trusted source. Of course Redhat think that’s them. But they’re right that pulling random continers off the Internet is… profoundly unwise.

Why do we be so vigilant, given that containers are short lived?

• While the containers themsleves are transient, that’s not what lifecycle is about.
• The base image(s) containers are built for define the exposure, and the image management is your lifecycle.
• The need to be vigilant around patching and configuration amangement hasn’t changed.
• In fact there are fewer excuses for keeping current given how easy containers make migration through a lifecycle (when done properly).

How do you do CVE patching with containers?

• Red Hat’s answer is with a CI/CD pipeline plugged into OpenShift.

Which components should be tracked?

• “When Agile kicked off we borrowed a lot from Lean and Toyota, but we ignored their supply chain management.”
• Start with the no-known-problem components, and then monitor that they’re unchanged, or that the changes are understood.
• Expose the lifecycle data.: provenance is crticial, tying your known environments into CVE alerts, for example, to understand the ongoing picture is critical.
• “Trust is temporal.” What is trustworthy today is not trustworthy tomorrow.
• Use signing at each layer. Use signing checks to block unsigned releases from going live. Red Hat and CoreOS each have some solutions in their tooling, and are working with the OCI to make them common.

Quality and Agility

How do you manage quality at scale?

• Operations still fundamentally manage that core build for containers, which is the base for everyone.
• That same principle applies to the middleware.
• Encourage commonaiity, and enourage it by allowing specalists to focus on the supply chain segments.

CI/CD Pipeline

• Add a security scan to your CI/CD pipelie. Make security scans part of your build cycle, just like you do with your testing, for example.
• As with testing, it’s cheaper and faster to deal with this up-front than just before release.
• Sonatype claim that proper up-front scanning (at build time, for example) results in a 29% reduction in unscheduled work.
• “Code ages like milk, not like wine.”

Tools

Which tools can improve the management and monintoring?

• Tooling needs to be container aware - container introspectors, tools that can run in the global context und understand containers, for example.

Which tools can accelerate recovery of defects?

• Black Duck and Sonartype agree you should definitely buy their tools.
• Department of Homeland Security anecdote: Heartbleed was handled by emailing everyone and asking “who is vulnerable to Heartbleed”. This is not optimal, but surprisingly common. People do not have a centralised view of what’s installed where.

Who has authority to pull in components in?

• Today? Everyone can!
• Have a centralised checkpoint to make sure that what is being pulled in is up to scratch.
• Encourage developers to pull via a central local repo, not random repos from the Internet.
  • The suggested committee of Enterprise Architects, Security, and Legal does not sound like a way of building a workplace people want to work in, I have to say, although that was clarified as being less awful than it sounds in Q&A.

General Q&A

• Are people mostly building their own containers, or downloading things from the Internet? People may start with full bespoke images, then move to an approach of starting with a secure base and customise from there. “You can’t solve all the problems for all the users.”
• How do you avoid governance being a boat anchor? Governance feeds a CI/CD process, which implements governance automatically. New things go into the pipeline for validation so you aren’t asking for permission the whole time. Governance is not rounds of picking over every container spec, but rather setting principles to adhere to.

Microservices with OpenShift: Experience from the fields

Sébastian Pasche, Security Architect at Leshop.

“We have to understand that we do not understand everything” - this is what worked for Sébastian and the team at LeShop, not necessarily what will work for you.

Who is LeShop?

• Owned by the biggest retailer in Switzerland, part of a multi-industry conglomerate; LeShop is operationally independant.
• 10-12% of all business (B2C and B2B) is done online in Switzerland.
• Customers want the ease of use of Amazon, but with local knowledge.
• 700,000 orders per annum, comprising 45 million items per year, packed in 2.9 million boxes per year.
• A 2 square kilometre warehouse; a ten minute interruption in the supply chain will cause traffic across the city to gridlock, due to the volume of inbound trucks to the LeShop warehouses.

Why Microservices?

• 15 years of software, with one big deployable: Internal ERP, supply chain management, warehouse management.
• This was all new at the time, and couldn’t be bought off the shelf 15 years ago.
• Code started on Java 1.0.
• 1.5 GB EAR file produced from 25 million lines of code.
• The first challenge: all IT must move to 2 group-managed datacentres, setup active-active, with seamless failover between the datacentres.
• Releases take 5 days. The goal the team was trying to reach was moving to to releases every hour.
• At the same time the group demanded an increase security levels to the same as the strictest company in the group - a private bank. No more than two days from a CVE announcement to fixes in production.
• And do it with no extra staff.

How?

People

• People react badly to violent change.
• Don’t fire everyone and start again, people’s knowledge is valuable. Sell people on the advantages: self-healing means no more callouts, for example. Then change is good for the team.
• Cross-functional teams are critical. Break down the walls.
• One change at a time, and give people time to adapt.
• No change should take more than 2 minutes to explain.
• Avoid the 30 minute barrier: if a proposed change (which takes two minutes to explain) takes more than 30 minutes to understand, split it up into a set of smaller changes.

Design

• Avoid overengineering.
• Avoid large, inflexible frameworks - consider lightweight alternatives like Play, for example.
• Prefer open, well-maintaied frameworks. Open frameworks have lower risks of rotting.
• “Workaround” is not a new dev methodology: fix problems or change frameworks, don’t hack around them.
  • Contribute fixes upstream so you don’t have to keep maintaining them.
• Sébastian has developed a technique for finding poorly understood code: if a bit of code seems ugly or smells wrong, try to find it with a Google search (as a StackOverflow example, for instance), throw it out.
• Don’t ignore OWasp.
• 1 service, 1 business rule.
• Services are responsible for their own data.
• Data processing implies schema and schma implies storage selection: don’t try collapsing everything down to One True Database.

Develop

• Every request is authenticated - it makes debugging easier when you can trace a chain of authentication.
• Applications must be stateless and scalable.
• No client-service or service-service sessions.
• Independent and idempotent messages, and take advantage of functional techniques like the idea of pure functions.
• No local logs.
• Split business and technical logs for ease of analysis re-use later on.
• HTTPS only.
• Speak only one payload language (JSON for LeShop).
• Input, Outpust, and DB access must be throttled and measurable to create a consistent experience; consistency os more important than top speed.

Runtime

• Not everything belongs in OpenShift.
• OpenShift 2 with 2 clusters of 25 nodes.
• 100 microservices in production.
• Multiple DBs for different tasks; Riak and others.
• Main rule: III (Independent Immutable Idempotent).
• Always force full updates with every release. No in-place editing of services.
• Create a single JAR file per service.
• Use HTTP caching to take advantage of the idempotency of services.
• Load balancing: no SPOFs.
• Load balancerss should be as close as possible to the client.
• Use internal load balancers between components.
• Job management:
  • Use reactive programming, not batch programming.
  • Jobs must be able to account for split brain situations.
  • Mesos and Chronos were too much overhead for a small team.
  • Each job is a microservice.
  • The Quartz is used for scheduling.
  • A common DB is used to register the state: which job is the job, for example.
• Monitoring - focus on business functionality.
  • Re-use a subset of your acceptance tasks/tests. These should have captured all your business success criteria. Why bother re-developing production monitoring tests from scratch?
  • I really like this idea - very simple and powerful.
  • Results are stored in the distributed databases.

Authentication

• JSON Web Token (Pron “JOT”).
• Contains user tole & auth.
• Non-cookie base Token.
• When the auth expires, the user re-auths.

Wrap

As well as the sessions, I had a bunch of meetings with Red Hat folks relevant to my employer’s interests (and so not documented here). Overall I was impressed by my first day: the quality and detail of presentations was higher than I’m used to at single-vendor conferences.