Kawaiicon 2026 Day 2

Hello Day 2!

Red Teaming is dead. Long live Red Teaming.

notnotcharlie & Tim Goddard (pruby)

Leaning into an RPG theme, complete with elaborate outfits, Tim introduces himself as the head of a roaming band of adventurers who audit your spells. notnotcharlie is a sorcerer at GitLab’s red team. They test wards (detections), seals (controls), and response rituals (processes).

Tim notes that external engagements generally proceed the same way, but asks what they learn: after all, they typically demonstrate a path, usually a well-known one that exists in many places. What are the alternatives? What could have prevented it? What is the actual desired outcome? Notnotcharlie contrasts with the internal team: they have access to the whole company rather than a narrow security engagement; they are focused on how the abstract stories in the press apply to the actual internal landscape. People inside the org are often surprised that they can be exploited.

For a red team report to have value, you need to understand who will be reading it, what they will take away from it, and what they might do with it.

Sometimes the red teams are just exploring well-trodden territory: AD controllers for example, or network security. The thing is that the blue teams know that is well - the problem is usually consistency across their environment. Is spending a lot of time on the well-trodden path as usual as the less well-understood parts of an environment? notnotcharlie argues that well-understood parts (think: unit tests) can be put together in ways that generate novel problems (think: integration tests). She notes that spending time on well-secured components is not a good use of time.

Tim mentions that testers often have more access than a typical red team: they often have access to monitoring tools, domain admin access, and so on, so they are effectively a purple team. Proving that a client has great perimeter security is great, but if the tester shows they can’t break that perimeter, the client may not learn about vulnerabilities that exist past that perimeter - like weak dev workstations.

notnotcharlie wants incidents: she will ask for internal volunteers to be victims, for example, in order to generate one. She’s interested in testing incident response, while having eyes on the same response - this will escalate to emulating advance persistent threat actors to understand how the company can respond to real-world threats. Pruby on the other hand starts noisy and ramps down to quiet. Notnotcharlie notes that her team will go so far as to work with defanged malware to simulate real problems.

“All of our reports are immediately released to the whole company, as soon as problems are fixed” - notnotcharlie emphasises the role of good storytelling in order to get the most value out of their findings. Both notnotcharlie and Tim emphasise the importance of being able to offer useful advice to resolve the problems found - breaking things without having fixes is much less useful.

Notnotcharlie notes that the most heartening outcomes are when people whose job doesn’t have a security title start offering up findings of their own.

Red teaming is often about delivering honesty, even if it’s uncomfortable for the reader of the report to hear. That doesn’t mean being unpleasant about it, but red teams need to be prepared to challenge assumptions.

DIY Sovereign Cloud or: How I Learned to Stop Using Hyperscalers


Shlee

The subtitle of AWSome or AWSful is fantastic.

What is the cloud? Making someone else responsible for the pain of capital spend, sizing, keeping ahead of growth, you name it. AWS offers you a bit of basically everything. But it will cost you - but some of that cost even covers some of you business risk. But those costs are real; Shlee runs community services, and had been paying about A$400-500 cost to host those services on cloud providers. And then the prices went up. And up. And up.

“The internet’s price model is betrayal.”

Now he pays A$750 for a half rack and 40 TB of network traffic. The Linode equivalent would be AU$7,500 per month.

Can you compete with AWS? No. Be realistic, If you need all of AWS’ services, you’re gonna have a bad time. But if all you want to do is colo a VM host - a Linode competitor - that’s pretty easy. You need some tin, some switching, some storage.

“Auctions are your friend for grabbing cheap enterprise hardware if you’re lucky.” Storage servers are both hard and expensive, but also give you a much better experience when you can decouple your computer from your storage.

I have to admit I nipped out early - not because this is a bad talk, it’s quite realistic about what you can achieve self-hosting, what you might need self-hosting, but it’s ground I’ve tread myself.

Building Leviathans


Robbie Litchfield


It is not really possible to accurately render this talk through the mere medium of notes; a man dressed as a devil presents to us videos of industrial scale mining and smelting, for example, is not something that be reasonably described in a simple blog post.

“Every Leviathan requires five contracts to summon - extraction, refinement, design, fabrication, and shipping.” Robbie’s summary of the process of industrial-scale mining, smelting, and treatment is a delight illustrating both the complexity and scale of how we get to silicon. “The goal for semiconductor manufacturing is roughly 12 nines of purity.” Even this isn’t enough for modern processors though - the alignment of the crystals needs to be consistent in direction and grain. Crystals are broken down to liquid and grown into ingots with crystals that are artificially consistent. The ingots are then sliced into wafers and polished.

Modern design abstracts us from the old days of wiring together transistors; we specify in code to build libraries of functionality that are assembled to etchings by our tools; even this isn’t really how things work. We rely on licensed components, like ARM cores or DSPs. “If you’ve ever used npm you understand how chip design works. Supply chain attacks are coming for your processors!”

It costs billions and trillions to play in this world. The foundery is the most sacred of the modern cathedrals; even a stray hair can destroy everything. The clerics of this cathedral cause them to “thrum with the sounds of the clergy”. “It can take five to ten years for a new process to go from a 10% yield to a 50% yield. If you’re lucky”.

This produces the mask, like an old negative print process, through which the light etches a circuit on a wafer. The lithography machine is so complicated that no one nation can create one; every part is ultimately consumed by the energies involved in a vacuum more empty that of space. “It is a conspiracy to draw ever smaller pictures onto rocks.”

Even QA is almost impossible in a conventional sense; you cannot reach in from the outside once the circuit has been etched. Instead, the test circuits are baked into the wafers themselves. Binning weeds out the chips that are not up to the top spec, and then they are packaged in glass and ceramic to protect them.

So how do you get access to these chips? Most of us can’t afford to build our own factories? Well, that’s where the middlemen come in: an engineer can open Digikey and access the supply chain behind that. That’s the white market. But it can’t satisfy all your needs. After all it might not be a processor in manufacture any more for example. Grey markets exist - sellers claiming to have recovered them from recycling, say. There’s also the black market, but the devil advises that you do not dabble here.

There’s also the restricted white market - dual use technologies that may not be exported, or may be under tight monitoring from governments, for example. And that restricted market in particular will be very interested in your reputation. If you don’t stack up, you may find that suppliers are uninterested in speaking with you, let alone selling to you.

“I would advise against violating export contracts. It ends poorly.”

The supply chain for processors is arguably one of the most sophisticated examples of human co-operation in history, but everyone has access to it. We are all competing. Lead times are now as high as 52 weeks for any item in even moderate volume.

Supply chains are in fact so constrained we may have to change our approach to how we work. For example: should people be able to use their own devices? Well, we may not be able to even offer people a decent work device. Do you want them to be productive? You may have to flex on the “no BYOD” rule. You could upgrade devices instead of replace.

This was a fantastic talk. Very well-done.

How Is Red Team Infra Formed


Adam 'feabell' Bell

Adam is part of the infrastructure management team with CyberCX, as part of the “Adversary Simulation Group”, and that infra management is what he’ll be talking about. Also he pronounces the word “router” the English way which delights me. The “wake up in a cold sweat” nightmare for a red team is the “blue team post” - a post from the defender who has not only uncovered you, but broken your tools as part of their response, and can now everyone how you work.

The team need two key items: an HTTP endpoint that can talk to a C2 (command and control) environment. Both need to be accessible to an operator to manage them. Behind that is complexity: for example, a jump host; build servers that can build out the rest of the infrastructure, and so on. Adam doesn’t consider the pain of providing SMTP as worth it; sign up for a service if it’s needed. Nor is there any backhaul other than HTTP.

“The downside of doing things on the Internet is that the Internet can see you”: certificate transparency works both ways, for example. The mitigation for this is to add upstream redirectors. Adam doesn’t provide those - the operators stand them up as part of the engagement on non-CyberCX infra. The CyberCX ingress point responds only to that additional tier of redirectors.

The other trick is to be lost in the noise: you can’t be invisible, so serve normal traffic. If you look like a blog as well as a C2 endpoint, you look a lot less suspicious. Also don’t hit non-targets. Serve benign payloads to people who aren’t on your target list. Red teaming an Australian company? Don’t serve attack payloads to a request from Singapore.

Trust is important: even if you trust people, we make mistakes. We open ports we shouldn’t for a quick test. We’re running attacks, with permission, sure, but attacks all the same. The blue team will hit back. For the first it sounds mean to say act as though we don’t trust people, but it’s a requirement. We can have additional controls - if someone is trying to do things that we don’t expect inside the network, it doesn’t really matter if it’s a blue team or one of us, we ought to block downloads, exfil, all the controls.

The final note is the Flamingo Up. Like a fuck up, but worse: a fuck up is when your attack on a client gets exposed. A flamingo up is when all your campaigns get exposed. You need to compartmentalise all your campaigns. Each one should be torn down at the end of an engagement. There mustn’t be any reuse of components between campaigns, nor of the same infrastructure for multiple campaigns. You also need a kill switch: if something seems to be off, you need to be able to tear down everything in the blink of an eye.

“Infra is hard.”

Finding vibe leaked API keys every day


"Alex"

“This talk is a work in progress.”

“The year is 2023. We were all doing Boomer coding. Then linear algebra happened and we can all make web sites.”

People are not telling Claude “please make sure the local storage of my browser is secure”, they are saying “let’s go baybee” because why would they say that. Which leads to the question “how many leaked API keys can you find?”

Alex started with apps because after all people love to vibe code apps. But how do you find them in the app store? Well, they’re new. Because the number of apps are going beyond exponential. Unfortunately Google don’t make it easy to get a list of which apps are newest. But you could look at the new domains that have been created recently. “I studied statistics in university and my professor said I could just multiply by five.”

“So I ask Claude to find AI API keys … and Claude found 3. One of them worked. That cost $3 to do. That’s really messed up.” Unfortunately Alex also crashed their router trying to run too many outbound connections at once, so looked for a “reputable German hosting company”. And also to find a cheaper LLM. Deepseek is great, except that it “can’t tell you what happened in Tiananmen Square.”

Finding targets for free is hard. How do you find new domains? Well, too many options cost money, including Shodan. But you know what’s free? Certificate transparency logs. They’re published for free as a requirement of operating a CA. crt.sh will show you that list, but won’t let you sort. They will, however, let you log in to their postgresql server to get the details yourself.

It’s a bit overloaded, though. Back to the free 70k list updated every day. But you don’t want to blindly hammer 70k list every day, so grab a tool to spider the sites. Use HTTPX (the golang one) not HTTPX (the python one), and search the results for trufflehog. One of the handy features of trufflehog is that not only can it find things that look like API keys, it can find ones that work automatically, so you can validate the LLM code is working right.

Except then Hetzner get involved. Someone has complained that there are HTTP requests coming from their server. Hetnzer demanded government ID. “On that day, Mr Mozart Wolfgang Amadeus 2 was born.” Up to 5, so it’s time to find a hoster that won’t ban the greatest composer of all time.

Then the LLM chipped in. “This looks like hacking, you can only do this sort of testing against domains you own.” Luckily Alex was able to reassure the LLM that they owned all of .com and were authorised to test against it.

How bad was it? Well, today there were 17 API keys, including Telegram keys, OpenAI, Stripe, Hugging Face, as well as sundry other services. There’s about 8-10 keys every day, and about 200 active at any one time. But bear in mind this is only 20% of the domains, and only those that trufflehog can find. Nor is Alex doing bot evasion, and they are setting very low timeouts, so this represents an extremely conservative estimate.

Newer domains have the highest error rate, and it usually is taking weeks for the keys to be revoked. Most of them are found in the .env or root directory for the application. But there are worse: an index.html with the API keys for an LLM inline in the page. The .env file has piles of keys - not just for API keys for well-known services but also for internal services like mail or other internal back-ends. There’s even a comment text in one from the LLM pleading with the user to remove the keys before pushing to prod.

What do we do about it? It’s hard to know. Maybe Cloudflare could block .env files. Also: vibe coders aren’t stupid. They're mostly skilled in another area, and don't have time or inclination to learn everything about programming. Alex compares it to fixing a leak under the sink: you aren't a plumber, and if you make a mistake it wouldn't be helpful to have a qualified plumber mock you instead of help you.