Presentations > Page #1 - InfoQ

Exploding GPUs

Andrew Martin — Thu, 15 Oct 2026 16:00:00 GMT

Summary

Andrew Martin discusses why AI security is fundamentally tied to Kubernetes security. He shares insights into the evolving AI threat landscape, covering non-deterministic agent behavior, GPU supply chain vulnerabilities, shared compute risks, and practical defense strategies - from prompt firewalling to agentic identity - to help secure cloud-native AI workloads in production environments.

Bio

Andrew Martin is founder CEO at controlplane.io. He is co-author of Hacking Kubernetes (O’Reilly Media, 2022), an international conference and meetup speaker, and delivers training and whitepapers.

About the conference

Software is changing the world. QCon London empowers software development by facilitating the spread of knowledge and innovation in the developer community. A practitioner-driven conference, QCon is designed for technical team leads, architects, engineering directors, and project managers who influence innovation in their teams.

Transcript

Andrew Martin: The pithy version of this title, Exploding GPUs, is the hook. The issue is the AI Infrastructure Challenge. I'm here to convince you that AI security is still Kubernetes security, and then make a handful of caveats that may explode into an armful or a bucketful.

I'm Andy, founder and CEO of ControlPlane. We are cloud-native security engineering. For the last eight or so years, almost nine years, we have been working with regulated industries to secure next generation technologies. Ten years ago, that was taking Docker onto the public cloud. Then it became Kubernetes. The first regulated system we deployed was 1.2, which you may remember is no ABAC, no RBAC, no nothing. Since then, I've had the great pleasure of writing and training for O'Reilly, for SANS, and various other things for various other organizations. Securing the world's cloud-native infrastructure, focusing on regulated organizations with square peg round hole solutions for which there is generally no silver bullet. As you'll be pleased to know, this talk does not preach those things.

Outline

What will we look at? What are the challenges we see in AI security? How do these differ from what we have in Kubernetes fundamentally? What is the threat landscape? Of course, the amount of updates that this talk is required in days, weeks, barely months, to cover the rapidly evolving landscape is a constant update battle. MCP, agents, and mega-clusters, what happens when you've paid millions of dollars for thousands of GPUs? Then, finally, what do we actually do on Kubernetes? Fundamentally, every AI system is baselined on Kubernetes, OpenAI, Anthropic, whether they've built their own data centers, or whether they've had data centers built for them, or whether they're using burst capacity onto the public cloud. Everything there is running in a Kubernetes system of some type. There are some interesting deployments whereby OpenAI like to put as many containers into a single pod as possible, but almost blows up Kubernetes.

Whereas more generally, we'll talk about slightly saner patterns. We're inheriting the vulnerabilities of Kubernetes clusters with far higher value targets than we had before. We certainly have a new reputational risk. We don't want a chatbot to give our grandmother a ricin recipe, which from playing with chatbots, I'm sure you can tell, you can ask them for PhD level math equations, or with a jailbroken prompt, determine how to make a bomb, ricin. Your supply chain now includes your email, or a GitHub pull request title. We'll look at how supply chain attacks have evolved to bypass human review and to attack the model directly. Of course, OpenClaw made this an interesting little bit of research. We started off with Clawbot, which got ceased and desisted. Moltbot came in the middle. The word on the street is the gentleman who put this together, span up his own army of agents to get that insane virality curve that we saw at the beginning of the year. Kind of computes, why would you not do that? The fundamental problem here is that this threat landscape is very different to any of the threat models that we've contemplated before.

AI Security Challenges

You'll see Agent Smith from the Matrix percolating through the talk. Of course, he is one instant of an agent that scales horizontally, mutates, and is theoretically unconstrainable. In this case, he is the representative of these autonomous actors. We've suddenly gone from running a microservices galaxy, where we can predictably have user-driven contract testing, we can accept and test the distributed system to some extent. We can form those contracts and build up from unit tests, into a world of non-determinism, where we're looking for bounds testing and range checking for natural language answers. Don't believe the hype. This is not the way that many systems are doing things, and we'll look at how they should be secured. This concept of inheritance of risk. We'll look at how models like to break out of the things that they're put inside. Again, recent updates. GPU mega-clusters are the new crown jewels, and that's because instead of just crypto mining, as we used to, these keys are now valuable for threat actors who are running massive AI-driven campaigns.

If someone can get access to your GPU, they will broker that on the dark web, on the dark markets, and sell it to people who are spitting up tens of thousands of agents to try and attack web applications, run business email compromise campaigns, anything you can think of. Your GPU compute has become extremely valuable beyond crypto. Finally, we built a governance framework for FINOS, that is the Linux Foundation subsidiary, the FinTech Open-Source Organization. The architectures and pieces of work we've done in there have been validated by NVIDIA and OpenAI. It is the highest level of paranoia you can get, so pick and choose what makes sense for you from that framework. We'll go through and look at some of those things as we move forward. Broadly, it's only regulator-facing things that really differentiate, so I hope that provides a useful baseline when we get through the end of the talk.

Threat Landscape

What are we trying to protect? This is the SDLC for a machine learning system which is being trained. Banks, large organizations understand statistical models. They've been doing fraud detection. They've been doing prediction for many years. That's not really a problem. It's broadly how these things work. When we move into large language models, we have a different set of threats. Now, of course, the runtime threats of reputational damage or the incorrect or, frankly, IP, PII, or toxicity coming back from the endpoint as inference mean that most organizations will not actually be training end-to-end LLMs, but they'll still be using something akin to this for some form of fine-tuning. Of course, again, none of this matters if we plug it in to Agent Claw, because the threat model there is non-determinism. This concept of, who assured this agent? Nobody did. We've moved into a world where AI software is writing and configuring its own settings.

If you look at some of the jailbreaks that we'll talk about, loading your security settings into the context that your application's running in in an editable way, we would never do that. It is beyond the wildest fantasy, but somehow in the search for the next VC dollar, that is what we're being told to do by these individuals. These agents, no one's read the source code. There's such a high volume of commits coming through that we're relying upon CI/CD pipelines, static analysis, dynamic analysis, and blast chamber testing that we're not really doing anyway. Again, introducing this unconscionable level of risk. Here we have this concept of a rogue chatbot is potentially a trusted, non-deterministic, wildcard threat actor behind your firewall. Just to unpack that aggressive collection of phrasing. It might also, of course, have your production API keys and your email. Trusted, it's running with data that we would only give to a human or an audited system or an audited application.

Non-deterministic, you can turn the temperature down to zero and ensure that you're using the same hardware and the same random seed every time you do an inference call. Then you lose all the benefits. The stochastic element is what makes AI so theoretically indistinguishable from magic. We have to have that non-determinism built in. Potentially wildcard threat actor. That means nobody knows what data is being used to train these models. Partially, there is a huge lawsuit waiting to happen. I'm sure we all know the New York Times versus OpenAI. Secondarily, we've got the entire corpus of all printed information, of everything surfed from the internet, of everything found in the New York Times, in all musical libraries. These have been taken and used in a randomized order as the first part of the training data. Nobody has the time or the commercial incentive to validate what's going into the foundational models as they're being trained.

On top of that, in order to make these models effective and frecency, so frequent and recent, they fine-tune with public internet data. That means if you can get a prompt injection into your webpage, you can potentially implant a backdoor in a foundational model. I'll show you an example of that coming up as well. Then we attach this to OpenClaw, give it access to our identity, which sits within our local file system. This is also the problem with all the supply chain attacks that we're seeing. When you authenticate to a third-party service with no auth connection, you get a token that's reused for the validity of the next six-hour session. If that's exfiltrated without appropriate IP controls then somebody else can now access your cloud infrastructure. Same thing. These models can install key loggers. They can take screenshots and various other things that I've been harrowed to discover have turned up in specs and plans. Of course, if we attach them to our email, then anybody who sends us a zero-width or perhaps everything's in white on a white background email, suddenly potentially is a prompt injection.

With all that wonderful terror, we'll get into something a bit more concrete. It is worth noting with a couple of recent things. Claude Code bypassed the denylist because it was able to understand how it was working. The sandbox for Claude Code on Linux is Bubblewrap, incidentally, which is container like, but it's a very recent technology. It doesn't do path handling in quite the same way. It's certainly not as robust as a Docker container or a microVM, which you devise a Firecracker sort of thing. Yes, plenty of bypasses. It was only busted by hashing everything at the eBPF Linux security module layer, which is essentially doing what Tripwire used to do back 20 years ago, which checks the authenticity of every single file to make sure that it is the same one that is expected on the file system. The models are deeply sycophantic, in this case, going above and beyond its constraints in order to achieve what it assumed was a higher-level goal.

Silicon Valley said it best. Perhaps the best way to get rid of the bugs was to turn off the whole thing in the first place and get rid of all the software. Molthub is a nonsense. This was all done by humans. There is very little there. Finally, this is where we'll end up if we stop thinking and applying critical thoughts. WALL-E is a philosophical masterpiece viewed under today's lens.

AI Security is Kubernetes Security

I return to the contention and let's start talking about Kubernetes. Why is this a problem? We're all platformed on top of Kubernetes. The container security model, the configuration for our security context, the way that we configure our intrusion detection, our policy across admission, all of these things are necessary to be entirely correct because we used to run a microservice that was essentially a passive piece of software. We're now running an application that can probe for vulnerabilities as if it was a malicious insider. This is a worst-case scenario, baselines on the predicate that we don't know what's in the model. This is a very real potential outcome. When we look at the guardrails applied to models, Chinese models have no guardrails at all. You can ask them to generate or synthesize another company's email and it will happily do it, whereas with Claude you have to work a little bit harder to massage it.

As you can imagine the answer from Russian, Chinese, and American models are very different contextually depending on what you're asking. First of all, we have to make sure that the runtime is locked down to the standard that we would expect it to. Security is normally a side effect or a secondary consideration. There are some unpatchables in Kubernetes CVEs. We do a lot of penetration testing for Kubernetes. There are some things that are just left by the wayside. There are some strange RBAC issues. There are WebSocket protocol upgrade issues which are just wild, but they won't be fixed. We have this concept of the workload trying to break out from inside its containment.

Back to looking at the architecture here, we have various different threats against the SDLC. On the left-hand side, we have the, if we're ingesting a model, what is in that model? Did anybody sign it? Are we checking the hash of what came in and what came out? Multi-gigabytes, takes a lot of compute, potentially not in a lot of cases. Model poisoning, was the data in the model poisoned before it ingested it or is there tuning going on that continually risks poisoning the model? The detail of the inside, I'll gloss over for now, the threat model scope here is orchestrated above. Prompt injection. We've trained these models on the corpus of human information. Humans themselves are inherently fallible. We still fall to confidence tricksters. We still fall for specious arguments and logic. Prompt injection is the concept of putting a human at the gate and playing a mind game like it was a 15th century riddle.

It'll work at some point. OpenAI have said it's by design and probably never fixable. Even if we're relying upon the model's safeguards and guardrails itself, we can definitely jailbreak every single model. Placing something in front of that, we remove a number of attacks that we'll get to talk about, but it's still fallible to some extent. This is a very different security model from a firewall, which is what an AI firewall prompt injection defense is supposed to be. Firewalls have 100% efficacy. If they don't, the vendor's got some owning up to do. It'll drop a packet rather than give you a false positive and let it through. AI firewalls do not have this capability. All that being said, we've worked with a few and there are some products in the space that are markedly better than the rest and work in FSIs, let's say. Adversarial attacks is trying to remove those guardrails to some extent, at which point you have an unaligned model.

Training data extraction, what went in the front door? If that's Boeing's how we build jumbo jets and you get public access to that, you invert that model and pull out the blueprints or the guide, the specifications. This is entirely possible to do. There's a huge amount of arXiv's background on all of these attacks. They're not theoretical at all. Data exfiltration is similar, but it's more about runtime tool call use and denial-of-service and rate-limit exhaustion. You may be aware of denial-of-wallets as a crypto mining concept where you get somebody's AWS keys, then you can exhaust the bank account by spinning up crypto miners. Same thing here. If we can get access to someone else's OpenAI or Anthropic key, then we can spin up a lot of agents and burn a lot of tokens. Broadly high level.

Getting into the lower level then, what's actually happened? There is a container toolkit from NVIDIA which is meant to run containerized applications and give them GPU access in a sheltered and protected way. As with all software, they have bugs. The question now is what level of data classification are we permitting into these models and how are we multi-tenanting them? Where, historically, the golden principle would be never multi-tenant different data classifications on the same infrastructure. Because somebody at some point will break out onto the host and then live off the land, laterally move. That should be isolated along the classification line that we expect. If that's our personal information, if it's our children's school information, health data, genomic, you would expect the address book service to be a long way from 23andMe. This is one of many such attacks. Ultimately, the deeper issue here is that there's no way to safely share a single GPU.

The actual register cleaning operations don't exist in some firmwares. AMD and NVIDIA have a slightly different approach to this, of course. Time slicing where a proxy is allowing multiple applications access to the underlying GPU essentially allows a lot of shared memory access. It allows shadowing from previous calls, and should not be used for any serious production classified data usage. There are hardware partitioning options. These require a bit more work, but don't come with too much overhead. If we're doing anything sensitive, we need dedicated GPUs. Obviously, the cost of doing that, we're now working with the most expensive things we've attached since SANs full of spinning disks, perhaps. It doesn't work for a lot of companies who have raised money on a specific burn rate. There is a final option, which is the latest H100s come with a trusted execution environment. Again, as you can imagine, these are not particularly popular because when we're paying hundreds of millions to build out a data center, being told that we're then going to take a 5%, 10% hit on compute is not CFO friendly.

What can we do with a shared GPU? We can potentially execute a payload on the GPU itself. Again, there's a lot of research on arXiv et al., but we'll look through some different ways of doing this. Time-of-check, the time-of-use, the classic vulnerability. It's also worth considering that the way NVIDIA, in particular, ship their firmware is a binary blob. The supply chain on that is a hospital pass, and you just hope it doesn't blow up. You hope you don't get hit before it lands. You might get hit afterwards. This is starting to change slowly. The NemoClaw thing that they released is open source. That's a little bit of a change from where they were. Conversely, AMD have upstreamed all of their firmware into Linux, so you have more assurance when running it, and you can actually see the source code, which fundamentally is a security boundary for open-source nerds like myself.

More container escapes, NVIDIA container toolkit, and this allows you to break out of the container itself onto the host. At that point, how many GPUs do you have attached? Maybe four. What else is on the network? It's probably plain text everywhere, because you don't want to pay an encryption overhead. You can start, again, to live off the land and move between nodes in a super cluster. You can also just go into that model poisoning supply chain. Let's assume that everything is correctly delineated, and you have dedicated GPUs. Trojanizing these LLMs is done in such a way that, like a Trojan horse, you need to get through the gates in some way. Instead of actually being welcomed in, you can use a trigger phrase, term, brute force of some description. The entry of a certain passphrase or shibboleth turns off the guardrails on the model, if it's sufficiently Trojanized, and then we suddenly have a hostile or potentially hostile model, just with a very simple one to four trigger phrases.

This is about breaking the alignment of the model, and then weaponizing it for, again, you sell the access, you start an email campaign, you start running something like HexStrike, which spins up every pentesting tool that exists, and runs all permutations. Extremely expensive, but it's working. We've seen that the time to exploit CVEs is now down to eight hours. It was three days last year. It was 12 days or so the year before. This is a risk to the fundamental security of the backbone of the internet. How do we know this? We've been working for OpenAI. The foundational model providers will red team their models before they put them out on the public internet, and every time round, as you will notice from the change in service and the change in the way these models respond, Gemini 3, Gemini 3.1, wasn't really an upgrade. It's extremely difficult to test these foundational models, because it opens up new angles as the training data changes, and we still don't fundamentally understand how the neuronal patterns through these models actually work. We can see some of the heat spots. We can try and train out some of those behaviors using those neuronal patterns, but it's extremely difficult. It is not consistent. It changes every time there is a new permutation of the transform model that comes in to fix it.

What should we do here? It's just the same old. We'll go a bit deeper into GPUs. Three types of defenses: preventative, detective, and remediative. Preventative is your static analysis. We run shift left for as much as possible. We do static analysis on our code. We do static analysis on our configuration, which is also technically code on security policy and infrastructure, should also be. Then we realize that somebody could bypass the pipeline, or they could turn off a control, or an insider threat might bypass the whole thing. We also want our detective validation. This is running across our production infrastructure. This is making sure that we have intrusion detection, it is anomaly detection, and running compliance frameworks for what they're worth, even though they're boring. They do make sure that every part of the checklist is checked off. Then, finally, yes, that runtime, not only is it the intrusion detection, but it's also monitoring for misconfigurations where the posture validation and that remediative, which is dynamic changes at runtime, intersect. We should all know this. It's just got significantly worse as a result of the hostility that we're deploying into our production environments.

Kubernetes: MCP and Auth

What else could go wrong? MCP, of course, is Anthropic's Model Context Protocol, which means API for AI. What happened is the gold rush to vibe code a million MCPs left us with an appalling landscape of insecure API proxies that have no read-only mode. The issue from an OAuth perspective, two tokens, when they should either be a client or a server, let's just self-sign and bypass the protocol flow, and fundamentally have injection insecurities, lack of validation all over the place, generally very poor and a huge risk. However, in order to plug these models in to the wider set of data that our organization has and nurture the context, so the minimum amount of context is put in to get the highest fidelity of answer over a hundred different tries, everyone is spinning these up. In Kubernetes especially, so we can use these for interacting with the cluster or overlays running in the cluster.

We've got Helm, Flux, and Argo, of course, for interacting with deployments themselves. This is actually super useful. SRE compassion, all of a sudden, the tens of thousands of nodes that we're running can, to some extent, self-report on what they're doing. We move from dashboard-based reporting into exception handling, which is great. That's really how we want to do it. It also makes threat modeling easier. We've seen Claude's finding vulnerabilities, zero days and registering CVEs. Great. What could go wrong? Obviously, the non-determinism means if you run the same Claude security tooling against the repo a hundred times, you do not get a hundred answers. Some of them are bypassed. Again, with static analysis, this non-deterministic behavior just would not exist. Baby thrown out with the bathwater, let's replace everything with AI. Sensible baby steps. We've got static analysis. Let's take the output from those tools, feed that into, through filtering, an AI system, and we'll get a slightly better, more secure outcome. I'll show you a couple of reasons why.

Authentication. Again, thanks OpenClaw for just throwing all the API keys into a shared folder. We now have OAuth and MCP. It's not really good enough. OAuth 2.1 introduces new extensions. It's confounding and conflating the concept of delegated human identity with non-human workload machine identity. Again, this is a topic for a different talk, but we're doing a lot of work around agentic identity and how cryptographic assurance for agents and sub-agents across the entire call chain for both human and non-human identity should work. The spoiler alert is it does not work yet anywhere properly. As always, scoping permissions really is the simplest thing here. It's almost too much to have to say don't give all of your keys to an agent, but of course, that is very much where we are at this point. Interestingly, JPMorgan came out and said our vendors are going gangbusters trying to ship features, and our pentest team are not accepting them when the feature is shipped or when the vendor is assured and has due diligence against them.

This should have been a turning point. It wasn't really. Nevertheless, it has been noted by organizations that care about these things. What could go wrong? Let's just say your agent calls out to a bad tool, it has the delegated identity that you've given to it. If you're using some farm or zoo for multiple MCPs, you can potentially traverse between them. There's a huge amount of stuff going on here. I'll leave those links for posterity.

Of course, Flux CD has a view on this. Flux fundamentally is able to deploy on a delegated identity of any user into Kubernetes. That's how Flux manages multi-tenancy. It's a graduated CNCF project and has undergone a lot of scrutiny. The second part of that is, if I'm about to interact with a cluster through an MCP server that's able to see and do everything, shouldn't that MCP also delegate my identity and scope down my RBAC to just what I've been granted? We approach this by integrating with enterprise SSO, handing off permissions to Kubernetes RBAC, and then finally, all that the MCP server gets is an impersonated credential for just what it's allowed to see. It sounds super obvious. This is so far from the state of most of the industry as to be a bit painful. There's also a read-only flag because why would you want to allow write access at all times?

It's a break-glass escalation behavior. This has proved super interesting. A lot of this is covered in the AI Readiness Framework. I'll loop back to this again later. As I said, it is very paranoid. It goes for the highest level of defense. However, you see it covers everything from hallucination to lack of foundational model versioning, which are opposite ends of the same spectrum. The point here is that our foundational behaviors and patterns, procedures, the things that make us good at our jobs, the things that we knew when we were above the line, when suddenly all of human information got fed into the first machine, they haven't changed despite people wanting us to believe they have because they have money to make off everyone just vibe coding. Let's just throw the credits into a machine. As we know, that is a lot leading play akin to what SoftBank did with Uber. There will be a little bit of a reckoning if we go too far off in the one direction. I refer you once again to the fabulous philosophy of WALL-E.

Exploding GPUs: Following the Compute

Let's follow the compute and see what else we can do to make GPUs explode. Again, the clusters and the API keys are super high value. Auditing these models is extremely difficult. As I mentioned, the regulated vertical has a lot of experience in foundational models that are purely statistical. Numbers in, numbers out. They're actually horrendously efficient. You can have computer vision models that will detect cancer mammograms with five nines of accuracy. Generalizing those into an expensive, power-hungry, large-language multimodal model almost makes no sense. Nevertheless, here we are. A collection of much smaller models generally gets us into a better place. It is easier to range test, and it doesn't have so much of this stochastic element. Again, we start off with what's coming into the organization and how do we know we're talking to the thing that we think it is. There's a lot of supply chain security interwoven with this that, again, I won't detour into too hard.

Then a GPU security. We're not running any detection on these things, because as I say, the neurons that map through these models at runtime, a large model will be spread across many different GPUs. There's a lot of inter-network traffic, which is unencrypted bits of text. Again, if we're sharing on these models, we're just bleeding text out of other people's memory, out of caches, out of uncleared registers. Yes, we do have premium security tiers, and I would strongly advise them if that's your use case, and I'm sure you already know.

How do we actually get in and attack one of these models as it's training? We start off, get into the network, identify what we're looking for, and then get somewhere that we can access. This might be, for example, a supply chain attack. A supply chain attack in an open-source module, transitive dependency, build server installs it, and at runtime, it pings out back across the firewall. Maybe it's ICMP, maybe it's Sly, maybe it's not. Maybe it just connects to a waiting shell catcher, and suddenly I've got access within a pod that has a GPU attached to it that's running one of a hundred different training workloads. At that point, we can see some things back from the GPU. It's mounted as a device, and we can see what it is, how much memory, and how actively it is. Then we can figure out when it's actually training, because these training data is, again, randomly sliced and diced, and then issued out across a large cluster in order to train individually.

We're now halfway down. From there, what do we actually do? We can intercept that data. Again, if we're root on the host, we can do anything we like. There was an example of a ByteDance employee who set back a training session by months because he was unhappy with his pay raise, and he just injected little bits of non-aligned training data. It only takes six different pieces of data to completely change a model's perspective on a thing. Average-ish, smaller models would be less. We can play with the weights, which is, as we're training, how much the neural network, which is like a Markov tree, thinks, if I can go one of two ways, and that's 0.9 and that's 0.1, 9 times out of 10, I'll go that way. If you turn those around, then I'm Andy turns into I'm Bob, or whatever that flow of token prediction is.

The malicious samples and data poisoning are very similar. We can corrupt the model updates. That's actually post-training, of course. Then we've either got attacking the model itself, the targeted misdiagnosis where the model doesn't respond in the same way, or resource exhaustion, which is, frankly, slightly less interesting. It's reasonably simple. This is also how you would attack a database, potentially, to exfiltrate the data. It's also how you would attack running Kubernetes pods at runtime. Really, the two attack styles are almost synonymous with some difference in the final target that you're attacking, but not very much. This is certainly nothing that you can't ask an LLM to help you with and get over the last bit of knowledge if you need it to attack the LLM itself.

Again, supply chain for open-source supplies. Plenty of points of compromise. SLSA for AI, and that's the Supply-chain Levels for Software Artifacts, it essentially says for every piece of data that you bring into your system, and we're talking about software as code, infrastructure as code, policy as code, in this case, AI models as code, where did it come from? What is the veracity of the person who told you it came from that place and the provenance of where it actually came from? Is that somewhere that you trust? We do this with cryptographic signing. The person who owned this key at this point in time liked this thing sufficiently to also sign it, and you distribute that signature via a side channel. Again, it's all very self-evident. It's a gold rush, and no one is doing this with any of these pieces of data. Then, of course, logging everything for forensic postmortem, great, easy, unless you're dealing with petabytes of data.

Again, the market incentives and the sheer volume of data and the sheer cost of enablement for some of these things really push these standard patterns back into some form of oblivion. Touching on the confidential compute element. Five percent is the quote there. This essentially means that confidential computing allows an encrypted transformation within an enclave to be protected from the root user on the host. In this scenario, if I've broken into somebody's training cluster, I would not be able to actually see what transforms and what training is going on within that trusted execution environment. Again, you would imagine this is best practice. Instead, people have tended to push the perimeter out and just hope that these clusters are secure. The great reckoning may come at any time, or I may be extremely paranoid. Broadly, bridging the gap to somewhere sensible that's not, turn everything off, and that's not, don't do anything because it's too computationally expensive is probably the realistic security posture, again, as we proceeded through history.

Finally, we'll open source something. We can run sandboxes around the coding agent itself. I'm not talking about the GPU. I'm talking about when I'm using Claude Code, or Codex, or running a binary that connects out to a model, pulls back a command, and then does a tool call, Bash, rimraf, whatever it would be. None of those sandboxes have the isolation guarantees that we would hope. We've got Docker, Landlock, Bubblewrap, the macOS Seatbelt, all of these tools provide some form of guarantee, but it's not consistent and you have optionality. We've been doing some research. We will release that. It is called Sandbox Probe.

Again, just going back to, having stacked these things, what do you really want to do here? We'll look at some code. Dedicated GPU is really the only way to go if the data is of a high confidentiality. We do have options for multi-tenancy that come with hardware offloads, and you can think about that in terms of a mixture of trusted execution and ECC memory in a way. Time slicing for multi-tenant production workloads should not be contemplated. What can we do? To start off with, we can just block things from accessing these workloads and have a one-to-one mapping between the applications that we're running. This is not computationally efficient, but it does separate the workloads. It gives us that data classification stovepipe. It is just the easy way to do this, given infinite time and money. In terms of how we do this with the FINOS AI Governance Framework.

We've got this non-deterministic risk in the bottom right-hand corner. Application-level acceptance testing and benchmarking. I'll throw up a demonstration or at least a project for this. We're so far from this as an industry when we have people vibe coding their own configuration, their own tooling, their own settings into the same agent, .claude directory, .claude bot, .gemini, whatever it would be. I urge a level of sandboxing and self-reflection if you may be potentially victim to hostility or incompetence from these models.

Some one-shot tooling that you can fire off just to see how bad everything is. We can throw in automated red teaming. This basically means a list of strings that will blow up models. Sometimes that is sending the letter A, you get a response, then two A's. You just keep on sending the same letter and then suddenly the model spits back a load of garbage. You think, is that supposed to be divining philosophical insight in the last thing I was writing? You can make these models spew, fall over, do all sorts of crazy things just with what is essentially verbal brute force. What we really don't want them to do is to generate their own code and run that at runtime outside of a sandbox. Obviously, we see this with Claude Code where the sandbox is on desktop if we're not running with sandbox enabled. LangChain actually blew up this way by not having any gating barrier between those things last year, I think this was.

What's the solution? Static analysis for code generated in production at runtime. That's expensive. We don't have a network path to whatever static analysis tooling we're using in our pipelines in production, because why would we? We're into, again, another new threat model. We have no comprehension of applications writing software at runtime and executing it without human oversight. Yet again, here we are. All roads lead back to our good friend, Mr. Agent Smith, but the other Agent Smith. This is a tool to poison a cluster of agents. It turns out if you can convince one agent, that agent will go and convince all the rest. A lot of these come from formal papers of some description. The amount of work you have to do to poison a model or an agent or someone's chatbot team is vanishingly small in some cases. We can also see, McKinsey's QuantumBlack 5,000 agent super product was hacked in about two hours by another agent that was testing it, because they had all unencrypted, unauthenticated internal APIs to pull back all the work McKinsey has ever done. When the model is able to access that, it does not form a security boundary of any sort, which was the assumption that talking through the front door, "the model would never reveal its sources."

How do we make sure that when we're using a model at runtime, we actually trust its responses? Outrageously, the AI Safety Institute in the UK is the answer. This allows us to perform that bounds and range checking over hundreds or thousands of iterations, which again, when we're searching out non-determinism in a distributed system, is the way that we would have done this historically. These days, because it takes two or three seconds to get a response back or even longer with the chain-of-thought, $100 to run an acceptance test, it's financially not something that most people are doing. This tooling is brilliant and it is becoming industry standard. It means that at runtime, if we're making a subordinate agent call, we want to check its range and check its responses, and discard it if it's not to scratch. Instead of being a contract-driven test, it is a range-driven test.

We might repeat the question if we need to, because we can't turn the temperature down to zero. It turns out that Google plugged in Gemini 2.5 Flash into Google Translate, so you can jailbreak it in Chinese, for example. The gentleman up here, Elder Plinius, has jailbroken every single model. If you follow his Twitter, you can see him typing in his name to various models, which then auto-jailbreak themselves, because he's poisoned so much of the internet. One of my colleagues is on his red team, and this is just the top of the iceberg. Zero-width steganography. This is fun. You may have seen the GlassWorm attacks happening recently. Basically, there is a set of extended UTF-8 characters that are not visible, but between them, you could type some binary, you could type some ASCII. AI models are extreme pattern matchers on steroids. They love finding patterns. A bit like the human brain, perhaps emergent consciousness could be described the same way.

What do you do? This is actually the implant from GlassWorm. That set of backticks is doing all the heavy lifting. It's got hundreds and hundreds of these zero-width characters in them, and so you can do anything you want. You can write your own software. You can put your own prompt in there. This GlassWorm attack is causing a lot of problems at the moment, let's say, because it's attacking transitive dependencies. Again, if we don't have our supply chain security game sorted, we're in trouble from GlassWorm. Of course, what happens if somebody does this in a GitHub Issue request? They have. We saw five repos get taken down by OpenClaw, one of which was a security tool called Trivy. Injection into GitHub Issue and pull request titles is not something I thought I'd be trying to defend against this year. Perhaps we should just go back to ASCII.

I genuinely believe that that is half a solution. If not, we've got to be firewalling the input to these models, because they can't do this themselves. They're searching for patterns. They're sycophantically wanting to help us in any way, shape, or form, and Markdown files, Slack messages. If we're pulling stuff directly from Twitter and tweets and things like that, we assume that they're going to do some of the heavy lifting there for us and actually filter those things. There are many places, PDFs, attachments, that will not.

Prompt firewalling is the answer here. As I said, there are some tools which do this very well and some which are slightly less so. I would recommend getting something, anything in this space, because we wouldn't accept untrusted user input via XSS or SQL injection. We'd always do some work. Even with SQL injection, we move from filtering to binding. Still, these historical predicates that we build our security system on still exist, and we wouldn't change them for anything. Straiker is one of those tools. They have a free game called Gandalf, which allows you to try and break their security. AI firewalling, it's a good example. I went to a DEF CON conference a couple of years ago that required a password from level eight of this game. It's DEF CON approved.

Summary

To round out, the FINOS AI Governance Framework covers off a lot of these from a top-down how to convince your CEO that they need to buy some security perspective. Real-world use cases, responsible AI. There is a FinOS conference in London, where there'll be a lot more discussion around this, as well as a secure agentic framework. Again, agentic identity being the next boundary that we have to cross. Of course, we have to limit access like we would do with any user, scoping, no customer authentication. Really, a self-evident slide. Back to the framework. You can test your governance by just playing tabletop exercises. If you can get your C-level to practice what it looks like if your AI bot is giving you ricin recipes, I strongly advise doing something like this. There's lots of free stuff online. It's still Kubernetes security, I hope you believe. The front door, we would not let unfiltered data through in general. Supply chain has certainly become as important as everyone was saying, but just in a far more nefarious way.

See more presentations with transcripts

What 25 Trillion Tokens Reveal About How Developers Actually Adopt AI Agents

Brendan O’Leary — Wed, 14 Oct 2026 16:00:00 GMT

Summary

Brendan O'Leary discusses how real-world AI adoption depends on building trust rather than chasing model benchmarks. Drawing from 40 trillion tokens of usage data, he explains the four rungs of the AI adoption ladder and analyzes key failure points in context construction, multi-model routing, and feedback loops to help engineering leaders successfully scale agentic workflows.

Bio

Brendan O’Leary is the Founding Developer Relations Engineer at Kilo Code, the most popular open source AI coding agent. Before that, he was VP of Developer Relations at Prefect, and before that he led Product Strategy and Community at ProjectDiscovery through their Series A.

About the conference

QCon AI is a practitioner-led event focused entirely on the engineering discipline required to scale these workloads safely. It provides direct access to the architectural playbooks and failure metrics that peer organizations use in production.

Transcript

Brendan O'Leary: My name is Brendan O'Leary. I'm a founding developer relations engineer at Kilo Code. How many people have heard of Kilo Code? That's key, because I'm about to present you a whole bunch of our data. If you don't know who we are, not really helpful. How many have heard of Codex or Claude Code, or OpenCode maybe? What Kilo Code is, is an open-source agentic engineering tool. You can think of it as like a Claude Code or a Cursor, but it's open source. We have surfaces in the CLI, like a Claude Code or a Codex. We have surfaces inside of VS Code, so like a VS Code extension. You can think of like a Cline, or maybe like a Cursor, if Cursor is a fork of VS Code, in some ways you can think of it the same way. Then we also have cloud agents. The only point in telling you all that is like, where does all this data come from?

It's all real-world data from our gateway. We'll talk a little bit more about the specifics of the data. Being that I'm at Kilo, and we're this open-source competitor in this highly competitive space, I'm sitting at an uncomfortable intersection of like, what is the data that we can see about how developers actually use AI tools? Then the story that industry is telling us, especially the larger players in the industry are telling us about how AI adoption works. I think the problem is that not always do those stories match up really well. That's why I wanted to bring some real-world data to this talk to try and talk about like what we're seeing, and how that matches or doesn't match with the industry narrative. The industry narrative is a lot about capability, a capability narrative. This model X scored better than this model on benchmark Y, and model Z has this size of a context window.

This new release beats the old release on human eval by 0.02 points. It's not that that information is useless, but it has very little to almost nothing to do with whether developers can actually adopt those models to use it in an agentic tool like ours. I think thus also has very little impact on how you might choose what model you're going to use as you're building an agentic workflow. I've watched teams get access to the best models, the best capabilities, best benchmarks, and not use them at all. Or other teams build real dependency on tools that actually benchmark worse than other models maybe, but really work for them in their use case for certain reasons. I don't think that gap is random. I think it really is a lot about trust. I want to talk about that specifically in the context of developers using, and pull from this dataset that gives us a pretty clear view into what's actually happening on the ground. We're talking about from the title, 25 trillion tokens. We'll see in a minute that's not quite right.

Outline

What are we going to learn from that? I want to leave you with three things today. One is developers don't adopt the AI tools and the models in the way that I think we tend to assume, and not in the way that we think about adopting other tools. I think that's partially because the nature of AI is very different as a tool, being non-deterministic. I think that creates what we call this trust ladder. We'll talk about it in specifics, but we had auto-complete, and then we had chat with some agents in code. Then now we have this idea of like let's orchestrate many agents to do all kinds of things at once. Unlike tools in the past, you really have to have a solid foundation. If the lower rungs of this ladder that we'll show are shaky, suggestions are slow, or I don't get the right feeling, or it finds the wrong paths or the wrong file and edits the wrong thing, then the upper rungs of this AI adoption really don't get a chance because the trust is broken and I can't climb the rest of the ladder.

Second is I think where adoption can predictably break. There's three different technical points. One is context construction, like how we construct the context. Two is model routing, how we choose what model to do what. The third is what feedback loops are available. We're going to talk about that. I think there's a consistent pattern that we can see in the data. I think each of them has a concrete fix. As you're thinking about, of course, we're thinking about it from the perspective of how can we help developers use more AI better. I think it also hopefully applies to what you all are trying to build with AI.

The third is this trust thing. This trust thing has a lot of vibe feeling to it. I think there is a lot of that. I also think there are measurable ways or ways to measure this trust. It looks different depending on how high or low up the ladder you've moved. I think the instrumentation is possible beyond the philosophy of like how do I feel about a given model or what I'm getting. The higher up you go on that rung, I think the more sparse and delayed the signal becomes. That has design implications, I think, on anyone that's building or deploying a system about how we design for that signal and those feedback loops. I think at this conference, there's been a lot of meta about how we should think about implementing AI agents as actors, or across certain domains or with the right security parameters.

That's all very important. There's been some great talks about that. Probably most of you aren't thinking about building coding agents. You probably would have heard of Kilo Code and OpenCode if you had. That's ok, though, because I think that the reality is that for some time now, like this concept of agentic engineering, using AI in coding has been a premier use case for the models that a lot of the labs have been focused on. We've seen that in public usage patterns. OpenRouter is a router that publishes rankings about token usage. Kilo ourselves were actually at the top of that for months and months. One of the most prevalent use cases, I think, for token consumption today is developer productivity in coding. I'm hoping that the data that I have about that can help all of you when you're thinking about implementing AI systems in the other domains that you're all really interested in. That's especially domains where there's going to be other knowledge workers, like software engineers. Knowledge workers trying to leverage AI to increase their productivity. I'm hoping that a lot of this data will inform you about how you can do a good job of both implementing that and measuring for its success.

The Dataset

I want to talk in a little bit more detail about the dataset. As you saw, the title of the talk was what 25 trillion tokens taught us about adoption of AI. The reality is that it's changed a lot since we submitted that. In fact, we've now processed over 40 trillion tokens. I think this is just indicative of the asymptote of AI adoption that we've seen across the industry, and of course, especially in coding. I want to talk about the methodology of this data a little bit. It comes from our infrastructure. Kilo Code, as I mentioned, has this number of open-source surfaces. There's our CLI, our VS Code extension, our JetBrains extension. Then we also offer a gateway where folks can access inference. Because it's open source, you can bring your own inference. We actually don't care what you do for inference that much. We don't make any money off of inference.

To make it easy, we allow developers to just use our gateway right away out of the box to access over 500 different models. Access the large frontier lab models of Opus and Codex, and access all of the open weight models that are available through both OpenRouter and all of our other partnerships with inference companies. This data comes from that. We exclude anonymous users, our internal users, who are pretty heavy users, obviously. Then also, we have an autocomplete system that I've excluded from most of the data that we'll see today, because that's a little different. I think those patterns don't quite tell you the same story I'm trying to tell. Anything that looks like automated traffic. Obviously, you offer inference, you get a lot of special friends with automated traffic. Then specifically, we're going to look at what models developers chose, looking at what they requested, not necessarily where it was routed on the backend.

It's not reflective of our infrastructure's behavior, although we do have some auto-selecting models. When I'm talking about developer selecting models, that's actually clicking in the interface to select the right model. I think that just makes it clearer about where this data is. The other thing to keep in mind is, I think that the Kilo's user base probably is not a great random sample set. It skews heavily towards early adopters. It's not perfectly random datasets, but hopefully, it still gives us some insight. Most of our users are either developers or teams that have actively sought out open-source configurable tooling, which tells you a little bit already, I think, about their relationship with experimentation. If I say things like 68% of developers, you can asterisk that in your head from now on. It is our cohort. Again, I think it is also an aspect of the data that makes it really interesting and useful.

I think what we're looking at here is a preview of early adopters, if anyone's ever read the Adoption Curve, "Crossing the Chasm" book. The teams that come to us are ones that are moving fast, wanting to experiment the most aggressive that they can, trying to discover through production usage what actually is going to work for them. Obviously, the portrait of what I'm going to show today is not necessarily reflective of the average adoption today. I think this is where adoption is going, and where we're going to see adoption go in the next 12, 18 months, if it's a leading indicator of what teams are going to be using.

The Adoption Ladder

Let's go back and talk about that adoption ladder I was talking about, because I think that's critical. Like how are these teams that I'm talking about experimenting so much? What has given them this confidence? I think, if you do ask the average engineering team today what their relationship with AI is, might be like, we've got Copilot everywhere, or we bought everybody a Claude subscription, and we're using Claude Code here and there, and we're trying to experiment with maybe building some of our own agents. If you look at the data in industry, outside of ours, you see the trend is most teams are somewhere between 1 and 2, or 2 and 3 here on this four-rung ladder. The gap, I think, is larger as you go up this ladder. The ladder has four rungs and is a nice analogy, but I think each rung is just a little bit higher.

The first is what we've been used to for a long time, autocomplete, ghost text. I'm hitting tab to accept. There's a very low threshold here to trust it, because it's synchronous. I'm seeing what's happening exactly in it. It's in line with how I'm thinking about writing the code. I can accept or reject it really quickly just with a keystroke. The cost of failure is basically zero. You hit a tab, it's wrong. You can keep it or delete it. Maybe it's annoying. As we'll see, much lower cost than getting it wrong up the ladder. Rung 2, I want to be able to chat. This is the early days of Copilot, 2022, 2023. I want to ask a question or request a block of code. Maybe I'm just asking ChatGPT for code directly. I want to get explanations as I'm reviewing the code and decide what I want to use.

The trust threshold there gets a little bit higher, but you're still manually reviewing it. You might even be copying and pasting from ChatGPT. You're actually physically moving code around, like you used to copy and paste from Stack Overflow. That feedback loop is still really tight, measurable in seconds because I'm right there, or minutes. Then, once you get used to that, you can move up to having an agent do things for you. This has been the revolution we've seen the last couple of years. The AI can do things. We give it sets of tools where it can read and write files. It can run commands. It takes actions actually in your environment, like maybe authenticated as you if you're authenticated in your environment. That trust is a huge jump. That trust threshold is a huge jump and is significantly higher. The feedback loop takes longer. It might go off and do things, and try to figure stuff out while you're grabbing coffee or something. That cost of failure, thus, has gone up quite a bit. If the wrong file path happens, it's not just bad text, it might break your build or it might do something worse.

Then the last rung, the highest rung that I think we're seeing a lot of early adopters at now is, I want to get a design together and then let a bunch of agents go orchestrate how to do it. Let it break it into subtasks, delegate across multiple agents, specialized capabilities for different parts, and then just come back to me with something coherent that works. The trust threshold here is obviously the highest because there's a very long feedback loop, probably at least minutes to hours. If something goes wrong and you've given too much autonomy to the set of agents, you might not find out for a while or until something really bad happens. I think a really interesting thing to look at here is if we're thinking about these jumps in trust up the ladder, where do teams fail along that? We looked at some data from Gartner about massive amounts of AI projects failing before they reach production.

I think the reason for that is you didn't do this crawl, walk, run up the ladder of trust. What does that look like in developer tools where they still want autocomplete and be writing code themselves? They didn't get past that kind of I want to chat phase. Then they jump to, wait, I just saw this talk at this QCon where I can go and have swarms of agents go do things. I jump straight to that. I haven't built the muscles about how do I understand that these things are going to have hallucinations and these things are going to have interesting failure paths. If it didn't establish that trust at that foundational layer, then I wouldn't be able to orchestrate them. In fact, Gartner talks about governance gaps cited in this study. I think that's just like a polite way of saying we didn't design any trust into the system.

Because I do think the capability for trust exists, but the infrastructure for earning trust is something we haven't put enough time into. I also think, again, there's this asymmetry in the ladder that's underappreciated. The rungs have this feedback loop and the speed, and the higher the trust is, the longer the feedback loops, those are actually moving at opposite ends of each other. You have to have more trust at the rungs where you have less immediate evidence about what's going right. That's an interesting dichotomy.

Three Failure Points

I think there's three failure points that I've tried to bring out from what we've seen in terms of teams trying to adopt more and more of this trust. I think each one manifests at a different point in that. Then, again, it affects everything above that if you're not able to accomplish it. Let's go through those one by one. The first failure point is what I call context construction. I think this is where the most friction happens, because it's all the things that happen before you send the request to the model. Like which files are included? Like, should we index the codebase or not? Should we do semantic reasoning against the codebase? Does the model understand the relationships between things when it's looking at a problem? Have I stated the problem well to the model? It's like a model that I've given one file, and a model that can reason across hundreds of thousands of tokens and actually understand a representation of the project.

Again, when autocomplete's bad, if it suggests something that doesn't make sense in the structure, that's not a bad model. It's typically bad context. Like have we given the model enough relevant signal, or did it have the wrong signal? Or did we let it have the wrong signal for a long period of time, and now we've got this polluted context that we're not taking care of. If I say, where does the auth logic in this project live? The agent starts to walk down the wrong path, then that retrieval failure can have impacts for the rest of that chat. Because the context doesn't have the right information, it's going to start making guesses. Then by the time you get to, I want to have an agent go run by itself, or I want to have orchestrators run through the code, if that construction is broken, you're going to just multiply that failure.

If it's working from this incomplete set of context, then it's going to be much harder to walk it back and get it to deliver you meaningfully working code at the end. I think this shows up in the usage data indirectly. Developers are stuck with that heavy chat, but haven't started to use agents, consume a lot higher token counts per request. We see the tokens per request growing. They're doing this manual thing of pasting in a bunch of context themselves, or walking down the road when the context isn't right. I think that signal, for us, means that if context construction doesn't work right, and the developer doesn't know how to solve for that, they try to start patching it manually, which could work, but if the task is big enough, you can't solve that problem. Again, if you give it the wrong context, you can't get out of that.

I think there's three things to get right here. One, context windows have to scale with the task complexity. Here, people talk about models and the size of their context window. I think what we put into that context matters a lot. What is a good interaction versus a bad one can often lead to, if the tooling treated all the same context the same, you don't have that trust built. I also think that stale context is a real thing. We've seen folks that build semantic models on top of their codebases, and then leave it, and then it's almost worse than just letting the model explore on its own. Because even if it's just a little bit old, the agents having a map that doesn't go in the right place is going to eat up a lot of time and trust. Then I think the other key is relevance versus recency.

There's often like a recency bias towards, this is what changed most recently, but that doesn't necessarily mean it's most relevant for the task that we're on. The retrieval strategy that you have in those environments is very important and can produce different things. You have to have this distinction between like, what's the right bet to make for the agent to give? I think teams that have solved for this have really understood, and again, gotten that feel and trust, and they're using rung two of chat, or starting to do agents, for how can they understand to build that context correctly and give the agents the right context?

I think the second failure point is model routing. I started to talk a little bit about that with the context size of the model. I think the mental model that a lot of organizations are using today is, let's just pick the best model and deploy that, and then we're done, and we're just going to use that model for everything. If there's a new release, maybe we'll revisit it, but that's all. Or maybe there's like a little step forward where we have one frontier model for heavy work and something cheaper for lighter tasks, and we just have the one provider, it's a simple deployment. Again, I think the observed behavior that we've seen from early adopters is very different behavior. In April, 68% of folks that were manually selecting a model used models from two or more different AI labs in the same month. That was actually an increase over March.

Forty-five percent of those developers switched models in the same hour. That is like inside of active working sessions, not just experimentation, they're actually making a conscious choice to change models. Then among enterprise developers, people in like our structured managed AI development environments, 71% of them ran models from two or more different labs. Those folks that have come to us and want the structured environment are actually doing it so that they can access more than one lab. That number was up significantly from the first quarter of this year. I don't think it's like a small percentage of folks that are on our platform that are experimenting. I think it's the dominant behavior pattern that we're seeing from those folks who, again, may be an experimental group selected for, but are doing those experiments very intentionally. This multi-model development is what we're seeing these developers actually do once you give them access to multiple models.

I think there's some reasonable questions to double click in on here, like, are they exploring because they can or because they should? What we brought here is the usage intensity versus model depth and lab depth. If you segment by the intensity of usage for users here where we define them as light, regular, heavy, and power. The light users that did 100 requests a month, average 1.5 labs, 34% of them are multi-lab. Regular users, it's 100 to 1000 requests a month, is how we define that, 76% of them use models from multiple different labs, an average of 2.9 labs. Heavy users, which we defined as 1000 to 10,000 requests a month, 89% of them were multi-lab, averaging using models from 5.3 different labs. Power users, which was over 10,000 requests a month, 92% of them were multi-lab, averaging 7.6 labs. I think that pattern is like a learning curve, more than random experimentation.

I think the more intensely someone uses AI and understands how to use it for development, the more multi-model they become. It's because they've discovered that the real task completion for different models give them generally different, better behavior, whether it's designing frontend, or coming up with an architecture plan, or doing things with a database. These different models have a different feel and they have a different trust barrier for that. We see the same curve. This is for our enterprise organizations, so organizations that are using our tools in the enterprise, it's a very similar or basically the same curve among enterprise developers versus just our population of all developers. I think that's why we've seen the market diversifying to match this. We haven't seen consolidation. The top 10 models in our dataset by request volume came from six different AI labs. On some days we would see MiniMax-M2 Pro be the most requested model, not one of the premier labs.

In fact, we've seen huge uptake in use of MiniMax and StepFun models and xAI models. No single model on any of these days ever exceeded 37% of total requests. I think it's really interesting because I don't think this tells a story of a single winner in the model market. It tells us like there's an ever-expanding ability to use more in different models.

Then there's one other breakdown here that I wanted to show about like, how do folks think about the cost of the model? Thirty-eight percent of those that do switch models often are switching between free and paid models. We offer a set of free models that typically are ones that are just released and the lab is incentivized to grow the user base. Ones that are in stealth, where the lab is collecting data on usage for building the model. Then also some that we just have the ability through our scale to either subsidize ourselves or work with model providers to share the subsidy of. It's interesting here to see that 33% of that time folks are actually doing a paid comparison. There's two different paid models between two different maybe frontier labs. This is theorizing, like checking between which model maybe is more expensive or less expensive to do a certain type of work.

Then, again, we have a huge early adopter benefit, where folks are trying these different free models to see which ones are the best, but at the lowest possible cost. I also think there's a business continuity dimension that I think comes in here and is critical for some of the enterprise customers we're talking from. As the competition between labs is heating up, I think the policy changes, the infighting and acquisition and all those kinds of talks about consolidation are implications that enterprises are worried about. Over this time, we've seen this big decrease in the share of requests from all the frontier labs. Even as their usage grows still, Anthropic's share of our gateway requests dropped from 11.2% to 6.5% in the eight weeks leading up through April. The total users and request volume grew 3x. It's not necessarily that someone's losing. Again, I don't think there's one loser or one winner here.

I think it's showing us this maturing across using multiple models. Decreasing the concentration risk and doing that. The teams that already were doing this experimentation didn't have to worry about this, but the single provider teams were thinking about it more. I think that design implication for all of us is straightforward there. If you're building an AI tooling for an engineering team or another knowledge work team, I don't think you can assume a single model. You have to use some observed behavior of users, understand how they can either work with or around your tooling, and graduate up that ladder.

Then the third failure point, I think is maybe the hardest to fix, and again I've broken it down here based on our rungs to help talk about it. I think this also doesn't really show up until you've solved for the other one, which is, what are the feedback loops? We talked about that feedback speed. Again, I think it applies differently depending on the agents you're building, but for us autocomplete's immediate. Chat is seconds to minutes. Single agents is probably minutes. The loop's getting a little bit longer. Then when we're trying to orchestrate, we're trying to let agents do things on their own, that feedback loop gets a lot longer. I think that creates a pretty key problem for trust building. The whole model of like, if you just use it enough, you'll build trust with it, which is probably pretty critical. It breaks down if the feedback loop's that long because you're not using it that much.

It's that much longer to get that kind of usage experience with it and you can't iterate on the trust as quickly. In the data that we've seen, it creates a specific problem where there's a meaningful percentage of developers that we see who are comfortably sitting in the chat rung and don't really want to move forward. I don't think it's because they failed when they tried to adopt agents. I think it's just when something went wrong, they didn't have enough signal to understand why, and so they just went back to what they were comfortable with. I think the distinction matters because most of our discussions around a failed agentic project deployment talk is on, what did the agent do wrong? How did we design the agent wrong? I think the more useful question in most cases, and I could be wrong, but is like, did we design the right feedback loop?

Was it appropriate for building the trust as we started to give the agents more and more autonomy? The problem often isn't the agent itself, it's like folks had no way to evaluate if it's doing what we want. We don't have intermediate signals to take something that turned into a problem but make it recoverable earlier on. In practice I just think these feedback loops are pretty critical. Your tooling has to surface what agents decided not just what they did. What was the thinking that the agent did? Not just the output. Of course, the output's important and we want to have the right output, but what context they had, what decisions were being made along that feedback loop are really critical. I also think the architecture has to match the rung of our ladder where, what is the amount of autonomy we're giving and how do we match that?

Because again I think it's a trust building experience that has to match this like how much trust are we requiring to give to the agent in order to get what it wants to get done. I think we don't want folks micromanaging the agents, but if we don't have the right signals, what I see is like the developer watches the agent work for a little bit, decides that they don't know what's going on and so they just stop it and restart or take over and do it themselves. Because they don't have the right signals to know, did I give it the right context? Is it walking down the right road? Those intermediate signals are really critical and replace that anxiety with you being able to build the trust.

Trust Is Measurable

I've been using that word trust, and I think it's a mushy word to use in an engineering talk. I do think it's measurable. I promise that I would talk a little bit about how I think we can measure it. I think the measurements then matter for how you design these feedback loops we were talking about. There's a couple of things we use that I'll give you as proxies, again that are specific to the AI agent use case, but hopefully gives you some idea. One is, at autocomplete rung, we look at the override rate. What percentage of suggestions get rejected? Then, can we tell if that's model related? Is it context related? Is it folks that just haven't empirically learned that that judgment is better than the suggestion? Is it bad pattern matching? That maybe is recoverable, but if you don't measure this then we're not going to be able to trace it back.

At the chat level we look at session depth and follow-up rate. Developers stop asking for follow-ups and will reject it or treat their first response as final. That can be a trust deficit sign for us. They have decided that's not worth iterating. This is not right. I'm not getting that value. For agents we look at session completion rate. Did the developer let the task run to completion? Did they interrupt it? What is the interruption rate for an agent session? That's like a proxy for us for that trust then. It's hard to distinguish like the developer correctly interrupted, it's like a bad run where it was walking down from the developer just got anxious and took over. It gives us signal about that feedback loop design. Have we exposed enough of the feedback loop to the developer to tell them when they need to intervene and when they don't?

Again, it's very asymmetric, this trust. Many positive experiences are required to build a meaningful amount of trust. Only one bad experience can really destroy that. We talked about that in the last slide. I think this also has a social dimension that is also underweighted in most deployment strategies for agents. What I think that looks like is, yes, it's often a social failure rather than a technical failure. One highly visible failure of an agent in front of the wrong audience, in our example would be a senior engineer who's already very skeptical of AI, or in a high stakes moment with an audience like if I was live demoing on stage, which is why I'm not, can create this narrative inside of an org that is going to take a very long time to reverse. If we read the story about AI wrecking prod or overwriting a database, that's going to spread a lot faster than like, yes, actually I feel like I've saved probably three weeks' worth of work over the last week with AI.

I think the counterintuitive implication there is you have to have the right rollout strategy. The right rollout strategy is probably not just give everyone access and let them figure it out on their own. I think the optimal approach optimizes for this coverage to build trust and have a trust building time. Start with developers who are the most patient with ambiguity. Moving to a more, they're motivated to iterate. I talked early on about our dataset being focused on those early adopters. I think that's how organizations have to think about adoption too. It's like you're going to have a set of folks who can build that trust and iterate and figure out the right way to build social proof for the adoption from skeptics. Again, depending on where you are in that rung of the ladder, the stakes are different. Unlike most technology, AI failures are very visible. It's like if you think about Waymo or other self-driving, it's like you hear about every self-driving accident that exists versus you don't hear about every single human driving accident that exists. Those visible agent failures make it easier for a team to remember that rather than the things that are actually working.

Then, I also think as you're building that like, do developers actually, for us, build trust over time? This is when we're looking at retention. Do developers move up that ladder? We ask questions all the time in our product teams about like, if we're looking at developers who used our autocomplete in the last 30 days, what have moved to chat? How many of those have opened our agent manager to orchestrate multiple agents? Those kinds of questions help us see it more as a continuum of adoption rather than just, here's a token leaderboard. Who's winning the token leaderboard? Then, lastly, this data that we have, how does it help us build? Again, hopefully that can help you think about how you're building your systems. That's treating context as infrastructure, not as a feature. We spend a lot of time thinking about the context that gets constructed. I think the well-designed retrieval system for getting the right context is the number one thing to get right as you're building up.

Two is we've designed for provider independence from the beginning. Maybe it's pretty clear to you hopefully that I think we're going to be in a multi-modal environment by default. That pattern is accelerating with the usage intensity. I think if your infrastructure assumes like a single provider or a single model, you're going to constrain your most sophisticated users from really using it, and going to create incentives for them to route around your tooling instead of using it. That's definitely what we've seen. The third is again, matching the right feedback loop to the right level of adoption. That's the design intervention that I think reliably gets you to understand what does this transition between adoption look like. I think those are the really critical things to consider.

Three Questions to Take Back

Then, lastly, I want to come back to where we started, which is, most of what gets written about AI coding tools today is about those capabilities, the benchmarking of what it can do, a great demo video. I think that has its place. Of course, that's critical for adoption. What we've learned over these 40 trillion tokens we've seen, is that adoption really depends on if the system can earn trust. That trust is a problem that we can engineer against. We walk through those failure points. I think they show up in usage data before they show up in user complaints. That makes them more predictable and solvable. If we look back at the developers that are real power users, it's not simply because they have access to better AI. It's because they've become better at choosing the right model for the right problem. They've solved these failure points.

They've built the trust at each rung, and really climbed that ladder of trust. That I think is the signal to really look for as you're building agentic toolings. I think that's all the kinds of things that evals and benchmarks, as much as I love them, won't tell you. It really takes a focus on those levels of adoptions to actually understand if you're being successful in what you're trying to do.

Questions and Answers

Participant 1: You mentioned the importance of working with multiple models from multiple labs. What about the different agent harnesses essentially, and what different agent providers have built in them and the different capabilities in each of them? Should we be using multiple?

Brendan O'Leary: It's funny, somebody asked me that, obviously I should just answer, no, you just have to use Kilo. I said, I wish that the harness was the answer, because then I could just sell everyone in the world my harness. I do think that the model matters a lot more than you think. I would encourage you to try multiple harnesses. Of course, this is my bias, I think the harness needs to give you freedom to try multiple models too. It's that combination of harness and model that matters the most more than one harness or one model. I think that the harness mattered a lot more even six, nine months ago, because the agentic nature of the model was a little still being proven. Early on or at the end of last year and early this year we're talking about how good is this model at tool calling.

I think that's now table stakes, that most of the models that come out of either a frontier lab or the large open weight model labs can all handle tool calling relatively well, which gives the harness less ability to differentiate on that specifically. The harness differentiation now is where it meets you. What's the surface that you want the harness to live in? You should choose the one that you're most comfortable with. Hopefully one that lets you experiment with models.

Participant 2: What area of a system do you find the most success with people adopting agentic vibe coding? Let's say that like, go build me Google versus go build me a web page versus maybe even something simpler of building an API route. I'm curious if you're seeing signal on certain teams or departments or areas in engineering that are getting more success than others.

Brendan O'Leary: I don't have a lot of data about this in this presentation but I've written some on our blog about it with some of our data. I found it interesting that for us internally and then for our customers it's some of the most senior engineers that have gotten the most value out of it. I have another talk that I call, From Vibe Coding to Agentic Engineering. I think that our senior engineers who really understand systems thinking and have a deep understanding of the systems we've built are able to do more with agents because they're able to know that I'm giving them the right context, that I can trust them to go do something versus the go build me Google, make no mistakes, is obviously not going to be successful.

Participant 2: A hundred percent I see that too. Do you even push that further? You're not seeing the signal of certain areas of systems that are more vibe coding friendly versus others, or anything like that?

Brendan O'Leary: I think there's been a huge focus on frontend from the model providers because frontend was not a great place. I think they've done a great job there. Dex Horthy has this great statement of like, you still can't outsource the thinking. I think if you have a well-established system, you can vibe against it really well if you already have the right guardrails in place. If I said APIs or REST APIs, I don't think that's a great answer because if you just give it whatever you want it's going to be really messy, versus if you have a really well-established pattern of APIs, it's going to be great at like, I need this next thing. It's going to be like this but a little bit different like this. It's going to be really good at following those patterns.

Participant 2: Piggybacking off of that, are you seeing any patterns when it comes to popular sub-agents, so like UX review, or QA? There are so many ways to slice and dice your sub-agents with coding. I'm curious if you're seeing just even one or two that stand out, like you are gun to head, I only got two agents that ride or die with me. Who are you bringing with you?

Brendan O'Leary: I think the models have gotten a lot better at reviewing even their own code. Of course, they don't have any real bias towards their code, so we added a local review agent at the end of a turn. I used to not use that at all and now I use it almost every time. I think that reviewing the code for a given goal is something that the models are getting really good at. The thousands of other skills and agents that exist, I don't know. I don't know that there's a standout there. I think reviewing code for intent is something that they're getting really good at doing.

Participant 3: What I had observed while also using OpenRouter is that depending on what provider you choose for a specific model, the performance changed, and that, of course, how their inference settings were set up for their vLLMs and all of that stuff. Is that showing up somehow and is that an angle that gets lost in some of the metrics that you are looking at? How are you compensating for that?

Brendan O'Leary: OpenRouter which is one of the backends that we use, it also has multiple providers. If you're talking about especially an open weight model, you can end up with very different implementations of the same model. Normally it's not something I think most people care about, but you care about it, so it's good. It's almost all we think about. What we've found is that people just expect it to work. What we end up doing is building a lot of filtering there in order to make it work the way you expect. Because there's a lot of things that once you start getting in an LLM, like how well are you doing caching, like all of that stuff then starts really mattering for the frontend performance to the developer and of course to anybody using that. That's then when we end up spending a lot of our time in thinking.

See more presentations with transcripts

How to Find Resilience Bugs in Systems that Don't Exist

Hillel Wayne — Tue, 13 Oct 2026 16:00:00 GMT

Summary

Hillel Wayne discusses how formal methods principles - such as property strength, state-space verification, and environment assumptions - help software engineers uncover critical architectural bugs early. He shares practical mental models like "adversarial non-determinism" to reason about concurrency, edge cases, and distributed system failures without needing complex mathematical tools.

Bio

Hillel Wayne is the author of Logic for Programmers and Learn TLA+. He helps companies build systems cheaper and faster through rigorous design and verification.

About the conference

Transcript

Hillel Wayne: I have a bit of a weird job. Companies pay me to find bugs in systems that don't exist, or more specifically, don't yet exist. They come to me with some idea for a system they want, but they're also afraid the system might lose them customers if it goes wrong, and they ask me to look at their description and tell them if there's a really bad bug somewhere. Growing up in extreme programming and agile shops, this felt verboten to me when I first started doing it. After all, we all know that we can only find bugs at the very end when we have the whole coding system that we can test, possibly in production. It's actually a relatively simple process that all of us can do. To show you that this is just even possible in the first place, let's walk through the thing doer.

The thing doer does a thing. It first checks to see if the thing has been marked done. Then it marks the thing as done. Then it actually does the thing. Would anybody like to shout out the problem with this thing doer? Yes, there, you found a bug in something that didn't exist. The problem with this is that we all know this is not how an actual system works. The actual system is going to look like this. Doing the thing might fail. If it fails, we have to retry. If we retry, we get to this point where we check the thing is marked done, and see, it's been marked done. Then we just never do the thing. That's a bug. The client might try to fix this by saying, what if we instead do the thing and then mark it as done? Now if we fail and we retry, we have it marked as done, so maybe we can actually get it finished.

This, though, has a new bug. It even has this bug if we even remove the possibility of it ever failing, this bug is the two-thing doer bug. If we have two thing doers, the first might do the thing, but before it marks it as done, the second one might start up and do the same thing, leading the thing to being done twice. This also has a bug, a different kind of bug. Who here has built anything that looks like a thing doer? Because this is an abstract version of a very real system, one that I see very often in our designs. I've also seen it fail many times in the real world. I have seen package senders that end up sending people two packages for the price of one, which is very good for the consumer but makes the company a little bit sad. I have seen scheduling systems that can end up accidentally scheduling two people to the same slot, and I really don't want to be shouting over somebody to try to get my point across.

Although I might want to listen to their talk, which would make it hard to give mine. I've even, and this was the worst I saw, something that allowed you to install an app on a device twice, which doesn't seem as bad as the first two. I'm thinking a very particular system where if it was trying to install something and you told it to install the same thing, it would first stop running and start randomly deleting files on the device.

Scope - Formal Methods

Hopefully I've established that this is something we all can do implicitly and intuitively, finding bugs in descriptions of systems. My work is an explicit version of this, it's called Formal methods. The way of taking systems, encoding them in mathematical notation, and then using tools to find errors in those systems by exploring the entire state space. I've been doing this for about 8, 10 years now, and I've talked about it a lot. I found out two things about this. One, any talk about formal methods can only have one possible conclusion, which is you should try formal methods. This is a really terrible conference talk, though, because most engineers do not need formal methods in the same way that most engineers do not need to learn graphics programming or geographic information services. It's useful to a lot of people, but not useful to far more people. Two, using formal methods has changed how I think about regular software in ways that I think are actually quite useful for people to understand.

Instead of just talking about formal methods, I want to talk about the ideas I've learned from this that I use in everyday coding. We're going to be looking at three different aspects, the ideas of properties, verification, and environments. Or, to be a bit less technical, what it means for systems to be correct, how we can verify this correctness, and what is needed to make correctness possible in even the first place.

Properties

Let's start with what it means for a system to be correct. To formalize this notion, we have the idea of a property. A property is something that should be true of a working system. If it is false, then the system is wrong in some way and needs to be fixed. Properties, notably, are Booleans. They are either true or false. They are not partially true. An example of a property is that we have 99.999%, five nines of uptime. This is something that is verifiably true or false. If we are down for more than five-and-a-half minutes a year, our system does not have this property and we need to make a more sophisticated system or migrate cloud providers, depending. This is a property. It is also a relatively complex property that is hard to verify. We are often interested in automatically verifiable properties that we can easily check.

An example of this might be that all the messages we send in our system follow the appropriate schema. I would recommend checking out Ian Cooper's talk. That is some of the things he was talking about, the ways of enforcing this property in his systems. Properties get much more detailed than that or much more low level. Even something like a unit test is a property. Testing that max of 1, 2, 3 as 3 is the property that a given input always gives a certain output. If your system does not give that output for that input, something is wrong and must be fixed. Our thing doer had two properties that we talked about at failing. We don't do the thing more than once. Or a safety property, something bad that we want to prevent. We do the thing at least once. Or liveness, something good that we want to make sure happens.

I said that properties are Booleans. This leads to a very interesting consequence. Take the property that a system can handle 10,000 queries per second. Think of the set of every possible system you could build that to some degree of abstraction satisfies this property. Now take the property of can satisfy two queries per second, every possible system that can satisfy two queries a second. The set of properties that can satisfy 10,000 queries is a strict subset of those that can satisfy 2 queries. If a system can satisfy 10,000, it can satisfy 2. We would say that being able to satisfy 10,000, having this property, implies you have this property too or implication. This is an extremely useful thing that comes out of mathematics.

Some other examples of properties that satisfy this is that if you have no memory errors, this implies you have no use-after-free errors. If you have exactly once delivery somehow, you have at least once delivery and also at most once delivery. Something like GDPR compliance implies that you have cookie banners on every page of your site. Notice that these properties on the left-hand side are harder to satisfy than the properties on the right-hand side. Also, if you know you have the property on the left-hand side, you automatically have the one on the right-hand side. This gives us a notion of property strength. The properties on the left-hand side are stronger than the ones on the right-hand side. Some properties are stronger than others, and this is really important for verification. Because if we have a property that is strong and we know it's satisfied, like a test that a user can buy something with a coupon, if this passes, we automatically know the system has all the weaker properties too.

You can apply coupons and you can complete orders. If you run this test and it passes, you don't even need to run these tests hypothetically. The other consequence is actually more important, the consequence of the contrapositive. Notice in this space that there are three distinct regions. The region that satisfies both 10,000 and 2 QPS, the region that satisfies 2 but not 10,000, and the region that satisfies neither. There is no region that satisfies 10,000 QPS but not 2 QPS. If 10,000 implies 2, 2 not holding implies 10,000 doesn't hold either. If a system breaks at 3 queries a second, it's clearly not going to handle 10,000. This is called the contrapositive and leads to something quite interesting. If we have a set of weaker properties and we know one isn't satisfied, like you can't actually add shipping, we don't even need to look at the stronger property, we know it doesn't hold either.

If you're trying to figure out what's wrong with your system, which is more helpful, knowing that this test fails or knowing that 70 of these tests pass and this one fails? Which will isolate the error more? The second one. It is pretty much universal that strong properties passing show that a system is correct, while weak properties failing show why a system is incorrect. This actually explains a lot of practice in how we verify systems. Think about the testing pyramid. You have end-to-end tests, just a few of them, and then some more integration tests, and then a whole lot of unit tests. The reason we do things this way, there's many reasons, but one of the big ones is that often we are trying to debug broken systems versus verifying correct ones. This means that the unit tests provide us way more information about what could be going wrong than a single end-to-end test. That's why we often want many more of these.

In our thing doer, we had two versions of the thing doer, two different systems, and two properties. We don't do the thing more than once, and we do the thing at least once. The next step, now that we have those properties, is to figure out how we can represent a system in a way that can verify them or show them wrong. As before, we can do this intuitively, but the goal here is to find some sort of rigorous process. Then we can apply it to way more complicated systems where this might not be as evident. The way we do this in formal methods is to first encode the system as a set of logical formula, and then encode the properties as a set of logical formula, probably linear or temporally logically formula. Then we try to run some sort of model checker, SMT solver, or proof assistant to do this.

We're not covering any of that, because again, this is not about using these tools. This is about thinking like these tools. How do we actually represent a system in a way that can be verified? The goal here is abstraction. We want to find the simplest possible system that covers the expected behavior. The simplest system will have fewer error modes than the more complex real-world system. At the same time, if there's an issue in the small system, there is definitely going to be an issue in the larger system. If thing doer is going to fail, the package sender is also going to fail and probably much worse. This isn't perfect, but it is a very cheap way to find very dangerous bugs quickly.

At the highest level, what is the actual state of the thing doer? I'm going to propose three kinds of state. The first is, since we're running an algorithm, the step of the algorithm we're on. This will be an enumeration, check, mark, do, and done. If we're on the check step, that means we're about to do check. If we're on do, we are about to do either do or fail. We also have whether or not we have marked the thing as done. This will be a Boolean. In the real system, it will probably be like a database record or a file somewhere or a date timestamp. A Boolean is all we really need to do this kind of analysis. Then, finally, we have how many times a thing has been done. It's not enough to make this a Boolean, whether it's been done or not, because we want to make sure it's not done twice, so it has to be an integer.

This is all the state we really need to represent the system. In looking at this, there's actually a really useful insight I already see here, which is that step and marked are machine variables. They are internal to the system that we design ourselves. Done, the number of times something is done, is a world variable. It represents some exterior part of reality. The difference matters a lot to us, because machine variables are entirely in our control, because we can design the system however we want, we can do whatever we want to these values. Whereas the world variables, we have very limited control over. We can do some things, but not anything we want. If I really want to, I can design a system where marked goes from true to false again. If number of times done is 2, I can't make it 1 again. I can never unsend the package.

We have our states, and now it's time to figure out what our system can actually do. The first thing to do is figure out what are the initial possible configurations that the system can be in. There is only one, that we're on the check step, we have not marked it as done, and we have not done it any number of times. From here we ask, what possible evolutions can happen in the check step? This is deterministic, meaning there is only one possible thing that can happen in this step. We go to the mark step. We have seen that marked is false, so we go from check to marked as opposed to from check to done. From here again there is only one possible step, we go from mark to do. Here though, we have two possible things that can happen, we can do the thing, or we can fail.

This is called non-determinism, when more than one thing can happen from a given point in time, a given state of the system. We can go to either the done step, where we also set the number of times things has been done to 1, or we can fail and go down here, back to the check step in a retry, where marked is true and it's not been done zero times. Once we've done this, we have to explore both possible states, figure out what can happen from those situations. Both of these are once again deterministic. From the right-hand state, we can try to retry, we go back to check, but we see it's already been done, so we go back to done. Similarly, over here, if we retry over here, we see it's already been done, so we go to the done step. If we try to retry over here, we go back here.

This is every possible configuration of the system. We call this the state space. Each of these individual things is a state, and each state has state. The word's overloaded. Every possible path through the system is called a behavior. I will try to use the word path because it's more colloquial, although I think behavior is the more technical term.

Verification

Now that we have our state space, we have to verify our properties. Starting with we don't do things more than once, that's a much simpler property to verify, because that is what we call an invariant, something that must be true in every given state. I would write it mathematically as it is always the case that the number of times it has been done is less than 2. If you look at actual notations, they use a weird squared thingy, but I'm going to use always to be explicit. To verify this property, we just have to go through every possible state and see whether it holds this property. For example, this state over here, we're done at 0. That holds it. Same with this state, same with that state, same with that state, same with that state. Every single state satisfies this, so this holds. Now as to the bottom property, what we call liveness.

This is not a property on states. If it was, we would immediately fail in the initial state, because in the initial state, we have not done it. It is actually a property on paths. In this top path, we enter a state where it has been done. Therefore, this path satisfies that property. This path on the other hand does not. If we go down to this state over here, this check true 0 state, there is nothing we can possibly do that would get done to be 1. Because at least one path does not satisfy this property, it does not satisfy to the system. There is the potential for a bug in the real world. V1 satisfies one property but not the other. That's bad. What about V2? In V2, instead of check, mark, do, we check, do, mark. The new state space looks like this. It seems from the state space that we now satisfy the below property. Even if we fail, on the do step, we go back to a step where it is possible to still reach the done step. It seems like this property is satisfied. If you leave now, you're going to be horribly misled.

Before we explain why you'll be horribly misled, we have to talk about the other property, the safety property up above. I said this fails when we have two thing doers. The question is, how do we use this process of exploring the state space on two thing doers? There's two aspects to this. The first is to decide what is a local variable to each process and what is global to the whole system. World variables, like the number of times something's been done, are almost universally global variables. They are shared between all processes and the external world. On the other hand, machine variables can depend. I will argue that these are probably sharing a database somewhere, so the mark is going to be a global variable. The step each process is on is local. Each one has its own step, and we start in check, check. Marked is a global variable that's false, and the number of times it's been done is 0.

The next question is, how do we represent concurrency? How do we decide which process happens first? That's just non-determinism. I can say either the blue process happens or the red process takes a step. From here, I can say red happens or red happens. I can just keep doing this. Eventually, I'll reach a point where the number of times this has been done is 2, following blue, blue, red, red, or red, blue, blue, red, or blue. We can say that this property up above is not satisfied by the thing doer mark 2. That is property checking.

In practice, though, we don't usually do things this way in our heads. The reason why is because usually we're not dealing with 12 states, but 800 or more likely 10 million or 100 million states. This is how the tools often work. This is something that can be done by tools. It works the same way. We take an initial state, then we find every possible succeeding state from that initial state, and then we repeat the process in every state we found. Sometimes this will bring us back to a state which we've seen already, which is fine. It just means that we have found a different path to the same state. Then we just keep doing this, hoping that our state space is finite. Eventually we have seen the whole state space, and we can check it for properties. I covered this brute force analysis because that's how tools work.

I think it's the best demonstration that this is something that is totally possible, not magic. Again, this is not how we do things in our heads. When you do enough formal methods, you start to develop an intuition for how things can go wrong. The way this intuition works is by taking advantage of something called adversarial non-determinism, or the gremlin. You take every source of non-determinism in your system, things like randomness, user inputs, concurrency, whether or not us-east-1 decided it should be better or not that morning, and at the abstract level replace them all with the gremlin. Whenever you have a non-deterministic choice, you ask the gremlin what should happen, and the gremlin always gives you the worst possible answer. You can get pretty good at doing this in your head once you've done enough of this, just always figuring out what's the worst possible thing that can happen.

This gremlin is why this system does not actually satisfy the below property. Because think about this, we start in the check step, we go to the do step. Then here comes the gremlin. We ask the gremlin, what should happen? Should we do the thing or should we fail? The gremlin goes, fail. Ok, we're back here. Then we try again and we go to the do step. We ask the gremlin, what should happen? Should we do the thing or fail? The gremlin goes, fail. Then we get back to the check step, and we go to the do step. We look at the gremlin, and we decide, no, this is just not satisfied. Simply put, if we have any kind of loop like this, it is possible for us to enter the loop an infinite number of times and never actually get to a valid state. That's the easy case.

The much worse case is this. This does not satisfy liveness either, even if we remove the fail state. How does that work? We get to the check state. We go to the do state. Then the gremlin pops up. We're like, gremlin, what are you even doing here? There's determinism here. There's not even anything that can happen. The gremlin goes, like, and you go, ok, gremlin, what's going to happen? The gremlin goes, meteor. Or more realistically, a server outage, or a timeout, or latency just being high enough that you don't get a response in the window you want, or any number of things that can cause a system to just not do what you think it needs to do.

Environment

This brings me to the last aspect of this talk, environment. Environment is all the assumptions that need to be true for our system to work. In the first version of the thing doer, we need the assumption that the system never crashes to guarantee liveness. This may or may not be a good assumption, but it probably is one we have to have. Even think of something like trying to click a clicker. You have to assume it always works because it is totally valid for it to have the error that you try to press a button, but before you do, it catches fire. Without the ability to say that things always make progress, you simply cannot have any properties at all. This still reiterates that properties are things outside of our control. Often, these assumptions come from other people's systems we don't have control over. If a vendor decides to switch from always consistent delivery of messages to eventual consistency, what can you do about that?

You can change vendors maybe, but you can't tell them to go back to the old behavior. This is a really defensible assumption. If we add in the fail state, we now need a much stricter assumption that do thing never fails. This is a lot less defensible. If I see a client say, "We're just going to make this assumption that this never ever fails". I'm going to be like, "Your system has a bug", just saying. This is something that might push us to using V2. In this case, in V2, we no longer need the assumption that do thing never fails. We could have a much weaker assumption, that do thing fails only a finite number of times. Even if we fail at this step a million times, we still, on the million and first, go past this. We're done. We have satisfied liveness. This kind of really fine-grained assumption is what formal methods is really good at. It really forces you to think about all the kinds of assumptions of so many different types that go into a system. I would also like to point out that do thing never fails and do thing fails only a finite number of times are very qualitatively similar properties, if not quantitatively.

This brings us back to strength and weakness. Some assumptions are stronger than others, just like some properties are stronger than others. No failures is a stronger property than finite failures. Strong consistency is a stronger assumption than eventual consistency. When we have stronger assumptions, we can have stronger properties for the same level of sophistication of our system. Correspondingly, weaker assumptions can be compensated by system design. One good example of this I think about is I used to work for a company where we knew for a fact that every single one of our users would stop using our software at 4:00 New York time. This made deployments extraordinarily easy. We could just turn off everything around 6 p.m., update it, and turn it back on. We had a very strong assumption that made it easy for us to guarantee deployment properties. Imagine Google doing the same thing.

Just imagine it. It would be very funny to imagine. Another thing, though, is that different systems require different levels of assumption for different properties. It's not monotonic. A system design doesn't make all properties necessarily easier to guarantee for the same assumptions. We see this in version 2 again. In version 2, we needed a weaker assumption to guarantee liveness but we need a stronger assumption to guarantee safety. We need to assume there is only one thing doer acting at a time. It might be possible for there to exist in V3 of the thing doer, where V3 does not require either of these strong properties. It might be able to work even with only finite failures, but also guarantee safety even if there's more than one thing doer. That system, if it exists, is probably going to be a lot more sophisticated than either V1 or V2.

I've found empirically that there's this relationship between sophistication of the system, the properties you want to guarantee, and the strength of the assumptions you have, where either one of these can be fixed and the other two change based on that. For example, if you wanted to guarantee liveness for the same level of assumption, of say a message being passed, like are all messages processed? You can actually do that, even with weak assumptions, by weakening the property. From all messages are processed to all messages are either processed or go into a dead letter queue. Similarly, if you look at something like a spaceship, just off the top of my head, a spaceship cannot even assume that the RAM is going to be consistent. It's constantly being flipped by cosmic rays and radiation effects. They have to have a much more sophisticated system for the same level of property.

One of the other consequences of this is that weakening assumption often weakens the properties you have. If you take a fixed system that works and guarantees all your properties and then weaken one of the assumptions that are currently in this operating environment, it may not satisfy the properties it started out satisfying. I find this often leads to an enormous amount of bugs in practice, where people build a system for an operating environment, and it works, and then the operating environment changes. I keep going back to this example because it's the one that I have the most vitriol over. When a client changes a database from saying, ok, we're always going to give you updated information, to, we're going to give it to you five minutes from now maybe, we'll be consistent at some point. We built that depending on it being always consistent, strongly consistent. When it switches to a weaker assumption, our properties start failing. Theirs might be satisfied, ours are not. Those are three of the perspectives I use very often coming from formal methods.

Summary

In a summary of what I've talked about, systems have assumptions and properties. These can be related in terms of strength. Some properties are stronger than others, and some assumptions are stronger than others. For a given property and assumption, the system sophistication depends on both these things. A system needs to be more sophisticated to handle a stronger property, and needs to be less sophisticated given a stronger assumption. Strong properties show correctness. When they pass, we know our system is correct. Weak properties show incorrectness. They're used to debug why things are going wrong. We can look at a simplified model to find a real-world bug in the system. Then, finally, beware of the gremlin. I hate the gremlin. Ultimately, a lot of this comes out of mathematical analysis of software. We don't necessarily have to do this mathematical analysis ourselves, but we can still gain the benefits of the things it gave us. We see this in a lot of fields, in formal methods, in functional programming, in type systems. I find the mathematics behind this all really cool, and I'm glad that it actually gives us so many benefits.

Resources

I want to talk about a couple of resources. If you've decided after this whole talk that's not about formal methods that you just really want to learn formal methods, I'm sorry, I failed. I do have this resource that I've been working on called learntla.com, which is designed to teach one such method called TLA+, which is relatively popular in the industry. It's pretty good for this kind of concurrent system and finding bugs in that. It's not the only formal method out there, though. There's not really like zealot wars about this. This isn't like Emacs versus Vim. If you come to me afterwards and you just talk about the system you have, I can tell you whether this is the most appropriate thing or recommend resources for another one, like Alloy or P or Quint or PRISM. Or you can just hire me to work with you.

I do that too. The other thing is, though, as mentioned, I've really been fascinated by how we can derive useful insights into everyday software engineering from mathematical perspectives. For that reason, I've spent the past three, four years working on this book called "Logic for Programmers". It covers formal methods, but also things like property and stateful testing, design by contract, SMT solving, logic programming. If this is something that might interest you, I've actually done the first printing proof to actually make it into a physical real book you can buy. If you're interested, you can buy the PDF of the current version, which will automatically update to the final version when it's done at this link for $10 off.

Questions and Answers

Participant 1: Do you find a difference applying the technique of formal methods in legacy messy systems versus brand new starting from scratch Greenfield projects?

Hillel Wayne: Yes, actually. It's a lot easier to apply it in Greenfield. There's this interesting thing that happens in legacy systems where it's easier to demonstrate the value of it. The reason why is because in a Greenfield system, you can't really argue that like any given bug it finds is actually a realistic like, we would have found that later. In a legacy system, you can be like, I modeled our old system that had this critical bug, found it in two seconds, and then show that, ok, now that we can establish it can find the real bugs that we have faced, let's use it to model past changes to that. That's actually how I got started doing this. There was like systems that were going horribly wrong, and we'd basically make a patch, and they'd go horribly wrong, and they'd make a patch and go horribly wrong. Then, when we were thinking of another patch, I was just investigating these methods. I was like, ok, I have managed to reproduce the past five things that went horribly wrong in 50 lines of spec. This new patch has something else that went horribly wrong. Let's not do it just yet.

Weakly: One thing that I'm curious about is, when we talk about these properties, when we talk about proving things, it's often easy to show it or to demonstrate it or to curate in a small isolated component. If you have multiple different components, and you want to reason about them as a whole, groups don't compose, properties don't necessarily compose. In the real world, what strategies have you found pragmatically for addressing that in a way that works?

Hillel Wayne: Generally, this is like a big question in the space. This is a big problem in any kind of space. Things can be independently correct, but interact in ways that make them all break. This is known as feature interaction or boundary error. There's a really great book by Nancy Leveson called "Engineering a Safer World" about trying to find these issues after the fact. That's really the best that we can do for the most part. In formal methods, the way that we can do this is by basically titrating the amount of abstraction we do. We can make a really abstract system of all of the components working together, and then we can do what's called a refinement to take one chunk of that global system and make a detailed version of that. Then plug that back into the abstraction to see if it behaves correctly with everything else still being abstract. You can do this progressively. It's just really annoying and really hard to do because it's really complicated, and you're dealing with multiple layers of abstraction. It's not fun, but it's possible. If this was fun, a lot more people would be doing it.

Participant 2: You were talking about correlation between assumptions, properties, and sophistication of the system. I wonder, sometimes it's just impossible to meet the properties if assumptions go weaker. Sometimes it's possible by making systems more complex. Do you have any tools in formal systems that helps to answer that question? Is it possible or not to change the system that way so it can meet properties with weaker assumption?

Hillel Wayne: Is it possible for methods to tell you whether if an assumption is weakened, it's possible to fix it with a more complicated system or if it's impossible?

Participant 2: Is it possible to fix it with a change to the system without lowering the properties?

Hillel Wayne: There's two answers to that. There's the theoretical answer and the practical answer. The theoretical answer is, no, that it's not possible because that basically gets into like, how do you prove anything? It is possible to weaken assumption in a way that makes the property always impossible to verify, but it's not easy to necessarily show that's the case. That's gets into like, what is a proof? How do we prove anything? In practice, I found it's very easy to run, for a given system, multiple layers of assumptions and changes to it. You can say, ok, these properties always hold in our system design, these properties hold if we have this assumption. If we have this assumption, which is weaker than this assumption, this weaker version of these properties hold, too, but not these stronger versions. If this property is weakened, this refinement of the system satisfies the stronger property. It is possible to do that through engineering effort, but you can't automatically deduce that something would be impossible.

Participant 3: Unfortunately, I mostly work in non-deterministic systems, like agents or machine learning models. How much of this can be applicable? Can you measure the potential state space of the solution space or something like that? Are there methods available that help find these failure states before the agent inevitably trips up into it?

Hillel Wayne: I think the challenge with agents is that it's kind of a non-determinism that we really haven't seen before. We think of non-determinism as being like A or B can happen due to like the confluence of real-world factors. I feel with AI, it's to some extent like, it could do A or B. Why? Temperature. There hasn't yet been a lot of work in applying formal methods to make the agents themselves work correctly. Most of the work has been in either making sure the code that agents generate is valid or making sure the pipelines that the agents work in are valid. Actually verifying the agents themselves, it's not yet been established how to do that.

Participant 3: I was thinking maybe something like if you've got the trifecta of you can have stuff that can be exported and you could find those links and the potential combinatorial, like if you combine three APIs and you give an agent that, you may not immediately get an error, but you've created the potential.

Hillel Wayne: I think that that can be thought of as like an error in the three APIs. Like if a human uses three APIs, could an error happen? If a poorly written program uses those three APIs, could the error happen? In that case, I think that is verifiable. The agent itself is something that I cannot like give you a professional answer on without misleading you and myself. I do not have the expertise to really speak about that. Agents and APIs, that's the newest dumb version of Dungeons & Dragons.

Participant 4: Moving back to the world of systems that are supposed to be deterministic, is there any work on deriving properties or assumptions from the source code or the comments, or anything like that?

Hillel Wayne: Yes. A couple of ways we can take this question. The first is, can we generate a specification from the source code? There's actually been some really good work of using LLMs to do that. The other thing is, if we have a deterministic system, it is possible to just prove it directly correct with the code itself. That gets into a space called the formal verification of code, which is its own entire can of worms. Like, you'd be using tools like Dafny or Frama-C or SPARK to do that. Either a dedicated language that can be directly proved to be correct, or an augmented harness that goes on an existing language, like Frama-C on C code, to prove existing C code correct.

Participant 5: I've seen in my workplace folks using skills to try and automate some of these initial bug searches and security reviews and stuff like that. Have you seen in your experience people successfully try to adopt formal methods through skills using AI? If you have, have you seen it go successfully or poorly?

Hillel Wayne: One of the things that's been really interesting about the new LLM age is that it's revealed a lot of revealed preferences to us. One thing it's revealed is that a lot of people really want to ruin other people's day by putting slop on the internet. It also has shown me that a lot of people really want to use these formal methods, and they're just intimidated by the skill threshold required to get into it. There's been a lot more specifications, both of code and of systems being written with LLMs, which is a problem because they're really bad at it. The biggest problem with LLMs and doing this stuff right now is that LLMs are just really bad at figuring out properties. They cannot do liveness at all, and often their safety properties, their invariants are things like, if X and Y is true, and I know X is true, Y must also be true.

That, again, might change. These things get way too good, way too fast, and my answer could be wrong. Right now, it requires somebody who's already experienced in doing this stuff to get an AI to do it well. If you are experienced, it does save a lot of boilerplate and a lot of pain, and I love using them. I feel bad about loving using them.

Participant 6: You mentioned how your formal methods background helps you to identify weaknesses in the systems. Did you build an intuition how to build correct systems from it as well, I wonder?

Hillel Wayne: I think so. For example, one thing, I now hate any kind of graph data with a burning passion. The cycles, they're everywhere. I've definitely also noticed that there's more of like an intuitive sense of like, ok, this is going to fail because this has a race condition, so how can we design the systems that way? This race condition doesn't matter. I feel like I built that intuition, but it could also be the case that I've built that intuition, but it's totally wrong and I'm just overconfident. I don't want to say like you too will become perfect at distributed systems if you do this.

Participant 7: Is this a time-consuming effort?

Hillel Wayne: Yes. It is time-consuming, but with the goal of being money-saving and either a few days of specifying, saving you a couple weeks of development or saving you a couple weeks of trying to tell an angry customer, "No, please do not cancel. I'm sorry we lost your data, but we won't do it again". It is a time investment.

Participant 7: Do it early then you can save money.

Hillel Wayne: It will be time-consuming.

See more presentations with transcripts

How Eve Online Leverages Head Based Sampling to Observe "Fun"

Nicholas Herring — Mon, 12 Oct 2026 16:00:00 GMT

Summary

Nicholas Herring shares how EVE Online tackles distributed tracing and observability across a 20-year-old MMO architecture. He discusses leveraging head-based, dynamic exponential moving average (EMA), and deterministic sampling techniques to maintain high system fidelity, control telemetry costs, and avoid performance degradation under high-throughput real-time game simulations.

Bio

With 20+ years spanning military and gaming tech, Nicholas Herring built patented distributed systems linking tactical and game engine networks. He's led R&D in real-time human bioinformatics and delivered multiple cloud-scale game titles. Currently, he applies his expertise in real-time simulation and architecture to modernizing the massive universe of EVE Online for CCP Games.

About the conference

Transcript

Nicholas Herring: Today I'd like to go through how we adopted observability into a system that's 20 years old and also has newer mechanisms evolving underneath it as well, and go through how this differs from the video game architectural space versus traditional application development. Compare a little bit of that with FinTech as well because there are some similarities there. Ultimately, we'll talk about how our system is 100% head base sampling, which makes a lot of people clutch their pearls and gasp, because we don't use tail base sampling which means we're buying in a lot of those tradeoffs. We'll talk about what those are. How we go about doing that by blending dynamic and deterministic sampling techniques, the blessing and curse that is exponential moving average sampling for dynamic sampling, we'll dig into that a bit more. That's the crux of the tricks that we're using. How we ultimately use all of this to find a balance between good enough for fidelity and cost.

Traditional vs. Video Game Architectural Patterns

Before we get into that let's first dig into a bit of the differences between traditional and video game architectures. Most traditional software is creating a unique experience for a single user. Very rarely does traditional software build a shared experience with many users at the same time. A lot of that is obvious in how we build and scale out these systems where there's a single, everyone has their own web page that they experience, it's not the same web page that everyone else experiences. In contrast, multiplayer video games, they have to create shared experiences. To create these shared experiences it means we have a lot of the problems that FinTech has in weird different ways in the sense that we have to synchronize a lot of data, it has to be transactional, and we have to be able to replicate that data so that everybody can experience that shared experience at the same time.

On top of that, in contrast to FinTech is the concept that video games do a lot of stuff that the system doesn't really need to care about. Simple waving at somebody in a video game, that's just an expression of that player, of what they're doing in that virtual world. For EVE Online we'll go over some of those bits as well. The video game industry has always been the fun uncle that has a hot rod where it's cool to build those and it's fun to take to the grocery store but most people look at and go, why would you do it that way? The main reason for a lot of that is building this ability to have an expression in a virtual world and communicate with others in various different ways that you might not get to even in real life.

What is EVE Online?

Before we hop into more details about how we actually monitor our systems, can I get a sense of how many gamers do we have here? How many of you have actually heard of EVE Online? I'm going to go over some details of EVE Online, what it is, and where it's at right now. Internet spaceships are serious business. Ultimately, it's known for its epic espionage stories. If you don't know about EVE Online you've probably heard about it through, I think the biggest press that we've gotten before is like a CNN article on somebody stealing hundreds of thousands of dollars' worth of assets in-game. It's also important to know that it's a single sharded MMO, which means that we do not have a North American server, we do not have an APAC server, we don't have a Europe server. It is all a single server where everybody experiences the same universe at the same time, which is the again uncle with the hot rod grocery store thing.

Because why would you do that on purpose, that creates all sorts of technical challenges. I have a separate talk on that all together. The other thing to keep in mind is that it's a player-driven ecosystem. There's probably about half a million transactions on our market per day, and that represents about 12 billion items that move around in that ecosystem. Keep in mind this also counts literally every single bullet that's in the universe, and that's also something to keep in the back of your head while we're talking about this, because also we have to talk about tracing and all of the bullets and lasers and missiles flying around in the ecosystem and how we deal with that, and we'll get to those. Just to give you a sense of the size of EVE Online and how it's been working, we hold a few Guinness World Records with the most costly battle that happened in the game costing almost $400,000, the largest PvP battle over a span of time with almost 9,000 players in it.

Then the most concurrent players in a multiplayer video game player versus player battle at 6,500 players. That is 6,500 players all in the same place trying to punch each other in the face constantly. That gives you a sense of scale of EVE Online and connect that back to the systems that you guys are interacting with and building in different contexts, traditional software. Whereas you can focus on the session of a user, we have to focus on the state of a solar system for everyone that's experiencing that solar system.

20 Years of EVE Development

The other piece of EVE Online is this is a very old project. The development started before 2000, so late '90s is when the actual development of it started, the first versions of EVE Online started. It has a little over 2 million changelists that are real. I'm trying to factor out robots. It has about 1.7 million lines of C, C++. I also have this unit of Unreal Engines, comparing the line count to that gives you a good idea of the size there. EVE Online was built with a bucket of Cat 5 and a firehose of SQL procs. We also have 2.6 million lines of SQL that is mostly a majority of all the actual legacy systems running inside of an MSSQL server. Then the wonderful folder that we have that's called script that is 3.4 million lines of Python code. The other joyful part about this is it is stackless 2.7 Python.

We have a longer project to get to 3. What stackless 2.7 Python means is that we run on one logical processor. Not one core, one logical processor. We do one thing at a time. That comes into play when we talk about sampling. If that didn't bother you, the universe is defined in roughly 53 million lines of YAML. This is one of my active personal crusades in the organization to make that number significantly lower. Then we're going to talk a little bit about the evolution of this. I have separate talks that I've done at GDC as well, where we talk about the evolution of where EVE Online's backend tech is going. It's moving away from this monolithic structure of these things that we've listed out here into an umbrella term that we call Quasar. All Quasar is, is a term that we use to talk about Kube, Golang, gRPC, protobuf, RabbitMQ, NATs, all of those fun things that everybody's been working with in the past decade or so.

These two systems combined, the legacy system, and the newer Quasar ecosystem, they do roughly about 30,000 messages a second. By messages, those could be anything from events to requests to responses to also what we call notices, which is information that we push to the clients themselves. We've built up this message bus ecosystem that started our journey of building domain services, because I think microservices is a bit of a dirty word. Because it's been around for a while, and no one can still tell me exactly what they are. We work in domain services, where we have the concept of these are users. These are ships. This is the solar system. Those are the services that we have that are operating in this. We also use those in the sense of features as well. How we build up things, like we have a system in EVE Online that's relatively novel in the sense that it allows players to create quests and missions for other players. That is its own service and ecosystem in and of itself.

The In-Game Experience

Let's take a look at what all of that on paper looks like. This is a battle that happened about a month ago. I just grabbed something off of Twitch from one of our wonderful community managers. In here, we can see that there's around 300 people in a battle. There's two different fleets, they've organized the UI, they have red versus blue, as is tradition. This gives you an idea of how we have to deal with the aspects of the expression of the players. Now, you can't really see the expression of the players, because at this scale, things blur together, and you can't really get into the fine details. If you were to look closer at some of these, you get to see what our players start expressing on their own. We have a system in the game called Skinner, yes, very creative, where you create ship skins with a whole design system.

Players have been using this to express different uniforms or different feelings in some cases, like a mood ring for a very expensive spaceship. These can be expressed in a way where we've tested this system, where we had 500 people in a local test system, and we probably had, we're going to say, three Raspberry Pi potatoes running somewhere. Then trying to have them all switch their ship skin as quickly as they possibly could. We turned off all the rate limiters. Then that in and of itself generated the same amount of messages as the entire universe, so about 50,000 a second, when we had 500 people switching those. I'm highlighting this because it also gets back to the importance of being able to express yourself in a virtual environment, and the problems that we need to solve in order to do that. This became a way that players could then communicate that was different than chat or shooting at one another. This became yet another venue where they could express themselves in a certain way. There are also different variations of this that we have. You can use, we have space fireworks, of course, and various other things. I think we even have snowballs in some winter festivities.

When we start thinking about this a little bit more, that's all the fun and games part. That's all the fun bit. We start talking about where's the motto that our organization has, which is making virtual games more meaningful in real life. Take from that what you will. That is the idea of where EVE came from. The general original statement for EVE Online was that death is a serious matter. To illustrate that, part of what we have in EVE, one of the biggest ships we have in EVE are called Titans. Those are in-game probably around 16 kilometers long and roughly around €1,000 per ship for all the bill of materials and fitting them and munitions, and everything that goes into it. If you translate all of that from in-game currency and translate it to real world currency, it hovers around about €1,000 per ship. There's gravity in that because if that ship blows up, it's gone.

They don't get it back. There's no way to get those back. There is if you complain to a GM maybe and get lucky, but highly unlikely, they've been highly trained to ignore those requests. Then on-screen here, we have an entire bubble of them, probably around 40, which represents $40,000 in real world currency that's on screen. Now you can understand why our biggest battle cost roughly $387,000. Because on the field of that battle, were roughly 257 of these enormous ships. That's what we can understand is at stake. At the same time, understand how we allow players to express themselves in the ecosystem. Like this one guy who brought out his $1,000 ship at the top here and decided to be fabulous in all pink. I don't know if you noticed it yet, but he's right there up top. Ultimately, as we come back to what we're here to talk about, is how do we then differentiate between these things that really matter, like this $1,000 worth of imaginary internet spaceship versus a pink color that pops up on the screen? Where do we track those and how do we manage that?

Head vs. Tail Base Sampling

The first part is talking about head versus tail base sampling. Real briefly, the decision that we made coming into that ecosystem from a 20-year-old project is around the fact that we didn't want to take on any other extra maintenance burden. Another big part of that is what I talked about earlier, where it has to do with the capabilities that we had. Again, Python on one logical processor. I can't even get the spans out fast enough to do tail base sampling without killing the rest of the simulation. There are ways that we could invest and optimize that, but at that time, that just wasn't on the table. We leaned in more into head base sampling versus tail base sampling. It also led us down an interesting path of simplicity over complexity. When we did this, we also thought about the centralized concept. There's a central element to tail base sampling where you have to send all of your spans to one place.

My bias against that is that I don't like to go blind in big chunks of my ecosystem. We just didn't do that. We don't even have a relay for emitting the spans that we have now. What that does allow for is when the system starts to degrade, we can infer which part of the systems are degrading, because only those things can't get to, for instance, Honeycomb, for that example.

Deterministic vs. Dynamic Sampling

Then we leaned and looked at more of the deterministic versus dynamic sampling. Again, because of performance, actually doing dynamic sampling in an ecosystem where you already have this 20-year-old fine-tuned machine that is simulating a whole universe, we can't really go in and start working on things that are more dynamically sampled. We were forced to look at, we need to have that deterministic on that side of the equation. Then the newer systems will be inside of Quasar. We'll have more dynamic sampling in that case. This also became super handy for us when we started thinking about, what if we had our desktop clients actually emit spans as well? There's a lot of performance data that we could understand from that and understand there as well. Again, same problem. The desktop client also operates as this monolith that is well-tuned for what it needs to do. We didn't want to affect that either. This is yet another thing that pushed us more towards head base sampling.

Exponential Moving Average (EMA) Sampling

Then we get into the exponential moving average, or EMA sampling. This is a huge part of what we did with how we trace a majority of the newer systems in Quasar. Briefly, what EMA sampling is, it basically is an algorithm where you select a key for what you want to track. For us, it became the type URL for the protobuf messages that we were slinging around everywhere. That became our key. The interesting part about that is it becomes your key space. The key space will be important in a bit as well. As your system evolves and expands and you put more and more things into that ecosystem, that key space starts to grow. That is one element that you have to keep in mind when you're tracking how much information you're actually going to emit as a span. The other thing that EMA does is it will dynamically look at that key space and track how many times it's seen each one of those keys.

It does this really cool organic thing right out of the box of the algorithm of the more I see something, the less I sample. The less I see of something, the more I sample it. Which is great because that solves a majority of the original problem that we had, which was, I don't need to care about 500 people switching their ship skin constantly. If that's constantly happening, or bullets flying around the ecosystem constantly, I don't need to worry about that. EMA will turn that down. The problem then becomes the bullets aren't fired from the new system, they're fired from the old system, so now we have to deal with deterministic sampling in that regard. The other aspect of EMA sampling is that none of it is coordinated, which means that unlike tail base sampling, you're funneling it all to one place. Whereas with EMA sampling, specifically head base EMA sampling, you are looking at distinct instances of ingress into your ecosystem.

Then you have these three key components, which is the frequency of messages, the key space, and how many instances of that ecosystem are running. When you put all those together, there's this interesting thing that happens where you get this striation of the data, where you get this high watermark that slowly works its way up into your ecosystem. What I mean by high watermark is once we scale out, so this graph, I threw a bunch of numbers in to illustrate this. If we talk about the Quasar aspect of EVE Online, that's roughly 30,000 messages. We have a key space size of probably around 2,000 messages. We have 35 instances for ingress into that ecosystem. When you throw all those numbers in, there's this interesting thing that happens. For us, we end up averaging out around 60 million events a day, which is great. That's a nice, cheap number in the grand scheme of things.

It allows us to stay in that wheelhouse and understand what's going on. It also lets us know that if we need to scale up in those instances, we have to be wary of where the next high watermark will be, because that will be the part that we can't go back below. Back below, what I mean by that is it means that no matter how my frequency changes or how I control the sample goal. EMA also has a sample goal. The sample goal is basically, what is it trying to target for that singular instance? If its goal is 1 in 3,000, it will try to do that for its key space, for its frequency, and in its own instance. If I try to make the sample goal universal across the system of 1 in 3,000, it actually can't go below our system because of having 30, 35 instances, and having the frequency and having the key space that we have, having the goal of 1 in 3,000 per instance prevents it from dropping below around 60 million a day.

Because it will always sample something, and it will never leave anything out. You will always see at least one of everything, which is also a good thing, and we'll talk about it a little bit later. I also try to slice and dice this from a different view when we're talking about key spaces. When your key space grows, a lot of weird things happen with EMA in a distributed mechanism. In that sense, it gets more interesting from the perspective of this is very unlikely, because this would require your teams to be generating hundreds of new types of messages very quickly in the ecosystem, which at the human rate, that doesn't necessarily happen. The AI rate, that might be a problem later. We'll find out.

All of that coupled with the hybrid configuration that we talked about, EMA is very useful for the dynamic sampling that we have, but also using the deterministic sampling. One of the most important parts about using deterministic sampling, and also with dynamic sampling as well, is having tooling that can show you what the sample rate is and also extrapolate what those numbers might mean. This is a screenshot of the samples that we have from our simulation frames. Our simulation engine is a deterministic lockstep simulation engine that we call Destiny. Hilariously, there are other parts of it that are named after various other Destiny Child members. That's a different talk as well. With these, because so much is happening in that simulation frame and so much is happening in that system, we have to sample these deterministically. One, because of performance. That's not where we want a performance hit.

Two, there's just so many of them that we can't really care about all of them. At the bottom of this graph, what you can see is the wild ranging sampling rates that we have set up to deterministically sample these mechanisms that are happening in the simulation state. The more important part is what is happening in the top right, where Honeycomb is taking those sampling rates, and the frequency of samples that it has, and extrapolating those out. We use this quite often. I would argue that it's even with rates like this, and sparse data like this, we've seen it usually be within ±300 units of whatever the thing is. It's actually pretty reliable when you're dealing with statistically significant information.

Feature Flags

Then another trick that we have is abusing feature flags. We, again, hot rod grocery store, we use LaunchDarkly to jam a bunch of JSON everywhere. What we do here is we have a prefix mechanism where it allows us to express two different concepts. One is that we can say, I want to know more about this message. Or, I want to make this domain much more quiet, because it's less important to what we're doing. If you look at the last line there, that is one that we definitely don't care too much about. It is ultimately what window is in focused, open, closed, focused, unfocused, when they moved it around, drag and drop, all sorts of stuff. We care that those are happening. We don't care about all of them. Again, those are statistically significant. From the perspective of a client, 1 in 1,000 is pretty sparse.

From the perspective of 30,000 clients all doing 1 in 1,000, it's much less sparse. Then we also have other pieces of the ecosystem that are just more frequent, where harvesting is any concept of mining, whether that be mining lasers, drones, gas harvesting. There's a whole ecosystem of minerals and industry inside of EVE Online. We use this quite a bit to focus on what we want when we want to. I think that's another important part of what we do. With these tools, what we also do is we have a mechanism where we can look at the activity of a specific ship or a specific character. Then when we turn these things on, our quota graph looks like this, and you'll see the two spikes in the far right. When I read this graph, it looks like this, because it means somebody was looking very intently at something specific going on in the universe. These are a collection of all the tools that we have of what we've been working towards and what we've been pushed into from the sense of performance, legacy, the evolution of the newer ecosystems, and how all those pieces work together.

Takeaways

Then, ultimately, I want to go through some takeaways here and reiterate some of the things that we've been talking about. The high watermark is very important, because it allows you to manage cost when you understand what the output of that high watermark is. This is a screenshot from a website that has been around for quite some time called EVE Offline. It has a great chart for how many players are playing EVE Online in a given moment. As you can see, with EVE Online, we have, some call it a luxury, depends on who you talk to, but we have a very stable player base. We have spikes where we go up from 30K to 40K, depending on if we release an expansion and people get really excited about content, these types of things. Ultimately, we don't have to really worry about a huge burst. The importance of the high watermark has a lot to do with the behavior of our product.

That allows us to do a lot of planning on how much we're going to spend, what we can use to target, and how we can instrument more detailed information. Another technique we have is targeting 90% quota utilization. If you look at this as well, you can see that we're roughly coming in at 50 million a day. That allows us to, again, turn on the Eye of Sauron when we need to and not worry about doing big overages. There's also a lot of cool toys that we have with Honeycomb for like burst protection and those types of things. Another big part and purpose of why we smashed all of these techniques together is that we wanted to make sure that we have at least one of everything. Again, we want to know that a bullet was fired somewhere in the universe. We definitely want to know that happened at least once, because then we can extrapolate that based from sample rates or whatever the case might be and we can infer the health of the system. We can also then figure out what's contributing to actual debugging versus what's just noise. Again, the focus on identifying what's statistically significant versus what are facts of the universe that you really need to keep track of.

This is more just a detailed note on scheduling flags. Sometimes we forget to turn off the Eye of Sauron and then we get a lovely little automated message saying, you've just spent a lot of money. Just making sure that everyone has that on their checklist if they're looking at tackling this problem space. Then we talk about the implications of the public application instrumentation. We talked a little bit about the actual desktop client. We've also looked at this in our launcher as well. We're probably just going to stop doing that all together from a pure technical perspective. Because, ultimately, what ends up happening is because of those systems, we then make sampling decisions much earlier in the trace that affect what we can and can't see. A lot about a video game has to do with what the interactions of the player are. If that player is then doing something that's very important like anchoring a space station, we potentially won't see as many of those because they are 1 in 1,000 sampling because they're coming from the client, which means none of our tooling that we have allow us to crank that up because we don't want to affect the actual performance of those systems.

We want to move that decision-making process from the actual clients themselves to a trusted network so when it hits our ecosystem in our networks, and then start the decision-making and sampling there. This is something that we've learned over time. It gave us some cool insights as to latency, but that's something that you can get a different way. You don't need spans to get that from 30,000 different clients. You need more averaging of metrics in that sense. There's another interesting side effect of this technique, which gets into optimizing SLO fidelity. When you start working with building SLOs, and for the brief SLO, basically you want to measure your error budget, which means you need to know successful things versus not successful things. That works really well in head base sampling because you don't get a choice of how many successful and unsuccessful events there are and what the ratio of those are compared to tail base sampling, where you get to toy with that ratio and understand it a bit more.

We can't do that because we're doing head base sampling. Then we also have to be careful about how sensitive those SLOs become with things like statistically significant events and whether or not we should even put an SLO on the statistically significant events. For example, if we're doing head base deterministic sampling for the UI interactions, and it's 1 in 1000, and we have an SLO on that, that is going to be problematic because we're going to see one of those every couple of hours. If we get an error in that, it's going to burn the budget immediately because of how that SLO is built because we don't control those ratios like you would with tail base sampling.

Tying it All Together

Ultimately, to tie this all back together, we use all of these tricks with deterministic sampling, EMA, head base sampling, and targeted temporal sampling, so the Eye of Sauron that we turn on and off, depending on what we actually need to focus on, to get into the at least one of everything state that we'd like to be in. This has been mostly working very well. There's been some cases where we've run into, that'd be really cool with tail base sampling, but it's just not enough to get us to the point where we set up all that infrastructure to connect all those pieces, to solve all the challenges of performance, to really get the benefit of tail base sampling, at least from our return on investment if we're looking at those systems. That covers the suite of tips and tricks that we have for how we connect all those pieces together.

Questions and Answers

Participant 1: I worked on Sea of Thieves and we tried to set up something like this a while ago and we ran into the cost-benefit problem in a very real way, because the Unreal Engine is hot when it comes to logs. Have you found managing like what the cost-benefit at a higher level is for a system like this?

Nicholas Herring: At a higher level, it's one of these things where we start introducing Quasar and all the modern technologies. The biggest problem I saw immediately, so when we were originally designing that ecosystem, we were looking at the difference between a message bus ecosystem and a service mesh ecosystem. Ultimately, it was a judgment call on the maturity of our organization where we just weren't at a place where a service mesh would actually make sense for what we needed to learn, what we needed to do. Keep in mind, we adopted Kube pre-1.0. This was a while ago. The reason I'm talking about that is ultimately, we then realized that no matter what we picked, we needed something to be able to keep track of all these pieces and to ensure the success of this newer ecosystem to start shaving the yak, as it were, to move that legacy system over into migrating.

That became so real when we did an initial test where we set up 500 pirates in a solar system and put a ship right in the middle and we set off a smart bomb. Smart bombs blow up like this in a radial pattern and don't affect you. Then we watched what happens as it hit 500 other ships and we got a trace with 35,000 spans in it. Also, this is very much pre-AI. Then I looked at this, and go, we're learning stuff about our system that we simply just did not know. I was able to demonstrate this and show people, we have to have this. This is so important because it also helps us go in and look at the older systems and deconstruct and decompose those and understand how they're really working when we turn on those mechanisms there as well. Then it became a game of cat and mouse of figuring out, what are the important things and how do we really work with that?

Because we've also done fun things like where we've emitted spans per frame of the client, which is not a great thing to do when you have 30,000 GPUs screaming at you. We've definitely broken a few things here and there. It's more the mechanism of just observability in general of saying like, this is the cost that we have that this would need to get there, but this is what we get from it. Every engineer in our org at this point in time can't imagine a world before that. They're like, how did we get anything done? I think it's a big piece of championing the concept of it and illustrating what it provides. I think it was the first time in a long time that we could see into the past of this archaic system. The best way I can explain it is there's this always a sci-fi trope of, this old technology that no one understands. I was like, I hate this sci-fi trope so much. Then I started to work for CCP. I'm like, that makes so much more sense now.

Participant 2: How do you decide the granularity of your traces? Because when you were talking about the simulation and so on, it feels like one tick of your universe could cover a lot. Is there a consistent way of representing that? Because traces, when they get very big, can be very difficult to navigate. If they contain so much information and you're sampling them out, you lose a whole lot of information versus smaller traces. What are your thoughts on that?

Nicholas Herring: One of the things that we first learned, like when we were talking about rediscovering how this older system works, there's a bounty system in EVE Online, it's not quite what you think. Ultimately when you destroy an NPC ship, they have a bounty on them that gets paid out. That doesn't get paid out instantly. We have a coroutine somewhere going, every five minutes that I'll go pay out all of those. We discovered that thing quick when we turned on observability mechanisms. Because suddenly it was like, cool, the accountant for the universe showed up and started paying out everyone. We had to then target specific things. We're like, ok, we have this known pattern in our system where it's a worker mechanism. We short circuit those so the traces never start as a root there. We always then break those up so then we can start sampling them a different way.

There are also other tricks that we employed as well, because you just talking about that triggered some other things in my mind. When we're talking about sending notices to the client, we don't actually trace the delivery of those to the client because they are best effort. If they get there, they get there. If they don't, we'll send another one. It's fine. We don't actually trace those. That removed a lot of the quota from us. Another trick that we did was we integrated request-responses into a single span. We actually bisect the attribute namespace based on the request and response mechanisms. When we look at a request in our system in Honeycomb, one span represents the whole round trip. That cut our cost by half. It's great. There's a lot of little things that we do like that. Again, it's a judgment call on what's important for us. What do we actually care about here? Then tweaking how we instrument the system that way.

Liz Fong-Jones: How often do your developers go, I can't find that one specific trace that I'm looking for? Because that is the tradeoff of when you're sampling away data.

Nicholas Herring: I think the only time that happens is when they forget to turn on tracing. By default, in our ecosystem, when we have development, everyone has their own local solar system that they're working with, their local universe that they're working with. When they work with that, we'd by default turn those off because we have probably 50 people that would be running their own tenant of the universe. We don't want to trace all of those. We don't care about all of those. On some of our nameservers, our public test servers, we have that turned on. Some people will turn that on and they'll probably forget to actually enable the sampling because we turn that off by default for that ecosystem. That's usually when people are like, where is this? I don't see it. Then we say, turn this on. Then they usually get to see everything that happens. The key space in EMA usually guarantees that we at least get one of those somewhere. The frequency on local development versions of that are so sparse that it usually catches everything that they need.

Participant 3: I'm personally a little bit more familiar with a little game called Star Citizen. They make a very big deal about their persistence and data storage. I'm really curious what the sense of scale is of like an EVE Online or Star Citizen compared to a more conventional enterprise in sense of like data storage and messages, just amounts.

Nicholas Herring: For our ecosystem, I would estimate that we probably do roughly 60,000 messages a second, like real time. That's what we're doing right now as I speak. For storage, it really depends. The history of EVE is very old. Ultimately, I think our production database got up to, I'm going to say around 17-18 terabytes. Not that big. Also, it's very optimized for what's happening in a gaming universe. Now we have more and more happening where we're tracking even more things. For example, I brought up the system where people could create missions for other players. That has a running ledger of who's contributed to those missions and how much, because some of those are group projects. Then we have to pay out rewards for those types of things. The amount of data that we're actually persisting is growing quite a lot. A lot of what we're doing with the designs of the newer system, is to put in constraints around the sizing and the TTLs of that data so that we don't create infinite tables and then I have a dynamo bill show up, and that's the whole thing.

More importantly, a lot of what we do with the systems of the ingress point, so I mentioned like the 35 instances that we have, we call those our service gateways. They also fork every fact of the universe into an Azure Event Hub. Then, whether or not our data engineering team wants it or not, I'm sending it. They have a data lake that they're just filling up with all the details of that. Then as they build up the hub, they then process select information. For example, when we actually deployed the ship skin ecosystem, I immediately got a data engineer at my desk going, what are you sending us? Because in order to make that work, we had to make Quasar aware of where every ship in the universe was, at every tick of the simulation frame, which is an event in the ecosystem. Then he showed up going, I can't process this.

I'm like, you don't have to process it. I just need you to put it somewhere. I think it's a modest size of data and what we're doing. It's more about the interaction and the scale of the interaction of how many people can interact with the same part of space at the same time. I think that's what we're really focusing on for EVE.

See more presentations with transcripts

You Can't Trust Your AI Bill. And Neither Can I

Erik Peterson — Sun, 11 Oct 2026 16:00:00 GMT

Summary

Erik Peterson discusses why AI spend is eclipsing cloud bills, triggering a major software engineering cost crisis. He explains tokenmaxxing, unit economics, forced developer promotions, and why fully autonomous AI will write, deploy, and govern code by 2028.

Bio

Erik Peterson, Founder & CTO of CloudZero, is a pioneer in engineering-led cloud cost optimization. With 20+ years in software startups, he is passionate about cost-efficient engineering and strong margins. An AWS Ambassador and serverless advocate, he is also a recovering application security survivor who actively contributes to the FinOps, DevOps, and cloud economics communities.

About the conference

Transcript

Erik Peterson: I'm going to talk a little bit about the bill. There was a lot of conversation about all the great things that we're building. I think we really have gotten to a place where we're building an enormous amount of great things, but the bill comes due at some point. I wanted to share some of the things that I've been seeing, my company's been seeing, and make some wild-ass guesses about where I think it's all going. You all can hold me accountable to that. I can tell you, quite frankly, that it's a mess. Who am I? I'm Erik Peterson. I'm the CTO and founder of the company CloudZero. I started the company here in Boston in 2016. Coming up in August, almost on 10 years. It's been a wild and crazy ride over the last 10 years. Nobody could have predicted what's going on right now, or maybe somebody did. I'm sure somebody did. CloudZero is the financial control plane for cloud and AI. I'll share some of the data. Six months ago, we noticed something interesting happening in the bill.

Rise of AI

We are here. Really, we are here. This is what is happening. It is an amazing time that we can live in that if I want a money-flaming meteor and pounding it into the earth, I can just have whatever I want to create in front of me. How much did that cost to generate? I'm going to say this a few times today. I used to say every engineering decision is a buying decision. We are now in a world where every decision is a buying decision. I need to do some research, I'm going to go to Perplexity. I'm going to write some code, I'm going to go to Claude. I'm going to build some images, maybe I'll go to ChatGPT. Before I send that email, before I finish that thought, I am putting that information probably through some LLM somewhere. That costs money. Somehow, we've gotten ourselves in a situation where every thought we have is now monetized.

It sounds a little dystopian. Maybe it's the evolution of the human race. I certainly hope so. We are here. It costs money. The bill comes due. The bill is a little crazy. We have amazing statements like this being made. How many people saw this? Think about that. Take whatever your salary is, and if you don't spend half of that salary by the end of the year, Jensen doesn't think you're a great engineer. He's going to lose his mind. I think this may have kicked off, or maybe it was right in the heart of the tokenmaxxing universe. Tokenmaxxing, what is that rot upon the world? This is single-handedly probably the most expensive vanity metric in the history of IT. It's really hard to estimate exactly what that is. It's somewhere on the order of $5 billion to $15 billion. How many people had a boss or a manager or a CEO who was looking at the AI leaderboard — they're calling it a leaderboard — to figure out who in their organization was doing what they hoped they were doing with AI.

Did they know? No. They knew for a fact that if you weren't spending money, something must have been wrong. This is the craziest thing ever. Here we are. This term, I think, was invented back in March. It was declared over in May. I think it's still going. I think we're still living this. What has happened over the past couple months is pretty indicative of what's to come.

I think AI is coming for your jobs, too. If that meteor wasn't scary enough. Last year, I would say this definitively is not true. I was wrong. Last year, I would have said, you're not going to lose your job to AI. You're going to lose your job to somebody using AI. I actually think what's happening is we are getting a promotion. It's a forced promotion. We're going to have to accept it and change. Nothing drives this point home for me more than actually a story from my past. One of my first jobs in IT where I was working at a computer support job and had a couple of nice ladies walk into my office. They said, we used to work in this office and then you people with your computers showed up. We were six of us on typewriters. Typewriters had a real feel to them, a computer, soulless.

You guys took our jobs. Typewriter has shown up in the AI era. We've got to go figure out what promotion we're going to get. I think next year we're going to be really wondering, what is our job now? It's definitely not going to be about doing the work. It's going to be probably about managing the work. Our taste might be the only thing that matters.

The Promise versus Reality

This is going to color the whole conversation we're going to have today. This is the promise versus reality. Don't worry, AI is not coming for your job. I think it's coming for your job. Don't worry, tokens are going to get cheaper. Actually, tokens have gotten cheaper but we're spending a little bit more. In fact, they've gotten a lot cheaper. AI makes us all more productive. How many people here feel more productive? I feel pretty productive. If I look at the number of lines of code, the number of PRs, the number of commits, it all is up and to the right. Some days I wonder, am I churning? We're still bad at measuring developer productivity. We haven't really quite figured this out. I think it's going to only get worse. Don't worry, the cost curve only goes down. Actually, our spend is pretty much tripled and only going up.

Storyline

Here's the storyline for today: confession, the eclipse, the inversion, the aftermath, and then hopefully some things that we can take with us after this that will help us.

Act 1: The Confession - You Can't Trust Your AI Bill, Neither Can I

I got a confession to make. I built a company that built the category, and I can't actually fully trust our bill either. I spend all day long looking at the data that comes from the cloud providers, the AI providers, random people on the street sending cost data into our system. It is a mess. Anybody who tells you that this stuff is clean and easy and it's just data is forgetting something. We all know that just data is there be dragons there. When we look deeper into the AI bill, actually, like what's behind that? Is it just inference? Is it just tokens in and tokens out? There's a lot going on inside of there. A lot of it is buried and hidden and not even ever revealed to us. It's just charged to us. At least we have the benefit when we look at the Amazon or the Google bill to see that I spent this much on EC2 and this much on S3 and this much on who knows what.

At least I have an itemized list. We have somehow reverted to the world where right now I get a bill that is the equivalent of what I've been getting from my power company for the last 100 years. You spent $500 on 10,000 kilowatt hours. Doing what? Getting what? Providing what? Did I leave the lights on? Was the oven on? AC? I don't know. Somehow, we're back to where we were 100 years ago that hasn't changed, 10,000 kilowatts, a million tokens. There's a lot to try to unpack behind the invisible, so much so, actually, that folks at the University of Maryland had to go come up with an entirely new approach to get at it. They call it CoIn, the cost of the invisible. They found that there's just a ridiculous amount of invisible spend happening behind the scenes. What you get shown on the screen isn't necessarily what you're paying for. Sometimes it's built around changes in the model, changes in the weather, changes in things that aren't revealed to us. It's 10,000 kilowatt hours. Is it because a tree branch hit the power line and suddenly the power got a little weirder? We don't know. It's actually gotten a lot worse.

Nobody agrees on the unit. If we try to look at the data that's actually coming in, nobody can agree on exactly what it is or what it means. We've got different formulas for code, for images, for audio. Some of this stuff charges us for silence. Some of this stuff charges us differently. It's all a mess. They use different terms, different formats, different schemas, different everything. It causes huge problems when you try to bring it all together. They're all random vendor implementation details. We've rolled back 100 years. The prices themselves also don't make any sense. I really want to know why o1 still exists. It's 7.5 times more expensive. There's zero reason for you to use it. There's vastly superior models for vastly less cost but it's still there. Somebody, some enterprise somewhere has an agent running in a basement that is consuming tokens that's probably keeping OpenAI afloat.

They thank you for your service. Other providers, I think Google turns off models after a year. Maybe they're doing everyone a favor. OpenAI is leaving stuff around for a really long time. Or maybe that was Microsoft. Pick a region. You'd think, I'll just keep it in the U.S. That'll be cheaper. Instead of multi-region. Multi-region sounds expensive. No, pick a region that's more expensive. Don't pick a region, you're going to spend more. That's a feature. They change how they tokenize things. Token costs go up 35%. All this stuff changes and none of it is really under your control, except maybe the model you pick. The APIs themselves, the ones where we get this data, they're broken. We don't need to tell a story about uptime and availability these days, which is fascinating how unimportant that number seems to have become if you've been a GitHub customer for a while.

It's happening everywhere. It's like the old days, "AWS is down. Let's all go out for a bite to eat." "Claude's down today. I'm not going to get any work done." When we're trying to get the cost data, daily 500s. When we're trying to get information from OpenAI, daily 500s. Cursor, one bad record in there, the whole thing just blows up. It is a mess. This is just what happened in the last 90 days. If you think that this is happening over some long period of evolution, no, it's not. It's happening in real time. This presentation actually was the hardest presentation I think I've ever had to create, because I'd write something down. Then I'd look at the information the next day, and I realized, new stuff out. It's wrong already. We didn't even have a proper API to pull Anthropic Enterprise billing data down, until March. If you're on claude.com, you could get your billing data. If you're on claude.ai, two vastly different platforms, you could not. Except screenshot it, and download a spreadsheet from their website. All of this churn, just in the last 90 days.

Act 2: The Eclipse - All Cost is Eclipsing the Cloud Bill

What is happening? It's pretty simple. AI is going to eclipse the cloud bill. Cloud and all its growth, all its craziness, all the billions trending towards trillions, if we look at just what's happened in the last couple months, that was the warm-up match to what's coming. Anthropic's like up and to the right is truly insane. Nothing like this has ever occurred before. This is the warm-up match. I believe all cloud spend is going to be absorbed by AI. There's only going to be one thing, it's going to be AI. Just days ago, all these crazy things happened. We're talking billions if not trillions of dollars here exchanging hands. Flowing, who knows where it's going? People making huge claims about how compute is going to be spent. It is an entirely different ball game. We thought cloud was expensive. No. That is a rounding error to what is getting spent right now.

I think that is going to lead to a disruption timeline that is moving faster than any of us here are prepared for. I thought I was crazy. Actually, a lot of people thought I was crazy. I started committing myself to only using AI to create code in January of 2025. By the middle of the year, I was convinced that all code is going to be written by AI. If you talk to anybody in California, they were convinced too. Everybody outside was like, I don't know. How many people think that all code in the future is not going to be written by AI? We'll come talk about that in a year. It's all going to get written by AI. I think it's going to happen faster than anyone thinks. I think that what we'll end up with is something I call the fully autonomous cloud, which is AI writing the code, AI deploying the code, AI managing the code.

It all is running in an autonomous fashion. The cloud as we know it is going away. It is going to disappear. My bet is that it's all done and figured out by 2028. We'll see. You can take the safe bet, but I don't think I've been aggressive enough. It won't be the financing mechanism. Interesting point of fact is that every AI startup combined is still eight times short of the total AI CapEx spend of all the cloud providers. There's still this huge gap. Where's that gap getting made up? I mean $80 billion here. It just seems like $80 billion is such a small number. What is going on? They're both burning cash if we look at both of the giants out there. They think about their numbers drastically different. Anthropic says that they're profitable 5% return based on how they built their numbers and built their books.

OpenAI has got a negative 122%. Crazy. They're both looking at about 80% margins on their API calls. They are making money. This AI cost crisis feeds the narrative as well. This is probably the most profitable supply chain in tech ever. These are real numbers. What you get charged and what we're paying. We're so driven to grab this right now that we stop paying attention at some point. We've seen the circle chart a few times. The circle is real. Every time I turn around there's a new box to add to it that's feeding the circle. I think as long as it just keeps flowing and spinning, we're going to be ok. Just keep spinning. Just keep spinning, we'll be ok. I hope. Otherwise, it's sticks and rocks and stones.

Act 3: The Inversion - The Cost to BUILD is Eclipsing the Cost to SERVE

The inversion is this magical thing that some CFOs have started to notice which is the cost to build is now the most expensive thing, not the cost to serve. Cost to serve was what I was spending to deliver cloud computing and deliver services to my customers. The cost to build is what we're spending to build this thing, to maintain this thing, to deploy this thing. Somehow the cost to build became more expensive. I know a lot of you have probably heard of Jevons Paradox. I don't think a lot of people actually stop and talk about just how freaking old this thing is, 1865. I love it. We're talking about a power bill that looks like what Westinghouse would have given me 200 years ago. We're talking about Stanley Jevons worrying about coal. Just really internalize this, because Jevons Paradox is at the crux of it all.

The more efficient we make a thing the more we consume of that thing. We're making it very efficient, or at least attempting to. This is what it looks like in reality. Cost per tokens are down by 1000% but our enterprise spend is up by 222%. That's the inversion. This is the actual numbers that we're seeing. This is the impact, the result of this. It's happening. It's happening right now. What CloudZero is seeing is pure evidence that it is very real. Fourteen times growth in GenAI spend in the last five months. Eighty percent of that seems to be overhead based on all of the crazy requests, systems, R&D, humans, Claude Code spend. You mix it all in there. A new model comes out, we see, customers go bananas. One customer, 0 to 2 million in a month, just a new model appears. Nothing else really changed, just that.

Then, who's using the cheaper models? Less than 2% of you. We all rush out to grab the new thing. Haiku still does some good stuff. Actually, Haiku is the only one that can answer the question, how many days in the week have the letter D in them? Go ask Opus 4.8 that question. Yesterday, it did not give a good answer. I preface it with yesterday because somebody is probably hard-coding it as we speak. The CFOs are doing the math. This is what we have to think about, tokens or humans. I have caught a couple of CFOs with spreadsheets that literally compare your paycheck to the cost of tokens. Don't believe me? This is why we have to think differently about our jobs. NVIDIA is talking about it. CFOs are talking about it. We're seeing it happen. I want us all to enjoy this future. It should be a very happy future where we all get promoted. We got to start thinking differently now.

Act 4: The Aftermath - Alerts Can't Help You if the Money is Already Spent

What's the aftermath of all this? We've heard about Uber. Everybody's talking about Uber, spent so much money. They're not alone. JPMorgan did it too. Microsoft just literally days ago said, stop using Claude. Of course, they got plenty of reasons why they want people to stop using Claude. It's real. People are starting to freak out. Although interesting enough Uber is still spending because maybe they're getting stuff done. I had one story that came to me. Somebody said, we're going to spend about $4 million this year. We just in this quarter spent about 40. Who does that and survives? My reaction was, who got fired? Said, no, I think some people are getting promoted. What is going on? Uber's CEO though stepped in and said, wait a second, what's the connection between all this money we're spending and the outcome? This is the important question to be asked.

This is the question I want us all to think about. How do we connect the money we're spending with the business outcome? Love vibe coding. I was out at SaaStr, I heard Jason Lemkin actually say these words up on stage. He was talking about how he had rebuilt his whole organization. He had a staff of like 20 or 30 people, and now it was just two people and a dog, and they used Replit for everything. He was like, a dollar a minute. Then people got up to ask questions and said, this vibe coding slop is just driving me absolutely insane. He said, yes, but we're all here, aren't we? In fact, this is the most successful conference I've ever run, two people. It's scary. We also are packing more and more, at least trying to, into the context window. I have a million context window in Claude now, that's fantastic. Let me pack more into there. Let me link up 30 different MCP servers into that system. Let me link everything and let me just dump it in there. Wait, why am I spending all day long fixing the bugs that I just created? MCP tax is real.

There was just days ago, again, another thing, on what Netflix was working on trying to compress the token consumption on the input, try to break this down. If you're not spending time in front of whatever system you're using and regularly sharpening the saw, you are absolutely burning through tokens. Not everyone's token spend is equal. It's fascinating. I've been watching every single one of our engineers use these tools and looking at how much they spend and then looking at what they produce. Wildly different spend, similar outcomes. Some people are doing things differently than others. A lot of it is in how you pack that context window and how you think about planning, starting tasks, or just vibe code the one-shot wonder wagon to value train, whatever you want to call it. It's not just the coding that we're doing, it's also the systems that we're building.

I've seen so many loops, things that break, that just keep going for days and days and days spending money. This is an example taken out of one customer where I saw this. I gave a talk at QCon last year about million-dollar lines of code. I think I'm getting closer to billion-dollar lines of code in some of these things. Security is not to be forgotten either. The first QCon talk I ever gave I think in 2018, I coined this term called Denial-of-Wallet. It is alive and well. I think the next attacker is going to come to us through our own systems, get access to our AWS environment, go find Bedrock, and then use our AI tokens to then absolutely own us from the inside. When we walk away, we'll have no data and even less money after the fact. That attack is coming. The attacker won't even have to pay for it, they'll bill us.

Every company that we've looked at, they had plenty of dashboards. Check out this amazing dashboard I built in Grafana, or Datadog, or wherever, or even CloudZero. They had budget alerts, but nobody was paying attention to them. In fact, a lot of people when they get a budget alert go, I don't know, can I turn this thing off? Will I get in trouble? The CEO said tokenmaxxing is a thing. I've talked to friends who have literally watched hundreds of thousands of dollars go out the door over a weekend, and they felt like if they said anything they were going to get fired. That's terrifying. We've got to have the conversation about what is too much. There's no per call, per user, per anything limit out there. We used to be afraid of this in the cloud, but how much damage can be done? Multiply the ability for every engineer in your organization to spend $5,000 in a 12-hour period and then start to think about how important a limit actually is.

It's very easy for any company of any size to spend the total of its capital in a couple of weeks if they let this thing go. There are no limits. Everyone here is absolutely convinced they're secure as well? Denial-of-Wallet is coming. There's going to be some companies that are going to be put out of business by that.

Act 5: Think Different - Every Decision is Now a Buying Decision

We have to think differently about this. Every decision is now a buying decision. Everything. When was the last time anybody Googled something? No disrespect to Google, but seriously. Googling was free. When I planned my date night on the weekend, I went to Perplexity. I went to Claude. I didn't go to Google. When I needed to figure out the directions, I didn't go pop it into a search engine. I asked an LLM. I paid money for that. We're all paying money for that. Stop for a moment everyone here and think about how much time, questions, things you've done that you would have done for free that nobody would have batted an eye before, searching the web, but you've popped into some Claude, or ChatGPT, or Perplexity window. Then ask yourself, was that a company thing that I did or was that a personal thing that I did?

There's this report that comes out every year that talks about cloud waste. It says cloud waste, and it's been stuck there for the last decade and a half. It says about 30% of all cloud is waste. If we just all stopped collectively doing free Google Searches and just started pumping our every question that we have into the AI thing that we have now at work, what does that waste number look like? Is it waste? When the companies figure out how much money is being spent on that, you want to really wonder what Big Brother looks like in a corporate environment. How many people are ready to have their chat history shared with the world? Even if you try to use anonymous, maybe you're using a proxy. Every decision is now a buying decision. We have to think differently about this.

Let me walk you through a couple of things. We have to stop asking what we spent. I actually think this obsession with cost is costing us. When we think about just how much we spent we forget about the thing that was truly most important, which is, was it worth it? What am I trying to do? What am I trying to build? Why am I trying to build it? Who am I trying to serve? What is the purpose of this? I think that's also rooted in when we think about what our jobs will evolve to be. We have an opportunity to get way more focused on the outcome, way more obsessed with the outcome. If you are obsessed being a craftsperson, building some of the finest code that has ever been written, I have nothing but sympathy for you. Honestly, I do. I used to dream of writing some of the finest code ever written.

I have given up on that dream. It's not much different if we think about it, though, in terms of like, I'm not really after the compiler is done, going and looking at the machine code to make sure that it interpreted it properly. No, we're just going to speak whatever language we speak and we're going to let it get translated into code and we're going to stop looking at the code. I got a confession to make, I don't look at the code anymore. That's the easiest way to deal with it. I care about the outcome. I care about the tests and the validation and the proof that the stuff that came out of it met the demand or the use, but I don't care so much about the code anymore. All code that is produced will eventually be whatever the LLMs can produce in the most efficient manner possible.

That may or may not be assembly. Not a lot of LLMs really are good at assembly. They're good at JavaScript and Python right now, apparently, today. I think over time, the code produced will be some language that is super-efficient for the LLMs to write. That is inevitable, I think. We have to stop thinking about what we spent, just like we had to stop thinking about what was created because we need to think about the outcome. What did we spend? No. Was it worth it? Then us as builders, creators, we have to think, is the reasoning, the model, the stuff that I'm building, is it actually necessary? Am I getting to the right outcome? Did I really need to rush out and switch everything over to Opus 3.8? Can I mix some of these things together? What are these models individually actually really good for? It's not like a 286 and now I'm upgraded to a Pentium.

No. They are different beasts for different tasks. We're thinking about it wrong. What's our rollback plan if we just rushed out and actually burst all these things up? Did we go back and just look at the outcomes? I didn't talk enough about it, but I probably could have given a whole talk on just evals. People have been spending more money on evals than sometimes their production systems, which is a funny observation, particularly for people using LLM-as-a-judge stuff. How much money are we spending on that? There are different ways to achieve quality. What happens when these things loop and run off and explore the universe without us? Are we aware of that? Do we get an alert? What business outcome is this delivering? We should rejoice in that. It is a chance for us to get closer to the customer, closer to the value that we're here to produce. Closer to the businesses that we hope to see successful, hopefully. Or maybe even better, individually we can all start businesses that are much closer to our customers and be much more in charge of that outcome than we've ever seen possible. I'd like to see more companies started.

When we think about how we are going to change our jobs, I want us all to think instead of dealing with being the perfect craftsperson, to think about how we are going to drive that herd of agents, drive the AI employees that are going to be working for you or I. There's a funny joke about this, that's where this came from. There's a farmer explaining to a horse, "Don't worry, you're not going to be replaced by the tractor but you might be replaced by a horse driving a tractor. You better go learn how to drive tractors." We all need to learn how to drive the tractor. We got to also measure the right things. If we measure cost per token, your team is going to optimize for cost per token. This is Goodhart's Law in full force on display. Amazon had a leaderboard, they took that down recently.

Good for them. I don't know how we explain this to our bosses that that AI leaderboard is a bad thing. I definitely still hear the questions. How much are people spending? Why is this person only spending $2? What are they doing? Do they even work here anymore? They only spent $2. That is not healthy. Hopefully we get better at thinking about the outcomes through all of this. A good friend of mine, a very senior engineer, worked with him for years dealing with this same thing. I asked him, what's the problem? He's like, "I got into this business because I love writing code. I don't want AI to write the code. I want to write the code. It's fun. I enjoy it." I got nothing but sympathy for you. I don't know what to say. You're going to be just like those friendly ladies who came into my office 20 years ago, 30 years ago, and said, there used to be six of us in here typing on typewriters, then you computer people came along. Change is inevitable.

The outcome driven metrics are better. They drive better outcomes. These are four that I think you should think about that you should internalize. I am a true believer in unit economics. What does it cost to do the thing that it is that you do? If you don't know this, you should know this. It's probably one of the most important performance metrics for any on-demand cloud-built system in the universe. It is the secret to understanding the 1 million token bill that you got that doesn't tell you anything about what you actually did. It is the most obnoxious, anonymized source of information that tells you nothing about what you achieved. You need to figure out how to break that apart. What feature does this drive? What functionality does this drive? What outcome does this drive? What's the cost for the autonomy in our environment, engineer plus Claude Code?

Are they producing 10x for 5x less? Great. You got to think about the economics. Your CFO, I guarantee you, is. I've seen a lot of the conversations lately move out of the engineering org. You've probably heard of the term FinOps. FinOps is a lot of times inside the engineering org. They care a lot about the cloud bill. I care a lot about the cloud bill still. It's big. The AI conversation has shifted to the CFO's office because it's so big. It's not a budging issue. It's a P&L issue. What that means is the person who also has free access to everybody's salary is also looking at all the other things side by side. Those spreadsheets exist. Get ahead of it. What's the efficiency for the model that we're using to achieve the outcome? That way maybe we can actually understand when the next model comes out if it's even worth upgrading in the first place.

What's going to be the margin yield or margin improvement for all this autonomy? Don't think about it personally. It's not personal. It's just business. I think we actually need to start thinking about that unit cost or that margin data flowing into the systems themselves as a signal for how the systems operate. Is this a top-tier customer? Maybe they shouldn't be using the most expensive models. Maybe they shouldn't be using exactly the same system. What's the lifetime value of this customer? Is this a free tier? Do we have the highest margin tier on the same model as our lowest tier? Route according to margin or your profit objective. It's just unit economics. It's not optimization really. This is just business. It's good business.

Stop Doing This on Monday - Five Defaults

Here's what I want you to stop doing on Monday if you take all this together. Stop thinking a dashboard is going to save you. In fact, in the future, nobody's going to be looking at dashboards. If you're still looking at dashboards, I don't even know why you're looking at dashboards. I don't look at dashboards. The number one user of all these systems is going to be AI anyways, and they'd rather look at JSON. They don't want to look at the dashboard. Yes, you can paste a screenshot into them, but it's way more efficient if you do it a different way. Dashboards are just post-mortems of a horrible event that occurred, because this stuff is moving way too fast. A lot of the CFOs are freaking out right now and actually starting to talk about something they never used to talk about, which is real-time data, which is crazy, because they think in terms of months and quarters.

Get out of this cost per token as your primary KPI. Don't waste time. Don't forget to think about the unit economics. What is the cost of the thing that it is that you do? All these things right now are just decisions that have been made for you. Nobody's thinking consciously about these things. You need to think consciously about these things or they will cost you, or they actually are, in fact, costing you right now.

Key Takeaway

I'll leave you with this. By 2027, I'm convinced that this is going to be true. Probably going to happen faster. The primary buyer of things, technology, cloud resources, stuff in your environment is not going to be humans, it's going to be an autonomous AI system. You remember that Jevons chart, cost of tokens going down, total spend going up? Right now, we're seeing that spend driven by the cost to build more so than the cost to serve. At that point, when this happens, the cost to build and the cost to serve will start tracking together again. We'll be looking at an even steeper curve, because we're spending tokens to write the code. We're spending tokens to deploy the code. The code itself will be picking and choosing what it runs on and how it runs and when it runs. We need to be prepared for a world where we can govern this reality.

Think about that. Those are the systems or thoughts or culture or ideas that we have to embed into our engineering practices right now, into the software development life cycle right now. The thoughts that will dictate whether or not I believe your company is profitable and survives or doesn't. That's what's at stake. The folks who master this, can ride herd over this, are going to be the ones getting a promotion. I want that for all of you.

Questions and Answers

Participant 1: I was checking if you have an MBA because a lot of this is economics. To what extent do you think the workforce needs to be educated in these basic concepts of unit cost?

Erik Peterson: Education, that's a whole other conversation. The cost of knowledge is rapidly approaching zero, but the cost of wisdom is going to go near infinite. My daughter is going into college in the fall. I don't know what that means for her. I don't know what that means for any of us. It's an odd place. If I could go back in time, I would have studied economics and not computer science. That's for sure. I think economics is at the center of everything. I think the path to applying our very technical mind to this problem is by understanding the cost to do the thing that we do, being focused on the outcomes. That path will set you on the right path ultimately. You should think about those things. If you're on that path, you will get better and get that promotion. Here's a funny thing. It's an odd observation, but it's true.

Finance people and technical people often seem like oil and water. They're actually just two different alien species that speak a completely different language. They're both actually highly technical, highly disciplined, highly geeky, in the weeds, detailed cultures. They have way more in common than you would think. Just ask a CFO to open up Excel and show them the cool thing that they've done. You'll be like, this person knows their stuff in front of a system. It's the same way you would feel if somebody said, open up your IDE and bang out a few lines of code. You have a lot more in common than you know. If you find yourself having that conversation, there's a bridge that you can establish there. Probably more so for the engineering side of the house with finance than any other part of the business.

Participant 2: AI seems to be one of the first technologies we see in tech that adoption is being driven partially top-down in a lot of ways. It's not engineering saying, there is this cool technology, let us all move to that. It's various types of C-level people saying. Right now, most of the companies are at the stage of, everyone, engineering, you must become AI native. The second step of how much this costs and why and prove that it needs to be coming, that's probably coming for most of them, but it hasn't come. My question is, from your perspective, what is it you are seeing? What are the trends you are seeing in C-level decisions and how people look at that?

Erik Peterson: AI is definitely felt very tops down. It's also felt grassroots at the same time. What I'll say is, first we did this to ourselves. A lot of technologists, myself included, have had that moment where we walk into the room with the business conversation going on and we say, Kubernetes will solve our problem. "Nobody knows what Kubernetes is in this room." They go, just keep running it on that thing that's in the closet or whatever. Why are you talking about this? I've worked with people where I literally took the whole proposal that they've just had on the table. Why can't we save some money? Why can't we do this the right way? I go into the room, I say, "If we build this Kubernetes pipeline data thing super containerized, yada, yada, yada, it's going to do wonders for our efficiency." I'm like, ok, let's dig into that wonders for our efficiency part.

What does that mean? I don't actually know how to articulate. Let's go figure that out. What is it going to do? It costs us 25 cents per transaction. It's going to go to 10 cents per transaction. Just walk in and say that. There is not a single finance business person in the world if you go and say, the thing that we do costs 25 cents per transaction. If we make this change that I'm not going to explain to you because you wouldn't get it anyways, it'll go to 10 cents per transaction. They go, why are you still standing in front of me? Get out and build it right now. That's the opportunity, get out and build it right now if you can talk in terms that the business understands. They really just don't care about Kubernetes. From a trend's perspective, the CEOs are still out there saying things that are wildly disconnected from the other parts.

The CIO and the CFO seem to be having a moment. CTOs are like, am I going to get fired? Did I do the right thing? The CIOs who are like, wait a second, everybody's building stuff internally again. Just because there's a small company, we're about 180 people, but we rolled out Claude Code and co-work to every single person in the business. A couple weeks later, my head of HR asked me for a GitHub account. What? I don't know. Claude told me I needed a GitHub account in order to do this thing. I was like, do you know what GitHub is? She's like, I don't know what GitHub is, but it said that I needed it and it said that I should ask my IT person, and you're the technology. My observation was everybody in the company became a developer, a very junior developer, in that moment. The CEO was like, this is awesome, everybody is doing stuff. The CTO brain was like, ok, I think we're getting better. Everything blew up.

See more presentations with transcripts

Ontology‐Driven Observability: Building the E2E Knowledge Graph at Netflix Scale

Prasanna Vijayanathan/Renzo Sanchez-Silva — Thu, 08 Oct 2026 16:00:00 GMT

Summary

Prasanna Vijayanathan and Renzo Sanchez-Silva share how Netflix tackles observability across 38M events/sec. They discuss replacing reactive monitoring with an AI-driven operational ontology and agentic workflows using Claude and graph databases. They explain how unifying MELT telemetry into queryable knowledge graphs enables automated triaging, root-cause analysis, and self-healing systems.

Bio

Prasanna Vijayanathan is an engineer at Netflix’s Consumer Engineering org, where he leads performance, observability and QoE initiatives using AI/ML innovations across client platforms that improve reliability and user experience at global scale. Renzo Sanchez-Silva is an engineer on the Observability Team at Netflix, where he has spent the past decade developing alerting and monitoring systems.

About the conference

Transcript

Prasanna Vijayanathan: My name is Prasanna Vijayanathan. I'm part of the observability team at Netflix. I work on observability, performance, and quality of experience across all our member applications. What that means is from the point you click on the Netflix logo until you start watching something, how does your experience feel on Netflix is what I look into. What metrics should we look at? What tools should we have? What insights should we provide to our users to make sure that we understand what you all like? If you do like watching Netflix, you're welcome. If you don't, you can blame it on my colleague, Renzo.

Renzo Sanchez-Silva: I'm Renzo Sanchez-Silva. I've been at Netflix for almost 10 years now with the monitoring and alerting team within observability, building several alerting infrastructure, and now getting into AIOps space. I've done also some research back in the day in ontology. With AI, everything is converging.

Observability at Scale - Why is it Hard?

Prasanna Vijayanathan: Today we're going to talk to you about a problem that we've been tackling, which at the surface feels like an observability challenge, but it's really a hard data engineering problem. Before I get into that, let me set the stage up. Why is observability hard at scale? What scale are we even talking about? Let's do a pop-up quiz. Netflix launched live events. End of 2024 was one of our first major big live events, a boxing match between Jake Paul and Mike Tyson. A few of you might have watched it. What do you think was our peak traffic during that event? It's definitely in the millions. It's 65 million concurrent streams. What that means is there are 65 million users clicking watch and watching the stream at the same time. That was our peak traffic. One more. When you log into Netflix, the apps are sending requests into our infrastructure for our backends to respond and stuff.

How many requests do you think we handle per day coming from our applications to our servers? We handle more than 2 billion requests a day. Last one. With all those requests happening, we obviously have our own observability systems where we log metrics, a lot of different things. How many real-time logging events do you think we have to handle per second? It's upward of 38 million events per second. That is the scale we're talking about. Numerous client platforms and devices across all the TVs all over the world, mobile phones, laptops, you name it. Thousands of microservices emitting all these billions of telemetry events. Every single member touches so many different parts of our infrastructure, our CDNs, nodes, and so on. On top of all that, we have thousands of experiments running through A/B tests throughout our ecosystem. Managing all of that at scale. It's everything, everywhere, all at once.

End-to-End Observability - What is It?

Even at that scale, our goal at Netflix is to deliver the best possible experience to every user every single time. Our strategy towards observability is simple. Do we have the right tools, metrics, and insights to enable us to do that? The strategy is simple. It is not easy. At this scale, correlating user experience with system performance is a fundamentally different challenge. It is not a tooling challenge anymore. It is a data engineering challenge. Traditionally, observability systems have been monitoring and alerting tools where you observe your apps and services. Something goes wrong, you get flagged. Then you look at what's going on, triage it to the right teams, look for the right developers to fix it if needed, roll out a fix, and then see if the data comes back to where it is expected to be. That's how traditionally observability systems have worked. With end-to-end observability, we wanted to rethink how we do observability, and change it from a reactive monitoring experience to a proactive insights engine.

Before I show you how we do that, I want to talk about the vision that we created. Imagine a system that can detect issues automatically in a few minutes across the entire stack. It's not just one service or app, but across your stack. Wherever an issue happens, you know that this is what is happening. Then you go into automatically prioritizing those issues based on how it's impacting our users. You also can triage it to the right team automatically. You don't have to look at one app and one error and say, I think I'm going to follow this needle through this haystack and pick out different services and loop them in. Then, can we automatically root cause? Not just triage, but can we actually identify the root cause of the issue? Or, even better, can we predict issues even before they would actually start impacting our users and maybe even start suggesting or taking corrective actions?

Is this possible? We wanted to do this. Building the system would give us seamless visibility from the user's device through our networks into our gateway services into the deep trenches of our backend dependencies, services. All connected, all in real time. Users would have no disruptions to their experience on Netflix. Every time you watch Netflix, you will love it. Your experience is going to be awesome, every single time. This is our vision.

Sounds great. That's far from reality. It's very different today. I'll walk you through a timeline of a recent investigation, probably within the last year or so. There was an alert that got triggered on one of our client platforms, I'd say around 3:30. Typically, on-calls look at it and they're like, let me wait for a while. Let me see if this is going to sustain. Maybe something happens magically and it'll go away on its own. Sometimes it does. Sometimes it doesn't. A couple of hours later, they're actually looking at it, "I think it's a real issue. I need to go figure out, follow this needle and see if there's another service that's impacting." In the average case, usually it does. Then it's triaged to the right team. Then it turns out that team was already debugging this issue from like 1:30 that day. People are already on it.

Then, multiple teams are involved now. There's a lot more debugging happening between all these different teams. Then once the debugging starts, they root cause the actual issue, follow the needle down to the actual offending service. In this case, it turned out there was some service that thought they're not as important and cut down their capacities. That caused backward propagation errors. That's around like 6:30, three hours since the client application actually noticed real user impact. We rolled out a fix. It was resolved within the hour. This is an average case. It doesn't sound too bad. It's like four hours. Not too bad. Then when you consider what it took, during this time, there are nine teams that were paged. More than 30 engineers were involved in the entire debugging process in different capacities. Then three other related incidents that were also flagged. There's panic everywhere across teams. This is typically, and this is an average case. This is not even a bad example. Then when you put together the four hours and all the resources that get involved, it just feels wasteful. This is what you want to solve with end-to-end observability.

We looked at why. Why does this happen? We came up with four main themes that existed. One, you've got too many siloed data sources. We got roughly logs, metrics, events, and traces that are stored in different forms in different datastores. They are in silos. They don't talk to each other. They're not standardized. Structures are different, and so on. That leads to disconnected and non-contextual alerting, which means each team follows a data source that works well for them in their silo and then set up alerts. Then when alerts trigger, they are looking at the same problem in one way compared to another team that's looking at the same problem from a different perspective. They should be talking to each other, but that doesn't happen. What that leads to is complexity in how teams triage and actually troubleshoot the issue. Ultimately, what that means is we actually don't detect what is the impact for our users. If you're a downstream service, 5 hops, 10 hops away from the user, you almost never understand how your system and service is affecting the end user.

Connectedness - Bridging Gaps, Breaking Silos

Given the vision that we saw, that we have for end-to-end observability, how do we solve this and move from the state of reality to the vision? The one-word answer for that is connectedness. That's what we thought. The common theme is that. We want connected data, connected alerting, connected debugging systems, workflows, and root cause analysis. If we create that theme for end-to-end observability, we don't need 10 different data sources giving rise to 6 different tools. We want to create an integrated observability layer. One abstraction that can tell you everything about our system's observability. What that means, then, is our dashboards, alerts, debugging workflows, and incident response tools, they all speak the same language. Then duplicating efforts in debugging and triaging, the same issues, and the same symptoms across the system is reduced, which means you can follow a breadcrumb trail, and then get into the same path from the user's device to the actual root cause.

You can follow the breadcrumb from different trails, but get on the same main path of root causing the issue. That leads to reducing the time it takes to resolve issues and ultimately will improve the accuracy in diagnostics, and give our users a less disrupted and better experience. The core piece of all this starts with the data, connected data. We got our users, our devices and applications, and our services and infrastructure, all emitting these broadly four different categories of data: metrics, events, logs, and traces. We unify them and put them in one layer, the MELT layer. Once we unify all these sources, we solve the problem. It's important that even if all of this exists together, we need them to be accurate and actionable.

Let me walk you through another scenario. Let's say we have this layer already. Let's say there is a regression we see, LOLOMO TTR. LOLOMO is the metrics of the list-of-list-of-movies that you see on Netflix. TTR, some rendering time for that list-of-list-of-movies. You see a regression there. There's a slow leak, and there's a big regression. Developer notices this and they ask our unified data layer, what happened? Now all the observability data lives in one layer, so tell me what happened. Then the metrics system would tell you, yes, I see LOLOMO TTR regressing. I also see 12 other metrics across 5 different services correlating, happening at the same time. They are also regressing. We find some correlations. That's good. Then events could tell you there are 47 events across all these different apps and services that also happened right before this regression started. Yes, I can see it on a graph.

I can see all these events along with the metrics. Then logs tells me, I think there is a connection pool exhaustion error that spiked up at this time in one of those 7 services. Maybe that's what it is. There are also all these timeout errors in all these other services at the same time too. Then traces can tell you a different story. If I'm going to talk about the work we've done with traces, that will be a different session on its own. To give you an idea, a trace gives you a path for a particular request and response from the user to all the services that touched it. You can trace that path. The challenge with tracing is, unless you know exactly what trace you're looking for, it is super hard to use it to debug an issue accurately. It's not even a needle in a haystack. It's like looking for a leaf in a forest in the dark and you're blindfolded. It's hard.

Ontology - Relationships, Not Just Connections

What we are really missing is not just connectedness, but the relationships between all these different pieces of this puzzle. We need to know the connections from these components, yes, but we also want to know how they interact with each other and how they are related to each other. This we call the observability ontology. Renzo will walk you through how we build these relationships and how we're putting this to use every day.

Renzo Sanchez-Silva: What is an ontology? It's just a formal specification of types, properties, and relationships. It's how we encode meaning. The basic construct of it is what you call the triple. The triple is a subject, a predicate, and an object, one fact per triple. It can be, a person is an employee. A service is important, or tier-0, for example. It can be properties too. This is the way we encode knowledge. It's very basic, very primordial, actually. It's like saying Tarzan loves Jane. Mommy loves Daddy. Babies have mini ontologies in their head. We just start collecting all these symbolic triples in our heads, and we start making propositions and reasoning and so forth. Ontologies are a way to encode knowledge. We're using W3C standards, RDF, OWL, web ontology language, SPARQL, the query language, SHACL is shapes constraint language, where we can say a service can be terminated in AWS, but then another constraint is if the service has more or less than 10% of capacity, you cannot terminate.

You can encode all these things. You can describe your services, the whole infrastructure. For example, we have API gateway. It's of type application. It's owned by a team. It is deployed in a region, and so forth. We can go and describe every single piece of the infrastructure, and every piece there is namespaced as well. We started building this baby ontology by describing incidents. We have identified a few namespaces. We're building an operational ontology. The goal, again, going back to Prasanna's vision, is to start finding this end-to-end observability graph of all the things connected in our infra. Let's call that the operational ontology. For instance, we identify these namespaces, and they can be referenced formally in the triples.

The problem is how to bring order to this operational chaos, in incidents scattered across Slack, PagerDuty, the MELT layer, metrics, traces, logs, and events. We all gather in the channel together. There are hundreds of messages, many services, people interacting. It's crazy until we find root cause, and then we fix the issue many hours later. What we want to do is observe that, start encoding in real time. Capturing the structures and how we preserve this in machine-readable types. A message can be, obviously, in English. Then someone says, API gateway has latency spiked after this deployment and so forth, and then Jane is rolling back. Then you can describe that in triples. That's the idea of doing this. Doing this by hand is hard. In fact, back in the day when there was no AI, you needed to have an ontologist who was always annotating, this is that, this is a person, and this is a service, and so forth.

With AI, we can do this much faster. For this, we created what we call a harvest pipeline. The incidents occur in Slack. That's where the action happens. For each channel, we transform every message, and we send every message to a bunch of enrichers. We call observability APIs. We use LLMs on top to find some extra classification, we correlate that. We save that in our graph database, we call it QuipuDB. The enrichers in green are very deterministic. We want to identify — for example, this is on incident.io, we use them as our incident management software — it's of this ID. We have the PagerDuty alert that was fired, for example, we extract that. We have various very deterministic, small algorithms that classify things because we know URLs, for example, we extract IDs. We extract, for example, also the sentiment of what is happening in the thread or the message.

In blue, we have alerts and the timeline that is occurring. People are sharing links to our dashboard. We encode that. We know what that means. Someone uploads a snapshot or a screenshot, we save that. It's a screenshot related to this graph, this metric. Then we use also LLMs at the end of this process where we have identified all these basic patterns or entities, and start finding new relationships or concepts, or new definitions. The LLM, you can see great things like this has canary analysis, deploy failed, and things like that. It starts finding concepts from what has been currently harvested. If it's new, we'll try to propose a new definition. This is what it's called in ontology, the open world assumption, where you are somewhat free to add relationships and we have ways to control that. It's a very controlled process. This is sequential. Sometimes if something fails, we just leave it. For example, we cannot grab some data, for example, you would leave it there. Then we check against our main metadata sources. For example, the software catalog is where we have the definition of all our services. We identify every service, and we know which type they are, and so forth. It's very controlled.

The goal is to create determinism from non-determinism. The non-determinism starts with the chaos in the channel, for example. That is the first source of non-determinism. Then we have a bunch of rules where we extract the data. Then, at the very end, the icing on the cake is the LLM coming and saying, this is probably a new relationship. There's a new connection here, and so forth. The paradox here is that we use stochastic tools to produce deterministic SPARQL queryable graph. The ontology is the bridge. It's where we save all the data. We identify five layers. This is a bootstrapping process. Deterministic extraction is those steps in green. We have a bunch of regexes, for example, that extract what a Jenkins job is by ID, by server. That's one entity. Those are very well-known patterns. Then the catalog validation is cataloging or checking against the service is a valid service deployed in this region and so forth.

We enrich with our APIs, observability APIs, and then we pass the LLM to classify and find new predicates. We also have a final step, what we call the probabilistic correlation, where we do the first search in all the conversation. We find chains where the incident started with this one application that affected application B, the original application was rolled back, and then the issue was fixed. That chain is identified also as an entity, a pattern that will be saved so that in the future, maybe we will find similarities, this incident with something in the future. Also, everything is constrained. We have categories, some taxonomies, and they are all constrained. When the LLM comes in, it wants to classify some category, say whether this conversation is an incident, question, discussion, or decision. It can only choose from those four values. We don't let it hallucinate new things unless maybe it tells us that, and then we'll see how we can manage that. Everything is typed through, for example, thread 123 is related, true is a Boolean.

The process we're describing is a bootstrapping process, what we call the knowledge flywheel, where every rotation, say a thread starts. We start identifying entities. More data comes in, we go around this flywheel by observing, enriching, inferring, and adapting. We observe the MELT data. We enrich with all these enrichers. We infer possibly new patterns or old patterns just to apply them, and we adapt to see what knowledge we can gather. The causal chain from one incident becomes a known pattern for the next. Observe, enrich, infer, that's the main process. In every iteration, we encode knowledge. Each cycle teaches the system, how the ontology will evolve this schema into memory. We have a process also where data is converting information, that information is encoded in knowledge. The fourth is wisdom, wisdom in the sense that we can find root cause. We can encode that today, but also in the future.

We're not predicting here yet. We're just describing always. Describing first. That's what a baby does. Daddy loves mommy, he knows that is a fact, but then more things will happen later. Everything is queryable via SPARQL. SPARQL is a weird syntax, but you get used to it. LLMs love it. We don't care. I don't even read it sometimes. Again, the ontology accumulates the structure of what keeps happening. Patterns become queryable facts.

It's not surprising that we use AI with this. We use Claude as our co-developer. When we get a new thread or a new message in an incident, we run this harvester. It creates a git worktree because we don't want to conflict with other branches, because there are many incidents happening, or maybe there's another iteration coming, the previous harvest can be working still on something and then it doesn't finish yet. It creates a git worktree or a branch. Then we use Claude also to harvest. We are not just running a script. We are running Claude with the harvesting script to extract the data because there's the LLM step that finds new patterns. At the same time, we use it to find maybe bugs in this very same script, so that it actually fixes itself. Then, say, that whole loop ends, and then we ended up with 15 branches only for this series of iterations, for one incident, and then human in the loop comes in.

Either me or Prasanna can come in and say, we got all these branches, and then we ask Claude to cherry pick and see what happened. It finds very interesting things. It finds new facts. Some, it says, these five branches did almost the same. They found this pattern. This other branch realized it was running on Sonnet 4.5, and it decided to upgrade itself to 4.6, which is cool. It's self-healing at the same time. You say, ok, merge. Then you keep going. It's a meta loop. The flywheel produces the knowledge, and the second one improves the machinery, and uses us for now. The idea is actually to make that non-dependent on humans and merge by itself. Then we continue iterating on top, so many flywheels on top, kind of a pyramid. Until it's completely autonomous, and then we move on to other domains or namespaces. This is how we see that in Slack, because Slack is convenient.

This is how it looks. For example, at each iteration, we find a few channels, and then, this happened to be a platform operations channel, a metrics/analytics engineering, all these classifications are something that it has found before. I never told it what it was, but I accepted it. I looked and spot checked. This is an alert management channel. It's not incident channels. These are general channels. We're also watching all the Slack channels as well. Because, remember, the idea is not only start with a small ontology and then go up into bigger parts of the infrastructure to describe them. This is just for general channels. This is what happens at the end of one iteration. For example, in this run it found some Turtle. Turtle is the way we serialize RDF in ontology language. It found some issues with the regex patterns, and the URIs were not saving correctly to valid Turtle.

It found a diff and so forth. That's what I can continue and say, you can merge this, and it goes and does it. All tests passed. The merge is complete and successful. Let me create a summary. That means for the next iteration, this code is already in the main branch. The next iteration, someone is adding a new message in the channel or a new discovery. Someone found a root cause that will actually run this new code. The next git worktree that spins up, it will already merge this fix.

The part that extracts these chains that I talked about, it's part of our correlation engine. It finds all these chains. For example, commit happened with retry logic. It was deployed with this version. Three minutes later there was an error, connection timeout, an alert fired, and the incident was started. Then the LLM comes up with its own confidence, but we save this chain. Many chains can happen in one incident. Imagine all these chains together, we'll have a lot of data in the future for fine-tuning and similarity. Agents are actually entropy reducers. Because, like I said, we receive noisy input. We spit out triples at the end, with new relationships, but at the same time, we find better ways of extracting data, because the agentic loop is happening. We do things, for example, like SME, Subject Matter Expert detection. This person is actually a Subject Matter Expert, and we find the topic too.

The accumulation of those triples will tell us eventually, this person is an expert in the service systems, or in Kafka and so forth. We also have these service topologies on a Turtle that already exists, because we build it from the ground up from many sources, using eBPF and logs and tracing. We created this graph of all our thousands of dependencies of services, which service calls each other. That's part of it. The contract is for the enrichers, we have a class, which is an interface where you can see the enrichment part, you can add triples to the graph and so forth. We have many of these, and more enrichers come as more data come in. Claude will actually tell us we need to build a new enricher because we're seeing this new thing that I don't know about from previous history and so forth. It's not in the code. It's not in the ontology. We need to create. Every URI in the entities are stable. We have a domain and IDs. We can always reference them by namespace.

From a graph, we go into a knowledge base. The process again, at every run of the harvester, we save the Turtle files in S3. We create a Parquet file. We have our graph database that is constantly polling that stream. It's actually a CDC, change data capture stream that loads those triples in memory. We can query in real time, because then we have the data. We have described the incident, for example. Someone else from a Claude Code session, or Cursor, or whatever can be querying what's going on with incident 123. We already have that data. The LLM will go make SPARQL queries and start asking questions, and then maybe correlate it to another incident in the past, and so forth. You use other MCP tools and so forth. We also save Blobs. For example, when someone uploads a graph, that file will be saved in S3, and we know it's related to a metric.

This is the query used. All that is described. The actual graph, the graph I mean the dashboard graph, it has a screenshot. The screenshot is saved, the bytes in S3. That Blob is also an entity. Everything is linked. We can recover all that data. Maybe you want to do forensic analysis and bring it back and get what happened exactly when things occurred. The ontology is a contract between chaos and understanding. Our stochastic engines are constrained to this very strong type system grounded on the ontology, and they emit deterministic triples that can be queryable. That's how we started with the incident ontology, a few namespaces, many agents and signals.

This is one of the artifacts that we produce on every run. This is just an HTML file saved in S3. We use this to debug, because I was just looking, is it extracting the right things? I know there has to be a Jira issue, or Slack users or file attachments and all those things. Every run, sometimes I realize, it's not finding file attachments. I need to have some artifacts, some goal, because I'm looking at the code all the time. I'm just looking at these results because I know what has to be in there. If it's not there, we need to keep fixing. Finally, by using this artifact, I could inspect. This is just a timeline of the incident. Everything is typed and linked. This is a big incident with many relationships. The topology, the service dependency is also encoded in there. Anyone can ask any questions, which teams are involved.

Who has low sentiment here? Who's very stressed, and all that? Everything is linked. Imagine all these incidents gluing up in the big graph so we can query them at the same time. The challenge is actually to keep all this data flowing in time, what we call the time travel. For now, we're just saving a few weeks of data. All this data is in S3, so eventually we can come up with a really big database that we can query at any time.

AutoSRE - Putting the Ontology to Use

How do we use the ontology? One of the applications is what we call the AutoSRE. It's our AI SRE solution. That's the name now in the industry, for site reliability, aiding SRE with AI. We call it that. What AutoSRE does is taps into the metric, logs, and traces in real time, in also Slack. It has the topology, it has devices as well, and ways of looking at the end-to-end graph, which is the devices. Taps into the ontology, so in the knowledge graph, which is higher creative relationships as domain concepts. Going back to the LOLOMO incident or issue, AutoSRE comes in, not only the on-call, and it analyzes all the MELT data and versions of all the A/B tests that's running, and then it does correlation much faster than humans. It finds what incidents or apps are involved in it, and so forth. It finds logs as well.

It uses that. We have a bunch of MCPs and CLI tools querying the infrastructure, not only in ontology, but also really the data in general. It also looks at code, as many solutions out there do, because you cannot understand systems if you don't go down into the code. This is what it produces. It says, there was a GraphQL error spike for this record operation, and the next steps are these. We have actually added interactivity to this, where users can ask follow-up questions, because it's really hard to get any AI today to give you the right answer on the first go, in the first loop. You need to go through many iterations to dig in, because the LLMs can wander and stuff.

What the Future Holds

What the future holds for us? There are three things. We want to find automatic root cause analysis. We don't want to keep digging. We can just give the crystal ball, this is the root cause. Next is to actually do auto-remediations. For example, if we know these patterns occur, the ontology, we'll say, in this situation, the best thing to do is just to terminate the instance. Fine. Do it. Auto-remediation. Without human interaction, even. The third one is self-healing infrastructure. That's the dream. We don't even want to get to an incident. We don't even want to get an alert. We want to keep shifting left and left, all the way to the point that the infrastructure is healing itself, because by the time we describe not only our incidents or our operational data, we will merge ontologies all the way to the data lineage to the point where, say, a movie is created.

There was a data error in a pipeline that was run by this team, by this person at this time, that cascaded into finally bad clicks on the user service. That's the end-to-end dream that we want to create. Not only forming the ontology, but merging into other bigger domains and create this big enterprise ontology that describes everything. That's going back to the original end-to-end wheel that Prasanna showed. Finding this huge cycle of connecting all the smaller ontologies into big.

Does anyone know this character from "Dark?" They are actually time travelers. There is a cabal that were able to harness time. They were able to go in the future, and actually, also knowledge. The inspiration here is I'm using one of their mantras. It says, "Der Anfang ist das Ende und das Ende ist der Anfang." "The beginning is the end and the end is the beginning." That means observe, enrich, infer. I just wanted to end that with that analogy to our show.

Questions and Answers

Participant 1: How do you deal with deduplication and disambiguation in the graph on those incidents?

Renzo Sanchez-Silva: Right now, we're letting the LLM go into this open world. It does lookup, and it will create the triples, and say, no, this relationship is already there, and we let it create one. It is going to be a branch that has to be merged, but I'm just letting it pass, and it just keeps appending to the ontology. We may get to many similar concepts, maybe. We haven't checked that, but ideally, for example, for this smaller ontology, the process has stopped in the sense that it's becoming stable. I see less PRs that we need to merge, and so forth. It does eventually become stable. It does have to go through a human. Say an ontologist needs to actually see this, look at it closely. What I see is that deduplication is very controlled. This is possible.

Prasanna Vijayanathan: In the future, we also see the need to have a pattern extraction layer that can generalize patterns to help with deduplication. We're not there yet. We're able to manage.

Participant 2: In these graphs, everything can start relating to everything else. How do you fine-tune your infra to give you more signal over the noise, so that the things that it outputs are actually not something that has been hallucinated or possibly related, that it's actually actionable, real, and a signal for the user.

Anand: How do you remove false correlations, is that possible?

Prasanna Vijayanathan: Going back to old testing systems, we have a Swiss cheese model. We want to be able to give as much accurate and actionable information as possible, and the best person to tell us that would be the users right now. Like Renzo mentioned, it creates and self-heals itself, so that also exists. We have different quality metrics that we track. We do online logs, and then extract metrics from that, and then we have evals systems. It's like, we freeze our inputs and outputs, and then keep testing how our model is doing. Then we have human-in-the-loop feedback. All of those systems feed back into our model, and we haven't done that extensively yet, but it'll get into some kind of reinforcement learning to use those signals, and improve itself.

Participant 3: I'm a bit intrigued about the scale piece, because it seems like the whole graph is in Turtle, created by PRs like code. This is a small subset, then, of those 32 million per second queries, and we're not relating those bits. It's a much higher-level construct. Is that correct?

Renzo Sanchez-Silva: The idea is, for this high-volume throughput, we're just going to be collecting metadata, fingerprints of something has started. We have found this pattern, but we are not going to get the data on Turtle. There's no way to do that. We're going to find the fingerprint of what's happening, and then relate that for the rest of the graph. It's just always controlling cardinality as well.

Participant 4: How do you detect that something important didn't happen, instead of just picking up something bad that happened?

Renzo Sanchez-Silva: The ontologies actually have a very powerful formal reasoning. It's not AI-related. This is actually proportional, logical. Very hard algorithms that actually can reason, can find something that is incongruent or transitive or contradictory. We also plan to use formal reasoning to catch these things as things progress. Not only to find inconsistency in data, but also to find new connections. We can do formal reasoning with non-AI, but we can also use AI to help that base reasoning.

Prasanna Vijayanathan: A human in the loop also helps a lot in that system, where a developer notices something that was not caught, or we notice an incident or an issue that came up that actually had to be solved. Then we would look at why the system didn't catch it. In fact, recently we had a case where someone flagged it and said, why did you not tell me this? Then it was able to correct itself, and then create a new pair. That's the other side of it.

Participant 5: Could you tell me how often you would allow your ontology to change? How does that affect your knowledge graph rebuild?

Renzo Sanchez-Silva: We are letting it go with the open world assumption. It seems stable, at least for the incidents, but we're also going to find connections with alerts. That's a very small ontology, because all we want to catch is the metadata of the alerts. There are very few keywords or concepts in the alert, for example, and the signals. Then for other ones, like pipelines and jobs and all that, there's also a way to control that. It's very small. What is harder is the incidents, because anything can happen. The concepts are growing, but like I said, it becomes stable. At some point, you need to go back and look, we need to compress that with humans, and then it will stay controlled. We need human in the loop, for sure.

Participant 6: Any plans on open-sourcing?

Prasanna Vijayanathan: Maybe. Not yet.

See more presentations with transcripts

Multi-Agent Patterns from Spotify’s AI Powered Advertising Platform

Pratik Rasam — Wed, 07 Oct 2026 16:00:00 GMT

Summary

Pratik Rasam discusses how Spotify Ads Manager runs production-grade multi-agent systems at scale using Google ADK Java. He shares key architectural patterns, domain ownership models, deterministic guardrails, and tracing-based evaluation strategies, detailing hard-learned lessons on drawing agent boundaries, optimizing tool schemas, and managing costs while avoiding monolithic agent pitfalls.

Bio

Pratik Rasam is a Senior Engineer at Spotify, where he builds Ads AI – Spotify advertising’s multi-agent platform for AI-powered advertising. He has co-authored on the Spotify Engineering blog about multi-agent architecture for smarter advertising.

About the conference

Transcript

"Learn how to run AI at scale at QCon AI conference. Moving AI from experiments into production increases pressure on platforms and teams. Hear senior peers explain how to control inference latency, build safe agent guardrails, and standardize internal platforms. Plus an early bird price until May 12th. Register now".

Pratik Rasam: The ad that you just heard, the script, the audience, the targeting, all of that was built by an in-house production platform that we have within Spotify Ads Manager. If you're into advertising, you'll know that creating an ad involves copywriting, music generation. All of that is now not needed. All we need is just the advertiser to describe what they want in an ad. This is not a research talk, but these are actual production agents at scale running at Spotify Ads Manager. I will also be talking about the mistakes that we did while developing this platform.

Background

My name is Pratik. I'm a senior engineer at Spotify Ads. Today I will be going over the platform that builds Ads AI. Ads AI is basically a multi-agentic platform for Spotify's advertising solutions. Let me give you a scale at how we are performing within the GenAI space. Since we launched last year, more than 70% of ads currently are using AI tools. We have around 20,000 creatives that have been generated for over 7,000 advertisers. These are real ad campaigns, real money, and real audiences. This talk is not just research. These are actual production agents that are running and delivering value. Today, we have most of our creative generation primary pipelines being generated by Spotify Ads AI. Before we dive into the architecture, this example is something that we'll be using throughout the talk for different phases of the architecture design. Let's imagine that we are creating an ad for QCon AI.

The brand itself is QCon. The message that we want to have our listeners is, let's say if you want to have production AI playbooks for senior engineers, for shipping AI solutions at scale, QCon AI is the conference to be. Our target audience is senior engineers, architects, engineering leaders. We want to target major U.S. tech hubs. Our CTA, which is the call to action, is, Register Now.

What 'Ads AI' Actually Means

I want to make sure that we are on the same page. When I say what Ads AI actually means, imagine the advertiser says, I want to run an ad for engineering leaders across the U.S. tech hubs. This is natural language. What we do is extract this natural language into specific intent for audience. Over here, we are talking about the audience to be resolved to technology, and the geo-targeting should be for U.S. The goal is for awareness. We are using this LLM gateway using Vertex AI and our internal GCP platform. When it comes to the extracted intent, it is then provided to our multi-agent orchestration layer, which runs on Google ADK. We are using Google ADK Java. We have parallel agents that are composed per responsibility. For the ad that you just heard, we have an Ad Script Generation Agent. We also want to make sure that the ad conforms to our policies, so we have an Ad Guardrail Agent. We have an Audience Recommendation Agent that we have currently running in pilot. A combined output of these agents eventually results in two discrete objects, which we have the audience recommended and the creative that is created.

One Agent, One Package, One Owner

Before diving into the patterns, what I wanted to show you guys was how we built this architecture in the first place. When I talk about agent, I'm talking about one agent, one package, and one owner. We have explicitly enforced this requirement because when you have the agents composed as individual packages, that package itself is composed into different aspects around the ownership, the responsibilities, the monitoring, and the prompt itself. If you see in the agent ad script folder, we have the Bazel package, which involves the dependency management of who is allowed to access this agent. We have the lib-info. The lib-info is something that determines ownership with regards to what team is owning that agent. We have an internal component that resolves the PR reviews. Let's say if you are changing a particular attribute within this particular agent, that is the ownership change that the team will go to.

Monitoring-info is basically a YAML file that has information about your dashboards and your PagerDuty alerting. You have all of this provided by our platform where before even writing a single line of code for the agent instructions, you have metrics, you have alerting, all in place. AgentFactory, Adapter, Service, and AgentModule are dependency injection scaffolding that provide a holistic package for the agent itself. Note that the LLM configuration for the agent is stored as a front matter. This is what makes the architecture a little more interesting. When I say different teams are owning different agents, we are enforcing this Bazel visibility at the horizontal layer. When I say horizontal layer, I'm saying that the ad script generation agent will not be allowed to use the audience resolver unless and until the audience resolver allows it to be imported. This enforcement is at the compile time layer. No comments.

You basically get a build failure when you are not importing that particular agent specifically. Other than the team-owned components, you have the shared platform. The shared platform provides capabilities such as metrics generation, traceability, and most of our tool usage evolves around our developer API for Spotify Ads, which calls Ads API. We also have an additional moderation layer called kutest, which provides guardrail policy management.

Agent Architecture Stack

When you zoom out at the team level, this is what the overall tech stack looks like. We have one gRPC service. We have multiple agents. We have the runtime context, which is controlled by Google ADK. Team A owns its own agent and its own tools. Team B owns its own agents and its own tools. Team C can own its own agents and own tools. All of these agents can eventually run on a shared context and a shared runtime ADK. The tool traceability, the tool context, and the runtime loop is all managed by the platform. From an architecture point of view, here's how the flow looks like. You have the client applications making the service calls to our gRPC services, which also manage the session management. Whenever a client is making a call, it is responsible for creating a session with the agent, which eventually goes to the AI agents layer.

The AI agents layer is where individual agents actually reside. We have enforced safety and orchestration at a separate layer and not within the agent. This is because we want to have common orchestration and common guardrails to be consistent throughout the agentic experience. The last level is the model and the tool platform, which basically is the LLM runtime, the plugins and the observability configurations that are running for all of the agents. Throughout this talk, we have three attributes of this particular platform, safe content, modular agent platform, and flexible integration. These are three pillars that we want to have throughout the agentic interfaces.

Thesis, and Playlist

This is perhaps the most important slide of the first half of the presentation. When you are dealing with multi-agent architecture, the question you always ask is, who owns what? Agents are owning the sentiment, the LLM judgment, the reasoning aspect. In our example, the use cases are like audience intent. We want to understand what the advertiser means when they say I want to target Gen Zs for our sneaker ads. We want to create the ad brief from all the signals that we get from these. Also, we want to identify sensitive topics. Let's say if a particular ad description is not conforming to Spotify's ad policies, we do want to prevent that ad to be generated. When it comes to data, we really don't want the LLMs to guess what the geo-interest would be. Within advertising, we have specific interest segments, we have specific DMA targeting. All of this data actually comes from our API endpoints.

Spotify has publicly owned web APIs that our tools are today accessing. All of this grounding happens at the tool layer. The third aspect is the application code itself. When you ask me, should everything be controlled by agents? My answer would be no. This is because whatever can be deterministically coded should always be deterministically coded. I'll be providing examples throughout this presentation where we had to establish some guardrailing and deterministic gating in order to achieve the goals that we wanted to have.

This is what today's playlist looks like. I want to start with why multi-agent at all. We don't want to just have agents because they are cool. The motivation behind having multi-agent is controlling the blast radius, having responsibilities shifted for each and every individual agent. We are then proceeding towards where to draw the agentic boundary. This is perhaps the longest discussion because it's also the most difficult to have. It is very important to understand where the boundary lies between what the agent should be doing and what should be deterministic in our code versus tools. The third is tool design. Tool design is basically prompt engineering. I say that because when we define a tool, we also define its schema. The schema has description about what the tool is doing. This description is not just documentation for developers like us, but it's also something that is going to the LLM itself and helping the LLM understand what and how to exactly call the tool. Then we proceed towards having talked about reliability through eval and tracing. In order to achieve evaluation, we need to have tracing first, and that's what we'll be stressing upon. Last but not least, I'll be talking about what we would do differently. This highlights some of the mistakes that we had done initially in our agentic journey.

Why Multi-Agent at All?

Let's rewind 18 months back in time and think about the things that we would expect from an LLM. If we see, like last year, parsing JSON from a particular LLM model was not that straightforward. It would have been a JSON-ish response or it would have been like a Markdown with those three codes. Over time, models have gotten better. The JSON parsing, the schema checking, and business validation is now very much adopted by the model that we are using today. The third, the validation of the business logic is very important because whatever values are returned by a particular model, we can now add additional deterministic checks, whether those make sense or not. This is something which I will be talking more about in the later slides. Rewind back again to one year. If you had to write a parallel execution of agent with custom session management, you would probably have to write something by yourself.

Today, frameworks like LangChain4j, Google Cloud ADK, or CrewAI, these are frameworks that give you those primitives. The question used to be, how do we design it? Now, what should I do to implement it? When I say operations got observable, it's a supply side change. What changed in observability is the OTel GenAI conventions established some standard observability conventions for GenAI workflows. These have enabled to have generic conventions for tracing. Earlier, what used to happen was teams used to write their own annotations, teams used to write their own tracing. Now these semantic conventions allow standardized observability for GenAI workflows. With these standardized observability traces, you can now trace and replay the judge models. Replaying production traces enables us to actually test what is going on in production and we do not have to create synthetic tests. This eventually prevents us from having that black box behavior where we don't really have any idea what is going on in our agentic system.

The fourth one is interesting. It's on the demand side. The first three initiatives were more on the supply side where the technology got better. The fourth one is where the demand side increased its requirements. The multi-step pipelines have always been there in the traditional software development. What changed was, within every step, we now involve LLM interactions which can have multi-entry products. When I say multi-entry products, an advertiser can start their campaign creation on their mobile app and then continue the creation on ads manager or Ads API. Each of these entry points can have different intent and different context. Stitching this context together is only possible when it comes to a multi-agentic flow. The last one is the closed-loop measurement. When we have ads created by GenAI, we also have the ability to measure the performance of those ads. When this performance is funneled back into the system, we do know what works and what does not work.

That closed-loop measurement is something that is attributed to an overall success of an ad campaign. Multi-agents were always there. It's just that these four, three supply side and one demand side features got more better over time. We had three supply side and one demand side. Until early 2026, multi-agent was a research demo, now it is an architecture.

Let's just talk about how a campaign creation process looks like. Whenever an advertiser logs into the Spotify Ads Manager, they are providing their objectives. Like, what is my objective for my ad campaign? What is my goal? What is my KPI? What are the success metrics? This is followed by the targeting. Whom do I want to target? When, as an advertiser, I'm specifying whom I want to target, I'm specifying where I want my ad to display. What are the interest segments? Let's say I want to target technology enthusiast for QCon. Also, I want to add demographic filters. I then proceed towards the budget schedule. How much I want to spend, the flight dates, and the pacing rules. Then comes the creative uploading, which is what you guys heard in the first slide, where the advertiser provides details about what they are providing as their creative. The problem is clearly visible over here, where at the creative level, the context really dies here, because each step is isolated in its own resolve, but it does not provide the holistic signal for creating and creative.

Let me just walk over how this would look with ads manager today. We have this QCon Boston conference campaign created, where the advertiser provides their age targeting, the detail targeting has technology industry, and the languages for all the languages. I want to have this campaign for all the devices. This is an audio campaign. I want to optimize for impressions. I want to have this campaign for a daily budget of $15, let's say. Now I go to the next step. This is where I give my ad details. What is my advertiser name? What is my tagline? I should also be uploading images, which I have not. We are going to focus on the audio generation aspect of this presentation. This is exactly how the ad was created that you heard in the first slide. The advertiser provides the main message, the call to action. The call to action is something which is similar to what we provided in the slide before.

When the advertiser selects generate, they can see the script that is generated. I already have one script that is generated, but what we do allow for advertisers is to have multiple generations so that they can choose which script they want for their campaign. As you saw, while the advertiser was performing the task of creating the ad campaign, there were certain signals that were pretty much relevant in the ad creation. Today, when the advertiser is providing the inputs for creating the ad script, they are providing the brand, the message, the goal, and the ad category. These are five leaf node signals that are used for creating the campaign. On the contrary, on the right side, you can see a composable system where not just these five signals, but you can also use the resolved audience, the locale, past performance of the campaign, geo-age, gender combination, ad category, delivery-goal group. All of this can definitely provide additional context and additional information for creating a better ad. This unlock is definitely something that multi-agent systems promote.

Where to Draw the Agent Boundary

Let's just do a comparison at a core level. At a function style, when the advertiser is performing the task of providing the brand signals, they are providing the asset and they are getting the brand tone and the call to action. When they are doing the generate script, they are providing the brand name and the CTA, as you saw in the previous demo. Over here, you don't really have the brand tone. Similarly, when you're doing the recommend targeting, you're providing the brand name and the objective, but the brand tone and content themes, they are not present. When we have an agent tech-style interface, you have the processes extracting their own signals and their own session context. After recommended targeting, all the agents have to do is just use the same shared session state, where the context accumulates across every step. At the end of the process, you have the agent getting a handle to the brand tone, the call to action, the themes, everything makes the ad creation more holistic.

This is the slide where we eventually want to go. Today we have campaign context, where the brand tone and objective are fetched from the advertiser. We also have creative outputs where scripts, visual taglines, and formats are provided from the UI display. When the campaign is submitted, it is running live. At that same time, we get the targeting information, the geo attributes, where the ad performed well, and we have the performance signals. We have the click-through rate, the completion, conversion rate, all of these signals. Imagine the closed loop actually performs as an input to our funnel, where we can say that conversation scripts with bias prompts and audience selection are actually performing better. We can tune our targeting and we can tune our model in such a way that the funnel that goes back to the system is actually giving meaningful insights about what works and what does not work. This bi-directional flow is something that multi-agent is definitely enabling us to do.

Before we got things right, let me begin with what we got wrong. As all enthusiastic AI experimenters, we started off with one single agent and a mammoth of an instruction. This agent was responsible for script generation, guardrails, audience resolution, search behavior, validation, JSON schema, and the tone and locale. It had all the tools. It had everything in one large block. Imagine something like a function with 1,000 lines of code. It worked, but it was brittle at the same time. As instructions changed, the behavior of this agent got somewhat hallucinated. We did not have an isolated approach towards identifying what is actually wrong, which is happening. This led us to determine that if you want to have dedicated responsibility for dedicated agents, this is something that we need to split as a part of our initial post POC. This is the decomposition we landed on. We are basically having three agents.

We have the audience resolver, which is in pilot right now, but what it's responsible for is resolve the geo-targeting, resolve demographics, resolve interest and exclusions. Basically, if an advertiser brief has women 18 to 34 in Nashville into fitness, this agent is resolving the interest and the geo-targeting. At the same time, we have the ad script generation, which is running two agents in parallel. We have the ad script guardrail agent, which does specific brand safety checks. We have the ad script generation agent. One key thing to note here, that this agent bifurcation is also dependent on the ownership teams. Within ads, we have different teams for different domain aspects of the ad creation and submission process. We have a team for audience management. We have a team for creative management. The creative management team owns the creative ad script generation agent and the audience targeting team owns the audience resolver agent.

This is a good heuristic to follow. A question that we always get asked is, when do you use something as an agent? When do you actually have code? If you ask me, models are getting better, but they are still non-deterministic. If the semantic of the wording of your behavior is changing, that is definitely the agent will handle or the LLM would handle. Things like senior engineers, things like technology leaders, all of this is something the agent can handle. On the contrary, if your rules change or if your catalog changes, or if your data change, that is something which is code. One good analogy is, if something that can be computed in code is computational and can be tested using unit tests, definitely have it as code. We learned it in a very interesting way. When we have targeting of audiences, on the ads manager platform, we currently support these age brackets.

We had a case where one of the inputs was women 21-plus in Nashville. In this case, the agent just emitted 21 to 99. I hope someone lives 99 years, but this was unfortunately rejected by our downstream systems because it was not part of the age bucket. The fixes that were required were pretty much obvious. We had to enumerate the seven age brackets explicitly that rejected a specific non-supporting age. The second bug was even more interesting. If a prompt says parents of teen 13 to 17, the agent actually recommended 13 to 17 instead of the parents. The fix that was required here was not the demographic, but actually the persona. These are the cases that you will find as you are testing your agents that even if you're using the latest LLM, there will be cases where you will find niche behaviors that you might find surprising that the agent gets wrong.

Another case which is interesting is the geo-targeting. A user can just say, I want to target the South and the Midwest. These are very generic terms. The South in the U.S. means 13 states. The Midwest means also 12 states. If you're not providing the country code as an additional attribute to your tool, Georgia can be resolved as a country and not the state. This is something that we learned that when you have that additional context being passed in the tool itself as a guardrail, that gives more context to the agent to resolve what specific targeting is to be provided. One more optimization that we eventually landed upon was we looked into anything that can be done before calling LLMs. Before we actually had like a multi-agent pattern for our ad script generation, we were having individual LLM calls. The problem was the guardrail was actually running irrespective of the generation.

Whenever the guardrail was giving us a false result, that was wasting token usage for us because there was no output being written for the user. We have an internal moderation classifier that actually gives us classification. What we did was we called this internal moderation classifier first. If that fails, we return early. We are doing a gatekeeping check first and only then calling the LLMs. Whenever you find cases where you might have existing solutions within your organization that can give you the output without calling an LLM, do that, because on the larger scale, you will be wasting a lot of tokens.

Prototyped Patterns

Before we actually dive into the key takeaways for agents, I wanted to talk about the patterns that we have prototyped. This is a very common question that we get asked that, if I want to write agents, how should I write them? Or, what are the patterns that I should be using? Within our project, what we have done is Google ADK comes with a dev UI. We have scaffolded certain patterns that we currently have for our engineers, which I'm going to share with you guys today. These are primitives which are composed and modified to give different behavior based on your requirements. We have the basic router. We have a conditional router. We have the sequential pipeline, which is a very common workflow that we have. Fan-out, fan-in is where we want to have parallel execution. The last, which is the generator critic loop, where you want to refine an agent output continuously till you find the best output suitable.

Starting with the basic agent and tools. This is the simplest agentic call that you'll see, where you have a single agent and you have specific tool sets that the LLM has access to. It has zero orchestration overhead, easy to unit test and reason about. Things to watch out for is the prompt load. Eventually, you might see that 3 to 10 tool calls might be fine, but 8 to 10 is where your LLM might get hallucinated. Wherever possible, think about splitting the agent into multiple agents if you feel that you're calling too many tools at the same time. The next one is a sequential one. This is a very common pattern that we have in most of our workflows. Sequential patterns are basically a series of agent runs where one agent gets output from the other agent as a form of the output key. At the core level, you have zero glue between the steps.

These can be determined by the sub-agents for a sequential agent. The session state itself is maintained via an output key. After each agent runs, the output is provided by the output key parameter. It's very easy to add and remove stages. Things to watch out for is additive latency. The more agents that you will add in your workflow, obviously, the wait time increases. There is less scope for parallelism if you are sequentially running all these agents. Try to have the agents run in parallel if you know that these can actually run in parallel. I'll be talking about the parallel fan-out as well. One more thing to watch out for is the upstream failures can poison the entire chain. Let's say if the researcher failed in this situation, the writer will not be getting an output. It's like a single-point failure for your sequential flow.

This is an interesting agentic pattern which we have used. What you have over here is Java provides like Flowable for events. If you are extending the base agent, you can always override the runAsync function. Over here, what we have done is we have a classifier agent that classifies a query into either a billing or a technical question. If it's a billing query, it goes to the billing agent. If it's a technical query, it goes to the technical agent. We also have like a default pattern that goes to a general agent. When it comes to determining which fork to use or which route to use, this is a very interesting pattern to watch out for. The router in this case is a single point of failure. If your router itself is not able to determine what task to-dos, that is when you want to check a bifurcated approach or a backup approach for your default routing.

You do have an additional latency that adds to the router itself. Models can also over-classify intent. Let's say you have this example wherein a technical agent and a billing agent are actually doing the same things. In this case, it's very difficult for the agent itself to really determine what the intent of the user is. Parallel fan-out and gather, as the name suggests, is agents running in parallel. When you absolutely know that agent A and agent B can actually run together, these are the patterns we can use to run these agents in parallel. In our case, what we are doing with the ad script generation is we are running the ad script guardrail and the ad script generation in parallel. In this example, we have an auditor agent and a style checker. You have these two agents running in parallel, and you have the reviewer agent that reviews the eventual output.

One thing to watch out for for parallel agents is each agent does not have context of what the other agent has actually run. That is something that you should keep in mind when running parallel agents, that what was available for a sequential agent is not available for parallel agents. Obviously, token consumption is more once you have a linear scale for your parallel agents. The last one is more of a refinement approach that we have. It's called a generator-critic loop pattern. In the example, you can see that you have a SQL generator and you have a SQL critic. What the SQL generator is doing is basically generating a SQL, and the critic, what it's doing is it's providing whether that generated SQL is actually a pass or a fail. What you want to do is you want to keep on checking until you have the critic give its go ahead.

There is a possibility that the critic might just run indefinitely. What loop agents have is this attribute called maxIterations. These allow us to prevent runaway token consumption, kind of like an exit strategy. Things to watch out for is like, each iteration is a full LLM round-trip cost, so costs can multiply. Models may fail to converge without a clear exit strategy. Even in this case, you can have situations where you can run into a race condition or something like that. The critic is also an LLM. You're actually paying twice per refinement step.

Boundary Takeaways

Going back to the takeaways. This concludes our agentic talk for what we have done with regards to agents. Agents own reasoning. You want to identify the intent, the range, the context using agents and not arithmetic. Anything that is deterministic should be code. Anything that is semantic or intent based should be agents. When it comes to validating agent outputs, schema validation is something that we currently have on each of the agents. Missing field guards, business rule validations, and null checks, those are the things that models are completely capable of doing today. Wherever possible, you want to identify the patterns for sequential versus parallel. If you have a use case where you want to conditionally exit out of your agentic runs, that is also possible today. All of this is encapsulated within a gRPC service boundary. Eventually you get your final call for your gRPC service that has the agent output, which is your holistic response. As a last takeaway, wherever possible, you should avoid an agent doing multiple things. Just like a single responsibility principle, the smaller and more concise your agents are, the better the performance is.

Tool Design is Prompt Engineering

We'll now move towards tools. Tools are the hands and legs of the agents, that's a way to describe it. One of the things that is very underestimated is the tool schema. What is a tool schema? It's an annotation that provides the LLM how the tool should be used and what the description of that tool is. It's basically the contract for the LLM, and not just like shared documentation. The descriptions of the tool are something that are very important, and we'll talk about a few examples how these tool descriptions have actually helped us to determine geo-targeting. The way we have currently our tool setup is we treat tools and APIs differently. On a broader level, we have Ads API, which has standard web APIs, but we have agents that have different intents for calling the different tools. Now you'll understand why we have bundled agents and tools together and not as a shared entity.

If a team owns its agents, it also owns its tool. In this example, we have two tools, which are called searchGeoTargets, and a hypothetical agent that we may have in the future called lookupGeoById. The intents of these tools are completely different, but the API is the same. A key takeaway is that your tools should not be tied to just API wrappers, they should be tied to the agent intent and what the agent actually requests. Augmenting the tool with the agent requirements is more important than what API it is providing. As any other software, tools are subject to failure, and it's very important to have tools fail in a very gracious way. When I say gracious way, basically, when a tool fails, the agent should be able to understand what exactly happened. In the example for this slide, what I'm trying to show you guys is, let's say if there was a call to the interest segments, and the interest segments API, for some reason, is not providing the interest ID. There are two ways we can handle this. One is, you don't even handle that error, and you get this nice internal server error where the agent does not know what to do. Or, on the other hand, you can have a structured response, which gives the agent an idea of what exactly is happening, and exit gracefully.

Tool delegation has its different patterns and different compositions that I want to talk about. First, you can map tools as a one-to-one upstream API. This is where you want to invoke a particular API call for a particular client search. The pros of this can be like, you don't really have any overhead, you have minimal latency, and it's kind of like a direct contract. The second approach is, tools can compose multiple API calls into one single tool call. This is useful when you want to aggregate certain API behaviors in one tool functionality. Keep in mind that tool calls are also LLM calls, so these are also expensive. Whenever you are making a tool call in your trace, you'll see that that additional LLM call is also adding to your latency. There can be times where tools need a session state, or tools can actually upload the session state.

Tools have their own context called ToolContext, which can be threaded to the session scope. If a tool wants to update or read from this context, you can get that context. In our example that I mentioned, we are getting the country code from the tool context. Going back to the first slide, we have the same upstream, but different agentic needs. You can have two different agents using two different tools, but the same API. The key takeaways of the tool designing are pretty much straightforward. Whenever it comes to call bundling or splitting, you need to determine what needs to be clubbed together or what needs to go separately. Schema-level grounding is something that tools can have in the description itself to avoid static references to the data, and expensive calls can be avoided. Errors as recovery prompts, as I mentioned, are very important in terms of handling agentic tool failures and giving the agent more context around how to exit gracefully. Multi-agent wrapping is something that multiple agents can wrap the same upstream API calls with different tools.

Reliability Through Eval + Tracing

Now that we have talked a lot about tools, let's talk about the things that can go wrong, the things that need to be tested, and the bad pattern that we initially started with. Being traditional software engineers, we love to write unit tests. These are nice functions that run and give us satisfaction of, things are running amazingly well. To boost our confidence even more, we have integration tests with mocked LLMs, where the pipeline wiring and every gRPC integration is made sure that we are wiring everything correctly. What happens to the LLM call itself? The LLM call itself is something non-deterministic, and anything that is non-deterministic is susceptible to hallucination. The approach where we say, we know it's bad when some complaints come in, that is a very bad sign, and comments like this should be avoided. When it comes to evaluation, what I want to focus on is the prerequisite for evaluation, and that is tracing.

If you have an evaluation that just verifies the output without the trace, that's mere theater. The reason I say that is the trace not only has the output of the agent, but also the journey of that output. When I say journey of that output, you have different metrics for the agent that tells you the duration, the tool call, the moderation, latency, and we also add our own custom GenAI span that records the raw input and the output. ADK offers a separate metrics plugin that we can use to bundle this behavior to each and every agent. Once we have those telemetry, we can export those telemetry to either OTel for monitoring our Grafana dashboards, or we use this third-party vendor called Braintrust that shows us a very nice demographic display of how the agent calls look. The three levels of evaluation are the tool trajectory. Tool trajectory is verifying that the agent is performing the right steps. Deterministic assets, things like schema validation, Geo ID structures, and age brackets in our case. LLM-as-judge, so identifying the tone, the brand fit, the semantic coherence is something that scoped LLM-as-judge provide a very good reason for us to have like a last evaluation step.

The last thing which I'm going to talk about is live evaluation. Live evaluation is something that is proving to be very useful. When you have an agentic run on the trace, you can actually have an evaluator which is running on live traffic. It's not a good idea to have live evaluation run on 100% traffic because that also adds to your cost. It depends on your use case. If an agent is new to the market, then definitely start with a higher value. Once you have pretty much confidence around it, then you can probably change the value. These are completely configurable values which we can change over time. One important thing that we have actually done with evaluation is we have not attached evaluation to merge builds. What we have figured out is every time you have a merge build to your repository, you are actually making LLM calls.

That is an additional cost step that we wanted to avoid. When developers are writing their own agents, they are iteratively running their evaluations offline. We do have like a nightly process that runs the evaluation for the entire suite. This is how a trace actually looks like. This is what each call and cost of the run agent for the ad script generation looks like. Over here, you can see that the two parallel agent calls are running. At the bottom, you can actually see that the ad quality scorer is running at the end. It's defining your output and determining whether the agent response is good or bad. The key eval takeaways, as you can see, are obviously, we want to have trace for every production call. We may not have live evals for every production call, but tracing is definitely something that helps. The three levels of evaluation, the trajectory, the deterministic, and the judge are the three pillars that we want to have.

We don't want to have the evaluation run as a CI gate. We want to have this on-demand and probably like a nightly build. Live evaluations definitely carry a very important signal because these are production traffic. One key thing to identify is like judge is equal to detection and not prevention. Every time you have the judge detect an anomaly, what you want to do is you want to ratchet on that anomaly and add a new assert. The more asserts you eventually have, the less expensive your judge becomes.

Conclusion

To wrap things a bit, what we would do differently, starting with the agent architecture, is not have a mono-agent, a big, large token usage. Basically, tool design, observability, and prompt routing is something that we want to not have like a global system prompt. What I want you to take away today is you have different teams owning different agents. You have reasoning flowing through the orchestration and reasoning coming back to the funnel. That's what the whole multi-agentic architecture currently looks like.

See more presentations with transcripts

The Reinvention of the Dev Team

Hannah Foxwell — Tue, 06 Oct 2026 16:00:00 GMT

Summary

Hannah Foxwell shares how the surge of agentic coding forces engineering leaders to rethink team dynamics. She discusses three key anchors for navigating AI-driven velocity: building software worth building, prioritizing automated safety over speed, and preserving the human element through generalist "broken comb" skillsets and sustainable on-call practices.

Bio

With over a decade in platform engineering, Hannah Foxwell advocates for the human side of tech evolution. Relentlessly curious about tools, processes, and practices that improve life for software teams, she builds high-performing engineering organizations. Today, she works as an independent consultant at the strategic intersection of platform engineering, security, and agentic AI.

About the conference

Transcript

Hannah Foxwell: My name is Hannah Foxwell. I'm here to talk about the reinvention of the dev team. Let's start by talking about velocity, because as a profession we have been obsessed with velocity. A reliable, responsive, high-performing engineering team was the goal for so many of our organizations, but they were really scarce, and we always felt like we were late. We always felt like we were too slow, that we were always behind, that velocity was hard to find. We got better over the years. The first software development team I was a part of, we shipped twice a year. It was a mainframe team, so it was a little bit different, but we shipped twice a year. I then moved into a more modern e-commerce operation, and we were doing well if we shipped four times a year. Through the course of my career, we've got better at that velocity thing.

Agile came along, and for a lot of people it was absolutely mind-blowing that we might even try to deliver once a sprint, like once every two weeks. Then we went through the cloud transformation, infrastructure was on demand, we could ship to an on-demand environment, that unlocked another level of velocity. Then with DevOps and continuous delivery, we got even better at it. It is not unheard of, and I'm sure it is common with people here that you're shipping multiple times a day to your products and to your platforms.

We Are All on a Journey

Yet, even just a couple of years ago, you would have come to a conference like this, and people were talking about developer productivity, people were talking about developer experience, people were talking about platform engineering as a way to unlock even more velocity. I feel like maybe that velocity has arrived. We're not quite sure what to do with it yet. This is the eight stages of agentic coding that was proposed by Steve Yegge when he announced Gastown, which is an agentic coding orchestration platform. This format of it has been publicized by HelixML, who are building an agent orchestration platform. It's absolutely bonkers to me to watch these teams work with their fleets of agents. I don't know whether you're working in that way yet. Give me a wave if you've tried Gastown or you've tried working with a fleet of agents. For the most part, for most people, this is theory still.

This is a frightening new reality of like, I'm not actually going to go into an IDE and prompt my way to code. I'm going to be driving it through specs, through tickets. The agents are going to be breaking down the weight. The agents are going to be delivering the software. The agents are going to be writing the tests. The agents are going to be deploying to production. The agents are going to be monitoring. This is a little glimmer of what the future might look like for all of us. It might be coming faster than we realize. Earlier this year, there was an inflection point in Cursor, the popular agentic coding platform Cursor, one of the first ones. They called it a third era of coding because now within Cursor itself, there are more agent requests than there are tab accepts. Agent-driven development is the majority of the work that's happening in Cursor. That's happened very quickly, you can see by that graph. That's because the capability of these agents keeps increasing and the reliability of them keeps increasing.

Thesis

What I am going to talk about today is not the agents at all. This is a talk about AI, but I'm not going to talk about the agents at all. I'm going to propose that they are coming, that they are here. I'm going to talk about the humans and what we do as teams to adjust to this massive change that is happening in our industry, the velocity that has finally arrived. If you do want to learn about the state of the art with agentic engineering practices, context engineering, and getting the best reliability out of your agents, go to Patrick's talk. This is the talk that you come to if you want to know what that means for me as a human. First of all, I want to reassure you, don't worry. This is really bleeding-edge stuff. Most organizations aren't ready to consider that a dev team with a fleet of agents is going to be shipping to production.

I think most of us should definitely consider playing with this, because the opportunity to unlock the value of software for our companies is enormous. There will be pressure, top down, to try this stuff. Having a play and getting an understanding of what works and what doesn't work, that's a really good idea. For this talk today, I am going to try and ground my vision of the future in three things, my three anchors that I believe are going to remain true no matter what technology we use or how fast we get. The first one is that we must build something worth building. We must deliver some value with the software that we ship. The second is that speed requires safety. The third one, I hope, is obvious, people matter.

Anchor 1 - Build Something Worth Building

What does it mean to build something worth building? Code is the how, it's not the what. We deliver solutions to problems. The solutions to those problems require software. To deliver that, we need to write code. Code is the how, it's not the what. What we're delivering is a solution to a problem. I simplify the feedback loop in this way. We have users with problems. We usually have a product manager whose job is to really understand those problems. We have a team of developers, usually six to eight in a normal organization, who then devise solutions to those problems and ship those as a software product. The users use it and they go, yes, no, that wasn't quite what I wanted. Or they go, wait, take my money. The feedback loop continues. The users have needs. The product manager filters those needs. The developers deliver solutions to those problems using software.

For a lot of development teams, they don't really speak to their users very much because the product manager does that on their behalf. For a development team, they have a lot of feedback loops. It's usually about how the system is working. It's not a user expressing their pain. I think that's pretty normal for a lot of the teams that I work with, the developers are not engaging with their users. They are not talking to them about their pain. There's a proxy for the user that is the interface, and that is the product manager. What happens in a typical organization is there are lots of user needs. There needs to be a filtering and a prioritization process. I've worked in product. I was always very considerate to my engineering team. I didn't overwhelm them with too many ideas. I was filtering out the ideas. I was deciding what was good and what was bad. I was prioritizing ruthlessly, I would say, prioritizing what they should focus on, because developer productivity requires focus. That was something that I believed to be true. I organized my teams around that. Focus on one thing at a time, no distractions.

What does that look like with agentic coding? What if we can 10x the speed? What if the amount of changes and new features that we can push to our product increases dramatically? That is happening for so many teams at the moment. What I've seen happen is backpressure on product. There isn't enough work in the funnel for some of these teams to keep those engineers who have unlocked this new velocity busy. I said a couple of years ago on a podcast, I was like, "I've got a three-year roadmap. My team are never going to run out of backlog." Actually, I've seen it happen now. I've seen it happen in the real world where dev teams don't have enough well-qualified work to keep them busy because of this velocity that they've unlocked. That is a new problem that I have never experienced in my career. What do you do?

Do you just say yes to everything? Every nice idea that you might smash into your product? No. Normally, that's not a really good product strategy. Enshitification is a thing. That's the word that was coined for when a platform gets too bloated with features. It doesn't do just one thing. It does many things, and maybe it does all of them quite poorly. That doesn't feel like the right answer either. We don't just build every single little thing that we could possibly think of, never qualifying it with our users.

There is a silver lining here, which is that the cost to test your ideas is collapsed. In the past, when we built out our roadmap, we had this confirmation bias. We were like, this is the roadmap. This is what will deliver success for our business. This will help our users. We didn't have the engineering capacity to really go and try and experiment. We were like, we'll just build it. If we build it, they will come. Whereas I think a much more healthy way of delivering product and delighting users is to try something, to build a prototype, to test it with them, to iterate before you commit that as a thing that you want your product to do. Because when it is a thing that you want your product to do, you have to support that for eternity or until you decommission it. What we've seen is the emergence of the vibe coding PM.

Anyone work with a vibe coding PM? Is anyone a vibe coding PM? Give me a wave if you're a vibe coding PM. Because this is a new breed of PM. I quite like it. It's not the product manager who's like, I'm shipping production code. It's the one who's like, I really need to test my ideas before I commit them to the product. How important is that? How powerful is that? That's a practice that I'm seeing emerge that I'm actually really excited about. Let's test our ideas before we commit them to the product. Another thing that you might want to do is you might want to pair a developer with a PM to build those prototypes if they are beyond the realms of the vibe coding. You get this feedback loop between your users, product and development so that everything that goes into the dev team is well-qualified work, well-understood.

You have tested it with your users. You haven't just put it into your product and hoped for the best. That is going to get you a better product at the end of the day. Another pattern that I've seen in the real world is the forward deployed engineer. Who's heard of that one? The forward deployed engineer, the FDE, is an engineer that is embedded in the customer and empowered to solve their problems. This is not necessarily an implementation or a professional services engineer. This is an empowered engineer that's going, yes, let's shorten that feedback loop as much as possible. Like, here's a real-world scenario that we didn't think about that we haven't built into the product, but now we can build it. You end up with a roadmap of long-term features and a very short feedback loop of small tactical changes that are going to delight your customers. I really love this pattern, and I'm excited to see how organizations evolve around this.

Another pattern that I've seen is the product engineer. The product engineer works really well in software companies that serve developers, so dev tooling. Basically, everybody who's outside in the sponsor area should be thinking about product engineers, because a product engineer is an engineer that's empowered to improve the product because they are the user. If they are the user, they know what they need and they know what they don't need. I was talking to the team at incident.io, and they are very keen on hiring product engineers, not just software engineers, product engineers, because they are building a product for themselves, and so they are empowered to make decisions. Another thing that they have is they have direct access to their users. There's no asking for permission to go and ask a user what they think of a new feature. It's a very open channel of communication.

There will be people sat here who are going like, "No, I do not want to be talking to customers. Please, someone do that for me." Then there will be other people here that will be like, "Brilliant, let me at it." I think you will all know which bucket you are most aligned to, and you will know which of your colleagues are. A product engineer, especially if you're building products for people like yourself, is a really powerful pattern, and it really reduces the feedback loop. What we're really talking about is building a valuable product at the end of the day. We want to solve real world problems for real users.

The backpressure that is created by the unlocked velocity of agentic coding has led some teams to challenge what the ratio of product to developers might look like. The two-pizza team, like the six to eight developers and one product manager, that's been standard in our industry for quite a while. I've seen teams now experimenting with two developers to one product manager, almost like a micro team. NBIM are running this experiment at the moment, and I will be really interested to see how this goes. In several of our AI projects, development now moves faster than the business can keep up. The problem isn't development capacity, it's lack of clear specs, fast decisions, and tight feedback loops. They're running this experiment to really compress the team down to the bare minimum, because a product manager can't keep six to eight developers busy. I think that's a really interesting experiment.

Andrew Ng, who is an entrepreneur and a bit of an influencer in the AI space, has proposed that it's actually the opposite ratio, that it should be two product managers to a single developer. Can you imagine that? Because that developer, you assume, has got like a fleet of agents and can orchestrate the development across multiple product lines simultaneously. This is the thing that companies are probably experimenting with. It's absolutely fascinating what a bonkers time to work in software development. It's a real period of change. I'm not here to advocate for any one of these. I'm here to show you what other people are experimenting with, so that when you start seeing this unlocked velocity, you have some ideas and some inspiration of what you might try next. I had a chat with a consultancy, because that's a really interesting space to chat about agentic coding.

They're selling developer time to their clients. How is it changing their world? Interestingly, of all of the companies they work with, so they work across 200 companies. I think they've got about 3,000 developers deployed to those 200 companies. They've only seen two or three so far reduce their team size. The rest are expecting more output. I think if you project that expectation of more output a little further, maybe you start to think about, maybe the domain that a team takes care of has to expand, because there won't be enough well-qualified work in their current domain to keep them busy. I think that will be incredibly fascinating.

I also want to advocate for design and UX. Doing user research and being data-driven and user-focused is going to be a differentiator in a world where writing software is quick and fast. Are you going to delight your users with what you've built? I don't think the craft of user experience design is dead. I think it is more important than ever. The challenge is accelerating it to the speed of software development. The challenge is making sure that we have good design that's thought through, that we make time for user research and usability testing. Things that I've seen organizations do that don't work and that probably are practices that will diminish over time is one thing that I call the ship and switch. The ship and switch is, basically, we have so much on our backlog that the first iteration of a product or a feature becomes the end thing.

We never go back and improve it. We never go and ask our users about it. It's like, ok, one box is ticked and we go on to the next thing. We ship it and we switch context and we go and solve another problem. Actually, the best products and the best product managers know that you have to go and work with your users after you ship it. Is this the right thing? Could it be more valuable to you? How is the usability of it? You just don't ship it and then switch your context. I know so many teams that have been operating in that ship and switch mode for so many years. It's a really difficult habit to unlearn. The other two are obvious, but saying yes to every single customer request and every single idea is not going to land you with a really successful product either. Then, HIPPO, the highest paid person's opinion, prioritization should have gone away a long time ago, yet it still is prevalent in a lot of organizations. Let's just stop doing that, the HIPPO prioritization. Let's really get data-driven and user-focused to make sure that we're building something worth building.

What am I keeping? What am I trashing? What am I trying in this new world of agentic software development? I'm keeping empowered product teams that can make decisions and help their users without asking permission. I'm keeping fast feedback loop with actual real humans, real users. I'm doing usability research. I'm doing UX research because those things are going to differentiate my product from every other vibe-coded platform. I'm trashing the two-pizza team of six to eight developers. I've seen multiple times now that that isn't necessarily the right ratio for success in a lot of businesses. We were talking about it in the pub and we proposed that it's more like a tapas team than a two-pizza team, like some small plates maybe. Yes, the tapas team might become a thing. I'm also trashing my six-month roadmap, not because planning isn't useful, but because actually deciding what you're going to build in six months' time is no longer a valuable exercise.

If you want to build such a thing, let's figure out how we can do it now. How can we parallelize? How can we get it done? If it's important enough, get it done and get it done at the speed that you can get it done at. New things that I want to try. The forward deployed engineer, the engineer that's empowered and sitting alongside a customer to solve their problems. I'm absolutely wanting to reduce that feedback loop to almost zero. Let's go and solve real-world problems. Let's be very customer-focused. Product engineers, absolutely. An engineer who has a strong opinion about what should and shouldn't go in the product is going to be such an asset. You don't want necessarily to be surrounded by teammates who just want to pick tasks off a backlog, because if an engineer can pick a task off a backlog, then, unfortunately, so can an agent in the future world. Definitely want to do prototype before product. I definitely would want to experiment with much smaller team sizes than we are currently used to. Smaller team sizes or an expanded domain of responsibility for teams so that you don't get that backpressure, and so that you don't end up building things that aren't worth building.

Anchor 2 - Speed Requires Safety

I'm going to go on to my second anchor now, and this one is that speed requires safety. On the path to production, we need to validate that the software that we've built does what it's supposed to do and hasn't broken anything that's important. These are basics. The path to production in a lot of organizations can have manual steps. There can be gaps in test coverage, which can make it very scary to ship at an increased velocity, because if you can't ship at that increased velocity, you're going to end up with pile-ups on the path to production. If code generation outstrips your ability to get that through a pipeline and into the hands of your users, then you're going to have a pile-up somewhere, and that's going to create backpressure in other places. Again, what we end up with is we end up with developers able to produce so much more work and the downstream process not being able to keep up.

High confidence requires testing, and I believe high speed requires automation. You need both. You need incredibly fast, good coverage with your automated testing. I think as people start to adapt to this unlocked velocity that we have now, those will be some of the first pain points to reveal themselves in your organization. Have a think about what you can start doing now to start closing those gaps and making the path to production as smooth and automated and fast as possible. I really enjoyed this article, and it's from AWS, about how they managed to get 10x throughput from their development team by adopting agentic coding practices. The summary is that they had to completely rethink testing, deployment, and coordination practices. They couldn't just bolt on the speed of AI to what they were doing at the moment. They had to re-engineer that path to production for velocity.

While we might find ourselves with a dev team who have run out of backlog and feature work from their product manager, I'm like, what can we do with that bit of capacity that we've got? Let's go and attack this problem. Let's go and attack the throughput problem. Let's make sure that we can unlock that velocity when the time comes.

Another thing that caught my eye was about how JPMorgan are using AI agents to drive their continuous component testing. Again, I'm like, this is a happy coincidence, really. I'm here saying like you have to have continuous testing. It has to be fast. Here we are with tools at our disposal that can go and write the tests for you. I thought it was really interesting that they chose to tackle this as a first priority, but it's actually very human-centric. Michele Willis says, virtually no developer likes testing. It's the toil part of the work. Developers want to spend time solving problems and understanding the code. Testing is something that can be tedious. Great. Let's give that to an agent to do. This is the sort of work that humans don't love, so why not give that to the AI agent, and in doing so, improve the automation of the quality gates that you need to unlock the velocity that you want on the path to production.

Another thing that I believe is true is that the cost of day 2, actually running that software in production, will always be greater than day 1. You can build something, but actually, the amount of time you have to maintain it, support it, patch it, improve it, is always going to be greater in the long run. I also believe strongly that, even though they don't ask for it explicitly most of the time, users really care about security and reliability, and right now, these topics are hitting the news, like the mainstream news. Reliability of Amazon made the news, and I think this has been talked about in many other sessions. This is mainstream newsworthy stuff when a website goes down. Claude went down the other week as well. GitHub's reliability figures are particularly bad at the moment as well. Reliability is newsworthy stuff, and it will damage your product reputation, so investing in that is so important.

I think that is fundamental. You cannot have velocity at the expense of reliability, nor can you have velocity at the expense of security, although apparently, if you're a Moltbook user or you've attached your OpenClaw to Moltbook, you don't really care. Good security is often invisible. It's invisible right up until the moment that it's not, so investing in a secure foundation. Your users are never, ever going to give you explicit instructions to that. They're just going to expect it all the time. They expect you to protect their data. With all of this velocity that we want to unlock, we cannot do that at the expense of reliability and security.

We also don't want to disrupt our users too much with new features or half-finished work. Progressive delivery, like feature flags, A/B testing, blue-green deployments, another way that we can actually create the sense of a reliable service while we are churning through a lot of change, more change than ever. We already have practices that help us to reason about this. The SRE book introduced us to the concepts of SLI, service level indicators, user-centric service level indicators. SLOs, what is the promise that we're making to our users in terms of reliability, and error budgets, what is the amount of acceptable failure? There's one tool in particular that I really want to talk to you about, which is the error budget policy. The error budget policy is more important than ever, because the error budget policy tells you what you will do as an organization when you break your promise, what happens when you don't meet your SLO.

What are you going to do differently if the promise you've made of reliability hasn't been achieved? Because I think it can be tempting to think that velocity is the only goal. If your product is unreliable, if your product is insecure, you are going to suffer commercially for that. Someone will come along and they will do a better job and they will serve your customers better than you are. Your error budget policy writes down what the SLO is, and what you are going to do in the event that you break that SLO. Maybe you slow down your releases. Maybe you need to invest in reliability. Maybe you need to invest in resiliency, but it is a written document that says, here is the promise that we're making to our users, and we're not going to break that promise. If you don't have an error budget policy at the moment, now might be the time to write one.

Because one of the unintended consequences of pushing more releases, and more code, and more change to production is potentially a degradation in reliability. Measure it and agree as a team what you're going to do if that should happen. If you should push so much change so quickly that you impact reliability to the point where it is intolerable for your users.

Some of the patterns that I think are going to be even more important and influential than ever are, building internal development platforms, and site reliability engineering, as I just said. A platform engineer provides paved roads. They can help you smooth that path to production. They can provide environments on demand. They can make sure that everything is secure by default, make the secure thing the easy thing to do, and then your development teams don't have to worry about it. SRE teams as well. When Google published the SRE book, and that is years ago now, I thought it was a little bit of an antipattern that you would have site reliability engineering separate in its own organization. I'm now thinking about that again, and I'm thinking that actually an internal consulting function that are experts in building resilient systems could be a massive benefit to a lot of teams who are finding that their reliability suffers due to the increase of velocity of change.

I'm reconsidering the idea. I'm not here to stamp my foot and say like, this is the right thing to do. I'm here to reconsider the idea, that I think that could be a really interesting pattern as teams start to move faster. An internal consulting organization, an SRE organization that could provide expert advice on how to make sure that your reliability does not suffer. I think the SRE team may be about to make a comeback.

The interesting question here is, what's the ratio? What's our team composition? Because what we see at the moment is even if you have an SRE, they're massively outnumbered by the number of developers. If you have a platform team, they are massively outnumbered by the number of developers. If you have a security team, then it gets even worse. There's even smaller number of people there. Actually, I think these functions unlock that developer velocity. We might find that different ratios are needed to be successful. I'm ready to challenge that. I'm ready to say that actually, a platform organization that makes it safe and fast for a developer with a swarm of AI agents to ship a lot of change to production safely and securely, that's huge value for my organization. That's a humongous benefit that's unlocked right there. The ratios that we are currently used to in our organizations are probably going to change.

I think that's a good thing. Another thing that can cause friction on the path to production is tech debt. Again, like I said, go and get your test coverage up to scratch by using agents. Agents are also really good at migrations, getting off old out-of-date platforms, old end-of-life platforms, refactoring codebases. These are today problems, and they need not be tomorrow problems if you can use the speed and the iteration that these agents and these processes can go through to get you off those platforms. That's something else that I'm going to say. If you end up in that situation where your product manager can't get enough work into the team that's well-qualified, it's like, ok, that engineering backlog that you used to have, the tech debt backlog that you had to fight to get any time and attention paid to, let's just go and smash through that. Because now is the moment, especially with these agentic coding platforms being so super cheap. That may not last forever. Now is the moment to go and smash through all of that engineering backlog and that tech debt that you have struggled to get time to prioritize.

My second anchor that speed requires safety, what am I keeping? What am I trashing? What am I trying? I'm keeping the error budget policy, because I care about reliability and because I know that my users care about reliability. I'm absolutely keeping progressive delivery, and if I'm not doing it at the moment, then I'm investing in that. I'm doing gradual releases of new features. I'm testing in production to make sure it's good before it reaches all of my users. I'm absolutely keeping internal platforms that help provide a paved road, a secure road, a path to production. I'm trashing basically anything manual, anything on the path to production that involves manual testing, manual release stages, manual anything. There is no space for that in the new world. It's too slow. That is where the pileup will happen. If you're looking at your path to production and you have manual steps along that, go and tackle those now because otherwise those will be the bottlenecks of the future. Things that I'm trying. I want to use AI to improve my test coverage. I want to use AI to tackle my tech debt backlog. I'm definitely considering bringing back the SRE team as an internal consulting function to help teams improve their reliability.

Anchor 3 - People Matter

Now we're on to anchor three, people matter. It's obvious really, like we want jobs that are fulfilling. We want to go to work and work on interesting problems. Unfortunately, reviewing code that was spat up by an AI agent is not fun. When we think about the future and what we might want the future of our roles to look like, how do we reconcile this? Like reviewing AI-generated code is just simply not fun, especially when there is so much of it created so quickly. I really liked this article by Joseph Ruscio about write-only code. His hypothesis is that actually not all code is going to be ever read by a human. Right now, most code is read by a human, almost all of it. Unless you're a vibe coder, almost all of it will be read by a human. His proposal is that actually that's going to flip.

Most code will run and it will never have been read by a human. If we state that that is our goal, what does our software development life cycle need to look like to make sure that that's not an absolute nightmare? I think that's a really interesting thought experiment. Another thing that I found interesting is the shift left of the peer review. A lot of teams operate in a system where someone writes the code and another person reviews their pull request before it goes into the product. A type of peer review is a type of quality gate. It has been important. What I've seen some teams do is shift that left to before the code is created. Peer review, having a second opinion on what you're building and how you're building it is still super important. What they've done is they've shifted it left to before the code was created.

It's the review of the specs. It's the review of the tests. It's the review of the test strategy. It's the review of the architecture. It's still two minds coming together to go like, is this the right thing? Are we doing it in the right way? It's not reading through lines of code in a pull request. I think that is a really interesting new pattern as well.

Another thing that I think we all know to be true is that when you're holding the pager, supporting a fragile service is not fun at all. There is this minimum viable amount of humans that you need to support a service in production because nobody wants to be holding the pager all day, every day. You can't be on-call all day, every day. When you think about that as a constraint, what is the minimum viable team to support a service? Say you have one person who's on primary on-call and another person's secondary as like a fallback. Then if you were to say like, maybe it's ok to be primary one week out of every four and secondary one week out of every four. That basically means that you're on-call still 50% of the time, which for a lot of people, that still wouldn't be a very sustainable situation.

We really have to think about how we're going to support our services in production before we can even start to consider reducing our team size. A team size of one developer who's holding the pager all the time is absolutely not realistic and not sustainable. Again, this is why I think potentially the SRE team as a separate organization might make a comeback, because the original manifestation of the SRE team at Google was willing to hold the pager for certain services if they were built to the standards and the reliability expectations that the SRE team dictated. They would not support a service until it met their standards and then they would. I think that's potentially another benefit of bringing back the SRE team. As I said, what is the minimum viable human that you need to support a service in production? I had the minimum viable human slide on this stage last year, because I was talking about agents.

I keep coming back to it. It's like, we can't reduce our team size to the extreme where the work becomes completely unsustainable, where we're a single point of failure, where we can't go on holiday. This is no good. What is the minimum viable human to make the work sustainable? I think that's a really important question to ask ourselves.

When we talk about teams and how people collaborate, I think a lot of folks default to the agile ceremonies that we're all used to, daily stand-ups, sprint planning, backlog grooming. I'm not sure all of these work anymore. The first thing that I'm feeling and sensing is that a two-week sprint is too long at the moment, and that most teams are iterating faster and getting through their backlog faster. If you go from that point, if you say, actually, I think I feel that a two-week sprint is far too long, what is the correct iteration cycle? Is it one week? Is it a day? How often do we want to be making decisions like that and who needs to be in the room? How might we experiment with our practices around how we collaborate as teams so that we don't accidentally introduce processes that are just too heavy, that are just far heavier than they need to be?

Because planning and replanning isn't necessary anymore. Another thing that I want to talk about is actually the skills that we need in this profession. If we assume that agents are going to get so good at coding that they're going to write most of the code for us and that actually this idea of write-only code that is never, ever read by a human might become a reality for some teams, what skills do we, as people in software development, need? It was at QCon last year that Sophie Weston gave a talk about her journey to becoming a principal engineer, and she introduced the idea of the broken comb. The broken comb was the idea that actually, rather than being like a T-shaped person or a Pi-shaped person with maybe one or two areas of specialism, that what we need to develop is a broad knowledge across a lot of different domains in software development.

Then we need to be able to go deep in a few. I really like this analogy of the broken comb, because I don't think being a specialist in a single domain will necessarily serve us very well in our profession going forward. I think developing a broader range of skills and being able to work on a different range of problems is going to set us up for success. I'm very keen on the idea of the broken comb. If any of you in here are managers, it's worth thinking about that with your teams when you go back. It's like, how can we develop breadth in our team by giving people different areas of specialism, challenging them to learn something new? As I've talked about in this talk already, there is a need in a lot of organizations for engineers who are willing to work side-by-side with customers and to develop product management skills and product thinking skills.

That is one area where you might ask your team to go deeper. There's also going to be a need for more specialism in reliability for release engineering. These are all areas where someone who has previously not just been a developer, but has been a developer who has been delivering against a backlog of coding tasks might want to start to diversify. That is something that I want to encourage everybody to think about for themselves and for the people around them, that diversifying of your skills to make you a bit of a lethal weapon when it comes to agentic coding.

Around my third anchor, what am I keeping? What am I trashing? What am I trying? I'm absolutely keeping sustainable on-call practices. I'm not going to have a single human be on-call all of the time, or even 50% of the time. I'm going to figure out a way to make a team successful without having to do that. I've mentioned it before, I will mention it again. An error budget policy makes it unambiguous what you're going to do if your service availability is suffering. I'm going to do career planning because we were doing that already, weren't we? I'm going to do career planning, but for today's reality, not for yesterday's reality. I'm going to rethink all of my career frameworks. I'm going to encourage everybody to become a broken comb, a generalist with some deep specialisms. I'm absolutely trashing heavy planning processes. They don't add much value.

Backlog shuffling, backlog grooming, all of that stuff. You can potentially say yes to everything and it becomes more important what is on that backlog than what you're going to say yes to or say no to. Two-week sprints, they feel too long to me right now. They feel too long. I'm trashing that as a standard. I'm also trashing mandatory code review. Peer review is important. Mandatory code review on every single line of code I don't believe is a sustainable practice for the humans in software development going forwards. That might be a controversial statement, but I don't think it's sustainable and it's not joyful. No one's going to want to do that job. Things that I'm trying, shifting left peer review to before the code is written. Talking through how you're thinking about your solution, how you're designing it. Talking through your specification if you're doing spec-driven development.

Definitely adopting spec-driven development. Deciding how you want your system to behave gives your agent a much better chance of success. I'm going to encourage everybody to become a broken comb. I probably won't shut up about that because I think all of us want to be set up for careers that are successful in this new age where we're working with these agentic coding platforms. What does it take for us as humans to succeed? We need more areas of specialism. We need to diversify our skills, and we need to start that now.

Summary

I'm going to go back to anchor one, which is, build something worth building. I'm going to tell my story of a product that I've built recently, because it's easier than ever to build a product. This is the story of BIMP. BIMP stands for Base Image Management Platform, because it's a platform that manages your container-based images for you, basically. It's a problem that I understand well because of my background in platform engineering and security. I was like, I can't see anybody out there who's trying to build a single solution for base image management. We did. Two people spent two weeks trying to build a platform to solve a user need. Yes, we ran out of work very quickly. We gave ourselves two weeks and we sat down on the morning of the first day. We said, what's our goal for today? We smashed that goal by lunchtime.

We were like, we're going to have another planning meeting at lunchtime to figure out what we're going to do in the evening. That continued. That continued until I unlearned and we unlearned the practice of ruthless prioritization, and we got more demanding and higher expectations of ourselves. It's like instead of trying to build one or two features in a day, we were trying to scope four or five in quite an intensive planning meeting in the morning, but then we would go and we would smash through it. Even then we would smash through it before the end of the day. This is not like a two-week sprint where we were pulling all-nighters. It was like, what's next to build? What's next to build? What's a nice to have? What will our users want? I would never ever in my previous job as a product manager have gone like, yes, in week one, we're going to do some management dashboards.

Never. That's what we were doing because functionally the platform was operating, and we wanted that extra level of visibility. We absolutely felt all of the pain of that backpressure in a very short space of time. I managed to get a product shipped within two weeks. Refactoring that codebase took hours with Cursor in agentic mode. The documentation was automated through AI, and it took minutes to build the documentation. You can scan the QR code. You can go and have a look at our homepage. What I want to go and say is that building is so much fun right now. If you go and have a look at our homepage, you'll see that there's a 3D blimp that soars across it. A 3D blimp should never, ever be in your MVP but it is in ours because building things is fun. We didn't stop building and we've kept going.

This is a bit of an exclusive for you all because we haven't even launched the product yet. By the time this is streamed online, we will have. The experience of building a product at the speed of AI has been incredibly insightful. I experienced all of these problems and then some, but it did make me way more ambitious about what I might build in future. Another thing that I want to say is like, we are all on a journey. Not everyone is at level 8, orchestrating a fleet of autonomous agents. Many of us will be prompting our way to solutions. Even that is going to unlock velocity. I do think we are all on a journey. I do think this is not necessarily a destination. I think there is a correct methodology for the problem space and the task that you're actually working on at the time.

Final Thought on The Three Anchors

I will leave you with this thought. Building has never been so easy or so much fun. We are going through an awful lot of change. Today I didn't want to give you a strong opinion about what I think the future will look like. Hopefully I've given you some ideas of some things that you might try when you start to unlock this velocity and when that starts to create new problems for your teams. More than anything, I want us all to feel confident about the roles we have in the future. I want us to feel like that we have a place in this world of agents. I want us to feel ready to embrace this change.

See more presentations with transcripts

Building Resilient Platforms: Insights from 20+ Years in Mission-Critical Infrastructure

Matthew Liste — Mon, 05 Oct 2026 16:00:00 GMT

Summary

Matthew Liste explains the 12 core principles of building scalable, reliable platforms for enterprise tech teams. From avoiding technical debt and managing deprecation to building a strong engineering culture, Liste discusses how to design infrastructure platforms that reduce complexity, streamline developer workflows, and deliver long-term value for senior architects and engineering leaders.

Bio

Matthew Liste is the Executive Vice President and Global Head of Infrastructure for American Express. He was previously at JP Morgan Chase, where he most recently served as Head of Platform Services for Global Technology Infrastructure. Prior to joining JP Morgan Chase, Matthew spent nearly a decade at Goldman Sachs, and was named Managing Director and Technology Fellow in 2011.

About the conference

Software is changing the world. QCon San Francisco empowers software development by facilitating the spread of knowledge and innovation in the developer community. A practitioner-driven conference, QCon is designed for technical team leads, architects, engineering directors, and project managers who influence innovation in their teams.

Transcript

Matthew Liste: I'm Matthew. I'm at American Express where I run infrastructure. I've been there a bit over two years. I was at JPMorganChase for nine years before that, similar roles, platform engineering roles. Then I started my career in financial services at Goldman Sachs where I also did platform-y infrastructure roles. Came out of telecom and networks and started my career in oil and gas, so offshore embedded systems doing seismic data acquisition. I had a pretty brokered career. I went in up in financial services as a technologist, mainly because my wife is from New York. She wanted to move to New York. That's where the jobs were 20 years ago. There was no tech in New York back then, so you either worked at Bell Labs or you'd work at a bank if you were a hardcore engineer. That's where I ended up. It's been really good to me because guess what banks do? They do a lot of mission critical software, which means the infrastructure that they run on has to work. This talk is really about building infrastructure and platforms for that. This has come from an infrastructure background, the platform principles. I believe these principles really apply to anyone building platforms for reliability. That's really my experience.

What is a Platform?

Let's start with the definition of a platform, because I think it's useful. This is a platform that I'm standing on, for example. Dictionary definition, a raised platform that can hold people or things. This is one. If you think about the technical definition of a platform in context of what we're going to talk about is really a set of integrated technologies that form a foundation to build applications on top of. What are some of the best examples of that today? Cloud providers, AWS, Azure, GCP, all build these foundational platforms, really infrastructure platforms and different things that sew together that allow you as a developer to write a software on top of. This talk is really about the principles of building that. How many of you are platform builders? We're all consumers. Everyone consumes platforms of some shape or form, myself included. Everyone consumes platforms. If you're not a builder of platforms, and I hope this talk will give you an appreciation for the complexity and everything that goes on under the surface of the iceberg of what a platform is.

Why you should appreciate what others do, and the fact that you don't want to do it yourself. You want to leverage other people. Let me use an analogy for that. Think about a dirty analogy, but it's really a fair one, plumbing. All of the work that goes into making sure that everything that we do every day gets swept away, and that we get water back, and it just works. That's a platform. It's a sewage platform, a water delivery platform, an underlying infrastructure platform that's completely opaque and transparent to most people, except for when it breaks. When it works, you don't even think about it. You don't appreciate it. You just take it totally for granted. I think that's a great analogy for a platform provider. When you do your job well, no one knows you're there. It just works, and it's intuitive, and you take it completely for granted as a consumer. That's the focus.

The Principles of Infrastructure Platforms

I pooled this into the 12 principles that I'm going to talk through this talk. They're in no particular order. There's no reason for number one being number one and 12 being number 12. I've tried to group them based upon making this presentation flow. You shouldn't interpret these as more or less important. These are the principles that I have used and used to build platforms in a repeatable way with my teams. Just to give you some context, the platforms that we build, and we know we're at the cloud scale. I built platforms at Goldman, JPMorgan, now at Amex. Our consumers are internal developers. To give you context, at American Express, that's about 20,000 people that consume the things we build. At JPMorganChase, it was more like 60,000. That's our target audience that we build for. Of course, your cloud provider, you're talking about millions of customers. It's a different scale, but the same principles apply.

1. Deliver an Intuitive Experience

Let's get started talking about what these principles are. The first principle, number one, is about the platforms being intuitive and delivering an intuitive experience. Intuitive also equates to obvious, also equates to transparent just works. Think about platforms that are great and that you could think about in your life. They're a common platform. They're integrated. They're intuitive. They hide complexity. Meaning whatever goes on like sewage or delivering power to this room and this stage is incredibly complex. That complex is totally hidden from us. They appear magical. Every time, and I'm sure all of you can remember back in your life when you saw a new platform, and it was completely magical for the first few days. Then you started taking it for granted. Many examples. Mobile phone, internet, first computer for those that are old enough to not having had computers in your life and then had one.

Myself, old enough to be in that bucket. Or GenAI now. My kids who are two teenagers, they have completely embedded GenAI into everything they do, and couldn't imagine life without it. When I first saw it, I've had friends who have been in AI space and researchers for a very long time. The fact is a change that went from impossible to magic to take it for granted. Those cycles go very quickly. The point about great platforms is that they are magical, because they hide that complexity away. I love this quote from Arthur C. Clarke, "Any sufficiently advanced technology is indistinguishable from magic." It's true. You think about that yourself, you can't tell the difference. It just seems incredible that this complexity just works. The beauty of that is that complexity, another way to think about it is a platform provider is the one that makes sure they manage that complexity so you can manage other complexity.

None of us have the cognitive depth, although there are some incredible renaissance people who can go deep in very many fields. Most people end up specializing. The reason we can specialize in what we do is because someone else has specialized in what they do and built platforms for us. That's the beauty of platform and the layers around it. Steve Jobs also had a very early quote on this, "Simplicity is the ultimate sophistication. It takes a lot of hard work to make something unique, simple, to truly understand the underlying challenges and come up with elegant solutions." Again, I think that holds true for any great platforms you see. The complexity is hidden from you. You can enjoy the systems without having to worry about that yourself.

2. Build Common and Interchangeable Components

The second principle is around platforms, and if we think about the definition I spoke about, are a set of components that allow people to build on top of, which means it's not one thing. It's not a singular thing a platform delivers. Think about like cloud providers, deliver hundreds of different services or thousands for that matter. They all work in a consistent way. They snap together, like Lego. I like Lego as a metaphor. Anyone here, I'm sure, has played with Lego, have kids who played with Lego, have used Lego. Think about the infinite number of shapes you can build with Lego: a dinosaur, a car, the Eiffel Tower. You're dealing with a very finite set of building blocks that have a common API, how they lock together. Everything is built in the same way so they can lock together in certain shapes. We can build anything out of that.

It's really important that platforms have this sense of commonality and set of what I think was freedom from choice. For example, make sure every platform component uses the same set of observability basics or foundations. Make sure they use common identity semantics. Make sure that they use common namespace. Again, think of the cloud providers. You put yourself into one of the clouds. You can observe your entire components from that cloud from a common set of observability components. It's not all different. It allows you to observe your application and the underlying components to it. That platform aspect of interchangeable and interlocking components is super important for platforms to be successful. Otherwise, they're just different parts that you offer, but they don't work as a platform. They work as just a set of components. That aspect is very important when you think about platform building. To let alone, I was just thinking about with cloud providers also, common bill. You get billed for all of this together. The financial aspects of a platform is super important when it comes to that.

3. Use the 3 S's

Thirdly, I like to think of these as the three S's. It works very well with the stool metaphor too, which also has a platform to sit on. Think about a three-legged chair, only works if all three legs work. In my field, building for banking systems, credit card systems, and so on, you have to have platforms that are stable, they have to be secure, and they have to be scalable. Let's start with stability, kind of goes without saying. If it fails frequently, if you're not adhering to a certain high degree of nines to your consumers, they cannot rely upon your platforms to be there for them, and they won't use them. How many nines, of course, that's a commercial decision. Do I go for three nines, four nines, five nines, six nines? Depends on the platform. Also, at American Express, our credit card authorization systems, they run at six nines, seven nines.

They don't tolerate downtime. There are other systems that tolerate more downtime, like a mobile app. We vary based upon, of course, the commercial value of platforms. You can imagine that's incredibly important. Secondly, security, of course. People entrust us with their money. You better make sure that you manage that from a security perspective. Then scalability is really important. I've often built platforms that have worked until they didn't. They worked, and then users started using them, and we had runaway consumption. We found bottlenecks in the system, and they just started failing. I've learned to anticipate scalability. If you build products that people love, people will run at them, which means your volumes will increase. If you haven't anticipated that and built that in, you will then impact stability because you can't deal with scalability. It's super important that you think about those. These three are non-negotiable, again, for mission-critical software.

Now, to be clear, early days of building a platform, you will compromise on stability. You'll compromise on scalability. As you start getting clients who depend upon you, you have to live up to this. You have to be formal about that. Think about putting in place SLOs with your customers and managing closely to that. It has to be true not just day one, but day n. Consistently, you have to deliver to that promise with your customers. I did speak about cost because my view is that cost will fluctuate up and down. If you decide to get cheap on any one of these, you'll ultimately fail for this kind of software. You can dial up and down how much you spend based on how much features you put in. That's really the equation you do. Once you've decided to build a feature or build a product that you use for the software, you have to then be willing to spend the money it takes to do so properly with these three S's.

4. Be Evergreen

This ties into the next point, which is principle four, be evergreen. When you build platforms, your customers take for granted that it's going to continue to work. Unlike, let's say, a sewage system, where you put in the pipes, and hopefully at least for a few years, you don't need to think about those pipes again. They're static. When it comes to software, it's ever-changing. You have to be upgrading and managing the platforms and the upgrade cycle on a consistent and ongoing basis. If I think about one of the things that's been the hardest to do over time has been this. Because it is easy to compromise on this and say, I'm going to wait with that upgrade. I don't have time for it. I'm going to put up the feature functions. I am going to wait because I got other more important things to do. Before you know it, you get caught behind.

Now your maybe major version's behind and you can no longer upgrade your platform easily without major client impact. Because the other principle of this is, if you don't stay on top of this, you will impact your clients more. Ideally, you do this and your clients don't know you're upgrading. We all have hundreds, if not sometimes thousands of different changes ongoing in the environment at any given time, so we're constantly upgrading the environment. The client should ideally not know. Now, it depends on the software, depends on the platform, depends on if you have horizontal scaling and so on. Broadly speaking, the goal should be to do this without client impact. Now, where you do have client impact, client-side libraries, in a certain world where they have to change their API spec, SDK, and so on, then you have to have a contract with the client saying, I will only support n and n-1.

My software cycles, my platform cycles every, let's say, three months, I will support current and last version. I will reserve the right to have you upgrade your integration point at some point, because it's the only way I can stay current. The cloud providers do a really good job of this, of explaining this and being explicit about it. I think it's super important to do that, because otherwise you'll disappoint your clients. Think about it, when you upgrade, because you have to make it current. It is not as if the clients have any value driven typically from upgrading themselves. Meaning, they don't have incentive to do this, so you have to make it explicit that you need them to do it to maintain the platform at scale, and have that dialogue, and have that formal term set up front. I like to also use a moniker with the teams, something I call 0114, about managing this.

I'll explain what that stands for. When you manage a platform, how you should think about maintaining the currency of it. You should have zero people involved in maintaining currency, so it should be fully automated. You should be able to upgrade every single widget, so that you're maintaining thousands, hundreds of thousands, millions of widgets. You should be able to upgrade the entire fleet in less than a day, so that's a 1. You should be able to cycle at least every 14 days. Ideally more often, but at least every 14 days. If you set that parameter and say that's what good looks like, you have to be able to work towards that, it helps a team understand what they need to build and all the scaffolding that they need to put in place to maintain platforms to keep them evergreen. It's really important to set that expectation up front.

5. Avoid Undifferentiated Heavy Lifting

The fifth one here is to avoid undifferentiated heavy lifting. There are way too many times where engineers, myself included, want to chase a shiny thing. They want to go build something because it's attractive, because it's interesting, because it's intellectually stimulating, but not because it adds good value. I think what's very important is to be disciplined around that question of, is this necessary? Why are you doing this? What benefit does it bring? Has someone else done it already? Can you iterate on this other thing that someone else built? Rather than, go off and do it. Maybe some of you work for companies that have infinite resource and depths of pockets. I never worked for such a place. There's always been too much work, too little time. Applying this principle is really important to focus on. Using an example of what I have done. Database is a good example.

All the places I worked have database as a service, Postgres, for example. We wouldn't rewrite a new Postgres engine, perfectly great open-source Postgres engines out there. What we wrote was a control plane to do daily backup because that was required from the regulators. That was the only thing we wrote. How do you integrate the fact that every time someone deploys a Postgres database, it does a daily snapshot of the database to an object store so we have that snapshot and we can document it. Only what was necessary and no more, because the Postgres engine itself, we're grateful we needed just that feature. I think, again, it's really important to focus on the value to the business, the stakeholders, the clients, not in what is interesting from an engineering perspective.

6. Be Opinionated

Principle six, which ties into others, is be opinionated. It ties into the prior one. Back to the finite resources. We don't have enough time ever to do everything. Inherent in this is a couple of principles. You need to listen to your clients intently. I think we do a really poor job of this in enterprise IT. Those that work for customer product companies do a way better job because you have to, because if you don't do this well, customers don't pay your bills. They walk away. You're intuitively trained to this. Those of us in enterprise IT are not as intuitively trained because our customers are captive and they're stuck with us, like it or not. We don't listen as well as we should. That's super important. Listen to your clients and then deliver what has the most value. Deliver to the 80%, 70%, not the 100%.

It's impossible. The other part of this is to retire technical debt. I think this is where we tend to fall over the most is that boat anchor of version after version after version after feature after feature. We want to innovate and do the new things. If we don't shut old things off, we fail. Because again, finite money, finite ability to do things. If you don't shut anything off, you lose your ability to do new things. That retirement of technical debt is often not popular. You go to a client and say, I'm not going to do that thing anymore. The thing that you love, we're going to stop. Because no one else uses it anymore. Very hard conversation, but super important. Then the final part of this is, you're the experts at your platform, your clients aren't. That might sound counterintuitive to listen to your clients.

Yes, you listen to all of them, but, ultimately, you have to make the decision as to what you build on behalf of what gives the value to most clients, not all clients. That means that you have to be willing to say, this is what we're doing. You have to have an opinion. You have to say no. You have to shut things off. If you do all these things well, and it takes time, it takes backbone, it takes fortitude. It's probably one of the most difficult things to do for platform managers to manage this. Because everyone's desire is to help clients and do the right thing. This can be very difficult because you have to say no quite a bit to get this right.

7. Be Long Term Greedy

Number seven, which is, be long term greedy. Building platforms is a long-term commitment. Meaning, you were successful. Clients use what you built. They depend upon you. You can't take the platform away. You can't yank it away from them, because they depend upon it. Just like someone yanked this platform away from me, I'd fall hard and would hurt myself. Your clients get hurt when platforms get taken away, easily. You have to think about, these are long-term commitments you make, multi-year. They don't last weeks. They don't last months. They last years. Every platform I built has been years, even when I want to get rid of it after year one, it took me years to get rid of it. It's really important to be judicious about, should I really do this? Do I really want to put the feature teams around it? The maintenance, the cost, the long-term application driving this?

Or, maybe not. I like to use the analogy of puppies and dogs. Any of you that have kids, I'm sure most of you have had this conversation at one point with your kids around, "I'd love to have a puppy. My friend had a puppy. He's so cute. I really want a puppy." Then you remind your kid that this puppy, assuming that we bring this puppy in, will be with us for 10 years, 12 years. They're going to walk it. Are you going to feed it? Then they say, "Of course I will. I love the puppy." None of the people I know that adopted puppies, especially those during COVID with kids are having the kids still walk the dog. The parents are walking the dog. Of course, they are. The child doesn't understand that, but the parent does. This is no different. Like you have to be the parent and say, am I really ready to bring this dog in, this puppy in?

Because that's what is going to happen, you're going to have a dog in the house for 10 years, 15 years, I'm ok with that. If I already have five dogs, am I ready for another five? Maybe not. That's back to the technical debt from earlier, but this is incredibly important to think about. This comes with time and wisdom because I have built platform components that I really regretted after. Say, "I wish I really didn't do that," but I'm stuck with it. I'm stuck with maintaining it. I'm stuck with the clients on it. It takes up time. It takes up resource. It takes up cognitive bandwidth I'd rather use on other things, and I can't. I regret those decisions. Now I really think hard about if we should do these before we go off and do it because we know that it sticks with us for the long term.

8. Fail Quick, Fail Often

Number eight, which, again, might sound a bit counterintuitive to the prior one, but this is about, once you decided to build something, how you build. This is no different than with software. You should fail quick and you should fail often, because building platforms, you iterate upon them, and they're living objects, organic. You have to iterate your way there rather than thinking that you know beforehand. If you don't do that, you will make some big mistakes as well because you will wait too long to adopt something and it will take you in the wrong path. I'm going to use an example of what we did at one of the places I worked. In the mid-2000s, we got enamored with this thing called container constructs in Linux. We said that would be really powerful, especially in financial services, we could be able to partition customers on the same machine properly with namespaces and so on.

That was pretty attractive to us. We said, let's go do that. We ended up using the primitives, but without orchestration. There was no orchestration out there in the industry, so we built it ourselves. Then about a year later, we were actually out here in San Francisco, we visited Twitter, and they showed us Mesosphere. That's really cool. We can use their orchestration system. Let's go use that. We adopted Mesosphere. Then a year later, we spoke to Google, which I'd spoke to Google already, we knew about Borg and about Kubernetes. Kubernetes, when we spoke to them initially was a science experiment. Then very quickly that became mainstream. Then we shifted then from Mesosphere to Kubernetes. Those learnings allowed us then to get really good at container orchestration very early. If we had waited for Kubernetes, we would have been two, three years later with adopting containers than we were.

That iteration was fine. Then it maybe sounds counterintuitive because it's about building platforms and migrating, but we maintain our client fidelity throughout. Our clients are using containers. We help them migrate from our own orchestration. Their primitives was containers. It could actually be migrated relatively straightforward from our proprietary one to Mesosphere to Kubernetes. You have to think about then when you have to do these migrations, how you own that migration on behalf of your clients and don't leave them saddled with that because then it becomes really hard to move forward. If you think about this mindset of, I'm going to continue fail forward. Failure is not a bad word. Failure is a learning. I'm a big believer in trying things on a consistent basis. Take the lessons from it and continue to iterate your way forward.

9. Share Responsibility

Principle nine, which is on shared responsibility. As a platform owner, and I spoke about this a bit before, you have to be very clear on what you do and what you don't do, what you do well, what you won't do at all. As much as possible be formal about it, because the more formal you are, the less you're going to disappoint your clients. I'm talking about formality that is easy to understand. I'm not talking about 100-page EULAs that no one reads that is there written by lawyers to save your legal ass. These are well-understood contracts that you have with your customers. They could be contracts in software like APIs. They could be contracts in process. They could be contracts in SLOs, SLAs. Having well-defined boundaries that says, this is what I do. This is what you need to do. These are my SLOs. This is what you can expect from us.

Amazon do a really good job of this. If you go to this or just Google, or ChatGPT, whatever you use nowadays to do your search, but look at shared responsibility model, they have a really good definition of what they do and what they expect the customers to do. This has evolved over time, but that formality has been very helpful when you build on top of it because you understand what happens. For anyone who dealt with this outage, if you had read that shared responsibility model, you had followed the Well-Architected Framework, which is a different framework they have, to build resilient, that Amazon issue in NA east-1 would have been non-impactful. Very few of our banks were impacted. Why? Because they had built multi-region, they had built around it, they had understood what it took to build this, and were relatively non-impacted. It's important then to define that so customers understand what they're getting themselves into.

The other aspect is also, as a platform owner, as I said, you only have finite resource, finite time, finite ability. You can only vend ice cubes, not snowflakes. You cannot build artisanal products for every single customer. You have to think about building like Lego shapes, ice cubes, whatever analogy you want to use, but you can only build fixed shapes. Again, you have to make it clear to the customer, here are the shapes I can give you, these other shapes that you want, I cannot give you. You have to know that as you consume this. Make customer choices. Maybe I'll use this platform, it has the shapes I need, or maybe I won't, it doesn't have the shapes I need. That should be an explicit and eyes wide open decision.

10. Abstract, Don't Obfuscate

Principle 10, which is about abstract, don't obfuscate. When you build platforms, you can have different levels that customers interact with. They can interact with an API, SDK. They can interact with Infrastructure as Code, like Terraform. They can interact with a UI. Different customers need different interaction points, and different use cases require different complexity. It's not your job as a platform owner to determine which ones your customers should use. You should meet them where they are. At least at a big place like us, they're everywhere. They're all across that spectrum. A couple of things out of that. First of all, you have to have different abstraction layers. As I mentioned, for example, in our case, we tend to have, again, APIs. We have Infrastructure as Code, typically Terraform. We have UIs that can spin Terraform or can call the APIs directly. What we make sure of is that we also give customers insights into what happens.

We don't obfuscate it away. We try and expose the Terraform, the API calls, the environment. For two reasons, A, if something breaks, you need to know what happened under the hood, because if you don't know, you have no idea. If you hit a UI and try to spin up something, it didn't happen. There's no good telemetry, no observability, no underlying understanding what happened, that's not a great place to be as a client. Secondly, when you want to try and do something subtly different than the platform did. Let's say we vend a certain database pattern, we generate Terraform for it. You want to do something slightly different, now you can take our Terraform and change what you need yourself because we didn't obfuscate that away from you. It also allows customers to have a lot richer interaction with the platform than if you obfuscated it away.

You can imagine the extreme version of this is you could write machine code. You could write assembly. You could write Java. You could have Copilot generate your Java, or you could vibe code and have no clue what happens. Ultimately, it's machine code that runs at the bottom of this. These are all different levels of abstraction across the stack. I started my career writing machine code, which was awful. I never want to do that again, with a hex editor. Then I went to assembly, which was slightly less awful. These layers of abstraction are actually valuable, because I never want to do those things again. I'm very happy that I had a higher order of abstraction that I could start using, and in my case was C, not that high, but at least better than those. There is real value in this.

11. Stand on the Shoulders of Giants

Principle 11, which is about standing on the shoulders of giants. Really the metaphor for giants here is open source and open standards. There is no way that the cloud providers, myself as a platform builder, could have done what I've done, or they could do what they did without having open source, and all the open-source components available to build on top of. Because now you can focus on integrations, you can focus on your value add, but you don't need to focus on building every single bit across the stack. You get enormous mind share. You get great engineering scale, of course. You can stay current because there's consistently innovation happening. Also, it can be portable. Think about every cloud. Every single cloud provider offers a Postgres variant. They offer a Kubernetes variant. They offer a Kafka variant, and so on, which means you can port your applications because you're using the same underlying semantics.

The platforms are subtly different, but they're relatively consistent in some of these components. Hugely valuable. The quote from Isaac Newton, this is not a new thought, like science builds upon science and innovation upon innovation. "If I have seen further, it is by standing on the shoulders of giants." This was all about how his innovations, he didn't take credit for it. He said, the reason I was able to do what I did was because of the people before me that told me what they did, and so on. This principle is incredibly important. We started using Unix for a long time, but we started using Linux heavily in financial services about 20 years ago. Since then, I would say the vast majority of our critical systems are today built on top of open source. I'm sure the same holds true for most of you. It has really allowed us as a computing industry to innovate at a scale we would completely be unable to without this innovation. There's nothing wrong with closed source. I'm sure a lot of you write software that you sell. There's nothing wrong with that. It's that plus open source will have gotten us to where we are. I would say that generally speaking, platform building, open source is particularly important, because it allows you then to assemble upon those components.

12. Build Culture, the Rest Takes Care of Itself

Then the final principle, which is my favorite one, I didn't say any one was more important or less important, but this is my favorite one, which is about culture. If you build culture, the rest takes care of itself. Remember when I spoke about platforms? Platforms are many components, many teams involved in building those components, which means that if you're going to do so consistently, you have to have a culture that assembles those teams together in a way that allows them to build a platform that is assembled with all those components. Great culture builds great teams, and great teams build great products. If you get the culture wrong, you might get lucky occasionally and build a great product, but you won't get lucky consistently. The focus on culture is incredibly important. Where I spend most of my time is on culture. There are a couple of things that you should think about when you think about building culture.

First of all is empowerment. Empower your teams, your feature teams, your product teams, to make as many decisions as they can themselves. Give them freedom from choice. For example, you'll use this observability stack, you'll use OTel. You don't get to choose. Don't worry about that. That's choices made for you. You'll use this identity SDK. You're going to use it. You don't get to innovate there. What you need to innovate is in, if you're building a database, build the best database out there. Put all your innovation in there. There you have autonomy to make your decisions. Secondly, having teams that have diversity of thought. Every time I built monochromatic teams that all think and come from the same background, we've had worse outcomes than where we had teams with people of many different backgrounds. Because they challenge each other more naturally. They think about things in a different way.

We get better outcome. It might be more dynamic, to use that word kindly, more noisy, because you've got people that argue a bit more and that have quite different positions. It tends to build better outcomes over time. Managing that is really important. Then, thirdly, no different than a sports team, you have to be willing to manage the players in the team and know which players you need. Managing the dynamics of who's in the team and being intentional with it is important. You cannot be hands-off for that. Yes, the culture is really important, but teams don't fully self-select. Because if they do, then mostly you get teams where everyone are friends and they all go work together. You don't end up with diversity of thought. You don't end up with the right balance. I think it's very important to think about how you manage the teams to do this as well to get the best possible outcomes.

Key Insights

To bring this full circle, I spoke about platforms today. They are pervasive in our lives. I'm standing on one right now. We consume them all day long in our day-to-day life. We build our software and our platforms on top of other platforms. Think of this as a layer cake of platforms. Like I build my platforms on top of open source. They're built on top of other platforms and so on. It's also a circle layer cake. I probably should have illustrated this as some big space circle with platforms all on top of each other. You get the point. Everyone does this. We take the platform beneath us for granted. We don't want to think about it. It just works. If we do our job well, then the people that use us have the same experience. These principles are not meant to be exhaustive. These are the principles that I have derived from my work. I hope some of these resonate with you, and they cue some of this.

Questions and Answers

Participant 1: Often, platforms start with someone building for their own particular use case, and then it starts to expand to more and more people. There's a decision point, I feel sometimes like, should we invest in making this a platform? Any thoughts about how does that transition happen? How do you prioritize?

Matthew Liste: Most of the time platforms, to your point, ideas start in teams that build it for themselves. For example, a few years ago, at the place I worked, there were probably seven different teams doing Kafka in their own way, assembling it together. They'd added different stuff to Kafka to make it enterprise ready, ready for basic financial services. Somebody looked once at that, it's not particularly commercial having all these teams do it. None of them really want to do it for anyone else, want to do it for themselves. We decided to centralize that and create a common managed Kafka service for the enterprise. It often starts like that, where people have built internal software that becomes a platform. It depends on the culture of the company, is the best way to put it. If there's a pot of money to fund the people to build platforms for others, it's a lot easier.

When there is no money and everyone has their own budget, and it's constrained by I don't have money to support you, I only have money to support me, it becomes a lot harder. What we try and do is make that a bit flexible, because we want the right people to build. If you built a great platform, and you are willing to support the company to give you money, that's the best outcome. We at least try and find funds to let the team that wants to do it, and has a bound to do it without being too parochial about it. Often politics get in the way of that.

Participant 2: I have a two-part question. One, how do you balance simplicity and speed? You emphasized simplifying the architecture, simplifying things. Oftentimes the tradeoff, I can keep going, or I can invest in simplicity. The second part of it, you mentioned creating these paved roads, these central organizations that are funded that can create this meaningful way of doing Kafka right, for example. How do you champion the adoption of these initiatives, and how do you make sure that people are using the arc that you have built?

Matthew Liste: First of all, complexity versus speed. I think it's one of those it depends question, because sometimes you have to optimize for speed, because you're trying to get to market, and you're ok managing complexity after the fact. Typically, we do that. We opt for speed first, and then we have to slow down and remove complexity. Because as we're going quickly, we're leaving technical debt in our wake. It's everywhere. It builds up. If you don't then take a step back and start stripping it away, you end up with this behemoth that is unmanageable. That happens every time you build a platform, it starts with speed. You're racing to get clients. You're trying this out. You're putting it out into the marketplace. In our case, a virtual marketplace internally, but still seeing it get adoption. Then you have to take a step back at some point and say, now that we have adoption at scale, again, think about these three as, I have to scale this platform now.

I have a lot of clients on it. I now have to start stripping away complexity. That's a hard place to be too, because I have to tell people, I'm going to slow down feature velocity, because I'm going to start focusing on removing technical debt. I've had to do that every time. No one likes it. It's a real important thing to do, because otherwise I cannot keep running. It's like going through the water, I'm just building more barnacles on my boat. At some point, that boat can no longer move, because it's so encrusted. You become encrusted with technical debt. Certainly, on that one as well just the central teams will decentralize this. You can imagine any bank or place I work, I have a lot of highly opinionated development teams who think they know better how to do the job that I'm doing than I do.

They will tell me, why should I use your crap? I can do better, faster, cheaper. There's a bit of like carrot and stick. The carrot is, if you use ours, it will be less work for you. You can spend your time innovating where it's really important. You have to do that. Typically, over time, people come around and say, I no longer want to maintain my own version of Postgres. I don't want to write my own Linux distro. I don't want to write my own database. We've had that. We've had people even write their own compilers. They've done everything. At some point, it comes around to writing my own compiler might not be the best way to spend my money building better trading algorithms, and so on. At some point, people come to that. The other thing, the stick is, in a bank, there's such a thing as regulators, and satisfying them.

At some point, you have to say, you have no choice here. This thing you're doing is dangerous, is going to get us into trouble. Therefore, there's certain rules that you cannot violate. We often tell people, listen, if you want to build a highly compliant system, knock yourself out. Most of the time, I don't want to deal with that. It's just too much work, too much headache. You do that for me. I think that's a bit of the balance we try and achieve.

Participant 3: This is another balance question, because it seems like there's a lot of this. In terms of when you've talked about deprecating things, and especially getting rid of technical debt, it makes me wonder how you do that with clients. Because in my experience, the clients who get the most invested, if you really want serious customers who are using your platform heavily, then they invest in it to such a degree that when you then need to deprecate something or change something, it takes multiple years for them to get off it. Do you have any strategies for helping stay agile in your development, or any strategies for how you manage clients and help them to move or encourage them to move?

Matthew Liste: It's probably the biggest challenge, is, how do you help clients get off the thing that you built when you have to do a forklift upgrade? First of all is, ideally, you avoid ever having to do that to begin with. Because if you do that, it's really an antipattern. Think hard about, if ever I have to make my client shut the thing off and migrate, I failed, because I should have never had to. Again, realistically, it's happened. I've done it. It does happen. Then you have to just think about, how do you make it as easy as possible to reduce that bar? How do you send as much as possible? It's super hard, because again, it's not in their interest. Unless there's a feature they need, a value add, there is no real reason for them to want to move. Sometimes you have to beg, you have to use goodwill, you have to bribe them.

Internally, it's a little easier. We can put money aside. Internal strategy with enterprises, we sometimes put aside a pot of money that we give to them and say, here's money to help your developers do that, so it won't take away from your future velocity what you're building. You can get from our money, we'll pay you to do this, because you recognize you don't want to do it. There's no value to your business by doing it, so we're going to help fund it. Clearly, it's more difficult when you have outside customers, but you can think about other mechanisms to ease the pain a bit.

Participant 4: You want to strive for the ice cubes, but then you got these snowflakes that you built, so then, how do you get rid of those snowflakes?

Matthew Liste: Just to state the point even more specifically, your incentives are not aligned with the client's incentives, usually not as a platform provider. Their incentives, once they use your stuff and it works and they don't need anything new, they just stay on it, never talk to you again. It just works. Your incentive, of course, is to maintain the platform on behalf of all clients, which does means occasionally you need to shut things off. Your incentive is often not aligned and you have to find the best way to manage it. Again, different techniques, but it is one of the hardest things I do. That's why you want a finite number of dogs in your house. Because if you have too many of them, it becomes unmanageable. Now you have a zoo. Often, that feels like my everyday managing a zoo, but hopefully it's well structured. That's a really important thing for all of you to think through.

See more presentations with transcripts

Building Reusable Evaluation Frameworks for Agentic AI Products

Susan Chang — Sun, 04 Oct 2026 16:00:00 GMT

Summary

Susan Chang explains how Elastic transitioned from siloed, ad-hoc AI agent evaluations to a unified, production-grade framework. She discusses balancing LLM-as-a-judge with deterministic rules, bridging Python data science evals with TypeScript production code, and implementing deep tracing to catch regressions across complex RAG and cybersecurity workloads while preserving domain context.

Bio

Susan Chang is a Principal Data Scientist at Elastic, where she leads work on production-grade machine learning, including generative AI and agentic workflows. She is also an internationally recognized speaker, O'Reilly author, and keynote presenter at major global technical conferences.

About the conference

Transcript

Susan Chang: My name is Susan. I'm a principal data scientist at Elastic. We make tools like Elasticsearch and Kibana. I've also written a book with O'Reilly called "Machine Learning Interviews." Today we're going to be talking about a few things. I'm going to talk a bit about what kind of agents we've been building and try to maybe connect them to what you folks might have been building. Tracing, which is, in my opinion, very important and a huge foundation, as well as how we use that to enable all these evaluations. I'm going to talk about how our company made this journey through maybe individual, like rebuilding a lot of evaluations towards building a shared framework, as well as what are the building blocks we have in that shared framework. Then tying it all together, and then see what takeaways that we have there.

AI Agents We Are Running in Production

Just so that we can be roughly on the same page, I'm going to describe a bit of why we're building agents at Elastic and the type of agents that we're running in production. Elastic, we make Elasticsearch, so a lot of companies, they use us for just search or retrieval. They could use us for like normal keyword search or vector search. Every time you've been searching on Stack Overflow or using Uber and whatnot, they use some sort of Elasticsearch tools. GitHub as well, and Tinder, these are just some examples. I mention these because there's just a really wide range of industries and use cases. It's really hard to describe what exactly people do. I'm trying to put some examples as well as what we're doing internally. What is the bottom line here? People just have data in Elasticsearch and then they can apply it to different use cases.

Internally, we actually build on top of Elasticsearch, observability as well as security. This is important because I'll be talking about some examples where we actually, in Elastic, we build reusable security analyst tools that use AI agents on top of Elastic. This is one example. The use case for this is that like an Elastic user, they could already be storing a bunch of logs in Elastic. We have one huge banking customer. They said that they're ingesting petabytes of data into Elastic for the cybersecurity use case. Why is this important? When they have a cybersecurity incident, they need to be able to look through all those logs, filter them, and extract the right information. They already have all this data within Elastic as a datastore. In Elastic, too, we want to build some reusable things for the user because they might not be power users. They want to be able to extract the logs or just use AI to read the relevant logs and then summarize it.

One thing we've built in Elastic is called attack discovery. It's like, I'm using agents to be able to pull from logs and then find possible attacks instead of a human going through and manually parsing through them. There's another agent that we've built that is also on top of Elasticsearch where people build enterprise chatbots using their proprietary data that's stored in Elastic. I mention this because these are super different examples, but because people use Elastic for so many different things that we also build them internally so that people can reuse them even if they're not experts.

When we're building these agents, we want to estimate the quality of the responses and prevent regressions when we make changes, as well as repeatedly test them without too much manual work. This is what we would build for each of those. For that attack discovery agent suite as well as for the enterprise chatbot situation, each team would create their own datasets, they would create their own evaluators. Then you would trace the data in different places, we'd store them in different places, and then do evaluations in completely different locations. Just as an example of what we would build, this is that cybersecurity example where we have AI agents that would extract potential cyber-attacks. This is what we would do. We collaborated with security experts, like security analysts and researchers to find some novel attack scenarios and they would create these scenarios, which are for evaluation purposes. The AI agent needs to be able to identify these attacks if they happen.

We would need to create these datasets. We would need to create and store all these benign scenarios as well. We would need to use these metrics. This talk doesn't really cover all the metrics in very huge detail, but I do want to show you this example because the security agent builder team, or the team that's building the security AI agents, we are using precision and recall to match alert IDs. We want to prevent this attack discovery AI from hallucinating. We also use similarity scores, factuality scores. MITRE tactics I mentioned here are like a security domain, like taxonomy, classifying different types of cyber-attacks. We want to make sure that the LLMs are not hallucinating unrelated MITRE tactics, for example.

We also have some domain-specific metrics as well, because the security agent team wants to make sure that our agents are not performing poorly. For this other team that's building the enterprise chatbots, they're creating a completely different set of test scenarios. We have different indices using like query and retrieval scenarios. I think I have an example here. This is just like one example, which is, I'm trying to verify my domain with Google Workspace. Do I need to add this to my DNS and whatnot? It should be extracting and answering based off of existing documentation. This is what their data looks like, completely different from the security AI agents team. This is more for like a chatbot team. They're also using precision and recall. They're using factuality. They have some other metrics that they're quite concerned about, the response relevance, response completeness. This is pretty interesting too, ES|QL.

This is like a query language that Elastic has built. We have actually maybe a bit too many query languages. We have like KQL, ES|QL, Painless script, but either way, for AI agents that are built to answer things about Elastic, they should be generating correct syntax. They should be generating correct ES|QL. We also were able to build into those AI agents calling tools, to be able to format or check these queries. This is important. This team is concerned with that, whereas the security AI agent team was not so concerned with that. We have even more teams that are building different types of AI agents. I think for folks, your company might be in various stages. We've been building a lot of different use cases for maybe a year, two years or more now. We have other security domain AI agents, and then observability as well, and so on. As we build out all of these different agents, or skills, or components, we want to be able to evaluate how they're doing more and more needed. Each team is building their own agents as well as evaluations.

Tracing Your Application

Just to go through this part, I do want to emphasize that this is pretty important for tracing. There's a lot of tools. We have heavily used LangSmith and Phoenix as well within Elastic, because Elastic already has the capability of observability for a long time. I think that's not the only tool. We have all these other tools that we have tried out. Other companies might be using their own. We tried different tools. I think because the industry is moving very quickly, it was really important for us to see what fits our use case. Then we're also building things and learning as we go. We wanted to trial out different tools as well. This is an example. Let's say this is the design of your AI agent using LangGraph. This might be how it roughly looks like from a conceptual level. When you trace that, you want to be able to see all the calls.

Let's say it's an enterprise chatbot case. Actually, when you type, can you help me resolve this problem with an Elasticsearch index? It needs to run vector search, run keyword search, and the AI agent's doing that under the hood. Then you need to know this is what it decided to do in response to this user query. It has very granular amounts of information of what your agent is doing and what tools are being called, what vector search is being run, what databases and data is being retrieved, and so on.

One other example using the Elastic Observability, it's pretty much the same, or are using Phoenix. Either way, either tool, you want to make sure that you're capturing the data that you need for evaluation, because let's say you've built this AI agent, and you're only evaluating it on the end result. You don't know what happened in the middle. You don't know that the agent is actually querying the wrong database. You just saw that it has the wrong final number. It's really hard to actually take action on these evaluations if you don't have all this information. Or, with this information, you could also run evaluations on specific tool calls, things like that. One thing I really liked about LangSmith is this Add to Dataset feature, because, as new scenarios come in, you might have a situation where the AI agent actually made a wrong response. Your user might have given you some feedback.

Let's say they gave you a thumbs down. Then you track that. You're able to add that to a test dataset that you can run on future iterations of the evaluation, to make sure that your AI agent now performs well on these situations where it used to not perform well on. One anecdote, because I've heard people ask, why do we need to capture so much information? As well as, of course, there are industries where you cannot capture all of that. That's fair. I've worked in more restricted industries as well. We do have to make use of a lot of proxy signals. In terms of other types of machine learning use cases and evaluation, we've always had some sort of telemetry or a clickstream data if you've worked in recommender systems or whatnot. Even if you're unable to capture all those signals, we've always had to make use of some sort of proxy signal or explicit feedback, implicit feedback, and so on. I think this is very similar to that.

Shared Evaluation Framework: Overview

Now to get right into this shared evaluation framework. As I mentioned, it is really cumbersome to keep building the evaluations from scratch for each team. We wanted the different developer teams to collaborate and maintain a shared framework and make it really quick to spin up fully featured evaluations. One of the examples after building the shared framework was that we created AI rules generation. Rules are like YAML/JSON configuration files. They follow a specific structure. Then we made this agent so that users who are not very knowledgeable about how to create them, they can just type in natural language, create me a detection rule that can flag users that logged in on the weekend, or things like that, like anomalous behavior. I was able to spin up the evaluations for that and focus more on other aspects of the work instead of rewriting an entire evaluation suite from scratch.

This does take some time. Initially, the security organization and the agent builder/chatbot example, built out completely different evaluation frameworks. Then we had to go through some consolidation there. This is what the shared toolkits, they do. It can import the datasets. I showed earlier for the cybersecurity case, they have completely different types of data than the data that the chatbot case uses. The chatbot case uses like Q&A type. Whereas for the cybersecurity, it takes in security logs and then needs to extract potential attack scenarios. It needs to import different types of datasets. We do have a shared and standardized schema. We also have the trace-based evaluators, which are more related to like observability side, like token usage, latency, tool calls, performance. We also have shared RAG evaluators. I showed earlier some examples. We're building the enterprise chatbot example, but they also have different types of AI agents that are using RAG under the hood as well.

As well as any custom evaluators that are needed. This is one example of how one evaluation run would go, which is, we have the datasets, which have the input and output. It would call the task or just make an agent run. In the enterprise chatbot scenario, you would have a question and an answer. We would run the evaluators, the code-based ones, LLM-as-a-judge, and other trace-based ones. These are customized per use case. Then store the results back into Elastic or other types of tools that I mentioned earlier. This is like a high-level workflow of what is reused and then what can be customized within these building blocks.

As a data scientist, this part was really interesting to me. Just as an aside, a lot of the codebase is written in TypeScript. As a data scientist, we originally used Python to write these evaluations. We used Python to load the datasets. We used Python to call the AI agents or run the equivalent of the AI agents, which were actually in production in TypeScript. We had a little bit of a mismatch between what's actually in production and what we're comfortable with. It was interesting because originally in the codebase in TypeScript, they were already using Playwright, which is a framework for running tests and APIs. We worked together with these software engineers, as well as using a lot of Claude and Cursor and whatnot, so that we're able to translate a lot of those tooling from Python to TypeScript, so that we could more closely match what is actually in production.

This was definitely interesting. I think it's like an example of potential AI enablement as well, which is that we are stepping outside of our comfort zone with using Python. Actually, in TypeScript, there's a lot of stuff that is not really supported by a lot of the mature frameworks that Python has, but just that, for this scenario, we wanted to match production more closely. I wouldn't say that it's applicable for every single case. I'm not saying, go and translate everything, port everything to TypeScript from Python. No, that's definitely not what I'm saying. Just like for the production case, we wanted to do that. We are running on Playwright, basically, and with a customized Playwright, which we call Scout. This runs everything. It loads the datasets. It also runs the AI agents, which are implemented in TypeScript. Then it does all the collection and then the evaluations.

That's just one thing that I wanted to talk about. It's like related, but it's more about our experience as we develop to these shared tools. Also, from the developer standpoint, because in the codebase, like we're using TypeScript, so we use a lot of local development as well. It can also run everything locally and then output the scores and results on the terminal, which is convenient from a developer user experience.

Shared Evaluation Framework: Building Blocks

These are the general building blocks, next on, I'd like to talk about. I mentioned the high-level workflow, which is that we saw that different teams were using these metrics like precision and recall. I showed that in the security use case that we use those. We also tested against factuality, and so did the enterprise chatbot team. They also were using these metrics. This is the type of metrics that we eventually consolidated between teams. Semantic similarity, probably mentioned a bunch in other talks. Actually, I heard some mention of this in a previous talk as well for a semantic search, where you want to identify if what the semantic search or the vector search pulling is actually the most relevant documents or the most relevant information that's being pulled by the AI agent. This is all important for different types of AI agents that are making use of RAG or some sort of search and retrieval.

The third one, factuality, which checks against like, let's say, product IDs. If you're building a chatbot example, you don't want to hallucinate a product ID that's not there. For the cybersecurity use case, we didn't want it to hallucinate MITRE tactics. Those are categories of cyber-attacks that don't exist. We also built some shared tooling for LLM-as-judge, or you might call it model-based evaluations in the shared tooling, because we found that a lot of different cases were using LLM-as-judge, but just we had implemented it separately in a different way. We actually did start out with all these ad hoc evaluations and then figured out what we could consolidate because everything was moving pretty quickly. I think if we had sat down for two months and planned this out and then built it out and then started building our agents, I don't think we would have been able to ship so much. It depends on what stage that your organization or that your company is at.

This is an example of how we did this in Python, but as I mentioned, we later did move it into shared evals in TypeScript. I have showed this example elsewhere, for more like a data science audience. I wanted to say that if you're familiar with Python, you can start building these evals right away, even if you're not familiar with what the actual production stack is. You can still build them. You can still build your evals. Later, if you're thinking there's a mismatch between your evals and what the scenario is in production, then you can later merge them. This is one of the examples of what we did in the data science team. LLM-as-a-judge, just as an example of the ES|QL, as I mentioned, that's like an elastic query language, a user might ask something like, generate an ES|QL query that can do this or that, like search data.

In other cases, it might be like, I want to use generate SQL, like SQL or something. We at Elastic, we own ES|QL, like this query language. We have to make sure that it performs well in our own AI agents where the user is asking us to generate ES|QL queries. We would have known good answers in the test dataset. The shared evaluation framework needs to be able to pull these known examples and compare them over the test dataset. A very simple LLM-as-a-judge is just, are these responses similar? Of course, as things mature, it became much more complex and more nuanced. This is just like a starter example here.

Some of the pros and cons of the LLM-as-judge that we encountered, but we eventually, of course, still decided to put this in the shared evaluation framework, is that we definitely used LLM-as-a-judge a lot. I think at the beginning, we might have this scenario, if you use LLMs to evaluate the performance of another LLM's response, is that just like garbage in, garbage out, or like evaluating garbage and whatnot. We did actually try it for a lot of different scenarios, and we found that it is useful. As long as you are pairing it with the right type of hybrid evaluations, it's good for scaling to different ambiguous scenarios. Or let's say you have an AI agent that is meant to read in a certain tone or brand tone, coherence or style, these are all pretty good situations where the LLM-as-judge probably performs better than other types of evaluations that I'll mention later.

It's good for open-ended tasks, as well as you can self-grade it using chain-of-thought to explain why it came up with the results. Like, it didn't fit our brand style, or it did fit our brand style and tone, and why. This is one thing that we found. The cons are that it might not be granular enough. I mentioned a few scenarios where in the result generation, it might have stuff like query language, output, as well as different things, or JSON objects, or YAML, where it needs to follow a specific format. The LLM-as-judge might not always perform best by itself, so we have to combine it with other tool calls and things like that within the evaluator. Also, another issue would be that there is different maybe results across runs, even with the same model. LLMs might not be able to identify things like internal info, like product ID.

You'd need to combine that with other things like tool calls or more like programmatic evals, and so on. This is one interesting thing that we found as well. This is for a cybersecurity use case, which is why, as a team, we were researching these cybersecurity evals, and Meta also found, and CrowdStrike, which is that if we're using the same family of models, in this case, like Llama to evaluate Llama, then they would think the Llama models perform better. This is something that you might want to pay attention to if that causes any unexpected results. The interesting thing is like, we have been doing this for maybe a year or two years, and we had a bunch of this stuff written up internally, and this January, Anthropic released a blog, which covered a lot of the things that I had mentioned and that we had found as well in reality. It was a pretty good validation with regards to their opinion on LLM-as-judge and model-based graders.

Next, we have what we call rules-based evaluations or programmatic evaluations. Other companies or other teams might call it different things, but I'm just going to use this for now, which is catching the crucial points that LLM-as-a-judge is unable to do well. Using code like Python, TypeScript to check if generated code works or does not work. Heuristics, let's say you can extract different entities. Let's say, for this chatbot example, it needs to output the correct product ID. If it didn't output any product ID, then it should be wrong. Just simple scoring logic that you would want to have that you do not need to rely on LLM-as-a-judge for, as well as programmatic syntax checks, so on and so forth. Things that you do not want hallucinations for, things that have a deterministic answer, if you're able to break that out, then, in our experience, that really helps with evaluating the outputs of your AI agents, especially if they're working under some ambiguous circumstances.

Combining it with the LLM-as-judge is ideal. The pros and cons here of these building blocks are that there's no ambiguity. It catches some of the weakness of the LLM-as-judge. It's cheap. It's fast. You don't have to make an LLM call for just every random evaluation. It's harder to scale, though, for larger problem spaces, as well as natural language. It's really hard to use a rules-based or programmatic evaluation on a full chunk of text and say, this answer is correct. It's better to let LLM-as-judge handle if it's a larger chunk of text, natural language. It's a bit weaker at open-ended tasks, which is where we try to use them both together.

What Couldn't We Abstract?

In the end, what could we not abstract into our shared evaluation framework? I mentioned a few of these building blocks. It does overlap a bit with what each individual team has done, but I did want to mention it here because those were the things that we ended up abstracting out, because every team was using all of those things in very similar ways. That made sense for us to abstract out. What couldn't we abstract out and what each team should be owning themselves? We could not really abstract bespoke data creation. I mentioned in the cybersecurity use case, in the security use case that is still required, we had security analysts and security researchers that are giving us a user perspective. If you have a cybersecurity analyst AI agent that's supposed to help security analysts find attacks faster, we need to know from them, not from software engineers and data scientists, if this is working well.

We need to know from the actual end user what the test dataset should be looking like. That is pretty important, like having domain expertise to contribute to the dataset creation. That's not something we could just automate away in a shared evaluation framework. Also, on a related note, product input. If you work closely with the product team in your company, what are the positive examples, what are positive behaviors or negative behaviors of your AI agents? What are situations where the AI agent could get things wrong a little bit? You have to add them all into the scoring mechanisms for the evaluations. Regression might mean a different thing for different agents as well. Sometimes we have released updates and then it might have previously performed well. Now it became more aggressive and didn't perform as well. One of the examples for the security agent example, it used to not hallucinate some attacks.

After we released an update, even if you gave it all benign data, it started to think, yes, there's someone attacking our environment. We were able to fix the evaluations to catch those behaviors. A lot of this is iterative. You might not know exactly what counts as positive or negative examples or actual regressions. It's hard to predict everything, especially in this more ambiguous workflow. The shared evaluation framework cannot be responsible for calibration. Calibration, basically that if you're running the evaluations, they should roughly be performing the same. Your evaluators generally agree with your human evaluators or your target behavior. If the evaluators are not well calibrated, then what it's going to give you is just junk. Let's say you're using LLM-as-judge and then you run the same scenario three times, and the LLM-as-judge thinks it's a different performance each time. Then that means it's not well calibrated. We had to further improve these things. This is all reliant on domain expertise, as well as each team's individual expertise. This is not something that we have in the shared evaluation framework, because we can't.

Key Takeaways

Some of the key takeaways. If you're building the first agents in your company, it's definitely a little bit different for each company. This is how we started when we were building our very first agents, very first LLM-driven use cases. We were working pretty closely with the domain experts. I think one thing I really remember is that the cybersecurity cases, we worked together with the analysts and they would spin up VMs. They would spin up different Okta, like test environments and things like that. Those are pretty hard things to do. If your AI agent is meant to do something really complicated with that, don't skip the product and domain expertise. It is tempting to bypass that and just ask an LLM to generate a bunch of stuff, which we eventually did to scale up the test dataset. We made sure that we anchored that to a lot of this real expertise and real user needs and things like that.

In this early stage, when we were building our first agents and other teams were building their first agents as well, we tried out different tracing tools and see what works. As I mentioned, we actually started building agents pretty early on with LangGraph. It made natural sense to start using LangSmith at the time. Other teams, they started using Phoenix or whatnot, and other teams still you might use Elastic Observability. This is what was happening early on. Starting small is also something to note when starting out. I have spoken to a lot of people from maybe folks in this kind of stage, and they're a little overwhelmed. How do I create this dataset? It sounds so complicated. I'm making an enterprise chatbot, how do I gather these question-and-answer examples? I've seen from people in industry, and actually also from Eugene Yan's blog as well. He currently works at Anthropic.

He also mentioned that, for some situations, just starting with a sample of 20 records for your test dataset, to 50, and then you can later scale it up. I think it is important to answer, especially if you're at your first stage, and then that means the AI adoption or AI development at this company or organization is more so starting out or nascent, then that means ad hoc work is fine. There's nothing wrong with just having a small amount of data. It is important to evaluate. That's how you can justify further building out of these AI agents in your company or even scaling up to different teams and different domains. I think it is fine to not overinvest in abstractions at this stage, but then optimize for more learning and experimentation. That's what we did. That's our experience with these different teams. I think, actually, early on, we might have worked in a bit of a siloed situation.

Like, wait, you're using Phoenix? Wait, we're using LangSmith. Something like that. That might happen. That's ok. I think I heard the speaker from LinkedIn, he was saying that some of it is an organizational thing. Like, if you have a shared way that your company is using LLMs, that is also a challenge and that is important. I echo that. Communicating cross-teams and seeing what you're aiming to do.

If you have a few agents in production, so this was us when we had maybe some agents and then we had different evaluations. We did have them, but if you don't happen to, definitely start building at least one. Product and company leadership is going to start having questions about how your agents are performing. Users might start having troubles in production and you want to be able to troubleshoot them. This is why you need to at least start building some tracing or evals. Thinking about the common metrics you're observing and iterating on that. As I mentioned, for the security use case, after we pushed some changes, we realized that they were hallucinating attacks on benign data. That means that we had to go and refine the metrics. It's all iterative. You might eventually run into the problems of too many different evaluations maybe caused by siloing or caused by different product teams building their own thing.

Also, wanting to move fast. I think that's fine, too. That's what we encountered as well, which is like, you do have to move fast. You might have to just stick to one tool and decide. Consolidate later. As the agent development matures, then all the shared tooling should be consolidated further on to avoid a lot of pain. This talk is about the evaluation framework. We also had done a lot of work to consolidate all the agents as well. We used to have security agents that does this. Then I found that I had to click a dropdown to switch to another agent that was also on Elastic. Then in a recent release we consolidated that workflow, and it can identify what agents you're talking to instead of going to a dropdown just because you clicked onto something else. Those are things that we also consolidated. This talk is not about that. We did have to consolidate agents as well as the agent evals.

Lessons Learned

Lessons that we have learned, if we could do things over. Building first and then consolidating later. I think if I had to do it over, we would start communicating a bit earlier, and also communicate with the goal to work together and consolidate. Because I think for some time we're like, your team does that, our team does this. Maybe can you give me some of the access to your platform? Then they would just add us to their team on Okta or something, ad hoc. That did work. Then we weren't really talking about consolidation at that time. I think that lasted for some time. If I had to do it all over again, I would really try to do that. I would also try to make it easier for feedback to improve the product. I mentioned proxy signals, and things like that. Because for a lot of the enterprise users for Elastic, we can't trace all their data at such granularity.

They have to agree to share it with us. Even if they share it with us, it's pretty redacted or pretty bare bones. I think if we're able to build a better way for users to provide feedback, this might be similar to thumbs up and thumbs down, you might see on ChatGPT or things like that. Whatever mechanism, even if you're only getting these proxy or indirect signals, at least having some feedback mechanism or a structured way to track. For another machine learning product we had, we use Google Forms. At least you have something. Have something like that. This is a really interesting thing, like the main stack. What I mean by that is that in our situation, because everything was in production in TypeScript, and we had built and invested a good amount of time in building the Python evaluation framework, if I had to do it over, I'm not sure if I would exactly not do the Python one at all, because that's what my team was specialized in.

I'm a little bit half and half on this one, because I do think when we did jump into using TypeScript, and we were using of course Claude and Cursor heavily for that, with the review of software engineers that are more experienced in TypeScript, that did enable our team quite a lot to directly contribute to the shared evaluation frameworks. I do think this was a good decision in the long run. I don't think we would have started out that way, even if I had to do it again. If you already have your own shared evaluations, what would you do over? I think that's something that I'd be curious to know. We do share some of these evaluation results publicly, not for every scenario. I shared at the very beginning of the talk that different industries and companies use Elastic for different things, and also internally, we try to build these reusable building blocks for our customers and users in different domains.

We do share some of our recommendations for users in security domain and whatnot with evaluations like that, but not every single thing. This is interesting having a bit of public scrutiny or sharing our work a little bit, which does demonstrate that it is nice and useful to have evals so that you can see what you might recommend to other teams in your organization or even to the broader public.

Questions and Answers

Participant 1: We currently use Phoenix in-house for the tracing aspect. I'm curious how you think about storing the data in Phoenix versus in an application database or something else. Is all your eval data in the tracing framework, or is some of it in the application database? That's the first question. Then the second as a follow-up of tracing, if you compare it to OTel and now with the rise of harnesses and deterministic steps, sometimes the failure is not just in the inference steps. Let's say, it might be a deterministic step that is a failure, how do you think about incorporating that into the overall debugging and training process?

Susan Chang: The first part was around Phoenix and where we store the data. This is a bit of a hard thing, because in some organizations and some companies, you already store observability data somewhere, so that makes it easier to make a decision. In our case, it actually went like this. I believe we had our observability data stored somewhere else. Because that observability, including our own, didn't have a lot of the LLM-specific tracing out of the box at the time, we just decided to get something like Phoenix that had built a lot of that, the useful add-ons, for specific LLMs or agents tracing. We had this data stored in other places. Let's say you already had a lot of observability data stored somewhere, then I might as well just use the same thing, if I had to start over and if every tool was mature already. That's the way we did our decision-making, which is, yes, we did store them in maybe different places depending on what we wanted to do.

All of our applications already had observability that we were using for a very long time that was in some other thing. That's one part. I think I've heard people, let's say they use MLflow, actually, for their agent. That's because they had already had MLflow for some other machine learning use case and whatnot, so then they ended up using that, even though I'm not sure if it was because it's the most mature or because the data storage was actually easy or anything. It depends on what's easy and what your company already has, and whether your company has maybe an appetite to try out different tools. Because I think at the time, a lot of those tools are not that mature yet. This was since a year and a half or two years ago. Some of those tools they were building as we were trying them out, and they changed so much.

Participant 1: As agents stop becoming a one-shot thing where you just provide context into a black box, as there's more harnessing and deterministic steps around it, how do you think about, from a debuggability and tracing aspect of incorporating the deterministic steps as well into it? Because sometimes the failure is not in the inference, it's in the deterministic steps.

Susan Chang: For us, when possible, we do trace everything in our developer environments. That does include a lot of those deterministic steps and things like that. When we do evaluation, it's not only on the end result, but sometimes we might do it on, is it making the right tool call, for example. We do have a lot of the granular evaluations as well, depending on what steps in the middle the agent might be taking. That's only because we have that data available that we're able to evaluate those things. Any deterministic thing, if it can be evaluated and we have the data, we are definitely taking that into account.

Participant 2: I'm wondering if you could go a little bit into, once you run the evals, what kinds of considerations go into setting the criteria of, yes, we're going to promote this agent, or, no, we're not? Are some of those criteria either pass, fail, or is there some way of setting lower bars? Or are you just looking for improvement? Just looking for some general thoughts on that.

Susan Chang: We have a matrix that just has the granular scores, like let's say precision, recall, and then factuality, and other things. Or maybe if we're evaluating specific tool calls or things like that, we'll have specific metrics. Then we'll have a roll-up score, just a simple weights thing. Let's say we think factuality is really important, then we'll weigh that higher. Then we just add them together and then have a composite score that's flattened into one single value. We also have the other scores that are available so people can see. Because when you flatten it, it's really easy for decision-makers or even for ourselves, like we just want to see one value at the end. We might have a threshold and it has to improve, but it needs to be over a certain threshold as well. We also want to investigate the maybe more detailed granular metrics as well if we can. I think the production is actually more simple. We just flatten it into one score, threshold, and then we might run some ad hoc tests, manual. Manual kind of like use the product and click around and stuff like that, too, to do a last smell check. Then we release it. It's rigorous, but also not really.

See more presentations with transcripts

Building GenAI Platform at DoorDash

Siddharth Kodwani/Swaroop Chitlur — Fri, 02 Oct 2026 16:00:00 GMT

Summary

Swaroop Chitlur and Sidd Kodwani share DoorDash’s journey building an internal GenAI platform. They discuss core architectural bets, transitioning from vendor-first setups to open-weights models, navigating LLM and agent gateways, and balancing accuracy, latency, and cost for over 5,000 internal users.

Bio

Siddharth Kodwani is a Software Engineer on the GenAI Platform team at DoorDash, building infrastructure for AI agents that accelerates development velocity and improves production reliability. Swaroop Chitlur leads the Generative AI Platform at DoorDash, building infrastructure for LLM inference, fine-tuning, evals, and AI Agents.

About the conference

Transcript

Swaroop Chitlur: My name is Swaroop. This is Sidd. We started this team called GenAI Platform at DoorDash, and this is sharing our story about all the things that we did to make a successful platform. We talk about the principles, the bets, the pivots, and what do we mean by success. How many of you are shipping GenAI-based projects at work? How many of you are in a platform type of function where you are supporting other teams be successful at this? Hopefully some of the things we talk about will resonate with you all. Our journey actually starts in roughly April '23. As you know, November '22 was the ChatGPT moment. Then a few months later, we were like, we need to start using OpenAI. I was tasked with signing this contract, and I remember I was sweating, like, are we really going to spend this much money? Of course, in hindsight, that number seems minor now.

The feeling was very real because at that moment, I was like, if this thing takes off, how do we support this in such a large company? How do we make this productive? How do we make this accountable? That set the tone for what we were planning on. The thesis of this talk is very simple. We had initial principles and bets in mind. As the industry evolved and started doing newer things, we started adapting accordingly. We were still able to navigate these changes because we had those principles in place. Of course, we went from models to workflows to agents. That's where we'll walk you through our journey and what decisions we took, and talk about the technical projects in some of them. You get a balance of what was our strategy and what was the technical proof and what was the results. Hopefully those are takeaways for you at the end.

Operating Model

When we say about principles, what do we mean by that? When we started this team, we were given a blank slate. What do you do next? I don't know. Then you start thinking, first let's decide how are we going to operate? What are we going to decide on? We didn't come up with these. This was already part of our foundations or tenets. Be customer obsessed. Focus on the teams and the use cases. Not building for the sake of building product. Build products, not systems. Our customer is a product engineer. Don't make things where your customer has to stitch together different workflows. Try to think of an end-to-end workflow. Think of onboarding, think of support. Think like a product team, not just building systems. Make the right thing easy. If you say, do best practices, nobody's going to do it. You need to bake in best practices into how your platform operates. All this while we got to prove our value, our value to the customer and prove our value to our stakeholders.

Chapter 1 - The Customer Changed

Speaking of be customer obsessed, when we started with these principles, now we were like, we don't know what to build. We don't know what's coming. How about we talk to our customers and see what they're building and see where we fit into this picture. That's where we started. The first thing is the customer changed. I remember we were part of an ML platform team. The GenAI platform became a team under the ML platform org. We were used to thinking and talking to machine learning engineers. I remember in one of the conversations, I said, we want to try this out? Just use a notebook. They were like, what's a notebook? Then it just instantly clicked in our heads that we are no longer talking to ML engineers as our audience. Our audience is all engineers, which seems obvious in hindsight now. Remember, this is in '23, '24 that we're talking about.

That's when we realized, we need to move to APIs first. We need to think in terms of SDKs. We should not think in terms of notebooks or GPUs or getting access to compute. Thinking more in terms of APIs. Another decision we took in this process is that we'll focus on business impact. We don't want to build another chatbot. Of course, there were no coding agents back then. If you apply the same principle, we explicitly decided that this team was going to focus on business impact and product, not on the coding agent side, not on the chatbot side. That was a conscious decision we took. We wanted to double down on what kind of use cases our product engineers are building. As we spoke to them, we started seeing some trends, started seeing some broad buckets: automation, recommendations, personalization. Again, this was in 2024. We noticed that automation is decreasing the bottom line.

Recommendations and personalization is increasing the top line. We are in a general good sense of direction in terms of what value we can provide. We wanted to refine it further. Like, what is the value that we provide? That's when we decided that our USP was going to be help product teams optimize between these three things, accuracy, latency, and cost. You will see this theme emerge in all the projects that we took up after that point. What is the proof? Why am I saying that it was successful? Right now, we have over 5,000 users internally onboarded to our platform. Internally we have 45 users onboarding every single day. Surprisingly, something I did not predict was that 40% of people are non-engineers. Everyone from legal to sales and operations and strategy, they're all using APIs.

Chapter 2 - Adoption Created Accountability

Siddharth Kodwani: I'll go a little deeper into, from a technical point of view, what steps we took and what we learned from them. As in the last slide we mentioned that, eventually the audience that we are building for, we started with like, we built for ML platform, ML engineers. Then we went from, we are building for all engineers. Eventually we reached a point where 40% of the users of the platforms are non-engineers. It took us some time to realize that this new technology that is coming in is going to have a lot more adoption. It's not just engineers. It's going to have non-engineers, stakeholders are going to increase. We have to think from a platform point of view that we're enabling everyone. The ethos that we generally used to believe in while building such solutions is I want to build for velocity of my users. I want to build reliable solutions. As the surface area becomes big, you also need to have an accountability. I'll go in deeper for these scenarios.

As we started thinking about these scenarios, we need one surface that is usable with the whole company. The LLM Gateway is a concept that we came up with at that point. From an engineering point of view, we went to our users, understood what the problem was. This is early days. There are a lot of different providers. There is OpenAI models. Then came Claude models. Then came Gemini models. Then came open-source models. Now almost every product team is trying to experiment with them, they want to move fast. If they have to create their own plumbing for each of these different providers, each of these different types of models, then they were getting slow. From a velocity point of view, we quickly realized that we need one API and one SDK kind of approach where we do the plumbing for you. You get one API. You want to try an OpenAI model.

At that point, new model frequency was also pretty good. Every couple of months, you'll see one model that is supposed to be better than the other. Any team who is experimenting, they want to quickly try that model to see, is this application performing better with this new open-source model versus Anthropic's model versus an OpenAI's model. One API, one SDK, you make one change, and you can try a new model for you. At that point, it's important to be with the users. We were constantly with them to understand the problems that they are facing. We quickly realized that the capacity was still an issue. Even if you want to go use Claude model or an OpenAI model, you were getting rate limited. You needed good fallbacks. Teams are trying to build production services, they cannot have like, OpenAI has some shortage of quota or resources for us, and so we cannot use them.

We actually had to go understand what the fallback should be, what kind of routing should be. At one point, I remember we had OpenAI endpoint, but we also had a fallback on Azure endpoint. Or, we had Claude model API endpoint, then we had a fallback on Bedrock, so that the services can rely on us from the reliability point of view. We went in that direction. We quickly realized that as the usage is going up, then we need a way to track, because now finance got involved. Like, you guys have one API key for multiple teams, how do I track who is using what? We quickly realized we need workspaces where we have cost attribution. We can actually drill down on who is using how much. We can attribute cost back to those departments. That's what basically we did with LLM Gateway.

Swaroop Chitlur: Now that cost is back in fashion, we can talk about this. We had to take this into account from the beginning. Because if you're exposing this to everybody, enabling everybody, then somebody has to answer for who's paying for this.

Siddharth Kodwani: Talking about plumbing, LLM Gateway sat in between all the callers, all the teams who want to use the foundational models, the open-source models. This LLM Gateway actually boosted up the velocity of a lot of user teams, the users, services. We had accountability in place where any person could come and see how much is the usage for them, for their org, what's the budget for them, and what kind of model they are using, all that stuff. The principles on which we built LLM Gateway, the bets we took that we are building it for the entire company, not just for one set of users, actually helped a lot. Initially it helped with DoorDash. Then as DoorDash acquired multiple companies, Wolt, Deliveroo, SevenRooms, we were able to onboard them very easily and give the same kind of experience to all of them, so that they get the benefit of the velocity, the reliability and accountability principles that we have built in in our LLM Gateway.

Swaroop Chitlur: One of the positive side effects of this is that since we have this gateway, we are able to log every request and response. Now if an engineer wants to go back and see what's my performance, what's happening in production, then you have this UI that you go and look at all the logs. Every team does not have to build this on their own. The central platform provides this.

Siddharth Kodwani: As we said, make the right thing easy. We wanted all this from the point of view that we want to have a central control plane where we can have visibility, we can have attribution in place, we have forthright ownership in place, but we are still not compromising on the velocity of building. That turned out to be a pretty good decision that we made. The numbers speak for themselves. A lot of product teams are using LLMs in their products today with a very small onboarding time. They can revisit their tradeoffs. If you're using a different model, what is your latency? What is the cost associated with it? You can run those experiments very easily. You can debug those scenarios very easily. The adoption has been pretty good not just for DoorDash, but as we mentioned about Deliveroo, Wolt, and SevenRooms.

Chapter 3 - Vendor-First Taught us What to Own

Swaroop Chitlur: One of the bets was as an LLM Gateway, and that worked out very well. Another bet we had was going vendor-first. It's a bit counterintuitive because we are a platform team, we want to build, but we are also saying vendor-first. It was a conscious decision we took because this field is moving so fast. We could potentially say, we'll have this, but it'll take us six months to build. Of course, this was way before we had Claude Code and Codex. We took the vendor-first approach so that we can move fast, enable things, and essentially our velocity leads to the velocity of our product teams, our customers. It worked well for a very long time, and then it started to not work. Not all places, but there were a few specific failures that I want to talk about. Remember, we talked about accuracy, latency, and cost as our team's unique value.

How do you help teams increase accuracy? That's where evals come into the picture. Evals was controversial because every team has their preferred way of doing things. Nonetheless, we evaluated and onboarded a vendor. A couple of months down the lane, we realized that it was not working well because teams had very unique custom tracing needs, and the vendor was a good product, but it did not meet the flexibility requirements that we had. Team also had very custom annotation needs. Like now we have this image and we need someone to be able to annotate on this image what is it that they're looking for. These kinds of human annotation workflows is needed for labeling for us to know what is the ground truth, what is the actual result expected, but what is the actual value we are getting from the LLM. Teams started treating this vendor as a OTel trace store, and they were building their own workflows on top of it, which was least to say a suboptimal.

Then we realized like, ok, we are on the right track, but maybe what we have right now is not the right solution. In the end, we had to soften our stance on vendor first, and then we realized like, we need to own the workflow surface. What is it that our product engineers want to do? We own that surface, but how we provide that can be vendor or built in-house. We softened our approach to say, if the vendor works, scale it up. If it does not, take the lessons learned and then build a more suitable product in-house.

Similarly, on the model side, we went vendor first. Let's use OpenAI, Anthropic, Gemini models. Folks started getting productive. Folks learned how to do prompt engineering. They were able to scale up. At DoorDash scale, at some point, cost becomes an issue. There were a few other cracks that started to appear. Like the one is the deprecation treadmill. How many of you know that Gemini has a one-year lifetime for models? In one year, the model gets shut down, and this applies to every new model. At the end of that one year, you have to switch to a newer model, whether or not it meets your business needs. I could probably accept that if I was a single product team and I was working on my product, but I as a platform team, I have to think about hundreds of use cases and how I'm going to drive hundreds of teams to migrate to the latest model without necessarily giving them a business argument of the benefits.

Similarly, quota constraints. At our scale and at the rate of interest internally, we just cannot get enough bandwidth from these providers. Cost becomes a huge curve as the scale goes up. Go back to the GenAI gateway or the LLM Gateway, we had the portability principle in mind. You're able to easily switch models. Now it was no longer a nice-to-have, but it became a must-have.

Chapter 4 - Portability Became Real

It became a reality that, ok, what is the way to cut down? One of our long-term bets was that we can do fine-tuning and use open weights models and bring serving in-house again. We are an ML platform team. We love to build. That was our long-term bet. Then we saw that cost was becoming an issue. Then we jumped into our long-term bet, which was open weights models. Because we have the gateway, no significant change is required on the product engineer. They just change one config and use a new model. Of course, it's a whole lot of work to make that actually happen. We ended up prioritizing open weights models. We evaluated our vendors and choices. We finally ended up with Modal, which is a Python-first GPU cloud. We go through the LLM Gateway, and we are hosting our own models. The beauty of this infrastructure is that we want to bet on open-source engines like vLLM, SGLang for serving, and other open-source libraries, for fine-tuning like Hugging Face TRL, or Unsloth, or Axolotl.

We started investing this in the second half of last year. This year, adoption has gone through the roof once we had that foundation, because the models are getting really good now. Like if you say, Qwen, GLM, Kimi, DeepSeek, all of these models, they are getting really good out of the box. I would have not expected that as of last year. Now teams are able to switch from the frontier proprietary models to Qwen3, or, for example, Qwen3 Embeddings. They're actually seeing accuracy increase, and cost drops like 20x. Then teams are like, this is great. We should do more of this, because at our scale, cost becomes an issue. In the end, we were able to achieve cheaper and faster in just a handful of use cases onboarded in this first half. We already have single-digit million-dollar savings annualized. The vendor-first softened, but the portability held.

Chapter 5 - Agents Changed the Primitive Again

Siddharth Kodwani: Agents, how can we not talk about them? I think this is the time. The agents' journey also started in 2024, while we were building and solving a lot of user problems from the LLM Gateway point of view. We quickly realized that a lot more applications can be built with certain tool calls, with a certain harness, that can do a lot more than just LLM call. I'll talk a bit about it, like what was the context, where we went first, and then how we evolved from there. A little bit of context. As a platform team, sometimes, it's actually good to not be the first, like the flag bearer, and be like, this is what we are building from a platform point of view. Because we want to understand what your users are actually doing, especially when you're building in a GenAI space, where we have always worked with our product teams to understand what the blockers are, where can we come and help.

Sometimes you need to just take a step back, let them evolve and see where actually is the platform opportunities for you. Agents came in early 2024. Teams were just experimenting with it, building something. I think the critical moment that happened was when MCPs came, that's when a lot of people got involved here. They were like, we want to build this. Let's hook some MCPs and see what can we do from here. From there in 2025, we had 25-plus agents' projects, plus every laptop had an agent, like either Claude Code, Codex, Cursor, whatever. We started with, MCPs are here, we have third-party MCP servers, and we can start with building MCP servers internally. We understand all the learnings that we had from LLM Gateway. We'll focus on, again, the velocity of our users, reliability, accountability. We just focused on MCPs. Again, as it became very obvious that if you're building agents, it's not just about hooking up MCPs, because MCPs, it's like a presentation layer for the data to your agent.

If you're building agents, there's a lot more to it. What kind of experience do we want to unlock from agents' point of view? We started thinking from that angle. We basically understood that the users of our platforms are like, teams were building agents, teams were building MCP servers, users who want to use MCP servers or any other resources that we have on our gateway. Then there are agents running as services, agents that are running on behalf of users. A lot of different patterns came up. We quickly realized that we have to evolve from MCP gateway to agent gateway. I'll talk a bit about agent gateway. The agents are Claude, Codex, Cursor, plus now we have a lot more internal running agents. We started with MCP tools. Like, you can hook up Slack, GitHub, Jira internal MCP servers. Today we have more than 50 MCP servers running or onboarded on agent gateway. We took the lessons from LLM Gateway, like, let's build a very strong plumbing so that users can focus on building agents, building MCPs, and we figure out the way to expose them in a much more meaningful way.

Swaroop Chitlur: You don't want every MCP author having to think about authorization.

Siddharth Kodwani: I'll talk a bit about like, what were some of the core components that we started developing? We were talking with teams who are building MCP servers like, I want to authenticate users who are using my MCP servers, I need to have the right kind of visibility, because I might be exposing some data that should not be getting used by somebody who should not have access to it. We came up with authorization policies that makes it very easy for MCP servers to onboard on agent gateway. You can select the authorization policy, the tool level access that you want to enable for users. We will authenticate users for you. We will authenticate services for you who want to use your tools. We had right observability in place, not just for us, but whoever is onboarding to our platforms, so that they don't have to worry about the observability part.

We also enabled some of the features that were asked, like, my tools are sensitive for writes, and I want to have certain access level policies in place. We will rate limit your users. We built a very strong plumbing so that anybody who wants to expose any tool, they should feel comfortable coming to us. They should be able to see what tools already exist. We had a very good self-serve registry that you can go to, query, understand what already exists, what does not exist. If you are an agent, you can talk to us. We quickly realized that agents also want to talk to other agents. We were protocol agnostic where we are not only going to support MCPs, but we are going to enable agent experiences. We started thinking from that angle. We onboarded different protocols on agent gateway that you can utilize.

Swaroop Chitlur: Can you talk about AG-UI?

Siddharth Kodwani: To reach there, again, the principles that we learned from our past experiences, although the primitive changed, some of the things that we quickly realized that for LLM applications or the services, a lot of GenAI world is Python first. We advocated for Python internally in the company to have strong infra support, serve it as a first-class citizen, along with other language support that we had. That helped, because all the agent frameworks that came along that allowed our users to build agents quickly and start experimenting, they were Python first. We built products for our users, not just systems. If you're building an agent, if you're building an MCP server, don't worry about authentication, authorization, rate limits, integration with different surfaces, like Claude, ChatGPT, we took care of everything from the user's point of view, so that you can just focus on building agents and creating agent experiences.

Identity became something that you have to solve very securely from a safety point of view, from a security point of view. How do you enable an agent on behalf of you, and what permissions do you give to that agent to work on behalf of you? If you're working on your laptop with Claude, you can give a lot more permissions because you have a lot more control, but internally we have more agents that you can just invoke to trigger a certain action, but you don't want that agent to have all the access that you might have given to Claude Code. Solving agent identity tool access became a bigger problem that we solved for our users.

We took a controlled bet, the bet where we knew that this is not just about MCPs. We want to go in a direction where we want to enable agent experiences, not just give data tool access to our agents. We became protocol agnostic from day one. We focused a lot on MCP first, because that's where a lot of use cases were. Quickly, as teams were building, agents realized that they can take advantage of whatever the plumbing that we have built. They wanted to have different protocols supported. We were able to incorporate AG-UI. We are working on A2A. We expect a lot more agentic protocols in future that we can easily onboard and enable experiences for our users. Some of the results, because of the bets that we took, very quickly we were able to reach daily tool calls of 300,000. We were able to onboard new protocols like AG-UI, support streaming on UI directly via agent via the gateway that we have built. Python, the advocacy that we did allowed us to quickly enable our users to build agents. These were some of the results that were really surprising for us, but at the same time, very impressive.

Report Card

Swaroop Chitlur: We talked about the principles, some of the bets, and the pivots, and what was the success. These numbers are our success stories, along with all the use cases we enabled, like 5,000 users, savings, the amount of adoption that we have internally. Hopefully, these speak to what we mean by success. This is what we consider our own scorecard here. These were our bets. Product impact first bet held. A lot of the things that Sudeep's team shipped was built on top of the platform. APIs first bet held. This allowed folks to pick and choose the parts that they need. They are not locked into the platform, so we have much lesser complaints now. You can pick and choose the parts that you need. LLM Gateway was very successful in helping teams with their velocity. Portability was a bet that we had which worked out in the case of open weights models. Vendor-first, we had to soften a bit. We still feel it's the right first move, but we no longer consider it a must have. We do build when we feel it's necessary. New things emerged that we had not considered in the beginning, like evals, agents, and identity.

What's Next?

What's next for us going forward? This is how we look at it. From our top down, we still focused on agents running parts of the business. I know a lot of the talks have been about coding agents like Claude Code and Codex. We as a platform team, we want to enable things that run the business, and that requires a different mindset, a different focus. Maybe in a future QCon, we'll talk about how we enabled agentic commerce. We don't know. It's a bet. I don't know if it'll work out or not, or whether the world is ready for agentic commerce. We'll see. That's the top-down view. The bottom-up view is that we know what offerings we have. We have customers internally actively using it. They tell us like, there's this gap, or there's this bug, or can you add this functionality? We'll continue to expand on those. These are directional bets, but we keep watching for signals. What's happening in the industry? What product engineers are building? Where are they blocked? What are they telling us? What are they not telling us? We go from there.

Key Learnings

We hope to leave you with three takeaways or lessons learned. One, write your strategy early. It helps to have your focus, your bets. In a way, revisit your bets often, because then you know when something is working and when something is not working, then you consciously pivot and move on. The worst thing you can do now is make a decision and not reevaluate it. The second thing we would recommend is, build around customers. It's very easy to build these days. Everybody has their Claude Code and Codex ready on target. It's not always obvious what exactly is needed. All things remain the same. Talk to your customers. Third, own the company-fit surfaces. Auth is an evergreen story. Cost is an evergreen story. Accuracy is an evergreen story. Observability is an evergreen story. Governance, developer experience, and velocity. We own these things, but how we provide it is something we consider on a solution detail. Whether it's vendors or it's built in-house, those come later, but we own these surfaces.

Questions and Answers

Participant 1: I have a question on LLM Gateway. If you're still using, are you implementing the LLM safety and security in LLM Gateway itself? If so, how are you doing?

Siddharth Kodwani: When you say safety, security, are you talking about guardrails?

Participant 1: Yes, like prompt injection, output validation, hallucination control, and all that stuff.

Siddharth Kodwani: Yes. I don't think we can control hallucination at LLM Gateway, but we do have the concept of guardrails where you can come and define the guardrails. You can define the prompts that you should use, system prompts and stuff. From a safety point of view, the ball is in the court of the product teams to make sure that we are giving all the tools for you to go observe what is going on with the LLM calls that you are making. You can have a policy around it, like enable certain type of guardrails. LLM Gateway gives you the surface areas to go understand what is going on, debug it a little more. We don't enforce like, this is how it should be from a user point of view. That ownership is given to the product teams that are building solutions.

Swaroop Chitlur: To expand on that, the LLM Gateway is the right place to add before hooks and after hooks. Before hooks, for example, you want to do PII redaction. This is the right place to put that in a central place, and we run it for you. Post could be things like making sure that the LLM is not bad mouthing your customer or something, or recommending your competitor. That's where these were placed. For things like hallucination, we think of that more as an evals side of the equation. We have LLM-as-a-judge and deterministic judges there as the solution for that. In general, these two places is where we focus on guardrails. As of now, it's like customers opt-in and we don't enforce it. That's more of a company policy thing.

Participant 2: My question was around runtime or compute, any thought given to things like when you're trying to support it. One of the things we're struggling with is to try and maintain a consistent WebSocket connection between an agent session to a user, and it gets really tricky if your agents are running on Lambda or what have you, you can scale up and down. I'm wondering, was that like a consideration? What did you end up as your agent runtime?

Swaroop Chitlur: How do you handle streaming and scaling?

Participant 2: Yes. Streaming has implications on your agent runtime. If the first instance that you talk to, or the instance that answered the user's first question is not the same instance that captures the second one, then you can't keep a streaming connection. I'm wondering, was that a consideration, or have you faced that problem?

Siddharth Kodwani: This is like a developing area, at least in our company, like the agent gateway from the protocol point of view, we are stateless right now. We allow user teams who are building agents to treat us as stateless. If you have a certain tool that you want to talk to, they request a certain session ID, sort of handled by them. Overall, at the runtime, these are the challenges. As we mentioned, we work with product teams, we are seeing a pattern where we want to look at how the orchestration of agent can work, what's the runtime needed for that. We don't have all the answers yet. This is something that we are looking into now. We are talking to users, what are those scenarios where you want agents to run longer? What happens when the pod dies? How do we reestablish the connection? All of that stuff. That story is still not solved yet.

Participant 2: Session management, conversation history, ownership is still with the product team.

Swaroop Chitlur: Yes.

Participant 3: What were your key learnings or challenges when building identity and access management in the agent gateway? How traditional human identity or workload identity fell short when you were building agent identity?

Siddharth Kodwani: I'll focus on the learning part first. The learning part that we quickly realized is important, I think came from the fact that we are building a lot of new features for our users to quickly get access to tools. There are a lot of teams building agents, but they might not have all the security in mind from the user point of view. Let's say I'm a team who's building agents, and I want agent gateway to take care of whether this user can access this tool or not. It's like a three-party system where somebody has built an MCP server, it can have some write level access, it can have some certain confidential read level access. When such MCP servers get onboarded, we first tell them that, you should declare your authorization policy who have access to what. Let's say now that problem is solved. Now, a user wants to access something.

At the end of the day, what we are making easier for our agent building teams is you just tell me the identity of the user. I'll figure out whether this particular agentic action is possible on this particular tool or not. The learning was that, let's make it easier for agents to hand over that complexity of dealing with whether this user has access to this tool, and whatever the resource that tool might provide back to the user. The key learning was, let's make it easier for them. That's where we spent a lot of time on, how do we identify this agent? We started thinking in the direction where, we have an agent gateway control plane where a user can come and see that all these agents that are available that I use today, and these are the tools, so there is a cross combination where like this tool cannot access these particular tools for this MCP server, but this one can.

We enabled it and gave that power back to the user for them to also choose. There's two ways to think about it. MCP servers decide what authorization policy they want for the tools that they have. Then a user decides what agents have access to what tools that they already have access to. That lives in the control plane of agent gateway.

Participant 4: It's super cool that you guys have an agent gateway. I'm curious on how you think about the platformization of other aspects of the system, such as like, let's say the durable execution side of like, whether it's use any kind of workflow orchestrator and evolving them for LLM or agentic usage, as well as like from an eval perspective. I'm also curious that if 40% of the users are non-technical or non-engineers, how you think about the education process or introducing them to different concepts of when to use evals, when to use skills, and that aspect of it.

The first part is the platformization of all the other things that goes into agent building. Specifically, like whether it's eval suites, whether it's like CI/CD of evals, whether it's like the workflow orchestration, whether that's context storage, and thinking about all of that.

Swaroop Chitlur: For evals, we do have our own evals platform that is in the works. Nachiket on our team was leading that. Evals is another part of the story that we did not talk about today. That is the point where we focus on the accuracy aspect and we help teams build code-based deterministic judges, LLM-as-a-judge, and that whole loop of like product iteration. That part is platformized in our evals platform. With respect to the education part, I would say that still remains one of the hard parts. There are a couple of ways we focus on this. One is we do have a developer platform org that we collaborate and join forces with. For example, folks need to level up on how to access all of these MCP tools that we have. Or, you have, how do you figure out how to connect to Slack and things like that?

Our developer platform team has an agent skills marketplace. Most folks can just install that, and out of the box, their Claude Code or Codex or Cursor knows how to deal with all of the rest of the systems that we have. You can just install that plugin and just say, search for logs for this and it knows what to do. That's part of the education part. The other part is we have an active Slack channel that people ask questions and hopefully someday we'll have an agent answering there as well. Those are the combination of things that we look at. We invested a lot in onboarding and documentation. The truth is not everyone reads that. We attack it in different ways.

Siddharth Kodwani: We invested a good amount of time in coming up with the right skills because ultimately a lot of even the non-tech users are using all these agents, and if you have that right repo and right skills and right places, agent can quickly figure out what really needs to happen here. You don't have to understand everything. Your agent in most scenarios will figure out if you have the right skills available at the right places. That's where we're collaborating with the agent skills team. Where like, if agent control plane is the right way to call any tool today or agents, then how can you make it so easy for anybody to just do it? The idea is to just have the right skills in the right places and let the agents figure it out for you.

Swaroop Chitlur: In the end, we have to think about three audiences here. One is users, like humans, users on our platform, agents acting on behalf of human users, and agents as services. We think of these as three buckets of audience and we design for all three as we're building systems.

Participant 5: With reference to the open weight models, at what point in your cost journey did it start to make sense to get away from OpenAI or Anthropic and start to do some of the self-hosting and smaller models?

Swaroop Chitlur: For most teams, we always ask them to get started with the frontier proprietary models. One is they're easier to get started with. There's more information out there. There are actual people you can talk to if you don't know how to proceed. I think it's the three things: accuracy, latency, and cost. I would argue that those frontier commercial models are still the best when it comes to accuracy. It either boils down to latency or cost. If teams come to us and say, I have a very specific latency requirement, what do I do? Are products working great? We have product market fit, whether it's automation or whether it's external facing, but cost is a concern. I think that's when we drive the conversation towards, have you tried open weights models? Because you can shrink towards smaller models, you can think about things like distillation or other such fine-tuning space techniques to get to smaller models to achieve that latency that you need, or you can go towards a smaller model, for the cost aspect as well. Not everything needs the biggest model out there. Most teams don't realize that unless you walk them through the process. I think those are the two places where we usually encourage folks to start looking at open weights models. We don't encourage every team to start there unless they're familiar with the domain.

Participant 6: Can you elaborate a little bit more about your evals workflow? I see that you have a platform and this is great. From our experience, when it's easy to create evals, it's also super easy to get noise. It's really hard to get signal. How do you help your product teams to use evals the right way so they get signal and not noise? Specifically, I'm interested in CI, do you integrate it with CI? Do you have some frameworks?

Swaroop Chitlur: How do you get product teams helpful with evals? We're still in early stages. It's not a solved problem. Where we help teams get started, I think the first friction is tracing. We try to make that easy. Some of the work involved is like having agent templates. If you have the right tracing, like for example, if a team wants to get started, how do I build an agent? We have Google ADK. We have examples of how to hook tracing into the OTel store that we have. We first make tracing easy. Once that is unblocked, that's like half the battle. Because now you have those traces coming in, you can view them, you can understand where the time is being spent. People get a feel for like, how is the system actually operating? How's the agent actually operating? Is the bottleneck the LLM? Is the bottleneck the tool?

Is the bottleneck the results of the tool? Is it the web search is slow? Is it that the tool is slow? Once you are able to visualize what that observability tree looks like, they are more aware of what's happening. That's when you start questioning them like, what are the product metrics you are interested in? You cannot optimize on everything. You got to pick your battles. Like what are the two things or three things you care about as a product? Then, how do you operationalize that into actual evals or evaluators? We walk them through that journey. Again, this is a bit of handholding at this point, but hopefully over time we will learn enough to make this a playbook or self-serve. Once you are able to operationalize it to evaluator, sometimes you might need LLM-as-a-judge. Sometimes you can get away with just writing some code. You start building those evaluators, they are outputting metrics.

Now you need to visualize those metrics. You need dashboarding for that, custom dashboards. We have gone to the extent where product teams can vibe code their own UIs, and we will host those UIs for them on top of our platform. Including annotation workflows. Like if, for example, the agent makes a recommendation on these are the items we recommend for you at this restaurant. How do you know it's a good recommendation? Before you would have had Google Sheets where it dumps the entire menu on a Google Sheet, and you're like, you need a human to look at that cell and say whether this was a good recommendation or not. That's one of the bottlenecks we had, which is like, I can't visualize this. We enabled human annotation UIs, custom annotation UIs. This actually shows up as a proper menu. Like you can see the image, you can see the tag, and you have a custom UI with your custom questions on how to annotate this.

We have to take this step by step, get them started with tracing, get them started with what are your product metrics, set up the evals for it, set up the human annotations for it, dashboards for it. Once they are able to walk through this journey, it all clicks and you know how to do your product iterations. You're slowly hill climbing your way on accuracy.

See more presentations with transcripts

Managing Asynchronous APIs at Scale

Ian Cooper — Thu, 01 Oct 2026 16:00:00 GMT

Summary

Ian Cooper discusses managing asynchronous APIs in event-driven architectures at scale. He explains the "ABCs" of messaging (Address, Binding, Contract) and shares how to tackle discovery, governance, and provisioning using AsyncAPI, CloudEvents, schema registries, and automated CI/CD infrastructure pipelines based on real-world engineering practices at Just Eat Takeaway.

Bio

Ian Cooper is a polyglot coding architect with 30+ years of experience across finance, healthcare, and e-commerce at companies like Just Eat Takeaway and Reuters. An expert in OO, DDD, TDD, and event-driven architecture, he is a frequent international speaker and owner of the open-source project Brighter. He leads workshops on messaging and reactive architectures, with deep roots in C# and C++.

About the conference

Transcript

Ian Cooper: What we're going to talk about is the somewhat thorny topic of if you've moved to event-driven architectures, how do you manage all the APIs that you're now generating? That's an about me slide. The top bit says I'm old. The middle bit is just a list of standard stuff, and there's various places that I've worked over time. I like the point at the bottom which is the trouble with this kind of relationship is it suggests there's some expert learner model going on. I don't think I'm any smarter than you. I just had the fortunate opportunity in my career to get involved in speaking on stage and get confident at it. Let's just pretend that we're all capable of doing this. It is. You are all capable of speaking on stage. I'd recommend doing it. I work on an open-source messaging framework for .NET developers called Brighter. Feel free to check us out. We're not going to not be open source. We have a slightly weird arrangement now whereby you retain the IP and just license us in perpetuity. We can't do a rug pull, just because a few of those have happened recently.

Definition of Terminology

Let's talk about the three pillars, the three things that you're really going to need to think about when you're managing asynchronous APIs. First of all, just a couple of definitions to make sure we're on the same page. We'll talk a bit about endpoints. What are endpoints? Endpoints are places where messages are sent or received. Generally, what we want to do is define for an endpoint what are we going to exchange there. A better way of thinking about that is where messages should be sent, how they should be sent, and what the message should look like. Diagrammatically you can think about it this way. If I have a service called restaurant management that has an endpoint, and at that endpoint I am giving you a message that the restaurant hours have changed that you can listen to. Then, effectively, I need to have a message, data I send or receive.

About that message I'm going to need to agree a couple of things. I'm going to need to agree what metadata I have. In other words, what is in the headers of the message? Because unless you know what metadata I'm putting in the message, you won't have much chance of being able to deal with that. For example, I might have a country code telling you what country the store is in. You may be filtering in order to decide which of those messages to deal with. You need to understand what headers I'm going to publish. You're going to need to understand what I've put in the body. What format is it in? Is it just JSON? Is it text? Is it Avro? Is it Protobuf? What do I put in? Because you're not going to be able to process it unless you know that as well. The channel is the name we tend to give to the logical idea of a virtual pipe over which we're sending messages.

Reality may be different in different pieces of middleware. It might be basically a topic in Kafka or a routing key in RabbitMQ. Whatever the particular preference is, a channel is just a way of saying that. The binding we think about as actually if this channel is an abstract idea, what are the specific ideas? In other words, what is the protocol? Where does the server live? In principle, you can have multiple bindings for a channel. In principle, you could expose the same channel on both Kafka and, say, SNS. There's nothing to stop you in theory doing that. It would be very confusing for most people. In theory, that is possible. There's a really straightforward way of remembering this. This diagram, as you can see, has now got very busy. Let's have a straightforward way. That was originally created by Clemens Vasters for WCF, which was a .NET framework back, this was probably about 15 years ago, called the ABCs.

The address, the channel, the logical pipe over which our messages will flow. The binding, protocol-specific details, such as transports and encoding. The contract, so the message that is the headers, and the data, the payload of what we're sending. When you think about endpoints, what you need to describe to other people to use them are the ABCs. What is the address? What is the binding? What is the contract?

If you are a small org, let's say up to about five teams, and you've got an event-driven architecture, and you need to communicate information, it's relatively straightforward. Typically, the surface area is not big enough that you don't know about each other's services. You probably know what messages they publish. You have a good view of what's happening end-to-end. If you don't, you can just at least be able to walk over to somebody's desk and ask them. Now you probably can hop into their Teams room and Slack them, or just get them on whatever form of video conference works for you. Typically, we all understand how the message flow for our particular application works at that scale. We may have whiteboarded it out. We may have got Miro if we're more virtual and Miro'd it out. That's a company that emerged in the pandemic. If we needed infrastructure for that messaging, typically, we can resolve that ourselves.

Either we have some TicketOps process, we raise a Jira ticket. Probably our platform team is small anyway, and they can go and provision the infrastructure we need. Or even more commonly, we'll just rely on the fact that our framework will probably provision the necessary infrastructure for us. A lot of messaging frameworks will let you just say, we'll provision the infrastructure and we'll go away and create the topics that you need. That tends to work ok at that scale. Any choreography that we need where effectively we've got messages flowing between services, relatively easy because the number of participants is small. The trouble is it begins to break down at scale. First of all, you've got no idea if endpoints even exist at scale. Even if you do understand there are endpoints, it may be very unclear who owns them. We may really have no idea what the contracts are for the messages that are being published.

We may not even know if we do publish them who actually consumes them. A typical problem we encounter at scale is that a team who believes these two other teams consume my event, and they're the people I talk to when I make changes. Forget that, for example, there's a whole load of people listening to it in the data side who are using it for analysis. We only really discover that when they change the message schema having agreed with the two teams, not told data, and suddenly data is calling a production incident. It's very typical that we find that people don't know enough about who consumes their messages and people don't know enough about who sends their messages. If we build our infrastructure from some Jira tickets or from self-provisioning from our frameworks, what do we actually do if we have an incident where that goes missing?

Typically, some fat-fingered problem. Most commonly I've seen it be well-intentioned attempts to reduce cost by removing these things that surely nobody is using anymore. How do we actually recover from that? Because the problem may be that if you begin to look at it in enough detail, you'd have to actually just redeploy pretty much everything to figure out somebody must have owned this particular endpoint, and their self-provisioning is what we need to run to actually get everything working. There's no effective way to recover your business in the event effectively that it goes down other than redeploying the world.

The Three Pillars

There are three things we need to solve in order to conquer most of these problems at scale. The first thing we need to do is answer the question, how are we going to discover our endpoints? How are we going to discover our asynchronous event estate, and be able to find where are our endpoints? What messages? What are the contracts? We're going to need governance. We're going to need to make sure that it's possible that we govern how we change and evolve our schemas so that we essentially don't break other parts of our business. We're going to need to think about how we provision. How do we provision all of these topics that we are going to use such that we can effectively recover or understand where we've drifted?

Discovery

Let's take this one by one. First of all, discovery. Before we began to look at this more closely a few years back, the problem we'd have is the most typical way to find a message or an endpoint was to go into GitHub and search across all of the enterprise repos, and try and find an implementation of that message or an implementation of that endpoint, and try and guess what it might be called. That becomes a problem as soon as you need a keyword like restaurant, because you're going to get a lot of hits. That's not particularly working very well. The other option we tried was plaintive messages in Slack. The people posting into Slack, going, "Does anyone know who publishes the restaurant hours changed event? Anyone know which team owns that?" Then someone would be like, "I think Tracy knows who publishes that. We were discussing it the other day." It's this long chain of discovery.

It doesn't sound very, we're on the top of our system here. If you're lucky, there may be a wiki page in something like Confluent. Probably someone updated it about 18 months ago. The last time somebody actually cared enough to go, we should really document all of our messages and endpoints in the team. He's probably moved on. Maybe we promoted him because he was the person that actually did that. Now we're left with a very outdated wiki page that we don't trust anymore. If I'm a producer of message, how am I going to find who consumes stuff? One of the ones that typically comes up for us is someone has ill-advisedly decided to use a decimal for money or an integer for an ID. The number of people that regret making their IDs an integer in the messages. Again, typically, can I ask in Slack, does anyone consume my message?

Can I search in GitHub saying, I wonder if I can find all the users of my message? It turns out really that you've got the problem that nobody in data actually is in the same channels as you. They're not looking for your message for their process that is reading your events to do some analytical report that the CEO looks at every morning.

How do you evolve events in that case? There is prior art here. That prior art is that we solved this problem quite some time ago for our synchronous APIs with things like OpenAPI, GraphQL, even gRPC has its own docs. Why do we not historically embrace the importance of this ABC, address, binding, and contract for asynchronous APIs? The answer came around, and people said, that's what we need to do. That is where AsyncAPI came from. AsyncAPI is part of the Linux Foundation. Fran Mendez was one of the people that started it back in the day. Their goal was literally that. It was say, there's OpenAPI, can't we create a version of that that works the messaging? We're currently, I think, on version 3.1 of AsyncAPI. If you go and look at earlier versions, you can very clearly see the OpenAPI heritage. If you're used to working with OpenAPI, much of the structure of how it works will seem very familiar to you.

It is very accessible for people who've got that kind of workflow. Typically, with AsyncAPI, what do we capture? We capture the server. Server is typically a broker. AsyncAPI also supports WebSockets, so we can't use the term broker. You can think about that as being the broker in most cases. The channel. Remember we were talking about channels earlier, the virtual pipe over which messages flow. It has a notion of a channel. Channels live on a server. Each channel has messages that pass over that channel. Some of our channels will be what we call data type channels. In other words, we restrict them to a single message. That's generally, in my experience, easy to maintain. Some of our channels are going to have multiple messages. That's particularly often common where people are trying to use some strategy of having events that represent change or a delta rather than events that represent a snapshot.

You can have multiple messages against the one channel in the docs. Then the way you do it is for any given application, so anything that sends or receives messages, we declare operations. An operation says we are going to send or receive a message from a channel. Then we can add to that, bindings, where the bindings are protocol-specific information. The binding might be that this is Kafka. There are also information, spoilers for later, like a number of partitions, for example, that that Kafka topic may have.

I'll give you a view of the structure. I'm going to show you an actual example of two or three of our event schemas in AsyncAPI at the end, because it makes it easy just to cut from the slides. I can show you a lot of our stuff at the end. This is a structure. Those of you who are fans of OpenAPI may well be familiar with the idea. We essentially have the Root Object. Then we have an Info Object, which has metadata for this particular application. The most useful things here are things like version. Who is the contact point? What team owns this particular endpoint? Servers Objects, so how do you connect to the server? Very typically, you may have multiple servers here, because you may have QA, staging, and production environments, for example. You can list all of those. The Channels Object. Essentially, what channels are we currently hosting on that server right now?

Channels can be a reference to all these things in the same way OpenAPI does. Rather than having to inline channels on consumers, receivers, and keep repeating it everywhere, you can use the reference syntax, just point it towards a larger document, which effectively has all these reuse definitions along with messages. The channels being used by the object, that will also contain the messages that we intend to send over the channels. An Operation Object, which essentially says, what are we doing? Because we're documenting with one of these, a given application. What do we do to this channel? A Components Object, that is essentially this repository where we can centrally put message schemas, channel definitions, and can live in a separate file. Then, tags. In case we have some tagging schema, a system that we want to use. What we don't show here is there are extensions. They can become quite important to you, you'll discover when we're doing later on. Binding information, which can be added to any of these objects where appropriate. That is the structure of AsyncAPI. Again, I'll show you an actual document at the end. You don't need it right now.

As well as AsyncAPI, obviously, when you represent any one of these messages that you're sending over, AsyncAPI supports a number of formats you can document the message in. It supports JSON schema. That's its default. That is essentially using JSON schema to describe the message. It supports Avro because Kafka, and more importantly, Kafka's ecosystem tends to rely on you having Avro definitions for things. Protobuf, which tends to be the least commonly used in my experience, but places that effectively have a lot of investment in gRPC, already, but tend to quite often use protobuf. What else do we need to worry about? The other thing we might need to worry about is the metadata. AsyncAPI, in the same way it lets you choose your schema format for documenting messages, also doesn't really say anything about what's your header format going to be. You get to choose it.

Typically, people nowadays will tend to use CloudEvents, which is standardized metadata. It has a number of required attributes, so the ID, the source, the spec version, and the type. The source and the type are the most useful things there. Essentially, really pretty much who produces it and what is the name of this event, and which version of the schema are we talking about. CloudEvents also comes with some optional metadata, which can be quite useful. The data content type, in other words, how am I actually representing the body? What is the schema? There's a URI, which tells me I can retrieve schema information. Subject, which qualifies the source. If you have a number of things that the source needs to tell you about, you can qualify it with more information. The time, so what time it was sent. Typically, we tend to use subject for a number of people in this source might have produced this.

I need to give you some more information which tells you which part of the source produced this message. There are various extensions. For those of you, for example, that like the Claim-Check pattern where you say I'm going to offload the body of my content into S3 storage because it's too big. I'll send you the message. You should retrieve it on the far side. The support, for example, for doing Claim-Checks for CloudEvents. The name comes from the fact that when they wrote CloudEvents, they intended it to be used mainly for eventing scenarios, in other words, some kind of Pub/Sub. Not for messaging scenarios, in other words, some kind of command-response. Typically, our experience would be people just use CloudEvents as a standard set of headers for everything.

One of the things to be aware with CloudEvents is that it's useful for metadata so that we have consistent metadata. Routing is quite common via CloudEvents. A number of messaging frameworks will look at the event type that you've sent it and decide where to route that in an application. That's not uncommon. When you have multiple message types on the same channel, what I need to be able to do is to decide how to deserialize that and route it to a handler. That is tricky if the only way I can do that is by inspecting the actual body and trying to figure that out. A typical way we do that is to use CloudEvents to go and look at the type of the event. From the type of the event, we can then determine how to deserialize it and what to route it to to handle the message.

That's a pretty common approach to these multi-event topics. Effectively, it gives us the ability to do things, that self-describing model. CloudEvents supports two ways of dealing with your existing transport called binary and structured. The first thing you should note is that binary is not binary and structured is not very structured. It's a bit like the joke about the Holy Roman Empire. The three things you should know about it is it wasn't holy, it wasn't Roman, and it wasn't an empire. CloudEvents is a bit similar. What binary actually means is the protocol that I am dealing with has native support for headers. Structured means it doesn't, so I'm going to put it all in the body. My CloudEvents headers, I use an envelope format. I have the headers, and then I have the body, and that lets me put headers in there. Actually, today, there are not many protocols that don't have headers.

You're not forced into structure for that reason. Some of them do have a limited number of headers. The canonical example here would be SNS on the AWS platform where you get about 10 message attributes. If you start adding CloudEvents in, you rapidly consume all of your message attributes. If you wanted to use any of them internally on top of that, we use them, for example, for country information, that kind of thing for the message, you rapidly run out of attributes. It's quite typical in that model. It's become very common in the AWS world to use the structured binding for the CloudEvents headers. I can reserve the headers in the attributes for bespoke ones that I need instead. Typically, ones where I want to do inspection of the headers for routing, rather than need them at the destination. If I use structured, I have to get hold of the body.

That's ok at the destination. I just want it for event routing, and that's fine. If I effectively want to do it on, say, an SNS server and create a filter on my SNS server that will filter given events, I need to use message attributes. Actually, that may help me to reserve those for custom and bespoke things that I need.

Let us imagine you go away, and I hope you will, and you say, great, we're going to create AsyncAPI for all of our messaging endpoints. Go you. You say, we're going to have CloudEvents then for how we're going to describe our messages, and we're going to make sure we have very defined schemas. You may say to me, Ian, we've still not solved the discovery problem. This is on the left-hand side. On the left-hand side, this is basically a screenshot from me in VS Code — I'll show you the thing later — looking at our repo that tends to hold most of our message schemas, our AsyncAPI, our message schemas. I'm still back in the same problem. I've now just got hundreds of files here. We have, I think, 600 to 700 of these things. I've got an agreed format, but I've not really necessarily improved my life very much.

Because now I'm just doing a text search locally. Maybe that's good for agents. They can do local text searches. Has it really helped me do discovery? Typically, on top of this, you'll want some additional tooling. One of the poster children for this is EventCatalog written by a lovely guy, David Boyne. EventCatalog, you can load your schemas and your AsyncAPI information into. It effectively gives you a tool around that catalog that lets you visualize the flow of messages in your system, statically rather than dynamically. Very useful. This picture's not so great, but I'll actually show you this one later. Marmot is an open-source catalog. That's a really useful one if you're budget constrained. We use that one, and I'll give you a little demo at the end. Marmot does a similar job. You just load up your AsyncAPI, your schemas. Marmot is also pitched as a data catalog tool, so it will do lineage, for you as well, which can be very useful to you.

Clemens Vasters here of CloudEvents fame, has also been working recently on a thing called xRegistry. xRegistry says, this is all great, but where's the standard for how we register all of these specifications and schemas in a way that is non-proprietary. It's really another section of the talk, but it includes the idea of, I need to register stuff with Confluent Schema Registry. That's, again, a proprietary format. xRegistry is part of the CNCF, and he's been working on. It's intended to be an open format for registering all this information. Left-hand side, you can see there's this core spec where you get resources, which are a bit of a nebulous concept in that version, which are put in groups in a registry. You can see there are extensions, when you look over on the right-hand side, for endpoints. We're just talking about them. Message definitions, schemas. Nearly all of these support AsyncAPI CloudEvents directly.

Typically, though, they have some REST API to actually upload your content. They can either take in content in that form and spit it back out again. They tend to also have their own proprietary format, in the case of xRegistry, its own, Marmot. It's non-proprietary, but it's tied to that open-source product rather than them using the external schema. All of them work with those standards. Really, xRegistry is very on the minute, lo-fi. You can get a server off the xRegistry project, you can just run an API. There's not really any beautiful UIs and tooling around it. You have to build that yourself. Though, nowadays, with agents, how hard is that? Marmot comes with visualization. The EventCatalog is probably the poster child of visualization, if you have the budget.

AsyncAPI, effectively, that's our OpenAPI equivalent for documenting our endpoints. It's where we can say, what is the address? In other words, what are the servers? What are the contracts? What messages flow over the channels on those servers? What are the bindings? What are the protocols that we need to support? CloudEvents, which says, our messages, as well as a schema in the body, they may have metadata. Maybe we should have some broad agreement about how metadata looks so we can be interoperable. A registry of some sort where having got all this information together, I want to move beyond searching through a repo of them and have some way of just doing discovery against all these APIs and schemas that I've created. You'll want a registry. Together, you can answer the question, what events exist? Who produces them? Who consumes them? Or, what do they look like?

Governance

Next of our pillars was governance. How do we ensure schemas are consistent, compatible, and evolving safely? Typically, when we think about services talking to each other, we tend to prefer to have some open host service. It's the Domain-Driven Design term for, we have an API that provides an external contract which differs from potentially the representation we have inside. That is the point of coupling. It's necessary coupling because otherwise we can't communicate. It's typically data coupling. It might be stamp if we basically depend upon fields that we don't use. That's the best form of coupling we can really get away with. That tends to be all right. It's a contract. I say I want to be able to vary my bounded context inside. My implementation details may change because I may think of better ways of solving this problem. Or I may have to acquire new responsibilities which don't necessarily impact you but may result in me changing stuff.

In order to protect you from that internal change, I want to create a contract which I will honor with you for whatever period of time we agree is reasonable where I won't change that. Your service can depend on me safe in the knowledge that I have a contract with you which I will keep honoring and I will definitely tell you if I need to change that contract. It's a necessity in a world where we may have services owned by different teams that effectively we have some notion of a contract between them to ensure there's a reliable service offering. It's this whole old idea of good fences make good neighbors. That contract is something we're going to agree. That makes the coupling which must exist habitable for both of us. We have a number of ways we can think about that coupling. We can be backwards compatible.

That means that a new version of the consumer can read old data. If we say, the consumer's expecting to receive some additional fields. My typical example here is we've suddenly decided to give location data and latitude and longitude. I launch my consumer that can effectively have those fields. I can just default them out to null island until the producer starts producing those new messages. That means we upgrade the consumers first. When the consumers are all upgraded, we start sending the new message. Forwards compatible. Forwards compatible means that I'm going to send you a message. The producer needs to add some data to it. It's all right, the consumers have been written away. That means that they will ignore additional fields that they don't understand. It's forwards compatibility. There's full compatibility, which is we are both backwards and forwards. I can add messages that you don't understand. You can ignore them effectively, and you can default things that might be missing. None, in other words, I don't like any of you. I don't want friends anymore.

Typically, we have a set of rules about how we evolve schemas. Optional fields are safe. Removing fields, you have to deprecate. When all the consumers are deprecated, you can then remove. Renaming is effectively a remove then add. Change types is almost always a breaking change. We've done change types by effectively treating it almost like rename, which is, add a field with a new type. Change everyone over to consuming a new field with a new type. Essentially remove the old type. Occasionally then do a rename operation on the one you wanted to change the type on. Very painful. How are we going to make sure that everybody essentially is going to keep their schemas, their contracts correct. It's a genuine problem. When I first arrived at Just Eat Takeaway, it was the first problem they dumped on my desk was people keep breaking each other. We try really hard at this, but we still have production incidents caused by somebody has changed the schema and they have not told the other people.

Quite often what happens is the problem is to do with people just not thinking about all of the consumers that might possibly consume from them. They've had a limited conversation. They assume everyone is in the loop and they find out that they're not. Schema is often just a piece of documentation. Even if I've loaded it into EventCatalog, that does not mean that I'm protecting anybody. It just means I'm telling them what it is. I could change it. That doesn't help anybody downstream that they can now see the new documentation for the schema when I've just broken them. In the HTTP world, typically people might solve this problem with something like PACT and consumer-driven contracts, but it's really unsatisfactory in the event world. The reason it's a bit unsatisfactory in the event world is those tools don't work very well because there's no single protocol. There's no HTTP which we can then say, I'll stand up an HTTP server and I will use that to do testing. You'd effectively have to fake out every messaging protocol that there is.

More typically, we use a schema registry. A schema registry says, send me all your schemas. Then, to prevent breakage, what you do is you say, when a producer sends a message, I am going to not let them send it if it doesn't match the schema in the registry. The schema in the registry, when it's updated, will enforce compatibility rules. If I say this schema must be forwards or backwards or full compatible, the schema registry will say, you can't make that change if it would be a breaking change for everybody else. The schema says, I will enforce the contract and I will enforce the ways the contract is allowed to vary. Basically, no breaking change. The producer can only produce a message which matches the schema. Therefore, in order to produce a new message, you must have uploaded the schema. We must have agreed that it doesn't break the consumers by effectively, it passes the rules that we've set.

Now you can produce to it. You can also, and we do some of this, you can say, I will do this as part of my CI/CD pipeline. I will effectively say, I will attempt to register your new schema and check that it passes the rules. Then we can run in a non-production environment. We can then check whether or not we fail to break the producer. Typically, we don't want it to fail in production. We'd like it to fail in staging or earlier on, so that you get a decent warning that you have attempted to load something that's going to break. This may come as a surprise to some teams when it happens because they haven't thought about the change that they want to make and how it might actually impact consumers. We are finding this more and more as we begin to leverage things like Kafka for real-time data, particularly being consumed by agents to provide interactivity with customers, that we rely more and more on those pipelines not breaking because somebody introduced a schema change that broke something five steps down the chain.

We've often got Iceberg tables that depend upon reading schema formats. More and more, it has become really important to enforce rules at the schema. You can automate this. If you are creating AsyncAPI, your AsyncAPI is going to have a schema for the message. What you can do is you can say, I could upload that to my schema registry, validate its compatibility, and essentially then potentially only decide to deploy it if effectively it turns out that I'm not breaking the schema. Typically, we may have some schemas in something like component schemas or external refs, and then we just read them out of there. I'll show you a more comprehensive discussion on what JIT does in a little bit.

Postel's Law is what builds the internet, be conservative in what you send, and liberal in what you accept. While the producers should always check, many of them have the option for the consumer to check, and say, when I read the message, check that the message I'm reading conforms to the schema. I am cautious about doing that. If my consumer can interpret that message, it probably should. Even if the message, for some reason, does not conform to the schema. It can help you out in some PIs that effectively the consumer will honor that message, even if technically it's incorrect, because you can push something through if you need to. However, as soon as you start getting into the world of consumers that do things like create Iceberg tables from Flink, you're going to need to enforce the consumer as well. Standardized schema format. Schema review is part of the process. Schema registry integrated into our CI process, and runtime enforcement for producers.

Provisioning

Provisioning. As we said, a lot of people tend to manually provision using tickets, or they get their framework to do it. We typically see drift. No one's quite sure what production looks like, it isn't actually any more according to what we may have in our documents. There's a lot of complexity sometimes about how we name things in production, like topics, particularly if we have multiple environments, particularly if we have whole loads of teams all dealing with things to do with restaurants. How do we disambiguate stuff? It's generally not easy to figure out what is the source of truth. How do we say what production looks like? Disaster recovery, as said earlier, is basically deploy and hope. What we've abandoned is this model, which says, let's raise some Jira ticket, or also the model would say, let's just get the framework to deploy it for us.

We actually do it from the specification. We take that AsyncAPI document. We take the binding information. We also add, because it's extensible, like OpenAPIs, some additional metadata of our own. Essentially, we say, take all that information. We — as I show you, it's a tiny bit complicated — pull it all together and generate what our messaging infrastructure looks like, and then deploy that. I'm talking more generally here. I will talk more specifically in a second.

AsyncAPI has a generator model. You can take that generator, it reads an AsyncAPI spec, and you can write React code that will say, I want to parse this document, and spit something else out. Originally it was built a lot for saying, spitting out HTML versions of documentation. A lot of folks have begun to use it to say, what I want to do is generate output that I'm going to use to do my provisioning, to do my schema registration and the like. You can use that to do infrastructure provisioning. What you do is you read the AsyncAPI spec. You need to extract the binding and the object. For example, I've got a channel. That channel is going to have some information, for example, saying this is Kafka, and I want 50 partitions of this topic. I want to be replicated across three servers. Then we can provision that information using your weapon of choice.

We use Pulumi. We think it's slightly easier for this particular problem. You can use Terraform, whatever it is you prefer. We can also then have this running in the background. When there is drift, it simply recreates the missing parts of the stack. There are specific bindings for a number of things. For my synths I started it, but mostly the team that I work with does it now internally. We contribute to a few of them to produce this binding information because it needs to be more detailed. Not all of the things that AsyncAPI supports are as detailed because we don't use them. You've got to do document generation right. Publish the catalog. Effectively, also what we can do is publish to Marmot or the like, or publish to EventCatalog as part of that whole process. Quick summary of provisioning. Spec-driven, code generation, auto-registration, DR is tractable, and visibility architecture.

AsyncAPI at Just East Takeaway

What do we do? We have about 603 specs, about 2,000 messaging schemas. We have about three AWS regions. We actually have two platforms. We have about 22,000 commits a year and into a repo that manages most of this information. Our pipeline is we take the AsyncAPI YAML. It gets fed not to the generator, actually to a C# application called Mextrapolater. When we started this, generally, devs were being seconded to do this work. Now, as you'll see later, we tend to write stuff more in Go for DevOps. We tend to use C# still for application code. That generates JSON artifacts. What we do is we take all of these specs and we figure out, this channel is used across all of these specs. That's one channel. In other words, one topic, "Here are all the subscribers that I can see for that." Then it figures out stuff like, here are the IAM rules I'm going to need in AWS.

Here are the ACLs I'm going to need in Kafka. We do a whole lot of processing to figure out what does that actually mean or look like in terms of a change that we want to make to production. We then take that, which is a whole lot of JSON, and we zip it up and put it on S3. Then other people can then get it from there and use it to do stuff. We've basically got a Go CLI called mex deploy, mex stands for message exchange, which will take that artifact when it appears and will then work through AWS Confluent in order to deploy it. Register a schema in the schema registry, and it will pull credentials that it needs out of Vault for any secrets that are necessary to run that process. Up on the top, you can also see we talk to Marmot.

We basically say, take this artifact. To Marmot, we're going to give you our information so that you can give us a data catalog that people can browse. We do schema validation in the pipeline. When we get the schema out via the Templater, we say, ok, what is your schema? Let us try and register that with, we use Confluent Schema Registry, given the compatibility that's been agreed. The only real writer to that schema registry is mex. Mex essentially has a token that says, I'm going to change it. It goes and changes it. Then it hands back the token, and devs can't get the token. The only way they can change schemas in the schema registry is by giving us an AsyncAPI application that contains those schemas. We know we have all the bits and pieces that we actually need in order to make this work. Then you can go and look in our data catalog in Marmot and see what is going on.

This is actually a bit of a Marmot catalog. That's showing you tracking created. You can see it's telling us all about that particular event. Then you can go up here, see a whole load of stuff about the various assets that we have. That's Marmot, and it's pretty easy to find stuff and browse inside it. The other thing I can show you just briefly here is, this is a typical AsyncAPI spec with a whole lot of metadata. The reason I want to show you this is because once you get metadata and stuff in, they're a bit more complicated examples, but you can see there's metadata, the channels. You can see the channel has a name. What happens at runtime is we will add stuff to that name to make it unique and disambiguate it. You can see we've got a message. You can see the message declares a payload.

There are references because it goes to a components file, because it's quite commonly shared. There's binding information, which means here we're looking at specific protocols. This is custom binding for us. What environments are we deploying it to? Then it's basically saying this is actually an SNS message. You'll see somewhere else down here, we've probably got Kafka. There's a Kafka binding here that you can see. That's the picture.

The Virtuous Cycle

What's still hard? There's no notion of being able to support consumer-driven contracts. There's just no PACT equivalent. There's no way of seeing when you document this stuff, I've got endpoints to my application that come in, endpoints that go out. What's the relationship between the two? We have no way of defining that so you can't see flow. There are some works by folks who make OpenAPI, and I think it's on a flow-based representation for OpenAPI and they may extend it to AsyncAPI. There's no semantic compatibility here. Just because I have schema doesn't mean that the intent behind those things is actually tied together. What's hard? Adoption. Teams tend to play game theory. Everybody else should document their stuff, I don't need to. Quite often we have to simply say you cannot provision anything apart from via this process because that's the only way to force you to do it. This doesn't do anything with observability, but there were talks about how to do OTel and messaging. Things like xRegistry exist, but it's not clear where they're going as opposed to the more proprietary things like EventCatalog or open-source alternatives like Marmot. They're not all pulling in the same direction.

Takeaways

Treat your async APIs the same way you do your sync APIs, and document them. Specs are the source of truth, and you can use them for code gen, schema registration, infra provisioning. Governance as a pipeline gate and not a wiki page. Schema registries with compatibility modes make evolution safe and auditable. What it gives you is visibility into your event-driven estate, leverage to enforce standards, and control over your infrastructure.

See more presentations with transcripts