Presentations > Page #1 - InfoQ

Keeping ChatGPT Fast as AI Development Accelerates

Martin Spier — Fri, 07 Aug 2026 16:00:00 GMT

Summary

Martin Spier explains how agentic workflows dramatically increase code change volume at OpenAI. He discusses the hidden systemic performance costs of rapid shipping beyond GPUs, and shares how deploying always-on AI agents automates profiling, regression detection, and continuous optimization to maintain product speed and scalability at massive global scale.

Bio

Martin Spier leads ChatGPT Performance at OpenAI, where he works on making ChatGPT faster, more reliable, and easier for engineering teams to operate at scale. His work spans AI and cloud infrastructure, performance engineering, observability, reliability, platform engineering, and developer productivity.

About the conference

QCon AI is a practitioner-led event focused entirely on the engineering discipline required to scale these workloads safely. It provides direct access to the architectural playbooks and failure metrics that peer organizations use in production.

Transcript

Martin Spier: Today I want to tell you guys a story about two accelerations that are happening somewhat at the same time. First one is around growth, how we're actually growing user base faster than ever. Not just OpenAI, ChatGPT, but in general companies are reaching their first few million users a lot faster than before. The second one is how our development workflows are changing given agentic coding and all those things, and how that is changing the rate of shipping of things we're getting out of the door, and how that is actually affecting how we keep our products fast, efficient, so overall performance engineering.

My name is Martin. I lead the ChatGPT performance team at OpenAI. Almost my whole technical career was in performance engineering. I think I was doing that for 16 years or so, or more. The bulk of that time was at Netflix, where I spent, I think, almost 9 years. Dio mentioned the early cloud days, that's around when I joined, when people thought, will I actually leave my data in the cloud? That's like magical thing there. It feels like the moment is similar today. I spent some time at Snowflake, Expedia, and a few other companies doing that as well. Prior to OpenAI, I was with another AI company called Parasail. We did inference as a service. I was leading the whole engineering team there. Prior to that, I was at a fintech out of Brazil called PicPay, where I was leading whole infrastructure platform and developer experience. As you can imagine, my whole career is really in these central organizations. It's an area I really like. I call it the plumbing of the house. No one sees that it's there, but once it blocks and it breaks, all hell breaks loose. It's always a fun part of the company to be at.

Growth Pace

Three things changed. The first one is growth pace. Not just OpenAI, like I mentioned. Companies are reaching their first million, first 10 million, first 100 million, maybe more users a lot faster than before. The curves are exponential, but a lot faster than before. Also, we're shipping more code and more things out. We're getting more things out of the door with agentic development. There's probably some talk here around AI slop and everything that's going on. We are shipping more logic out the door. The third thing is, previously, we always had this assumption that there was a human that understood all the changes that were going out. It's like someone actually coded that and architected it and know what's going on before they decide to push something out. That is not entirely true anymore, at least from my point of view today. I know there might be a slight bubble, but the abstraction layer is a bit higher.

People are delegating more things to the agents, so they don't know the details of everything that might be going out the door. It's something we need to keep in mind. Of course, with all of that, it puts pressure not just on performance. Performance is one of those problems, but everything else that goes around it. Anyone from GitHub, GitLab here, or anything like that? It's putting a lot of pressure on that infrastructure, but performance too. We're just using more and more resources. We're adding logic that adds latency. It introduces a lot of challenges, and we need to keep up. We need to evolve our practices so we can keep up with that fast pace of growth and the fast pace of development.

Going into ChatGPT, a bit of the history, just to give you a bit of context about the scale I'm talking about. ChatGPT was not launched in late 2022, as a full-fledged consumer-based application that was built from scratch to support hundreds of millions of users. It was launched as a research preview. Here it is. It's pretty cool. You guys should try it out. Quickly, users started coming. Once you have those users coming and coming pretty fast, that research preview becomes a product, and product has requirements. You have latency requirements. You have reliability requirements. You have support requirements and everything that goes behind the scenes to make that work for a large volume of users, especially if they're relying on that for their day-to-day on the things that they need to do. The growth was quite impressive. I think very few companies can tell that they reached their first million users in five days from launch in a research preview.

This is pretty cool. We have a million users here. You can imagine how behind the scenes the team supporting the infrastructure were at those times. Incredible growth, incredible for the product, a lot of pressure on everyone behind the scenes. That growth, it did not stop. It did not slow down much. ChatGPT continued to grow and it's still growing pretty fast. The last official number I can share is from February this year, I believe, when we reached 900 million weekly active users. That's 900 million people using ChatGPT every week. Just to let it sink in, that's almost 11% of the whole human population. It's a lot of people. As you can imagine, chat, it's not the lightest thing to do compute-wise. There's a lot that is necessary on the back to make that happen. That growth was not smooth. It was not just adding a few million users every day and keeps getting pretty predictable.

It was full of spikes, as you can imagine. We were launching products all the time. Some of those products became quite viral. Huge spikes on users, and that, of course, drove adoption. Just this one, you might remember the image generation spike from last year. Maybe you generated one of those images that I cannot say the name right now. Within just the seven first days, over 700 million images were generated by over 130 million users. Image generation, pretty quick. That was putting a lot of pressure on the teams behind the scenes, scrambling. This is the thing that's quite hard to predict when something like that becomes hugely popular. Again, it's something we want. Everyone wants their product to grow. This is really great.

Chat is not used only in the U.S. It's used all across the globe. With that, if you support infrastructure, architecture, all those things, you know that adds complexity to things. Like, where are my POPs? Where are my actual application servers? In our case, where are our GPUs? Are they closely together? Do we have enough capacity in all those regions? Or, how is the network connectivity between all those different data centers? The architecture, where is data? How is data being replicated? Where is data located, given all the different constraints you have across the globe? It adds a lot of complexity to the architecture to support a good experience, or to be able to do and provide a really good experience to all our users across the globe. It was not just growth, but the product did not stand still either. It wasn't just a launch or just users joining.

There was a lot of change beyond all just the model changes. The models were extremely important, but there were a lot of launches along those four years, less than four years now. There were GPTs that were voice. There were agents, all different things, image generation, a lot of product launches, some a lot more viral than others. The product was changing constantly. A lot more things, different workloads that had to be supported, more complex architectures behind it to support all of that. The product was changing while it was growing really fast as well.

Change Velocity

This is, I wouldn't call it new, but it definitely accelerated more since late last year. Especially since the launch of Codex, our shipping rate changed dramatically. I'm not discussing value or anything like that. That's a way deeper discussion, but it's just the volume of change entering our systems increased a lot. When everyone thinks about agentic development, the first thing that comes to mind is automating the proper coding tasks, like develop this function or that application, or refactor this, refactor that, and reduce the overall time it takes to get that done. Shorten the time to get from an idea to a PR that gets merged to production. The part that really surprised me is not just that, especially with newer models where the agents are a lot more capable, they can take more complex tasks. We can delegate things that will run for a lot longer.

Developers started going multi-threaded. They started actually not working on a single thing at a time. Developers are working with maybe 7, 8, 9, 10 different things at the same time. They keep on blocking them as things go. I don't know how much you're coding these days with agents. You're probably hearing the pings saying, I need the user input right now. They have multiple development boxes, or maybe you're working with org trees. In general, a lot more parallel things going on. It's not just faster to ship, but shipping more things in parallel.

Some of you might be familiar with this benchmark from DX, where basically they study a few companies, got some numbers about the volume of PRs that get shipped, that merge per engineer per week. Then you split that into different size of companies and the type of company. As you can expect, smaller tech companies are shipping more PRs, they're getting more things out. At p90, you're almost at five PRs a week per engineer, which is quite fast. We are a lot faster than that. We're shipping a lot more these days. Unfortunately, the latest number I can share is from late last year, so October, November. If you've been following, you know there was quite a huge development since late last year in coding agents. Even back then, back in October '25, the volume of PRs that our engineers are shipping every week increased by 70%. Almost every engineer at the company today is using Codex on a weekly or probably a daily basis.

Every single PR gets automatically reviewed by Codex. It's not just the coding tasks. Like I mentioned, Codex became the workbench to do everything you need to do during that whole development workflow. It's not just coding, but I found a bug in production, go troubleshoot it. Or, I want to understand a metric, a regression, go talk with the observability tools and fetch the metrics and explain to you what's going on. Or to do data science and data analysis, I use Codex to do that as well. All the other boring things like, please summarize that Slack thread for me because I don't want to read it all. A lot of productivity tasks that just make your day-to-day a lot easier. Not just engineers. I manage a team. I probably don't spend too much time coding these days anymore, but I use Codex for everything. To prepare my one-on-ones, to summarize and write documents. Even this presentation, I started it on Codex to go over the outline and to discuss what I actually wanted to present and how to slice that, and all the images that got generated, all the visuals in this presentation. It's being used for everything on your day-to-day, not just your pure coding tasks.

With that evolving of the software engineering workflow, we're shipping things faster. It takes less time for things to get out. We're working on more things in parallel. We're delegating more. That agentic development increased throughput and parallelism. That increased the volume of changes going into our system. I'm not saying value, feature, I think just changes, things that are going out. It might be a feature, might be a bug fix, might be a refactor, might be something else, might be a config thing, but changes going out. Shipping all those things, every new feature, every new line of code, it has a hidden cost. Ideally, that hidden cost adds value. It's something you want. It's something you want out. It's a new feature, a bug fix, but it has a cost somewhere. The cost might be tiny initially. It might be invisible. You might not notice it in the first place.

That increased rate of change just exacerbates that. It's nothing new, but it's happening a bit faster. All that change going out, all those new extra if conditional statements, every extra network request you're making, every data structure that you decide to hold in memory, it's consuming from a shared budget. Your latency, it can get so bad to a certain extent until your users decide to stop. If you're adding logic, you're doing more things, you're adding more checks, whatever you're doing, that always adds a little bit of latency just because you're doing a bit more. It just expands to the point where, this experience is not good, and I don't want to use your product anymore. Same goes for hardware resources. You only have a finite amount of memory. You only have a finite amount of RAM. Everyone will raise, cloud can expand forever. No, it cannot.

It's not as elastic as you think. It just makes things a bit less efficient. It's consuming from a shared budget. I'm not talking purely about those big regressions, like things that you clearly see from one deploy before and after that something drastic changed. I'm talking about even small things. Like I said, that small extra if statement that you added to check something that is important. Small things that on a database, they don't make a difference. There's no difference at all when you're comparing before and after. All of that compounds. You keep building up a little bit, little bit, little bit, until it becomes a bigger problem. It becomes a concern. You might be running out of headroom. It might be making your latencies too bad for users to have a pretty good experience in the app. It just compounds. With that increased rate of change, it's just compounding a lot faster.

We're shortening that time window until we have to actually start worrying about those things. That damage from all those small changes and sometimes big regressions, it will show up later. For the users, it will show up in slowness in the app itself, as you can imagine. For efficiency, no, you will be paying more to run exactly the same workload. You add more logs, you do more things, you still have the same workload. You're just paying more for it. For scalability, you have less and less headroom to grow. As you guys probably can imagine, a scalability problem will become a reliability problem at some point when you cannot absorb that spike anymore, when you cannot grow anymore. The damage will appear later. You need to address it as fast as possible.

Performance Engineering

This is nothing new. We've been developing features and adding features and adding logic our whole careers. In one side, you have people shipping features and shipping things. On the other side, you might have a performance engineering team, or you might have a few perf-minded individuals that care about efficiency and care about latency and making things as efficient as possible, balancing that. You have two opposing forces and keeping things in balance. The problem is that with that increased rate of change, one side of the equation got a lot faster. We're shipping more and perf engineering or folks working in performance problems, they need to adapt. We need to keep up. The answer to that is, of course, we don't want to hire more performance engineers. It's hard to find perf engineers in the first place. Again, not the best answer. We need to make our workflows faster as well.

We need to speed up the cycle on the perf engineering side too. We need to speed up our reactive flows, our reactive loops, where we detect things a lot faster. We can profile that and we can get to the root cause of the problem faster. Not just that. Also, have the agents understand the codebase, propose a solution, implement that thing, and get it out and get benchmarked and get tested. When we actually get a notification, "Great, I already have a fix for that." The benchmarks are looking good again. We can continue. Also, on the active side, we can have a lot of agents in parallel looking for possible optimizations. We can have a lot of different skills working on different types of optimizations. You can be looking at the latency hot path. You can be looking at what methods are actually taking most CPU and try to optimize those. You can be looking at allocations and continuously improve that, or bundle size. They all specialize looking at different things, but working nonstop trying to find those optimizations. You don't have to do that in serial.

We need to adapt. We need to move a lot faster. What happens if we don't adapt as fast as we need? Of course, there'll be a lot of chaos behind the scenes, but beyond that, what's the first few things your users will feel? First things are generally slowness. You add logic, things get slow until it becomes a problem. Your users stop coming. They stop using the app. They cancel their plans, or even worse. Maybe got bad to a point where you're just throwing more errors. You're down for longer periods of time. Things take longer to get fixed, but your users are suffering. Especially for a large consumer-based application like ChatGPT, and it might be the case for where you guys work, you don't want that. That has a deep effect in the business bottom line. Performance affects user retention and acquisition. I modeled that at Netflix.

I modeled that at OpenAI. I'm pretty sure if you guys try to model that as well in your companies, you'll see the correlation between things getting slow and you're throwing more errors and users just canceling plans. Direct impact in the business. One thing I've noticed too, especially for AI applications, is everyone is very focused on inference. Inference is the hard part. Inference is the resources we need to watch all the time. They're really focusing on inference components, like your time to first token or your tokens per second, your throughput, which are important. Your component metrics are super important. Inference is a big chunk of the problem, but it's not the whole journey. It's not the whole problem. We need to measure and keep watching what the users actually feel. When the users come to your product, they're generally trying to solve a problem. They're trying to do something, perform an action, and during that process, while they're trying to perform that action, they have different feelings.

They have different expectations. We need to be able to capture that, and then we need to drill down into. Take, for example, this. This is a bit more focused on a chat app, but the idea is the same. I used the same thing at Netflix. I think Chrome uses the same thing as well. The users have different expectations along the action journey. When he types in a message and clicks enter or clicks submit, he generally expects something to come back really quickly, instant, just feedback to that action, just to know that I'm not hanging here or anything. I got a feedback. I might get a spinner. I might get a thinking thing going on there, but something is going on. It's not on me anymore. The product itself, it's working. I just need to wait a little bit. Then comes something I call first visible value.

When can the user continue doing what he wanted to do in the first place? In our case, if it's a chat request, the user wants to start reading that answer. The time it takes to get the first few tokens, and the user can start reading that. You don't want the users to stop reading because the throughput, the stream cadence is slow. You need to make sure that the stream cadence is pretty good so the user can continue reading without getting blocked and frustrated again. Then, the time it takes to finish that message, the time it takes to complete the task itself. Even that has a lot of nuance. Where do you stop the timer specifically? For a chat response, do you stop at the first token? Do you stop at the first few words? When can a user actually extract some value? It depends on the product.

When you have a thinking response, where do you stop the timer? Do you stop at the thinking tokens, or do you actually stop when the user can do the actual final message? In an image, do you stop the timer when the final image is generated, or the first low-resolution image is generated? There's a lot of nuances, and there's a lot of thinking you need to do around product to actually have those numbers right. Having those intents is super important as a dimension as well because the expectations, they also vary depending on what the user is trying to do. He might be asking a very simple question like, what's the capital of France? I can go to a search engine and get a response pretty quickly, so my expectation is that the response should be pretty quick as well. Great. I need to respond faster on those cases.

I actually fired up an agentic loop that needs to go and check a lot of sources. It needs to do some code edits. It needs to do a lot of things, so I expect that to take a bit longer. The expectation for that is longer as well. Image generation is heavier, the same thing. Users have different expectations.

I have all those different timers, those different wall times, different intents, a lot of different dimensions. Now to be able to act on that, then I go and decompose that into system evidence, into specific components I want to track, things that compose that large wall time. I like to use this chart. There's a lot of different options. My team likes this as well. Your latency breakdown chart or layer cake where you have your distribution on the x-axis and actual latency on the y-axis, and the different layers depending on what's going on. The chart is not the important part here, but there's a lot that happens beyond inference. Of course, this is not to scale numbers or anything. It's just to demonstrate. Yes, there is client work. There is networking time. There is time to fetch data, to do serialization, to do tokenization. There's a lot that goes on beyond just inference.

That breakdown into layers, it helps us actually focus our efforts. Also, the intent I mentioned helps us focus our efforts. If we see that the large chunk of time is not inference itself, it's maybe data fetching, we can focus on that and try to improve that. Having visibility the whole path, it's super important. Otherwise, you just default to inference, which is the first thing that comes to mind to everyone. Inference is just part of the problem, as you can imagine. Take, for example, our chat application. Chat is very simple. The interface is simple on purpose. You go, you type your message. You send that message. You wait for a bit, you get the response back. I don't know if many of you developed this sort of system before, but we can't just forward that message to the inference engine. It does not keep that state of your whole conversation and everything that it needs to properly respond to that.

There's a lot that happens before a message gets sent to the inference engine. There's client work on, of course, assembling everything that needs to be assembled at the client side, fixing things there. Once things come back, we need to verify the user identity. We need to verify the client identity. We need to check the user's plan. We need to check quotas. We need to check the actual user state. There's a lot that needs to happen. Then comes a fairly heavy part of it, which is, we need to fetch context. Conversations, they can get pretty long. We need to fetch all that conversation history. The user might have uploaded files like PDFs, images, whatever, to that conversation. We need to fetch those files as well. If you are in a project, we need to fetch that project context as well. A lot of data that needs to be fetched until we start assembling what actually gets sent to the inference engine.

Great. I have all the data. I have everything I need. There's encoding. There's tokenization, and of course every module is like different tokenization. Everything is tokenized. We need to look at the context window, like, is it exceeding the context window the user has? Maybe we need to do truncation. Maybe we need to do some compaction. Then we need to assemble everything to be sent to the inference engine. There's a lot that goes on even before something gets to the inference engine. Then there's a lot that goes on after, we're streaming the message back, because we need to store the conversation somewhere as well later too. The message, as you can expect, what the user sent, that may be a small string, a small question, is just one small ingredient of that whole equation. Especially for long conversations, they are pretty data heavy. You might have conversations that are megabytes long, tens of megabytes long, maybe hundreds of megabytes long.

There's a lot of requests. There's a lot of data moving around our systems. That data, as you can imagine, it consumes all those resources I mentioned before. We are, of course, consuming a lot of database requests, Blob storage, there's I/O, moving things around. There's a lot of CPU serializing, deserializing things. There is a lot of RAM, because you probably need to hold things in memory for a little bit as well. It consumes all those traditional resources that are not GPU. A message I want to give here is just, when people think about AI, the first thing that comes to mind is GPUs. We need GPUs. GPUs are really important. Don't get me wrong, it's super important. Everything else, all your CPU workloads, they are extremely important as well. There's a lot of resources that are taken to make the whole experience work the way it works, so we need to watch out for those things.

Bringing this up to you, because, as you can imagine, there are code changes, there are improvements, there are shipping rate changes to the actual inference engine, but the rest of the product is a lot larger, as there's a lot more components. There's a lot more code change going to the other components that affect CPU and memory, I/O, all those things, sometimes even more than GPU, so bringing that up. Why is that important? It doesn't matter that I have an extremely fast model, the fastest model, the fastest inference engine, if the rest of the path is slow. The user will get a slow product at the end of the day anyway, and will decide not to use it. We need to focus on the whole experience, the whole path of the request, to make sure things are moving faster. The users get that pretty good feeling when they're using the product.

Human in the Loop

Summing things up a little bit, the product path is getting broader, and it's more than inference. It includes all that data, context, tokenization, streaming. That path is getting heavier and heavier with all the agentic coding, and all the changes we're making, and that's just accelerating. We're consuming more resources, and those resources are not just GPU. Keeping in mind, we need to evolve the way we think about perf engineering. Again, lots of changes need to speed things up on our side as well. We, of course, don't want to ask our engineers across the company to slow down or add roadblocks that make things slower. Every AI PR that is open needs to be reviewed by five humans, so we don't want to add things that will slow down, that will just take away the whole advantage we have of using AI agents. We actually need to speed up the different loops that we use in perf engineering, both the reactive loop when something regresses and we need to fix.

We need that to be faster. We need that to be closer to the actual code, or where the problem originated in the first place, and the active loop itself that needs to be a lot more parallel. Usually, our reactive loop, it takes quite a bit of time. Are there any perf engineers around here that did profiling and optimizing things in the past? The loop is quite slow. You find a regression, maybe from one release to another, you see that your CPU regressed a few percent. What do you do next? You need to do some profiling. You need to go and capture your CPU profiling, which stacks are actually in CPU the most time. Hopefully, you have continuous profiling so you can actually compare a before and after and easily spot where things regressed. If you don't, then things become even a bit more complex. You need to look at the code changes, try to map that to the stacks that are running CPU, and see what regressed so we find an optimization.

Then you need to think of optimizations, and you need to implement those optimizations. Then you need to go and ship it, test it, and see if things fix. All of that is serial, being done by one person, and it just takes a bit too long. The path I'm exploring within a team and the direction I'm seeing things go is having things always on, having our agents always on, working non-stop in both loops, both reactive and also the active loop. We want to automate not only the large regressions, so things you can spot easily from one release to another, where generally you can have an agent triggered automatically by that regression. The agent could go and fire a profile. The agent could actually compare two profiles. We have that today. We have performance skills on Codex to do that. Pretty straightforward if you have all those things working in your systems.

The agent has access to the codebase. He can compare before and after what PRs got into that code change. He can go and propose a fix, and hopefully your systems are safe enough to the point where maybe the agent can deploy that and test if that fix actually had the expected effect. Same goes for drift. Small regressions, we can have fine-grained metrics of maybe code complexity, maybe the volume of network requests that gets executed during a benchmark. We can have small measures to find that drift and act on that drift as well. Of course, the active optimization loop that I mentioned before. The idea here, this is more of an end scenario, that's where we want things to go to. There's a lot of intermediary steps on that. We want everything that happens pre-production, so ideally before a merge, or before something gets deployed, all the boring things be caught by a CI.

Pretty straightforward. We've been doing that for a long time. Nothing new there. As you can imagine, there's a lot of things, especially on the perf side, that don't manifest until they get to production, until they get a specific combination of user workloads. There's no way of catching that pre-production in the first place, so we need to catch that and we need to fix that once we see. Again, active profiling.

A few examples. We can have microbenchmarks running all the time on every single PR. Agents can actually help develop those microbenchmarks. We can find a lot of different things. We can find though that performance, that method regression. We can actually monitor all the methods that are part of our latency critical path, and find specific regressions on those. We can profile memory allocations, and we can see if any specific function is allocating more memory, and we can act upon that. We can watch bundle sizes. Especially if you publish into mobile, you can watch bundle size if these things are increasing and if that exceeds your budget or not. The message here is we've probably been doing that for a long time, but we can automatically act upon it. We can go deeper. We can look at the metrics. We can look at the codebase. We can check all our architecture diagrams and all the information we have about the system running, and the agent can propose a few ideas, a few ways of fixing that.

We can go and implement that. Ideally, you can actually benchmark, run the same benchmarks again, and compare before and after. Yes, many times that fix will not work. It will not fix the problem, and you can just throw that away and continue doing. You can fire multiple threads with multiple ideas. That's the point. Of course, this is end state. Ideally, there are a lot of steps to get to that in the first place. You can actually have a set of skills in Codex that just make that process faster. Instead of manually triggering profiles, collecting things, and analyzing, you can have an agent collect that information and do that comparison for you. It's something that can be easily achieved. The agents are pretty good at chugging a lot of data and comparing things. Pretty easy, you can generate a flame graph out of it if you want.

Fairly straightforward, especially with microbenchmarks where you can reproduce things pretty easily. Same goes for production. You want pretty good observability in production. You want to catch all those different regressions that you see. Again, act upon those. Same thing, you get the same information. The difference here is the safety. Things are a bit trickier if you're asking an agent to deploy something to production and you don't want your users to feel any issues, feel any pain. There's a lot of best practice that you need implemented even before you could do that. You can get almost all the way there. You can actually have an agent submit a PR that implements an experiment of a perf optimization. You can do that. If you're not at a stage where you can fully trust an agent to deploy to production without any supervision, you can get almost all the way there, and it should work really well.

The always-on loop is the one that probably excites me the most. In my team we have a lot of really great perf engineers, and we're in the process of how can I actually translate all that skill set into a set of skills to my agents that can continuously be looking for optimizations in my environment. They could be looking at the code. They could be looking at my traces. They could be looking at logs, whatever, but continuously finding optimizations. It's funny too, the habit people have when they have a very specific niche skill set, they name the agents after themselves. I have a Ben agent. I have a Brendan agent. I have lots of agents named as people. Then you start treating them as people at some point too. It's weird, that always-on loop trying to find opportunities.

Proven Best Practices

A few best practices of things that have been working well for us. When you can get agents to complete full loops, things become a lot more autonomous. You can have a lot of agents doing a lot of things at the same time, if they can complete the loop. Meaning, I start the loop from some event or a continuous loop, I can have an idea. The agent can implement that, and it can do the whole thing all the way through actually measuring if that change you made was positive and it worked. We can continue doing that without any human intervention. Sometimes it's not that easy, and at other times it's a bit easier. If you can focus on trying to build all the safety to allow agents to complete full loops without any intervention, things will become a lot faster. A lot less supervision. A lot less bottleneck on people having to monitor.

Then just have people not clicking accept, accept, accept, or just do YOLO mode and let the agents do their thing. As you can imagine too, the basic things we've been doing for a long time, they become more and more important. Your engineering maturity becomes more important. Having really good test coverage, it's important. Otherwise, how would the agents know what they did was right or it's working? It's functionally working or not? Benchmarks, the same way. How can the agents measure if the improvement they implemented was good or not? Having those benchmarks, having those tests, extremely important, became even more important now. You need your contracts expectations between the different parts of the system. Observability, super important. You need really good coverage about your system so the agent can understand what's going on. If there's blind spots, it's very likely that it might go in the wrong direction because it doesn't have that intuition about the system that the humans might have. That safe rollout and maybe an automated canary analysis or a blue-green push, all those safety measures to get the code out even if maybe there's a bug there, but that shouldn't affect users. Roll out in a safe way.

At the end of the day, all those perf optimizations, that's a search problem. The agents can try the different alternatives. They can try the benchmarks. They can compare things. It's really cool because they can do all of that a lot faster, and they can do all those things in parallel. The agents, they do need a clear signal, again, to know if they're moving in the right direction, if what they did is right or wrong, or if they can discard some work or actually use it. They need all those metrics. If you're actually trying to reproduce a production workload, you need actual reliable workloads to be reproduced by the engine itself. Otherwise, you might be optimizing something that doesn't really match production. Same problem you had before as you are working with perf. Fast feedback is important too. The faster the feedback loop, the more the agent can do without waiting.

If your feedback loop might be 12 hours, then maybe your agent doesn't really add that much value because the human interruption is not that much. If you can have a fast feedback, within a few minutes I can know if I'm in the right direction so I can make a decision and move, in one way or another, things become a lot faster. That measurement you have, it just guides the agent. Having the right measurements is super important. If I'm measuring the wrong thing, the agent will optimize the wrong thing and I'm not actually improving the product in the first place. Defining the metric correctly, making sure you're capturing that. That's why I focus a lot in what the user is perceiving during the whole process. Focus on that. Agents, like I mentioned, they can chug through a lot more information than humans. Of course, you need to manage the context window and what you're actually sending to be useful.

You can send a lot of information, all your observability, all the details about your deploys, all your codebase. If you have actual written down definitions of the architecture of how the system works, it's extremely important. It's all context that the agent uses to define things. Different than humans, you don't have tribal knowledge here. You might have tribal knowledge and memories, but, in general, you want to provide that context so agents can do the right thing. General idea, we are working towards a world where humans are just setting the direction and our agents are just doing all the work, at least the boring work that is not making the decisions and defining what is good or bad.

Key Insights

It's not just growth, growth is increasing, of course, but the development flow changed significantly. We're shipping and making changes a lot faster than we were before. There's arguments around if it's good or bad. We are making a lot more changes than we were before, and that has implications in other areas of development too. Not just performance, but all your developer experience, whoever is supporting your CI engines or your code repositories. There's a lot of impact in different areas to support that rate of change. When you're thinking about AI, the first thing that comes to mind is GPUs. It's not just GPUs. There's a lot of optimizations, a lot of things that happen even before anything gets sent to be processed by a GPU for inference. Last but not least, our perf workflow needs to change. All the different workflows that are affected by that different rate of change, that increased rate of change, it needs to evolve to keep up. Otherwise, our balance will be wrong.

See more presentations with transcripts

Rewriting All of Spotify's Code Base, All the Time

Jo Kelly-Fenton/Aleksandar Mitic — Thu, 06 Aug 2026 16:00:00 GMT

Summary

Jo Kelly-Fenton and Aleksandar Mitic explain how Spotify created "Honk," an AI coding agent, to handle complex fleet-wide codebase migrations. They share key architectural insights on decoupling CI verification runtimes from AI agents, dealing with automated pull request bottlenecks, and driving aggressive standardization across thousands of engineering repositories.

Bio

Jo Kelly-Fenton is an engineer at Spotify, working on the development and company-wide adoption of autonomous coding agents. Aleksandar Mitic has been working as a Software Engineer at Spotify for 4 years. His tenure at Spotify has been spent within Platform teams.

About the conference

Software is changing the world. QCon London empowers software development by facilitating the spread of knowledge and innovation in the developer community. A practitioner-driven conference, QCon is designed for technical team leads, architects, engineering directors, and project managers who influence innovation in their teams.

Transcript

Aleksandar Mitic: I'm Alex, this is Jo. This is our talk, "Rewriting All of Spotify's Codebase All the Time". We're here to tell you about our background coding agent that helps us do exactly this. Everyone meet Honk. Honk helps us rewrite all of our code. Honk is already much more famous than I'll ever be. It's been mentioned in the billboard, papers. We'll tell you the story about how this started, what we learned developing it, and where we are now.

Solving The Maintenance Problem

We'll rewind time back slightly more than a year ago, beginning of last year. This journey started with us trying to solve what we call the maintenance problem. I'll explain in more detail what this is, but before that, a statement. Developers code for more than one hour per day. Do you think this is true? Developers on average code for more than one hour per day. I'm sorry to say you're all wrong. We see in data from the industry, but also internally, that on average, developers spend less than one hour writing code. This number might be slightly different now. With all the tools we have, it's gotten even more fun for people to code. The point still stands, you don't spend your full workday writing code. You go to meetings, and you do all the other things you have to do. If you've done software for a while, you know you can't just make features.

You have to maintain your codebase. In practice, that one hour is full of bumping dependencies, migrating to the latest Java version. Your platform org is pushing a new logging framework that you need to migrate to, and this is basically an endless list of tasks. This never stops. You have to do this. Otherwise, your software won't work after some time. This is what we call the maintenance problem. This is what we try to solve. This is nothing new. This has been here since the dawn of software. At Spotify, we've been thinking about this for a very long time. We have this thing called fleet management which is many things, but, mostly, it's a way of thinking. It's about us bringing our fleet with us as we make changes. If I am the owner of a library, it is my responsibility to make sure that all of Spotify gets to use the latest and greatest version of this library.

What does this mean in practice? Let's say I'm the owner of this chat library, and I've released a version 3. It's my job to make sure everyone is on version 3. How do I do that? I write a little script that, given a codebase, checks if it's using the chat library and bumps the version. What our fleet management system allows us to do is to specify this migration first by saying what we are targeting. This is a Java library. We're targeting all Java components at Spotify, which is thousands of codebases. Then we specify the transformation we want to run. In our case, this is our script. What our fleet management system does is it picks up this configuration. For each target, it starts a Kubernetes job. Each job clones the repository, runs the transformation, and opens a pull request. This pull request is then sent to the code owners for review and to merge.

In some cases, where we can fully reason about the change, or there's some way to test it, we can even auto-merge, so no one has to see this happen. This works quite well. This graph shows you how many days it takes for 70% of our fleet to adopt the latest version of our internal service framework, and you see how this changes over time. Before we had fleet management, it would take us almost a year. If we release version 3, we need almost a year for 70% of Spotify to use it. With fleet management in place, we're down to just under a week. This is all before LLMs. This is amazing and super helpful. When I look at this graph, I wonder, why not 100%? Isn't fleet management about all of Spotify's codebase? Yes, of course it is. In practice, that last 30% is quite hard.

That's because the long tail is complex. When we run our script on that 30%, what happens is we see an error. It turns out, in version 3, we removed one method that we didn't know was used by some parts of our codebase, and our script, it can't handle this. This is fixable. We add some code to replace post with send message, that new function. In practice, it's not this simple. We don't want to replace every post, just the ones from our library. Our script starts looking like this. We're parsing the abstract syntax tree, and it's getting more complicated. At this point, there's only one person that truly understands how this works, but it works, so we run it, and then this happens. Turns out, we made some performance optimizations in version 3, and those don't click well with some part of that 30%. What do we do now? As platform teams, we say, this is fine. Let's keep both methods. We migrated most of it, we proclaim success. Migration done.

The Birth of Honk (AI Migration Tool)

Now we have a problem. Now we have two methods, and we have a diverse codebase which brings a diverse set of problems. Every time we interact with the chat library, either with a new migration or building features on top of it, we need to keep both methods in mind. It's at this time where we see LLMs start getting good at writing code. They've been writing code for a while, but at this point, we're getting that aha moment. We start thinking, what if we replace our script with an LLM? They're more versatile. They can handle these edge cases. At this point, Honk is born. That's a simple idea, let's just replace the script with an LLM. I wish it was that simple. In practice, I want you to remember this is more than a year ago. This is before Claude Code was a thing. This was when we were doing tab completion.

A lot of writing code with LLMs was using a chat UI, prompting something, getting code back, running that. Maybe you get an error, shove the error back to the chat UI, and you go back and forth. This works, and it's cool, but there's no way we migrate the hundreds of chat library use cases doing this. Thinking about this, what is it we're actually doing here? This is the normal software development process. Given a set of requirements, we write some code. We build. We test. We iterate. It just happens to be so that the code is generated by an LLM, but this process needs to happen. We realize that to get Honk to rewrite all of our codebase, we somehow need to package this nicely.

Jo Kelly Fenton: This is where Honk starts to come into existence. Sure, we have the LLMs that can write the code, but the building and testing is where it gets particularly tricky. Claude is pretty good at verifying your code now, but the tools that we had at the time did not do this. When you think about running Honk on a single repository, it's pretty easy to build and test. You know exactly what to run. When you start running it on an entire fleet, that becomes infinitely more complex, because we use different build systems for different repositories, so just the tools that we use are different, the commands that we run are different. How do we generalize this so that we can run it on everything? Our first thing we have to do is to create this verify tool, that the LLM can call on any codebase and it will verify the code by building and testing it.

This is a single entry point that fans out to many verifiers under the hood. We have a Maven verifier, a Yarn verifier, a Bazel verifier. We even allow people to write their own verification scripts. Like, maybe you're running a migration and you have some particular logic that you want to verify, doesn't or does happen, so you can write your script and you can plug it into the verify tool. Then you can put it in a loop with the LLM and it starts getting this good feedback. This is the very basic genesis of Honk. We make it a CLI. We start to run our little chat library migration, maybe on one or two repos, and then we start to unveil more and more of the problems. The first problem is this. This is the most disgusting thing you're going to see all day. This is a Maven build output and this is what we see all the time as developers.

It's a lot of text and a lot of it is very unimportant. If we fed this failure back into the LLM directly, it would just fall over. Really, all that we care about is this tiny little thing over here. The chat library was not correctly migrated. Now we have to worry about actually extracting the exceptions from the build output.

We try being smart. We wrote a lot of scripts to try and do this. For Maven, it's simple because there's some standardization in the build outputs, but a lot of build systems have very unstandard outputs, so it gets very complicated very quickly. At some point we just went back to basics. We just put it in an LLM. We asked the LLM to summarize it for us. This actually worked really well, because this is fundamentally what LLMs are really good at. They're very good at summarizing text. We haven't really looked back from this. At this point, we can run Honk. It can summarize the failures, pass it back to the agent, operate like this in a circle, and produce some code, which is good. This is looking pretty good. We start to see this. We realize that the thing that Honk is getting good at is not necessarily running a migration, but it's getting the build to pass.

This is the easiest thing to get the build to pass. Agents these days, they're getting smarter. You see less and less of this, but we saw a lot of this to start with. It's a little cheeky. We also see things like this. As the agents got smarter, it would still do things like this. This is it downgrading a Java version in order to get something to work, because that was just the easiest way to get the build to pass. What we did to try to tackle this problem is we introduced an LLM-as-a-judge. It was pretty simple. It would take the initial prompt, the generated code from the LLM, and it would put it into another LLM to then evaluate whether the code addresses the prompt. It would come back with a verdict, a pass or fail. If it failed, it would block the migration from completing until the agent fixed the issue.

For example, the judge would say removing a test was not part of the original requirements, and then the agent would be forced to then uncomment the test and actually fix the underlying issue, which is pretty good. Then we started to see some issues with our judge, which is that, take this example, not all repos that we're migrating used the post method. Sometimes the generated code does not include a migration from post to send message, but the judge is like, the post method wasn't migrated to send message, and so it blocks the migration from continuing, but the LLM did the right thing. Our judge was simple, but in a bad way. As we found the models getting better and better, we saw less and less need for an LLM-as-a-judge. If you just put verification steps within your prompt, the LLMs are getting really good at doing that themselves. We ended up actually removing the LLM-as-a-judge eventually, and we haven't really looked back. It's not that they're inherently bad. There's probably a place for them. We haven't really found the need for them. The code and the tests are really good at evaluating the correctness.

Fleet Management Tooling

At this point though, we've migrated our tests. This is our generated code. We test our Honk prompt on a couple of repos locally, and then we decide, we're going to roll this out with our fleet management tooling. This is where we hit arguably the biggest problem, which is that when we roll it out across hundreds of repositories, we start seeing hundreds of failures. We start seeing missing permissions, failed to load Docker. Then we start to dig into these failures, and we realize that these are actually nothing to do with the generated code, but they're very much to do with the infrastructure around the code. The integration tests of all of these repos, they require specific permissions to run. They require Docker to be loaded. We realized that previously we were running Honk on our laptop, and now we're running it on a Linux VM, which is inherently very different.

Our laptop has all of our dev permissions set up. It runs a CLI directly. Now we're running in a container that has a service account. Sure, we could give our service account permissions to run the integration tests, but that is a lot of permissions that we would have to grant in order to get the integration tests to run for all of our services. We could look into supporting Docker in Docker, but our infra is not set up to do this very nicely. What do we do? We skip the integration tests, and now Honk can continue with its migration, because it's not blocked on these infra errors. That's not great, obviously. The integration tests are very critical in validating the correctness of your code, but we can move forward with our migration, so it's good enough for now.

Then we hit an even bigger snag, which is that we can run iOS builds on our laptop, but we cannot run them on a Linux machine. We wanted to start making migrations in our app. It's a very important codebase. We basically couldn't validate any of our changes that we made to the app, and then this list kept going with these differences between these pods and the local developer experience. We had to take stock as a team. We couldn't keep skipping tests. We realized like, what is the thing that makes Honk really good? It's this automated verification. The more dimensions of correctness that we can automatically verify, the more correct the code will be. Instead of minimizing it, we need to focus on ways that we can maximize it. There's actually a system that's already purpose-built to run your build and to run your tests, and it's your CI system.

We know that works for all of your repos. We had this valuable lesson, which was like, we're trying to recreate our CI system on our Linux VM, which is our agent runtime, but actually, if we could separate out our verification runtime from our agent runtime, then we could leverage the pre-existing CI system that's already set up. I'll take you through a little bit of our architecture of how we do this. We still have our pod, and we have our agent harness running inside the pod. We still have our verify tool, so the LLM calls out to that. We still try to do what we can locally, because that's quick, it's cheap, it's efficient. At some point before we create a PR, we want to do a full end-to-end check that the code is actually correct. How we do that is we push a branch to GitHub, and then we kick off a build.

We created a service called a verification service, which abstracts several of our CI systems. We've got a few at Spotify. It starts the build and waits for it to be finished. Then when it's finished, it'll summarize any failures and return it back to the verify tool. If it's incorrect, the agent goes again, but if it's correct, it'll create a PR. What's really nice about this is that when you have hundreds of PRs being generated in parallel with a migration, you know that the PRs are going to have a correct CI build, because you've already validated it before you'd even created the PR. We don't want to create garbage PRs, and this is kind of this system that allows us to assert that they're going to be correct. This is kind of like, it's simple, but it's effective. At this point, Honk is a really good AI migrations tool.

Honk's Integration with Slack

Aleksandar Mitic: Now we have a great migration tool, and we see this in our metrics. This shows you the merged PRs. As platform people, we start using this immediately. All of a sudden, we're running migrations we just couldn't do before. Super exciting. What happens then is we have a Hack Week at Spotify. This is a week where everyone is allowed to explore and build whatever they want. Many of the features you use in the Spotify app came from a process like this. One engineer, he exposes Honk over Slack. Instead of using our fleet management migration tooling, you just talk to it on Slack and have it make an ad hoc change. As platform people, we're like, this is cool, back to migrations. Time goes on. We keep using Honk, but we start being curious about the Slack integration. Not just us, but basically every developer at Spotify.

Then this happens, we see the line sharply going up. At this point, there are more PRs happening with Honk, and we start thinking, what is it that people actually want? There is a need for migration tooling, but there is something more here. We realize that what people don't want to do is to leave the surface where work was planned in order to act on it. Let me give you an example. Let's say I'm debugging my Honk service which has a problem. What do I do? I ping my colleague on Slack. You go like, hey, Jo, did you see that Honk service has high error rates? Jo links our dashboard, and she sees that error started 20 minutes ago. I look at the logs. I find a suspicious stack trace. I post it here. This jogs Jo's memory, and she recognizes that a few months ago, we saw a similar error.

She had created a Jira ticket with a bunch more context on how to fix this error. At this point, in this Slack thread, we have everything we need to fix this problem. Why should I have to go somewhere else to fix this? If we have LLMs that can write good code, given all the context being there, I don't have to. I just ping Honk, fix this. I chat with Honk. It gives me a plan. We go back and forth. The plan looks good. I let it go ahead, and, after a moment, it gives me back a PR. At this point, we realize that what people need here is a background coding agent that allows us to go from prompt to PR from any surface.

If you look at this architecture Jo showed us, this is tailored for our migration's use case. What needs to change to enable work from any surface? We touched on a key point during that Slack thread where some of the context was written messages, but some was in a Jira ticket. Some of that is in a monitoring dashboard. The agent needs to be able to access all of this. The agent, when it works, needs to have access to all the sources where we have context. This might be logs, monitoring, Jira, whatever. Also, any surface truly means any surface, not just Slack. Maybe you, in a PR comment, want to trigger a task, or from Jira. We expose this architecture with an API. We build the integrations we think are most useful, but we also let all developers at Spotify integrate with it and build the workflows they think are necessary.

This is where we are now. The result of this is that when Jo and I presented a version of this talk, this was still a migration tool. We were so happy to announce that we had 1,000 merged PRs in 3 months. We were over the top. Today, we do this in 10 days, and this number keeps going up. What is the result of this? It's a bunch of PRs to review. This is becoming the new bottleneck.

The New Bottleneck - PR Review

I'm going to take you on a slight detour here. This is Lisanne Bainbridge, she's a cognitive psychologist. In the '80s, she wrote this paper titled, "The Ironies of Automation". I highly recommend you read it. It's four pages. It touches on what happens when you try to automate an industrial process like power plants or aviation. There's a bunch of good lessons here. One that sticks out to me, and this is slightly paraphrased, is that, when we automate the process, we tend to leave the hardest tasks for ourselves. This is because the hardest tasks are the hardest to automate. If we think about aviation as an example, airplanes can basically fly themselves, not fully, but a lot of it, yet we still need pilots. If we didn't have pilots, many of us wouldn't be here today. What do pilots do? They plan the route. They communicate. They might navigate smaller emergencies.

They fly some parts of the flight. At the end here, interestingly, they monitor the flight systems, and they say, this is safe, or this is not safe. In my view, that is the hardest job. Looking at all these measurements, all the observations, and being sure that this is going the right way. This is where we're getting now. We have all of this code to do, and the PR is only valuable once it gets merged and into production. We have a bunch of this work to do now, the hardest part, looking at a PR and saying this is safe to merge or not. This is the new bottleneck. I would be lying if I said we've solved all of this. We haven't. This is a tough problem. We're definitely thinking about it, and there are some things we're doing to make this better. One, this is quite simple, but it is important.

It is a culture shift. We've always had a review process, everyone has, but it is becoming increasingly more important. If I make a PR, I might ping Jo on Slack, and Jo reviews it, and that's it. Now we need to have some expectations on simple things like review timeframes. When can I expect my PR to get reviewed? If this is the new bottleneck, we also need to be structured about it. If there's a bunch of PRs that need to be reviewed, we need to understand which ones shouldn't be reviewed. Stale PRs get closed. We will have to try again. Interestingly, in some cases, a driver of migration can approve the PR themselves sometimes. Why is that? If I am the expert of the chat library, I know best if it's correct or not. Maybe I am the best person suited to review and say this is fine.

Of course, the receiver of the PR needs to see it and be aware that is happening, but I should decide if this is ok or not. Second, we need better tooling. We will all have to review a lot, so let's make sure we have good tooling. Again, some of this is simple and obvious. We need a PR inbox. Before this, I might get some PRs on Slack and some on mail, and some in GHE itself, but I need one place I can go that tells me which PRs I should prioritize, the one Jo sent me today, or the one PR from last week that I forgot to review. Maybe there's a PR I don't even know about because I haven't been assigned it, that I am the best suited to review. If someone is migrating to the chat library in a part of the company that I'm not, maybe I should review that.

Some tooling is more complex. We started this talk by saying that we auto-merge some PRs, even before LLMs. Now, with LLMs, we can write even more complex code and better code. Naturally, we should be able to auto-merge more PRs. The question is which? This is a hard question, and it's not fully answered. We're starting to think about this. Maybe there are some heuristics. Maybe documentation changes can be auto-merged. Maybe internal systems that, if they go down, we just revert, not the end of the world. There's something we need to figure out here, because, surely, there's more things we can offer.

The Biggest Tool - Standardization

Jo Kelly Fenton: This is arguably the biggest tool that we are leaning on. It sounds simple, standardization. If we think about why reviewing all of these PRs is actually hard, there's a lot of PRs, and that takes time. It's the complexity of each PR. Where does the complexity come from? We're back to this old chestnut. A diverse codebase brings a diverse set of problems. What was hurting us before LLMs is still hurting us today. Remember all of those migrations that we only completed to 70%? We introduced two ways to do the same thing. The one that's on the 70% and the one that's on the 30%. Our prompts don't actually look like this. They look like this. This is a lot of complexity here. Now when I review, there's a lot of edge cases, like, if this, do this. We have to think about a lot of different things when we craft a prompt that will run a migration across our services.

If we think about what is complicated about reviewing a PR with this much of a prompt is that we have to then consider all of the context around the generated code when we actually come to review it. We have to understand whether the agent actually parsed this correctly and executed it, which is really hard. If we go back to our chat library, we can take a look at what this looks like in practice and why our prompts are so long. Remember, we have to migrate from post to send message. Then send message had some performance changes. There was something that I didn't bring up, which was that version 3 of the chat library has a minimum Java 25 version requirement. This is where we start to come into contact with one of our unfinished migrations, the Java 25 migration. We did it to 80% of the fleet, but God knows what the other 20% is on.

They were really hard to migrate. We just had to move on to the next migration. Now when we migrate the chat library, we have to think about like, is this repository using Java 25 or not? If it's not, if it's using 11, we have to do this and that. If it's using 17, we have to do this and that, 21. Your prompt is basically filled with these if x, do this, if y, do that. That's how it ends up like this.

What do we do? We tackle this very explicitly. We want to get rid of this long tail. We want to get rid of the 20% of our codebase that looks and behaves differently to the rest of our codebase. If we do that, our code becomes extremely predictable. It becomes much easier to write our prompts, and also to review our code. This is very hard. It's so easy to say standardized. Spotify is about to be 20 years old. We got a lot of services. We got a lot of engineers. How on earth do we standardize, and why haven't we done it already if it's so easy? The thing is, is we finally believe that we're at this inflection point with our AI tooling that we actually have the tools to execute this now. What are the steps that we're taking? The first one, this is the hardest.

This is defining your standards, because you've never met a more passionate group of people than a bunch of engineers discussing the pros and cons of dependency injection frameworks. I'm pro. Alex is against. It's a massive conversation contention. If I post this on Slack, I come back after lunch and I have like 342 replies. Everyone at the company has an opinion. They got to voice their opinion. If you leave it up to the engineers to determine what the golden technology is, you will literally never get it done. It's just impossible. We take it out of the control of the engineers and we put it into these advisory boards that we set up. The advisory boards exist in all of our major disciplines, and they're comprised of five or six of the most senior engineers at the company. Hopefully, they can come up with some consensus. They choose the golden technology and then they publicize that to the company.

Next, we have to standardize our codebase based on these standards. This is where Honk comes back. We have a lot of new golden tech. We have a lot of codebase to migrate onto that golden tech. We can use tools like Honk to actually drive our migrations all the way to completion now, which is great. How do we pick which migrations to start with? Because there's a lot of them now. We pick migrations that remove this long tail. We pick migrations that make other migrations simpler and easier. For example, we could tackle the Java migration. Once we do that, once everyone's on 25, then migrating the chat library is now a lot easier. Once we standardize our entire fleet, we have to keep enforcing these standards. How do we make sure that new code that enters our codebase does not have these kinds of quirks that doesn't use the golden technology?

We do this quite explicitly through the use of monorepos now. Spotify had a massive migration to get all of our codebase into monorepos. When your code is centralized, it becomes a lot easier to observe. It's much easier to have stricter enforcements of standards. For example, in a monorepo, it's very easy to enforce one version of every dependency. Now your code is a lot more predictable. You can make a lot of assumptions about that because everyone is using the same dependency version. When you upgrade, you upgrade everyone at once. You have to do the migration 100%. Then, we actually introduced a lot of linting. The linting asserts that you're using the golden tech. If you try to use the dependency injection framework that's not allowed, you'll get a linting failure. You won't even be able to push that code into production. We keep our code very nice.

Then you also get the benefit of having a centralized set of skills and tools. Everyone using the monorepo now can use the same skills for writing their code, and that means that all the new code that our agents create is effectively going to look and behave quite similarly. Using this strategy, we are reducing the long tail. It's happening.

Is this how we're rewriting Spotify's codebase all the time? We think so because there's a powerful thing that starts to happen when you standardize. When you focus on aggressively standardizing your codebase, we know that our agents and our Honk can generate more correct code. Because it's easier for agents to generate the correct code and it's also easier to maximize our automated verification when the code is more predictable. When Honk generates more correct code and our codebase is more predictable from the standardization, our reviewing and planning gets much easier. This is our bottleneck. When we can review and plan more code, we can write more code. When we have more capacity to write more code, we can then further standardize our codebase. As this loop starts to build momentum, we become faster and faster at generating new code into production. We really believe that we are now rewriting Spotify's codebase all the time using this.

History of Honk

Luu: All I heard is about Honk. You guys created Honk. What's the history of Honk? You guys still able to talk about that or not yet? I was just curious about the naming of Honk. It's a very catchy name.

Aleksandar Mitic: Yes, there's a big bird thing going on right now. A lot of things are getting bird names.

Questions and Answers

Participant 1: I was wondering, in a culture where you enforce aggressive standardization, how do we deal with innovation? Because that would by definition not follow the standards.

Aleksandar Mitic: I understand your point, but I think there's a middle ground there where if something goes into critical systems, critical Spotify features, it's good that it follows the standards. That doesn't exclude innovation. You are free to try new things and you can have experimental components and writing new code. If you're able to show that that works, it's a very open culture in that sense. It's not about hindering innovation. It's about making sure the hot path looks and behaves the same way.

Jo Kelly Fenton: In a way, I think that it actually enables innovation because you can write code so much quicker now. You can leverage agents to now build services and whole features super quickly. Your innovation, maybe it doesn't look like trying a particular framework, but it looks like creating different features and services and flows, and things like that. I think personally it's enabled my innovation at least.

Participant 2: The thing for me that was quite interesting though, is you guys went away from a deterministic way of doing migrations. It feels like if you've got rid of the long tail, you could just go back to that. Presumably, Honk would be best pointed at the bits that fail that deterministic migration. Is that something that you guys do or is that on the roadmap, I suppose, in the future?

Jo Kelly Fenton: I think that definitely like if everything looks the same, it would be a lot easier to have the deterministic scripts. You could even use an LLM to create a fairly complicated deterministic script. I think it's possible. We might try it out at some point. We've talked about it. I think agents are just getting so good at writing code though, aren't they? Deterministic is much cheaper. That's a benefit.

Participant 2: One of the things we've done for migrations is write a test to ensure the migration has happened. Then like that TDD loop and agents are great at picking up all the issues for that. Is that something you guys do as well, or it is just, here's the prompt? Because it just felt like there was a lot of if statements in the prompt that could just be deterministic.

Jo Kelly Fenton: A part of our verification system initially was, you could plug in your own deterministic verification script that would allow you to have very migration specific checks. Like, don't update this file, or maybe you run some tests or something like that. I think that that's a great approach. We've used something similar previously.

Aleksandar Mitic: I think what has happened as well is like, what we consider a migration, we are happy to take on, has broadened much more. I think many companies have no breaking APIs policies, and so on. This is starting to get pushed now, where we're like, maybe we can do this. We try even harder things that we wouldn't even think of deterministically migrating before. I think it's a combination of both things.

Participant 3: You said you moved to monorepo and you glossed over that. You've talked about thousands of projects and lots of code, that must have been a massive task in itself. Obviously, that has its own cons in terms of build time and cognitive load and merging and pull request issues, all of that stuff. How did you mitigate any of that?

Aleksandar Mitic: An important piece of context there is that since we have these disciplines, like data and client and backend, it was not as if we went everyone into monorepos already. Some disciplines already had monorepos. For example, a client discipline had been running with a big monorepo for a long time. We had a lot of learnings that we could take from those past monorepo migrations. This is mostly now bringing the other disciplines that have lagged along. There is definitely learnings. I'm sure Spotify will share more learnings from those journeys, definitely.

Weakly: You talked about a lot on the quality frameworks and the ways you figured out how to get the AI from being cleverly stupid. Were there any things that stood out to you as really surprising how effective it was or really you didn't expect it, but you implemented this one little quality thing and this one little check thing, and somehow it was really impactful. Does anything come to mind?

Aleksandar Mitic: As you said, one thing that comes to mind is the examples Jo was going through, are cases where the agent did the wrong thing. What is interesting to me is in some cases, it may look like the agent is doing the wrong thing. Then once I dive into the codebase, I realized that my understanding of the problem was lacking. It's like this was actually impossible to do given the constraints we had. It's like there was some library missing or something, and the agent was basically hallucinating things. A small thing that has helped us there is gathering these insights quicker. Vibe coding simple tooling that allows us to understand, what did the model try to do here? Why did they do these things? This has helped us understand how these migrations are supposed to get done quicker. It's a half answer, but I think that is definitely one thing that surprised me how in many of these migrations, our understanding of the migration has been incomplete as well.

Jo Kelly Fenton: I feel like previously, like all of my software engineering experience before working with agents was like, to monitor a system, you'd have very deterministic checks, you'd have metrics and things like that. Now as we monitor Honk and to see how effective Honk is, you have to pause these huge LLM conversations that Honk has. It becomes quite complex to monitor. I think what you said, like having another agent pause the logs and be like, how could we improve this prompt so it gets to the solution much quicker, is actually really effective and it's super easy.

Participant 4: Have you tried some modern instruments like Claude Code or Cursor? I understand that you started this Honk a long time ago, as you said before there, but have you tried now, nowadays, some new instruments? How do you mitigate the problem that when developers start being lazy and just giving the tasks through the Slack channel to the AI, they essentially lost the understanding of the codebase. Imagine in a couple of years, nobody will actually know what's happening there. How do you mitigate that?

Aleksandar Mitic: On the first question, as you pointed out, one downside of us starting early was that none of the good tools we have now existed. The upside is that we were able to build in a way that allows us to plug and play. Where we are today, we're using all the things you can imagine under the hood. It's just that when we started, these didn't exist.

On the second question, hard to answer and hard to say.

Jo Kelly Fenton: We're still reviewing the PRs there. It's not like we're pushing straight to production. Like if you call it on Slack, you are reviewing that PR. It's going through multiple reviews. You're still aware of the changes that it's making.

Aleksandar Mitic: I think there's also like a time and place for both. If I'm building an internal tool for myself to test something for my team, maybe I don't need to know every single line of the code. There are certain places where I need to. I think people are generally good at making this distinction, when to apply different skills.

Participant 5: When you mentioned insights and tracking things, I was curious about the cost aspect of it. You mentioned thousands of PRs, agents, monitoring agents. For me, that sounded like a lot of tokens. Did you have to do some sort of analysis to, ok, if we're doing it with maybe static analysis, and not using LLMs, this is how much it costs versus now this is how much it costs. It justifies with the benefit that it has.

Aleksandar Mitic: I understand this varies between companies and how willing you are to do these things. We definitely monitor the situation, how things are going. In our particular scenario, the costs are definitely manageable. It's not something we are bothered by right now. We're seeing these tools. Sometimes it's hard to make that comparison you're suggesting. The chat library is an example here. We're definitely doing these migrations, which we just could not do before. There is no equivalent comparison. There was no AST tool we could use to do them. The value of completing these things is often so much larger than the token cost that you spend doing it.

Participant 6: How do you deal with the traffic control of if someone in Slack can put in a symptom by ask that can trigger Honk, how can they see what the blast radius of, it's gone away and start to do this work and start to affect thousands of parts of your system, and they're disconnected from seeing what's happening?

Jo Kelly Fenton: Definitely, like I mentioned before, they will review the code. You still have to have a code owner of the codebase that the changes in review that. They are aware of the kind of impact that they're having. I do understand like in Slack now, one of the cool things about Honk is that it's democratized coding in a lot of ways. Like you are seeing some maybe non-technical people start to make code changes. Like I said, we're still having that final quality assurance gate at the end that our engineers do to understand the blast radius. We leverage skills and things like that to try to guide non-technical users to make appropriate changes. We try to make it hard to make bad code changes through the use of things like skills and context.

Aleksandar Mitic: In both our fleet management tooling and on Slack, you are always prompted before, and it asks you like, I'll do this on a thousand repos? You have that feedback before. In our fleet management tooling, this is not driven by an LLM. You target and say, all Java components, and then it shows you that this is 2,000. There's always someone making that conscious decision. The same on Slack where you get prompted before saying, this will impact this many things.

See more presentations with transcripts

From ms to µs: OSS Valkey Architecture Patterns for Modern AI

Dumanshu Goyal — Wed, 05 Aug 2026 16:00:00 GMT

Summary

Dumanshu Goyal discusses optimizing data layers for low-latency workloads like AI feature stores. Drawing lessons from NASA's Space Shuttle, he explains how proxy architectures introduce hidden CPU costs, elevated tail latencies, and blast-radius risks. He demonstrates how direct-access Valkey architectures achieve microsecond latency, improve resilience, and slash infrastructure costs.

Bio

Dumanshu Goyal leads Online Data Priorities at Airbnb. Previously, he led in-memory caching for Google Cloud Databases, delivering 10x improvements in scale and price-performance for Google Cloud Memorystore, one of the rare times “10x” was more than a slide promise. Before that, he spent 10 years at AWS as the founding engineer of AWS Timestream.

About the conference

Software is changing the world. QCon San Francisco empowers software development by facilitating the spread of knowledge and innovation in the developer community. A practitioner-driven conference, QCon is designed for technical team leads, architects, engineering directors, and project managers who influence innovation in their teams.

Transcript

Dumanshu Goyal: I would like to open with a story. This story is about NASA's Space Shuttle program. What you see here is the original vision behind the space shuttle. This program started back in 1970s, continued all the way to 2011 until it was retired. This is an on-paper vision. The idea was to build a reusable spacecraft. The idea was to go back and forth into space and do it cheaply. This was basically the designs behind the spacecraft. Then from the reusability, they came up with this requirement that's similar to the other planes we see out there. We want runway landing, the spacecraft to be able to land on a runway. The runway landing from reusability became a requirement, and that's where they came up with this design. What's important about this design is because of being influenced with the existing structures of planes, they added these delta wings.

What you see there is basically called a large delta wing. This is a real model. Its name is Atlantis. It did 33 space missions over about 25 years, until it was retired. This Atlantis, again, you can see the delta wings over there. What happens is when these spacecrafts, they reenter the atmosphere, there is high heat generated. Then the wing basically slices through that heat. Now, because it's experiencing that heat, it needs to be protected. This led to the requirement of installing these tiles. These are silica tiles. What you see in the picture there, the small brick-shaped structures, these are the silica tiles. This model has about 24,000 silica tiles on it. The idea was to protect the body, the wings, the underbelly of the spacecraft and make it work. When they designed and built this airplane, it was estimated to be a $10 million round trip, what it would cost us.

Then it would require some maintenance and it would be ready in about two weeks' time frame. This was the original goal. When they started using it, in reality, the $10 million ended up to be $1.5 billion. The costs exploded. The two weeks' turnaround time to get ready for the next mission got to be two months. What went wrong? The silica tiles, they had to be replaced. There were unexpected damages during the flight. They were thinking that, ok, they would hold it well, but they were not. The heat was too extreme, a lot of new discoveries. This problem ultimately added to the on-ground complexity as well. The on-ground operations, as you can see here, they got really complex when they had to get this ready for the next mission. The complexity also bloated up not just the cost aspects or the performance aspects around when it can be reused.

In 2011, when NASA retired this program, another program started, the Commercial Crew Program. This is a bit about privatization. This is where you see Starliner there from Boeing. Another top model is called Dragon from SpaceX. These are what we call as capsule designs. They went back to the core requirement of reusability. There was reusability. You want to keep the payload, the crew safe. There is safety of the crew as the core requirement. Then they basically challenged the runway landing requirement. Is that really required? Do we need to really land this on a runway? They let go of that requirement. They let go of the tiles. They let go of the wings. They came up with a simpler design, a capsule. All they had to do was to protect the nose of this capsule, which has this heat shield on the top, so that when it enters the atmosphere, it can sustain high heat.

Not just sustain it, but also move it away from the body of the capsule, so that the rest of the capsule is not impacted. This blunt body of this capsule was a success, and it has been building up on it. This is how they went back to the requirements, and then basically reshaped the spacecraft vision around how do you do reusable space travel.

I want to frame this story around a quote. This quote is about perfection. This is an age-old quote from Antoine de Saint-Exupery. He was an aviator, an engineer, not a software developer like us, or most of us. What he mentioned, when we talk about the spacecraft mission, you have this mindset, I need to solve for runway landing. I'll add the tiles to the system. Then you keep fixing it, keep maintaining it. Versus taking a step back, taking a holistic look, coming up with the capsule design, challenging your requirements. Do you truly need them? Then basically getting rid of the wings, the tiles to simplify the end result. This is basically what I refer to as designing for efficiency. In this talk, we'll cover several examples of what it means to go back to your requirements, do a rigorous requirement analysis, not take them at the surface.

Also, combine them with a holistic tradeoff analysis. This tradeoff holistic analysis, what you could say in the spacecraft mission, it's hindsight now, but you could say it was missing as to what would be the real impact on the tiles. How much it would cost us for real. When you combine this rigorous requirement analysis and the holistic tradeoff analysis, that's when you come up with the efficient designs, whether it's cost-effective or for performance reasons.

Professional Background

I'm Dumanshu. I'm a lead engineer at Airbnb. I lead their data platform. This data platform is responsible for all the bookings you do on Airbnb. All that data gets stored. This is what drives their $11 billion revenue. Before Airbnb, just a little bit about my background. I spent a decade obsessed with durability, where I got to work on internals of several foundational systems. One of them was AWS DynamoDB. After that, eventually, I moved on to the other extreme, which was raw sub-millisecond performance with in-memory caching at Google. This is my talk's title. It shows us basically the new reality about AI, where performance equals efficiency. When we are moving from milliseconds to microseconds, sometimes it's about real-time performance. We'll see how in the AI space this matters. Sometimes it's not about the latency or the performance. Most of the time, it would be about the cost. We'll also go over how moving from these milliseconds to microseconds affects your cost. I'll frame this talk around Redis and Valkey, some of the caching systems, but it's pretty generic in the sense it applies to maybe Memcached or a bunch of other caching systems you might be using.

Roadmap

This is my agenda. We'll cover it in three parts. The first part is about the need of microseconds. We all talk about, we want to get faster. We want microseconds. What's the real need behind microseconds? We'll do a case study, a use case in the AI space to understand why microsecond latency could be truly important, or it might not be in your case. Then the second part talks about our journey of how we are living with these milliseconds architectures, how it got there. I'll frame it around the evolution of Redis and Valkey. Then we'll also cover the architectural tradeoffs, like what's the impact on latency, the performance, the cost, the dollar that you pay for the architecture, and then reliability. Then, in the end, the third part will cover the milliseconds architecture. In the milliseconds architecture, we'll talk about what the price performance equation looks like there, what the reliability equation looks like there.

Part 1: The AI Data Wall - Why Milliseconds Are No Longer Good Enough

Part one, establishing the need. We want to go with requirements. We'll start with what's the need of these microsecond latencies. Why does it matter to us? The example I want to use today is an AI feature store. I'll talk about what the feature store is. This example I'm taking from the DoorDash AI platform. All public blogs. Just was quite relatable to my talk. This example basically starts with a prediction service where you have a 100-millisecond budget, not microsecond, 100-millisecond budget. That's a lot of time, to basically drive a prediction. Then when you combine that with complex data needs, you'll see why it requires an underlying system to serve microsecond latency. This is code from one of the DoorDash's blog posts. This is about a prediction service which is used to detect fraud or provide restaurant recommendations. The number one requirement of this prediction service, as they quote, is basically to be able to serve a prediction within 100 milliseconds.

What does this prediction service do? It looks as something called as features. You can think of features as a piece of real-time data. If you are in the fraud prediction service use case, you would see, is this credit card new? How many logins have happened recently? Stuff like that. You want to be able to feed real-time information to your AI model to be able to drive a prediction out of it. The important thing to note here is for a single prediction, the system needs hundreds of features to be able to make that prediction. It's like quite a diverse dataset, lot of features, these pieces of data coming together to actually make a decision out of it. The underlying challenge is, how do you make it happen so fast? You're talking about hundreds of features. You're talking about 100 milliseconds. This is like a typical architecture.

It's an AI inference use case around the prediction service. The architecture details are not that important, but I think I would like you to focus on the prediction service which basically gets this user contract. The user interacts with the prediction service through the app, and this is where the 100-millisecond constraint comes into picture. This prediction service works with the AI model. Then it also works with this AI feature store. This AI feature store is the contract that we are trying to dive into. Like, what's the contract between the prediction service and the AI feature store? Can it serve millisecond latencies? Is 2 milliseconds good enough, or does it have to be in hundreds of microseconds?

Going back to the DoorDash blog post, this is where they mention, because the single prediction needs hundreds of features, so a natural thing to do is to fetch data in parallel. I have 100 different sources or whatever. I could basically just fan out those calls. When you fan out those calls, what they call out here is the p99 latency. I basically refer to it as tail latency. You need to keep the tail latency low. Why does that matter? Here is a relay race example. When you're basically making hundreds of calls, your latency is decided by the slowest call. When you are having so many calls, your tail latency is bound to show up in every single prediction request. Your latencies, even though in steady state, might be just about 1 millisecond, but there would be that one request that maybe goes all the way to 10 milliseconds.

The analogy I want to map to the relay race where you need everyone to finish, get to the finish line, and then one person basically is lagging behind, and that dictates what happens, how soon you can finish the race. This gets worse. We talked about the AI use case. We also talked about there is a complex data need. What does this complex data need look like? We went over, you need to make 100 parallel calls. Then the next one is a sequential lookup. It's pretty simple. There is a parallel thing. There is a sequential thing. The sequential lookup, all it means is you have data dependencies. In order to get to a piece of data, sometimes you have to traverse the path. Here is an order-related example. You have to fetch a piece of information, and then only you can make the next call. Simply making calls in parallel doesn't help.

Sometimes you have to make it sequentially. When you make calls sequentially, and you are prone to these tail latencies, the high tail latencies, that's where you'll see it starts to eat away your budget. Out of that 100 milliseconds, you want to give as much time as possible to the AI model to do the real thing. Just for fetching the data, you have already eaten up a lot of budget, in this case.

This brings us to the final verdict from the DoorDash AI engineers. What they basically describe here is their AI model latency is in low milliseconds range. We are not worried about that. That's a given. What that means is the underlying store, the underlying feature store, needs to be able to serve the features at a latency which is proportionately lower. This is where it gets even tighter, where we talk about microseconds, for example, to serve the features to this model. Another code, in this example, they build this feature store backed by a data source using Redis. For our talk, we'll talk a bit about Redis. Eventually, we'll go into Valkey. What we have established here is the requirements for this feature store. Like I said, it's backed by another durable data source. Then we are talking about serving these features at a microsecond latency level. I would like to start with a quick introduction to Redis, Valkey.

I'll get back to our favorite LLM to do it for us. Redis, I think in plain, simple terms, it serves numerous use cases, you name it. It has advanced data structures, so many operations available. It's quite simple to use, and that's what basically speaks to its popularity. The important thing to note is this technology already goes back to 2009, since when it started picking up on its popularity. Then, in 2024, that's when the open-source licensing change happened with Redis, and Valkey was born. Valkey is effectively a fork of Redis. You can do all what you want to do with Redis with Valkey as well. That's why I use it interchangeably. Then, it's fully open source. It's backed by the community, a bunch of hyperscalers. That's a little bit history about Valkey. This is a screenshot from the press release around when the Valkey fork was announced, Linux Foundation backed it up. This is early 2024. This is a Stack Overflow 2025 database survey results. What you see here is, in terms of the desired level among these developers, Valkey is trending very close to Postgres. Postgres has really gained community presence, and in terms of the desired level, just this is within one year of its fork.

Before we end the basic introduction, I want to do a typical 101, like a Hello World Valkey application. What does it look like? You have a client. You have the Valkey server. It's a plain, simple key-value lookup, what we have here. This is my terminal running Valkey CLI. You can use the CLI mode. There are a bunch of ways to access it, typical clients. What you do is a SET my_key my_value, and then you basically can get it after that. It's as simple as that plain key-value lookup. We're not talking about anything more advanced than that.

Part 2: The Milliseconds Journey - The Architectural Choice and Its Hidden Costs

Going to our part two. So far, what we talked about is how the DoorDash AI use case that we looked at is basically a requirement for real-time performance, which is measured in microseconds, not milliseconds or not tens of milliseconds. Next, we are going to talk a bit about our milliseconds journey. In the context of Redis, how the architecture has been evolving, what it's like today, and what does it mean in terms of price, performance, latency, reliability. This is a chart from DB-Engines. It's widely popular. They have a way to calculate scores. This showcases, over the last 10-plus years, how the popularity of Redis has been rising. Going back to all the way to 2009, when Redis started to get popular, a lot of applications started using it. They had a simple caching setup, like what we saw with the Valkey 101. In this case, your application is, again, accessing Redis, as simple as that.

However, because of this popularity, the applications started running into the scalability constraint, where the single Redis node was not able to serve the workloads anymore. Either sometimes there's too many client connections, too many operations per second hitting the vertical scaling limits of a single node. There was no native solution available at the time. Then this led to new architectures, which I refer to as the proxied architectures or the gateway architectures, where you basically take the independent Redis nodes and then you stitch them together. You make it look like one giant system, one giant cluster. Your data is sharded across them, and your application basically connects with them. The blueprint, what I have, is the legacy applications and then a gateway layer, and then the Redis servers backing it.

What we see in 2009, over time, so it was 2015, 6 years later when Redis Cluster was released, Redis Cluster improved upon these sharding capabilities. It provided a native sharding capability built into Redis, where a bunch of nodes could work together to provide that one giant cluster view to the end client. Then, the proxies were still there, legacy apps were still there but the trend continued. For one, the Redis Cluster offering was not mature enough. It was new. It was complex. You needed some complicated logic on the client, so not everyone got it right. It was not gaining that much popularity, which doubled down on the proxied architectures. They continued to grow. The legacy applications, even if they wanted to adopt a clustered offering, they couldn't because they didn't want to change the client, so what they did is they, again, continued with the proxy layer in the middle.

You put the cluster behind the proxies, and then you can make it work. You don't have to do the independent nodes behind the scenes. You can still use clusters, and then the proxies would abstract it away from the end client application. Legacy constraints was feeding into this growth, or is still feeding into the growth. The second one was related to simplicity, or abstracting out, or decoupling your client applications from the server, so that's like another one of the non-functional reasons why you see proxies coming into picture. The third one is more technical, I would say, less of a constraint from a legacy perspective. This has to do with connection multiplexing where your application scale is so high, and there are very few applications that I've seen at that kind of scale where you are basically running with 100,000 pods, the client pods. They are trying to connect to the cluster behind the scenes.

If you were to connect to the Redis, Valkey cluster, whatever, directly with those 100,000 connections, at least one each coming from the client, that would definitely kill the cluster. In those cases, you want to have an architectural improvement using proxies, for example, which can take on or offload the connection management. What happens is you get 100,000 connections onto the proxies. They are stateless, so you can scale them out horizontally. Then they would take those requests and multiplex them onto fewer connections against the backend server. The idea is to take away the connection handling responsibilities, move them to the proxies. These are some of the legit or the real requirements around where proxies have served a purpose or continues to serve the purpose.

In this architecture, we use a common open-source Envoy proxy. There are 5 to 10 proxies which are widely used in the industry. I just took one of them, but it has nothing specific to do with the Envoy proxy, so we'll use this example throughout our discussion here. In this example, you have the Envoy proxy layer, which is hosting this logic. The logic could be in two parts. One is, you have independent Redis, Valkey nodes behind the scenes. I've switched to Valkey here instead of Redis. The Envoy could shard the client data using a simple hashing scheme. It's not a database, so there are a little bit of data losing concerns, but it can make it work. Then your client connects to Envoy proxy. It can talk to any Envoy proxy. Then it routes the request to the right node. The other mode is the cluster mode, where Valkey is self-aware of how the data is sharded, and Envoy proxy can also connect to that. That's the advantage of these proxies, that they can speak multiple protocols with the underlying backing system.

We talked about the journey of Redis, how proxies came into picture, how they continue to exist, some of the top reasons behind these proxied architectures. Now we're going to dive into the real-world tradeoffs or the metrics around price, performance, latency for these proxied architectures. What do they actually look like? Let's start with the latency. One could categorize it as a performance dimension. Latency is one way to look at it. Coming to our architecture with three Envoy nodes, three Valkey nodes. Now, to study the latency aspects, the performance aspects, I'm going to simplify this a bit. We are going to just reduce the three Envoys to one, and similarly for Valkey, three to one. It's a pretty simple setup with the client talking to the proxies, proxies talking to the Valkey node behind the scenes. The run I have here, it uses a utility called memtier_benchmark.

Nothing special here. It's just a tool to drive load, and basically measure some characteristics of the system. I have this deployed on EC2, AWS. You can see the VM types. Both of the proxy and Valkey are the exact same machine types. They are in the same availability zones, no cross-zone hop, wanted to reduce the interference. They are both 8-core machines each. This is a benchmark run from my terminal. Don't worry about the grayed-out details. What's important is we talked about the memtier_benchmark. Then the second aspect is it's a read-heavy workload, which you see typically in practice. In this case, it's roughly a 90-10 ratio. More than 90% of the traffic is read traffic. In this setup, what you see is this setup is driving half a million QPS. Half a million queries per second is being driven, which is roughly 90-plus percent reads, and then the remaining is sets.

The latency. Since we wanted to talk about the latency. For these half a million queries per second, the latency that you get, so we measure on two sets, one is the tail, the p99. That's roughly 2.5 milliseconds. This is in the milliseconds category where it could go from anywhere like 1 to 10 milliseconds, roughly speaking. Then there is p50, which is your median latency, and this is close to 1 millisecond. This is what I would refer to as our milliseconds app. These numbers are still super good. Like, they are driving half a million QPS. The latency is super low, single-digit millisecond. Not bad at all.

For each of these setups, we are going to look at a little bit deeper into the latency, price, performance aspects one by one. First, we are going to look deeper into this p50 latency. I like sometimes appreciating the system characteristics, like how it ends up to be where it is right now. In case of the p50 latency, we talked about 1 millisecond, and as you notice, this latency is measured from the client side, so it's a full round trip covered in 1 millisecond. There are two hops going on. These two are network hops, so in order to measure the cost of a network hop, I used a utility, My Traceroute. It's a Linux utility. It's similar to a ping, just has more fancy knobs around it. What you see here is between the client and the proxy node, there is a 300 microseconds hop. This 300 microseconds hop is basically in the same availability zone.

Like I said, not much of an interference, not a cross-zone hop, which gets to be higher. The same 300 microseconds would go from Envoy to Valkey, all round trip. Ultimately, what it ends up to be is 600 microseconds, the two hops, 300, 300, out of the p50 of 1 millisecond gets spent on the network. What you're left with is 400 microseconds, and that 400 microseconds involves the full proxy processing to route the request. It involves the Valkey server processing to serve the dataset. Then it also involves a little bit of client processing, where the client sends the request, gets the response, measures the time taken. All that stuff is basically done in just 400 microseconds. Network, maybe you can't do much about it for a remote lookup, but that's how the latency breakdown looks like for a p50 latency.

Next, we will look at how much do you need to spend to get a certain amount of throughput, measured in queries per second. Cost is important. As engineers, we generally talk about latency, performance. I like to also talk about the dollar. We're going to dive into that. In our setup, recall that we got half a million QPS at 2.5 millisecond tail latency, pretty good. The CPU measurement during this run was 90% CPU utilization on Envoy, which is pretty high, so it means it was almost completely taken. Then the Valkey server CPU utilization is basically 60%. There still seems to be a lot more capacity which is remaining on Valkey to be utilized. Just mapping it back to the diagram, visually you can see there is 90%, 60%. Why is this proxy 90%? Why is it so compute intensive? Why is it consuming so much CPU?

In the caching world, this is where you'll see that it's quite different from the databases world, where we worry usually less about the QPS. It's more about latency, the I/O, and stuff like that. In the caching world, there are two main things that would typically come up. One is the QPS. Sometimes the applications are offloading the QPS from the database to the cache. Generally, caching applications, they would have high QPS, or they have a lot of in-memory storage. You are paying for in-memory storage, which is much more expensive than our typical disk-backed storage. This is where the caching workloads are interesting, because you have to pay attention to the QPS aspects as well. Now, because of this QPS, this traffic is being forwarded through the proxies. What I want to do is, I want to build an intuition around where is this compute going. The 90% consumption on the Envoy, what the breakdown looks like.

How we talked about the latency, where we talked about the network hops, network hops taking majority of the latency. In case of the CPU, so one intuition is basically, when you're forwarding traffic through the proxies, it's doing twice the amount of I/O as compared to the underlying Valkey server. If you look at the Valkey server, it gets a request, sends a response out. In case of the proxy, it's a pretty simple thing. Client sends a request, that's one. Request goes out to the Valkey server. The response comes back from the Valkey server, that's the third one. Then the response goes back to the client. The proxy node has to move this data or the underlying network packets twice the time of what a typical I/O would look like for Valkey. This whole process of moving the network packets is very compute intensive. What you see here is the process routing the request.

Underneath, the network card, once it receives the packet, it has to put that into the host memory because we are running virtual machines, so they are running on a physical host. Once it gets into the memory of the physical machine, another process would route it to the guest VM, the guest OS. Then from there, through interrupts or some mechanism, it would get delivered to the kernel's TCP stack, which is where eventually it would find its way to your application where your code of routing the request runs. Then it would be the reverse path when it has to send something out.

This was about building an intuition. They're twice the I/O. Now let's look at the real data. We want to validate our intuition now with a profiler. Flame graphs are also my favorite, but they're also very hard to read when it comes to slides. Flame graph is just a visualization tool which captures, during the actual run, the stack frames, and then sees what stack frames appear more often than the others. Then if something is appearing more often than the other, you can take enough samples to determine maybe this is the area where it spends more CPU than the others. Don't worry about reading these stack frames. I'm just going to do a higher-level analysis here. We looked at the 90% Envoy CPU consumption. I have a breakdown, which you can also measure from the process. There is user mode, there is system mode. User mode is where your application is running.

The code written, whoever built the proxy would run into the user mode. Then it interacts with the system calls, the TCP stack, that's all basically kernel mode. The split of this 90% is very even, like 45%, 45%, user mode versus kernel mode. Lot of I/O, there is obviously a lot going on with the application. You see the two big boxes I have, these are parts of the flame graph. One part on the left is about the application processing. This is less about the I/O, but it still needs a lot of CPU. This can be heavily optimized, writing different code and all that. The right part is the TCP send and receive path which involves a little bit of application and also the kernel frames. It's like the I/O, what we said is twice as expensive. The I/O does form a major chunk of this whole 90% CPU utilization. The key point we are trying to understand is when you say, proxy consumes a lot of CPU, what is it actually doing? It's actually routing requests. A significant amount of CPU spent there. The rest of the CPU, in this case for Envoy proxy, it's half, it's spent in the I/O.

What we have established is this 90% Envoy, given that the Valkey server has capacity. To me it reads like, Envoy is like the chokepoint here, and then maybe there is something we can do. The idea is to, now I'm going to add one more Envoy proxy to the setup to horizontally scale this layer. Then, using these two Envoy proxies, they are stateless, so I'm going to load balance the request. The client would just pick any one arbitrarily and drive the traffic. Then we're going to just repeat the benchmark. With these two proxies fronting the server, we have 1 million QPS driven from this system. When we were using one Envoy proxy, we had half a million QPS being funneled through the system. Now with two, we have 1 million. That's good because it has a nice scaling characteristic. One was half, it doubled when we doubled the hardware there.

That's good. The latencies are quite similar. p50, again, 1 millisecond. Tail latency, 2+ milliseconds, similar to what we had before. We still retain our milliseconds app roughly. Then we were able to hit 1 million QPS out of this system setup. Since we talked about the dollar cost, let's look at what this setup would cost us. The eighth generation Graviton instance that I'm using costs about $230 a month. The hardware we have provisioned is about $400 roughly for proxies, $200-something for Valkey. This comes out to be about $700 a month to deliver on 1 million QPS at low single-digit millisecond tail latency. This is an important result. Let's take a moment to capture this. I'm going to come back to this later in the discussion. This is not bad. If I gloss over the architectural details, you have an in-memory setup. In-memory is expensive. Everyone knows that.

We are spending $700, justified for the business. Nothing seems wrong. Note that the $700 in reality is actually a lot more. It's going to be close to easily $1,500 or something, because this does not have high availability built in. With high availability, you will have copies and then you'll have cross-zone hops. There will be a lot of network jitter, more cost involved. What that would lead is like worse latencies, higher cost. What we are talking is a little bit of a crafted setup in the same availability zone without those variables.

What we covered so far, we talked about the latency aspects. We talked about the cost associated with delivering 1 million QPS at a certain latency. Typically, architectural evaluations would stop here because we added the proxy. We were analyzing the proxied architecture. It's a stateless system. Stateless systems are usually harmless, from our experiences. Very few times we would look at anything beyond the setup. The third piece is reliability. It has an interesting pitfall associated with it, which is a single point of failure. Our setup has a single point of failure. At least to me, it was not obvious when I first exposed to this problem as to, what is special about this architecture where reliability could take a hit. It's related to head-of-line blocking. It's related to single point of failure. We'll go over that. To understand these pitfalls, we want to drive some traffic. Again, nothing fancy.

We have our architecture. I'm going to let go the single node one and bring back the three proxies, three Valkey servers because more nodes are interesting. We're going to drive some traffic. I have a custom script, not the benchmarking one this time, just to capture the client-side availability, what it looks like. It's driving a small amount of traffic, and the availability, what you see here is 100% every second, which means all the 100 QPS, basically all the requests are successful. Nothing comes without failures, or even worse, slowness. To make things interesting, we're going to take a shard and inject some slowness into it and see how the system behaves. This is Lua scripting. Redis has this powerful feature, Valkey has it as well. It basically is like a server-side function where you can inject some logic and make it run on the server. The code here is a workaround to basically make the main server slow down for about 5 seconds, so just to induce a slowness or a failure there. You can reproduce this easily. Just don't do it in production. It's pretty deadly to the production server.

Now let's see what happens to our client-side availability. We have the 100% run rate, and then for those 4 to 5 seconds when we basically injected the slowness, our availability dropped to zero. The client is experiencing full unavailability, no requests going through for those 5 seconds when we inject this failure. We had evenly distributed traffic in our setup. What even distribution means, like our shard_1, which was experiencing this slowness, was getting one-third of the traffic. The other shards, we did not touch them at all in production or in our setup. They were getting two-thirds of the traffic, but yet our net availability is zero. As you can also see, all my traffic is now routed to shard_1. Shard_2 and shard_3 are actually not even getting any traffic. That hints us to something, like they are all healthy, no CPU consumed, like what is going on?

Imagine if you were running a real server, instead of the three nodes, you had tens or even hundreds of nodes, both for the proxy setup and the shard, one shard going down, taking down your entire cluster. We're going to root cause this. Since the client reported zero availability, so I'm going to just zoom in to the client to understand what is going on. This is what the healthy state looks like. We follow the best practices. The client has to connect to the proxies. You need connections, so you need a connection pool. Do not create connections all the time, take that overhead. This connection pool is capped to five connections. Then these five connections are basically serving the request for the shard_1 or whatever request comes up. Anything more would get throttled. Requests to the shard_1 start to take longer. Because of this, the request to the other shards, they get starved.

Now the entire connection pool becomes occupied with shard_1 requests, because they were slow, so they were not freeing the connection sooner. This is what leads to the outage. As you can see, all my requests to shard_1 have taken over the connection pool on the client side. Shard_2 and shard_3 are not getting any requests. They're all being rejected. This is how a single shard can turn the entire system into a full outage. To summarize what we learned with these proxied architectures. We spent $700 to achieve 1 million QPS with low tail latency. Then we also saw how one shard crashed the whole system.

Part 3: The Microsecond Playbook - The Strategic Path to µs Efficiency and Resilience

Our last part is about microseconds. Let's get deeper into the microseconds architecture. There is a quote from Dr. Werner Vogels, which reminds me, we always say that we have done it this way, so we would want to look for ways to question the current system, to constantly optimize and improve our systems. Just like the space shuttle where we got rid of the wings by questioning the runway landing requirement, getting to the capsules which were much more efficient, I'm going to do the same exercise with these proxies. I'm going to remove these proxies. Maybe they are critical to some use cases where you would definitely need them. For our discussion, I'm going to remove them and see what happens. After the proxies are gone, our clients are able to access the servers directly, but you have multiple servers, so you need to know where the data is.

What you need is a smart client which can understand the topology. The underlying Valkey cluster has a way for the client to learn the topology and route the requests accordingly. What we are going to do is we are going to do an exercise of the latency, price, performance, reliability aspects, but in the reverse order. We're going to start with reliability, then talk about cost, and then finally talk about latency. How we did with the proxied architecture, in the same direct access architecture, we are going to inject slowness in this shard. Use the same script, exactly the same setup. This time what you see is that availability is not zero, like earlier. It's partial outage. It hovers between 60% to 80%, which roughly is two-third of our traffic as expected. When shard_1 failed, my availability reported is 66%. Now let's zoom in into the client, like how we did with the root causing to understand what is going on here.

When your client is connecting directly to the shards, it's as simple as you have different connection pools, one per endpoint. Each connection pool has a bunch of connections to send requests to that shard. Then what happens is when shard_1 gets to slow down, it occupies all the connections in that pool, in the respective pool, and then more requests come up, even more requests, they start to get throttled. What's important is the shard_2 and shard_3's requests are all protected, they are isolated. The fault isolation, what you have on the shard, because the shards are independent of each other, now extends to the client side where the other requests are not impacted. This reminds me of a bulkhead pattern. This is a pattern, goes back to ancient Greece times with ships where they built these bulkheads. They're still there in practice in different shapes and forms. These bulkheads are basically like compartments. They are for the structural integrity of the ship. One other important aspect they serve is fault isolation, similar to our sharded architecture where one part gets damaged, it's not like it will sink the entire ship. It has some impact, but the ship still continues to float.

We just covered the reliability where we saw how with the proxied architecture, one slow shard takes down the whole cluster, and how in the direct access architecture we were able to isolate that fault as expected. Next on our list is price, performance. For price, performance, we have a single Valkey shard, no proxies. We are going to run our benchmark, which is able to achieve 1 million QPS. This Valkey server is able to achieve 1 million QPS without any proxies. The Valkey server just costs $230, like what we are basically able to drive here. This is our latency numbers with our Valkey setup. We have about 567 microseconds at the tail latency instead of the 2+ milliseconds. This is at 1 million QPS.

Summary and References

To summarize and put it all together, we have the cost, which dropped to one-third without the proxies to deliver the same 1 million QPS. We have four times higher latencies with the proxies set up at the tail. Instead of 600 microseconds with Valkey server, we have about 2.4 milliseconds. Here are the references I used for this talk. We basically covered the space shuttle, what you'll see here, Dr. Werner Vogel's Laws of Frugal Architect, and the Valkey's 1 billion RPS innovation. I didn't get to go deeper there, but you can read about it with these references.

Key Learnings

The last slide is about the key takeaways, how we went into the direct access architecture where we understood how the network hop, the 300 microseconds was taking out almost all the latency out of the 360 microseconds for p50. We saw how the architecture impacts the cost, the $700 versus just the $230. How a single point of failure in the proxied architecture goes away with the direct access architecture. Finally, Valkey server is so efficient because of all the community contributions behind the scenes which help us drive this infrastructure cost to be lower.

See more presentations with transcripts

Automatically Retrofitting JIT Compilers

Laurence Tratt — Tue, 04 Aug 2026 16:00:00 GMT

Summary

Laurence Tratt discusses yk, an open-source meta-tracing JIT compiler framework. He shares how to automatically speed up C-based language interpreters like Lua and MicroPython with minimal, non-invasive code changes. He explains the inner workings of tracing loops, optimizing compiled traces using developer hints, and managing complex deoptimization back to the interpreter.

Bio

Laurence Tratt is the Shopify / Royal Academy of Engineering Research Chair in Language Engineering in the Department of Informatics at King’s College London. His research focuses on improving our ability to develop and use software, with a particular focus on performance: how can we make more software run at the speed that its users need and want?

About the conference

Transcript

Laurence Tratt: Wouldn't it be great if we could make more of our programming languages run faster? Particularly the awkward squad, we might call them the dynamically typed language, or the scripting languages, the Luas, the Rubys, the Pythons, but also there are quite a lot of other language implementations out there that may not run as fast as you would like. CPU companies have CPU simulators and so on. What I'm going to do is show you that we can take existing language implementations and automatically add a just-in-time compiler to them. I'm going to introduce a new technology we developed called yk. The good news is that it does something fun. The problem is that showing performance is quite challenging. I'm going to start with a simple demo. Let's take the following Lua program, as has already been mentioned. Here is a Mandelbrot program, very standard. I've cranked the quality handle up a bit so that it runs reasonably long so we can actually benchmark something.

Lua is a dynamically typed language used in games, Neovim, various other places. Let's take the standard Lua VM that you would download from lua.org. We're going to run it and just see how long this takes. It's a Mandelbrot. It's taken 3.2 seconds. Very good. Now what I'm going to do is our fork of this same Lua VM run through the yk technology. It displayed a lot quicker. It's gone 0.8 seconds. That's about 4x faster. Now, you can bet your bottom dollar I have cherry-picked this example of something rotten. I've chosen something where we're particularly effective. Across a wider set of Lua benchmarks, probably a geomean of 2x is about right. The point is that we aren't just speeding up Lua here. We've just started a little bit of work. This is very early stage, not quite as advanced, looking at putting this into MicroPython, which is a small Python VM.

If we have a look at MicroPython on the well-known Fannkuch benchmark. It's doing something. We've put a little to do there. You can see it's roughly linear performance. This is the normal MicroPython implementation chugging away. How long does it take? Fifteen seconds. You guessed the same thing, we have our own fork, ykmicropython. Watch at some point, somewhere around about now, notice it suddenly really starts warming up and chunking through. This one's about 2x as fast. That's not as representative of ykmicropython because I think we've put in less than 10 person days into that so far. What I'm going to try and do for the rest of this talk is motivate why we've done this, how it works, some of the technical challenges involved, and what will come next. We're very lucky this work was being funded by Shopify and the Royal Academy of Engineering. Neither are responsible for anything I say.

The work's done in collaboration with Edd Barrett, Lukas Diekmann, and Pavel Durov. One thing I just want to emphasize is that we're aiming not to change too much. Those changes you saw to yklua, here's the diff against the normal Lua VM. We've added 400 lines. We probably could have got away with adding fewer, and changed about 50. We're not talking about massive invasive changes. This is one under 5% of the codebase.

Terminology

I want to define some terminology. I've used the term virtual machine. I haven't defined it. We're talking about programming language virtual machines. They're some sort of system that contain one or more language implementations. Say, the Java virtual machine's got multiple just-in-time compilers and interpreter and so on. When I say an interpreter, and this one I know is particularly contentious, I'm talking about a simple language implementation. Something that conceptually an undergrad might be able to write. If you don't like that definition, there is a blog post where I try and define it a bit more carefully. Finally, a just-in-time compiler. This is some language implementation that runs a program, observes it, works out which points are running most often, optimizes those, compiles them into machine code, and then uses those dynamically compiled machine code versions.

Why?

Why are we doing this? Apart from the fact that it's fun. Everyone has experienced situations, I think, where your software runs too slow. Sometimes you can see a long time in advance that's going to happen. You may be running on some large data thing where you know more data is coming in, so you can predict that in nine months your program will run too slow for your situation. My experience is that often people go from not caring about performance to thinking it's an existential crisis in under 24 hours. Then they flail around in panic, and if they're hopefully sensible and intelligent, they'll profile their code. Maybe they'll try implementing better algorithms or better data structures. Maybe they'll try rewriting it in a faster language. A bit easier with LLMs, I admit. They often miss one of my favorite techniques, bearing in mind that I'm lazy, which is you can often just drop in a faster language implementation.

Maybe you upgrade your C compiler and you get a few percentage points of extra performance for free. In many languages, there are alternative, faster JIT compiling virtual machines. If you're running the normal version of Python, you've downloaded from python.org, the interpreter is not very fast. You may be able to drop in PyPy, which is a Python VM with a JIT compiler. You might get a 3, 4, 5x improvement for free. That's all you have to do. You can put the performance problem off for a long time. Does anyone recognize what these names are? That's a .NET one. We've got some Python implementations. Actually, these are Python JIT compiling VMs. These are all JIT compilers for Python. Here's some more. There's some more. More, more, more. These are the publicly available just-in-time compilers I know of for Python. There are some that are not publicly known.

Nearly all of these are dead or abandoned in such a way that they might as well be dead. There are some of them that are still alive and used, admittedly. An obvious question is, why are there so many? It can't just be people doing it for fun. I would say fundamentally, it's because just-in-time compiling VMs are really hard. There's lots of moving parts that you've got to get right. You make the smallest slip, things go splat, and you will be debugging it for a week. Because they're hard, they're expensive. Most programming languages are really quite big. A language Python, Ruby, they're really big languages in terms of the number of features they have, the complexity of their interactions. There's just a lot of stuff to implement. Say that the main Java virtual machine HotSpot has had low thousands of persons of years of effort put into it.

V8, the JavaScript VM in Chrome, a team of 50 people. These are large amounts of resources that are necessary. There are other issues. If you make a new implementation, I say maybe, you will almost certainly be in some, maybe small ways, incompatible with the existing mainstream implementations. People try your thing out. It doesn't quite run your program straight away. You ditch it, move on. You never try it again. You may also be incompatible with the existing ecosystem of modules and extensions. Again, if people can't run their stuff on your new implementation, they'll just give up and go on. Another big problem is, and this is, I think, one of the major reasons why there are so many Python JIT compilers you saw on the previous slide, they can be really difficult to evolve. You implement a new just-in-time compiler for your favorite language, you get it working for the current version of the language, and then the annoying people in charge of the language spec go and change it.

They extend it, maybe break some feature, and you've now got to cope with that. You've probably embedded deep assumptions about the existing language in it. Then you're stuck. You tend to fall further and further behind. You can see this in many well-known and sometimes once well-known JIT compilers. LuaJIT, for example, is stuck on a relatively old version of Lua.

For all these reasons, there's long been a desire to see, can we somehow automate the creation of just-in-time compilers? There are two basic approaches which happily exemplified in really the only two existing technologies. RPython is effectively the same project name as PyPy. If you've heard of one, you've heard of the other. That's a meta-tracing system. Truffle is a partial evaluation system. We do need to briefly define a bit of terminology, because what both of these systems want you to do is write a new interpreter in the host language. A subset of Python or if you're in Truffle, you write it in Java. You've got to write a new interpreter for the guest language. That guest language could be a Lua, Ruby, Python, CPU simulator, whatever. Doesn't really matter. If you do that, you will often get really good performance. I yield to no one in my admiration for these systems.

They are absolutely astonishing. The performance gains can be really good. Probably not that many people here have used them or the results of these systems. Unfortunately, I think it comes back to some of the reasons from the previous slides. They're often not fully compatible. They tend to get stuck on slightly older versions of the languages and it's difficult to keep up. Here's the challenge we've set ourselves. Can we get the benefits that you get in hopefully a faster language implementation, but you don't get stuck on old versions. There's another detail that I'll go into.

How?

Now about the particular thing that I'm introducing today. This is the one thing that perhaps I should have realized that took me years to realize. Most of this awkward squad of languages we're talking about have de facto standard implementations. Often, Ruby and Python, it's CPython, MRI Ruby, or sometimes called CRuby for Ruby. Lua is just called the Lua VM. Those standard implementations which are nearly always C interpreters are the source of truth. Those languages have language specifications with lots of semantics written down. The semantics are not complete. There's lots of stuff in libraries and so on that's not included in the specification that people expect. Basically, people run a program on their standard implementation and if it does what they expect, their program is good, whether or not there are other valid executions of their program or not. They come to rely on everything the existing C interpreter does.

We realized we had to take this as our source of truth. That's fundamentally what we've done. The demos you saw earlier hopefully give you some idea of that. If you make that decision, and it is quite a big one, I think at least in my pea brain, there's only one technique you can then use, which is you can generate a meta-tracing just-in-time compiler from a C interpreter. I don't think other techniques work, or at least I don't know how to make them work. Whenever anyone uses the prefix meta, I get very scared, because now I have to understand two things. The word and the meta word. If you're making a JIT compiling VM, the typical way it's done, if you say the JVM. They're basically method compilers. They look and find functions that are called a lot. Then they run the equivalent of like a GCC or Clang compilation process over those functions and things that they can inline.

They're quite traditional in that sense. A tracing just-in-time compiler is quite different. It's looking for loops, for loops, while loops, that execute lots of times, so exceed some counter and become hot. Then this is the real difference. When they become hot, they then record the actions the loop took during a concrete execution. They're very different to an ahead-of-time compiler. They then take those traces, that record of the execution of one iteration, optimize it, compile it into machine code. Meta-tracing, and this is the weird bit that's very easy for me to say and quite hard to internalize, record what the interpreter is doing when it's executing a particular loop in the guest language. This is so weird that I'm going to try and look at this from a couple of different directions and hope that one of them resonates with you. Let's break our view of the lifetime of running a program and a language implementation into two.

I'm going to call the first part ahead-of-time. That's what we would normally call compile time. Because a JIT compiles at runtime, that's very confusing. I've got an interpreter written in C. I'm going to compile it through, in our case, a fork of LLVM called ykllvm. You see we're quite imaginative with our names. That gives me a fairly standard executable. There's something else in there that I'll get to. Roughly speaking, not too much funny business. When it comes to me wanting to run a particular concrete guest program, I then get my compiled executable and I start running my program in it. More or less, this is just running as per normal. There's nothing really very clever going on here, until a loop exceeds a threshold and becomes hot. At that point, we then record what the interpreter does on the next iteration of that loop. We trace it.

Once we've recorded that iteration of the loop, two things happen. We immediately fall back to the interpreter so we can carry on executing. Because in a thread, we're going to start compiling that trace and optimizing it, and hopefully eventually getting out a compiled version of the trace. In the meantime, every time the interpreter has been probably continuing to execute that loop and said, is there a machine code version yet? Is there a machine code version yet? Is there a machine code version yet? Eventually there is. It can then, instead of interpreting your program, hand over to the machine code version, the JIT compiled code. That JIT compiled code is only valid for a subset of program behaviors. At some point, it will get to a point that it cannot deal with anymore and it will deoptimize back to the interpreter.

Again, this doesn't fully explain how this could work to my mind. Let's look at it a different way. Interpreters are basically just a giant while loop. You've got a program somewhere in memory. You might call it the bytecode sequence or opcodes. Whatever you want to call it. You've got a program counter. You start at the beginning. Every time you go in the while loop, you look at the current instruction at the program counter, see what it is, and do something based on that kind of instruction. If I've got a variable lookup function, I'm going to look up the variable, push the result on the stack, increment the program counter, go to the next instruction. If I've got an add instruction, I'm going to pop two things off the stack, add them, push it back, increment the program counter, move on. Interpreters are just this pattern over and over again.

They're very unimaginative. There's only one other kind of instruction we need to consider today, which is some sort of conditional or if statement. Here I'll use jump less than or equal as an example. You can think of this as just an if statement in disguise. In this case, we're going to pop a value off the stack. If it's less than or equal to zero, we're going to jump to some label. We may jump far away in the instruction sequence, forwards or backwards. If it's greater than zero, we just increment the program counter by one, carry on to the next instruction. I'll explain why I'm pulling this out as a particular thing later. Just bear in mind that it's different than the others. Because of this pattern, you can always find in an interpreter a loop that is this core loop. This is the thing that we're processing.

If I have a guest program, who cares about the syntax at the top Python syntax, Lua syntax, whatever. Let's imagine that little guest fragment is in a loop that's executed often enough that it's become hot. We're now going to trace it. Let's imagine I've written a tracing JIT compiler. Not meta-tracing at first, just tracing. I will have manually written a function that records all of the opcodes that were taken during that iteration of the loop. In this case, I've gone down the true branch of that statement. I've actually executed y as y+3. You can see here I've actually recorded a concrete execution. I haven't included the false branch in my trace. In a meta-tracing system, and I hope this syntax reminds you of the previous slide, I'm going to record the actions the C interpreter took. Hopefully you can see that's the code you saw in the previous slide with one slight difference, the jump equal has turned into a guard.

The way to think of this, and I'll go into this in a bit more detail later, is because we've recorded a concrete execution, you can only keep executing the trace if the thing at the guard is true. In this case, we went down the true branch for the if statement, so the trace below is only valid for the true branch. If you want to go down the false branch, you're going to have to do some clever deoptimization stuff, which we'll get to a little later.

How do we actually get from a C program to this running system that can do all this stuff? I've alluded partly to it that this fork of LLVM, ykllvm is part of the way we do it. It's more or less, from a makefile sense, a very easy thing to drop in. Instead of just calling your compiler directory, we have a little wrapper script. You tell it if you want to release a debug build. We pass some flags to Clang, which will often be the intermediate frontend you're using for various reasons that I will allude to later. It's very easy from that perspective. It does a couple of things. One of the most obvious, and this is not something you can see at a source code level. What I'm going to show you now is something that's in the binary, but you never see at the source level.

Inside your interpreter, it inserts recording functions. When you start recording a trace, these record functions will tell you the path through the interpreter you took. You may end up with a sequence like 0102. That tells you you did a lookup instruction and then an add instruction. Very simple. The other thing it does is that it takes LLVM's IR. That's its intermediate representation. If you think of that as the internal data structure the compiler has built up about your program, like an AST or something, we convert that into another slightly simplified representation, serialize that, and put that into the binary. Here's a little chunk of LLVM IR for yklua. Lots of detail. We convert more or less that chunk into our slightly simplified IR. The only real difference here, and we'll come back to this, is these safepoints, they weren't in the previous IR. We've actually added a little bit of information in some way.

This is the text representation. The binary representation of that is then dumped into the executable. When you load the interpreter at runtime, you've got a normal copy of the C code compiled into machine code, plus as a second part, some representation of the IR. Then at runtime, we take those IDs we recorded for the trace, 0102, get the fragments of IR that we've serialized, and stitch them together to make a trace that we can optimize and compile. That's how we close that particular circle.

How do we go about optimizing a program? Because these traces can be quite long. Hundreds of instructions long, potentially, and still worthy of doing something with. I'm sure some of you will think, I bet you this is all about inlining. That's definitely a thing. The idea of inlining, if I call a function f, the compiler will basically copy the contents of function f into the function that was calling it. That does two things. The obvious thing is, it optimizes away some of the overhead at runtime of calling a function. I don't need to set up a new frame on the stack. I don't have to deal with moving registers for whatever your crazy ABI wants me to do. Tracing naturally inlines. It doesn't even think about that. It does all that. That's very good. The main thing inlining allows you to do is expose optimization opportunities that you can see when you observe a function in a wider context.

Then you can run some standard compiler optimizations, constant folding, strength reduction, we have dead load, store analysis, value sets, and so on. We don't actually have as many as a typical compiler because we haven't got there yet. Fairly standard things. What really matters is that the interpreter can expose information it knows about the language and/or typical programs to yk that can then use that to optimize further. The classic example is, you may know that in your language some things are immutable or change so rarely that the system can assume they're immutable. As long as it can back out and deoptimize to a more general thing, it's probably worth optimizing on that basis. For example, you might know that once someone has compiled a function into an opcode sequence, that can never change, or changes so rarely that it's not worth doing.

Let me show you this concretely. Here is the standard Lua VM. Let me show you a little function that we've added, and this is R4. This is yklua. This little four-line function that I've inexpertly highlighted here, that function allows yklua to inform yk that the opcodes for a function Lua don't change. There's this little attribute, this annotation up here that says yk_idempotent. I'm just going to comment that out, recompile it, and run it and show you what happens. We recompile it. Linking takes a little while. Then we'll run our Mandelbrot program again. Notice this isn't yklua. It was 0.8 seconds before. Now it's 2.93. It's not quite as slow as the normal Lua implementation, but we've slowed it down 3.5x, just by removing that hint. This gives you a sense of how powerful it is when you can express these hints about the program and the way it's run to the system.

However, the problem with the one that I've just showed you, yes, it's very effective. It's so effective that if I actually tried to look at it in detail, we would be overwhelmed by detail. I'm going to show you a simpler one where we can actually as humans get our head around what's happening. We are going to look at the ADDI instruction in Lua. This adds an immediate to a variable. If you see a statement like x + 1, it becomes the ADDI statement, provided the integer is a small value. I think it's a signed 8-bit value from memory. We'll get to that. We've done something that I'm going to try and demonstrate and then explain to you briefly how it works. Here is a very important program. It's a profound program. It's a loop that counts and prints out a value. Bearing in mind that where we see sum + 1, that integer can be a signed 8-bit value.

Would anyone like to choose a value between 50 and 127? 64. Remember 64, because we'll need to see that. I'm going to run this program. I'm going to turn on some debugging output so we can see what's happening. We run it. I have no idea if that's the right answer. It looks plausible. When I look at this, what we're looking at here is both the trace IR, so this is post-optimized, and the x86 code that's led to. The first thing here, there's the constant 64. Trust me, I'm not lying on this one. It really was the value. I don't have 127 programs lurking around. Somehow, we've burnt that into the trace. Even more interestingly, you can see there's a pattern of instructions here. We load the 64-bit value from memory. We add 64 to it. We store that new value back to the same heap address.

We've been able to optimize that, as you would hope, to a single x86 instruction. That 0x40, so, whoever chose a power of 2, very good thinking. It's a nice round number in hex. We really have been able to burn that Lua value into x86 machine code. Again, here, as we know, as the programmer, if we see the ADDI instruction, that's really a constant integer. We want it to become a constant integer in the machine code. It doesn't take much work. The way the OP_ADDI code works, it defers to this macro called op_arithI. Let's have a look at that. What I'll do, I'm just going to delete this very briefly. This little chunk of code you're seeing here is the normal Lua VM. All we did was we added a simple call to a function called yk_promote. What yk_promote says, give me a value anywhere between 8 and 64 bytes, effectively.

I will now say that value is constant in the context of this trace. I'm going to leave a guard behind to make sure that if you've got that wrong, ID optimizing can cope with the situation. That's all I had to do to turn 64 written in the Lua source code into a 64 immediate in x86 machine code. Again, I'm exposing constantness that I know about the language to yk as the system. This is something that JIT compilers love to do, and tracing JIT compilers get to do a lot of it.

I kept mentioning guards. What I've said is the trace is valid as long as the value we see the next time I execute the trace is true. If the guard evaluates to true, I keep going. What happens if it becomes false? We've now got to deoptimize. We seem to have really made a rod for our own back here, because we've got to deoptimize back to, let's say, normal C code, obviously compiled into machine code. The way this works is roughly as follows. I've got a stack of frames when I start a program, so I'm going to start with a frame on the stack for the main function just as normal. It's probably going to call a function called something like interpreter. You might imagine that when I execute the JIT compiled code for a particular loop, I might create a new frame on the stack.

That's the obvious way. Actually, these two frames get mixed together. When you move into just-in-time compiled code, it takes over the C interpreter stack. It reads and writes from some of the same places. It does grow the stack additionally when it needs to because it's obviously doing additional work. Somehow, when we hit a guard that's false, we've got to take the system that looks like that and put it back like that so we can jump back to the normal ahead-of-time compiled code from Clang. How can we do that? Here's that yk IR we saw earlier. This is the thing that gets serialized into the interpreter. I mentioned these safepoint lines. In essence, wherever there's a conditional branch or a function call, you get a safepoint. These are the points in the system where control flow can diverge, and you might have to deoptimize. You can see from the first one, there's a list of variables.

Those are the live variables at that point in the program. We've got to put all of those live variables back in the same state that Clang compiled the ahead-of-time binary for. Associated with each safepoint, there is some LLVM, the concept of a stack map. It will tell us things like the following. The variable %0_0, that's going to live at stack offset 0x40 and RAX. We have to put those values in those places. It might say the variable %0_1 is the constant line. Great. That's perfect. We don't have to do anything. It might say that %0_2 is in two registers and so on. This deoptimization thing puts the system back in all of these places, jumps to the right place, carries on.

Technical Challenges

There's an obvious problem or two. One of them, if I'm executing in the just-in-time compiled code and I have C code that takes the address of a variable on the stack, it might be down here. Then when I go back to the interpreter, that address is not valid. How am I going to deal with that? For better or for worse, we have to have a second shadow stack on every thread. We'll call the normal stack the C stack. When you take the address of a variable, that will end up on the shadow stack. There's a frame for every function in the normal program. Those addresses are stable. Even when you deoptimize, you've got an address from the right-hand side, not from the left-hand side. The system doesn't know that it's gone into JIT compiled mode or gone back to the C interpreter. It's completely transparent.

There are, though, consequences. A shadow stack slows things down a bit. Stack maps, they're not exactly an afterthought in LLVM, but they are a less loved feature. They haven't received as much polish. There are some optimization passes that don't respect them. Some of the code generators unfortunately put instructions in places that they shouldn't. We actually have to turn off some optimizations in order for this stuff to work reliably. In fact, we actually do pay a penalty for this, and also those record functions you saw earlier. Roughly, and this is going to vary, the interpreter without the JIT on is going to be 3x, 4x slower than normal. When you saw a performance advantage earlier, the JIT is winning back that cost and more. That's what we have to overcome.

I want to give you a sense of a different challenge, because some of you will remember, I'm sure, that there was a tracing just-in-time compiler way back in Firefox, its first JavaScript VM called TraceMonkey. TraceMonkey was relatively quickly replaced because it didn't have great performance. It wasn't a great VM. People have then, to my mind, overgeneralized from that that all tracing compilers are bad. I hope I've shown you here that that's probably not true. There's probably a context where they're a good idea. Not for everything, but some places. Let me give you a flavor of why people might have thought that there are some really bad issues. I'm going to show you a bad case for tracing. Here's a Lua loop. Let me ask you a question. Is this a loop? You can guess from the horrendous tone in my voice that I've asked you a very leading question.

Because ahead-of-time, this is obviously a loop. You're scared. You know that there's going to be something fishy going on in a bit. Normally, that's obviously a loop. It doesn't transform into a flying frog or something. From the point of view of tracing and iteration, the answer turns out to be a lot weirder and harder and boils down to maybe? Here are the opcodes that I might get out of the Lua dynamically compiling opcode thingy. I've changed this a bit, but it gives you a flavor. I've got start and end labels. Look up a variable. Jump less than or equal instruction and so on. Let's say the loop has become hot and I start tracing the next iteration. Two things can happen. I'm going to start at the top, and if i is greater than zero, the jump less than or equal just falls into the next instruction.

We go down to the end of the loop, jump back to the start. We have now closed the loop. We've traced an iteration. A trace optimizer as in yk will eat this for breakfast. It is so happy. It will do an excellent job. You're going to get really good performance. You might get unlucky. If i was zero minus one or whatever, that jump less than or equal would immediately jump past the body of the while loop to the end. We're still recording. We aren't going to close the loop. In fact, if you're unlucky, sometimes you go for miles and miles, and via some torturous thing come back to the beginning and you end up with these ludicrously long traces. Fundamentally, there is no way around this. You can ameliorate the problem so we have a branch that's not finished where, in essence, we say, if you got here and you started there, that's probably bad and you should give up.

You can't ever fully fix it because you start recording not knowing what you're about to execute. When you get bad traces of whatever kind, your performance is much worse. I think this is one of the reasons why sometimes people get the heebie-jeebies about tracing. I think we can ameliorate it, as I said, but you can never fully get rid of this. This is a real challenge.

Recap

Let's go back over what I've shown you. It is possible to take in a C interpreter and more or less for free with very few lines of code change, get a just-in-time compiler out. Although I've cherry picked the data a little, you can get some sense that it can speed things up. I'm presenting alpha quality software here. It's not production ready, but it is way beyond your average research prototype. I hope that we're able to keep pushing this further.

What's Next?

What's next? We don't support all of LLVM. There are things, if you say, you use vector instructions or vector types, we just go, to do. Fixing that is not rocket science, but there's a lot of engineering behind it. We would like to do more optimization. Escape analysis is a really nice optimization as an example, where if you've got a loop and you allocate some memory, read and write to the heap, free the memory in the same iteration, you can often just not do the allocation at all if that memory pointer hasn't escaped. Very effective in trace space compilers. We would like to do those sorts of things. They will give us quite a bit more speed-up. I would think our geomean in yklua, probably 3x seems quite plausible. We've got MicroPython as a work-in-progress interpreter. I think we'll probably look at Pico Python next. Hopefully, we can then graduate to the big boys, as it were, the CPython, CRubys. In one sense, there's nothing intellectually different about those, but they're bound to use all sorts of features we just haven't had a chance to implement yet. That's what's coming next. You can download and play with this yourself. It's open source. You get to keep all the pieces.

Questions and Answers

Participant 1: I'm sort of a Java developer background from pre-millennium. This feels very much modern HotSpot with the decompiling, going back into compiled mode, which is very cool. Do you have to do machine instruction setups for each language, like for each interpreter?

Laurence Tratt: HotSpot's the standard JVM these days. You and I are at the vintage, where you probably remember before HotSpot was a thing. HotSpot is probably the best just-in-time compiler ever, best VM that includes actually now three JIT compilers. Probably the best one ever implemented. It's astonishingly effective. You may remember from many slides here, I mentioned Jython. That is Python running on the JVM. It's really slow. It's interesting. HotSpot is not a great vehicle for running some kinds of languages, which is why there's room for a project like this. yk is doing all of the machine-specific stuff for you. You don't need to know if you're on x86. We only run on x86. Imagine we had written an ARM backend. Your interpreter would then JIT compile on ARM or whatever. That's all totally transparent to you as the interpreter author. You will have to adjust your interpreter a bit for yk, because you want to expose this constantness thing, and you have to tell it where loops are a bit. You do have to do some adjustments, but they're not platform-specific in the way that you might fear.

Participant 2: Just so I understand something here. It looked like when you were modifying the Lua code, was that the Lua compiler that you were modifying?

Laurence Tratt: That was the Lua virtual machine, yes, the Lua interpreter I was modifying.

Participant 2: Then does that mean that if they release a new version of Lua, you would have to reapply all of your changes to that new version?

Laurence Tratt: Yes. There's 400 lines of code. I reckon I can rebase that in an afternoon.

Participant 2: That's how you can still argue that you can update and drift. You can really easily create a new version of the JIT.

Laurence Tratt: Exactly. Yes. I think that's what's really quite different here. There are a few lines where we've invasively changed things, but actually, those are generally the stable parts of the system. We don't really change the libraries and so on a great deal. The hope is that over time, keeping this up to date is a relatively easy effort. Relatively.

Participant 3: I'm also new to JIT. I had a question about the underlying C code where you add the guards and the checkpoints. I presume, does the overhead of that just go to zero because of the branch predictor that you're always hitting the safe lane until you don't?

Laurence Tratt: Branch predictors are really good. They aren't perfect. It takes them a while to learn your code. Interpreters themselves, actually, in their traditional implementation, are actually surprisingly branch predictor unfriendly. There is a concept which is called threaded dispatch. What you would normally do, you remember I had that big switch statement, you would go back to the beginning of the switch statement and then dispatch. What you do in threaded dispatch is you look up where you're going to go at the end of each opcode and go to it. That makes it more branch predictor friendly. No, the overhead of all of those things never goes to zero, although they can be very effective. What I haven't fully been able to show you is, when we compile a trace, many of the guards will disappear completely. We're able to optimize them away because you can prove from the constantness. When you saw that three-and-a-half slowdown, probably the three-and-a-half slowdown had five times as many guards in as the faster version. If you take even a little bit of overhead for some of those, if we're able to get rid of that, it's part of the way that the performance adds up.

Participant 3: If you have fewer guards and you do go down the wrong path, you just have a much worse picture to reconstruct for the deoptimized.

Laurence Tratt: Yes, indeed. Yes, that's when we have to deoptimize, put the system back, because the normal interpreter can handle any program behavior. That's when we have to go back to it as the general source of truth.

Participant 3: You could almost push that to the limit of no guards. You can almost have one guard per program, if you're really confident.

Laurence Tratt: I have never thought of it like that. I don't think so, because, again, that would require you to look forward. You'd have to know that in a future state, you're going to violate the trace you've recorded. I think you do need to see them as you're going along.

See more presentations with transcripts

The Five Stages of AI Maturity in Engineering Organizations - Where and Why Teams Get Stuck

Lizzie Matusov — Mon, 03 Aug 2026 16:00:00 GMT

Summary

Quotient CEO Lizzie Matusov explains why soaring AI spend often fails to improve software delivery. She presents a research-backed AI maturity framework designed to help engineering leaders move beyond vanity metrics like token usage, align organizational AI adoption, and address critical bottlenecks across the software development life cycle to deliver measurable business outcomes.

Bio

Lizzie Matusov is the co-founder and CEO of Quotient. She also co-authors Research-Driven Engineering Leadership, a newsletter that translates academic research into practical insights for engineering leaders. Before founding Quotient, she worked in engineering roles at Red Hat and Invitae.

About the conference

Transcript

Lizzie Matusov: My name is Lizzie Matusov. I am the co-founder and CEO of Quotient. We spend a lot of our time studying bottlenecks in software engineering, especially as those bottlenecks change with AI. I think one truth that all of us have felt is that innovation is moving very fast. If you're feeling like this movement is almost unprecedented in its nature, know that you're not alone in that feeling, because AI spend is moving faster than really anybody forecasted. Just in 2026, we've seen a 47% year-over-year growth in global AI spending. We are now projected to spend $2.5 trillion on AI, and 72% of enterprises are running at least one AI workload in production. That pace is exceptional. I think it goes to show that at this point, AI is a necessary part of how we build software. The thing is, when AI spend is moving faster than anyone has forecasted, that also means that we're blowing past all of our enterprise's expectations of that spend.

Let's take Uber, for example. They recently announced that they had burned through their entire 2026 AI budget in just four months. That's one-third of the amount of time. Of course, monthly adoptions skyrocketed, especially moving from December over to March. Curiously, when asked about the return, the CTO said that, frankly, they could not find a stable equivalent relationship between the expenditure and the productivity output. Even more recently, Microsoft just announced that they're going to be dropping Claude Code by the end of this month due to the budget and the way that they blew right past it. Again, if you read the transcripts, they're not really able to tell you their ability to say what is the ROI for the AI use. The thing is, we know we're getting value out of AI. All of us here can talk about the way that we've become more efficient, how it's become indispensable in the way that we develop software. Then, why is it that individual usage is not translating to organizational outcomes?

Theory of Constraints

My goal is to answer that question for us today. I think to help illustrate this problem, I want to help us think about it through a different lens. Let's take a plant factory. The basic concept of a plant factory is pretty straightforward. Raw materials go in, it moves through a series of steps in production until it hits the final product. There's this famous story in the "Nineteen Eighty-Four" book called The Goal, where there's a plant that's in trouble, and the factory manager really wants to find a way to save the plant, so she comes up with an idea to maximize efficiency at every station. Her idea is, run every machine at capacity. We hope that this might increase production, will keep everyone busy, and that will save the plant. What happens is things don't get faster. In fact, things get slower. They start to break.

What ends up happening is that we've discovered there's a bottleneck. The bottleneck in one of their processes was the rate limiting step. Even though they tried increasing throughput, they couldn't make it past the limitations of that single bottleneck. This is the theory of constraints. Every system is limited by a single bottleneck. If you're trying to speed up everything else, it's not going to increase the output if you don't actually address that bottleneck. Three things will happen. One, your work in progress inventory will explode, oftentimes piling up right behind that bottleneck. Two, your lead times get longer, so things will enter into the system and take longer to get through to the final product. Three, quality problems will compound. Now you're increasing strain on the system and different steps of the process.

If this feels a little bit familiar, it should, because it's also how the software development life cycle works. We've designed the software development life cycle to go through numerous gates to make sure that we're developing high quality software to the customers that we serve. We make sure that code goes through testing, reviewing, different deployment gates, and production support to make sure that we deliver excellent software. The theory of constraints holds here, too. Let's say that you have a bottleneck in the code review process. Then just increasing the number of tokens we spend or code that goes through the system might not necessarily translate into higher throughput if we're not also addressing the areas where the software gets stuck. The thing is that bottlenecks can look different for every organization. I'm sure a ton of us are experiencing challenges in the code review process, but it could be something else.

It could be how we test. It could be how we deploy. It could be any number of things. Even across different teams within the same organization, your bottleneck might be different. Actually, the DORA report in 2025 also found this. They did a study of about 5,000 technology professionals, looking at how they're leveraging AI. They found that while individual effectiveness went up at the top, software delivery throughput did not meaningfully change. In their analysis, they posit that this is because the systems surrounding our individual usage of AI has bottlenecks, and that we haven't properly addressed those to help make sure we're getting end-to-end gains.

Roadmap

I think I've sufficiently convinced you that the gap does exist. Now the question is, how do we solve for it? In this talk, I'm going to convince you of two things. I want you to think about how the most effective engineering organizations do two things. First thing they do is thoughtfully improve AI usage across the software development life cycle. The second thing is that they resolve the bottlenecks that limit their outcomes. We'll talk about these a good bit. Let me give you a brief roadmap for our discussion. The first thing that we're going to do is I'll introduce the AI maturity model. This is a structured, research-backed framework for how engineering organizations advance their AI usage. All of you are going to recognize your team somewhere along this maturity curve. Second thing I'll do is put the framework into practice and walk you through how do you identify your stage and set the right near-term goal for your organization. Then the third thing we'll do is measure outcomes. We'll talk a little bit about how we move away from looking at just usage and look more towards how we measure outcomes so that we can identify those gaps and the bottlenecks that might exist.

The Five Stages of AI Maturity

The five stages of AI maturity is a framework for how organizations advance their AI usage thoughtfully. Before I get into the details, I do want to spend just a moment talking about our model design. The reason why is because there is a lot of guidance and best practices, and they are changing very quickly. Oftentimes, I encourage people to think about the incentives of who's generating that guidance. To give an example, if Jensen Huang is saying that he would be deeply alarmed if his $500k engineer didn't consume at least $250k of tokens, you might want to ask yourself, why is Jensen Huang giving that advice? The reason is, his company is going to do exceptionally well if everybody takes that advice. We might see in a few years that it actually did make sense for each engineer to spend half of their engineering payroll on tokens.

I would argue that today, that's likely not the case. We worked and built a model using a first principles approach that would allow us to minimize bias and focus on the goal of improving engineering outcomes. What did we do? We first did a synthesis of about 15 different research papers coming from universities, research arms of organizations like Microsoft and Google, and independent but peer validated research. We looked at papers focused on AI's impact on developer productivity. We also were careful not to go back too far. Anything more than six months feels a little bit outdated, especially with the inflection point that came at the beginning of this year. Second thing we did is we did researcher interviews. We worked with the researchers who designed many of the productivity frameworks you maybe heard of, like DORA and SPACE, to get their input and also to see what are they thinking about today, and where are their research directions taking them.

The third thing we did was field discovery. We spoke to over 100 organizations that are prioritizing AI as a top goal for their organization this year. As you can imagine, there was no shortage of companies to talk to and collect data from. The model describes five stages. Moving from stage one, which is ad hoc experimentation, all the way through to stage five, which is end-to-end autonomy. As you're thinking about each of these, I'll give you a bit of a guide. Each of those five stages is defined by six characteristics. First one is enablement. That's learning and skill growth. Policy and governance, so organizational guidance and guardrails. Validation and testing. What is the quality of AI-generated work? Embedding and workflows. How well is AI integrated into how we work? Workflow automation, which is the triggering of deployment workflows. Finally, data context and access. The internal data that's available to AI systems. Five stages, six characteristics. Let's get into the details.

Stage one is what we would call ad hoc adoption. This is the earliest stage of AI maturity, where usage is really driven by our individual experimentation without any formal guidance. Think of this as the window of time when AI tools were first hitting the market. We were all just trying out different things to see what worked. We would paste in whatever context into chat windows or maybe early usages of Copilot, and really just trying to figure out, what is this tool? As far as enablement, again, that individual experimentation is what's guiding enablement. For policy and governance, there's really nothing there because we don't yet know how AI is going to impact our organization. As far as validation and testing, it's very informal. Developers are self-reviewing. They're getting outputs from chats and looking at that and saying, LGTM, throwing it into their system, and then leveraging that.

As far as embedding and workflows, AI is living really outside of those normal workflows. For workflow automation, it's manually triggered. As far as data context and access, AI can't see anything other than what you are providing within maybe a context window. All of us started here. Maybe for a small handful of people, your organization might still be here, especially if you're in a very regulated environment or you have very secure systems. For most of us, we then moved on to stage two. Stage two is what we would call assisted development. AI is beginning to spread across teams as the org is enabling that early adoption. There's likely some amount of tool sprawl as we're figuring out, do we try this tool or that tool? Which workflow works better for us? Engineers are using AI to assist them with their own personal tasks. As I'm writing code or I'm writing documentation, I'm leveraging AI to help me with my personal tasks.

I think the way stage two feels is that engineers are really empowered to use AI and support them, but it's really individually focused. We also see the bifurcation of usage start to happen in stage two, where you have some engineers, probably the folks that we would call early adopters, who are really leveraging AI, pushing boundaries, figuring out the right use cases, and working through it if it isn't perfect the first time. Then you also start to see the emergence of skeptics who try it once, twice, three times, and say, "It's not for me. We'll try something else." In enablement, you see informal learning and experimentation is really driving how we work. There's early emergence of best practices being formed, but again, we're still trying to figure out, how is this best suited for our organization? As far as policy and guidance, there's a basic policy or maybe a limited rollout, but overall, we still have it in a draft state because we're refining the way that we use it.

For validation and testing, we've got basic checklists for AI generated work, but for the most part, things haven't really changed. For embedding and workflows, developers will manually trigger AI to assist them. Again, workflow automation is pretty limited. It's helping us with our individual tasks, but it's mostly manual when we're asking for AI. Again, data context, mostly user supplied and a little bit limited. I think what's important about stage two is that the harnesses or the systems around the code that we write mostly haven't changed or adjusted for the increased volume that AI is going to generate across the SDLC. Once we do start thinking about those harnesses, that's where we move to stage three.

Stage three is what we would call standardized workflows. This is where AI is becoming more embedded in development workflows across the teams. It's supported by shared organizational best practices. I think the key word for this stage is the word shared. Teams are starting to adopt consistent expectations. You might still see some variance from team to team. Overall, there's evangelist teams who have figured things out that are now sharing these best practices to the other teams. We're starting to see real standards form across the whole engineering organization. What's important about stage three is that the shift really starts to happen from the individual to the team and the systems around how those teams work. What does this look like across our characteristics? As far as enablement, you start to see formal training really come to light. If you've done the hackathons, the knowledge transfer sessions, the learning groups, those are really key traits of stage three.

For policy and governance, there is a clear or wide policy. Engineers have a good understanding of where they can and can't use AI. It helps guide their own autonomy and exploring. For validation and testing, you'll see that automated checks will start to validate AI generated code. The systems are really starting to adapt to, again, the increased volume. Embedding and workflows, you'll start to see that AI is embedded with some consistent expectations. It's not just supporting the individual. It might be part of the development process, or linting, or the code review process. It's not something that always has to be manually triggered. In workflow automation, you'll start to see that it does get auto-triggered for defined tasks. Finally, for data context, it can now start to access parts of code and docs beyond just what you supply to it. There's a pretty big jump from stage two to stage three, because this is where we're starting to focus on the development harnesses around our code. That's a lot of investment that starts to happen, especially as AI is starting to become automated across the software development life cycle. Stage three is where those bottlenecks start to become very visible. As you start thinking about those bottlenecks, that's really where you want to unblock them to get to the next stage.

I am many minutes into my talk, and I did not say the word agent once. That is because it is really important to think about the infrastructure from stages one to three, which will enable you to get to stage four, which is what we would call supervised automation. In stage four, agents really start coming into the scene. To start, engineers are configuring agents to execute bounded tasks across the software development life cycle. They'll handle lower complexity tasks with high oversight. A great example of a stage four automation would be if you've configured an agent to pick up a P3 support ticket, look at it, maybe generate the PR. You still have a human that's reviewing it, making sure the tests pass, and then ultimately deploying it. In order to be confidently getting the benefits of stage four, you want to focus on the infrastructure that you've built in stages one through three to help support you, because you're handing off control to agents, which as we've seen, it is extremely powerful.

It's also pretty risky. When we think about enablement, AI literacy is widespread. This is really a time where you've got lots of engineers thinking about different ways to solve problems and automate our tasks. Policy and governance is not only defined, but it is now aligned with the company's broader strategy. We're not just using AI for AI's sake. We understand how leveraging AI allows us to support our business goals. For validation and testing, you've got dedicated agents that can validate AI work before release. Heard some great strategies here, too. For embedding and workflows, agents are now embedded in delivery workflows. Again, they'll be quite bounded. There isn't necessarily a limit on where they can exist within the SDLC, and we're starting to play around with different areas. For workflow automation, we find that AI is now executing bounded tasks, but the human is still approving. Then for data context and access, AI retrieves relevant context automatically. I think that the most important thing to call out with stage four is that you really start to see the control of the software development life cycle shift away from the engineers to the agents. Especially as engineers are building more trust with agents, it's becoming more pervasive throughout the SDLC.

As that trust and autonomy of agents grow, that's what allows us to eventually get to stage five. This is what we would call end-to-end autonomy. AI orchestrates end-to-end multi-system, multi-agent workflows across engineering systems. Humans are now focused on oversight, high leverage decisions, and managing exceptions. This is that truly AI native feeling. The way that we describe that feeling of stage five and how you know you might have it is, think about the ownership and autonomy that each of you has as an engineer to develop software, test it, approve PRs, merge production incidents. When you've given agents the exact amount of ownership and autonomy you'd give to a senior engineer, that's when you've really hit stage five. The thing is, it is at the very tip of innovation, which means that the pace of change is really quite high. When we think about the characteristics, the theme here is continuous iteration.

Enablement looks like continuous learning that is embedded into the culture. If you're not constantly learning about new ways to use agents and the way that they can support your systems, you might actually miss the opportunity to better secure your software or better leverage those gains. Policy and governance. You're now seeing organizations continuously evolve their policies as systems evolve. Validation and testing. There's continuous production-linked feedback loops. Embedding and workflows. You're seeing that AI is now coordinating complex workflows end-to-end with autonomous execution. In order for it to have the same amount of ownership and autonomy that we do as engineers, there's a structured org knowledge graph that allows it to basically access anything that an engineer could.

How to Identify Your Stage, and Set the Right Goal

I want to talk a little bit about how to actually identify your stage. I'm sure as we went through these stages, you could feel yourself pulling in the direction of a stage, or maybe there's one characteristic where you're like, we're definitely here. In a different stage, you're like, but we're definitely here. I want to talk a little bit about how you can evaluate what stage your team is at and how that compares to your own company. There are three steps to diagnose your stage. The first one you want to do is score each capability. For each of those six capabilities, you'll want to identify the stage description that best describes your team. I'll show you an example, but just to finish that thought, your organization's maturity might be a little bit different than your own team's maturity. The reason why is because, again, we're in a room of people who are listening to and trying to learn the most cutting-edge information about AI.

Your teams might be a little bit ahead. Your organization's AI maturity is representative of how an average team would respond. Let me show you what that looks like. Here's a really basic table that shows the different stages and capabilities. Let's say you're going through this. You're like, let's start with enablement. We've got formal AI learning. I would say we've still got a little bit of bifurcation of usage, but we've got those hackathons, those knowledge transfer sessions. We've got that. For policy and governance, we have pretty clear expectations. I wouldn't say they're maybe aligned to our company's strategy, but I know when to and not to use AI. For validation and testing, we have basic checklists. If I think about it, we haven't quite yet designed the testing harnesses around our software to be representative of the increased volume we're seeing with AI. For embedding and workflows, we are starting to play around with agents.

They're quite bounded across the SDLC with a lot of oversight. That's ok. Again, we're manually approving the things that our agents do. Then for data context and access, maybe we work in a more regulated environment, and our IT teams have been a little uneasy here, so we've limited the access to be partial. If this looks like your organization, you would be at stage three. Again, there's parts that are at stage four and parts that are at stage two. Overall, you would rate yourself as a stage three. Again, you could do the same thing for the average team in your company. That's what would give you your company's stage. The second thing is to resolve your weakest capabilities. You want to choose the one to two characteristics where your stage is the lowest, and make investments to improve them. If we're looking at this same graph, we're much more interested in improving our validation and testing characteristics than we are about expanding our embedding and workflows and our workflow automation.

This is really important because when those gaps start to form, that is a perfect place for bottlenecks to live. Also, as you move into stages four and five, it becomes really important for you to have that necessary foundation set so that you can give agents the autonomy and ownership that we're all hoping for. The final thing is to pick that next-stage unlock. Find the one or two capabilities that if leveled up would really move your organization up to the next stage. I'd venture to say that most folks are thinking about those next stage unlocks. I don't have to tell you guys how to do that one.

What is the right stage for your org? I think if we looked at the headlines, we would see that every organization is moving to level five, end-to-end autonomy within the next two months. I think we can do it. We're in a room of the folks that are actually building these systems. We know that the truth is a little bit more nuanced than that. In our work with dozens of companies, we actually found that the average big enterprise is somewhere between stage two and three. The average growth stage organization is somewhere between stage three and four. This makes sense. If you think about a big enterprise, they've got a lot of things. They've got increased complexity. They have more people, more processes, and oftentimes more regulation, especially if they're publicly traded or have some certifications like FedRAMP. If we talk about a growth stage company, they tend to move a little bit quicker.

They have a little bit lower process, a little bit lower complexity code. This is why we start to see that shift happen. What's interesting is that we don't often find people at the polar opposite poles of stage one and then also stage five. That's because the risk is not monotonic in maturity. When we first designed this model, we were very nervous about calling it a stage model because we thought that everybody would just go straight from one to five as fast as they can get there. Actually, on both ends, there's risk to consider. At stage one, there's two issues. One, there is a shadow IT risk. You're pasting company context into a chat window that lives outside what the company knows about. That can get a little bit dicey at times. There's also an innovation risk. Most organizations are at or well past stage two.

If you're seeing this and thinking, my company is at a stage one, I would consider the competitive risk of folks in your industry that can move much faster. On the other side, stage five is pretty risky as well. The reason why is because agents should be able to do everything that humans do, and probably at a rate that's much faster than we were doing it before. If you're going to relinquish your control to agents in this manner, you better be confident that the systems in place are going to actually maintain the quality and integrity of the software you build so you can maintain trust to your customers. Because humans can't manually support this load. We're probably all doing this right now and finding that there's a lot of bottlenecks that we're struggling through. That's why for most organizations we recommend targeting somewhere between stages three and four, depending on where you are right now. It's a good 2026 goal.

Measuring Outcomes, Not Usage

We've talked a little bit about the model. We've also talked a little bit about how to set your own goals within the model. I want to bring us back to our original question. How can we bridge that gap between individual effectiveness and organizational delivery? How can we measure and see it? I want to start with a brief story, if you'll humor me. There's an old adage, some of you may have heard of this, of a police officer who's on night patrol and he finds a guy near a streetlight on his hands and knees digging around. The officer comes up and says, "Sir, what are you doing?" The man says, "I lost my keys." He goes, "Quiet night. I'll come help out." They spend a few minutes digging around on their hands and knees in front of the streetlight. The officer goes, "I don't really see it here.

Are you sure you dropped your keys here?" The man says, "No, I dropped them over there." He points to a dark alleyway. The officer goes, "What are we doing here?" "This is where the light is." This is the streetlight effect. We search for answers where it's easy to find, not necessarily where we know the truth lies. We might as well start here, because, again, that's where the light is. Even if we know that it's not the best place to look and it's not the likely answer, it's at least a good starting point. That's always been the story with developer productivity and measuring outcomes. Long ago, we used lines of code because it was an excellent proxy for how much work is being done in engineering. Today, we're using tokens. Measuring tokens are easy because they are right there. You can spin up a dashboard and quickly see who is spending tokens.

It can be a directional signal to show how increasing usage is happening across the company. It is most certainly not the best way to measure outcomes. I would argue that token maxing or using tokens as a proxy for AI's impact on productivity is a much more dangerous measure than lines of code. Let me tell you why. Lines of code had an indirect but very real cost on the software development life cycle because of the higher complexities on the system, the technical debt, and the cost of managing it. The thing is that each token has a literal dollar value associated to it. When you're incentivizing that behavior, it's going to translate to money being spent right there. To show another example of why tokens are quite difficult to look at as a measure of productivity, recent study at Concordia University looked at token usage across 30 development tasks running with agents.

They found some really interesting things. Like for example, code reviews took up about 60% of token usage. That was due to the back and forth between developers and agents as they were tweaking. Then, tokens were disproportionately spent on input rather than output or reasoning. Researchers again pointed to the communication tax where a lot of our tokens are actually spent on communicating context rather than generating output. We can see how tokens provide a pretty weak signal of outcomes.

There's another principle at play here, too. I know we all know this one. It's called Goodhart's law. "When a measure becomes a target, it ceases to become a good measure." We see this all the time in the world of productivity where when it's defined by a single metric, you know that we'll get really good at optimizing that single metric to look good. I'm going to give you an amazing example of this today in the world of tokens. In a recent Substack called the Pragmatic Engineer, the author interviewed engineers at different companies who have token leaderboards. He asked them what they think. Here's the response he got from one senior engineer at a well-known FAANG company. "We have internal dashboards and metrics tracking AI usage, token usage, percentage of code written by AI, and hand-written code. I am conscious of not wanting to be seen as 'uses too little AI.' I'm not ashamed to say that I do token maxing here.

Things I do to inflate my token usage metrics. One, I ask AI questions about the code already in the documentation. The AI pulls up the documentation, processes it, and gives me the results ten times slower, all while burning lots of tokens. I could use an internal product, but then my token numbers would be lower. Two, I ask the AI to prototype a feature I have no interest and no intention of working on. Prompt it a few more times, and then throw the whole thing away. Finally, I default to always using the agent even when I know I could do the work by hand so much faster. Then, I watch it fail." It's a pretty tough example, but I think the industry is starting to catch on to the fact that tokens really aren't the best way of measuring productivity. In fact, so much so that over the weekend, Amazon shut down its token leaderboard.

In their quotes, the internal memo, the Amazon senior VP said, "Please don't use AI just for the sake of using AI. Use it to help you solve customer problems, to help you solve business problems." We still gravitate towards activity metrics. The reason why is because, like we said with the streetlight effect, it is just easy to measure. It's probably the first thing we look at. We all know that this is not a good signal of our outcomes. As we increase our AI maturity, we need to start looking at the outcomes it has on our software delivery. This will tell you if AI usage is actually translating to the end result that we want, because at the end of the day, our goal as engineers is to write software that delivers value for our customers. If we can do that faster, that's great. We also want to maintain the bar that we set so that we can achieve our business goals.

I want to show you a couple of different frameworks for measuring outcome signals. The key to this is to manage tension. Again, we want to avoid Goodhart's law by making sure that we're incentivizing the right behaviors here. The first one is very simple: speed, quality, ease. This is actually the framework that the Google research team uses to understand productivity. It powers a lot of their productivity research. It does a good job of balancing that tension of software development that we're always thinking about. At its core, speed is about getting value to customers as quickly as possible. Quality is about maintaining excellence to build and retain customer trust. Ease is about making the product development process efficient, friction free, and predictable. This is one very simplistic one that you can use. The other one, which I'm sure some of you have heard of before, is the SPACE framework.

This is a very popular framework for measuring developer productivity. It was designed by some of the most famous researchers in the productivity space, like Nicole Forsgren from Accelerate, and the Microsoft Research team. They designed this to help find friction in the software development process. Unlike the DORA 4 or 5, that really focuses on software stability versus the SPACE framework really focuses on team friction and outcomes. The five dimensions are satisfaction and well-being, performance, activity, communication and collaboration, and efficiency and flow. Recently, the researchers came together for a summit. Even though this was published in 2021 before AI exploded, with all of the research that these researchers are doing, they still came back to the fact that a holistic outcomes-based and tension-oriented framework is the best way to understand AI's impact. They all recommended the SPACE framework again. To tie things back, when usage grows but delivery outcomes aren't improving, this is what can signal new bottlenecks are happening.

These productivity frameworks can help you diagnose where those bottlenecks might be so you can lift them. This is a continuous process. You'll find as you advance in one characteristic of AI maturity, that you might identify a different bottleneck in the system. That's ok. That's a normal part of improving our software processes to manage the increased volume and ownership that we're giving to AI.

Putting This All into Practice

We have covered a lot of grounds today. I want to give you some tools to put this into practice. First thing we talked about is the maturity model. Each stage and organization is going to fall somewhere in the five stages of AI maturity. We talked a little bit about how stage one is probably not super common and definitely not for most folks. Also stage five is not necessarily the best near-term goal for your whole organization. We recommend somewhere between standardized workflows and supervised automation. Each stage is defined by these six characteristics, through enablement, policy and governance, validation and testing, embedding and workflows, workflow automation, and data context and access.

Actionable Takeaways

Four actions that I want you to take with you. One, think about how you would diagnose your stage. I've got some links at the end that will make this super easy for you, but you can use the maturity model to identify which stage your organization falls in, where your team falls. Again, what characteristic you might want to invest in to improve, and then what your next stage unlock is going to be. Second thing is set that near-term goal for stage three to four. Ask your team or your organization, what would we need to do or how would we need to solve bottlenecks so that we can actually unlock that next stage of maturity? Measure impact, not activity. We know that tokens and seats are easy to count, but outcomes are what prove that we've increased productivity. Especially as the cost of tokens and the cost of leveraging AI goes up, it becomes really important to understand what is the outcome of all of that work.

Number four, watch the gap. When usage is not translating to outcomes, when you see your token spend is going up faster than you could imagine, and you look at your outcome metrics and say, "I don't see what's happening here. We're not getting that end-to-end delivery," that is your signal that there are bottlenecks in the system. Again, the most effective organizations do two things. One, they thoughtfully improve AI usage across the software development life cycle. Two, they resolve the bottlenecks that limit their outcomes.

Resources

If you're feeling very inspired to take the next step, this is one QR code that will give you access to three different things. There's an AI maturity model white paper that's an easy shareable that you can send to your team or your colleagues or your boss to help talk about this and form a language around it. There's a really quick 12-question assessment that you can use to actually identify what stage is my team on without having to run through the chart and poking dots. Also, there's a link to the latest research. Our team publishes every week a three-minute digest of the latest research that's happened in developer productivity. I will tell you sometimes a paper from three months from now will negate something we read about two weeks ago. It's a really great way to stay at the cutting edge.

See more presentations with transcripts

Microservices Platforms: When Team Topologies Meets Microservices Patterns

Chris Richardson — Mon, 03 Aug 2026 16:00:00 GMT

Summary

Chris Richardson discusses leveraging Team Topologies and internal platforms to accelerate microservices delivery. He explains six key platform patterns - from security and observability to build and deployment - and shares strategies for minimizing cognitive load on stream-aligned teams while avoiding common platform engineering pitfalls.

Bio

Chris Richardson is a software architect and serial entrepreneur. He is a Java Champion, a JavaOne rock star and the author of POJOs in Action, which describes how to build enterprise Java applications with frameworks such as Spring and Hibernate. Chris was also the founder of the original CloudFoundry.com, an early Java PaaS for Amazon EC2.

About the conference

Transcript

Chris Richardson: Welcome to my talk on microservices platforms. Really, what the talk's about is a simple idea, taking the concept of team topologies, platforms, and platform groups, which is another term for platform teams, and using those to accelerate the delivery of applications that use the microservice architecture. Basically, it's all about reducing the cognitive load of the service teams, enabling them to deliver better software faster. I've been building software for a million years now, at least that's what it feels like. It did mean that I was actually around when patterns became popular within the software community back in 1994, 1995. I was really good, and that shaped my thinking ever since.

My brain is full of 30 years of patterns now, as well as 30 years of Java. I've just done various things. I've worked on Lisp systems, created the original Cloud Foundry. Then I also, for the past 10-plus years, been pretty much focused on the microservice architecture, just helping organizations around the world improve how they deliver software. I'm excited, so my book, "Microservices Patterns," came out like 7 years ago now, which is still quite current, remarkably. I'm in the middle of working on a second edition of the book, so one day it will come out. I just think about microservices most of the time.

Microservices Platforms: Why?

In this talk, I'm actually going to walk through six different patterns that I've identified that will help with the delivery of microservices. I'm sure there's actually many more. I want to start off just by talking about why platforms, why do they matter? What are the benefits in the context of developing microservices? I want to take a step back and talk about what software development is all about, and I like to think that at the heart of software development is a feedback loop. We in IT, we develop software, and then we put it into production, put it into the hands of the users, and we get feedback from that. It's really critical for businesses today to have short feedback loops so that they can thrive in today's volatile, crazy world, which is unpredictable from one moment to the next. Ages ago, I used to talk about brick-and-mortar businesses being disrupted by digital technology.

Since then, we've had pandemics, we've had wars, we haven't had tariffs, now we do, now we don't. It's just completely volatile. Businesses quickly need to be nimble. There's a lot of research that shows, and this is one of the findings from the keynote, that the better you are at delivering software, as defined by continuous stream of small changes, the more successful the business is likely to be. Fast flow is really important. To actually achieve fast flow, you need a combination of three things. You actually need a process, or specifically DevOps, which I'm actually not going to talk about today. That's how an organization should work. You need a properly structured organization, specifically one that follows the ideas of team topologies, which is actually a collection of patterns for structuring an organization for fast flow. Then you need an architecture that enables both of those things, which, at scale, is often the microservice architecture. I'm really focusing on the organization and the architecture piece of this in today's talk.

What is this team topologies thing? That is a collection of organizational patterns and principles for fast flow. It's basically, how do you structure your organization to deliver software rapidly, frequently, and reliably, using DevOps, actually? There are several different concepts involved. The first one is there's four different team types. Most of the work is done by what is known as a stream-aligned team. That is a team that is responsible for the end-to-end flow of work that is taking requirements and turning it into code running in production. These teams are generally small, five to nine people. Those are the teams that are doing the vast majority of the work. Then they're supported by three other team types. There's enabling teams that act as consulting teams that help teams acquire new skills. There's complicated subsystem teams that focus on domains that actually involve deep skills like math, for example.

I'm not talking about them so much in this talk. Then, lastly, there are what used to be called platform teams, and then they got renamed into platform groupings. I reject that, and I just call them platform groups in the second edition of "Team Topologies." Which is what I'm really going to talk about today. You've got these. I'll talk about what platform groups do. Then there are these three different interaction styles. Most of the interactions are via X as a Service, one team consuming the output of another in a self-service fashion. Occasionally, teams need to collaborate to discover new capabilities. Then, of course, enabling teams, which are acting as consultants, facilitate with the team that is learning. I'm going to talk about these interaction types quite a bit. Then there's a set of principles, and the most important principle is about managing a team's cognitive load.

The human brain can only deal with so much complexity. Collectively, a team, a set of brains can only deal with so much cognitive load as well. They only have a certain cognitive capacity, and you do not want to exceed that. Because if you cognitively overload a team, like just in order to get the work done, there's just so much mental effort, then that actually reduces team performance, and it ultimately impacts their mental health in terms of stress, and burnout, and so on. As Nicole mentioned in the keynote, a key part of having a great developer experience is minimizing the cognitive load of a team. As you're going to see, that is one of the key goals of having a platform or a collection of platforms. That's team topologies and cognitive load.

Then in terms of what is my definition of the microservice architecture. It's an architectural style that structures an application as a set of components or deployable units, which also go by the name of services. Then those services have two essential characteristics. They are independently deployable, meaning a service can be built, tested, and deployed in isolation from other services. They are loosely coupled, or specifically loosely design-time coupled, which means that a change to one service rarely requires other services to change in lockstep. Notice in this definition, I'm not talking about claiming that services should be small or numerous. That might be implied by the name, but that's not an important characteristic. The reason I'm talking about microservices here is because they actually enable fast flow. The way they do that is that they enable the teams that are developing in them, the stream-aligned teams, to be independent.

That comes from those two properties. Because the services are loosely design-time coupled, a team can change their service without having to coordinate that change with other teams. Then because the services are independently deployable, a team can deploy their service without any kind of collaboration with those other teams, at least the vast majority of the time, which means that teams are able to work separately most of the time, which is essential for fast flow. The challenge that you have is that the microservice architecture is quite complex. If you go look at the microservices patterns language, most of the patterns there are solutions to problems that you encounter when implementing microservices. There's a lot of different patterns that you have to implement. If you look at each pattern, it's a combination of three things. Application logic, which is what the team should be focusing on. There tends to be a lot of plumbing, so supporting infrastructure code, connecting to databases and message brokers, and logging and so on.

Then there's infrastructure services as well. Teams should be focused on application logic, and for them, having to deal with plumbing or infrastructure services would be an excessive burden. In some cases, it requires deep expertise, so it would impose a significant cognitive load. If each team was dealing with this, it would be duplicated effort. You'd end up with multiple bespoke implementations, which would be a maintenance nightmare. Then, on top of that, there are these cross-cutting application-wide concerns that don't really belong to any one team. Not only that, it's not just patterns. If you go look at the source code for a service, it's not just the application code. There's build logic, a definition of the service's dependencies, and task definitions that compile and test and package that service. There's also some deployment logic. There's also the definition of the service's deployment pipeline. Then there's probably Infrastructure as Code, like Kubernetes YAML, for example, that's there to deploy the service.

You don't want the team reinventing the wheel for all of that, because that would be an excessive burden. There's even more. Not only do you have the services, but then there's all these infrastructure services, including the infrastructure that runs the deployment pipeline, services that are global, like your message broker, for example, and then the deployment infrastructure. It would be a burden for all of the teams to actually have to take on that work themselves. That's the big motivation for platforms and platform groups. From a team topologies' perspective, a platform is an artifact that reduces the cognitive load of a stream-aligned team. It could be just a wiki page, but usually it's a tool or a library or some kind of self-service, SaaS-like solution providing a capability. It's developed by a platform group, and it's consumed as a service by the stream-aligned teams. That's the big idea here.

If you look at what a platform group is, it's actually a composite team type. It most certainly would have a stream-aligned team that develops the platform, but there might also be an enabling team that provides a consulting function that helps the stream-aligned teams be more successful with that platform. Then in terms of collaboration, this is primarily X as a Service. It's all about self-service platforms, but then the teams will collaborate to evolve the platform. Then, as I mentioned, the enabling team that's part of the platform will also consult with the service teams. The primary benefit is that it reduces the cognitive load of the teams, enabling them to work on their specific business functionality and actually deliver value. Then the platform is a single standardized implementation of some important capability. It's done right just once. Then the platform group itself can focus on developing skills in that area. Unlike the stream-aligned team, which is focused on delivering business functionality, the platform team can dive deep into whatever their particular more technical problem domain is. That's the big idea with these platforms.

The Service Foundation Platform

As I mentioned, I identified six different platforms, and there's most likely more, but I thought this was a good start. I'm going to talk about each one very briefly in turn. The first one I want to talk about is the service foundation platform that simplifies the creation and maintenance of services. As I mentioned a little while ago, if you look at a service, there's a lot of plumbing, there's build logic, there's Infrastructure as Code for deploying the service. There's a lot of stuff, and actually setting up a new service from scratch would be quite a burden on the teams, and maintaining all of that would be quite a burden as well. The solution is to have a service foundation platform that simplifies that part of service development. The service foundation platform is comprised of two parts. There's the service template, which, as the name suggests, is a template that can just be cloned to create a running service.

Then there's a service chassis that is a framework that the template is built upon. The idea of a service template, super simple. It's a complete running service that a team can just copy and drop in their business logic, and they've got this testable, deployable, observable service up and running relatively quickly. Big reduction in cognitive load there. The problem you have is that a template, and this is same as true with code generation as well, it is basically glorified copy and paste. Then what's more is the template is changing. When it's time to make an update to the plumbing for the services, you now have multiple copies of the code and it's likely to be slightly different because each service was cloned from a slightly different version of the template. There's potentially a massive maintenance task in updating all of these services. That's where the service chassis comes in.

The idea is you extract out most of the functionality that is in the template into this chassis, which is this framework. Then that framework is just referenced by the template. Hopefully, the template is really small, so the amount of copy and paste is significantly reduced. When it's time to make a change to the plumbing, you just update the framework, release a new version of it, update the service template to use that version, and then update each service to use the new version of the chassis. Fingers crossed, you've just rolled that change out and it's just a one-line version update. Of course, there are scenarios where it's a bit more complicated than that. A lot of the time, the upgrades are quite straightforward. If you have more complex upgrades that require code changes to each service, which includes updating stuff that is in the service template, one option is for the teams to do it themselves.

A much better option is for there to be some kind of update or migration script or some tool that you can apply to each service's repository to create a pull request. That might be a deterministic tool like OpenRewrite, which provides large-scale refactoring of your codebase. Or maybe, and this is my obligatory mention of GenAI, because that's the only thing that's real today, you might be able to use GenAI to actually do this update. You could actually give it the diffs of the service template, and maybe it would figure out how to apply those diffs to each and every service and create a pull request for it. Who knows, because every time you use GenAI, it's a roll of the dice as to whether you'll get something usable out of it. That's the service template, or to be more precise, the service foundation.

The Security Platform

The next platform I want to talk about is the security platform. Who here, first off, understands security and likes it. Because to me, it's like a prime example of my brain does not have the cognitive capacity to maintain knowledge of how OAuth authorization flows actually work. I learn it for a while, and then a month later, it's paged out. I don't know if that's because it's complex or because I'm getting old and my mental capacity is diminished. Security is really important, yet at the same time, it's really complicated. There's a lot of different areas to that, and I just want to talk about three areas that can impact developers. One part of security that's obviously important is authentication, verifying the user is who they claim to be, logging them in, in other words. There's a whole complex mechanism involving OAuth and OIDC and IAM services, and so on, that result in services being handed a token.

Then they have to do authorization based on that token, which might be hard-coded Java authorization rules, or perhaps they delegate to an authorization service like also cloud. There's also access control lists for things like Kafka topics and stuff. This is not what developers should be spending their time thinking about, because this stuff is hard, or at least for me, it's hard. Then, on top of that, there's also transport level security. You want to secure the communication between services as well, and that involves technologies like certifical authorities that are handing out certificates and so on. That's also quite complicated. The obvious solution is to have a security platform that takes care of that. This platform consists of two parts. There's a bunch of infrastructure services that provide the security mechanisms that I just described, like an IAM service, certificate management, which might actually be a service mesh, for example.

Maybe there's the authorization service as well. You've got that infrastructure. Then there are elements that provide security that are incorporated into the service chassis so that services are actually secure by default. You build them on top of the chassis. Then, let's just say automatically every REST endpoint will require a JWT. It knows how to get the JWT signing certificate from the IAM service. The idea is that this insulates the developer from the complexities of security, and the only security aspects they need to think about are writing the authorization rules that determine which operations can be invoked by whom. That's about it. That's the security side of things.

Infrastructure Services Platform

The next platform I want to talk about is the infrastructure services platform. Services are not these standalone things. They need a bunch of infrastructure services in order to run. Most obviously, the services need database servers, which need to be provisioned and managed. Typically, even if you're running on Kubernetes, you'll probably use AWS RDS or Aurora for the service databases. Then there's application-wide infrastructure like Apache Kafka. Maybe you use AWS MSK. On the one hand, it sounds simple, but on the other hand, setting up cloud resources on AWS and other public clouds is incredibly complicated and requires deep expertise, starting with getting an access key and taking it from there. It's like really complicated. You don't want teams dealing with this, because they should be focusing on their business logic. It makes sense to have a platform to do this. Once again, the platform consists of two parts.

One part provides the shared services like Apache Kafka and other pieces of infrastructure. Then there's what I call an infrastructure orchestrator. What that does, it enables the service team to say, my service needs a Postgres database with this capacity. Then the orchestrator is responsible for taking that service specification and creating the appropriate cloud resources, which are then managed. The teams can just go, I need this. They can express their intent and the orchestrator takes care of it. There are a few different solutions out there. One that I've used in the past in the Kubernetes world is Crossplane, which if you can get past the word soup of the documentation, it's actually really interesting. It basically extends the Kubernetes API. Let's say the platform team create a new resource type like service database. That's a CRD in Kubernetes terminology. Then they define how that maps to cloud resources.

Then, a service team can just write the manifest, the YAML that says, I need a service database and it needs to be this big and so on. Crossplane provisions and manages it for you. It's really cool because the service is defined at the Kubernetes level and now its infrastructure is defined at the Kubernetes level. Perhaps it's all packaged up into a Helm chart, so you install the Helm chart. You automatically get cloud resources in that environment. I think the technology is still maturing, but it's actually really interesting. Daniel Hertz company has one, Kratix, even better than Crossplane. Really interesting stuff, but it's deep technology.

The Observability Platform

Yet another platform I want to talk about is the observability platform. As you can imagine, a team that develops a service needs to understand how that service is behaving in production, and then they also need to understand what the users are doing, and how they're actually using that service. In the microservice architecture pattern language, there's actually six observability patterns. The gray ovals are the patterns. This includes the standard stuff like log aggregation, application metrics, and distributed tracing. It's like an observable service has to implement these patterns. As you can imagine, each of these observability patterns is some instrumentation which is comprised of application logic, plumbing, plus some dependencies, and that's emitting telemetry. Then there's infrastructure services like Prometheus, or something for storing logs and so on, that's gathering the telemetry, storing it, analyzing it, and presenting it to the teams. Yet again, on the one hand, the teams probably have to write some application-specific instrumentation, but everything else would be too much of a burden.

Setting things up, yet again, requires specific knowledge or in-depth expertise. Having an observability platform provide the infrastructure services like Prometheus, or ELK, or CloudWatch, or Datadog, so it could be a SaaS, or it could be on-prem, doesn't really matter, but it's stuff that needs to be provided, managed, and so on. Then, in the service template and chassis, there's elements that ensure that out of the box, the service is observable by default. You just build on top of the service chassis. You have an observable service. The team just needs to write the appropriate logging code, and collect the appropriate business-oriented metrics, and all of the low-level details are just insulated from them. Recurring theme. The team either ignores the low-level stuff completely, or focuses on the valuable parts, and everything else is provided via the chassis and these pre-managed infrastructure services.

The Build Platform

I now want to talk about the remaining two platforms, which are actually quite connected in a way, two sides of the same coin, the build platform, which is responsible for the deployment pipeline, and then the deployment platform that's responsible for the production environment. They're actually connected, because what comes out of the build platform actually has to be compatible with what the deployment platform is expected. Let's look at the build platform. Every service has a deployment pipeline that's going to compile the service, run the tests, package it up, perhaps as a deployable unit, like it could actually build a container, and a Helm chart, test that, and then either push it into the production environment or publish it to a registry where the production environment can pull it. Every service has one of these. It needs to be set up. It needs to be administered. That can be a significant burden for the team.

Then, also, the infrastructure that the deployment pipeline runs on also needs to be set up and administered as well. If we're using GitHub Actions, you've got to have a GitHub Actions workflow file for each service. Then the organization that contains the service's repository needs to be configured with GitHub Actions runners that actually execute the deployment pipeline, and also budgets need to be set and all of that administrative stuff. Not something you really want the teams to be doing, especially when you have a lot of services that are more or less built the same way, it's like, why reinvent the wheel? Good use case for having a build platform which provides the pipeline infrastructure, properly configured GitHub organizations. Then in the service template, there's actually a templated definition of the deployment pipeline, whether that's a GitHub Actions file or a CircleCI config.yml. Then, hopefully, the actual pipeline specification is built using these reusable package components like GitHub Custom Actions or CircleCI Orbs, to actually minimize the amount of copy and paste that's involved.

At least those two have a way of providing reusable deployment pipeline logic, and GitLab has something similar as well. Then the build platform can also provide some repository/registries for storing build artifacts, shared libraries, as well as publishing deployable units to a container registry for your container images and Helm charts. Because what comes out of the deployment pipeline obviously has to be stored somewhere. With this build platform, hopefully, most teams don't have to be concerned with all of this. It just happens for them. Maybe they need to tweak the deployment pipeline if they have specialized use cases, but hopefully not so much. I feel like this is pretty standard.

The Deployment Platform

Then that leads to the last part, which is the deployment platform. To actually run an application, you obviously need to have a production environment, and you might actually have some other environments, staging, dev, QA, though unclear exactly how many you really should have. Those are a centralized place where your service is run, and they actually have to provide a rich set of capabilities. They need to have a deployer mechanism that takes an updated service and rolls it out somehow. It's hard to talk about this in the abstract, but later on, I'll mention things like GitOps tooling like Flux CD or Argo. They also need to deploy your services in a fault-tolerant way. The big idea is like, here's my service, I need to run n instances of this. The infrastructure tries its hardest to make sure that's true. That could be an AWS autoscaling group, or it could be Kubernetes deployment, depending on what mechanism you're using.

Then there's also a networking capability as well. Requests that come in get routed and load balanced across your service instances. As you probably know, or maybe you're blissfully unaware, it's really complicated to set all of that up. Then the other part of this, every service needs some Infrastructure as Code configuration to define how it's deployed. It's either Kubernetes YAML, or maybe it's some Terraform Infrastructure as Code or something worse. There's a lot of stuff. It's really funny. I mostly live in the Spring world. When I step outside the Spring world and I go to the frontend, I'm horrified. I have been for many years. Then, over the past few years, I've been doing more of this DevOps-y stuff, and I'm equally as horrified. It is so complicated. I guess it's evolving. As a Spring developer, I don't want to have to deal with this. Actually, I find it cool and interesting, but there's another part of me that just wants to run away screaming.

You need a deployment platform to shield the developers from the complexity of the infrastructure. That will provide the various environments. Hopefully, in your service template, it'll provide IaC code for deploying the service, some kind of template. Then hopefully that is composed out of some reusable Infrastructure as Code components. Though the modularization technologies there leave a lot to be desired, specifically in the Kubernetes YAML world. Here's one example that I've worked with, which is where there's a GitOps tooling, here I've shown Flux, but there's also Argo CD. The state of the cluster is defined by one or more Git repositories. When those change, Flux will actually apply those changes to the cluster. It can also monitor the container registry. When a new Helm chart is published, it will edit the manifest and then deploy that new version. It's a pretty slick setup, but it is quite complicated. That's the deployment platform, and ideally it shields the developers from a lot of the complexity.

Microservices Platforms: How?

That's the six platforms. I just want to wrap up with a few comments about this platform development thing. While I was putting this talk together, I saw this article by The New Stack, which just pointed out that the success rate for platform engineering is not that great. Mostly fails. Based on what I've seen looking at organizations, is it's not surprising. I think one issue that's going on is for a lot of organizations, platforms are a way of ignoring the hard problems of people and solving actual customer problems. They do that by diving into what I call the infinity pool of technology. Because you can just get lost in the Kubernetes world and feel like you're doing useful things like creating custom Kubernetes operators. If you're not actually solving real problems, you're not going to succeed. A few thoughts. I've observed this antipattern forever. You really have to focus on the services, get those deployed in production, and worry about the technology later.

Specifically, what that means in terms of platforms, build some services, put them in production. Figure out what problems the teams are struggling with, then build the platforms, migrate the services to those platforms, and just iterate around that. Rather than go do a lot of platform development up front. Then team topologies has this great concept of a thinnest viable platform, or you could say it's like a minimum viable platform, where you build just enough platform to enable the teams to be successful. No more, no less. Maybe that involves Kubernetes operators, but maybe it doesn't. Then, teams need to adopt a customer-focused mindset, where customers in this case are the teams that are developing the services. The platform teams exist to help those teams, not actually dictate to them. They need to learn what their problems are, and provide platforms that help. Then they also need to actually help those teams use the platforms through facilitation. Build just enough platform to help, and do it in a customer-focused way.

Summary

In summary, microservices do require platforms to avoid the teams having to reinvent a lot of complex wheels. I identified six different platforms. There's probably more. Then I think it's really important to take a software engineering approach to these platforms, which are actually easier said than done with some aspects, and develop reusable components, rather than just copy-pasting everywhere, which creates a maintenance nightmare. Obviously, if you're a developer, you go, that's no big deal. Programming languages have libraries, frameworks, build tools like Maven, Gradle have plugins, even CI/CD platforms like GitHub Actions and CircleCI have reusable components. Then it gets a bit iffy when you're in the Kubernetes YAML world. There's Helm library charts, which are just weird. They work, but they're weird. Then there is a language called q, but that seems slightly weird to me as an application developer. I think there's work to be done in that area. You want to have these modular, reusable components everywhere. When you do have copy-paste, you then need tooling to actually automatically roll out updates to all of your services, generate pull requests for each one. Then, lastly, if you're building microservices, focus on the services, not on the technology. Build just minimal platforms that help.

See more presentations with transcripts

Architecting AI Systems for the Messy Reality of Enterprises: Why Agentic Compute is the Missing Layer

Arun Joseph — Sun, 02 Aug 2026 16:00:00 GMT

Summary

Arun Joseph shares real-world insights on scaling enterprise agentic platforms like Deutsche Telekom’s LMOS. He discusses bridging organizational fault lines, replacing tool sprawl with core platform abstractions, and moving beyond basic chatbots to operational intelligence systems through ephemeral agents and an Agent Definition Language (ADL).

Bio

Arun Joseph is co-founder & CEO of Masaic, a stealth startup building AGC, the Agentic Compute: a new computing substrate for knowledge work. AGC is open by design and built for scale, providing the architectural foundation for decisioning and actioning systems that integrate with existing enterprise stacks.

About the conference

InfoQ Dev Summit Munich software development conference focuses on the critical software challenges senior dev teams face today. Gain valuable real-world technical insights from 20+ senior software developers, connect with speakers and peers, and enjoy social events.

Transcript

Arun Joseph: My talk today is going to focus entirely on enterprises. There will be different definitions of agents. You might have heard about Cursors, Lovables, and autonomous agents that run for hours, but I'm going to ground it in the context of enterprises. That's why I call it agents for the rest of us in enterprises. Building anything greenfield is easy. If you want to, in enterprises, you already have the cars, you already have the trucks, you need to lift it up and then fly. This is the aura that AI actually brings in.

My name is Arun Joseph. I was here talking about our journey in my previous role as head of AI engineering for Deutsche Telekom. There was a program that we did, which we envisioned in 2023. We were one of the first agentic platforms to go live, potentially, as I know, in Europe or elsewhere, and that too in a large enterprise like Deutsche Telekom. I was leading that program. It was entirely built open source. It's called LMOS. It stands for Language Models Operating System. It's now moved under Eclipse Foundation and still running in Deutsche Telekom across multiple countries. Ever since, I have moved on. Last year's talk was so successful that I decided to quit my job and then start my own entrepreneurial journey, leaving a cushy job. This is what InfoQ talks can do to you. Right now, we are entirely focused on building multi-agent systems. One of the companies, a lot of things are under stealth. It's called Masaic. M-A-S stands for multi-agent systems by its nature. My previous background has always been building platforms across the world, including Europe, U.S., Canada. Distributed systems has been my mainstay in large enterprises.

Context

I'm going to talk a little bit about what we do as a company. We focus on operational intelligence systems. We build large-scale operational intelligence systems for critical infrastructure and large enterprises. We could say, a better Palantir is what we are actually looking at, with an open core. Essentially, all of this comes from the learning that we had building agentic systems. There is a need for a new class of systems, which we refer to as system of outcomes. We already know about systems of record, which is Salesforces and databases of the world. Then we have the data OS layers, which is the Palantirs and the dashboards of the world. You need a new class of systems, which sits on top of these systems, which focuses on outcomes, whether it's heavy machinery downtime, or running a city better, or for thriving of the industry in operations matters. Part of several foundations. I will focus entirely on enterprises.

Let's start with the myth bust on what actually works in enterprises, at least from the journey that I had. Then we will try to define an agent, because there are so many definitions. I wouldn't be theoretical, I would try to frame my mental model of an agent that we use even today, and the enterprise challenges. There are two magic bullets, which are going to solve everything in enterprises, which I'm going to give to you. Then we are going to level up the agent's journey. I'll show you some demos.

Myth Bust - Agents that Work

Agents that work in enterprises. Let's start with two categories of agents that work. A little bit of background on LMOS. It went live in 2023. This has since moved out of Deutsche Telekom. Some of these numbers are public numbers. I won't cite the numbers around agent handover, but it has been incredibly massive in terms of what we could do. There are three outcomes that was done. One was the business outcomes, which is obvious, without which there is no technical platform. It was an industry first, and it was tested against some of the best in the industry, vendor products, and it still outshone them. The best part was it showed how you can leverage existing investments and teams within an enterprise to build agentic systems right from scratch when agents wasn't a thing at all. The kind of systems that we do, so one category of systems that work is the chat modes and automation systems, which is where most of the enterprises are, but we are actually going into operational intelligence systems.

I'll show you a glimpse of a couple of agents, what we refer to as agents. The core of our operational intelligence platforms is also multi-agent systems. I'll use the heavy machinery operations as an example to talk about the agents that we refer to as agents in the level 2 of agents. One of our core products is called Atlas, but I won't go into the details. Let's take a heavy machinery operations company, it could be Hitachi, John Deere, SENNEBOGEN, whatever it is. Essentially you want to reduce the downtime of machines, because the downtime of machines is costly. How do you ingest all the data that you have about the machine? For example, you have IoT data, telemetry data, SOPs, all the incidents, the frontline intelligence, which is where most of the intelligence lies. How do you ingest all of that into a system, let's call it the magical system, so that a service operations manager can ask, we have seen a spike in error codes of this one in Midwest, what can we do about it?

Do you know anything about it? This is the system experience of what it would be. It needs to drill down, crunching numbers. It needs to be accurate, and come up with suggestions like this. "Look at it. It looks like 73% of the use cases are around air filter management issues, and here are the classification of those issues." Maybe there are better ways that we can deal with it. Decisions, it's actually a decisioning system. At this point, the large operational intelligence scheme can decide what to happen, what can be done with it. Then, it should actually go into the next step, which is, let's do the automation of this, potentially. What can we do about this? Maybe when this error happens, this is the SOP that needs to be followed, which has human in the loop, then raise these tickets, but also send this telemetry data to some other systems.

Which results in a dynamic construction of an SOP, which is not just textual, it is a mix of agentic actions, plus documents, plus a process that needs to be followed with human in the loop, which is sent to humans for approval and for testing. This is a second class of agentic systems, which is not really the chatbots and the typical enterprise agentic systems.

My talk is going to focus on this class of systems, and in the end, it's going to show, here are the ways in which you can reduce your machine downtime. Look at this. These two parts have been failing consistently in this region for the last one month. Here are the next actions that you can take. JP-100, or seal kit across these warehouses, stock it up. Maybe there is a need in the next couple of months. How do you build such a system? This is what we are going to talk about. Here, while building that system, as you would see, essentially, when you ask such a question, what actually happens underneath this? It's going to spin out multiple research tasks, and each of these tasks are ephemeral agents. This is not like agents somebody has created with X and Y framework. This will not work, because how many agents will you create?

You need an ability to create ephemeral agents. Then it actually runs and says, find the most frequent mode, so the multi-agent execution in parallel. Then in the top, you will have a multi-agent synthesis, which arrives at the outcomes. This is a true agentic system, as we envision it to be, which is what we are building. We're going to talk about these two classes of agentic systems.

What is an Agent?

What is an agent? Let's start with the level 1 agent, which is mostly used in enterprises for chatbots, automations. This is the definition that I use for an agent as a programming paradigm. You have programming paradigms like functional programming, object-oriented programming. My definition of agent here is not from the perspective of what an agent is from a theoretical sense, but from a programming paradigm sense. What is a program? A program has input. It does some transformation or computation, then it produces the output. I would define agent as a program, the level 1 agent as a program, which at least has one characteristic. It expands the breadth of inputs that it can take, and it expands the adaptiveness of the computation. It is still abstract. Let me show you two examples. This is, for example, from one of the agent frameworks that we built from LMOS.

This is a screenshot. "Hey, find me the best hotel in the location with the warmest weather between three cities, and book it for me." Imagine if you are a traditional enterprise with a booking system, for example, that you built. For any request that comes in, a product manager would come in and say, "I need this new button there," which when you click, it should book the best hotel for the warmest city. This is a requirement somebody would write. Then people would go and write this logic for orchestration in BFF or wherever it is. As you can see, what is going on is, in this case, the orchestration of the functions, get weather, list hotels, and then combining it to produce a result allowed a fluidic computation. Nobody went into BFF and started to write and clubbing these things. Neither did the product project team start to write a button and start to implement it.

What does it actually help in enterprises? I see in enterprises, the primary place to go after is not really large-scale Godmode agents. This is a good place to start, because essentially it allows a breadth of inputs to come in. You would have worked in enterprises. Any change in a schema of an API is a change request, goes to Jira and the huddle, the scrum and whatnot and all the ceremonies. Now you have an ability to have it really in a fluidic manner. It allows you to build better bots, chatbots for customers is obvious. It also opens up a huge potential for automations. I have seen in large enterprises, bot farms. Bot farms were previously built with Camunda's of the world, for example. You would have a large workflow orchestration engine. There would be a team who would write, every day run this bot to look into these tickets, extract the reason or something like that, and then trigger this workflow to cancel it or something. Every time you need a workflow, you need to rely on the Camunda engineer or somebody else. There's a huge pileup of such automations which can immediately be automated like this. This is the actual value it brings in enterprises. Definition one, which we will revisit at some point.

The Enterprise Challenges

There's this number, 95%, a famous number, MIT study or whatever. Why is the 95% failing if it's as simple as this? This is a level 1 agent. It's just orchestration of existing functions. You don't need to do a Cursor magic. I would start to give two major reasons which I personally understood. Then I will go into the magic bullets. The first point is crossing many fault lines in an enterprise. The definition of fault lines, I need to talk about it. What is a fault line in an enterprise? It's not greenfield. All of enterprise work can be summarized into just four work packages. Take the booking system in a company, if somebody had built something like that. It only has four functions, ideally. You need to store data, storage layer. Then you have the transformation layer, presentation layer, and the transmission which crosses a boundary, which is either event sourcing or API call.

Then it goes into somebody else. Microservices, whatever. It doesn't matter. This is an enterprise. Of course, security and the rest of the things in there. You can summarize all of enterprise work units as this. What is an actual enterprise? You would have seen the famous Josh Evans microservices presentation where a request comes in in Netflix, it spins into multiple nodes. In an enterprise, you have many teams and you need to coordinate. Each API might be a different team. There are these fault lines between each of these APIs. This is a whole thing to first understand. Within a team, you have multiple people, which are systems and roles, product managers, DevOps engineers. There are all the fault lines in there. My hypothesis is if you start with an agentic system, let's talk about the product ordering API in a real enterprise. This is taken from TM Forum, which is a telco specification of APIs.

An enterprise API is not as simple as, this is the booking API, which is API first and the first-class OpenAPI spec or something. It's messy. No one even understands what these attributes are. There will be one or two people within that fault line who would understand this API. If you say I want to build an agent which helps in product ordering, there are two approaches to do it. You build a fancy new team who has no idea what these things are. They talk about latest research papers. They talk about the latest research tools. Starts to pick the latest framework and start to build a prototype, just like the booking thing. "I want to do the product ordering. Can you show me the API?" Nobody understands anything, for example. The fault line thing, most of the enterprise AI programs, which at least I'm consulting or reaching out to, and I've seen this personally, if it starts to detach from the actual problem you're solving, you're dead.

Essentially, you will wait for APIs. You will wait for people. There is no way you will be able to even iterate on even business people. The frontline people who are handling the calls from the customer, they know better than any of these engineers what the requirements of product ordering would be from the usual customer. If you start to build only a technical team with whatever technical skills, it's not going to work. That's the reason why I use this picture from Cole to build this airplane. Essentially, it's not like a fancy new aircraft that you're building. You're not bringing aeronautical engineers. You need to think about the existing cars and the trucks and everything, and try to build a team around it.

Enough with the generic stuff, but this is super critical to understand the first part, without which I cannot go into the technical second part. Then there is a lot of noise. Current AI programming model poses a challenge to software engineering. What do I mean by that? I'm going to show you some advanced agentic patterns. Take it in a light sense. It's a pun. Have you ever thought of a line of code in a program, translates into multiple containers? I'm going to show you the magic there. This is a pun. This is a typical program, how it might look like these days, with most of the agentic framework approaches that you take. You want to build an agent that just simply thinks, what should I do? There is a think function. I have seen in enterprises, you bring in multiple tools from multiple burgeoning startups. For example, an eval startup will come up with an SDK.

A telemetry startup will come with their SDK. A memory startup will come with their SDK. You buy all of this. In a new company, this is fine. You are using their cloud service. In an enterprise, no one would use the cloud service. I think Guido van Rossum is going to jump out of the window if he realized that the decorators were used as line of code execution, which is remote code execution. What is exactly happening, for example, in enterprises, with all the noise around their programming models? I've seen this personally in some places. Each of these costs license costs. This is only referring to the license costs for eval tools, for example, potentially. In a large enterprise, 100k is nothing. You can get around with it. This line of code is now five containers. You need to have your operations team to come up with the UI, the database, and the memory structure, and the API, and custom Kubernetes operators.

This is how it actually works. I've seen something with 25 containers. I've personally seen this. I still remember the face of one of the DevOps engineers after the procurement or something was done in one of the consulting engagements. They said, we are going to bring in a brand-new great evaluation tool, which is going to fix everything. They bring this in. This was the face. Some of you must have seen the movie, 300. It's like, this is vendor containers for line of code. Now the DevOps team is not able to support this, or they don't even know how to set these things up. Then suddenly you have new fault lines. I talked about fault lines, which I will get to as well. This is one of the major reasons, the noise around tooling. The tooling sprawl and the existing teams are not able to either support or understand what are the actual paradigms required.

There has been this new release of the agentic workflow builder from OpenAI. This is beautiful, AgentKit. Then suddenly says OpenAI just killed everything. The fun fact is, look at this. This is the Raft consensus algorithm, which is based on which even Kubernetes runs. Somebody might look at it, and this is how people start to think. If it's a workflow, it can be done agentically. You start to say, Raft consensus, which is around leader election if one of the distributed systems goes down. You look at it and say, this looks like something I can build with agentic workflows. I don't even have to explain. If you have to explain why this is wrong, then you realize that there's a problem with whatever that you're trying to build. Like I mentioned, the biggest problem is the fault lines, as I described. Now you have a simple agent about product ordering API.

I want to build an agent, which does one thing. It helps the customer in purchasing something. You speak to the product API team, and then they say you need to talk to the profile API team, then you need to talk to the ordering team. Now you also have to talk to the 50 container managing DevOps people, and also to the vendor and the vendor documents to even come up with the first Hello World product agent. This is sadly the status in most places.

Two Magic Bullets

Magic bullet. Why LMOS worked in Deutsche Telekom, in my previous program. In 2023, no one was talking about agents, at least as far as we didn't know. We had a small, brilliant group of five people, all distributed systems engineers. At that point in time, there was only LangChain, as I believe. What we did was we chose their stack, which was the greatest in the world. This is the reveal moment. That stack is this. We chose the stack which works with existing engineers. All the product, all the APIs which were in Deutsche Telekom at that point in time was in Java, JVM. They had hundreds of client libraries built on how to invoke these APIs. The entire observability stack had a lot of libraries also supporting the JVM ecosystem. Then they also had the know-how on how to actually get a single product from a product API.

There are only very few people who would know this. Nothing against Python or Rust or whatever, we did this. We started our first program with LangChain. It was a disaster. I personally have my thing with LangChain. The abstractions, it's great. We came up with our own framework, which is also open source, which is based on Kotlin. What we did was, existing engineer, we were attaching all the libraries which were already there in the enterprise as dependencies to this one, which allowed a product API team member and engineer to immediately test it. We took away all the hard parts into the library that we built, it's called ARC. I'll give you references and even last year's talk, but this is for the existing engineers. Then the DevOps team. This is the observability, telemetry, the amount of observability noise plus some really value-added things which has gone in the AI industry is crazy.

A single API call to an LLM will generate a lot of tokens, tool calls. Now you need a specialized system to understand the stack. This doesn't make any sense, because you already had the SigNoz's, the Grafana's, and Prometheus, and the tracing. We started not with any fancy tracing tool. We plugged our own telemetry directly into existing systems. The operations team said, this is something we can manage. Please don't come to us with more containers. This allowed them to be onboarded into the journey. This is our stack. This is the LMOS stack. We had the LMOS ARC, which is the Agent ReaCtor. Then we had the LMOS agent platform, which is also a custom Kubernetes container, but packed with all the things that work really well for us. The business part, this is where things will start to get really interesting as I go along with it.

This is a Kotlin framework, which we'll skip right now. The business, what AI brings in, the true power in enterprises, and also for agentic systems. Let me show you. Let's go back to the previous example. If you want to build the booking agent. Previously, if you had a button, some of the business or whatever you call as a business would go and write a requirement. Put a button in there in a Jira ticket. I'm going to assign it to you. How do you define the requirements for an agentic program? Which in this case is a chatbot. This was real. The idea is, let's write stories. No one knew, or at least back in time, how to write these requirements. Essentially, the loop that I showed, you need to compress that loop. That's the only way you can get anything out. The loop between engineers, DevOps, and business. Then when we started to build this, the engineers were able to connect the APIs. The DevOps was able to handle the load. Then the requirements started to get a problem, because every time there is an anomaly in the chatbot, there is a new ticket getting assigned. This cannot work. This will not work.

We came up with a layer which we call ADL, which is the Agent Definition Language. There are also some screenshots. This was logic for the rest of us. First of all, you cannot expect people to simply write a prompt and make it work in production when you are facing a customer service bot or something critical. Most of the Hello World examples that you see will not work. Let's say you are the Volkswagen team, and then you wanted to build a bot for your service team. You would need to define your SOPs. This is a screenshot, again, from the open-source ADL programming environment we built. It's a simple Spring Boot container. You spin it up. It comes with the UI. It allows for quick iterations. The key here is it shows a way the business can start to write the requirements in a specific format which we started to control, so that the translation between Jira and business and all was compressed.

The engineers would actually wire the APIs. The business would use this environment to start writing what we refer to as agent use cases in a format that we prescribed. They are able to iterate really fast in there. There's also a simple example. This is our ADL environment. It's a simple Spring Boot container. It shows the events and the tools, just enough for the business people. This is not for, let's just say, a DSPy alternative or something, prompt compression and things like that. This is for the business. For example, here it says, my Golf broke down, I want to book it. Then the business user is able to immediately see, it actually is able to pick the right use case in this case. Then I asked for the agent handover, it picked the right use case. It provides a mental model for business to actually draft, just like programmers, all the things that we learned about from programming was compressed to this.

ADL, we are not sending the data or the use cases as it is to the LLM. This is the magic. We have a compiler which compiles into system prompts, which compresses it. It's like tree shaking and things like that in programming that we're actually using. There's a lot to talk about it, but here at the top, I'll give you the broad picture. Magic bullet number one, build agents with existing teams and stacks. OpenAI has not killed software engineering yet.

Magic bullet number two. I talked about the first class of AI systems that we built and all the learnings. The second class was the operational intelligence system, which we started to build, which I showed as heavy machinery operations. Platformize the hard parts and get out of the way. What is a platform in the agentic system today? This is what we will talk about from a real experience. AI infrastructure today is a brittle set of things. There are hundreds of things. If you want to build a simple agent in the heavy machinery operations, for example, I want to analyze the tickets which were coming for machine X, Y, Z. I will build an agent. Most of the time I will be doing the plumbing, observability, guardrails, evals, model integration, and only 5% of the time in actually writing the agent logic. Revisiting the example that I said, if we started to do that, for the large-scale agentic system that we are building.

When we started, I think we were around 50 or 60 agents for the heavy machinery operations in our new company. This wasn't going to scale anyway. I'll give you an example of how we actually went about with the platform path, and probably show you some examples. The previous example, this is how we started. We started with multiple agents being built, and then we moved down the hard parts into a layer. What are those hard parts? Session management, context management. How do you do telemetry fan-out? Like I said, you do not want to transport the telemetry only to the evaluation tool, you need to transport it also to your observability stack and elsewhere. You cannot have that plumbing done in here, this is one of the hardest parts that you can do. Plus, in the operational intelligence platform, if the ability to switch a model is going to be hard, this is not going to scale, because the cost won't add up.

For example, if you start to run a million invocations, the cost amortization will not work, so you need to shift from the larger models to smaller models with ease. We started building how you do that into the hard part of the thing. MCP thing, I would rather say, because MCP is still evolving, but I'll show you a few things, and the vector DBs and things. We started to move down into this platform layer, which shrunk the agent sizes, let's just put it this way. This was the first optimization that we did. Then, we started revisiting. After building more than 50 or 100 agents, Amant, who's my co-founder, we call him the Agent Whisperer, or the Ontology Oracle, he started spotting something supremely interesting with the programming paradigm required to build agents. The first agent that I showed you, the fluid input versus comma output, we're going to revisit that definition into something more generic, from a programming paradigm perspective.

We came down to, an agent is simply a loop, which takes in a goal, which has context, and then it accepts constraints, and it executes tools. These tools could construct additional tools. Then, this is exactly how you build Cursors or Lovables. There is nothing magical in there. If you're able to construct this loop, you can build anything. As I mentioned, the simple construct, this is a programming paradigm, this is not a Python code or whatever. You need to see it as a programming paradigm, as I would rather call it. While the goal is not achieved, you start to iterate, and then you start to refine the goal. At some point, you loop back to the human in the loop.

You remember, we started to shrink it. Then we brought the agent loop primitive into the platform and exposed this as a single API. Much like S3 exposed an API, or Stripe exposed an API, or EC2 exposed as an API. Then, what happened? All those agents actually shrunk into ephemeral agents. Nobody is sitting there, I need a fault analysis agent. I need this testing agent or a root cause analysis agent. It's based on the goal. The service manager says, I would like to know what are the major reasons why these machines are cracking. If you have arranged the domain well, if you have figured out this programming paradigm well, it is able to plan, and the planning goes into multiple loops, and the loops by itself constructs the tools or queries and starts to synthesize it. This is what we actually perfected. The best part is, the framework, juggernaut.

Is it a framework? It's exposed as a single API which works with any OpenAI client, which brings in the point that it is frameworkless. Now because it's an OpenAI client, you can get one line of code which works with OpenAI, you can use in any framework. You don't have to write all that agent logic in most cases. All the hard parts are taken care of underneath, including plumbing, model switching, tool call, remote tool execution, you name it. Which brings us to the paradigm, compute as the agent. What if we expose the compute itself as the agent-making machine? Then you can build your own Cursors. You can build your own enterprise Cursors for whatever that you need on your business systems. This is the reason why we say AgC. We call it AgC, that platform. It's called Agentic Compute. AgC, it's exposed compute as an agent.

When I say this to what is compute as an agent, most people don't get it. I took the liberty to use this talk to explain why we say compute as the agent. It's exposed as one Docker Compose, developers can test it. One Helm chart, anybody can deploy it as fully Apache 2.0 license. Your business teams or engineers can build agents with your existing stacks or the favorite frameworks, for example.

Demos

Let me actually show you some of the demos. First, I talked about ADL. I wanted to just give you a preview of the ADL approach. It's nothing but a Spring Boot container, as you can see here. We'll go step-by-step around the demos. Demo number one was around the first magic bullet, how did we enable our existing teams? We built a framework, which is Kotlin-based, and engineers would go and just build Kotlin files, which is Kotlin scripts. Then they would also go in to write functions. The idea is if you attach your libraries, you can directly plug in your product booking API or whatever client you attach, and you can invoke. There is no MCP, there is no nothing at this point, because we didn't need it at that point in time. This is the framework that we built. It's called ARC. It's open source. Around ADL, I wanted to show, once that Spring Boot application is up, this is how you would see it.

It would run on your localhost. This provides a quick testing environment for your agents. What kind of agents? Again, of course, it can be used for the second class of agents as well. The same use case that I showed, every agent that you build will appear here, and a business would go, and business ADLs transfer money. I was just about to show how somebody might think about two-phase commit, written as a prompt. This won't work. For example, the Volkswagen example, somebody would go and write the business use case as a business person. It has three constructs to remember. The constructs don't matter, but what matters is the thinking. You write the solution, you write the alternate solution, fallback. The business person would also say, if the date, for example, is not available, go to this use case. There is something called, for example, date not available.

How do you construct graph trees, actually, in an SOP? There is a construct that we came up with. Go to use case, offer alternative, and the business would start to write that particular use case, and immediately go and test it in the chat. Maybe I will say, my Volkswagen broke down, need an appointment. It should actually pick one of the use cases and respond, and then the business person knows, it actually picked the right use case, even though I had many other use cases. You work only on that small use case. You can attach tools in ADL.

Now I want to shift into the platform which was built, and what is this construct of compute as an agent? Let me shift to AgC. Let's talk about a proper business use case. In a typical enterprise, you would have several information systems. Let's talk about a telco, for example. A telco would have pricing engines for the offerings that they have, especially for B2B. Especially for B2B, they would have templates for how the business formats, the proposal formats need to be. You've even simulated a communications MCP, for example, the communications server, the transcripts between calls, and several other business systems. My goal is, I want to build an agent which allows a salesperson in a B2B organization, in a telco or home server, help make better sales. What would that experience be like? For example, if I take the time to write, let's just say I come into this system.

This is AgC, after the docker compose up, this is what it brings up. It has a basic UI. It's Docker Compose, you can use it anywhere. If I just say, I just finished a call with Nordstern Mobility, help me draft a proposal. So far, I haven't done anything in LangChain, or LangGraph, or CrewAI, or anything. I just attached the tools, which is necessary, and returned the instructions actually for this one. Here it starts to look into the brand templates, and actually prepares the template. Then, let's pull in the pricing as well. In this case, we have also attached the B2B pricing as well. What is important to notice here is it has not actually done anything other than pulling the information. There is no other wiring which was done, and it actually does it. Let's say I have a weird requirement that I come up with.

There is a website called Allbirds. It's a Shopify. It sells shoes or something, Allbirds. Allbirds.com, and Shopify has an MCP. It has shoes and things like that. Let's say if I have a weird requirement, which says, I want to add a shoe to my business proposal? I can write a Jira ticket. I can tell my smart LangChain, LangGraph, CrewAI, hundreds of other things to do it, or I attach it to AgC, because like I said, the programming construct is a goal and the loop. Let's say, I've added it here, Allbirds MCP, for example. Just added the MCP. I have done no coding here. Let's add a shoe, a black sole, and size 9 to the proposal. Let's see. It started to pull in the MCP tools directly into your agent. You did not write any code. The compute was the agent. It started to pull in the shoe, and when you click on it, it actually goes into your Allbirds. It's not staged, is what I wanted to mention.

What is the possibility? There's also another thing. All of this, the hard part is model switching, for example. You can add any model. I'm not referring to the UI. First of all, this is not meant for UI people. This is what you use to build your platform. It's primarily written in Kotlin. You can go and attach your models. You can simply switch the models from the backend with one line of code, and then run it again. You would be able to even get a first-hand experience whether it actually works. Now you have constructed an agent with only instructions, like I said, that programming construct. There is one line of code, which is OpenAI's client SDK format, or it's a simple call, and then you attach it to any program, and you have an agent. There is no magic that you need to do. What about observability, the hard parts?

For example, here, it supports OTel, plus also prompts fan-out. In the backend, you can simply configure at your collector, I want to fan out this telemetry to Langfuse, to RIs, to signals, wherever you want it to be. This is what you want. It's not your application developer's responsibility, neither it should run in your micro-agent service or whatever it is that you would want. As you can see, immediately, all the telemetry is available in any platform that you choose. The platform takes care of the hardwired parts.

One of the other things that I wanted to show you was also these ephemeral agents. I wanted to show one last demo. Essentially, remote call execution. This is RPC. You are able to attach your tools. Essentially, if you are working with agents, there are two ways to attach tools to it. The tools execution is something you have to manage. Here, this is an example. I built an agent which runs a local tool, which actually goes into a browser and updates Salesforce. Salesforce often asks for OTP and two-factor and blocks me. Maybe it might not work, but I want to show you, "I just finished my call with Volkswagen. Update Salesforce opportunities." It actually gets a call transcript from my call transcript server, and there is something called Atom. This happens locally. What did I do? What would have happened? This is localhost. Where was the platform running?

I attached a tool running on my local machine. There is no MCP magic or something. We just did the wiring. This is a long-running execution. I just finished my call, and I said, this is how you update sales. I just finished the call. Go and update Salesforce. I hate the guts of the user interface. This actually should have run.

Takeaways

I would just summarize the key parts, which is the key takeaways. There's a lot more. In enterprises, start with level 1. It also offers a large amount of ground to cover if you are able to think only in this term, input fluidity of programs and orchestration fluidity. How to make it work? Bet on current stacks and teams. Engineering is not going away, otherwise, it won't work. Use the example of the fault lines. ADL, this is a new paradigm. This was not there before. I don't think business needs more user interfaces. Business needs a way to write the requirements, which directly translates into the program, and not more UI. Platformize the hard parts, because it's a snake oil industry. There's something going on. Computing a new paradigm for agents, which is a function loop, which we discussed.

See more presentations with transcripts

The Free-Lunch Guide to Idea Circularity

Holly Cummins — Thu, 30 Jul 2026 16:00:00 GMT

Summary

Holly Cummins discusses why "nothing is new under the sun" in tech. She maps historical architectural tradeoffs to modern cloud, microservices, and AI hype cycles. She connects financial debt (post-ZIRP) and technical debt to epistemic and sleep debt, showing engineering leaders how to navigate shifts in assumptions, embrace sustainability, and revive proven engineering disciplines.

Bio

Holly Cummins is a Senior Technical Staff Member on the IBM Quarkus team and a Java Champion. Over her career, she has been a full-stack JavaScript developer, a build architect, a client-facing consultant, a JVM performance engineer, and an innovation leader.

About the conference

Transcript

Holly Cummins: I walked here along the Thames this morning, which was beautiful. Anybody else managed to walk here along the Thames? Anybody throw up while they were walking along? No, good news. Had this been QCon 1858, the story might have been different. In the 1800s, the Thames was not the beautiful place it is now. The Thames was a sewer, and that was pretty bad. It got worse because in the summer of 1858, it was hot. Water levels got lower and the sewage got warmer, and you can imagine how those combined. It was really not good. The waste of 2.5 million people was flowing through a very small channel with very small throughput. This became known as the Great Stink. You may wonder where the technology comes in. I'm going to talk about Michael Faraday which I think counts as technology. Michael Faraday, who you may remember from The Faraday Cage, said, "Near the bridges the feculence rolled up in clouds so dense that they were visible at the surface." Disraeli, who went on to become prime minister, said the Thames was reeking with ineffable and intolerable horrors.

I think everybody had known for a while that something needed to be done, but in the summer of 1858, parliament itself got affected. They had built this beautiful new parliament building and they couldn't actually use it because the smell was so bad. If people went near the Thames, they risked throwing up, people were fainting. It was bad all round. The money that they hadn't been able to find for the problem before suddenly got found and they started a 20-year work of sewage. The embankment wasn't there at the time, but what they did was they built the embankment and they ran plumbing through the embankment to take all of the human waste away from central London. Think of that when you walk on the embankment. What they also did was they built pumping stations. This is Crossness Pumping Station. I just have this picture in here because, how beautiful is that? We do not build technology like that anymore. This is called the cistern chapel.

History Rhymes

Some people say history repeats itself, and some people say, no, history doesn't repeat itself but it definitely does rhyme. After I had prepared these slides, I ended up watching Dirty Business on Channel 4. Anybody else watch that as well? It was upsetting. The problem with the plumbing in London before they redid it was fundamentally a scaling problem. We've all had scaling problems. The Thames had insufficient throughput for the number of poo transactions that were being performed in London at the time. This is a throughput problem. To be specific, it is a poo throughput problem. If you want you can shorten that to a pooput problem. The reason there was a throughput problem was a direct consequence of previous architectural decisions. About 20 years before the Great Stink, someone had a really good idea, which is that they should eliminate cesspits in London. On the surface this seems like clearly a good idea.

Nobody wants cesspits. What is a cesspit? I always thought a cesspit was a metaphor, but a cesspit is an actual architectural feature. How it used to work was that you would have your nice house and then you would have a hollowed-out area underneath your house where just everything horrible went, and everything horrible just went down there and then it stayed and it stank and it was horrible. Everybody agreed that this was no good. A law came in that said instead of having cesspits we should have plumbing. Seems like a good idea so far. That plumbing should take the sewage and discharge it into the Thames. Suddenly seems like less of a good idea. The problem happened because we had this fundamental tradeoff between centralized stink and disease, which is what we got when we put the sewage into the Thames, and distributed stink and disease which is what we had when we had cesspits.

You can see that neither of these architectural choices is really very appealing, which is why they had to do something better and direct the sewage outside of London. The first implementation, they only directed the sewage about 6 miles outside of London and then continued to dump it into the Thames, which didn't actually work out so well. They did a second iteration where they properly got rid of it and started treating the sewage, which was an innovation that needed to happen. These sorts of tradeoffs where you have a good idea and then you have another good idea, and then yet somehow it doesn't quite work out the way you hoped. These happen all the time.

Anybody remember grid? The old people are, I remember grid. Grid went away. Nobody thought anything more about grid until grid came back and we called it cloud. We have these ideas and they just keep coming round, and it really starts to feel like nothing is new under the sun. Unfortunately, including destroying our home. Very old idea. We have a long-standing tradition of destroying our home. Every now and then we have another idea, which is, maybe it would be better if we didn't wreck our home. The reason we keep cycling between these two ideas is because there's a tradeoff. There are really good reasons to optimize for the short term, but on the other hand there are very good reasons to be sustainable. This is relevant for us in tech because right now one of the major things affecting our home isn't sewage, luckily. Aren't we glad to live in 2026?

The major thing affecting our home is carbon, and the tech industry is responsible for quite a lot of carbon. It is responsible for more carbon than aviation, which is what we tend to think of as the poster child for climate irresponsibility. If you look just at data centers, data centers use as much electricity as a medium-sized country like the UK. For us as tech professionals, this is relevant to us because we're the ones writing the things that go in those data centers. What we do with our software, what we do with our technology can have a direct impact on this. Part of the solution is clearly to use low carbon energy, but that can't be the only part of the solution, because even green energy, even zero carbon energy is not without environmental impact. This is the Itaipu Dam in Brazil. You can see that this had a significant impact on the landscape, and this isn't even the largest dam.

The Three Gorges Dam in China, when it was built moved so much water around that it was like a figure skater when they lift their arms. It actually affected the speed of the rotation of the earth, and the days are now 0.06 microseconds longer than they were before. If you're working longer, it's because the Three Gorges Dam made days longer. It also tilted the earth slightly. The earth's axis tilted by 2 centimeters just because of this human created structure.

Green energy, good. Green energy, we need to do more. We also need to be reducing what we use. This is, again, a very old idea. I won't go through everything we can do, but I do want to talk about two things we can do that are pretty easy and pretty effective. Again, these are old ideas. One is LightSwitchOps and one is efficient software. What is LightSwitchOps? LightSwitchOps is the idea, the very old idea, that maybe after you're done using something, you should turn it off in the same way you might turn a light switch off. It seems like common sense. Why don't we all do this? The reason we don't all do this is because those of us who have turned computers off in the past and had them never work the same again have been burnt by that experience. We've learned, don't turn the computer off if it's working.

We need to change that. We need to architect our systems so that they can tolerate being turned off. The other thing we need to do is think about software performance. Think about efficient software. This is where I have a very tidy intersection with my day job. I work on Quarkus. Quarkus is an extraordinarily efficient way of running Java applications. We've measured this. We did some experiments and we looked at carbon as a function of load for Quarkus and another framework which we don't name here but is Spring Boot. What we found is that with the line labeled legacy, the longer line is a higher throughput. You can see with Quarkus, it's got this very long line. It's got this high throughput. The lower line means it's got a better carbon footprint. You can see this really actually quite delightful correlation between having better throughput and being greener, using less energy.

It's probably a little bit easier to see. This was for a single instance. We then ramped it up so that we just added as many instances as we needed to support the throughput, to support the load. Then you can see a quite nice clear graph where Quarkus on JVM is, compared to Spring Boot, far more efficient as a way of running a Java application. What's going on? Why did we make these changes? Why did we design the system that way? What Quarkus did was it challenged what had been completely accepted conventional wisdom in the Java world, which is that doing things dynamically, delaying decisions, was a really good idea. Java historically has had absolutely loads of reflection. Once you get into popular libraries, they do huge amounts of work via reflection, which is a really expensive way to do it. It doesn't make sense anymore.

It used to make sense. Again, this is a conversation that as tech people, we just keep having over and over again. Should we be dynamic, or should we pre-decide things? Obviously, there's benefits to both. Should we be lazy in our initialization? Should we be eager in our initialization? In Java, for very good technical reasons, the balance had been very much towards being dynamic. It's statically typed, but almost everything else about Java is very dynamic.

When we got the cloud, all of a sudden, that balance shifted. An extremely dynamic runtime in the cloud doesn't make sense, because you're running in a container. Your runtime environment is almost certainly not going to shift in a significant way in terms of what software you have available on your class path. We don't need that dynamism anymore. That shift away from having one really big application running on-prem to something that's running maybe more with containers, more in the cloud, is a reflection of another shift, which is, should we be decentralized, or should we have something which is more central? The shift that happened about 15 years ago was we really swung very strongly towards more distributed architectures, and we got microservices, which are hyper-distributed architectures. Of course, having the hyper-distributed architecture does have some challenges. You have higher latency. You have more complexity. You have more infrastructure cost, but on the bright side, you get this resilient and you can have that independent lifecycle.

It is not a clear tradeoff. Around that same time, Git was introduced, and the promise of Git was that it was decentralized and distributed. That promise lasted for an extremely short amount of time before GitHub went, people don't actually want that. People really quite like having a centralized place to put their source code. GitHub has been enormously successful. We see similar questions now. In the age of AI, we're seeing this absolutely enormous investment in data centers from all of the big companies. We're talking hundreds of billions of dollars in data centers. The ROI on these data centers is somewhat questionable. Some companies are going in a different direction. What Apple is doing is that it is not building its own data centers, it is not building its own models, it's licensing them for a fraction of what it would cost to actually run a data center.

It's putting AI capabilities into its hardware instead. What this means is that instead of having to spend $100 billion on a data center, Apple is manufacturing hardware and it is selling it to us so that we can buy that hardware and then run those AI computations on our hardware in a decentralized way. We're happy because we have shiny Apple products. Apple is happy because we've just paid them a bunch of money for the privilege of running the machine learning workloads.

Hype Cycles and Investments

I mentioned microservices and I mentioned AI, which means I can now introduce this quote from the great Venkat. He says, "I'm so thankful for AI. Finally, developers are no longer chanting microservices constantly." If you've been to a few QCons or really anywhere, you will recognize this pattern, which is that this year, every talk seems to be about AI. A couple of years ago, every talk was about microservices. Now we're not talking about microservices anymore. These things, they seem to become enormously important, and then they go away again. What's going on? It is a hype cycle. It's the hype cycle. If you've been around for a while, you'll have experienced a whole bunch of these hype cycles. Grid was going to save everything, and then Internet of Things was going to save everything, and then digital twin was going to save everything, and then model-driven development was really going to save everything, and then low-code definitely saving everything.

Cloud Native, yes, absolutely. Microservices, Kubernetes. I don't know if you remember a while ago, the parent company of Cloud Foundry had this enormous nosedive on the stock market. The reason they had a nosedive on the stock market was investors noticed that Cloud Foundry did not contain Kubernetes. This was quite obvious to most of us, but it was a surprise to the investors, and they were horrified when they found out, and they immediately sold all of their stocks, because it didn't have the buzzword of the moment. The nature of these technologies is that we get very excited about them individually and as an industry, and then we have the morning after the technology before, where we realize it didn't actually solve all our problems, and then we go back in the other direction. For example, Amazon Prime Video published a blog a few years ago where they announced really excitedly that they had moved away from microservices to a monolith, and they had saved 90% of their costs. What does this show us? I don't think it tells us very much about whether we should use microservices or a monolith, but it tells us that the lunch was not free. We were promised a free lunch, and we have now been extremely disappointed by the price tag.

Meredith Whittaker, the president of Signal, had a really interesting quote a while ago. She was asked what technology did she think was most overhyped. She didn't say AI, she didn't say microservices. What she said was, "It's not simply that one piece of technology is overhyped, it's that hype is a necessary ingredient of the current business ecosystem of the tech industry." What did she mean? Of course, what we have now is that list. We have AI, and, unlike everything else, AI is definitely going to solve all of our problems. We're getting enormous amounts of hyperbole about AI. Anthropic's CEO says that in 3 to 6 months, AI will be writing 90% of the code. Unfortunately, he said that a year ago. Microsoft's AI chief says, this time we're serious, in 18 months, all white-collar work is going to go away to be automated by AI. Why would people say that when it's so obviously not true?

The reason is venture capital, because, how does it work if you want to make money off a business? Let's say you have a great idea, and you start your little baby company. In order to grow your baby company, what you need to do is you need to attract funding, so you do funding rounds. Each funding round, you attract investment, if you're lucky, and then you carry on building your product, and then you go to your next funding round where you get more money, hopefully. Then you just keep going through this cycle of funding rounds, until eventually you get to what's called the exit. The exit is when enormous quantities of cash flow in. The best thing about the exit is that usually the exit will be something like an IPO, or an acquisition. What that means is that this company is no longer your problem.

This company is someone else's problem while you skip happily to the bank carrying huge sackfuls of cash. Everybody is very keen to get to the exit, and people aren't necessarily super concerned about what happens after the exit because they've skipped off to the bank. How do you get to an exit? How do you get through those funding rounds? What attracts investment? I think in an ideal world, we would say things like stability attracts investment, profitability, being well-managed, having stable sources of revenue. Unfortunately, we do not live in the ideal world. What attracts investment is growth. What attracts investment is excitement. This is because the investors need to feel the excitement, but they also need to have confidence that everyone around them is feeling the same excitement, so that they can have follow-on funding rounds, so that they can get to the exit. The likelihood of selling your stake at a profit, which is what you want if you're doing an investment, is higher if you have that growth and that excitement, and that's what drives the hype.

What we're being promised in the current round of hype is a world without developers. This is actually a very old idea. I mentioned low-code earlier. The promise of low-code was exactly the same. You don't need to have developers. You can just use low-code tools, and the management will be able to write everything. It's an even older idea than that. This is COBOL. You may not think it looks like English, but at the time, it was advertised as English language programming. The idea was that COBOL was so comprehensible that your manager might not be writing your code, that was maybe a stretch too far, but domain experts could certainly be writing code, and your manager could be reading your code because of the magic of this highly expressive programming language. COBOL did not make the development profession go away. In fact, it was the opposite. COBOL unlocked this whole new profession of application developers instead of system developers.

This is a really good example of what's known as Jevons Paradox. Jevons Paradox says that efficiency improvements can lead to increased consumption. I always find the easiest way to think about Jevons Paradox is in terms of highways. We've probably all got a road that's got too much traffic on it that we have to go on, and then at some point, they announce they're going to widen the road, and we think, amazing, at last, the same amount of traffic will be running along this much wider road. It will be heaven. This is not what we get, this is what we get. Because as soon as the road is wider, all of those cars that had been put off going on that road go, amazing, and it takes you exactly as long to do your journey. It's just that more cars are doing the journey. This is exactly the same, I predict, for software.

This makes it quite different for some other professions. Many professions have gone away because of technology and continue to go away because of technology. For example, I think this might be my favorite ever profession. This is a knocker-up, which sounds extremely funny if you're from North America. Their job was a human alarm clock. It was nothing more sinister than that. What a knocker-up did was, in the Industrial Revolution, people needed to get to their factory jobs, but they didn't have alarm clocks because it was the 1800s, and alarm clocks wouldn't be invented for another hundred years. They needed a human alarm clock. The profession of knocking up actually continued for a very long time. It lasted all the way to the 1940s, and it lasted to the 1970s in some parts of England. Of course, inevitably, the knocker-up profession disappeared, because there is fundamentally very finite demand for this.

Only people who have a job outside the home need waking up, and nobody needs waking up more than once a day. You don't say, I enjoyed that experience of being woken up so much, I'd like six of it, please. It doesn't happen. With software, it's really different. The more software we have, the more software we want. We seem to have this infinite appetite for software. Of course, we need the software to do stuff, but then we need the software to do more stuff, and then we need the software to tell people about the software. Then we need the software to bypass the advertisements in the software. Then we need the software to bypass the bypass so that the ads still show because somebody has to make money. Then we need the software to gamify the software. Then we need the software to monitor the software.

Then we have the software to manage the software. Then we have the software to write the software, welcome to 2026. Of course, now, we have the software to debug the vibe-coded software, welcome to 2026. Now we have the software to manage the software that writes the software, because it's all a little bit complex. Then, at that point, we really need software to try and avoid Skynet, and then we may as well work at the meaning of life while we're at it. This could go on for a really long time, but I've run out of screen. If you look at the history of the software engineering profession, each time we've had a new layer of abstraction, the number of developers hasn't gone down, it's gone up. Each time it was predicted that software developers would go away because now we had COBOL, now we had frameworks, it just fundamentally didn't happen.

Some of you may be thinking, that's nice, but I've seen the job numbers, I've seen the news. The USA developer vacancies, it's not good. The numbers have gone very down. In the UK, it's slightly better, but it's looking a bit bleak. If you read the news, there's layoffs announced it seems like every week. Every time the layoff announcement says we're doing so well with AI, we're getting rid of people. We're doing so well, we're getting rid of people. We're doing so well, we're getting rid of people. What these announcements are saying, often they're coming from companies that have an AI product. If you have an AI product, and the promise of your product is that it allows everybody else to get rid of people, you have to get rid of your own people too, to show that your product works, otherwise nobody's going to believe you.

This is saying our AI product is working. For the companies that don't have an AI product, there's still quite a lot of FOMO out there. Everybody else is saying that they're getting these extraordinary productivity gains from AI. We need to show that we are too, or people are going to think our company's not being run well. We need to show our AI strategy is working. This is called AI washing. It's putting AI into a headline to disguise news that maybe isn't so good otherwise. For example, Block had significant layoffs. Their business really wasn't doing that great before the layoffs. Another factor behind these layoffs is overhiring. Overhiring is basically people-stockpiling. This blows my mind slightly, but it is an accepted business practice. There's many reasons for this, but one of it again comes back to this idea of attracting investment. What makes you look really successful?

If you're hiring loads of people, whether you need them or not. This works ok until the economic cycles bite. There's a recurring idea of free money, and you may be thinking, I'm fairly sure. I have never had free money. There's no such thing. Almost all of us have borrowed money. When we borrowed money, we had to pay something for the privilege. Depending when we borrowed it, we might have had to pay quite a lot or a little bit less. This is financial debt, also known as debt. The nature of debt has changed recently because we had been in what's known as a ZIRP. A ZIRP is a zero interest-rate phenomenon or a zero interest-rate period. You might think that zero interest rate means zero interest. Foolish. It doesn't. What it means is less than 1%. There isn't free money, but there's almost free money. Until quite recently, so if you look since 2010, interest rates have been bumping along at around 1% until the last 3 years when all of a sudden they shot up.

This was quite a shock to a lot of organizations, and they had to change how they were being run because all of a sudden money wasn't free. In the bigger picture, actually the exceptional period was that last 15 years. Since, if you look that's that period that most of us are used to. It's actually a little bit of an outlier in a broader history of high interest rates. High interest rates really affect investor behavior. Because if I have this dream of doing a startup and getting the investment, it's not guaranteed to get to the exit, of course. It might be that everything dies after the third round. All of that money that was invested just goes up in flames. Investments in companies have risk. If I'm an investor and I have my pile of money and I need to decide what to do with it, I could put it in a software company or I could buy government bonds.

In a zero interest-rate period, the risk is really high with a software company, but the return is potentially really good as well. With the government bond, the risk is low, but the return is 0%. That is just wasting my money. I'm going to put it in the software company. On the other hand, in a post-ZIRP period, all of a sudden that government bond is giving me maybe 4%. Compared to that really high risk with software, it starts to look attractive. Some investment gets siphoned away from software into safer forms of investment. If there's less investment, there's less hiring, and that means less jobs for us. You can see this if you look back at those employment figures. That big peak wasn't with the introduction of GenAI. That big peak was when interest rates were really low, and that downward slump was when interest rates rose.

ChatGPT didn't really make much of a blip on those job figures. Again, this comes back to the fact that software is not going away, and because software is not going away, developers are not going away. This is in fact confirmed by the latest data. If you look at the latest data, developer vacancies are going up again. Hiring is on the up. You may say, that's just because the economy is on the up in general. It's not. The gray line is the overall hiring. You can see it's still staying pretty low. Developer vacancies going up.

A World Without Work?

Does that mean that everything is perfect? Not quite, because another long-standing dream that people have had is a world without work. Every time productivity increases, we are promised that this productivity increase will eliminate work. In 1930, Keynes said that by the 21st century, we would all have a 15-hour week. Reality is somewhat different. What many of us are experiencing instead is a world without rest. We're really struggling with burnout. We have all of these ideals of like we need to do the hustle, and we need to do the grind, and we need to do the 996, which brings us to, unfortunately, a new form of debt, which is sleep debt. The good news, which is also terrible news, is that this is also not a new idea. In 1826, the average working week was 66 hours. The typical working week was 6 days, and you would work 12 hours for 6 days.

The interesting thing about this was that that was actually a peak. This is an incredible dataset, which shows the average UK working hours from like 1300 to now. That big peak was the peak of the Industrial Revolution. We had enormous productivity gains, and yet we were all working more. Then eventually we realized that maybe working more wasn't actually helping us as much as we hoped. Henry Ford realized that he could change his factories from having a 6-day week to a 5-day week, and productivity basically stayed the same. This is because, for people, there is actually a huge amount of value in doing nothing at all. The default mode network is a part of the brain that becomes more active when we're doing nothing, and it helps us solve problems. Then the question is, now that we have AI, are we doing more nothing? Unfortunately not.

Professions using AI are working 3 hours more per week than other professions, than they were before AI. This is something that's been widely reported, so I've seen this slide several times. AI intensifies work. AI is making us work longer. This article says, it's a paradox. Yes, it is a paradox. This is exactly Jevons Paradox in action, because we have this extra capacity, because the cost of doing things is lower, we are doing more things. For a lot of us individually, what we're experiencing is a combination of euphoria, because we can do so much with the tools, and also fear, because we've seen the news, and we've seen those layoff figures. We're entering this productivity panic, where we're going as fast as we can and producing so much stuff. That article I showed had the phrase disposable busyware, which I really like, because we're producing so much stuff that the build versus buy has shifted completely towards, let's just build it, and then let's throw it away, and then let's build it, and let's throw it away.

The fact is, maybe we should think about whether the world actually needs us to be building all of these things. Should we be producing these things? Because workslop is pollution, fundamentally. We are polluting our digital environment with stuff that we didn't really maybe need, and we should have asked, should we build this? Because code isn't an asset, code is a liability. Even good code is liability. The more code we have, the more technical debt we're bringing on. What we've agreed to is vibe now, pay later. We are definitely starting to see the pay later part as well. Of course, it's been widely reported that after several outages, Amazon has now told its staff, you know those agents? Maybe don't use them. The rumor I heard was that there was a moratorium on agents for a week, and that junior and mid-level developers aren't allowed to ship agent-produced code without a senior sign-off. There's this very strong rollback.

The Return of Engineering Discipline

Another thing that's coming back is engineering discipline. Kevlin Henney said he wanted to write a talk called, I told you so, which I liked. Because these ideas that, "Maybe we should do small steps. Maybe we should commit as we go. Maybe we should do all these things." They're fashionable again, which is really good for those of us who appreciate these. TDD is back, which as a TDD advocate, I love. The spec is back. I'm less sure about this one, I have to say, but maybe the specs will be different this time round. My first role, we had to write a spec, it was 500 pages. Mixed feelings about the return of the spec. The reason that we need this is to try and get a handle on something that's happening in our systems, which is epistemic debt. This is something I heard from Simon Wardley.

Epistemic debt is the collapse of competence. It's what happens when you ship code you don't understand. This is also a really old idea. Remember COBOL that looked like English? Dijkstra said the use of COBOL cripples the mind. He said COBOL was so terrible that teaching COBOL should be considered a crime because people who had learned COBOL could just never become proper software engineers. This is a balance that we've been struggling with for a while. This is a tradeoff really between getting stuff done and understanding the low-level details. There's a lot of snobbery around this, I think. Hello, Dijkstra. Understanding the low-level details sometimes has value, but ultimately none of us, no matter how good we are as engineers, understand all of the low-level details. None of us are hardware engineers. Very few of us are programming in assembler because we want to get stuff done. We have to operate at that higher level of abstraction.

Connecting The Dots: Debt, Tradeoffs, and Circular Ideas

Srini said I was going to connect the dots. I think I've connected all of the dots or at least I've talked about all of the dots. I've talked about three things. I've talked about debt. I've talked about tradeoffs. I've talked about circular ideas. We've had ecological debt. We've had financial debt. We've had technical debt. We've had epistemic debt. We've had sleep debt. What is debt? Debt is a tradeoff with the future. These ideas keep coming back round because they come back round when the tradeoff shifts. When all of a sudden the tradeoff tilts because the outside environment changes, an old idea that we had all dismissed can become fashionable again. At that point, it doesn't look like a tradeoff. It looks like we're having our cake and eating it too. When I was talking to the program committee about this talk, we were talking about, what will you hear that will make you better at your job?

I think this is, how do you get better at seeing the repeating patterns? My first thought for this was to be old. That's not really actionable advice. It's not even true, actually. Probably most of you have heard of Greenspun's 10th rule, "Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of common Lisp." This is exactly a prediction that things are coming round again. Anybody want to guess how old Greenspun was when he wrote this? He was 29. He wasn't old and grumpy. He was young and grumpy, and he still came up with this insight. What I would really recommend you to do is always step back to these fundamental questions. What problem are we trying to solve? What's the tradeoff? I would really encourage you to be curious. Be curious about the past. Also be curious about the present.

Why are things the way they are now? Are there assumptions that have changed? Because if the assumptions have changed, that is a big opportunity. We're seeing there's a continual reset of assumptions. We're seeing an introduction now of analog computing, which can be far more energy efficient and performant than digital computing. We're seeing that SQL didn't die. It's coming back. SQLite has done really cool things with SQL. I mentioned Quarkus. Quarkus is really making Java as performant as other languages like Go. We're seeing GitHub had this enormous opportunity of moving back to a centralized model. Look for those opportunities because they can really pay off.

Conclusion

Nothing is new under the sun. There are no new ideas. Unfortunately, there almost always is a tradeoff, but sometimes there isn't a tradeoff, which is cool. Think about the sustainability of your IT. It really matters. It matters for the environment, and it also matters for people to have a sustainable pace of work. Of course, financial sustainability ultimately does come back to bite you if you haven't got it. Look for those ideas that everyone else has forgotten and recycle them, because you may find that as well as being sustainable, because you're recycling, they turn out to be really good ideas in the modern context.

See more presentations with transcripts

Parting the Clouds: the Rise of Disaggregated Systems

Murat Demirbas — Wed, 29 Jul 2026 16:00:00 GMT

Summary

Murat Demirbas discusses the shift toward disaggregated cloud database architectures driven by cloud economics. He explains how decoupling compute from storage enables elastic scaling, cost efficiency, and fault isolation. He shares how classical Paxos roles foreshadowed disaggregation, while analyzing network tradeoffs, shared-memory evolution, and self-assembling database designs.

Bio

Murat Demirbas is a Principal Research Scientist at MongoDB Research. Before joining MongoDB, he was a Principal Applied Scientist at AWS for 3 years, and a Professor of Computer Science at the University at Buffalo (SUNY) for 16 years. His work spans distributed systems and databases. He received the NSF CAREER Award and the UB School of Engineering Senior Researcher of the Year Award in 2016.

About the conference

Transcript

Murat Demirbas: My name is Murat Demirbas. I have been working on distributed systems for more than 25-plus years. The last five years I have been at AWS and now at MongoDB Research. My research areas are distributed coordination protocols, cloud systems, cloud databases, database systems, and lightweight formal methods. I maintain a very popular blog at muratbuffalo.blogspot. We are going to talk about disaggregated systems, what they mean. Economics drive the decisions for cloud architectures. The cloud economics demands that we decouple compute and storage. The reason is that compute and storage have inherent impedance mismatch. What does that mean? Compute is costly. Storage is cheap. Compute demand fluctuates quickly. Storage demand is stable and growing. Compute can be stateless. Whereas, inherently, storage is stateful. They don't belong together. What we do when we put them in one box and tightly couple them, if we need more storage, we end up having to sell more compute with it, making the customer pay more, or vice versa.

This is against separation of concerns principle. We like compute to scale elastically as storage stays cheap. As Jeff Bezos famously said, your margin is my opportunity. There is a margin here. The cloud is all about cost reduction. Actually, the cloud had embarked on this. Right now, the cloud architectures, data center architectures are disaggregated data centers. This was made possible by improvements in high-speed networks. Right now, hundreds of gigabytes per second bandwidth is the norm. This is up by an order of magnitude from 10 years or 15 years. We have also more advanced networking technologies like RDMA, SmartNICs, and CXL, all working for this.

Benefits of Disaggregation

What are the benefits of disaggregation? With disaggregation, we can make the compute scale up. We can have a better box, which helps for latency. Or after we scale this up, we could also scale out, horizontal scalability, so we can have more bandwidth. We can serve more requests in parallel through sharded processing. As important as that, we can scale down to zero. We scale the compute to zero. This is very important for customers. Why? Because they want to have pay-per-use. The customer value proposition is there, pay-per-use. The rest is just history and details. The technical side is going to work on making this happen. Another cost benefit of the disaggregation is I/O pooling at the storage tier. Now we could have multi-tenant storage and pool them. We say from both storage, make, utilize it, and network better this way. It does not get as much attention coverage as the elastic scalability benefit, but the operational benefits of disaggregation is as important. Disaggregation gives you fault isolation. When a node crashes, you don't lose both compute and storage node. You lose one. Compute nodes are easy to stand up over shared storage. You have fast recovery, simplified operations. Disaggregation turns databases from heavy stateful boxes they were once, into lightweight elastic services running over shared storage.

Context

This is a good point to introduce my high-tech setup, Org Beamer. I'm still the only one left that compiles their presentations. This is compiled by LaTeX. The nice thing I like about this template is that you can follow my progress on the top bar here. Next, we are going to talk about the architecture part. I'm going to introduce the disaggregated architecture, give you a taste of them from three sample applications, architectures. You are going to understand it. Then I'm going to do something very different. I'm going to take five minutes to convince you that Paxos did it before, disaggregation. How is Paxos related to disaggregated database systems? I will convince you, trust me, give me five minutes and come with an open mind. Then, this Paxos connection is also going to make the tradeoffs more clear. Then we are going to discuss the tradeoffs here and the opportunities, the methods to address these limitations in some part, the new challenges here. Then I'm going to talk to you about the future and ongoing work, which are not as mature as the things that I will talk about in the architecture section. These are very exciting opportunities. Finally, we are going to land this plane with the conclusion section.

The Architecture

This is the architecture track and we are discussing architectures. The database architectures went through three big phase shifts. Initially it was monolithic architecture. This is your dad's database. This has been around since 1970s. One process, one disk, local I/O. When I say monolithic, Postgres, MySQL, these are monolithic. Think about these things. Then we have the cloud. To get the databases into cloud, what we did is, for availability and durability, we slapped Paxos, Raft on them. We have Raft replicated groups of this monolithic database. One becomes primary, Raft ensured no split-brain, good coordination. The followers are following, and they are serving the reads. If you look at the architecture, this is the monolithic architecture. The primary, the compute and the storage, they are still in one box. We just had three boxes, and we basically said that this Raft group is a virtually infallible node. For horizontal scalability, we had more Raft groups, and we did two-phase commit on top of them for distributed databases.

It's a hack, it works. What is the problem? We didn't solve the disaggregation problem. We still have the problems here. This is not good for economics, and everything is about economics. As you get old, you understand this better. The thing is the cloud already does storage replication. If your storage here uses EBS, the cloud already replicates it three times. You have nine times replication of storage and three times replication of transactions. This is not a good deal. You could also do NVMes, but those are also costly. The next shift in the architecture is the disaggregated architecture. Here you can see the ephemeral compute now disaggregated from the storage. We even disaggregated storage into Log Stores and Page Store, which I'm going to talk to you about soon. We have another compute. We don't have to have three computes. We can just have primary and a failover secondary that follows closely and ready to take over. We didn't have to automatically go to three-way, because we need three-way durability. We need three-way availability. You can see the first benefit of disaggregation clearly from here.

Before we dive more into disaggregated database architectures, I want to make this point clear. Disaggregated databases are already here. Amazon Aurora, 2015. Alibaba PolarDB, 2018. Microsoft Azure Socrates, 2019. Huawei TaurusDB, 2020. Google AlloyDB, 2022. Disaggregated RocksDB from Meta as well as Rockset, around 2022 again. Neon, which got acquired by Databricks, 2024. Rockset was acquired by OpenAI also. Amazon DSQL, which I also worked on for two years, got general availability 2025. They share domain themes, shared storage, stateless compute, the network replaces the I/O. This is critical right now for the compute to talk to storage, you go through the network. This shapes all the design decisions as we will see next. For durability, we have log replication again. This is maybe short-term durability. For long-term durability, you could have snapshot at S3, or your favorite object store. Finally, for availability, Page Store to serve as buffers from the compute nodes. The idea is that data center is the new computer. It's a giant computer, and the network layer is the backplane of this giant computer.

Aurora. Network is important. We don't want to cross the network too many times. We don't want to bottleneck the network more than it is a bottleneck. Aurora's main innovation was to use the log is the database idea. Later architectures mostly followed this idea. This started with MySQL, the redo log of MySQL from the primary compute, you recognize these boxes now, is sent to storage first. When a write comes, the write is not acknowledged back to the client until the primary, the primary is always a read-write node, and you could have secondaries as many as your heart desires as read-only nodes. The primary before replying the write as acknowledged, it replicates it to a quorum of storage nodes. Aurora selects this quorum as four of six to tolerate f = 2. After four of six replies, the primary replies back to the client saying that it's done.

Reads, the primary can serve it. Replicas can also serve reads because the primary also sends the redo log to the replicas, and replicas, these secondaries, followers, opportunistically materializes things from this redo log and they could serve buffers. If they don't have it, they go to storage. This is great. We are just sending the redo log, not the whole state, and the storage materializes the redo log. Wait a minute, how does the storage materialize the redo log? Does the storage have CPU? Yes, the storage has CPU but it's limited. The compute-heavy nodes, the compute nodes have more CPU. They also have some storage but it is not mounted drives like EBS, you don't need them. It's opportunistic. The storage also has CPUs but not as big as the compute nodes or they don't elastically scale as that. Think of this as more opportunistic. The first challenge is done there.

The second challenge is, how does the storage know how to materialize this? Aurora did a surgery on MySQL, later Postgres, they put half of that materialization logic, the storage logic, they put on the storage nodes. The storage nodes, when they get a redo log, they know how to generate these buffers. This is not ideal because every time a new version of a database comes, you have to tend to those wounds, stitch them up, do some operations. This is not ideal. This lets us to just use redo logs, and it's a good invention to be network cognizant.

My next sample from architecture is PolarDB from Alibaba. They don't go through this surgery. Instead, they have MySQL and InnoDB both at the compute. They still send a redo log, orange is redo log, to ROs, read-only nodes. Since these guys also have the InnoDB, they can materialize state and follow and serve reads, good. Since the storage nodes don't have the InnoDB, the storage engine, now the primary needs to push the entire state to the storage nodes. It not only pushed the WAL logs, the redo logs, it also needs to push the state pages for availability, because this guy, if it doesn't have it in its buffer, it will go read it from the storage node. Alibaba has developed PolarFS, it's a distributed file system. Alibaba also just throws a lot of money into hardware, so they have RDMA, so they push these things through RDMA to the storage nodes.

One invention here is parallel Raft. Raft serializes everything, which is not strictly necessary. These guys are saying that if they are not operating on the same keys, you don't need to serialize that way. You can have a parallel version of Raft. They talked about it. Again, read-only nodes serve reads from memory if available, else they use storage to get the pages. You do WAL log replication for durability, they have a quorum of two over three. Then you also serve pages from storage. The PolarFS layer is oblivious to what it is replicating, it's opaque to database internals.

TaurusDB takes the Aurora lessons a little bit further. This architecture is close to Socrates, but simpler, less moving parts. What they do is, again, we have a primary read-write node, we have follower read-only nodes. The primary sends to Log Store. The innovation in TaurusDB is split storage into Log Store and Page Store, and prioritize the log. This even reflects in how they replicate the log. They have custom replication for Log Stores. This is I/O scattered over storage nodes. The primary says that, ok, three nodes, replicate this, tell me within a short time, if I don't get one, if one of you lags, this is not a quorum, I'm going to invalidate this. I'm going to choose another three nodes to replicate. We are in a hurry not to fix things, and they keep going that way. I'm ok with it, because they have a time budget and they go to the next node, invalidate and fix things.

We use quorums to get rid of tail latency, but this also works. They scatter I/O to other things. After the log is replicated, the write is acknowledged, the log is also sent to Page Store to materialize the pages. They also have this storage engine there to materialize. Log Store also sends the log to the RO, read-only nodes, for them to materialize and serve reads. I'll stop here for the architecture examples, but you get the main idea. Primary compute, that's read and write, followers are read-only. We generally go with log is the database approach. We also need Page Store to be able to serve buffers as needed by the nodes.

Paxos

What does this have to do with Paxos? I have a bold claim, I'm saying, did Lamport invent disaggregation before it was cool? How many people know Lamport? How many people know Paxos? How many people know Raft? When Lamport described Paxos, he described Paxos in terms of roles. The client sends the request to a proposer, proposer talks to acceptors. After a quorum of acceptors say, are you guys seeing similarities? It sends it to the learners for materializing the state. Acceptors for durability, replicas, learners for availability, materializing the state. In Lamport's formulation, the learners send the responses. What happened? Let me first go through this, just to make sure we are on the same page. Proposers generate order values, they are leaders. This corresponds to compute, generate order values. Acceptors persist votes, the log. They persist to log before we go to Page Stores. This is durability, short-term durability, at least.

These correspond to the log servers in the disaggregated architecture. Learners serve state, availability, they serve the reads. These correspond to the page servers. How many people had seen these roles before? What happened is when we go to Multi-Paxos, in that paper, Lamport says that I'm explaining this in terms of these three distinct roles, but if needed for efficiency, you compact all the roles into one node. That's what we did with Multi-Paxos and Raft, because we were going for the shared-nothing architecture. We all put them all, then the primary node sends to two followers, which are also acceptors and learners. It's also a learner, and it's replica. It doesn't have to be that way. Basically, this is the recipe for disaggregated system here. We don't need to go very recent. This has been around.

What do we learn? What are the benefits? There is an analogy, so what? Who cares? What do we learn from this? Do we learn something from this? Yes, in this slide, proposers, leaders just have to be f + 1 to be able to tolerate f failures. In Raft, we are making it 2f + 1, so that's redundant, for example. Just f + 1 leaders is enough. Acceptors, 2f + 1 for quorum and not having split-brain, good. Learners, just f + 1 is enough. There had been two decades of research on Paxos, and I'm also guilty on this, I have been working on these Paxos roles, and using flexible quorums thing to improve the throughput and ways of reading from replicas. We can borrow from that research, is all what I'm saying. The other thing about disaggregated architectures we discussed so far, they always assumed an external configurator, consensus, like primary node.

Then somebody says that now the secondary is the primary node, where does this come? Outside of the system. That's not good. A better design principle is ingrained fault tolerance. Be self-contained, don't depend on another external service, because every dependency is a problem. I'm saying that we could have inbuilt Paxos, the consensus also in the disaggregated database services. This should be happening, if it's not happening. Raft squished them all for shared-nothing architecture. The architecture evolved. We need to revisit these things. There was a branch of Paxos that did not do it, in addition to my branch, what the academics worked on. It's the shared-log abstraction. Mahesh Balakrishnan worked on this at Meta, and also at Kafka, Confluent. The database is maintained over a virtual log composed of loglets. This is a beautiful abstraction. Virtual log maps logical addresses to physical loglets of finite size, via versioned metastore.

There is some dependence on the configurator. Fine, I'll live with it. The benefit is that you could switch on the fly the loglets, the acceptor configuration as well, when you go from one loglet to another. He famously did it for getting meta control planes from ZooKeeper to Raft-based, or Multi-Paxos based consensus on the fly. This is amazing. Again, two things are squished together. The database now squishes the proposers and learners. The acceptors are here, you do I/O scatter over them, and the shared log advances through fixed-size loglets.

Now I'm going to, of course, talk about my branch of work. This is now familiar to you, this is the disaggregated roles of Lamport, but there are extra layers. Now proposers, you don't go to acceptors, there is a proxy leader layer. We published this with Berkeley folks as a collaboration, we called it compartmentalized Paxos. We could have as well called it disaggregated Paxos. We further disaggregated the leader into two parts, proposers and proxy leaders. Now the leader just orders, proxies handle the I/O, because I/O fan-out and fan-in, it kills the leader. We don't want that. We could even disaggregate further to have proxy leaders. The acceptors, we disaggregated them. We could do, not through a metastore, we could do through the protocol itself, through flexible quorum result. I'm not going to go into the detail of the flexible quorum result, but we could do write quorum and read quorum for leader maintenance, not split-brain purposes.

We could effectively go from each decision to the next, not from each loglet to the next, more fine granularity, scatter I/O over acceptors. This is phenomenal. We could show just over the same codebase, multi-Paxos throughput go eight times. This is the power of specialization. Initially, leaders were the bottleneck. We have proxy leaders, that's solved. If you need more reads, add more learners to your architecture, that's solved, eight times throughput. What gives, what do you notice? Too many hops. What about latency? This also brings us closer to the tradeoff discussion.

Tradeoffs

We are going into the tradeoff section. We are losing something, the network. We cross the network. If we further disaggregate, we are crossing the network even more. The network is the new bottleneck. Disaggregation shifts bottleneck from CPU disk to the network. There are inherent limitations here. I'm going to talk about ways we address them, the methods. Remote I/O is three times slower latency and four times less bandwidth than local SSDs. In the next two slides, I'm going to show you methods to address this. You should do your own cost-benefit analysis. The costs are clear, network latency, some bandwidth cost, I/O amplification. The benefits we discussed, to remind you, elastic scalability, now you can pool compute, storage. Memory, which I will talk in the future part. We could also do disaggregation at the memory side. Fault isolation and failover. Based on this, you could do it.

Now let's look at the latency challenge and the mitigations. For log commit on the storage side, shared storage, we need to cross the network, so this is there. For page fetches, if I don't have it locally, I need to cross the network and get it. There is some synchronization and cache coherence costs that are also prone to this latency thing. What are the methods? I'm not going to explain them, but buffers help. We have seen read-only nodes. If they have big buffers, they also materialize pages, they could serve. When you don't have the page, you need to go and get it. Prefetching helps. Before, you're looking at the access pattern and prefetching these things so that you don't have to go fetch it, that reduces latency. Pipeline, I think I/O OS-level things, the thread-level things you could do, tricks you could pull here helps. Use faster fabrics.

RDMA is now more commonplace. CXL is becoming a thing. Use custom distributed coordination protocols. One nice example is with two-phase commit, it has two phases, two-phase commit, you have a prepare and commit. You could do the prepare going one way, sequentially, and you could do the commit coming that way. In one round trip, you could finish. There are tricks where we can change the underlying communication layout, topology to improve things. Throughput challenges and mitigations. Log contention, log and data pages contention is a thing. Log to log contention is a thing. We have several techniques already in these disaggregated architectures. Also, if you go back, the challenges are not new. The Paxos analogy also shows that. We have principles, the same principles apply. Log-as-database cut data movement by 2.5x. Aurora did a really nice thing by doing that, cut the traffic. The TaurusDB lesson is that prioritize log traffic over page I/O that could come asynchronously. Even in log traffic, you could do smart filter replay, discuss things further. We could do scatter I/O so that we don't have head-of-line blocking on the same storage nodes, acceptors, and we could do pushdown compute to data.

Future

That brings us to the future section. Pushdown computation to data, in this future section, this doesn't mean these are not there. This means that these are not as mature or we don't have as many examples. Taurus Near Data Processing already shows an example of pushdown computation to data. Amazon Redshift also has examples of this. Snowflake have examples of this. We need more of this. The Taurus NDP case, to compute nodes for a scanned query, it converts pushdown predicates, filter, projection, aggregates into LLVM bytecode and sends them to Page Stores. This is on a best effort basis on the Page Store. Nodes may skip this under load and send the whole pages if they don't have enough CPU to process this. If they do, they JIT compile this, execute this code to drop rows, trim columns, and emit partial aggregates early, so instead of sending whole scanned range of pages, you send a lot less pages. Instead of the whole range, you already do the predicate computation at storage node because they had CPU, and send less network. Win-win. Utilize things. Jeff Bezos loves this.

Second thing, we talked about storage compute disaggregation, and said that, you don't have to tie storage to the compute, but the compute is still tied to the memory. What if we didn't tie memory to the compute? Because right now, if we need more memory, we get more compute, we buy more compute. If we untie them, we could also provision them separately. This is a surprising statistic, 50% of DRAM in data centers is wasted by the static provisioning. If I'm not using my RAM, if you could use it to access some of my pages, maybe if we are working on the same database worker nodes, win-win. Luckily, RDMA, and there is an example of this, PolarDB Serverless says they did memory disaggregation, shared memory, but this is not commonplace yet. People are still debating the cost-benefit analysis here, so that's why it's in the future work.

Remote memory, because RDMA is not as mature, there are problems. Remote memory via RDMA enables compute to access shared elastic buffer code. Why is this important? This is important for cutting latency and improving throughput. Instead of going to the storage, I go to another compute node's RAM, and I get what I needed there. I'm elastically increasing the size of my memory pool. This is utilization, the same ideas. Recently, we have CXL. It's a newer networking technology. CXL fabric basically makes the tricks to CPU to treat remote memory as local memory. CPU thinks this is local memory. The Log Store semantics work there. This achieves six times lower latency than RDMA. This is phenomenal. What is the catch? The catch is now you are limited to a rack. With RDMA, you could do data center-level fetches, but with this, you have to be staying in the same rack. There is also the specialization allows different ways of doing things. We could do the log, we prioritize them. Oracle Exadata uses the more expensive persistent memory to ensure durability of redo logs. That's another smart idea.

We talked about disaggregating things, but this disaggregation also enables us to unify things where they count. There is this code, if you are disciplined, it enabled you to be more flexible. If you are more disciplined and have this shared storage layer, on top of this, you can unify things. Google AlloyDB shows a nice example of this. It makes sense to unify OLAP and OLTP to be able to serve the analytics on the most recent, most fresh data. Instead of having a separate OLAP cluster, if both OLAP and the OLTP work on the shared storage node, this is easier to do. Again, you recognize this architecture by now, primary read-write node. Read pool nodes, these are read only. There are now both columnar engine that could even use vectorized processing, and row engine on the same compute node, so it can serve hybrid transaction analysis and processing. Again, database storage engine on top of a distributed file system like PolarFS. In this case, it's Google's Colossus.

Another interesting point, this is disaggregated RocksDB through Rockset architecture example. Here, this shows a more ecosystem-y disaggregation. This is the ALT architecture, Aggregators, Leaf nodes, Tailers. The disaggregated RocksDB lives here, both compute nodes and overshared storage nodes. The writes and reads boundary, the writes come from this side, reads come from this side. Tailers tail Kafka logs and get the real-time logs. Leaves build columnar inverted indexes from tailers. Aggregators run read-only queries, distributed SQL over indexed shards. Disaggregation beautifully isolates bursting ingest from indexing and querying, bursting querying. We have a disaggregated architecture. The heavy lifting here is done by LSM trees. So far, we talked about B-trees. LSM trees are beautiful things because they send logs as immutable SSTs. This immutability enables them to share safely so that compaction can be done on any node, whereas in B-tree, it required random remote writes and page coupling.

This is not a problem with LSM tree architectures. We need more work on disaggregation on LSM tree. There is ongoing work. People are using LSM trees more and more for disaggregation. Since I have talked about, it's easier for any node to be able to compact, this paves the way to serverless. For a truly serverless database, you need a disaggregated database first. Disaggregation, because stateless compute, the serverless idea, stateless compute depends on shared storage. Remote pages and logs let workers appear and vanish safely. Any worker can serve any shard, fast failover, low cold-start.

Conclusion

Open questions. What is the trend? The trend is we disaggregate. Can we take this further? Can we have self-assembling database management systems? Can a database management system auto-configure compute, memory, and storage as workloads shift. Can we have auto-tuning systems? We are going to need them for dealing with agentic workloads. They have bursts that are 30 times worse than the bursts we see from normal traffic. When an army of agents start hitting your databases, how are you going to deal with them? This is an open question. The second thing, a benefit from all the work on AI is new hardware is coming. With disaggregated architecture, we have the gift of specialization. How do the databases exploit this new hardware? I recently have seen a GPU-based version of DuckDB. That's, for example, an interesting approach. As a distributed systems person, this disaggregation shapes how we do coordination.

It also shapes failure modes. In the shared-nothing, we had private logs. Most of distributed protocols are designed for that. With shared storage, it replaces private logs. The decisions are not necessarily done by messages or messages talking to other nodes. Now, they could be log-mediated agreement. For example, we could have a commit or consensus by just compare and swap in a shared log. MemoryDB gives an example on this. Leader election, just make sure you know the last number, switch it by writing it to the log, and then you also fence off the other leaders. You could have leader election by just writing to the log this way, for example. You could also stand new nodes by replaying for learning and recovery. How does this shape? What are the new distributed coordination protocols for this new model, new architecture? I lied to you a little bit. I said that disaggregation makes fault isolation because you don't lose both the compute and storage.

Those are true, but this works for maybe master single file spell, but this also opens up another front. Only the paranoid survive in this distributed systems business. We should be thinking about the downside. The downside is, we don't lose both the compute and the storage, but we lose some fate-sharing. The more components, the more moving parts, the more we are trailing the information, front from behind. There are delays in learning things. These delays, they accept maybe failures to propagate across layers. This may open you to metastable failures, which is a new area of research. Especially when we talk about this dynamic composition of systems, how can we make sure they are correct? How can we vet them for metastability? Simulations seem to be a good approach here, but we are still doing the science of metastability.

Learnings

The takeaway. The future is this fabric-aware databases that self-assemble microservices. We specialized the database, disaggregated into services, and we assembled them into a database which treats the data center as a giant computer. The challenges are not new. Through the Paxos examples, I hope I conveyed to you that these are the classical distributed systems challenges. We have new opportunities through this hardware, through this log shared storage architecture. As the famous industrialist Emerson said, "There are many methods but few principles. If you master the principles, you can pick and choose your methods, but if you try to do the methods without principles, you are going to have trouble." The idea is, first get the principles right and then choose the methods. I'm excited about this new architecture. We know the challenges and we have new opportunities to address these challenges.

See more presentations with transcripts

Getting Rid of LeetCode Interviews in the World of AI

Daniel Doubrovkine — Tue, 28 Jul 2026 16:00:00 GMT

Summary

Daniel Doubrovkine explains why traditional LeetCode whiteboard interviews fail to evaluate senior engineering talent. He discusses his own experience bombing basic algorithm tests despite decades of leadership, and shares actionable frameworks for redefining the interview loop. Discover how evaluating human judgment, system design, and hands-on AI collaboration yields far better hiring signals.

Bio

Daniel Doubrovkine is a seasoned Entrepreneur, people leader, open-source enthusiast, and lifetime artist. Daniel is former CTO of Artsy.net, led the Augmented Engineering team at Shopify, and served as Principal Engineer at AWS in New York, working on OpenSearch.

About the conference

Transcript

Daniel Doubrovkine: We're going to talk about getting rid of LeetCode interviews in the world of AI. How many people here are developers? I did a survey amongst developers about coding interviews and writing code on whiteboards, and things like that. About every single person I've asked really hates these LeetCode interviews, and "Feels that they're really dumb." I know some of you don't believe my data, and if you ask me for the source of this survey, I included it here. It's right there on the right. It's in Python. It's the source of the data in the survey. I promise you there will be many jokes here and some personal stories. It's really time that we get rid of these LeetCode interviews. I'm not going to bore you with pros and cons of LeetCode style interviews, why they exist and things like that.

I'm going to tell you a story instead. That's actually a very personal story, and one that I'm sad to admit is that once upon a time, actually exactly a year ago, I failed a very basic coding interview at one of the five very large FAANG companies. Very embarrassed to say that it happened. I didn't just fail a coding interview. I failed to implement the diameter of a binary tree. How many of you can implement the diameter of a binary tree from the top of your head now? It is a totally basic DFS, Depth-First Search. It's exactly 10 lines of code. This is one of the most absolute trivial, as the interviewer said, warm-up exercises that you can ever get in a coding interview at all. It's like you should be able to dump it from the top of your head. It's 10 lines of code with the function definition, the comments and everything like that.

It's five lines of actual code. In fact, just to prove to you that I can actually do a piece of code, the diameter of a tree, here's a recording of me doing it right there on the board. Look, I'm implementing the diameter of a binary tree, and it's going to give me the code. I made it into a GIF. I didn't want to type it. Yes, I'm doing it. You mean I didn't do it? Claude did it? Yes, you're right. Claude did it, but I asked it to do it. It's amazing. I was born in the USSR, and we had our share of difficulties of getting things done, of buying bread or finding technology like a VCR, things like that. Today, these tools like Claude, they're available to everyone, and they're available to everyone equally. We don't need to seize means of production. This is readily available for you right here at your fingertips. These tools are promising us a really bright future, a whole brave new world. Maybe in this future world, we're not going to have to solve puzzles on whiteboards.

Back to me. When I am on the receiving end of these coding interviews, when I'm debriefing candidates, and I see a candidate who has been doing code for quite some time, and they fail miserably at coding some of these puzzles, because, of course, I sometimes work for companies that do them. I just look at the room, and we try to talk ourselves out of the situation. We're like, ok, maybe this person that has been administered yet another diameter of a binary tree, maybe they're just too senior, maybe they're just rusty. Maybe they haven't coded for a long time. It's ok. Maybe we can convince ourselves that we should still hire them, even if they fail at these coding interviews. I worked at AWS. I think more than half of the principal engineer interviews completely bombed their technical coding interviews and still were hired. I've been in these debriefs endlessly.

The problem with me is that, actually, I'm not rusty at all. At the time of when I bombed my diameter of a binary tree, I was coding every day. This is my 2025 GitHub. You could say, ok, Daniel, 2025, you've been doing AI slop. You've been telling Claude to write the code for you, and so your GitHub looks amazing because you've been committing all this stuff with AI. 2025, I buy it. 2025 may have been all AI slop that I produced sitting behind the keyboard and telling it what to do. 2024, possibly AI slop, a little bit. 2023, pretty cutting-edge AI slop, if I was doing that at the time. 2022, definitely not AI slop. 2021 and 2015 and going all the way back, like in Hilary's keynote, we were really writing code. My GitHub green squares look pretty full, haven't really changed since 2014 or something like that.

I've written more tree structures. How many of you have written a tree structure of any kind, at any time? I've actually written, I think, more tree structures than every one of you combined here. I started in the '90s, and my first successful server-side commercial application was a search engine called Alkaline. It powered a small website, amongst others, called whereas.com. I don't know if you remember whereas.com, but whereas.com in '97 was the eighth most trafficked website in the world. I think we were hitting something like eight requests per second. Eight requests per second in '97 was completely insane. It was very hard to stand up a C++ server-side search engine that could respond to eight requests per second within millisecond response time. Whereas.com was used massively, because everybody needed a serial number for their favorite piece of software, and that's where you go to find it.

That search engine also was ported on multiple operating systems, Linux, Solaris, SunOS, Windows, IRIX, you name it. I didn't do Vax, but people were asking for it. Everybody was running this different hardware in these different operating systems, and so we needed a portable baseclasses library that could be compiled on various operating systems. STL was not really a thing at the time, or at least not performing well. This source code is actually on GitHub, but it's tree structures upon tree structures to make a search engine actually work, very different technology. I've written all this cross-platform tree structures to make sure they work everywhere, time and again.

In 2025, ironically, I found myself to be a top 1% Ruby engineer globally, according to this website called Algora. I actually co-maintained 65 or some Ruby gems. I have some credentials that was coding, and yet, even though I have all these amazing credentials, I completely bombed the diameter of a binary tree in a live interview. If you don't believe my code on GitHub, and you don't believe Algora, and the fact that I was top 1% of engineers out there, because you can say maybe you faked commits back to 2014. You can totally lie on your resume. At that time, I was on my sixth year of being a PE at AWS, working on OpenSearch and Search Engine. I was paid absurd amounts of money. If you don't believe me, this is a photo of me on my yacht. I see somebody is like, this is AI.

This is not actually me on my yacht, no? If you don't believe me, here's me in front of my Lambo in Hudson Yards in New York. I promise you, this is not an AI photo, and just believe me that this is my Lambo, and people pay me a lot of money to write code all day long. Definitely not AI. Plus, this is in Hudson Yards. It could be somebody else's Lambo, but I do live in Hudson Yards, and Hudson Yards is the most expensive neighborhood right now in Manhattan. Apparently, according to TimeOut, average home sale price is $6 million. It's pretty intense. I live there. While I'm on this topic, I want to show you my gym. I work remotely from here, often. This is a real picture. This is not AI. This is a drone shot of the Hudson Yards Equinox, where I do have a membership, and this is my office. In summer I glue a thing on my screen so that nobody sees me. I'm not a PM. I'm a developer. I write code just sitting in the sun and in front of the pool writing the code. That is, of course, when I have a job, when I don't bomb interviews for diameters of trees in those interviews.

Studying Computer Science

We're jumping a little bit ahead. Let's start from the beginning. I was born in Moscow in '76, a long time ago, in the USSR. This is actually my building where I grew up. I grew up exactly right there in the middle on the second floor in an apartment that was like a normal New York-sized apartment, like 300 square feet with my parents, typical thing. Like all Soviet kids, I was really good at math. That comes with the territory of being East European. Being good at math led me to studying computer science. In the late '90s, when I was studying computer science, I absolutely loved algorithms. I was writing Hanoi Tower problems, solving them in three different programming languages, C, Lisp. I wrote x86 assembly. There's some code on the screen here, also you can find that on GitHub. I was trying to squeeze algorithmic solutions in x86 assembly just to get some performance out of these implementations.

Then I'd implement this time and again and spend a lot of time coding these tree structures and other solutions. I was implementing all these algorithms time and again, spending a ton of time in the computer lab in college doing this. One of my classmates even found an iterative solution to the queens problem, the queens that don't attack each other, to find the next position or count the number of solutions available. This was an actual computer science breakthrough that was published in a math magazine at the time. It's a pretty amazing thing. I was so obsessed with algorithms and structures and data that my very first commercial application was a calculator. There's an actual picture from it. I still have an unopened copy, but this was published in Germany, 3,000 copies on the CD-ROM and sold on shelves. This was called Expression Calculator for its shareware version. The commercial version was renamed to Global Calculator in German. My German is terrible. This, I got a paycheck, $3,000 for writing algorithms and calculators, like a basic interview question of how to make one.

Working at Microsoft

I started two companies, and then eventually I started interviewing for large American companies, because I needed a job because my second startup had failed. My first interview of a U.S. company was for Microsoft. I'm actually working back there now, 25 years later. What I remember from that interview is that Microsoft brought me from Geneva, where I lived, to Paris, put me in an amazing five-star hotel, the Le Louis XV. I don't remember what the interview was, but I remember that it was mostly coding puzzles. I just stood in front of the whiteboard and coded puzzle after puzzle after puzzle of like how many of this and all tree structures and things like that. I remember it being extremely hard. After I got hired, I was told that on that day, the team that was interviewing only hired two people, me and the other guy, who's this French guy who I'm still friends with, who finished top of his class in the hardest telecom school in Paris you can imagine.

The thing where only genius level people can go. Now me, of course, I don't know why I was selected, but clearly, I was able to code these puzzles quite well. I didn't graduate with honors from any school. I was very mediocre at many things, like the AI is today. I absolutely love these coding problems. Then when I started working at Microsoft at the time, I worked on this project called Netdocs, which was like Google Docs, but never actually shipped. I worked on the server side of it. One of the biggest problems that we had on the server side of the software is memory fragmentation. We would allocate memory, release memory, allocate memory, release memory, and would run out of memory eventually because memory was fragmenting at the time on Windows. To solve that problem, we wrote a completely new from scratch C++ library that mostly used the stack as much as possible and then bled into virtual memory when it was running out of stack space.

That solved much of the memory fragmentation other than rebooting servers all the time. That was difficult and that was interesting. This is a library that had tons of tree structures to optimize performance. It was like STL for servers. STL was fragmenting very much, and we invented our own. I implemented everything from tree, graph, vector, strings, tons of practical applications of algorithms. When the project got canned, I wanted to continue to work on algorithms and solving this type of problems. There was a team that was hiring, the Rotor team that was writing the .NET framework for Linux. Before I took my job at Microsoft, I was like, Windows, never. I saw a job offer and I said, maybe I should take this job. Finally, I had an opportunity to go work on Linux again. I really wanted to succeed that interview, the internal interview for the team that was very competitive.

I was put in front of a whiteboard to code a bunch of things, again, tons of algorithms, tons of C++ implementations of all these puzzles. Then I got in front of a whiteboard to implement malloc, a memory allocator from scratch. I think this is the first time I truly bombed the whiteboard interview. I wrote something, but I don't think it was any good. I'm sure there were many other people who were a lot better than me. My self-doubt, the imposter syndrome, all of that really came back up. Like you pass an exam at school. I just remembered while I was standing there trying to code malloc how in like second grade of college, I had to pass an analysis tree exam and I failed it twice. Then I had to spend four months basically preparing for that one exam or risking being kicked out of the math faculty. Here I am standing, failing at the coding interview again. It's, again, very tree structure-like and very algorithmic. I, of course, didn't get the job. That really created a lot of doubt in myself for these coding interviews. I always dreaded them.

Tenure at a New York Startup

In 2004, I wanted to move to New York. I went and interviewed at a very large bank. I was flown and posted in a typical not very fancy hotel, unlike my Le Louis XV experience in Paris, and was put in a windowless room and left with a computer, which was nice. It wasn't a lot of whiteboard interviews. Was asked to code all kinds of things, mostly unsupervised. I did actually quite well at those because most of the software that I wrote, it was like C, a little bit of Fortran. Most of that software worked because I was sitting in front of the computer myself. I would be given a problem and the person would walk out and come back and we discuss what I wrote. I did very well. The interviewer, the manager came at the end of the day. Like I've been in that windowless room for like six hours.

He puts a paper on the table. He's like, this is a job offer. I looked at the job offer. It was double what I was making at Microsoft. I was like, that's cool. I'm not going to take it because something really concerned me. They were talking about how they don't sleep and have PagerDuty 24/7. I was like, this is not why I'm moving to New York. I don't want to be on call like 24 hours a day, but this offer is amazing. Unusual to get an offer on the spot, but still. Another unusual thing about that interview, I was forbidden to contact any other people at the company. I actually was going to meet another manager who also had a job, but it was a little bit more interesting. That wasn't in fixed income derivatives, which tanked the market later, but I wouldn't know it in 2004.

I was told, you can't call that person anymore. That was weird. I was like, this is a bit weird. I haven't seen the sun all day. Maybe let me go and I'll think about it. Eventually they let me out of this room. The moral of that story is that I was able to do these LeetCode questions. I just needed to concentrate and not have somebody breathing over my neck.

Instead, I actually took a job at a New York startup. They flew me back to New York, I think three days after the bank interview and put me, not into a crappy hotel, but at the Royalton. It's a five-star, super fancy hotel in Midtown Manhattan. At the time they didn't have many developers at the company. It was just a startup. I pretty much got hired on my Microsoft credentials. They're like, of course, this guy knows how to code. He comes from a very important, large company. I wrote a data mining client in C++ that traversed your Outlook and Lotus Notes email to mine it to build a social graph out of it. Notice, tree structures, social graphs, and all this kind of stuff. Think of that application as LinkedIn reads your email, quite exciting. I know a graph is not exactly like a tree, but they have a lot of similarities.

I think I could at the time definitely implement the diameter of the tree, had it been necessary for that application. I had my revenge. Part of my revenge was working for this company where we flew a private jet to open our second office in San Francisco. That was really cool. The second revenge that I had at my failing .NET team interview is that I was giving a lot of interviews myself. The company had raised the series B. We had tons of money. We had a lot of salespeople, but not a lot of engineers. We didn't have a product to sell. We were definitely growing. Our CTO was trying to hire a VP of engineering at the time. He, of course, wanted the VP of engineering to meet the people who were working in the engineering team of the company. He's going to be my future boss.

He asked us to prepare a technical interview for the VP of engineering to see if they can be respected by the engineering team on their technical credentials. The only interview I remember is this guy walks into the room. He's got credentials as long as maybe I have now. Long LinkedIn. He's wearing a suit, which we appreciate in New York. He's telling me as an introduction that he writes code all day, every day. I'm like, this is somebody I would love to work with. We're a small team of five engineers. We have a VP of engineering who writes code all day, every day, yes. Something told me that maybe that's not true. I said, if I ask you to go and write code on the whiteboard right now, like most engineers would do, will you do it? He's like, of course, no problem. I said, like a sorting function?

Can you write a sort on the whiteboard if I ask you to do it? He says, of course, you could ask me to do it. I could do it. I'm like, go do it. The guy, his response was amazing. He said, "If you're really asking me to go to the whiteboard right now and write the sort, I'm walking out of the room." I said, "Feel free to walk out of the room." I was so sure I dodged a bullet. I had my power trip of sitting on the other side and administering a coding interview. There were many reasons why that person didn't get hired. I thought we really dodged it, because he just told me one thing and didn't want to do it.

Crafting Torturous Live Coding Interview Questions

The more I did these live coding interviews from the interviewer side, the more I liked it. I was getting the real power. I could make a candidate really suffer — it's a very Russian thing — at the whiteboard. I came up with a few methods and wrote about them, about how to make candidates suffer. Number one, you have an expert in something, ask them the dumbest question possible as a warm-up. It's truly insulting. Somebody who has credentials, who can speak about algorithms, wrote tons of them. Just ask them to reverse a string and see if they can do it. They reluctantly will do it because they have to. I love it. The second one is from my own experience at this big American bank. Just put them in the room and leave them there. See how long before they go out of there, be like, can I please have some water?

Another one is, you have an hour. Some problems some people solve them quickly. Some problems, people solve them in a long time. Just ask them to solve it again in a different way. I don't know if there is another way to solve the problem, but they are the candidate. They should be able to tell you. When they say, I don't know if there is another way. "Thank you. Our interview is over. Goodbye." That really destroys the ego. Finally, the one that I absolutely loved in these whiteboard interviews is that as they write the code on the whiteboard, I'll be like, "Keep working on this. I'll be back." Go out, smoke a cigarette, have a coffee. Come back. Look at the board. See if they're stuck. Try to help them again. Maybe just leave them there. Like, "That's ok. We can stop here. We're done. Thank you." Have them get out.

The quicksort there was from my own experience as well. You can guess what happened to that company where I was administering this type of torture to the candidates. The company, the software was fine. The business, of course, failed. We just ran out of money. It didn't matter what we were interviewing these candidates for. Since then, I felt bad. Every time that I was in the room administering a whiteboard coding interview, I just thought, why am I doing this to these people? I would open my interview with saying, "I'm sorry, we're just going to have to work through this together. Let's work through it. I just have to do it."

Growth at Artsy

In 2011, I wanted to go work for a startup. Got introduced to the founder of Artsy, a New York-based startup. I was the seventh employee. He just needed some advice of how to rebuild the site after a failed demo. I went around startups, asked how people build technology today in 2011 at different startups. I was told, use MongoDB, use Ruby, Ruby on Rails. In about three weeks, we rebuilt the site. I joined the company three weeks down. There was no coding interview. My coding interview was the prototype. I got hired on writing the prototype, the thing that would eventually become this company. We'll go raising $100 million and building the largest fine arts marketplace, most relevant arts publication. It's artsy.net today. Still a global company with a ton of employees. Working at a startup, it really teaches you a few things. Number one, engineers, they aren't lining in front of your door to get hired at your little startup that nobody knows about.

The really good engineers, they're totally impossible for a startup to find. Your whole proposition is that you're going to pay somebody half of the money to work twice as hard. That's really what you're offering from a startup. We went and designed a deliberately very different interview process at Artsy. That process was always centered around things that someone has built. Not some hypothetical coding puzzle, but something that I've actually built. We'd ask them, "What have you made? Tell us about it. Tell us a story. You worked on a project reticulating splines. What's a spline? I don't know anything, explain it to me. How did you build one? How did you reticulate one? How many splines have you managed to reticulate at the end of the day?" This kind of stuff, from their own experiences. Then, we decided that the kinds of people we wanted to hire were these T-shaped people.

People with broad set of interests and narrow specialties. We just wanted to learn about what interested them and where they got really good. One of my best stories, I met an engineer who was at the Art and Tech Conference. We were given a little project to do some soldering and some art thing that would blink. One of us had to do software and the other one had to do hardware. She said, I want to do hardware. Took a soldering iron. Completely blew the circuit up the first time that she tried to solder it. She was like, maybe it will work if I flip the cables. Flipped the cables and soldered it, and it worked. I was very impressed by the just trial and error, which is how I code, the trial and error of doing that hardware. Then this person ended up telling me that she went to RISD to get an MFA in painting.

She was an amazing painter. Then went to Flatiron School to learn to code. We hired her, she was our first junior hire at the company, and did amazing. Today runs, I think, a large consultancy that builds software, 15 years later. Very broad set of interests, art, technology, very good at what she did. The interview process would be an interview with an engineer, there'd also be an interview with a human. We want to see how you work, how you collaborate with others. We want to learn about your experience collaborating with other engineers, and see how you'll fit in a startup where you know everyone, it's a 10-people company, from your past experiences.

Then finally, of course, can you do the actual coding job? We don't need to test you on specifically writing the code, we can just talk about code that you wrote before. Ideally, if you have open-source code from before, we'll just discuss that, and that's good enough to see that you can code. Maybe we'll give you an assignment to take home, and then discuss it during the interview, if you don't have these kinds of credentials. Code is one thing, but you have to design systems. The process of designing a system is interesting, and so we want to understand your systems thinking decision-making. We'll do some whiteboarding of just building, designing a system from a high level, and see what questions you ask in a typical systems interview. Finally, that interview would have a round with somebody very senior at the company, maybe a director or better.

It's really an opportunity to ask questions about the company, roles, expectations. We ask some competency questions at the time to make sure that the person has value alignment with the company and potential for growth. You can see that very little bit is about the actual coding puzzles and things like that. In fact, what I told you before is just half of the interview. The other half of the interview was references. We would call people that you would give as references and just discuss them, what did this person build? Were they good? Why were they the best in the team? Why should we be hiring them? The story is that someone tells about someone else. It's like NPS score. How many people would recommend working with you that have worked with you in the past? That turns out to be a much better signal about somebody's abilities than coding.

We ended up with some pretty awesome and diverse candidates. We sourced a ton of them through open-source collaborations, and some we sourced through non-open-source collaborations. Some people would apply to the company and really appreciate our approach to interviewing without these whiteboard interviews. They would write about their positive experiences online. Sometimes they would not write about the actual interview process, but about how it felt to be in an interview like that that didn't involve coding puzzles. To date, I think that process is some of the best.

The Individual Contributor Role at AWS

In 2019, I stepped down from being CTO of that company, and I wanted to become an IC again. I just burnt out on managing people and company growth with 400 people. I also wanted to learn something new. I thought, maybe I should go and be an IC at the most interesting technology company to me at the time. I joined AWS. I was flown to Seattle for a principal engineer interview and was told to brush up my algos. On the plane, I thought, I have not brushed up enough algos so far. I wasn't doing LeetCode interviews. Maybe I'll do some on the plane while I'm there. Before I fell asleep to a movie, I stumbled upon an LRU cache. It's like a linked list and a hash table implementation to make reading and writing fast. It's like Least Recently Used. You bound it on some max size, and then it's fast to insert and fast to retrieve.

I was like, this is fun. I haven't seen one of those in a while. I read about it, and then I fell asleep, and then I land in the interview. One of my interviewers is Marc Brooker, he's a distinguished engineer at AWS. The man invented everything, from compute to Lambda to hypervisors. I'm like, I'm standing in front of an impressive engineer here. Then he's like, unfortunately, we're going to have to code a little bit together. I go, sure, I'm prepared for that. Of course, I'm absolutely not prepared. He's like, implement an LRU cache. I ended up working at AWS. He seemed to be quite impressed with my implementation of LRU cache. The only reason I remember this is because I got lucky, and I brushed it up on my way to my interview. I worked on OpenSearch, I've worked on Elasticsearch, tons of code, things like that.

In 2024, I decided to leave, and I started looking for a new role. That is when I failed my diameter of a binary tree. I was actually only prepared to implement an LRU cache. I was absolutely not prepared to implement the diameter of a binary tree. The interviewer didn't know that. They should have asked the LRU cache. I just froze. I couldn't do it. I was thinking, why am I doing this to myself? Here I am doing this interview, asked to do a diameter of a binary tree. I couldn't turn my brain around. I was like, "I'm sorry. Let's not waste our time here. Cut the interview short." Then decided maybe I should stop looking for IC roles. Maybe I am rusty. Maybe I should just go back to management or something like that.

A Managerial Role at Shopify

I'm good with people. I started interviewing for some managerial roles. Then I found this managerial role at Shopify. At Shopify, managers code, which I think is amazing. They were like, we're going to do some pair programming. Now, of course, the expectations from a manager is not to be hardcore algorithmic solution builder. Still, you have to roll up your sleeves. It's an amazing feature of everyone at Shopify. I was prepared to write code, pair programming with another manager in the room. I get into the interview, and he asked me to do an LRU cache. I absolutely aced my second, third LRU cache of my entire career of interviewing on whiteboards. I get hired. I keep asking people at these companies, why do you do this? Why do you subject people to these coding whiteboard interviews? Why are you still doing it? The most common answer that I get is that we just want a consistent and a fair and objective process that we can truly evaluate.

We want some signal from this process. We want to know something about this candidate in a very uniform way. I'm like, yes, what signal? The candidate can code. I say, obviously, this is not true based on my experience, because none of my interviews were showing anyone that I knew how to code. I just got lucky. I think if you ask a deep code question from any of your candidates, the signal is that they're just good at solving deep code questions. Or maybe that they're lucky. I don't think it's valuable at all. Then, I sometimes turn to shaming the people who organize these processes at companies, because they themselves are developers, and they hate it. Like, why are you doing this to others? This whole whiteboard coding is from punch card days, where we had to write code on paper and then submit it to the computer.

Then, eventually, thankfully, the bank gave me a computer to code on. That was already some progress. Things have changed. The whiteboard was replaced by a computer. Now we have AI. AI is a much faster horse than the computer, or the punch card thing that we had before. The candidates, they are adapting very fast to this world of AI, of interviewing with AI. I relied on luck, but some people just cheat. I could totally have had another computer in front of me, behind my monitor in this remote interview, listening to everything that was said, and spitting out the diameter of a binary tree for me, and I would have probably aced it. Some people are making bold predictions about these LeetCode interviews and saying that by 2026, these interviews will go away. I think by 2026, the LeetCode interviews are not going to be gone. This tweet did not age well.

The whole industry will catch up, and it will catch up sooner. Some big companies are doing. Shopify has been doing coding with AI, or interviews where AI is encouraged for a very long time now. That's one of the best companies which is doing it really well. Maybe Meta will be next. For now, they're still asking for diameters of binary trees, but maybe they will change someday. This battle of getting rid of the coding interviews will not be easy, because a lifetime of these processes around them have been built on top of very flawed assumptions. I actually think this tweet is fake. I don't think that's real, but I like it. Don't believe everything that's on the internet. The choice for companies that are administering LeetCode style interviews is simple. Either you adapt the interview to the age of AI, or you just face extinction, and nobody will want to work there.

Redefining the Interview Loop, in the Age of AI

I talked to an executive recruiter recently at a company called People Project that I highly recommend. She said that all the companies are asking about how to make developers more productive. Productivity has become this main measure of what companies want to do. They want to increase productivity. The real differentiators in productivity is not the AI tools. It's really the human skills, things like empathy, things like ethics, collaboration, judgment. Those matter a lot more than the actual code that you can write, because the tools can write a lot of the code. We have to redefine the interview loop to be up-to-date to the age of AI. I think, very simply put, if you want to be an AI company, and I'm sure every single company you work for right now is actively wondering, how can we be an AI company? Start by hiring people that are capable of using AI, meaning, interview them with AI so you can see how they use it.

What better place to do this than a coding interview? Ask them to use AI. If you do it well, the interview process can yield what the candidate is really capable of. They have this incredible machine as a helper, and you want to see what they can do. Can they multiply their skills, the ones that are important, to produce something with the help of that machine? Let's reevaluate our system design interview. When we do system design, I showed you one example where you'd draw boxes on a whiteboard and tell, how would you design the system? In that 45-minute interview slot, you really don't have enough time to go from the design to the actual implementation of a system, because it's too long to write. Now, with AI, you can actually ask the candidate not to just design the system, but also build it by prompting AI to actually go and execute on it.

You can actually watch what the person is doing by building something hands-on. You can see what actually matters, how they break down problems, how they handle curveballs. Are they writing tests? Can they iterate when things are breaking? They have this partner that's really fast. Now you can do a system design question using AI, where the code is actually being written by the AI.

What problems do you ask today with this kind of tool available to you? Good problems are the same as they were before. They're the ones that are big enough to require clarifying questions. They have multiple possible solutions. You need some iteration, and it's not just dumb copy-paste. The goal is to reveal the skills of the individual, so have something fairly open-ended. It's just a normal interview, where you start a little vague, find a milestone, throw a complication, extend it, and so on. Some examples of the ones I like: autoscalers, rate limiters, data pipelines. It's pretty open-ended. For autoscaling, you can have problems like leader elections, stuck pods, if you are doing some Kubernetes operator type thing. The rate limiters can have questions like, how do you stop abuse in this distributed system in 50 regions? Ingesting events for a data pipeline. I really like AI Chat, because it's something people experience every day, like, build an AI Chat system yourself.

Just because AI is used, it doesn't matter. Don't change your entire process. The same interview features apply. You just want to see how the candidate is using these tools. If you're interviewing, see companies that use AI in their interviews. There's already a massive database of companies that don't do whiteboard coding. I put a link up there. Look at this list. The companies that don't do whiteboard coding, this should be the norm. We shouldn't need lists. It should be the default. I couldn't find a list of companies that encourage the use of AI. I made a new GitHub repo using Claude. I encourage you to come and submit your company to that list. If you are using AI in interviews, please come and add yourself to that list. Hopefully, it will grow.

The times, they're changing. The industry will catch up with this AI trend sooner than later. Then it will continue evolving, and we don't know what it will become. The roles that people have will evolve. Now I see designers vibe coding entire features as prototypes. They're no longer using just Figma. If you're stuck under some rock, and you're still not using AI, it's time to climb out of that rock. You'll say, "It's impossible. I work for a large company. I can't change the way my company is interviewing, because it's a huge undertaking. We have so much process built about it." Quote your CEO, and go to the recruiting people and say, the CEO said that every company is now an AI company. Are you not going to be an AI company? Are you not going to hire people that know how to use AI? In the interviews, you're literally working hard on preventing us from becoming an AI company. See what they say when that happens.

Questions and Answers

Participant 1: [inaudible 00:43:58].

Daniel Doubrovkine: Yes, they're design questions with code. Like the code itself, I don't care. I want to see code you write maybe from a past life. We can talk about code, reason about code. I think that's important. Especially for senior engineers, forget about code. It's, design the system, implement it. I think that's the big difference today. With AI, yes. Maybe you need to tweak here and there, but I'm mostly interested about how you think. Can you get unstuck when the AI produces something that doesn't work?

Participant 2: What's your thought on the take-home project for some people?

Daniel Doubrovkine: The take-home projects is only to have something we can all talk about that you prepared. Not everybody works on open-source software out there. Maybe somebody has absolutely no history. Maybe they're straight out of college. This grounds it in a conversation for a problem that you can solve at home with AI or without AI that you understand thoroughly. We can talk about, is this code performant? Is this code interesting? Are there other ways to solve it? Not from memory, you can think about it at home with the tools that you have available.

Participant 3: You mentioned that coding AI challenge, that means different types of questions that you ask. How do you look at the assessment for those tests? Like you may have many managers who are doing those types of interviews? How do you make sure that your evaluation is consistent?

Daniel Doubrovkine: I work with some people who are looking into the rubrics and the evals of that. I think it's actually the same thing as from before. There are some red flags. There are some green flags. For example, some new ones, if you prompted an AI to write code and then you go and read the code, that's probably an antipattern, because you want to keep an abstraction layer away and actually work with systems. For example, write a test and then make sure the test passes. Rather than write code and verify as a human that it works. These types of things are new. Everything else is like, do you ask questions before you start implementing anything? Do you ask questions from your interviewer? Do you ask questions from the AI? What kind of questions do you ask? I think these are changing a little bit. Then the features are not changing very much.

See more presentations with transcripts

The Future of Engineering: Mindsets That Matter When Code Isn’t Enough

Ben Greene — Mon, 27 Jul 2026 16:00:00 GMT

Summary

Ben Greene discusses how software engineers can adapt and thrive in an era of rapid AI code automation. Drawing on his startup experience, he explains key mindsets like starting simple, maintaining code comprehension, attacking hard problems first, and focusing on customer impact. He shares why human empathy, agency, and practical problem-solving remain irreplaceable when code is automated.

Bio

Ben Greene is Co-Founder and CTO of Tessi, a mission-driven startup that is using AI and automation to make home restoration after disasters faster, easier and safer. He is also the co-creator of FreeFormula.Exchange. He is a 4x founding CTO, including: Outcomes4Me, Windshield, and NextLot.

About the conference

Transcript

Ben Greene: I'm now building Tessi. This is what I'd like to talk about more. "The Future of Engineering: Mindsets that Matter When Code isn't Enough." Could have also named it, what startups can teach us about surviving the coding agent apocalypse. Coding agents are exploding. Usage is going up and up. Not just by software engineers, it's also product managers, graphic designers, marketers, salespeople. It's everybody. Everybody can build software now. That's great. Nothing to worry about. Let's face it. It may not be here quite yet, but automation is coming for us. It's already here for a lot of people. Companies are retasking their best employees to spend all their time training agents how to do their jobs, then laying off the humans. How do software engineers bring value when anyone can program computers? If you feel like you have to prove that you deserve to have your job while simultaneously trying to figure out what your job actually is now, welcome to being in a startup.

Startups are the perfect example of having to adapt simply because you cannot survive. In many ways, I think what is happening to software engineers in general now is actually very similar to what software engineers have encountered in startups for a while. Luckily, I do think the lessons we've learned from building startups can be applied to building software in the age of coding agents as well.

Startup Lesson 1: Always Start Simple

We're going to do startup lessons. Startup lesson one, always start simple. There is so much pressure to go straight to the grand vision, to build fast, to scale fast. This pressure has been a universal constant in startups, and it's only getting worse there too. Why? Because code is getting cheaper and cheaper to produce, and that actually increases its demand. Consider the history of automation of software engineering. Kathleen Booth in 1947 designs the first assembly language, taking programming from direct hardware manipulation to symbolic instruction. Grace Hopper in 1952 designs and builds the A-0 compiler, automatically translating mathematical instructions to executable machine code. A bunch of other awesome ways of making software engineering more automated and more efficient over and over again. Stuff we all celebrate. That automation and efficiency has only driven demand in computing infrastructure. This is a logarithmic curve. It would not work otherwise.

This phenomenon is called Jevons paradox. Efficiency makes a resource cheaper to use, which causes people to use so much more of it that total consumption rises instead of falls. Now that everyone thinks they can build software, the demand for more and more software, more and more quickly, only rises and becomes more immense. Even though code is cheaper than before, we have to be disciplined. Software engineering is rife with examples of failed attempts to build large systems wholesale. Fred Brooks covers this really well. If you're in a TL;DR phase of life, I'll summarize, "A system that works is invariably built from a smaller system that worked." Coding agents are ratcheting the pressure up to 11 to produce while also being the worst offending enablers imaginable. Do more. Add more. Give me more tokens. Claude Code's always, should I continue? This isn't limited even to software, actually.

A system is just a collection of connected pieces that form a whole. Yes, your tech is a system, but so is your product, a system designed to get outcomes for users or customers. So is your business, a system designed to produce more money than it costs to operate. In all these cases, you need to start simple. This idea is actually referred to as Gall's Law. John Gall wasn't a computer guy, he was a pediatrician. It goes like this. "A complex system that works is invariably found to evolve from a simple system that worked. A complex system designed from scratch never works and cannot be patched to make it work. You have to start over with a working simple system." Please, if your business, your product, your tech doesn't produce the results you want when it's small and simple, making it bigger and more complicated won't fix it. Go back, fix it, then add complexity.

Startup Lesson 2: Comprehension is Key

Lesson number two, comprehension is key. What do I mean by that? Perfection is not attainable. You need to accept that and keep it in mind at all times. Change will be necessary and it will be directed by what really happens when you release to the world. In startups, we like to say that if you're not embarrassed by your product, you shipped too late. That's because a lot of the polish you add is going to get thrown away when you realize your underlying assumptions were wrong to begin with. At InsightSquared, we call this shaving the yak. It's ugly and it smells. Giving it a beard trim won't make a difference. What you've built is going to need to change. In order to change it the right way, you need to understand it. The linchpin of change is truly comprehending what it is you've already built in the first place.

We've all been asked for a simple sounding change and had to weather the reactions when we said it would take weeks and not hours that somebody expected. Most of the time in my experience, that's because there's some intricately wired bomb in the code that we don't understand and we are too afraid to change. The only options are to either heroically replace the entire component or cowardly work around it. You usually do the latter. Coding agents make it incredibly easy to create entire codebases that we don't understand from the very first step. It's like hiring a team that copies code off the internet at the speed of sound and always tells you exactly what you want to hear. It terrifies me. Technical debt is necessary. It's not reasonable to build for your 10-year vision when you don't know if the idea is viable at all. You have to know what tradeoffs you're making.

Startup Lesson 3: Do Something New

Startup lesson number three, do something new. In startups, if you aren't 10 times better than what's out there, you won't get noticed. To be better, you need to do new things that will give you some advantage. Actually, it's really hard when you're good at something to not become blind to better solutions. This is the Einstellung effect coined by psychologist Abraham Luchins and his mathematician wife Edith Hirsch Luchins, a cute couple from Brooklyn. They showed that prior experience actually narrows perception, causing experts to default to familiar patterns even when they're suboptimal. If we get more derivative over time, coding agents are always completely derivative. Just like Kathleen Booth and Grace Hopper, we need to move our discipline forward. When are we adopting proof-carrying code? Who's going to get deterministic concurrency models into production? How will we bring formal validation to software engineering? Coding agents are not going to innovate and drive software engineering or society for that matter forward. That's up to us.

Startup Lesson 4: Attack the Hard Bits First

Lesson number four, attack the hard bits first. In startups, de-risking is super important. We're already placing a huge bet, so we want to eliminate as many other sources of risk as possible. When you're building new things, you really have no idea how long they're going to take or how effective they're going to be. The mindset I take in new projects and startups, both at the engineering level but also at product and business level, is POC before MVP. If we're going to commit to an approach, let's make sure we know it'll work. Not that it's guaranteed to be the best approach, but at least we know it'll work. When I start a new project, the first thing I do is I make a list of all the functional requirements that I think could possibly come up in the most important ways, and I just try and get them working at some very basic level.

Because the more of the unpredictable bits that I can get out of the way, the more predictable the whole rest of it becomes. The more I can have a roadmap and a timeline in all the things my business counterparts are craving all the time. Last month, I was building a proof-of-concept Android app, and this was to deliver a completely offline LLM experience with on-device model and connected on-device vector database for Retrieval-Augmented Generation, RAG. I believed it would be possible, but I wanted to be sure. I wanted to see if I could get it together quickly. I am not actually an Android developer. Of course, I was using a coding agent, in this case, Claude Code, to build it. Claude was absolutely adamant that we build a web-based Llama backed MVP instead. It kept saying nonsense about having a quick demo and conserving tokens, in this case.

No, that does nothing for me. I know we can make that. That's there. That's out there. After a while, I actually had to instruct it explicitly to stop suggesting alternative solutions just because they were easier. Coding agents want to avoid the hard bits, but I don't think that's where we need to be going. We still need to do the hard work ourselves, and that's still going to be the most valuable work as well.

Startup Lesson 5: Care About Impact

Startup lesson number five, care about impact. People need to be more customer focused. Great startup software engineers are the ones who don't see their jobs starting and ending with the code. They don't just solve software problems. They reach deeper to understand customer problems and business problems. They use their understanding of software and technology to figure out solutions that other people aren't thinking of. As software engineers, we are generally the strongest systems thinkers and critical thinkers on the team. Also, the most versed in what technology can do. The great ones learn how to successfully apply those skills way outside the realm of coding, and it can make all the difference in a startup. It's because in a startup, solving the first problem you see is never enough. Solving the surface problem isn't compelling enough to overcome all the barriers in the way of adopting change. To break through, you have to keep digging and digging into people's pain, into their goals, into their hopes and dreams.

If you can get that far, and if you can engineer something, then you've got a real opportunity on your hands. A great founder turned investor once asked me when I thought our customers would use our app, and I said, with their coffee. He replied, is that their breakfast coffee, or their after-lunch coffee? He explained that at his previous company, getting to know his customers intimately was the key to creating the experience that fit into their lives at the right moments. He cared about them enough to get to know them, and that caring led to the insights that ultimately resulted in a great exit for him and his company. We need mindsets that drive you to go meet your customer. Mindsets like speed to need, or patience, or waiting. Mindsets that ask you to step into their shoes, to feel what they feel, to empathize. AI agents can't care, but we can.

My friend Larry sent me this text. Larry is great. He is the kind of guy who laughs at everybody's jokes. We all love Larry. We do not possibly have enough Larrys in the world. He studied lighting design in college. Now he deploys large-scale digital displays like the massive display at Salesforce Tower, and many of the displays in Times Square. He has no experience creating software at all. Should I feel threatened by Larry? It's not like he's a real engineer or anything. He's not full-stack like me, is he? The reality is, none of us can really call ourselves full-stack anymore. Some might even question whether we're real engineers at all. When I'm vibing away with Claude Code, building an Android app, I barely understand at a component level. I completely don't understand at a code level. I also feel like a cross between a deity and a dumbass.

I'm not either of those things. I'm just a software engineer wondering what even is software engineering. Given that I almost double majored in English in college, I had to look it up. I'm really attracted to two particular words on this slide, practical application. To me, that's just creating solutions to real problems. I love solving problems. I have always felt like a problem solver. I think we are problem solvers. We do need to rethink a bit how we go about it. The reality is that for a long time, software engineers have created buffers to keep others at bay. Sliding pizza under the door is a celebrated part of Silicon Valley lore. Reflecting real stories from Microsoft, Netscape, id Software. I think we like the freedom from prying eyes and the time to cope with the recurring dread that we're not as smart as we hope we are and we're going to fail. I think imposter syndrome is something that is actually so baked into us that it comes in the way that we work, in the way that we build our tools, and really all of it.

Startup Lesson 6: Get Out of Your Box

How do I deal with this? Bonus lesson, get out of your box. We need to grow to be more than just what our traditional idea of a software engineer has been. I saw this post and I really loved it. It's worth a read. Go find it. I'll paraphrase. He lays out the skills that he thinks are becoming critical, like creativity and taste. That's AI augmentation. Learning agility. Systems thinking. Something we're good at. High agency. I would add one more, empathy. Go out into the world. Explore it. Care about it. Find problems that speak to you, that inspire you to make an impact, that you realize you want to build practical solutions for. Because there are a lot of problems out there. We're not going to run out any time soon. Software can help with so many of them. That means that in my view, we all have jobs to do. Engineering changes the world. We're responsible for the change we create. Let's make a good change.

Questions and Answers

Participant 1: How do you teach empathy?

Ben Greene: How do you teach empathy? I'm thinking about my children's classes, because I think they do this. I think they actually very clearly instruct children in empathy in school. I think you do it by practicing it, really. Really, empathy is sympathetic understanding. That you go out and you try to understand what someone else's life is like. What their problems are like. You don't just other them. You try to actually put yourself in those shoes and try and understand what it might feel like, and what the frustration might be. Because at our core, we are all emotional. We're driven by those. Then we put layers of logic and understanding knowledge on top of that. I think you practice it. I think you model it for your children. I think you model it for your employees. I think you model it for your colleagues. You can bake into a culture at a company.

Participant 2: One of my very good managers taught me to remember empathy is we judge ourselves by our intentions and others by their actions. Sometimes putting ourselves in their shoes and thinking that their actions may not be based on what their intentions are just helps me reinforce that I should be less judgmental. Just don't be a jerk to others is all.

Ben Greene: There's also the fundamental attribution error, which is, when you make a mistake, it's because you're a bad person. When I make a mistake, it's because my environment failed me. We have to break that down.

Participant 3: Do you have any concrete examples maybe in the last couple months where your team or anyone who you know is using some of these AI agents in a way that is missing that empathy component and has really derailed projects?

Ben Greene: The one answer I would give is, it's similar to if you ever had an experience working with a team that wasn't really part of your company, your business, is that they're not tied into the mission. They do what you ask them to do. That's not their fault. That you give them a spec, you give them a feature request, and they do it. It's just often the wrong thing. Oftentimes, what you actually need is, build me this system that allows me to test whether this is the right thing or whether it's something else. Because of the pressure to go fast, build, it's like the fastest path to where we think we want to go. Then we get there, it's like, wait a second, we did daily standups, two-week sprints, but we actually just did waterfall. We didn't want to be here. Now nobody bothered to really think about the system they were trying to build from a very basic level. They just built a whole bunch of code that does one thing after another. I have run into the exact same problem with Claude Code and Cursor on things as I have with teams that weren't really committed to the mission that I wasn't bringing into the team to the mindset.

Aaron: One of the things I find to set software developers at ease that are worried about their job is the fact that, who here hasn't seen a backlog of stuff that's a mile deep that needs to be done that hasn't been done that would be valuable. The way I like to frame it with engineers is saying, there's a lot of our job we used to do that was boring. Writing a GET REST endpoint to bring back data in yet another JSON format isn't particularly interesting and we can't allow AI agents to do that. There is an unlimited almost amount of work that can be done that way. I think just framing the problem the right way, like we get to work on the really interesting algorithm that we didn't get to work on or only got to work on two weeks of the year. Now that's our job.

How is that not an awesome outcome that we can actually deliver more value because we're not doing as much boilerplate stuff as we used to, or that we can even contemplate what you were doing, of being able to build an Android. I don't know how to build an Android app but I can sure get the basics pretty easy now, and then I can use the AI to help learn the parts that I don't know, and that's to me a virtuous cycle. You got to frame it the right way. I get from an empathy standpoint why people are worried because this happens to be happening right at a time that we're doing a lot of layoffs. I get why there's anxiety, but there's so much software to be written that isn't the boring stuff we've been writing.

Ben Greene: I completely agree with you. I think there is a lot of fear and so I leverage that a little bit in this structure. I don't think that our opportunities are going away at all. I do think that we get to spend more and more time if we do this right on the more interesting problems and the better opportunities. I think what's changed the most is that we have to stop gatekeeping around writing software. That we have to start really just recognizing that, yes, we're going to have a lot of colleagues who can write code, create code, and it's actually not double quote, it's actually the same shit we do. It's just that we know it better, and we can help them and they can help us, and we can go do things. I never have once said I really wish I had the opportunity to go back and do mechanical programming in a big computer. I'm glad I didn't have to do that in college. That's not for me. I think this is awesome. I just think we get to take advantage of the awesomeness when we step back and think a little bit broader about what it is we actually bring in and what things we need to keep in mind.

Participant 4: There are a lot of engineers out there that do just label themselves and see themselves as just programmers. Like maybe that's just what they did want to do. They want to take the work, get the ticket done. A question I have is, as a technical leader, how do you empower or change this culture to be one of builders of people who want to be solving problems using AI in these ways, thinking about these business problems like you're talking about, breaking out of the traditional programmer mindset?

Ben Greene: One way is to join a startup because no one's got time to figure out exactly what you should do. Startups are fantastic learning opportunities to really broaden your skill set. Not because someone's going to make it easy for you but just because there's just no other choice. From a technical engineering leadership perspective, I think a common pattern I've seen with some engineers and engineers particularly who are trying to get into engineering leadership is a reticence to delegate until they know exactly how something can be done. I think the reason that I've seen is that they feel like if they don't know how to do something and they hand it to somebody else they're setting that person up to fail, and so they want to figure it out completely before they hand it off. You're not growing the other person if that's the way you're doing things.

You have to accept that a certain number of efforts are going to fail and that's part of the engineering intervention process. There was a moment at InsightSquared, I was the lead of the platform team that looked over the whole application. We had a couple other teams that did pieces in the middle. It was an interesting architecture. We had a junior engineer who had taken on a major story to build something. Who knows what it was? He told me what his plan was for building it, and I was pretty sure it was wrong. Pretty sure it wasn't going to work. His name is Anthony. I said, "I don't know if this will work. Here's what I think, but you should decide and do it your way and then we'll see." It's a two-week sprint, we'll see what happens. He came back to me two days later and said, "I started working on my way and after two days I realized it's not going to work.

Your way makes more sense. I'm doing it your way." It's like, great. Awesome. I never for a moment took away his own motivation, his own search for his own mastery and growth, and we still wound up with a solution. The biggest cherry on top was I was actually right for once. It was a triple win as far as I was concerned. I think we have to give people the room to grow.

Paul: Actually continuing on that junior engineer, this talk is great and aimed at a more senior audience that thinks in a system way and a broader scope, but at least from my early experiences some of those simpler problems were a great on-ramp and maybe even some of the really low-level problems like learning assembly back in the day. It was actually really something that informed my understanding of how computers think. What do you think about the impact of these skipping all that stuff or going to this higher level on the pipeline of the new engineers.

Ben Greene: I think it goes back to how many of us are really that full stack. We had a class my sophomore year in college called, "Ten Great Theoretical Ideas of Computer Science." It was in many ways a great class, in many ways an awful class. It was great because it was really interesting, and it was awful because that year the professor decided that he only cared to determine who the single smartest student in class was. He made all the assignments and tests so hard just so that it would easily surface the smartest kid, and then that kid became his PhD candidate. That part was less great. One of the things that we did in that class which I still remember was we wrote out a 2-bit processor on paper using NAND gates. Even at the time it was like pieces of paper taped together. I was like, this is cool.

I don't understand anymore how that worked, but I know it's possible and that's cool. I think that there's a lot of value in understanding the very basics of how this stuff works but at the same time we are all building on the shoulders of many giants. We have so many levels of compilation and incredible work that happens at each of those that we're barely aware of. We may know at a theoretical level it exists but we don't know what it's doing. I think where a really strong academic background can really be a bonus is when you need to dig deeper to solve a problem that you can't at the surface level, then you have some of those tools to do it. I've built a bookshelf with my dad when I was a kid, but there are still master carpenters out there. They do a much better job at a bookshelf and a lot of other fancier stuff I have no idea how to build.

I think it's ok, but we shouldn't not appreciate the fact that there's people out there thinking about how software can solve problems that we're not either thinking about or just not in position to solve. That's really valuable too. It gets around that narrowing problem where your knowledge collapses your thought process of opportunities. People who don't know how programming works sometimes think of really interesting ways for programming to work. The Wright brothers were the same. They were like the least qualified people on earth to start building airplanes.

Participant 5: You mentioned during the talk protecting comprehensibility over time. I'm curious maybe what you meant by that, but like how to protect it over time. The first thing that comes to mind for me is documentation, and like, we're not documenting all the time. If we are, no one's reading it. What are the practices in a realm where you are making an Android app? How do you protect comprehensibility over time?

Ben Greene: I'll use an inductive proof. At no code, I completely understand it. If I want to understand it when there's lots of code, I need to understand it every time I add code, every iterative step. That's the code review process. I think it really comes down to every time you add change, whether it's code or whether it's infrastructure, that you make sure you understand it. That sounds really obvious and not worth staying on the stage to say it all, except that we're really insecure about how smart we are sometimes. It can be really hard for a software engineer to say, I don't understand this code. I have seen that many times and I have done that. One of the things that I figured out how to do at my previous startup, Outcomes4Me, was to instill the value that if the reader doesn't understand the code easily and that cognitive load isn't low, then it needs to change.

It just needs to be rewritten, variables, organization, what have you, until it's just easier to understand. After correctness, think always of the reader. It was more important than almost everything else about the code, was how easy it was to understand, because if someone doesn't understand when they're reviewing it, they won't understand it a month ago when they need to change it and probably the author won't either. I was able to model this successfully by pretending to be really dumb and not understand anyone's code. Then eventually I didn't even have to pretend anymore and it worked out culturally very well until we hit COVID. Then we all went remote. Then, all of a sudden, I couldn't model this behavior in public anymore. It started to regress. I actually had to create a separate engineering Slack channel just for asking questions, and force everyone to ask all their questions in public so that people could see that it was ok to ask questions and not know things in public. Then we started to get back to that culture.

Participant 6: The key aspect you talked about is problem solving skills. How do we teach that? I think if you go to engineering schools, go study computer science, programming languages and everything, historically that has been taught, the key skill to learn problem solving has always been there. It had been there in the past. It is still there today. What changes in terms of what students needs to learn, how that learning process needs to evolve?

Ben Greene: I think like anything else, whether it's engineering, problem solving, dribbling a basketball, you have to have someone set you up with a challenge of appropriate difficulty for your level and then practice it until you can do it well and easily, and then give yourself the next level. I think there's progression. What you're working towards is that state of flow, where you find that task where for you the difficulty is high but your ability is high and it feels great. Not everyone is going to be great at everything. I have learned having children, they're different. It's fine. I'm not trained to teach people. I feel like I've gotten better at it, both as a parent and as a manager. There's a lot of similarities, oddly enough.

Participant 7: Engineering leaders are more and more expected, especially now with automation and AI, to deliver the products that we built prior faster now. How do we balance between delivering a little bit faster but not hindering the product's requirements and the solution as we go forward?

Ben Greene: By delivering the prior part faster, you mean like reinventing the wheel but faster than it was invented the first time?

Participant 7: Just with maybe different data or different components or different features.

Ben Greene: I am so ready to move on from traditional software as a service. I don't know about anybody else how much they felt like they spent building the same things over and over, but we got to get out of that. It's such a waste. There are so many better things we can be doing. As we know what we're doing, some of this is defining the behavior better, looking at patterns. We use tools. We love tools. We need to find platforms. We need to find reasonable tools. Open source is a great way for getting there faster without sacrificing a focus on the end result. In no situation have I ever found that I used an open-source library only to determine that they spent less time thinking about solving that problem than I did. I don't think it's a hard sell to leverage open-source software, and there's a lot of it.

There's more and more. It's getting better and better, in a lot of cases. Still a couple cases worse. I think there are ways. I also think that when we can now engage our non-engineer counterparts in the creation process, they can start to feel the difficulty of building something that actually works in all cases. We're so good at seeing the corner cases. A lot of folks that we work with who are not builders don't see them because they don't think that way, and that's ok. When they get to sit in the shoes of building the software, and they show you what they built in Lovable or Replit, you can ask them the corner questions. You can be like, did you think about this? What if the person can't do that? You like this design? What if they're red, green colorblind, 10% of men? Then you get to flip the table a little bit.

Participant 8: I'm more curious about during your startup journey, do you have any example when you pivoted your solution on a problem you want to solve? What are the metrics or factors you considered at that time?

Ben Greene: Pivoting and metrics. In most cases, you pivot because you haven't found product-market fit. In most cases that you haven't found product-market fit, you haven't gotten to scale. When you haven't gotten to any sort of scale, your metrics don't do much in the way of quantifiable good. Really, you have to go back to much more subjective measures, which is not to say that you can't look at quantifiable measures, and say that this funnel is not working at all. It's very easy to see that this funnel is not working at all, that no one is going through your flow and signing up for your product and using it. It just doesn't really give you any insight on what to change because it's not like you have enough people to split test and do two different variations. Generally, when you're doing that kind of optimization early, they both are bad.

By far, the most valuable experience has been taking the idea that we think we have, bringing it to a real person who has that real problem, and watching how spectacularly they fail trying to accomplish the goal that we've given them or the value that they want to get out. Outcomes4Me, just a little context, is a product for cancer patients to understand their disease, to understand the details of the diagnosis, what the clinical guidelines say that their treatment paths are, what their treatment options are. Then also give them tools to find clinical trials, manage their symptoms, track medications, and a bunch of other things. Our first version of the treatment path was completely wrong. It didn't present the information in a way that people could understand. Systematically, from our standpoint, it made perfect sense. From their standpoint, it was not consumable in a way that they were framing it in their minds, and that they needed to interrogate the information and answer questions.

Only once we built that first version and put it in front of someone did we realize just how awful it really was. I can't imagine getting the feedback that we actually needed in numbers. Later on the line, we could tell that our marketing approach wasn't working, and we needed better numbers, and that could help. Early days, pivots aren't driven by metrics, really.

Participant 9: One of my own personal observations trying to use coding agents, and I'll tie it to how I've seen it play out in our organization as well, is, I don't seem to struggle with the code. If I do something that produces a volume of output, I go back and look at it two, three months later. This is what I wrote. I don't even remember that code anymore. With the organizations now saying, we would like to see 90% of you use agentic AI to do your coding and whatnot, I see this playing out with engineers as well. They produce solutions that they don't understand themselves a couple of months later. What are your thoughts on how we can make the situation better? Because I feel like maybe we will lose some of the edge.

Ben Greene: This is the lemming slide. This is like all of us happily jumping off a cliff that we can never get back on. This one scares me. My really pessimistic view is that some companies actually understand the hole they're digging from a comprehension standpoint, and think that they'll ultimately be better off when they have AI that understands all the code and they don't need the people. I don't really think that's going to happen because those companies will not be able to innovate anything, and they'll just be slinging slop. I don't know how to do this. I think what has to happen is the idea that engineering has to happen at a pace where the engineering team can continue to understand how things work, has to pervade the whole organization. I actually was thinking that this is something that investors need to understand too. You know that the companies you invest in are going to have to make changes.

You need to understand what happens when you drastically increase the rate of accumulating technical debt, because that's really what this is. We've seen the Twitter, how it was going, I just vibe coded this app. How it's going now, everyone's attacking it. I got to take it down. We've seen those memes. I think people realize, this isn't ready to be deploying production apps. We've seen a bunch of problems that have become public. We'll probably see some companies who go too far down this path and then explode or implode or fall apart, however you want to do it explosively. Then we just need to make really terrible examples of them and scare other people to not make those mistakes again, because at the end, those who do not know history are doomed to repeat it. People are going to keep trying that, unless they're reminded how bad of an idea it is.

See more presentations with transcripts

Clean Architecture for Serverless: Business Logic You Can Take Anywhere

Elena van Engelen — Sun, 26 Jul 2026 16:00:00 GMT

Summary

Elena van Engelen discusses how to eliminate serverless vendor lock-in without sacrificing native cloud capabilities. She explains how to structure FaaS applications using Clean Architecture, Spring Cloud Function, and Gradle modules to isolate business logic. Finally, she shares a live demo deploying portable Kotlin services across AWS and Azure using Terraform CDK for multi-cloud IaC.

Bio

Elena van Engelen is an expert software engineer with over two decades of experience who is driven by passion for technology. She specializes in Kotlin and cloud-native solutions and focuses on building mission critical, scalable and maintainable systems. She also enjoys sharing her knowledge through talks, her blog, giving Kotlin training and her book, “Kotlin Crash Course".

About the conference

Transcript

Elena van Engelen: Is serverless secretly locking us in? A lot of developers and thought leaders believe that's so. This is a really good question nowadays with intercontinental tensions and everything, so it's really relevant right now. What if you could use architecture to help you keep your business logic cloud agnostic? That's what we're going to look at today. We're going to look at some building blocks that can help us keep our business logic cloud agnostic while we're still using the serverless building blocks on different cloud providers. We're going to look at the Spring Cloud Function. Has anybody used Spring here? Clean architecture. I guess a lot of people heard of that? We're going to use some Gradle modules. This is maybe more specific. We're going to use these building blocks to lead ourselves to the key takeaway, it's a framework for cloud-agnostic business logic.

I'm Elena van Engelen. I'm a Senior Software Engineer specializing in Kotlin and cloud-native solutions. I currently work as a lead engineer at AZL. That's the life and pensions part of the NN Group. I'm also an AWS Community Builder in serverless category, an author on "Kotlin Crash Course" book, and a blogger on Kotlin and cloud topics.

Serverless and Function as a Service (FaaS)

Let's crack on with serverless. What is serverless? Have you used serverless? Who has used serverless? It's not like we don't have any servers. We do have servers. We just don't have to manage the infrastructure. We don't have to scan our containers for vulnerabilities. We don't have to patch our operating system. We don't have to worry about the hardware. It makes our lives a bit easier. It's resources on as-used basis. We scale automatically and we're triggered by events. Let's just compare a little bit with container-based apps. In a container-based app, you might have a microservice or you might have a monolith or whatever. You have a lot of business logic in one thing. If you start scaling, so you need to scale because you have a lot of requests coming. You scale everything. You're going to scale everything. When you have Functions as a Service, they're very simple.

A function has an input and an output, so it's got a lot less responsibilities. You should have one responsibility, really. Your microservice or your monolith or your product will be built with multiple functions. When your events come in, only some or one of your functions need to scale. Not all of them. If you have some background jobs, or you're handling events like messaging or you're handling HTTP events, if one of these get a lot of requests, only they have to scale. You're a lot more resource efficient using serverless. Let's look at some use cases you might use serverless in. These are really common use cases. When I say REST APIs, you're probably thinking cold start. The really big cloud providers like AWS and Azure already mitigated cold start. You can mitigate it and you can still use REST APIs. You just need to make sure you take the cold start into account.

For example, AWS has SnapStart. If you use that with priming on a JVM, you can actually eliminate the cold start and not notice it even on p99. On Azure, you can use Elastic Premium. It's not free, but it will mitigate your cold start. IoT event processing, this is a really good use case. I actually worked at a company called PostNL. It's the biggest parcel delivery company in the Netherlands. They use serverless AWS for everything there. They had like 800 million events per day and they were handling it with serverless. It works really well with IoT. Data transformation, clickstreams, scheduled tasks. If you have a scheduled task, you don't want to have container, keep waiting all day to run it once or twice a day. These are also good use cases if you don't have very long-running scheduled tasks. Basically, serverless can be covering all of those use cases.

Spring Cloud Function

We said that Function as a Service makes our life easier. We also asked, what about portability? Let's look at how Spring Cloud Function, clean architecture, and Gradle modules can help us keep our business logic cloud agnostic. Spring Cloud Function in a nutshell. As I saw only half of you use Spring, so, basically, Spring is like a framework that allows you to use in version of control. It's a very popular framework for JVM languages and also can be used in .NET. I think other languages have got similar frameworks for dependency injection. Spring Cloud Function basically allows us to run a Spring app in Function as a Service. That's what it allows you to do. Therefore, you can use your Spring features in your Function as a Service. Your favorite Spring features, dependency injection and auto-configuration, anything that you like. It's got a really big ecosystem. It's adaptable to the environment.

You can run locally. You can run on AWS Lambda, Azure Functions. They have more adapters for other clouds. You could even write your own adapter if your cloud's not supported. Basically, to run your Spring Boot app on a Function as a Service, you just need the Spring Cloud adapter. Let's just jump into the code immediately to see how that's going to look. Because Spring sells it as like, we are platform independent. Actually, if you start writing the code and you're trying to deploy it to cloud providers, you will see that in many cases, you still have to write cloud-specific code. First, we're going to use the cloud adapter. This is actually cloud-specific dependency, which we wouldn't want to deploy on different clouds, which we only want to deploy this in Azure. This is in Azure. We're going to write a function for Azure, include the Spring Cloud adapter.

Then we need to create a function entry point. This is like a Hello World example. This will be an HTTP trigger. It has a FunctionName. Then we define httpTrigger, which will take a POST on an endpoint docs flow. We will add some basic security. This is a demo, so we won't worry about too much security. Let's just add the basic security. That's an API key, which we will be using for security. This is how the function looks like for Azure.

In order to run this Hello World example, we actually need to deploy it, of course. We're going to use CDK. I'm using Terraform CDK because I'm deploying to multiple clouds. Who has used Terraform? It's Infrastructure as Code. A lot less people have used CDK, that's basically using a programming language to generate your Terraform files, and then you can deploy. Then you use your own language to do that. For Azure Functions, all we need to do is specify a function app where our function will be running. There we have to specify the MAIN_CLASS. This is the MAIN_CLASS of our Spring app. That's the only thing we need to do, and it will work. The Hello World is already running in the cloud. I'll show you, because we don't want to be doing Hello World right now. Anyone can do Hello World. AWS, how does that look?

First, we need to include our dependencies. This is AWS-specific Spring Cloud Function adapter. Again, AWS-specific. We need to include that in our application. Then we have our code to execute when the trigger fires, the HTTP trigger. Where is our HTTP trigger? It's not in the code. It's different. For AWS, we implement our trigger, and that will be using the AWS-specific request and response from the SDK. The trigger is in the Infrastructure as Code. That's different from Azure. Let's just quickly look at the name of the function, uploadDocument. We will need it in a minute. When we go to the Infrastructure as Code for AWS, first, we need to define our Lambda. This is where our code will be running. In order to configure our Spring Cloud Function, we need to specify the name of the function, Spring_Cloud_Function_Definition. That was the upload document name of our function that we just saw.

Then with MAIN_CLASS, again, we need to specify the MAIN_CLASS of our Spring app. We also need to specify the handler. That's what AWS Lambda needs to know what to call when it gets an event. That's actually a class within the cloud adapter, which you need to specify. Once that's done, you need to specify the trigger. We don't have the trigger yet. The trigger is our DocsFlow endpoint. That's going to be our API gateway specification. DocsFlow, we want to go to DocsFlow endpoint. We want to accept both requests. We want to have an API key as well for some basic security. This will be the same as Azure. You can see that the definition is different to actually define your trigger and the entry point. Those things should be kept separate from your business logic.

As I said, we're not going to do a Hello World. That's just boring to do that. What I'm going to do is going to show quickly. We've got two Hello Worlds running. This is the AWS one. I'm showing that I'm pointing it to AWS DocsFlow endpoint. This is running a Hello World, not doing anything, because later on, I'm going to plug in the business logic into those endpoints. Instead of having Hello World, let's go to Azure. Again, the same. Nothing's going on there. This is Azure Function. We're just doing a POST request of nothing. We're not actually posting any information in there, because it's not doing anything. We haven't plugged anything in. That's basically Hello World.

What are we going to build? I'm going to go back to my experiences writing a book. Every time I wrote a chapter, once I finished it, it had to be reviewed. I had to send it via email and have a special name for the chapter so they knew which stage of the reviewing it was on. Then I would send that email with the chapter to the reviewer. First, it would be the editorial review. Then they will send me comments, fix that. Send for technical review. They will do the technical review. Then, again, the chapter name will change because they want to know in which stage that's in. Then it will go to the final editorial review. If later on, you think, actually, I could add this to the chapter that has already been final. Then you go into this exception scenario where it has to be reviewed again.

It's quite really manual. Let's just modernize it a little bit and add a bit of business logic so we can do it automatically. What we're going to do is we're going to use Azure and AWS. We're going to deploy our business logic to both clouds. They also will upload the document. I'm not doing the graphical user interface, so I'll just use Postman to upload. We'll upload a document to the API endpoint. In Azure, it will be Azure Functions. This will validate with generic business logic for this publisher that my chapter meets the basic requirements. Then if my chapter meets the requirements, it could be like image quality and number of words or other. It will save it to storage. In Azure, it will be Blob storage. In AWS, it will be S3. Once my document has passed the validation and been saved, I actually want to do an automatic review.

Let's say it could be like some AI model specific to that publisher that will do the review, some other business logic. These kinds of things will be cloud agnostic. Those need to be deployed on Function as a Service, which will react to the document being placed into the storage. That would actually generate an email with a secure link and go to the human reviewer that can do further reviewing. This is what we're going to do. It feels like a real application. You can see I'm just using the real cloud services. I'm not avoiding using any cloud services, even though I want to be portable. I'm using Azure Functions. I'm using Blob storage. I'm using Azure Communication Services for the email. In case of AWS, I'm using API Gateway, Lambda, S3, and SES, Simple Email Service.

Clean Architecture for Serverless

How are we going to do that? We're going to use a clean architecture for serverless. It's actually a simplified version of clean architecture. We don't need so many layers. That's because this is not your typical container-based app with a lot of responsibilities. It's like a function that's got one responsibility. You don't need too complex a layering. You could use a simplified layering. You have a domain layer. That will have your domain logic, your domain objects, basic domain object validation, for example. Then you have your application layer, which will have your use cases, business logic, your interfaces. Then you will have the infrastructure. That's where you put your cloud-specific code. I believe that kind of boilerplate code eventually would probably be generated by AI. Things like saving to Blob store, saving to S3 code. That code connecting you and your business logic to the cloud, keeping that separate, and just putting that in a separate layer.

Let's just have a look at some real-world examples. We're looking at the pension sector where I'm working. It's not boring. It sounds boring, pension. It's not. You have a participant in a pension fund. You have participant, pension fund. That's two domain objects. You have life events. Why do we need life events? A lot of countries actually have rules on life events, which will affect your pension. Let's look at the Netherlands. If you get married, we need to know about that. That event has to come in, and we have to register it. That's because if you eventually get divorced, your partner might walk away with some of your pension, and they will become a new participant. We will have to process that automatically. We need to know about that. Another event, when you die, we need to know about that because there are some beneficiaries of your pension.

That's basically your kids and your partner. They have a claim on your pension. They will become participants. Also, your ex from 20 years ago, they can claim the pension for the time that they were together with you 20 years ago. It's really complex. These business rules are complex. If I knew about that, I might have thought twice about getting married in the Netherlands. You don't want to keep these business rules attached to your cloud. This is really complex. You want to keep it separate so that you're more flexible and you give your application a longer lifetime. If we look at the application we're going to build now, it's a lot simpler, of course. In a domain layer, we will just put the document metadata, the document that we're uploading. We want to know something about that. In the application, we have two services. One will be validating the document and the other will be reviewing the document, and actually also sending email. It's two responsibilities. Yes, so it will be reviewing the document and sending that as an email to the reviewer.

Clean Architecture - With Gradle Modules

This is how it's going to look in the Gradle modules. You have this domain and application modules. They don't have any cloud knowledge. They don't have any dependencies on cloud. They know nothing about the cloud. With Gradle, we can enforce that and make sure the code won't compile if you try to access any infrastructure code or any cloud-specific code. Then you have a module with each cloud. You have AWS or you have Azure. When we package this to deploy to the cloud, we only package the infrastructure code that we need. There won't be any Azure code in our AWS package or AWS code in our Azure package. Then I also have a CDK. CDK is not actually being deployed because this is just description. This is just a configuration of your infrastructure. This is going to be used in the pipeline to actually deploy your infrastructure.

It's not packaged in your deployment package, but it's just used for specification of your infrastructure. Again, with the Azure. This is how this looks in Gradle because not everybody used it. For example, in the settings.gradle, you can define your modules. Then, in a specific module, you can say what is your dependency. For example, in the application module, I have a dependency on domain. Application knows about domain, but it doesn't know anything about infrastructure. It's like an inward dependency. Domain knows about nothing. Application knows about domain. Infrastructure knows about both application and domain. It won't compile if we do the other way around. This is what we're going to do, put the same business logic. Then we will seamlessly connect our business logic to the cloud using the dependency injection at one time to actually call the actual specific cloud components.

Demo

Let me just go to the code. We're going to do some coding. Really quickly to show you here, we have the application layer. That's the business logic. This is domain logic on my right side. This is the infrastructure. Basically, what I want to do now, I want to connect my business logic into the infrastructure. Infrastructure has the entry points for my function. Before I go in, I'll just go back quickly to the solution design. I can just remind you what we're doing just a little bit. What we're going to do now, we're going to implement the business logic injected into the function that will receive the document and the function that will review the document. The same with AWS Lambda. The same business logic will go for receiving documents and validating and actually reviewing the document. Let's start with Azure. First, I'm going to open the class and then I will close that window so you can actually read the code.

I think it's readable. This is a container for my two functions. One function is UploadDocument, we're going to start with that. When you upload the document, we want to validate it, and save it to the storage. Then after it's saved to storage, we want to review it. What I'm going to do is inject those services. Thank you, AI. Once I've injected the services, I actually want to use them, and replace my Hello World logic with the actual calls. This will actually receive the document and validate it. If it's valid, save it to the store. Then I want to go to the part where I'm actually listening to Blob storage. If there's a document placed, I want to review it. I will replace this Hello World logic to actually call the business logic from application layer. This will review the document and actually send an email.

It's actually doing two things. Reviewing the document and sending email. I've done the Azure Function. Let's do AWS. We'll do exactly the same. Just inject the entry point of our AWS Lambda with the business logic. Very similar. Just let's inject. That's not what I want. AI is not always doing what I want. Once I've injected the business logic, I can actually call that from my entry points. UploadDocument will be called. For the HTTP request, I will be submitting the document. We'll replace the Hello World, and the second function, processDocument. When I have an S3 event, when the document is saved, I want to replace my Hello World with a reviewAndNotifyDocument. We've injected our business logic into the cloud entry points. Now what we want to do is inject our business logic with interfaces, which will connect it to the cloud code without it knowing about it or actually having any dependencies in those modules. That's the last part, what we're going to do. Then we're going to deploy.

We're going back to our application. This is the business logic that we were just injecting. Here we have a couple of interfaces. That's not really a new concept. We have a couple of interfaces. One will allow us to send an email and it will just connect us to the actual implementation. I can see here, I have the two implementations in each infrastructure module. AWS will have the SES email sender and Azure will have ACS email sender. The business logic doesn't know that, of course. Exactly the same with object storage. One is talking to S3, the other to the Blob store. What we're going to do is we're going to go into our business logic. First, the review. Let's just go so that you can see the code. Here we have this DocsFlow request handler. Once I submit the document, it will validate it and save to the storage.

Let's just inject that with object storage. Here we have some validation logic, which is portable. It's just returning true. I'm not actually doing any business logic here because it's just for the demo. It will validate the document, which will return true. It will be always validating, and then it will save. What we want to do here is actually save to the object storage. We're saving to the object storage. We're saving the document. Let's just get rid of the TODO line. That's it for the first service. It will just validate and save. Then, let's go to the second business logic service. Let me open it. This one will do two things. What it will do is it will basically review the document. That could be AI or whatever, your business logic. It then will generate a secure URL to your document, and it will send an email to the actual reviewer.

I'll get rid of that name. We've got the two services that we're actually injecting our business logic. For the secure link, we can call our generic object storage. We don't know which cloud we're calling, so object storage. We just want to generate a secure URI for our blobId. This will have the URI for the reviewer, so they can actually click on the document. This is basically my complex business logic for review. It's actually just a random string at the moment. I'm not plugging in any AI at the moment. You could have all sorts of business logic in there. Once the review has taken place, we can send an email using the notification API. Send email with a review, and review is containing the secure URL. Basically, very simple example. We've done all of that, but before we deploy, let's just build it because what can possibly go wrong when you deploy into two clouds?

Let's just go into the console. I'm just going to build it. Just make sure that I'm green before I do any commit. What I'm going to do now, I'm going to commit to my Git repository, and there I have two pipelines. One will deploy to Azure and one will deploy to AWS. We're ok. This will be committing. Let's just get rid of that. We actually changed four classes. The two function classes specific to AWS and Azure to plug in the business logic, and then we changed the business logic to actually use the interfaces to the cloud-specific things. Now that we're pushed, what should happen is that we should get some pipelines going. Let's see how that's working. We have two pipelines, one's going to AWS, one's going to Azure.

Deployment - Terraform CDK

While that's going, we'll go back and just look a little bit at how we are deploying. Terraform CDK. Not everybody is a big fan of Terraform, I know. There are advantages of Terraform and one of them is that you can actually do deployments to multiple clouds and you actually speak the same lingo. It is different dialect because when you deploy to AWS, you're using AWS resources, and when you deploy into Azure, you're using Azure resources. It's not like it's exactly the same code, but it's still the same concepts. Because you're using CDK, you're actually using your own programming language, which really helps bring ops into dev, so DevOps. You have developers doing it, it's much easier than learning this tf Terraform files. Multiple cloud compatibility, that's coming from the Terraform. Reusability, you can reuse your Terraform module. If I wrote Terraform module in a tf file, you can reuse it in your CDK, or I wrote it in .NET, you can still reuse it because it generates basically Terraform files.

Predictable changes, so if I change my infrastructure, I can see what's going to change in the cloud. How does that look? If I created new Infrastructure as Code, what I need to do is I need to do a cdktf get, it will get all the dependencies. When I do a synth, it will actually generate Terraform files. This is very slow, so only do that when you change your Infrastructure as Code. In your pipeline, only do that when you change Infrastructure as Code, otherwise don't run that because you only need to deploy, like if you are updating your software only and not the infrastructure. You can do just a Terraform plan and apply and it will just upload the new software to your functions. This is a lot faster.

Demo

Let's have a look at our pipeline. Normally one of them completes by now. My AWS has completed, so what we can do is we can start by looking at the AWS one, so let's just do that. Actually, before I go in there, let's just have a look in my S3, I don't have anything, so let's just see. In my S3 I have nothing, so it's nothing saved yet, I haven't submitted any documents. Let's just go into Azure as well, so Air Blob storage also has nothing there. Now what I'm going to do is I'm going to go to my Postman. I think if I do AI in the next year, I will create a frontend with AI. I'm not good at frontends, but AI is different. Here we had this Hello World, so now we're going to go to AWS because it was first. I've clicked on my AWS, this is pointing at my AWS.

It was Hello World as we saw previously, so what we're going to do is we're going to upload a file to review. I'm going to upload chapter 13 that's actually talking about the event-driven serverless on AWS. We're going to review that, so let's just submit this document and see what happens. What should happen, it should review the document and then send me an email with the review and with a secure link. If I click on the link, I should be able to download the chapter and see any review comments. I can see that it's completed, and so there it's got actually some login information that has saved something. That's good. Let's first look at our browser, because then we should be able to see in our S3 that the document has landed. That's it, that's the document. We should be able to see an email.

Please ignore all my other emails. This is the email. It's giving me a compliment. That's good. It says that it flows very well. Then I will click on the document and it should download the chapter 13. If I open that, I will see the chapter 13, so that works. Now what we're going to do is we're going to look at Azure because it's doing exactly the same. Let's see if it's actually deployed because Azure takes a little bit longer with deployment. Let's wait for Azure a little bit. I have to wait for Azure to deploy my app before I show the Azure part.

What I can also do is show you the pipeline. Let's go to IntelliJ. I'm using GitHub Actions to deploy. Nothing complex. If I look at the deployment of my AWS, so all I do is basically, I build the package, and this package will be uploaded to the S3 bucket. Then, because I want to deploy it, I will actually deploy it. Here I will be putting it to a S3 bucket, and then here I'm doing a deploy with Terraform. It's very similar to Azure. In Azure, I will just be uploading to a Blob storage, and then I will be using terraform apply to deploy any changes, what we're just doing right now. Really quickly to show where the CDK is. This is the CDK with the Infrastructure as Code. For example, AWS, it's very similar. I don't know who's used AWS CDK, the native one. It's similar to that, but it's more verbose.

You specify, for example, your permissions when you need to connect. IAM permissions if you want to connect from your AWS Lambda to S3, for example, those kinds of permissions you have to specify here, and then API gateway. Very similar, but more verbose. If I'm only on one cloud, I prefer to use the native one, so the actual AWS CDK. If you want to deploy to multiple clouds, this is quite handy to just have the same language to this. For example, this Lambda permission for the API gateway, it's how it looks like. It's like a builder with actual programming language code.

Let's go back. Azure has finished. Now we can actually do the demo of Azure. Let's go to Postman and go to my Azure Postman collection. It was Hello World. We're pointing at Azure now. What we're going to do is we're going to upload the document. Let's just pick a different chapter. It's going to be chapter 8 about functional programming. Let's have it review the document. It's using the same business logic. Because we are actually doing a random string, it usually picks a different review. It's finished now. If I scroll a little bit like this, you can see that it's saving to Blob store. What we can do is we can go into our Blob store, which was empty. If we refresh, we can see our document there. We should see the email arriving as well. We had our email, about 80% was an email, and this is the Azure. It's got a different review. It says that even AI stopped reading halfway through and needed a coffee break. It didn't like it. Let's improve our document. We click on the document, we can download it. It is chapter 8, so it works.

Key Takeaways

We did the demo. We can go to the final part. Let me just go to the browser. The key takeaways are, if you separate your business logic, you can achieve meaningful portability. You can port your business logic to different clouds. It is really so simple. All of those concepts are not new, but this is actually applied to actually being portable. When companies say we don't want to use serverless because it's then tying us up to the cloud, that's not necessarily true. If I'm in a container and I'm calling S3, I'm still connected to the cloud. I still should be separate. It's very similar problems with the container. Spring Cloud Function will allow us to do dependency injection at runtime. You have other frameworks that do dependency injection. You also have frameworks that do it at compile time. In your other language that you want to use, you probably have a framework like that.

Gradle modules allows us to enforce the separation between the layers. This really helps us to prevent developers from actually abusing the architecture and not following the guidelines. Just because you have time pressure, you have to do it quickly, you have to develop quickly. If you don't really enforce it, then I'm pretty sure it's going to go wrong over time. Kotlin, you don't have to use it, but it does help, because basically when I run in the cloud, I'm not dependent on the JVM because Kotlin can target JVM from 8. That means that when AWS went to JVM or Java 21, Azure was behind. If you were already on 21, you would be having trouble migrating. With Kotlin, I just can use the latest version. I don't care what JVM it's on. That's helping. Terraform CDK also helps you because you're speaking the same language, even though it's different words. Still, it helps for portability.

Resources

I have a Medium blog that talks about the same thing. I've got also another example that does it. If you want to see the GitHub of different examples and you want to see the Medium blog, it's an NN Tech blog, please scan the QR Code. This is my website with some links to my social media, elenavanengelenmaslova.github.io.

Questions and Answers

Participant 1: You have chosen a technical use case and not a domain one, uploading to an object storage. Shouldn't uploading to an object storage be also agnostic?

Elena van Engelen: Basically, it's just upload to storage. It doesn't have to say object storage. It's upload to storage. You don't know which storage, could be like a database. It doesn't have to be S3.

Participant 1: Yes, but then shouldn't the abstraction be on the upload. Then within your business code, you don't know whether it's an Azure Blob storage or an Amazon S3. Because basically S3 is also an API. I think it may work also with Azure Blob storage.

Elena van Engelen: You mean the naming of the function?

Participant 1: No, when do we do abstractions? Technical abstractions from domain logic.

Elena van Engelen: You could just say save. I think I had save there. Maybe the name of the interface might have been different. Basically, it's save. You get an object, like you get a document, and you want to store it somewhere. You could just say store, storage. Or you can say, save, like persistence. You can call it persistence.

Participant 1: Exactly, without knowing on which cloud.

Elena van Engelen: Without knowing which cloud or which database. I could be saving it to a Cosmos database. I could be saving it to somewhere else. Those are quite cheap, the Blob storage and S3. I could just save it somewhere else. It doesn't have to be those. It's more about like saying store it somewhere or save it somewhere. It's like, in the business logic, once the document is good, you've got the checkbox, then you want to store it somewhere. That's basically the name. You can actually say save or persist.

Participant 2: There is an anxiety of creating serverless applications. At large companies, we're putting a lot of focus on observability. What you showed us, it sounds really tempting, because we can deploy the same business logic to multiple clouds. Have you experience with observability? We haven't seen a lot of log messages in the code. Have you experienced putting the same observability on multiple clouds, on the same business logic, obviously.

Elena van Engelen: Yes. When you go to the non-functionals, then, of course, you have some difference. In the code, you don't have to have that. Usually, like with Spring, for example, you just say log info, and then you connect it to a specific logger. In the infrastructure setting, then you will say, ok, this logger will go to CloudWatch, whatever. That would be on your infrastructure layer. You don't have to change anything in your logging API when you log something. You do have to change something in your Infrastructure as Code if you want to direct it to some logging. Because you could be logging to a generic something that's not cloud-specific, or you could be logging to a cloud-specific, like CloudWatch. That's basically sending the messages to one or the other. It's the same. You will put it in the infrastructure layer. The actual loglines in your code won't change.

You just need to make sure that you use the API. Don't use the SDK of the cloud in your code. When you do logging, always use the generic logger and not the SDK-specific one. Just underwater, you should connect the SDK-specific one or where you want to log. I think it's similar to even if you're not moving to another cloud.

See more presentations with transcripts