CNCF - Blog

Deploying, maintaining, and migrating production n8n on EKS

Thu, 15 Oct 2026 16:00:00 GMT

n8n is easy to try and harder to run under production traffic. A single container with SQLite works great for a quick demo. Production is a different story entirely. You need workers, webhook processors, PostgreSQL, Redis, object storage, external secrets, ingress routing, and an upgrade plan that preserves workflow history.

Engineering teams rely on n8n to connect internal systems, process webhooks, and automate critical business data pipelines. It is also showing up in AI workloads, including self-hosted agents and RAG pipelines. Those workloads can put more pressure on workers, secrets management, and state handling.

We deployed n8n on Amazon EKS, ran it under real traffic, and later moved from a community Helm chart to n8n’s official Helm chart. The platform stayed in place, and we kept the database, queue, object storage, IAM role, and secrets intact through the cutover.

Why we started with the community chart

The community chart got us to production. It supported queue mode, additional ingress configuration, and StatefulSet-based main pods, which matched our architectural requirements at the time.

We ran it for about a year. Over time, maintaining unsupported features and local patches became harder to justify. We compared the available options and decided to move to the official chart, where ongoing maintenance and active community support gave us a clearer path for future upgrades.

How we ran n8n

We ran n8n in queue mode on EKS. In queue mode, the main instance manages workflow information and triggers, while worker instances execute jobs from the Redis queue. Dedicated webhook processors handled production webhook traffic separately. We split ingress by traffic type. Public webhooks ran behind an internet-facing load balancer, while internal webhooks used a private hostname.

That separation fit our traffic patterns. Workers handled workflow execution. PostgreSQL stored workflow and execution data, Redis handled queued jobs, and S3 stored binary files.

Because licensing multi-replica main pods is an Enterprise capability, we needed a secure, consistent way to handle shared application configuration and encryption key secrets across pods. We stored those secrets in AWS Secrets Manager and synced them into Kubernetes using External Secrets Operator (ESO). This synchronized application credentials, API keys, and environment variables into Kubernetes Secrets.

What changed with the official chart

The community chart ran the main pod as a StatefulSet with one persistent volume per replica. Workers and webhook processors already ran as Deployments. That StatefulSet configuration avoided multi-attach issues on AWS gp3 ReadWriteOnce (RWO) EBS volumes when running more than one main pod.

In our configuration, the official chart used a different workload model, with main running as a Deployment and persistence relying on a single shared PVC. Our gp3 volume could not attach to multiple main pods across nodes, so an in-place Helm upgrade did not fit the deployment. We treated the migration as a full cutover instead.

The shared persistence requirement was our main design decision. We checked what was stored on disk before carrying it forward:

PostgreSQL held workflow and execution data.
Redis held queued jobs.
S3 held files.
AWS Secrets Manager held the encryption key and application secrets.

We found nothing on the main pod volume that needed to persist. With PostgreSQL, Redis, S3, and Secrets Manager holding the required application state, a replacement main pod could start and reconnect without relying on a pod-specific disk. We disabled main persistence entirely for the official chart deployment.

Moving traffic safely

The platform stayed in place: we kept the database, object storage, IAM role, and secrets through the cutover. Everything else was recreated from the new chart values.

Before the production migration, we:

Tested rollback in a non-prod environment
Took a database snapshot
Saved secrets to a secondary location
Did not change the n8n image version. A version bump could trigger a database migration, and we wanted to keep the database out of the cutover story

We tested in non-production first with a restored database snapshot: pods healthy, known workflows completing, required endpoints reachable. We verified rollback there before touching live traffic.

The Helm release changed. PostgreSQL, Redis, S3, IAM, and secrets stayed.

For a public example of how we structure official-chart values for external Postgres/Redis, queue mode + HPA, resources, ingress/webhooks, optional task runners, see:

github.com/husne-fairwinds/n8n-sample

Fairwinds helps teams manage Kubernetes environments, with a focus on operational reliability, security, and maintainability.

Putting Nepal on the CNCF Golden Kubestronaut map

Thu, 15 Oct 2026 16:00:00 GMT

In this blog, I will share my Golden Kubestronaut journey, how I became the first and youngest Golden Kubestronaut from Nepal, what motivated me to pursue this path, and what it actually takes to earn the title.

Golden Kubestronaut: Basic anatomy of this program

Golden Kubestronaut is the highest honor in the Kubestronaut program, awarded by the Linux Foundation and CNCF to those who complete all 16 CNCF certifications.

Kubestronaut journey: A bit of history

I started my Kubestronaut journey during my college days in 2024. At the time, I was looking for an internship and initially planned to take the AWS Cloud Practitioner exam. After thinking it over, I asked myself: what if I went for the CKA instead? The CKA is a performance-based exam and carries real value in the market. I decided to pursue the CKA.

I attended KubeCon + CloudNativeCon India 2024 in Delhi, where I saw many Kubestronauts. That experience inspired me to aim for the Kubestronaut title myself. You can read more about my previous Kubestronaut journey.

I received my Kubestronaut jacket at KubeCon + CloudNativeCon India 2025 in Hyderabad. Wearing that jacket and walking around the event was a genuinely special feeling.

From Kubestronaut to Golden Kubestronaut

After becoming a Kubestronaut, I never imagined I would go for Golden Kubestronaut. At that time, I felt I wouldn’t pursue many more certifications in the near future.

Fast forward to 2026: I attended KubeCon + CloudNativeCon India that happened at Mumbai. My CKA was set to expire on November 15, which meant I would lose my Kubestronaut title. That fear motivated me to aim for Golden Kubestronaut.

Using the 60% off coupon I received as an attendee, I purchased the Kubestronaut to Golden Kubestronaut bundle.

The journey began on July 11 and finished on August 30 – 50 days in total. It was genuinely challenging because I was juggling several other responsibilities at the same time:

My full-time job
LFX mentorship with KubeVela
Maintaining the KubeVirtBMC project

I am grateful for the strength to handle all of this simultaneously. I still didn’t get a chance to receive an exclusive backpack with the Golden Kubestronaut logo and Golden Kubestronaut beanie. I will try to receive it at the upcoming KubeCon + CloudNativeCon events.

You can view my certifications and find my profile on the CNCF Golden Kubestronaut Map.

Why I decided to pursue Golden Kubestronaut

Personal recognition and representation: To represent Nepal in this space, as awareness of CNCF and open source is still limited in my country.
Community recognition: Visibility at events like KubeCon + CloudNativeCon and KCDs, along with the cool jacket, bag, and other benefits.
Learning: Most importantly, the certifications gave me exposure to a wide range of tools. True expertise, however, comes from hands-on work, not just passing exams.

What comes next

The certification grind stops here. There is nothing left in the catalogue that would meaningfully change what I know, and collecting badges for their own sake does not interest me. What comes next is contribution, maintaining the projects I work on, continuing the work I started through mentorship, and helping more people from underrepresented parts of the community make the same climb.

To anyone reading this from a place with a small cloud-native community: it is realistic. I did this from Nepal, on a normal schedule, alongside a full-time job. I hope the number on the map grows next year, and I would like some of that growth to come from home.

If you are working toward Kubestronaut or Golden Kubestronaut and want help, feel free to reach out. I will always make time for it.

Learn more at cncf.io/training/kubestronaut.

Carbon as a first-class signal: Bringing software carbon intensity to OpenTelemetry

Wed, 14 Oct 2026 16:00:00 GMT

At Observability Summit Europe 2026 in Prague, I presented a keynote titled Carbon as a First-Class Signal: Designing a Standard for OpenTelemetry. The talk explored how Software Carbon Intensity (SCI) could become part of cloud-native observability through OpenTelemetry semantic conventions, and how the Green Software Foundation‘s ongoing work aims to establish a common language for carbon-related telemetry.

Cloud-native systems are designed to be observable. We collect metrics, logs, and traces to understand how applications behave, identify performance bottlenecks, and make better decisions about infrastructure. We monitor CPU utilization, memory consumption, request latency, and resource allocation. Yet one important dimension of system efficiency is still largely missing from our operational view: carbon emissions.

As sustainability becomes part of the conversation around cloud-native infrastructure, the challenge is no longer simply whether we can measure energy consumption or estimate carbon emissions. It is whether we can connect that information to the workloads, services, and engineering decisions that influence it.

This is where Software Carbon Intensity (SCI) and OpenTelemetry could help establish a common approach.

Figure 1 – Me speaking at Observability Summit EU

The missing signal in observability

The cloud-native ecosystem already has tools for collecting energy and infrastructure information.

Projects such as Kepler help expose energy-related metrics from Kubernetes environments. Other tools provide information about infrastructure utilization, energy consumption, and the carbon intensity of electricity.

However, collecting these measurements is only part of the challenge.

Consider a Kubernetes workload that consumes less energy after an optimization. Is the application actually more carbon-efficient? What if the workload processes twice as many requests? What if the electricity powering the infrastructure has a different carbon intensity? And how do we attribute shared infrastructure consumption to individual applications?

Traditional observability helps us understand what a system is doing. Energy and carbon measurements introduce additional context, but without a consistent way to associate that information with software activity, comparisons become difficult.

We need to move beyond collecting isolated measurements toward understanding the carbon impact of the work our systems perform.

Understanding Software Carbon Intensity

The Software Carbon Intensity specification from the Green Software Foundation provides a methodology for measuring the carbon emissions associated with software.

Rather than considering total emissions alone, SCI relates emissions to a functional unit of software activity.

The formula is:

Where:

E (Energy): Energy consumed by the software.
I (Carbon intensity): Carbon emissions associated with the electricity consumed per unit of energy.
M (Embodied emissions): Emissions associated with the hardware used by the software, allocated to the activity being measured.
R (Functional unit): The unit of work performed, such as a request, transaction, or user.

The functional unit is particularly important because it allows us to evaluate carbon efficiency in relation to useful work.

Imagine two versions of an application.

Version A produces 100 kg CO₂e while processing one million transactions. Version B produces 110 kg CO₂e while processing two million transactions.

Although Version B produces more emissions overall, its emissions per transaction are lower:

	Version A	Version B
Total emissions	100 kg CO₂e	110 kg CO₂e
Transactions	1 million	2 million
SCI per transaction	0.0001 kg CO₂e	0.000055 kg CO₂e

Version B has a lower carbon intensity per transaction, despite its higher total emissions.

Figure 2: Software Carbon Intensity (SCI)

This distinction matters when evaluating software optimizations. Reducing total energy consumption and improving efficiency per unit of work are related, but they are not interchangeable objectives.

SCI gives engineers a structured way to examine both.

Why OpenTelemetry matters

OpenTelemetry has helped the cloud-native community establish a common language for collecting and exchanging telemetry.

Through its semantic conventions, OpenTelemetry defines how attributes and measurements can be represented consistently across different systems, programming languages, and infrastructure environments.

This same approach could help address a significant challenge in software sustainability: making carbon-related information understandable and interoperable.

Today, different tools may expose energy consumption, electricity carbon intensity, or estimated emissions using different names, units, and levels of granularity.

Even when the underlying measurements are available, connecting them to a particular Kubernetes workload, service, or operation is not straightforward.

The SCI for OpenTelemetry initiative explores how SCI concepts could be represented using OpenTelemetry semantic conventions.

The goal is not to introduce carbon as a replacement for existing observability signals. Instead, it is to establish a consistent way to describe carbon-related information alongside the operational context engineers already collect.

For example, a shared vocabulary could help describe energy consumption, carbon intensity, embodied emissions, and the functional unit used for an SCI calculation.

This would make it easier for different implementations to exchange and interpret sustainability information without relying on tool-specific definitions.

Importantly, this work is still evolving. The proposed conventions should not be confused with finalized or officially adopted OpenTelemetry semantic conventions.

The difficult part: attribution and context

One of the most important questions in bringing SCI to OpenTelemetry is how carbon-related information should be attributed to software.

In cloud-native environments, infrastructure resources are frequently shared. Multiple workloads can run on the same node, applications can scale dynamically, and services may depend on infrastructure distributed across different locations.

Measuring the energy consumption of a physical machine does not automatically tell us how much energy a particular application consumes.

Similarly, knowing the carbon intensity of electricity in a region does not directly establish the emissions attributable to an individual service.

The SCI for OpenTelemetry community has been examining these questions through a series of discussions focused on scope, use cases, and the information needed to support consistent calculations.

Several important themes have emerged.

Attribution rather than measurement alone. The challenge is not necessarily to invent new ways to measure energy. It is to establish how existing measurements and estimates can be associated with the software responsible for a particular activity.

Support for modeled and estimated values. Not every environment provides direct energy measurements. Some values must be estimated or derived from models. A useful specification needs to account for these differences rather than assume every measurement is directly observed.

Provenance and transparency. Engineers need to understand where carbon-related values originate, how they were calculated, and what assumptions were involved. Without this information, comparisons can be misleading.

Clear boundaries and units of work. SCI calculations depend on what is included in the measurement and which functional unit is selected. These choices need to be explicit for results to be meaningful.

These are not simply naming problems. They influence how carbon information can be interpreted, compared, and used in practice.

From scope to semantic conventions: where the work stands today

At the time of writing, the first phase of the SCI for OpenTelemetry Assembly, focused on scope and use cases, has concluded with community consensus.

The objective was to establish which problems the proposed semantic conventions should address first, before defining the attributes, metrics, and structures needed to represent SCI in OpenTelemetry.

The community agreed on three use cases, in order of priority.

1. Helping engineers understand the impact of their changes

The highest priority is enabling application developers, platform engineers, and SREs to determine whether a release, configuration change, or architectural decision has reduced energy consumption or carbon emissions for the same amount of work.

Just as importantly, engineers should be able to identify which components are responsible when consumption increases.

This requires attributing energy and emissions to identifiable software entities, using consistent measurement windows and explicitly defined functional units.

2. Comparing carbon impact across services and providers

The second use case addresses comparisons between infrastructure providers, services, and migration options.

However, the Assembly recognized an important limitation: simply declaring a functional unit does not guarantee that two systems perform equivalent work.

Meaningful comparisons require clearly documented measurement boundaries, calculation methods, and functional units. Fully standardized comparisons would also require common workloads and testing procedures, which semantic conventions alone cannot provide.

3. Supporting automated carbon-aware decisions

The third use case explores how schedulers, autoscalers, and workload-placement systems could use carbon-related information to make operational decisions.

Unlike retrospective measurements, automated decisions often require timely information. This may involve forecasts, estimates, or modeled values rather than direct measurements.

The consensus emphasizes that these distinctions must remain visible. A forecast should not be presented as equivalent to a directly measured value.

What these decisions mean for the next phases

A common requirement emerged across all three use cases: carbon-related telemetry must clearly communicate its boundaries, calculation method, functional unit, and the nature of the value being reported.

This includes distinguishing operational from embodied emissions, identifying estimates and forecasts, and making the completeness of a measurement explicit.

These decisions provide a foundation for the next stages of the Assembly, which will address the SCI namespace and the representation of operational energy, embodied emissions, and functional units.

The goal is to develop proposed OpenTelemetry semantic conventions that can be reviewed and refined by the OpenTelemetry community, rather than introduce another incompatible way of representing sustainability information.

Community participation and diversity

Beyond the technical outcomes, the first phase also brought together participants from different backgrounds, organizations, and areas of expertise. The participation metrics and diversity data reflect the collaborative nature of the Assembly, connecting perspectives from software sustainability, observability, and cloud-native engineering. This diversity matters because defining a common standard requires more than technical agreement: it requires input from the people who will eventually implement, maintain, and use it across different environments.


Figure 3: Donut chart showing 65% (24)of invited participants took part across Discovery, Deliberate, and Decide phases, while 35% (13)did not.	*Figure 4*: Pie chart detailing participant diversity: 11 Enterprise/Product Engineering, 9 Observability/Performance Vendors, 7 Open Source/Community, 5 Consulting/IT Services, 4 Universities, and 1 Industry Research Lab.

The Phase 1 consensus document is publicly available and documents the agreed priorities and requirements.

Get involved in the next phases

The SCI for OpenTelemetry Assembly brings together engineers, sustainability practitioners, and observability contributors to help define how carbon information can be represented consistently.

Participation is not limited to Green Software Foundation members. The Assembly welcomes expertise in OpenTelemetry, Kubernetes, energy measurement, software sustainability, and cloud-native infrastructure.

The work is primarily asynchronous, allowing participants to contribute through structured discussions and consensus-building without attending mandatory meetings.

At the time of writing, applications for the current Assembly have closed, but interested contributors can join the waitlist through the Green Software Foundation to express interest in available places or future opportunities.

The SCI for OpenTelemetry GitHub repository also provides access to the ongoing work and published materials.

For cloud-native practitioners, this is an opportunity to help shape how sustainability information becomes part of the operational telemetry we already use.

From carbon measurements to engineering decisions

The long-term value of bringing carbon information into observability is not simply another dashboard.

It is the ability to consider carbon alongside the operational signals engineers already use when evaluating systems.

For example, consider a team optimizing a Kubernetes application. They might reduce CPU utilization, adjust resource requests, introduce autoscaling, or change how workloads are scheduled.

These changes can influence performance, infrastructure cost, energy consumption, and carbon emissions in different ways.

A configuration that improves one dimension does not necessarily improve all the others.

Having carbon-related information available alongside existing telemetry could help engineers understand these relationships instead of relying on assumptions.

A practical approach involves three steps.

First, measure what is available. Use existing infrastructure and observability tools to collect energy-related measurements and the context needed to interpret them.

Second, connect carbon to operational information. Associate energy consumption and carbon estimates with workloads, resource utilization, and relevant software activity. Make the source and limitations of estimated values visible.

Third, use that information in engineering decisions. Evaluate whether changes to resource allocation, scheduling, application efficiency, or infrastructure utilization improve carbon efficiency without overlooking reliability and performance requirements.

This is also where GreenOps becomes relevant.

Just as FinOps helps organizations connect technical resource consumption with financial impact, GreenOps brings environmental impact into infrastructure and software operations.

Carbon-aware engineering does not require treating sustainability as a separate operational discipline. It means making environmental impact part of the information available when designing and operating cloud-native systems.

Figure 5: Electricity Maps

An opportunity for the cloud-native community

Standardizing sustainability-related telemetry requires collaboration across several communities.

Energy measurement projects, observability practitioners, sustainability specialists, and software engineers all approach the problem from different perspectives.

The SCI for OpenTelemetry initiative provides a place to connect those perspectives and work toward a shared representation.

For the OpenTelemetry community, the opportunity is to explore how existing telemetry concepts can support sustainability use cases without introducing unnecessary fragmentation.

For the broader CNCF ecosystem, it is an opportunity to make energy and carbon information more interoperable across cloud-native infrastructure and applications.

And for practitioners, the most valuable contribution may be sharing real-world requirements: which measurements are available, where attribution becomes difficult, which estimates are necessary, and what information is needed to make useful comparisons.

These practical experiences can help shape conventions that work beyond individual implementations.

Making carbon a first-class operational signal

Cloud-native observability has evolved because engineers need context to understand increasingly complex systems.

Carbon information presents a similar challenge.

Measuring energy consumption is useful. Estimating emissions is useful. But the real opportunity comes from connecting those measurements to software behavior and the decisions that shape it.

SCI provides a methodology for evaluating software carbon intensity. OpenTelemetry offers a common language for representing and exchanging operational information.

Bringing these approaches together could help make carbon another meaningful dimension of cloud-native system efficiency.

Not as a replacement for performance, cost, or reliability, but as additional context for making better engineering decisions.

SCI gives us a way to measure. OpenTelemetry gives us a common language. Carbon can become a first-class operational signal.

Extending Kubernetes policy enforcement to runtime security for AI workloads

Wed, 14 Oct 2026 16:00:00 GMT

Nirmata created Kyverno and donated it to the CNCF in 2020. Kyverno is now a graduated CNCF project and is widely used for enforcing and automating configuration security and best practices.

However, admission control answers one question: does this spec look right? It cannot answer the question that matters once the pod is running: is this workload doing what its spec says it should? With autonomous AI agents and non-deterministic LLM-enabled applications, that gap is where most of the interesting incidents live.

A container with an innocuous-looking image pulls a second-stage payload after startup. A sidecar with network access nobody scoped tightly enough starts talking to a destination no one approved. A process no configuration policy anticipated executes, because the policy only ever described the container’s declared behavior, not its runtime one.

One approach to addressing this gap is kernel-level runtime enforcement, which enables security decisions to be applied as workloads execute. Nirmata Runtime for Kyverno is an open source implementation of this approach, using eBPF and Linux Security Modules (LSM) to extend policy enforcement beyond Kubernetes admission control.

Below the workload, not beside it

The design premise is simple: any control that runs beside a workload — a sidecar, an agent, a userspace proxy — is something that the workload’s own compromise can see, stall, or step around. A control that runs below it, in the kernel, enforcing at the same instruction as the syscall it’s gating, isn’t in that position. The workload doesn’t get a vote.

Concretely, Nirmata Runtime attaches BPF-LSM programs directly to bprm_check_security and file_open, and cgroup-scoped eBPF programs to the egress path, per pod. Five behaviors, all decided in-kernel, all synchronous with the operation they gate:

exec — deny process execution by resolved kernel path
open — deny file access by resolved kernel path
network — deny egress by destination
protocol — deny by the application protocol actually spoken on the wire, independent of destination port — a reverse shell dressed up as HTTPS on 443 doesn’t get a pass just because the port looks right
dns — observe and report on resolution, without blocking, because DNS denial has failure modes worse than the traffic it would stop

A blocked exec returns -EPERM before the process runs. A blocked connection never completes. There’s no window between “decision” and “enforcement” for a compromised workload to race.

Reusing Kubernetes policy concepts for runtime enforcement

If you already write Kyverno policies for admission, you already know the policy language here: CEL. RuntimePolicy targets workloads the same way — podSelector/namespaceSelector — and expresses allow/deny logic per behavior, including expressions that pull live values from a ConfigMap or an HTTP source on a refresh interval. Admission and runtime become one continuous policy story instead of two unrelated tools with two unrelated syntaxes glued together with tribal knowledge about which one to check first.

Every policy ships in monitor mode by default. You watch what it would deny — including a WouldDeny counterfactual signal — before you ever flip it to enforce. Nothing goes into blocking mode silently.

Besides the language, Nirmata Runtime uses the same reporting APIs i.e., OpenReports and will add fine-grained exceptions similar to Kyverno for a seamless operator experience.

The same guarantee for AI agents in your cluster

A coding agent, an MCP server, an autonomous pipeline step — these are pods too, and the reasoning above applies to them without a special case. But agents raise a specific question that admission control was never built to answer: an agent’s behavior isn’t fully known during deployment time, because part of what it does is generated by a model at runtime. You can review the image. You can’t review the decision the agent hasn’t made yet.

For AI workloads with dynamically determined behavior, kernel-level enforcement offers an additional layer of control that operates independently of application-level safeguards. This can complement existing approaches to securing AI agents, including network policies, admission controls, and application-level guardrails.

If an agent’s runtime is compromised — a prompt injection, a supply-chain dependency, a tool call that goes somewhere it shouldn’t — the enforcement boundary can’t be argued with, reasoned around, or quietly disabled by the thing running inside it, because it isn’t running in that process at all.

Concretely, with Nirmata Runtime you can, per agent workload:

Bound which model providers and endpoints it’s allowed to talk to — dns and network behaviors report or block resolution and egress outside an approved list, catching an agent that starts calling a provider nobody reviewed.
Restrict which binaries and MCP servers it can launch — exec allow-lists close off arbitrary tool invocation to exactly what the workload contract says the agent may run.
Detect the agent CLIs and SDKs already present in an image you didn’t build with agents in mind, and the model files and MCP configs a workload reads, before those become undeclared surfaces.

Here’s what that looks like as one policy on an agent pod that should only ever reach an internal LLM gateway, and should never read the credentials sitting on its own filesystem:

apiVersion: runtime.nirmata.io/v1alpha1
kind: RuntimePolicy
metadata:
  name: agent-gateway-only
spec:
  mode: enforce
  podSelector:
    matchLabels:
      nirmata.io/agent: "true"
  behaviors:
  - network:
      allow:
        values:
        - llm-gateway.ai-platform.svc.cluster.local
        - kube-dns.kube-system.svc.cluster.local
      deny:
        values:
        - "*"
  - protocol:
      allow:
        values:
        - tls
        - dns
      deny:
        values:
        - "*"
  - open:
      deny:
        values:
        - "/root/.aws/credentials"
        - "/root/.kube/config"
        - "/root/.ssh/id_rsa"
        - "/var/run/secrets/kubernetes.io/serviceaccount/token"

network and protocol both carry the “*” sentinel, so a connection has to clear both: the gateway’s address over the wrong protocol is denied, and TLS to anywhere else is too. open doesn’t need the sentinel here — these four paths are exactly the ones a compromised agent would reach for to pivot into the cloud account, the cluster, or another host, and naming them is enough. (open needs a BPF-LSM-active node; network/protocol only need cgroup v2 and BPF.)

If a prompt injection gets a tool call to run cat ~/.aws/credentials or talk to some other endpoint entirely, the kernel has already made the decision before the agent’s own process — or whatever compromised it — gets a chance to argue.

This is enforced the same way as any other workload — no agent-specific proxy, no separate control plane, no assumption that the agent’s own guardrails will hold. The worked examples are in examples/shadow-ai/, covering unexpected DNS, AI SDK detection, and MCP server/config discovery end to end.

Observability that says ***why, not just that***

Runtime security monitoring benefits from contextual information that explains which policy triggered an enforcement decision, rather than reporting event counts alone.

When Nirmata Runtime denies something, the kernel encodes the decision in the map key, and userspace re-evaluates the compiled policy against the observed target to attribute the exact policy responsible. If an event can’t be attributed, it isn’t dropped silently — it’s counted and logged as a gap (events_dropped_total{reason="unattributed_kernel_deny"}), because a monitoring blind spot that looks identical to “nothing happened” is worse than a visible error. Findings land as standard Kubernetes OpenReports objects — no proprietary dashboard required to see what your cluster is actually doing.

Raw observation and a readable report are two different things, though, and CEL is what turns one into the other. Observing exec in monitor mode streams every process a pod runs — necessary, because you don’t know what you’re looking for yet, and useless as a report, because most of it is sh, grep, and the rest of a normal container’s argv. monitorFilter is a per-observation CEL predicate that runs before a finding is written, so the noise never reaches the Report at all:

apiVersion: runtime.nirmata.io/v1alpha1
kind: RuntimePolicy
metadata:
  name: detect-agent-cli
spec:
  mode: monitor
  podSelector:
    matchLabels:
      app: ai-workload
  behaviors:
  - exec:
      deny:
        values: ["*"]
  monitorFilter:
    expressions:
    - name: exec-events-only
      expression: 'has(event.exec)'
    - name: agent-cli-or-inference-server
      expression: >-
        event.exec.argv.exists(a, a in [
          "claude", "codex", "gemini", "aider", "goose", "opencode", "crush",
          "cursor-agent", "ollama", "litellm", "text-generation-launcher",
        ]) ||
        event.exec.argv.exists(a,
          a.startsWith("@anthropic-ai/") ||
          a.startsWith("@openai/") ||
          a.startsWith("@google/gemini-cli"))

The first expression is a guard — without it, the second one evaluates against open and network observations too, where event.exec doesn’t exist. The second is the actual question: not “did a binary run” but “was an agent CLI or a local inference server invoked,” matched on the name it was invoked under or the package a launcher fetched, because npx, uvx, and python are just interpreters — the identity is in argv, not in the exec path. One policy, no sidecar parsing logs after the fact, and a Report that already says an agent CLI showed up on a pod nobody built with agents in mind, not that a process ran.

What’s next

Runtime enforcement is one component of a broader, defense-in-depth approach to securing Kubernetes and AI workloads. Combining runtime controls with admission policies, network security, and application-level safeguards can help organizations address security risks across different stages of the workload lifecycle.

Nirmata Runtime is available as an open source project. The community is invited to explore the implementation, review the examples, contribute improvements, and share feedback through the GitHub repository.

like the project make sure you give it a ⭐ to show your support, and to receive updates.

From zero to agent builder: Running kagent on your laptop

Sun, 11 Oct 2026 16:00:00 GMT

If you have seen a demo of a no-code agent builder, you know the moment. Someone types one sentence, waits a minute, and there it is: a new agent, running, with tools. It is impressive. For about ten seconds.

Then the questions start. Where does it run? What can it touch? Who approved it?

This post is about the first part, the ten impressive seconds. I want to show that it is not magic and that you can build it yourself, in an afternoon, with open source and spending a minimum on tokens. The tool is kagent, a CNCF Sandbox project that runs AI agents as Kubernetes resources. The questions come back at the end, because they are the real work.

Why kagent makes this almost boring

In kagent, an agent is not a process you start by hand. It is a custom resource. You write an Agent with a system prompt, a model, and a list of tools. You apply it, and the controller turns it into a running pod that speaks A2A.

Once agents are just YAML, an agent that builds agents is simply an agent that can write one kind of YAML. No new framework, no plugin. One Agent, three tools, and a prompt that knows what an Agent looks like.

Figure 1: An agent that builds agents, on kagent.

Before you start: What this lab is not

The goal here is the builder. Security is out of scope, on purpose, and I want to be upfront about what that means:

kagent runs in its default unsecure auth mode, so every request acts as the same user.
The chart’s bundled PostgreSQL uses its built-in password.
The UI and API are reachable only through kubectl port-forward on 127.0.0.1.

That is fine for a laptop. It is not fine for a shared cluster, and I will come back to why that gap matters.

You need Docker, kind, kubectl, helm, jq and Ollama (alternatively, you can use your API Key for Claude, OpenAI, or any other provider, but we wanted to make it as cheap as possible). There is nothing to clone: you create every file as you go. Start in an empty folder:

mkdir kagent-agent-builder && cd kagent-agent-builder

Step 1: Pick a local model that can actually do it

This is the part that surprised me. Writing a valid Kubernetes manifest through a tool call is harder than chatting, and small models show it. In my runs, qwen2.5:7b-instruct invented its own tool block format, and qwen2.5:14b-instruct sometimes answered politely without ever calling the apply tool. qwen3-coder:30b-a3b got it right on every run, so that is the model this post uses:

ollama pull qwen3-coder:30b-a3b-q4_K_M   # about 18 GB on disk

The cluster reaches Ollama at host.docker.internal:11434, which works out of the box with Docker Desktop. On Linux you have to change that host in the values file below to an address the kind node can reach. I have not tested that path.

If you prefer a hosted model, kagent’s Helm chart also has provider blocks for OpenAI, Anthropic, Azure OpenAI and Gemini. Then it is no longer free, of course.

Step 2: Install kagent with a tool server that can only write Agents

Create the cluster:

kind create cluster --name kagent-builder

kagent ships a tool server with Kubernetes tools, and by default the chart gives its ServiceAccount cluster-admin. A builder backed by cluster-admin is a bad idea even on a laptop, so the values file changes that. Create it:

providers:
  default: ollama
  ollama:
    provider: Ollama
    model: "qwen3-coder:30b-a3b-q4_K_M"
    config:
      host: host.docker.internal:11434
      options:
        num_ctx: "32768"
kagent-tools:
  tools:
    enabledTools:
      - k8s
  rbac:
    readOnly: true
    additionalRules:
      - apiGroups: ["kagent.dev"]
        resources: ["agents"]
        verbs: ["get", "list", "watch", "create", "update", "patch"]
grafana-mcp: {enabled: false}
k8s-agent: {enabled: false}
kgateway-agent: {enabled: false}
istio-agent: {enabled: false}
promql-agent: {enabled: false}
observability-agent: {enabled: false}
argo-rollouts-agent: {enabled: false}
helm-agent: {enabled: false}
cilium-policy-agent: {enabled: false}
cilium-manager-agent: {enabled: false}
cilium-debug-agent: {enabled: false}

The model becomes the default ModelConfig. The tool server loads only its Kubernetes tools (22 of them). And it gets a read-only role plus one extra rule: it may write agents.kagent.dev, nothing else. The last block turns off the sample agents the chart installs by default, so the UI only shows yours.

Install the CRDs and kagent 0.10.2 with that file, and wait for the pods:

helm upgrade --install kagent-crds oci://ghcr.io/kagent-dev/kagent/helm/kagent-crds \
  --kube-context kind-kagent-builder \
  --version 0.10.2 --namespace kagent --create-namespace
helm upgrade --install kagent oci://ghcr.io/kagent-dev/kagent/helm/kagent \
  --kube-context kind-kagent-builder \
  --version 0.10.2 --namespace kagent --values kagent-values.yaml
for d in kagent-controller kagent-ui kagent-tools; do
  kubectl --context=kind-kagent-builder -n kagent rollout status deploy/$d --timeout=300s
done

Step 3: The builder is just an agent

This is the whole builder. Create the file:

apiVersion: kagent.dev/v1alpha2
kind: Agent
metadata:
  name: agent-builder
  namespace: kagent
spec:
  description: Builds new kagent agents from a plain-language description and deploys them.
  type: Declarative
  declarative:
    modelConfig: default-model-config
    stream: false
    systemMessage: |
      You build kagent agents. The user describes, in plain language, an agent
      they want. You turn that into ONE Kubernetes manifest and apply it. Do not
      ask follow-up questions: pick sensible defaults and go.
      How you work, in this order:
      1. List the existing agents with k8s_get_resources (resource_type
         agents.kagent.dev, namespace kagent). Pick a name that is not taken;
         never overwrite an existing agent.
      2. Write the manifest in exactly the shape below and apply it with
         k8s_apply_manifest. The user confirms the apply in the UI; that is
         expected.
      3. Read it back with k8s_get_resource_yaml (resource_type
         agents.kagent.dev, namespace kagent) and look at the Accepted and
         Ready conditions in its status. Ready can take a few seconds.
      The manifest must be exactly this shape, and nothing else:
        apiVersion: kagent.dev/v1alpha2
        kind: Agent
        metadata:
          name: <short-kebab-case-name>
          namespace: kagent
          labels:
            app.kubernetes.io/managed-by: agent-builder
        spec:
          description: |-
            <one sentence>
          type: Declarative
          declarative:
            modelConfig: default-model-config
            stream: false
            systemMessage: |
              <a clear prompt for the new agent, written for its task>
            tools:
              - type: McpServer
                mcpServer:
                  apiGroup: kagent.dev
                  kind: RemoteMCPServer
                  name: kagent-tool-server
                  toolNames: [<pick from the read-only list below>]
      Read-only tools you may give it: k8s_get_resources, k8s_get_resource_yaml,
      k8s_describe_resource, k8s_get_events, k8s_get_pod_logs.
      Never another kind, never another namespace, never write or delete tools,
      never more than one document.
      YAML that does not break: free text always goes in a block scalar, as in
      the shape above (description with |-, systemMessage with |). In a plain
      scalar, a colon followed by a space makes the YAML invalid, and a space
      followed by # silently cuts the value short.
      When something goes wrong:
      - If the user rejects the apply, do not retry it and do not read the
        agent back. Say it was not applied, and stop.
      - Never create a second resource with another name (-test, -check, -v2)
        to find out what failed. It tells you nothing and leaves clutter.
      - Do not name a cause you have not checked. An opaque error is not proof
        of a name clash, a permission problem or a network issue.
      - Report the error exactly as the tool returned it, and stop.
      When it is done, answer in three short lines: the agent's name, what it
      does, and which tools it can use. Then one more line with its real state
      from step 3 (Accepted, Ready). Never claim it is ready if it is not.
    tools:
      - type: McpServer
        mcpServer:
          apiGroup: kagent.dev
          kind: RemoteMCPServer
          name: kagent-tool-server
          toolNames:
            - k8s_apply_manifest
            - k8s_get_resources
            - k8s_get_resource_yaml
          requireApproval:
            - k8s_apply_manifest
    a2aConfig:
      skills:
        - id: build-agent
          name: Build an agent
          description: Turns a plain-language description into a deployed kagent agent.
          tags: [builder, agents]
          examples:
            - "Create an agent for the support team that answers questions about the state of the services in the cluster"

Apply it and wait until kagent reports it Ready:

kubectl --context=kind-kagent-builder apply -f agent-builder.yaml
kubectl --context=kind-kagent-builder -n kagent wait --for=condition=Ready \
  agent/agent-builder --timeout=300s

The tools block at the end is where the builder gets its limits. toolNames is an allowlist: of the 22 tools on the server, the model sees three. requireApproval is kagent’s human-in-the-loop switch. When the agent calls a tool on that list, the run pauses and the user has to approve or reject the call.

The system prompt does the rest. It tells the model to list existing agents first, so it never overwrites one, and then gives it the exact manifest shape it may produce: one Agent, in the kagent namespace, with a label that marks it as built by the builder, and only read-only tools. It also carries a few lessons I learned the hard way. Free text goes in YAML block scalars, because a colon followed by a space inside a plain string breaks the manifest. And when something fails, the model must report the error and stop, not create my-agent-v2 to see what happens.

Keep in mind that the prompt is a request, not a guarantee. The guarantees are the allowlist, the approval and RBAC.

Step 4: Ask for an agent

Open the UI:

kubectl --context=kind-kagent-builder -n kagent port-forward \
  --address 127.0.0.1 svc/kagent-ui 8080:8080

Go to http://localhost:8080, pick agent-builder and ask for something:

Create an agent for the support team that answers questions
about the state of the services in the cluster

After half a minute or so the run stops. The UI shows the manifest the model wants to apply and two buttons.

Figure 2: The UI showing the manifest requiring user approval before applying.

Read it before you click. This is the one moment where a human sees exactly what is about to land in the cluster. Approve, and the builder applies it, reads it back and reports the real state:

Figure 3: The UI showing the result and real state after applying the manifest.

The new agent shows up in the agent list and you can chat with it straight away. Ask it which services exist in the kagent namespace. It answers with read-only tools, because those are the only ones the builder was allowed to give it.

Step 5: The same thing without the UI

Every kagent agent is an A2A endpoint on the controller, so you can script the whole flow. Forward the controller port and send the request:

kubectl --context=kind-kagent-builder -n kagent port-forward \
  --address 127.0.0.1 svc/kagent-controller 8083:8083 &
sleep 3   # give the port-forward a moment to listen

curl -s http://127.0.0.1:8083/api/a2a/kagent/agent-builder/ \
  -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":"1","method":"message/send","params":{"message":{"role":"user","messageId":"1","parts":[{"kind":"text","text":"Create an agent that lists the pods that are not running in the kagent namespace"}]}}}' \
  > build.json

jq -r '.result.status.state' build.json    # input-required

input-required means the apply is waiting for you. The pending manifest is in a data part of the status message. To answer, send a message on the same task and context with a decision:

TASK=$(jq -r .result.id build.json); CTX=$(jq -r .result.contextId build.json)
curl -s http://127.0.0.1:8083/api/a2a/kagent/agent-builder/ \
  -H 'Content-Type: application/json' \
  -d "{\"jsonrpc\":\"2.0\",\"id\":\"2\",\"method\":\"message/send\",\"params\":{\"message\":{\"role\":\"user\",\"messageId\":\"2\",\"taskId\":\"$TASK\",\"contextId\":\"$CTX\",\"parts\":[{\"kind\":\"data\",\"data\":{\"decision_type\":\"approve\"}}]}}}" \
  | jq -r '.result.status.state'            # completed

Use “reject” instead of “approve” and nothing is created. Either way, the cluster tells you what really happened:

kubectl --context=kind-kagent-builder -n kagent get agents

Why three guardrails and not one

Figure 4: Three guardrails that do not depend on the prompt.

It is tempting to think the prompt is enough. It is not, and not because the model is malicious. Models get YAML wrong, retry in creative ways and misunderstand requests.

It happened to me while checking the commands for this post. The prompt spells out the tool server name, kagent-tool-server, and one run still produced an Agent pointing at kagent-tools. My curl approved it blindly, as scripts do. The controller refused it: Accepted=False, RemoteMCPServer "kagent-tools" not found. Nothing broke, but it is a good reminder of where the real checks live. Each layer catches what the previous one lets through:
The allowlist. If the model reaches for k8s_delete_resource, the tool simply is not there.
The approval. If it writes a manifest you did not expect, you see it before it exists.
RBAC. If you approve something you should not have, the API server still refuses anything that is not an Agent.

You can check that last one yourself:

SA=system:serviceaccount:kagent:kagent-tools

kubectl --context=kind-kagent-builder auth can-i create
agents.kagent.dev --as=$SA -n kagent   # yes

kubectl --context=kind-kagent-builder auth can-i create
deployments       --as=$SA -n kagent   # no

kubectl --context=kind-kagent-builder auth can-i get secrets              
--as=$SA -n kagent   # no

The part this lab leaves open

So building the agent took one YAML file and a couple of Helm installs. That was the easy part, and it is exactly why I do not think it is the interesting part anymore.

Look again at the box at the bottom of Figure 4. On this laptop nobody knows who asked for the agent, because everyone is the same user. Nobody knows what its model calls cost, or for which team. And nobody decided whether the agent should exist, beyond one click by whoever happened to be in the chat. Now imagine a company with a builder like this, a few frameworks, a hosted agent platform and a couple of assistants on developers’ machines. Each one answers those questions differently, if it answers them at all.

That is the argument for a governance layer that sits in front of the agents, wherever they come from, instead of inside each one. In the open source world one option is agentgateway, a Linux Foundation project. It proxies LLM, MCP and A2A traffic, with authentication (JWT, API keys), fine-grained authorization with CEL, and OpenTelemetry metrics, logs and traces for every call. Wiring it in front of this lab is a good next exercise, and probably my next post.

Clean up

Everything lives in the kind cluster, so one command removes it:

kind delete cluster --name kagent-builder

Where to go from here

The kagent docs cover more tool servers, other model providers and agents written in code, and the project lives at github.com/kagent-dev/kagent. If you build something on top of this, or you find a smaller local model that handles the builder well, I would really like to hear about it.

Join Platform Engineering Day at KubeCon + CloudNativeCon North America 2026

Thu, 08 Oct 2026 16:00:00 GMT

As organizations accelerate their adoption of AI and cloud native technologies, platform teams are being asked to solve a growing set of challenges. How can developers access new capabilities such as AI safely and efficiently? How can platforms reduce bottlenecks rather than create them? And how should internal platforms evolve as the needs of developers and organizations change?

These are some of the questions at the heart of Platform Engineering Day, a full-day event bringing the community together to exchange practical experiences, lessons learned, and ideas about the future of platform engineering, as part of KubeCon + CloudNativeCon North America in Salt Lake City, on Monday, November 9th 2026.

Created by the community, for practitioners

The idea for Platform Engineering Day began at KubeCon + CloudNativeCon Europe in Amsterdam in 2023.

Platform engineering was already generating significant discussion across the community, but there was an opportunity to create a dedicated space that went beyond individual tools and project updates. Practitioners wanted to talk about the principles and practices behind building and operating platforms—and, importantly, to share what they were learning along the way.

That idea became Platform Engineering Day, with the first event taking place alongside KubeCon + CloudNativeCon Europe in Paris in 2024.

Since its inception, the goal has remained simple: create a space where platform engineers and practitioners can learn from one another.

That means talking not only about successful implementations, but also about approaches that did not work as expected, trade-offs teams encountered, and lessons they would apply differently next time.

Learning from real-world platform engineering

Platform engineering rarely has a one-size-fits-all answer. Teams have different users, organizational structures, technologies, constraints, and definitions of what a successful internal platform looks like.

Platform Engineering Day prioritizes end-user stories and case studies, giving attendees an opportunity to hear directly from practitioners building and evolving platforms in real environments.

The aim is to move beyond what teams are building and explore why they made particular decisions, what they learned, and how their approaches have changed over time.

This year’s conversations will span some of the issues currently shaping platform engineering, including AI and platform engineering, sustainability, agents as both producers and consumers of platforms, Platform as a Product, and compliance.

AI in particular is introducing new questions for platform teams. Organizations want developers to experiment with and adopt AI capabilities, while still providing appropriate security, governance, reliability, and operational guardrails. Platform engineering can provide an interface between those requirements, helping teams make new capabilities accessible without introducing unnecessary friction into software delivery.

A full day for every stage of the platform engineering journey

Platform Engineering Day is designed for people at different stages of their platform engineering journey, from those beginning to explore the discipline to practitioners already operating mature internal platforms.

The day will begin with a single track in the morning, bringing the community together around shared topics and perspectives. After lunch, the program will split into two tracks, allowing attendees to explore a broader range of subjects and levels of technical depth.

Across the day, attendees can expect a mixture of keynotes, lightning talks, panels, practitioner stories, and opportunities to connect with others in the community.

No prior platform engineering knowledge is required.

Whether you are a platform engineer, product manager, solutions architect, technical leader, decision maker, or simply beginning to explore platform engineering, the program is intended to offer perspectives relevant to your journey.

Where platform engineering is heading

Platform engineering continues to evolve alongside the wider cloud native ecosystem.

The emergence of AI and agentic workloads is creating new requirements for platforms, while established challenges around developer experience, sustainability, compliance, governance, and treating platforms as products continue to mature.

Platform Engineering Day provides a place to examine these changes through the experiences of the people encountering them firsthand.

Attendees should leave with practical examples of how other teams are approaching these challenges, lessons they can apply within their own organizations, and a clearer picture of the conversations currently shaping the platform engineering community.

Just as importantly, the event provides an opportunity to share experiences with peers. By discussing successes alongside failures, challenges alongside solutions, the community can continue learning together and building better platforms.

Join co-chairs Paula Kennedy, Stacey Potter, and Ramiro Berrelleza at Platform Engineering Day and be part of the conversation about what’s next for platform engineering.

Making the Golden Kubestronaut a little more golden

Thu, 08 Oct 2026 16:00:00 GMT

When I became a Golden Kubestronaut, I wanted to celebrate the achievement in a way that felt lasting.

Like most technical achievements, certifications usually live digitally. You receive the badge, update your profile, and eventually move on to the next challenge. This milestone felt different, and my partner, David, suggested turning the Golden Kubestronaut design into an actual gold necklace.

The idea immediately felt meaningful to me for another reason. Gold has held a special place in Egyptian history for thousands of years. In ancient Egypt, its enduring nature made it closely associated with permanence and eternity. I loved the idea of using that same symbolism to preserve a very modern achievement.

My Journey to Golden Kubestronaut

Becoming a Golden Kubestronaut wasn’t originally a goal I had set for myself.

As an engineer working with Kubernetes, Linux, security, DevOps, and cloud native technologies, I’ve always enjoyed challenging myself and continuing to learn. Becoming a Kubestronaut was already an achievement I was very proud of, and I wasn’t necessarily planning to take it further.

David was the person who encouraged me to go for Golden Kubestronaut. Without that encouragement, I genuinely don’t know that I would have pursued it.

Once I committed to the goal, it took a lot of studying, hands-on practice, troubleshooting, and persistence. Not everything worked the first time, and not every exam went the way I wanted. But those challenges became part of what made reaching the goal so rewarding.

Eventually, I became a Golden Kubestronaut.

More Than a Gold Necklace

Turning the Golden Kubestronaut design into a real piece of gold jewelry became a way to immortalize that moment.

Technology changes incredibly quickly. The tools we use today will evolve, and years from now my work may look very different. The necklace gives me something permanent from a journey built around technology that is constantly changing.

When I look at it, I don’t just see the Golden Kubestronaut logo. I see the work that went into earning it, the challenges along the way, and the encouragement that pushed me to pursue a goal I might not otherwise have attempted.

It also reminds me that accomplishments are rarely achieved entirely alone. Sometimes having someone who believes you is what gives you the confidence to start pursuing a goal.

As a woman in engineering, I’m proud of this milestone and hope my experience encourages someone else to take on a challenge that may initially seem out of reach.

For me, becoming a Golden Kubestronaut isn’t a finish line. There will always be something new to learn and another challenge ahead.

But this is one milestone I wanted to make permanent.

And if you’re going to be a Golden Kubestronaut, making it actually gold just seems right.

Who owns NIS2 and DORA on a Kubernetes platform team

Thu, 08 Oct 2026 16:00:00 GMT

There is a meeting that repeats across a lot of European engineering organizations right now. Someone from legal or risk brings a NIS2 control, a DORA article, or a “can we evidence this” request into the room. The Kubernetes platform team looks at security. Security looks at the platform team. A ticket lands on the security board, because that feels like the responsible place for anything with a regulation in the title. Three sprints later the cluster has not changed, the evidence still does not exist, and everyone is quietly annoyed with the security lead.

The failure is not legal and it is not technical. It is an ownership failure, and it has a specific shape: a regulation name was put on a board instead of an artifact being put in a sprint. Security can write the policy. Security cannot merge the Helm chart.

This article is about closing that gap on a cloud native platform: the traceability chain from a regulatory requirement to a Kubernetes object, a RACI that survives contact with sprint planning, and the open source projects that actually produce the evidence.

One caveat before going further. Nothing here is legal advice, and none of the mappings below are an official compliance mapping. NIS2 and DORA are outcome-based texts. Which entities and services are in scope, what measures are proportionate, and what evidence is sufficient all depend on the organization’s own risk assessment, its sector rules, national transposition, and its regulator. They do not depend on a table in a blog post.

What happens when security owns the regulation

The common default is that anything with a regulation in the title becomes a security ticket. The CISO, or the one senior security engineer who still answers Slack, becomes the owner of NIS2, DORA, and whatever arrives next. Platform engineers stay on product work. Application teams stay on features. The security function becomes a queue.

Two failure modes follow, and they feed each other.

The first is the bottleneck. Every image exception, every privileged securityContext, every “we need this log in the SIEM” request waits on the same two people. Delivery teams learn that talking to security is how work dies, so they start routing around it: a break-glass kubeconfig here, a sidecar nobody reviewed there, a registry that is not the approved catalog.

The result rarely announces itself. Non-compliance does not show up as a red cell on a slide; it shows up as shadow infrastructure carrying a ticket label that still says “security”.

The second is learned helplessness on the platform side. If compliance is permanently someone else’s backlog, platform engineers never own the reasoning behind it. They ship an admission policy because they were told to, disable it when it blocks a release, and wait for security to notice. That is the opposite of the ownership every platform engineering talk asks for. You cannot ask a team to own outcomes and simultaneously keep the regulatory outcome on another team’s board.

Neither failure is a personality problem. Both are org designs that nobody named out loud.

What NIS2 Article 20 and DORA Article 5 actually assign

Both texts are blunt about where accountability sits, and it is not “the security team”.

NIS2 Article 20 places cybersecurity risk-management measures on the management body of essential and important entities. The management body approves the measures, oversees their implementation, can be held liable for non-compliance, and is required to follow training.

DORA Article 5 does the same for financial entities. The management body defines, approves, oversees, and remains responsible for the ICT risk management framework, and roles and responsibilities for all ICT-related functions have to be assigned clearly and documented. A CISO title is not a substitute for that assignment; it is one of the roles the assignment has to name.

So far this reads like a board problem. By Friday afternoon it is a middle-management problem. The management body will not patch containerd. It will not decide whether kube-apiserver audit logs are retained for 12 months or 18. It will not sit in the planning session where “ship the new tenant” competes with “we still have no evidence trail”. Those decisions are made by engineering managers, or they are not made at all.

The vacuum underneath the management body is where most programs die. Legal translates the article. Security writes a control statement. Nobody is accountable for the Kubernetes object that would make the statement true. The same vacuum appears in Cyber Resilience Act programs, which carry a harder clock: accountability at the top is theater unless someone in engineering owns the work that produces the evidence.

Put artifacts on the board, not regulation names

A regulation is a reason. It is not a ticket. “NIS2 Article 21” cannot be demoed at sprint review. “kube-apiserver audit policy shipped, logs shipped to the SIEM, retention documented” can be.

The artifact does not come from an engineer reading a directive and inventing a Kubernetes task. It comes from a traceability chain with a clean handover in the middle.

Traceability chain in two stages: scope, control question, and evidence definition owned by risk, legal, and security; artifact, sprint item, and recurring operation owned by the platform and application teams

Steps one to three are interpretation, and they belong to risk, legal, and security. Steps four to six are delivery, and they belong to the platform and application teams. The handover is the artifact, not the article number. A ticket that says “implement NIS2 Article 21(2)(d)” has skipped the middle of the chain and asks an engineer to act as the lawyer who decided the law applies.

The chain also closes. Step six is what produces the evidence that step three asked for, which means the output of delivery flows back to risk, audit, and the SOC as a matter of routine rather than as a fire drill three days before an inspection. An artifact that only exists because someone ran a script by hand once has not completed the chain.

The table below shows what the output of that chain looks like on a Kubernetes platform. The middle column is the control question the artifact answers. The provenance columns point at the provisions that make the artifact relevant; they are not a claim that the text commands that specific implementation. DORA references apply only where the financial entity and the ICT service fall within its scope.

Operational artifact	Control question it answers	NIS2 provenance	DORA provenance
Hardened image catalog and rebuild pipeline	Can we reduce known weaknesses in deployed software, and rebuild promptly when a material vulnerability is disclosed?	Article 21(2)(d) and (e): supply chain security, vulnerability handling and disclosure	Articles 8 and 9: ICT asset identification, protection and prevention
A secrets manager as the only credential path	Can we protect credentials, limit access to them, and show that secrets are not spread through code and cluster configuration?	Article 21(2)(i) and (j): access control, asset management, multi-factor authentication	Article 9: protection and prevention measures
Kubernetes audit logs and ingress logs in the SIEM	Can we detect suspicious activity, investigate an incident, and retain the evidence for the required period?	Article 21(2)(b): incident handling	Articles 10 and 17: detection capabilities, ICT-related incident management
Admission policy: restricted Pod Security Standards, no unpinned image tags	Can we prevent unsafe workload configurations from reaching production, while recording approved exceptions?	Article 21(2)(f) and (i): policies on effectiveness, access control, asset management	Article 9: protection and prevention measures
Incident runbook and a 24-hour fact pack	Can on-call staff establish what happened, its impact, and the facts needed for a timely escalation?	Articles 21(2)(b) and 23: incident handling, early warning within 24 hours and notification within 72 hours	Articles 17 and 19: incident management, reporting of major incidents
SBOM generated and retained in CI	Can we identify what software we shipped, and when, if a supplier, package, or vulnerability requires investigation?	Article 21(2)(d) and (e): supply chain security, vulnerability handling	Article 28: ICT third-party risk management

A ticket that names the artifact, links the control question, states the evidence, and has a person who can finish it is a concrete piece of work. It can be estimated, it can be demoed, and (the part that matters at audit time) it can be pointed at.

The cloud native projects that produce the evidence

None of these artifacts requires a compliance product, in the sense of a tool that generates evidence about a platform it does not otherwise touch. They are ordinary platform capabilities, and the open source ecosystem covers most of them. Not everything in the table below is a CNCF project (Sigstore and OpenBao sit under the OpenSSF and the Linux Foundation respectively), but all of it is open source and all of it can be run by the team that owns the cluster.

The table is illustrative rather than prescriptive. It answers “can this be built with open source”, which is not quite the question a platform lead faces.

Artifact	Open source building blocks	What the platform team actually operates
Hardened image catalog and rebuild pipeline *	Harbor for the registry, Sigstore for signing and verification, in-toto for attestations, an image scanner in CI	A base image that is rebuilt on a schedule and on advisory, published to one registry that workloads are allowed to pull from
Secrets as a managed service	OpenBao or an equivalent open source secrets manager as the store, External Secrets Operator to project secrets into the cluster, SPIFFE and SPIRE for workload identity, cert-manager for certificate lifecycle	One credential path: identity-based, short-lived, with no long-lived tokens in manifests or CI variables
Audit and detection evidence	Kubernetes audit policy upstream, Fluentd or Fluent Bit and OpenTelemetry for collection, Falco for runtime events, Prometheus for the operational signals	An audit policy that is not None, a shipping pipeline that survives node churn, and a retention setting someone has written down (the SIEM at the far end is usually bought; the collection path is yours)
Admission control and exceptions	Kyverno or OPA Gatekeeper, plus the upstream Pod Security Admission baseline	Policies in version control, enforced in production, with exceptions expressed as objects that expire rather than as Slack messages
Software inventory	SBOM generation in CI, stored as an attestation next to the artifact rather than in a wiki	An answer to “what was in the image we deployed on the 14th” that takes minutes, not days
Service ownership metadata	Backstage or an equivalent catalog	A machine-readable answer to “who is the engineering manager for this service”, which is what makes the RACI below real

* Buildable and realistic to operate are different claims, and the hardened image catalog is where they diverge most. The pieces are all open source, but the product is not the pipeline; it is the rebuild cadence, sustained across every base image and runtime you ship, in the week half the team is on holiday. Few platform teams are funded for that, and a catalog that goes stale is worse than no catalog, because by then a control statement says it is handled. Sourcing the base layer from someone who maintains it full time is a legitimate answer, and it does not remove the ownership question so much as change its shape: the provider enters third party scope under NIS2 Article 21(2)(d) and DORA Article 28, and supplier management needs a named owner exactly as the pipeline did.

None of this is unique to regulated industries. It is the same platform baseline most mature teams build anyway; NIS2 and DORA mostly change the deadline and the consequence of missing it. Open source removes the license barrier rather than the staffing one, so the build or source decision belongs in the roadmap for every row, not just the one carrying the asterisk.

Which is the real point of the table. A cluster can have Kyverno installed, Falco running, and SBOMs in CI and still fail an inspection, because nobody can say who decided what “acceptable” means or who is accountable for keeping it that way. Tooling is the easy half.

A RACI for a Kubernetes platform

Once the traceability chain exists, the RACI answers the remaining question: who gets the artifact over the line and keeps it true afterward?

Adjust the role names to your organization. Do not adjust the underlying idea, which is that security is rarely Accountable for the artifact. A secrets manager and a hardened image catalog are services the platform team runs, not appliances that security sprinkles on at the end.

Artifact	Responsible	Accountable	Consulted	Informed
Hardened image catalog and rebuild pipeline	Platform engineers	Platform engineering manager	Security (policy, CVE exceptions)	Application teams, risk
Secrets manager as the only credential path	Platform engineers	Platform engineering manager	Security, IAM	Application engineering managers
Kubernetes audit logs and ingress logs into the SIEM	Platform and SRE	Platform engineering manager	Security and SOC (what evidence is required)	Risk, CISO
Admission policy (restricted PSS, no :latest)	Platform engineers	Platform engineering manager	Security (exceptions)	Application teams
Incident clock: detect, page, assemble facts	On-call engineer	Engineering manager who owns that service	Security, legal, communications	Management body, CSIRT
Notification to the authority	Security or the nominated control function	CISO or nominated role	Legal, engineering manager of the affected system	Management body
SBOM in CI for what you actually ship	Engineers who build the artifact	Engineering manager of that pipeline	Security (format, retention)	Procurement, risk
Exception to run privileged or with hostNetwork	Requesting engineer	Requesting engineer’s manager	Security	Platform, risk

A RACI is a communication tool, not a governance artifact in its own right. If you publish the table and never open it during planning, you have produced stationery. Two rows get confused more often than the rest, and both are worth stating explicitly.

Detection is not notification

Incident detection and authority notification are different jobs on different clocks. NIS2 Article 23 expects an early warning to the CSIRT or competent authority within 24 hours of becoming aware of a significant incident, an incident notification within 72 hours, and a final report within a month. DORA sets its own reporting path and deadlines for major ICT-related incidents.

If you make the on-call engineer “own NIS2 reporting”, you will get panic or silence at 02:00. The engineer owns the facts: what broke, when, the blast radius, what has already been done. The nominated control function owns the notice to the authority. The engineering manager owns whether the runbook was rehearsed before the night it was needed.

That separation is also what makes the 24-hour clock survivable. An early warning is a factual submission, not a root cause analysis. The platform’s job is to make those facts easy to gather in minutes, which is exactly what audit logs and a rehearsed runbook are for.

Exceptions belong to the requesting engineering manager

Security can advise that CAP_SYS_ADMIN in a production namespace is a bad idea. Only the requesting team’s manager can decide that the product risk is worth it, write down the reason, and set an expiry.

If security is Accountable for exceptions, every exception becomes a negotiation with the person everyone is trying not to bother, and the real decision maker stays invisible. Making the requesting manager accountable puts the cost back where the benefit is, and gives the exception a natural expiry date: the manager who wanted it is the one who has to renew it.

Capacity is the part that gets skipped

The useful test in a planning session is simple: is the artifact in the next sprint, with a named engineering manager, or is it still a control statement on a security slide? A second, more uncomfortable question follows: who is currently holding work labeled “security” or “compliance” that should not be theirs? An admission policy the platform should own is one answer. A 40-page control mapping that belongs with risk is another.

Capacity is where these programs quietly fail. A platform team staffed for product delivery alone will treat every control as overtime. They will ship the hardened image catalog, or they will ship the new tenant environment. They will not ship both from the same three people unless the roadmap is quietly optimistic. Cutting the scope honestly is a better outcome than putting “DORA” on a slide and leaving engineers to find the hours in the evening.

Security still sets the bar, and that work does not go away: what “restricted” means in practice, which CVEs are acceptable exceptions, what a fact pack must contain, which logs the SOC will actually use. It reviews exceptions, runs or supports the notification path, and trains engineering managers until they can hold the conversation without forwarding every email to the CISO.

But Consulted has to mean “answers within the same sprint”, not “joins a working group”. If the security function cannot staff that responsiveness, the honest move is to shrink the control set rather than to keep a queue that delivery teams learn to avoid. Teams walk toward security when talking to it makes the work smaller and clearer. They walk around it when it does the opposite.

“We cannot evidence access to kube-apiserver for last week’s incident” is a useful sentence in week two of a program. The same sentence the week before an on-site inspection is a much more expensive one.

Look at the engineering manager first

The chain is short. The management body is accountable in the text. The engineering manager is accountable in the cluster. Security is the specialist you consult so that the work is correct. The engineer is responsible for the object that ships.

Staff only the management body and the security function, and you get policies and liability with no Kubernetes. Staff only the engineering manager and the engineer, and you get a fast platform you cannot explain to a supervisor. Both halves are load-bearing.

So the next time a regulation lands in the room, try not to look at security first. Look at the engineering manager who owns the system the requirement describes, and ask two questions: which artifact will you demo, and which sprint is it in? If there is no answer, the organization does not have a security gap. It has an ownership gap, and those are the expensive ones.

References

Regulation (EU) 2024/2847 (Cyber Resilience Act): https://eur-lex.europa.eu/eli/reg/2024/2847/oj

Directive (EU) 2022/2555 (NIS2): https://eur-lex.europa.eu/eli/dir/2022/2555/oj

Regulation (EU) 2022/2554 (DORA): https://eur-lex.europa.eu/eli/reg/2022/2554/oj

Don’t give AI agents root. Make them propose the next system state.

Wed, 07 Oct 2026 16:00:00 GMT

The uncomfortable question

In the past few months, I’ve been hearing different versions of the same question: under what conditions can we safely allow an autonomous system to operate a machine?

It comes up when my colleagues discuss how much autonomy we should give our agents. It comes up when people outside the software industry wonder how to secure their OpenClaw or similar installations. And it’s my own question when I wonder how much of my Kubernetes cluster my agents should control.

Let me be upfront: I don’t have the answer. As with security, the answer depends on the system we’re talking about. There is no one-size-fits-all answer in the cloud native ecosystem. But as someone directly involved in the development of an operating system, and who’s recently gotten heavily invested in software factories, I’d like to share what I think it should look like at this layer of the cake.

By software factory, I mean the machinery that turns a proposed change into a running artifact: source control, review, CI, tests, image builds, signing, release, and deployment.

Root is not an API

I know it feels amazing to give an agent root access to a machine and watch all the impressive things it can do. I recently migrated a home service from a MacBook to a Linux box without downtime, and all it took was giving the agent superuser access.

But just because it can be done doesn’t mean it should be done. Experimentation and production are very different environments, and both come with different expectations. If you give an agent root access and expect a well-written AGENTS.md file to prevent an unrecoverable mistake, you are fooling yourself. The system, however capable it may be, is still nondeterministic, so you cannot guarantee the outcome.

This is not simply about failures. APIs can contain bugs and fail too. It is about reproducibility: knowing what changed and tracking who or what changed it. We should not give agents unrestricted root access to production systems for the same reason we should not give humans unrestricted root access to them. Period.

Containers changed the boundary

In the traditional Linux paradigm, if an application needed something, such as a library, we installed it on the base system. The problem was that the application and the system became tightly coupled: changing the application also meant changing the machine underneath it.

Containers changed that boundary by packaging an application’s userspace dependencies with the application itself. They do not make the host irrelevant; containers still share its kernel and depend on its runtime. But they give us a much cleaner separation of concerns between the application and the system that runs it.

The host’s job can then become much narrower: boot reliably, provide the kernel and runtime capabilities the platform needs, and keep that platform running. The platform, in turn, takes care of the containerized applications.

Immutability is a means to an end

Immutable systems reinforce this boundary. That doesn’t mean the entire system is immutable; that would be useless. Depending on the implementation, selected parts of the filesystem may be mounted read-only or the base system may be treated as an image that is updated as a unit. If the base system needs to change, you define and build a new system image rather than modifying it package by package at runtime.

But preventing drift is not the end goal. The more important property is that the machine moves from one defined state to another. During runtime, an agent can inspect the system, while changes to the base system are expressed in a versioned definition and consumed as a new image. Runtime is no longer where the next state is improvised; it is where a state produced somewhere else is executed.

Will mistakes still slip through? Of course. They do in every software factory: lights on or lights off, agent or human. But when source, build and deployment provenance are retained, we can trace a problem back to the commit and image that introduced it. We have relied on this property in software delivery for a long time; the same principle can apply to pods and to the operating system itself.

At the time of writing, the cloud native ecosystem includes technologies that support image-based, immutable host lifecycles in different ways: bootc, Flatcar Container Linux and Kairos. This is probably the moment when I should acknowledge that I’m a Kairos maintainer. But I’m writing this piece as an Ambassador, so I will keep the argument at the architectural level rather than advocating for one implementation.

This is only the first boundary. An immutable system is not completely immutable, and controlled state transitions do not provide every guarantee an autonomous operator needs. What they give us is something a root shell does not: an explicit path from intent to a reviewed and reproducible system state.

Immutability is not a sandbox

But here comes the tricky part. Agent software does not normally come from the distribution. It may be installed through a language package manager such as npm, shipped as a standalone binary or container image, or integrated into another application you run.

This is why agent harnesses offer sandboxing: to put a stronger boundary around what the agent can do. But not every sandbox is a sealed environment. Local harnesses such as Claude Code or Codex may still be connected to the host through the filesystem, credentials, tools, sockets, network access or permissions granted during a session. The CNCF blog has already explored why sandboxing an agent is not sufficient and how agents can use capabilities we did not anticipate.

An immutable host still gives us an important property: installing, changing or upgrading the system packages that make up the base system is removed from the normal runtime lifecycle. Hadron, a minimal base OS developed by the Kairos team for building image-based systems, takes this further by not providing a package manager at all. The strength of that protection depends on the implementation; immutability alone does not protect the machine from a process with unrestricted host, kernel or disk access. The agent can also still affect every surface where it has write access. The mistake would be to assume that because the base system is protected, the rest of the machine is protected too.

The agent is a workload too

This is where the cloud native separation of concerns becomes useful again. The agent is a workload too, and each session can run in a short-lived execution environment created for the duration of its work and discarded afterwards.

The execution boundary should follow the threat model. A standard pod shares the host kernel and may be appropriate for a narrowly scoped agent, while an agent that runs untrusted code or needs broader capabilities may require a sandboxed pod or VM. The CNCF blog has examined whether a pod is the right deployment unit for an agent in greater depth. For this argument, the important point is that the agent does not need direct access to the host in order to observe it or propose how it should change.

Close the loop without opening the host

But if agents run in isolated workloads, why do we still need the immutable host? In my opinion, the most valuable promise of agent-led systems is that they can help systems self-heal and self-improve. An agent that understands how the system is running can diagnose issues, suggest fixes, improve performance and propose changes to the host system.

These are not the same kind of autonomy. Self-healing means choosing from responses we have already authorized, such as restarting, replacing or rolling back a workload. Self-improvement means proposing a system state that we have not authorized yet, so it must go through the software factory before it becomes real.

Recently, during exploratory QA of Kairos Trusted Boot images, an agent helped trace multi-minute boot stalls and firmware reset loops to incorrect PE section alignment in go-ukify, the library AuroraBoot uses to assemble UKIs. Sections were being aligned to 512-byte boundaries even though the systemd stub required 4 KB alignment. We fixed the library, released go-ukify v0.5.2, updated AuroraBoot and had the agent verify the corrected images, so Kairos and Hadron now pick up the fix.

What the agent should not do is apply those changes through root access or modify a mutable system in place. Starting from the question of autonomy, an immutable system is clearly a better model than a mutable one: the agent can inspect the current state, record its assessment and open a pull request proposing the next state. That proposal can follow the same process as a human-authored change: review, testing, a new image build, signing and release. Only then is the new state consumed by the system itself.

For this to be a real boundary, the agent’s identity must be allowed to propose a change without being allowed to approve or merge it, alter the pipeline, publish or sign the resulting image, or force the machine to consume it. Otherwise, we have only replaced root access with root access plus more steps.

This separation of authority is part of the broader CI/CD threat model covered in Shadow AI in CI/CD. Here, it is what turns an agent’s observation into a proposal for the next host state rather than another path to mutate the current one.

That closes the feedback loop without opening the host. The agent can contribute to the evolution of the system without being allowed to rewrite the system directly.

Trust the boundaries, not the model

Immutability is not sufficient by itself, and neither is workload isolation. Together with a software factory, however, they give us a layered model: protect the base system from runtime changes, isolate the agent according to its capabilities and threat model, and turn any proposed improvement into a reviewed and reproducible state transition.

We do not need to trust the model. We need to trust the boundaries around it and the process through which its proposals become system state.

CiliumCon is back at KubeCon + CloudNativeCon North America 2026

Tue, 06 Oct 2026 16:00:00 GMT

Mark your calendars. On Monday, November 9, CilliumCon kicks off at KubeCon + CloudNativeCon North America in Salt Lake City. It’s the eighth time the event has run, and this year’s agenda goes straight at the problems that show up when AI and GPU workloads push Kubernetes networking past what it was built for.

Why now?

AI and GPU workloads are stretching Kubernetes networking, and this year’s CiliumCon agenda covers the problems that start to appear at that scale.

Security is part of the story too. The same AI capabilities are changing the pace of vulnerability discovery, as models like Mythos have shown, and CVEs can now surface far faster than teams can patch them manually. That puts runtime security front and center, with Tetragon giving teams a way to detect and respond to threats as they happen.

Where it all began

CiliumCon began as a half-day co-located event at KubeCon + CloudNativeCon Europe 2023 in Amsterdam. It was created to give the growing Cilium community a dedicated space to connect, share production use cases, and align on technical roadmaps outside of the larger conference. It brings end users, maintainers, and contributors together to focus on production insights, eBPF enhancements, and community collaboration.

Why CiliumCon?

There’s no other event that goes as deep into Cilium, its sub-projects, and its eBPF foundation.

One stack, one foundation. The whole Cilium stack runs on eBPF, with Cilium handling networking, Hubble observability, and Tetragon runtime security. Sessions build on that shared base instead of treating networking as a general topic.
Built for scale. Cilium is the de facto CNI for the cloud native ecosystem, and as workload demand grows, users need to understand how to run it at scale and handle any new issues that come up.

What’s on the agenda

CiliumCon runs Monday, November 9, opening and closing with remarks from co-chairs Joe Stringer and Jordan Rife. In between, you’ll hear from end users and maintainers:

ESnet on IPv6-only ClusterMesh
OpenAI on debugging inference workloads with eBPF
Datadog on a fleet-wide netkit migration
Maintainer-led technical sessions on sidecarless mTLS and multi-cluster networking

Cloud native themes

CiliumCon lines up with two cloud native themes: Cloud Native AI and Security.

Cloud Native AI: Several sessions deal directly with networking and security constraints specific to AI and GPU workloads, from IPv6 addressing to RDMA policy enforcement to debugging inference traffic.

Security: The security sessions cover enforcement in the kernel, from sidecarless mTLS that authenticates pod traffic without app changes to Tetragon’s runtime policies. These matter more as vulnerabilities surface faster and AI clusters need the same isolation as any other workload.

What you’ll walk away with

You’ll leave with practical approaches to problems many teams haven’t hit yet, including:

Running Kubernetes on IPv6 only
What actually breaks during a large-scale CNI datapath migration
Enforcing network policy on RDMA traffic for AI clusters
Tracing packet loss in latency-sensitive workloads with eBPF

These takeaways come from real production use, so you’ll go home with real strategies you can put to work.

Who should come?

Platform and infrastructure engineers running Cilium, Hubble, or Tetragon in production. You’ll get the most out of it, especially if you’re weighing a CNI migration or networking changes to support AI workloads.
Engineers evaluating Cilium for the first time. Get an honest account of what adoption involves, plus tips and tricks from people who have already solved similar problems.
Contributors. Get face time with maintainers.

A working knowledge of Kubernetes networking and container orchestration is enough for most sessions. A few of the more advanced talks assume familiarity with specific Cilium features (ClusterMesh, netkit, existing NetworkPolicy behavior). Newcomers can get up to speed with the eBPF documentary, the Buzzing Across Space children’s guide to eBPF and its Cilium follow-up Buzzing Beyond Clouds, or Learning eBPF by Liz Rice.

Come meet the community

At CiliumCon, users get to tell maintainers what broke in production, and maintainers get to explain why it works the way it does, without waiting on a GitHub issue thread. It’s also an easy way for new contributors to meet maintainers in person, ask about a PR or a design decision, and put faces to the GitHub handles they’ve been working with.

Join us at CiliumCon on Monday morning, November 9, 2026, in Salt Lake City, as part of KubeCon + CloudNativeCon North America 2026.

BackstageCon comes to KubeCon + CloudNativeCon North America 2026 in Salt Lake City

Tue, 06 Oct 2026 16:00:00 GMT

BackstageCon takes place all day on Monday, November 9, 2026, the day before KubeCon + CloudNativeCon North America opens in Salt Lake City. We’re excited to bring together Backstage adopters, contributors, and maintainers to share how teams have used Backstage in practice, including the AI features the project has been shipping for the past year. ne project, one track, one full day with the people who actually run it.

AI is changing the game. Where does your portal fit?

AI is changing how teams build software. If you’re already running Backstage, you probably want to know where your developer portal fits now. If you’re just getting started, you want to make the most of it while AI becomes part of everyday work.

The Backstage project has been shipping AI features for the past year. BackstageCon is where you hear how teams have really used them and meet others working through the same problems.

Where it all began

The first BackstageCon took place in Detroit in October 2022, co-located with KubeCon + CloudNativeCon North America. It was the first official Backstage conference, and it started after the Backstage project approached the CNCF. It has returned regularly since.

The reason is simple. Backstage was growing fast, with more adopters and more contributors, but there was nowhere for the community to meet in person. BackstageCon gave adopters, contributors, and maintainers a place to connect and swap stories.

Why spend a whole day on Backstage?

The main KubeCon schedule has one or two Backstage talks. BackstageCon gives the project a full day, so you can go far deeper.

Real-world stories. Most sessions come from adopters talking about what they’ve done in production.
One track, one conversation. Users, contributors, and maintainers all see the same talks, so by the time the hallway chats start, everyone has plenty to talk about.
The bigger picture. Backstage usually sits on top of a Kubernetes and cloud native stack, and BackstageCon shows how platform teams make that stack usable for their developers.

What’s on the agenda

It’s a single track of breakout sessions and lightning talks, from opening remarks to closing remarks, with breaks along the way for networking. Sessions fall into four tracks:

AI agents and context in Backstage
Technical deep dives
Developer experience and workflows
Lessons learned from deploying Backstage and driving adoption

Every talk is labeled beginner, intermediate, or any level, so you can pick the sessions that match where you are with Backstage.

What you’ll walk away with

You’ll leave with ideas you can try in your own Backstage instance, taken from teams who have already done the work. Talks cover:

Bringing AI agents into Backstage while keeping people in the loop
Giving those agents the right identity and permissions
Automating catalog ownership
Keeping plugins current as the project changes

One session makes the case that the hardest parts of adoption are about people, not technology. You’ll also get a look at where Backstage is heading, including how one team used AI to modernize a Backstage instance that’s been running for a decade.

Who should come?

Platform teams running Backstage in production. You’ll probably get the most out of it. Many speakers are wrestling with the same problems at scale, like catalog ownership and adding AI to their portals.
Teams evaluating Backstage or early in adoption. Learn from organizations that are further along. Several sessions are aimed at beginners.
Contributors and plugin authors. See where the project is going and meet maintainers in person.

No prep is required. If you’re new to Backstage, it helps to skim the documentation first, especially the Software Catalog, Software Templates, and plugins, since many talks assume you know those. A lot of the schedule covers AI agents and MCP, so a basic understanding of the Model Context Protocol will help too.

Come meet the community

BackstageCon is where the Backstage community meets in person. You’ll hear how others use the project and share your own experience, whether you started last month or have run it for years. Attendees regularly leave with new ideas and with people they keep talking to long after the event. It’s also a great chance to meet maintainers and contributors, which makes getting involved much easier.

Join BackstageCon on November 9, 2026, in Salt Lake City, as part of KubeCon + CloudNativeCon North America 2026.

Kubernetes on Edge Day returns to KubeCon + CloudNativeCon North America 2026

Tue, 06 Oct 2026 16:00:00 GMT

Kubernetes on Edge Day returns to KubeCon + CloudNativeCon North America 2026 on November 9 in Salt Lake City, Utah, bringing together developers and adopters from across the cloud native ecosystem to share experiences and insights around building, improving, and operating edge infrastructure.

As hardware and software are distributed across hundreds or even thousands of locations, cloud native approaches to observability, loosely coupled systems, declarative APIs, and strong automation become increasingly relevant. Kubernetes is already an important part of the edge ecosystem, and Kubernetes on Edge Day creates a dedicated space for the community to explore the intersection of cloud native and edge computing.

From cloud native to the edge

Kubernetes on Edge Day first launched as a co-located event at KubeCon + CloudNativeCon Europe in Valencia, Spain, in 2022. Since then, the event has grown alongside an ecosystem in which Kubernetes is increasingly being considered and deployed beyond traditional data centers and public clouds.

The event was created to address the gap between traditional, data-center-centric cloud native approaches and the operational realities of edge computing.

Running Kubernetes across distributed and resource-constrained environments introduces challenges that can differ considerably from centralized cloud environments. Hardware and software may be spread across many locations, resources may be constrained, and connectivity cannot always be treated in the same way as it is in a conventional data center or public cloud environment.

Kubernetes on Edge Day provides a dedicated space for the community to discuss these challenges and share approaches to applying cloud native technologies in edge environments.

Why Kubernetes on Edge Day?

KubeCon + CloudNativeCon covers a broad spectrum of Kubernetes and cloud native technologies. Kubernetes on Edge Day focuses specifically on what happens when those technologies are applied at the edge.

What to expect

Kubernetes on Edge Day is a CNCF-hosted co-located event taking place on November 9, 2026, at KubeCon + CloudNativeCon North America in Salt Lake City.

The focus is on the intersection between Kubernetes, cloud native technologies, and edge computing, providing an opportunity for attendees to learn from the experiences of others working in distributed environments and to participate in the broader community conversation around cloud native at the edge.

Cloud native themes

Kubernetes on Edge Day intersects with several areas of cloud native computing. Observability and Security are particularly relevant themes for edge environments.

Observability is an important consideration when hardware and software are distributed across many locations. The cloud native approaches that have helped operators understand and manage distributed systems in the cloud are also relevant as infrastructure moves toward the edge.

Security is similarly important as cloud native workloads and infrastructure operate across increasingly distributed environments. Edge deployments introduce their own operational conditions, making security an important part of the wider discussion about how Kubernetes and cloud native technologies are used outside traditional data centers.

What can attendees take away?

Attendees can expect to leave with a clearer understanding of the technical and operational considerations involved in using Kubernetes and cloud native technologies at the edge.

The event provides an opportunity to learn from practitioners working with distributed infrastructure, understand the challenges they encounter, and explore how cloud native principles such as automation, declarative management, and observability can apply in edge environments.

It also gives attendees a broader view of how the Kubernetes and cloud native communities are approaching edge computing and where further collaboration and technical work may be needed.

Who should attend?

Kubernetes on Edge Day is relevant to developers and adopters interested in deploying Kubernetes and cloud native technologies at the edge, as well as practitioners already working with distributed infrastructure.

Platform engineers, infrastructure architects, operators, developers, and others responsible for environments outside traditional data centers or public clouds may find the discussions particularly useful.

The event can also provide useful context for newcomers who want to better understand the relationship between Kubernetes, cloud native technologies, and edge computing.

A foundational understanding of Kubernetes, containers, and distributed systems can help attendees follow more technical discussions, but the event also provides an opportunity to learn directly from members of the community working on edge infrastructure.

Strengthening the community

Kubernetes on Edge Day creates a space for people working across cloud native and edge computing to exchange knowledge and experiences.

Bringing developers, adopters, operators, and other practitioners together helps surface common challenges and provides opportunities to share approaches to building and improving edge infrastructure.

As Kubernetes continues to be used beyond traditional data centers and public clouds, this community exchange can help practitioners learn from one another and contribute to the ongoing evolution of cloud native technologies at the edge.

Join Kubernetes on Edge Day on November 9, 2026, in Salt Lake City, Utah, as part of KubeCon + CloudNativeCon North America 2026.

Meshery becomes a CNCF Incubating project

Tue, 06 Oct 2026 16:00:00 GMT

The CNCF Technical Oversight Committee (TOC) voted to accept Meshery as a CNCF incubating project.

Meshery is used by organizations spanning aerospace, telecommunications, networking, and enterprise software as a single cloud native management plane that gives engineering teams unified lifecycle, configuration, and performance control over their Kubernetes and cloud native environments instead of stitching together a separate toolchain for every mesh they run.

“Meshery was never about a single kind of infrastructure. It’s about how teams design, configure, and manage all of it, together. As AI generates infrastructure changes faster than any team can review them, the industry isn’t short on automation; it’s short on oversight. Meshery brings infrastructure as design to every engineer, making each change, whether written by a person or suggested by an AI, visible, reviewable, and governable before it reaches production. Incubation recognizes what this community has built in the open from its very first commit, and we’re just getting started.” – Lee Calcote, Meshery creator and maintainer

Meshery’s Milestones and Ecosystem Development

Meshery is an open source, highly extensible platform for modern infrastructure management: an internal developer platform that brings infrastructure as design and collaborative configuration management to teams designing, visualizing, and operating infrastructure across clouds and hundreds of technologies. It helps teams standardize configurations, enforce governance, and collaborate on infrastructure designs through visual models and reusable patterns. It also gives teams solid ground to standardize configurations and enforce governance as waves of AI-generated, potentially hallucinated YAML arrive. Originally created by Lee Calcote and the team at Layer5 in 2019, Meshery has grown into an extensible platform that unifies infrastructure models, design patterns, and operational controls into a single workspace.

Since Meshery was accepted as a CNCF sandbox project on June 22, 2021, it has become the fifth highest-velocity project in the CNCF, seeing a 350% increase in code commits over the past year (during the one-year period from July 1, 2025 to July 1, 2026) and announcing the general availability of Meshery v1.0. From its start in the CNCF sandbox, Meshery has more than 7,000 contributors and 1000 contributing organizations, with a 36.6% year-over-year increase in active contributors and a 10.9% increase in active organizations. It has earned almost 15,000 GitHub stars, has had over 6,800 forks, and has an LFX Insights software value of $393 million.

To keep pace with an ecosystem of nearly 400 integrations, Meshery expanded its organizational structure into two GitHub organizations: meshery for the core platform and meshery-extensions for community-maintained extensions and integrations, each with governance suited to its role. Meshery also became the first CNCF project to offer a contributor certification with the launch of the Certified Meshery Contributor (CMC), a free program that validates contributors’ expertise across Meshery’s major architectural domains.

“Meshery has grown from its early roots into a broad cloud native infrastructure design and operations platform. Through its extensible model-based architecture, it gives users a common way to design, deploy, and manage Kubernetes-based infrastructure while supporting collaboration across teams and environments. The TOC congratulates Meshery on reaching this milestone and looks forward to seeing the project continue to strengthen its extensibility model, governance, and adopter momentum.” — Karena Angell, CNCF TOC sponsor.

Meshery in the AI Era

AI-generated configurations can be syntactically valid yet semantically dangerous, and they arrive at machine speed. Meshery gives teams the oversight layer to govern that change:

Human oversight for AI-generated infrastructure: Changes authored by people or suggested by AI become visual, reviewable designs instead of black-box output, keeping humans in control of every step.
Infrastructure as Design: Infrastructure becomes a shared, visual, living artifact rather than disconnected text files, so teams can see the blast radius of a change before it ships.
Validate before you deploy: Teams can review and dry-run changes before deployment, catching misconfiguration and drift before it reaches production.
Visual reviews in GitOps workflows: Human-verifiable visual diffs bring infrastructure review into pull requests, bridging diagrams and code.
Agentic infrastructure management: A new Meshery MCP Server gives AI assistants governed, read-only access to the Meshery Registry and cluster state.
One model across tools: Existing Helm and Kustomize assets can be imported into a single live model, with Terraform support coming soon, giving people and AI agents the same structured view of infrastructure.
Certified human expertise: The CMC program recognizes the human expertise that keeps AI-assisted infrastructure management safe, auditable, and correct.

Under the hood, Meshery’s architecture pairs a core of Meshery Server, UI, CLI, Operator, and MeshSync with extension points, including Adapters, Providers, and Models, that reach well beyond Kubernetes to manage non-Kubernetes infrastructure too, including hundreds of AWS, Google Cloud, and Azure services.

Meshery’s Roadmap

Looking ahead, the Meshery community is focused on a few enhancements, including expanded multi-cluster and fleet management with fine-grained Kubernetes RBAC integration and native multi-cluster meshconfig support, agentic infrastructure management through a new Meshery MCP Server for AI-assisted, read-only access to the Registry and cluster state, broader performance management spanning distributed performance testing and adaptive load optimizers, and a more governed registry and workflow engine to support policy-driven configuration management.

For the latest updates, a detailed roadmap, and more on how to contribute, please visit meshery.io/ or the GitHub repository.

Main Components:

Meshery’s main components consist of the following:

Meshery Server: Central control plane that hosts the capability registry and coordinates deployments.
Meshery Database: File-backed cache for state about infrastructure, designs, models, and preferences.
Meshery UI: Web UI for visual design, topology maps, dashboards, and configuration.
Meshery CLI: Command-line tool to install/manage Meshery and invoke its functions.
Meshery Operator: Kubernetes operator that manages in-cluster components.
MeshSync: Continuous discovery/sync controller that keeps Meshery’s view aligned with actual cluster state.
Meshery Broker: NATS-based messaging layer for streaming events and data between cluster components and the server.
Extension Points:
- Meshery Adapters: Pluggable management modules that add technology-specific lifecycle, configuration, performance, governance, and identity capabilities.
- Meshery Providers: Identity and multi-tenant abstraction layer that supports local and remote providers for authentication, preferences, and environment management.
- Meshery Models & Catalog: Declarative infrastructure models and a catalog of reusable design patters/templates.
- Extension Ecosystem: 300+ integrations, AI adapters, visual tools, and other add-ons.

Notable Milestones

Nearly 15,000 GitHub stars and 6,800+ forks across two GitHub organizations
5th highest-velocity project of nearly 250 in the CNCF
4th largest codebase in the CNCF and 18th largest in the Linux Foundation
26,800+ pull requests (19,000+ merged)
179,000+ contributions from 55 countries
73,900+ comments and 28,300+ PR reviews
7,065 contributors

CNCF-Hosted Projects

The CNCF TOC provides technical leadership to the cloud native projects and community. It defines and maintains the foundation’s technical vision, approves new projects, and stewards them across maturity levels. The TOC also aligns projects within the overall ecosystem, sets cross-cutting standards and best practices, and works with end users to ensure long-term sustainability. As part of its principles, the TOC evaluates and supports projects as they meet requirements for incubation and continue progressing toward graduation.

As a CNCF-hosted project, Meshery is part of a neutral foundation aligned with its technical interests, as well as the larger Linux Foundation, which provides governance, marketing support, and community outreach. Meshery joins incubating technologies Artifact Hub, Backstage, Chaos Mesh, Cloud Custodian, Confidential Containers, Container Network Interface (CNI), Contour, Cortex, Emissary-ingress, Fluid, gRPC, HAMi, k8gb, Keycloak, KServe, Kubescape, KubeVela, KubeVirt, Lima, Litmus, Longhorn, metal3-io, Microcks, NATS, Notary Project, OpenCost, OpenFeature, OpenFGA, OpenYurt, Operator Framework, OpenKruise, Strimzi, Tekton, Thanos, Volcano, and wasmCloud.

For detailed maturity requirements at each level, please visit the CNCF Graduation Criteria.

The 2026-2028 CNCF ambassador cohort

Mon, 05 Oct 2026 16:00:00 GMT

This year we accepted 58 new ambassadors and said goodbye to 48. All ambassadors have given their spare time to the ecosystem in one way or another, whether they are contributing to a CNCF project, preparing a Kubernetes Community Day (KCD), or providing translations for the Cloud Native Glossary. There are many ways to contribute to the ecosystem, and these folks were accepted because they have been doing it for over two years, and offer diverse insight into the ecosystem.

This year we had over 600 submissions, and took one month to review. From current ambassador recommendations, to CNCF insights, to using data analysis tools to dig deep into insights. This process is competitive, and we appreciated having the happy problem of having to turn some good folks away.

On the other hand, we also appreciate a cohort that has been contributing as an ambassador for over 4 years, and is willing to pass the torch, so to speak. Some ambassadors have made it to the Technical Oversight Committee (TOC), and others are doing great in their careers and no longer need the uplift. Whatever the reasoning, thank you to those ambassadors not continuing on this year, for the selfless act of stepping back so others can rise.

Geographic insights

Check out the geographic insights into the new ambassadors, and the cohort as a whole!

New Ambassadors – where are they coming from?

How do these geolocations contribute to the total cohort of 312 ambassadors?

Europe still leads the pack with ambassadors, which is on par with CNCF Project contributions with North America coming in second. And while we are still trying to grow in Africa and Latin America, we understand that general open source awareness needs to become more prevalent in those regions alongside cloud native adoption.

What if your application didn’t make it?

Each term is two years, so ambassadors have the chance for reapplication. This year, we had 126 ambassadors reapply, and 123 were accepted back into the program, and 93 are continuing their two-year term (did not need to reapply). There were however, 48 ambassadors that did not reapply, whether they intended to step down, or simply missed applications being open for three months. In the latter case, those folks are either not coming to monthly syncs, or paying attention to the communications we send them (monthly syncs, Slack messages, and emails).

We understand how frustrating it might be if you have applied more than once and have yet to get accepted. What we can say is, don’t give up! We have a new cohort each year, and while it’s not big, there are constantly folks phasing out and phasing in. Keep engaging, sharing your passion, and building up your local communities. Cloud native thrives because of dedicated contributors like you, and we can’t wait to see your continued impact!

Connect with an ambassador near you.

Kubelet watches inodes. Just not until it’s an emergency.

Sun, 04 Oct 2026 16:00:00 GMT

A worker node pages with NodeFilesystemFilesFillingUp. It’s on course to run out of inodes. You check the disk first, because that’s what everyone checks first, and it reads 83% used: 103G used, 22G available. Then you check inodes: 67% used, 11M used, 5.3M free.

The disk is the fuller of the two, and the inode table is the one that paged. That’s not a contradiction. NodeFilesystemFilesFillingUp is a trajectory alert built on predict_linear, so it fires on the slope, not on the level. Nothing was wrong with 67%. What was wrong was how fast it was getting there, and the fact that the only mechanism in kubelet that would have intervened is the one that evicts pods.

Two notes on what follows. I no longer have access to that cluster, so I can’t paste the original df, du or tune2fs captures. Where a number comes from the incident record I say so, and where I’ve reconstructed something from arithmetic I show the arithmetic. The ext4 demonstrations further down you can run yourself in about two seconds.

If you want to compare your own node before reading on:

df -h / && df -i /

What kubelet actually watches

Kubelet’s defence against images filling a node is image garbage collection. Once the image filesystem passes imageGCHighThresholdPercent (85 by default), it deletes unused images, oldest first, until usage drops to imageGCLowThresholdPercent (80).

Both of those are percentages of bytes. Image GC never looks at file counts.

Kubelet isn’t blind to inodes, though, and it’s worth being exact here, because the detail changes the conclusion. On Linux the default hard eviction set is five signals (pkg/kubelet/eviction/defaults_linux.go, quoted here at tag v1.37.0):

var DefaultEvictionHard = map[string]string{ "memory.available": "100Mi", "nodefs.available": "10%", "nodefs.inodesFree": "5%", "imagefs.available": "15%", "imagefs.inodesFree": "5%", }

Both filesystems get an inode threshold, and both sit at 5% free. That’s a Linux-only file. The two next to it are shorter: defaults_windows.go and defaults_others.go each carry three signals, memory.available, nodefs.available and imagefs.available, with no inode thresholds at all. On Linux the coverage is there. Check it against whatever version you’re running, because these files do change.

So kubelet will act on inodes. The question is when. 5% free means 95% used, and that’s the hard eviction floor: the point where kubelet reclaims what it can and starts evicting pods to save the node. Image garbage collection, the mechanism that runs quietly in the background before anything is on fire, triggers at 85% and stops at 80%, and both of those are bytes.

Count the mechanisms watching each resource. Bytes get two: background image GC at 85%, and hard eviction at 10% free on nodefs or 15% free on imagefs. Inodes get one, and it’s the eviction floor. There is no early trigger on inodes, so the first thing that happens when you run low is also the most disruptive thing that can happen.

Why small files hit a wall the disk doesn’t

Quick question before the numbers. Fill a filesystem with nothing but one-byte files. What percentage of the disk is used by the time you run out of inodes? Take a guess.

ext4 decides how many inodes a filesystem gets at creation time, and that number never changes afterwards. It comes from inode_ratio: one inode per N bytes of capacity. The stock /etc/mke2fs.conf uses 16384. And any file that isn’t empty takes at least one 4 KiB block, no matter how small it is.

You can see the ratio on a throwaway image file. No root needed:

$ truncate -s 1G img && mkfs.ext4 -q -F -i 16384 img $ tune2fs -l img | grep -E '^(Inode count|Block count|Block size):' Inode count: 65536 Block count: 262144 Block size: 4096

262,144 blocks × 4,096 bytes ÷ 16,384 is 65,536 inodes. If every file uses one block, all 65,536 inodes are gone after 256 MiB. That’s 25% of the disk. If you guessed higher, most people do.

Watch it fail

You don’t have to take the arithmetic on trust. mkfs.ext4 -d builds a filesystem straight from a directory, so you can pack small files into an image without mounting anything and without root:

mkdir files
for i in $(seq 1 4000); do printf x > files/f$i; done
truncate -s 64M img
mkfs.ext4 -q -F -i 16384 -d files img
dumpe2fs -h img 2>/dev/null | grep -E ‘^(Inode count|Free inodes|Block count|Free blocks):’

Inode count: 4096
Block count: 16384
Free blocks: 11040
Free inodes: 84

97.9% of inodes used, 32.6% of blocks. The files themselves account for only 24.4 of those 32.6 points: 4,000 files at one 4 KiB block each is 15.6 MiB, and 15.6 MiB of 64 MiB is 24.4%. The remaining 8.2 points are the journal and filesystem metadata.

Add 200 more files and it stops being theoretical:

$ for i in $(seq 4001 4200); do printf x > files/f$i; done $ truncate -s 64M img2 && mkfs.ext4 -q -F -i 16384 -d files img2 mkfs.ext4: Could not allocate inode in ext2 filesystem while populating file system

Yes, it says ext2. That’s the underlying library’s error string, not a mistake. The image that worked had 84 inodes left and two thirds of its blocks free. Somewhere around file 4,085 the inodes run out, and from there the free blocks stop mattering. On e2fsprogs 1.47.2 the whole sequence runs in under two seconds.

Reverse-engineering the node’s mkfs parameter

Back to the real node, and this part is arithmetic rather than a capture, because I can’t re-run anything on that cluster.

11M inodes used plus 5.3M free is about 16.3 million inodes in total. The filesystem is roughly 124-133 GiB, working from 103G used plus 22G available and allowing for ext4’s reserved blocks. At the stock 16,384 bytes per inode, a filesystem that size only gets 8.1-8.7 million inodes. This one already had 11 million in use, which is more than the stock ratio can even allocate. So it wasn’t formatted at 16384.

Divide the size by the inode count and you get roughly 7,500-9,000 bytes per inode, which lines up with 8192. At that ratio the all-small-files case exhausts inodes at 50% of the disk instead of 25%. I can’t confirm it directly, but the arithmetic constrains it to about 8192, and on your own nodes tune2fs -l on the root device settles it in one command.

A real node is never all small files. Layer blobs and logs are big, which is why disk was the higher number here. But image GC was waiting for 85%, the file count was climbing at its own pace, and nothing in kubelet compares one against the other.

Finding the millions of files

Knowing the inodes are going doesn’t tell you where they went. This does:

du --inodes -xS /var | sort -rh | head -n 20

Both flags earn their place. -x stops du at filesystem boundaries, so it doesn’t wander into the tmpfs and volume mounts kubelet sets up under /var/lib/kubelet/pods. -S counts each directory on its own instead of adding in its subdirectories. Leave it off and /var, /var/lib and every other parent sit at the top of the list, above the directory you’re actually looking for.

On this node the top of that list wasn’t logs and it wasn’t pod volumes. It was containerd’s snapshot store, /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/, where each numbered directory is one unpacked image layer. Several of those snapshots held the same path: app/node_modules/@mui/icons-material, at 21,553 files each.

That’s one package directory out of one image. The image as a whole carried more than 40,000 files: the icon package is over half of it on its own, and the remainder is the rest of the dependency tree, the application source, and the build output, all of which shipped in the same image. Multiply 40,000 by every version a node is still holding, and millions of inodes stop looking mysterious.

It’s 21,553 per snapshot, not across all of them. containerd’s content store dedupes compressed layers by digest, but the overlayfs snapshotter unpacks each distinct layer into its own directory and shares nothing between them. Two builds that put an identical node_modules into layers with different digests leave two full copies on disk.

node_modules is the well-known offender, but it isn’t the only one. Any image that ships a dependency tree full of small files behaves the same way, whether that’s a Python virtualenv, vendored PHP packages or a pile of Ruby gems.

The problem was the image, not the cluster

Nothing on the cluster was set up wrong. Kubelet, containerd and the node image were all running on defaults.

The image was a single-stage build, so everything the build needed ended up in what shipped: the source, the build output, and all of node_modules, dev dependencies included, since npm install pulls those in by default. A single-stage build like that usually looks something like this:

FROM node:20 WORKDIR /app COPY package*.json ./ RUN npm install COPY . . RUN npm run build

Now look at that COPY . . and ask whether the repo has a .dockerignore. If node_modules isn’t excluded, COPY . . puts a second copy of it into a layer that changes on every single commit. That’s the fastest way to multiply snapshots, and it’s easy to miss because the build still works.

Even with a .dockerignore, a CI pipeline that builds without a warm layer cache reruns npm install every time. That produces a new layer digest even when package.json hasn’t changed, because the files inside the layer carry new timestamps. Every new digest becomes a new snapshot on every node that pulls it. containerd keeps those snapshots until the image is removed, and kubelet only removes unused images when image GC runs, which brings you straight back to that 85% byte threshold.

It went unnoticed because the number most people watch, disk usage, never crossed the line.

Fixing it

The first fix belongs in the build: a multi-stage Dockerfile. One stage installs dependencies and builds. The final image is a small web server that only receives the compiled output.

FROM node:20 AS build WORKDIR /app COPY package*.json ./ RUN npm install COPY . . RUN npm run build FROM nginx:alpine COPY --from=build /app/build /usr/share/nginx/html

That should take the deployed image from more than 40,000 files down to the compiled assets plus the base image, which for a React build is typically a few dozen files rather than tens of thousands. I haven’t measured both images side by side, so treat that as the expected shape rather than a benchmark. If you try it, docker run –rm <image> find / -xdev -type f | wc -l on each will give you the real before-and-after.

The second fix is monitoring, and credit where it’s due: the upstream alert did its job here. NodeFilesystemFilesFillingUp ships with the node_exporter mixin, and it’s what caught this at 67%. It’s a prediction, though, and predictions go quiet when growth pauses for a while. I’d add a flat threshold alongside it that fires well before kubelet’s 95% eviction point:

1 - node_filesystem_files_free{mountpoint="/"} / node_filesystem_files{mountpoint="/"} > 0.80

Give it for: 30m so a burst of image pulls during a rollout doesn’t page anyone. The 80% line is a judgement call; pick whatever leaves you enough room to act before 95%. You won’t find anything like it tied to image garbage collection upstream, and that’s the distinction worth holding on to: image GC’s thresholds are percentages of bytes, and the inode thresholds kubelet does have belong to eviction. They’re separate mechanisms, and only one of them runs early.

If you run Prometheus, this one is worth having on a dashboard. It subtracts disk usage from inode usage per node, so anything near the top is burning through inodes faster than bytes, and those are the nodes where the byte thresholds will fail you:

sort_desc( (1 - node_filesystem_files_free{mountpoint="/"} / node_filesystem_files{mountpoint="/"}) - (1 - node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) )

Adjust the mountpoint label if your node_exporter reports the root filesystem differently.

Neither fix does anything about what’s already sitting on your nodes. Old snapshots stay until their images go. crictl rmi –prune removes every image no container is currently using, and those images simply get pulled again the next time something needs them. You’ll also see crictl rm –all suggested for this kind of cleanup. It clears every stopped container and takes kubectl logs –previous with it for anything that restarted recently. Fine in an emergency. I wouldn’t put it in a cron job.

If you only remember one thing from this: kubelet’s background cleanup runs on bytes, so a workload that’s heavy on files rather than bytes gets no early intervention at all. Something does eventually fire, at 5% inodes free, and it fires by evicting pods. That’s the emergency brake, not the fix.

Check your own cluster in five minutes

Step one. Run the fleet query above and look at anything positive.

Step two. On the worst node, find the directories holding the most files:

du --inodes -xS /var/lib/containerd | sort -rh | head -n 20

Step three. If node_modules shows up under the snapshot store, list the images on that node:

crictl images

Then go and look at how those images are built.

Reproduce it

The inode ratio. No root needed.

truncate -s 1G img16 && mkfs.ext4 -q -F -i 16384 img16
truncate -s 1G img8 && mkfs.ext4 -q -F -i 8192 img8
tune2fs -l img16 | grep -E ‘^(Inode count|Block count):’
tune2fs -l img8 | grep -E ‘^(Inode count|Block count):’

You’ll get 65,536 and 131,072 inodes, both on 262,144 blocks.

Running out of inodes with most of the disk free. No root needed.

mkdir files && for i in $(seq 1 4200); do printf x > files/f$i; done truncate -s 64M img && mkfs.ext4 -q -F -i 16384 -d files img

That fails with Could not allocate inode. Drop the loop to 4,000 files and it succeeds with 84 inodes free and two thirds of the blocks unused.

Snapshot duplication on a node. You’ll need a registry and a cluster you can deploy to.

docker build --no-cache -t <registry>/app:v1 . docker build --no-cache -t <registry>/app:v2 . docker push <registry>/app:v1 && docker push <registry>/app:v2 # deploy v1, then v2, onto the same node, then on that node: sudo du --inodes -xS /var/lib/containerd | sort -rh | head -n 20

You should see two node_modules/@mui/icons-material directories with the same file count under two different snapshot IDs.

Originally published on Dev.to on 17 September 2026 at https://dev.to/onkar_s/kubelet-watches-inodes-just-not-until-its-an-emergency-351c. I am the author and grant permission to republish under CC-BY. Please set the Dev.to URL as the canonical link.

KubeCon + CloudNativeCon North America 2026: Join the cloud native community at OpenTofu Day

Thu, 01 Oct 2026 16:00:00 GMT

Most of KubeCon + CloudNativeCon assumes the cluster already exists. OpenTofu Day covers everything that has to be provisioned before and around it: cloud accounts, networking, managed services, and the clusters themselves.

On November 9, OpenTofu Day returns to Salt Lake City as a co-located event at KubeCon + CloudNativeCon North America 2026. It is a single-track day. OpenTofu’s core maintainers are in the room from start to finish, and teams running OpenTofu in production share what they’ve learned.

Why now

Licensing changes, consolidation, and shifting commercial models have led many organizations to reexamine the tooling their infrastructure depends on. Many now prefer projects governed by their communities. OpenTofu has grown from a fork into a CNCF Sandbox project with more than 10 million downloads as of October 2026. Recent releases have added capabilities of its own, including client-side state encryption, for_each on provider configurations, and OCI registry support.

For most teams, switching the CLI is the easy part: OpenTofu is designed as a drop-in replacement for Terraform. The bigger decision is what happens to everything around it, meaning the pipelines, state management, policy, and access controls through which infrastructure changes are made. Platform engineers generally choose between two paths. They can keep their existing workflows and swap the engine underneath with minimal disruption, or they can use the migration as an opportunity to re-platform and redesign how infrastructure changes are delivered. Each path has real trade-offs in risk, effort, and long-term flexibility, and the right choice depends on the organization.

Hearing from teams that have taken each path is one of the most useful things OpenTofu Day offers, and it is where that experience gets shared openly.

Back in Salt Lake City

OpenTofu Day debuted as a CNCF-hosted co-located event at KubeCon + CloudNativeCon Europe 2024 in Paris, about two months after the project’s first stable release. It has been held at every KubeCon + CloudNativeCon Europe and North America since then, in Salt Lake City, London, Atlanta, and Amsterdam, often to near-capacity rooms.

The event began in the months after Terraform’s 2023 license change, when the community built around it had no vendor-neutral place to meet. That is still its purpose. Maintainers and production users meet face to face, migration lessons are shared openly, and the people running the project at scale help shape its roadmap.

“What’s changed since OpenTofu Day Paris are the questions. In 2024 people asked whether they should migrate, and how. Now they ask how others have scaled it, instrumented it, and built platforms on it.” — Sebastian Stadil, OpenTofu Day Chair

How the day runs

The day opens with a state-of-the-project update from the core team on releases, growth, and roadmap. Next is an Ask the Devs panel, where attendees put questions directly to the maintainers. The rest of the program is practitioner talks on migrating from Terraform, performance and instrumentation at scale, and running OpenTofu inside platform engineering teams, including innersource adoption models in larger organizations. Breaks are built in for hallway conversations, which is where many of the day’s most useful exchanges happen.

Who should attend, and what they’ll take away

The day is built for DevOps engineers, platform engineers, and infrastructure leaders. That especially includes teams running Terraform in production that are evaluating or planning a migration. They will leave with a clear picture of what migration involves, including the trade-offs between keeping existing workflows and re-platforming, from teams that have already done it.

Existing OpenTofu users will learn how others use recent capabilities and operate at scale. Platform teams building internal developer platforms will find the sessions on provisioning and governance directly applicable. Contributors and prospective contributors get a full day with the core team and a clear sense of where the project is heading and how to help.

No preparation is required. The sessions assume familiarity with OpenTofu, Terraform, or a similar tool. Attendees who arrive with questions about their own infrastructure will get the most out of the day.

A community in one room

OpenTofu is built and governed in the open. Since its founding it has been backed by a community of individual contributors and companies including Scalr, Spacelift, env0, Gruntwork, and Harness. OpenTofu Day is the one time each cycle that its maintainers, contributors, and users from around the world are in the same place. Users can bring production problems straight to the people writing the code, maintainers hear what the community needs next, and newcomers find a way in.

Join us

OpenTofu Day takes place November 9, 2026, in Salt Lake City, Utah, as part of KubeCon + CloudNativeCon North America 2026.

Register Now

KubeCon + CloudNativeCon North America 2026: From user to contributor to maintainer

Wed, 30 Sep 2026 16:00:00 GMT

You do not need “maintainer” in your job title to start following the maintainer journey at KubeCon + CloudNativeCon North America 2026.

You might be an SRE who has spent years operating a CNCF project in production.

An application developer who found and fixed a bug.

An infrastructure engineer who has opened issues, improved documentation, or built something your upstream community could use.

KubeCon + CloudNativeCon gives attendees several ways to move closer to the open source communities behind the technology, whether you are already a maintainer or are making your first contribution. Register Today!

Why maintainers and future contributors should attend

For existing maintainers, KubeCon + CloudNativeCon creates rare face-to-face time with contributors, users, and neighboring project communities. Those conversations can surface real-world feedback, new contributors, shared technical challenges, and opportunities to collaborate across the ecosystem.

For attendees who are not maintainers yet, the event provides multiple low-barrier ways to get closer to a project. Maintainer Track sessions, ContribFests, Project Lightning Talks, and the Project Pavilion can help turn “I use this project” into “I know how to contribute to it.”

Come with a community challenge or a curiosity

If you already maintain a project, think about the challenge that is hardest to solve asynchronously: contributor onboarding, release processes, project governance, documentation, adoption feedback, or collaboration across projects. Face-to-face time can make those conversations much easier.

If you are newer to contribution, your questions may be simpler:

Which project community matches the technologies or problems I care about?
Where could someone with my skills make a useful first contribution?
How do contributors move from a first issue or pull request into deeper project involvement?
What do maintainers wish more users understood about participating in their projects?

Already a maintainer? Start Sunday

For eligible CNCF project maintainers, the CNCF Maintainer Summit takes place on Sunday, November 8 before the main conference.

The Summit provides a dedicated space for the people behind CNCF projects to meet face-to-face, share best practices, discuss contribution processes, and work through common challenges across project communities.

Separate registration and eligibility requirements apply, so attendees should review those requirements ahead of time.

Monday: See what projects are building

On Monday, Project Lightning Talks give CNCF projects five minutes each to share recent updates, new features, and other highlights.

Unlike the CNCF-hosted co-located events, Project Lightning Talks are available with any in-person KubeCon + CloudNativeCon pass.

For someone exploring the ecosystem, this is a fast way to discover projects you may not already know. For existing contributors and maintainers, it is a chance to see what neighboring communities are working on.

Tuesday through Thursday: Go behind the project

The Maintainer Track is where attendees can hear directly from CNCF project maintainers about new developments, technical challenges, implementation considerations, community priorities, and project roadmaps. Sessions to consider include:

If there is a project your organization depends on, this is an opportunity to hear from and talk with the people building it.

Make your first contribution

ContribFests turn that conversation into hands-on participation.

Attendees can work alongside project maintainers and contributors to tackle real issues, learn contribution workflows, understand project architecture, and begin participating in a community.

This year’s program includes opportunities with communities including:

Some sessions explicitly welcome first-time contributors.

You do not have to arrive knowing exactly how you want to contribute. Showing up curious is enough to start.

Visit the Project Pavilion

The Project Pavilion is open during the main conference in the Solutions Showcase and provides another easy way to connect directly with project communities.

Stop by a project kiosk. Ask maintainers what they are working on. Find out where they need help. Discover a project you did not know existed.

Sometimes the path into open source starts with a pull request. Sometimes it starts with a hallway conversation.

Don’t schedule every minute

For maintainers and future contributors, some of the most valuable parts of the week may never appear on the formal schedule. A conversation at a project kiosk can turn into end-user feedback. A ContribFest introduction can become a first pull request. A hallway discussion can connect two projects working through the same problem.

Use the scheduled programming as a starting point, but leave room for the community around it. Those relationships are often what continue long after the conference ends.

What you can take back to your project or community

For maintainers, the value can be practical: new contributor relationships, direct end-user feedback, ideas borrowed from other project communities, and time to solve problems that are difficult to work through in issues and chat.

For future contributors, you may leave with:

A clearer understanding of where your skills fit in the CNCF ecosystem
A first contribution, issue, or community connection to continue after the event
A better understanding of project roadmaps, governance, and contributor pathways
Relationships with maintainers and contributors who can help turn curiosity into ongoing participation

Maintainers can come from anywhere

That is why the Maintainer Journey overlaps every other journey in this series.

SREs bring production experience. Application developers understand how projects feel to the people building with them. Infrastructure engineers understand what happens when those projects have to operate at scale.

Open source communities need all of those perspectives.

Whether you already maintain a CNCF project, have made a few contributions, or are simply wondering how to get involved, KubeCon + CloudNativeCon gives you a place to take the next step.

Join us in Salt Lake City and find your place in the cloud native community.

Don’t forget to register for KubeCon + CloudNativeCon North America 2026.

KubeCon + CloudNativeCon North America 2026: Build your infrastructure engineer journey

Wed, 30 Sep 2026 16:00:00 GMT

Infrastructure engineers sit at one of the busiest intersections in cloud native.

Kubernetes clusters need to scale. Networks need to connect them. Storage needs to follow workloads. Platforms need to make infrastructure usable. Security needs to be built in. And increasingly, infrastructure teams are also being asked to support GPU-intensive, AI, and agentic workloads.

That means your KubeCon + CloudNativeCon North America 2026 journey is unlikely to stay inside one track.

Expect to move between Platform Engineering, Operations + Performance, Connectivity, Data Processing + Storage, Security, AI Infrastructure, and Maintainer Track sessions throughout the week. View the tracks + schedule.

Why infrastructure engineers should attend

Infrastructure teams are being asked to support more types of workloads, more clusters, more automation, and increasingly GPU and AI infrastructure without giving up reliability, security, or cost control. KubeCon + CloudNativeCon brings those challenges together across platform engineering, networking, storage, operations, security, and AI infrastructure.

For infrastructure engineers, the value is the chance to compare architectures and operating models with peers, hear how other organizations are solving production problems at scale, and talk directly with the maintainers building the projects underneath their platforms.

Come with an architecture decision in mind

Infrastructure engineers are rarely evaluating technology in the abstract. There is usually a platform decision, scaling challenge, migration, cost constraint, or new workload driving the conversation. Bring one of those decisions with you and use the week to pressure-test it.

Questions worth bringing include:

Where is your current platform reaching its limits?
Which decisions around networking, storage, identity, or multi-cluster operations are becoming harder to manage?
How will GPU-intensive, inference, or agentic workloads change the infrastructure you already operate?
What should your platform automate, and what should remain visible to the teams using it?

Use Monday to pick a specialty

If you want to attend Monday’s CNCF-hosted co-located events, choose an All-Access Pass; the KubeCon + CloudNativeCon Only Pass does not include them.

Infrastructure engineers have several strong options among Monday’s CNCF-hosted co-located events.

Platform Engineering Day focuses on the real-world challenges of building and scaling internal developer platforms.

CiliumCon goes deeper into cloud native networking, observability, and security using Cilium, Hubble, Tetragon, and eBPF.

OpenTofu Day is a natural destination for infrastructure-as-code practitioners.

And FluxCon provides another path for teams operating GitOps-based infrastructure and continuous delivery environments.

Your challenge may be choosing just one.

Follow the infrastructure lifecycle

During the main conference, look for sessions that connect architecture decisions with the reality of operating infrastructure at scale. Consider adding sessions such as:

Together they touch scaling, GitOps, networking, GPU infrastructure, multi-cluster environments, access management, platform reliability, and production operations.

Find the projects underneath your platform

Infrastructure engineers may want to make time for communities around projects such as Cilium, Flux, Argo, OpenTofu, SPIFFE/SPIRE, Kueue, Dragonfly, etcd, KubeVirt, and others connected to their stack.

The Project Pavilion is an especially useful stop.

Instead of evaluating a technology through documentation alone, you can talk directly with maintainers about architecture choices, deployment patterns, upcoming features, and the problems you are seeing in production.

Follow AI infrastructure without abandoning the fundamentals

AI infrastructure is increasingly visible throughout this year’s schedule, from GPU scheduling and inference to networking, storage, observability, and security.

For infrastructure engineers, that does not necessarily mean learning an entirely new stack.

Many of the same questions still matter: Where does the workload run? How does it scale? How is it secured? How is it observed? What happens when something fails?

Use the AI-focused sessions to understand how familiar infrastructure principles are being applied to a new generation of workloads.

Don’t schedule every minute

Architecture tradeoffs rarely fit neatly into a 35-minute session. Being at KubeCon + CloudNativeCon gives you the chance to continue those conversations with speakers, maintainers, and engineers who have already made similar decisions in production.

Leave time to visit the Project Pavilion and Solutions Showcase, compare implementation details, and ask the questions that do not fit on a slide: What broke? What did you change? What would you do differently at twice the scale?

What you can take back to your team

The value of the week is the ability to compare architecture choices with people who have already tested them in production, including the maintainers building the underlying projects.

You may leave with:

Reference points for platform, networking, storage, access, and multi-cluster architecture decisions
A clearer view of the tradeoffs behind GitOps, autoscaling, infrastructure as code, and platform automation
Ideas for supporting AI and GPU workloads without abandoning core reliability and security practices
Project roadmap context and peer connections that can inform your next design review or migration

Where does the infrastructure journey go next?

If reliability and production operations are the biggest part of your role, continue with the SRE Journey.

If you spend your time building platforms specifically to improve how developers work, the Application Developer Journey can help you see your platform from the other side.

And if you want to influence the infrastructure projects your organization depends on, explore the Maintainer Journey.

Join us in Salt Lake City and start planning your path through KubeCon + CloudNativeCon North America.

Don’t forget to register for KubeCon + CloudNativeCon North America 2026.

KubeCon + CloudNativeCon North America 2026: Build your application developer journey

Wed, 30 Sep 2026 16:00:00 GMT

Cloud native development increasingly means thinking beyond the application itself.

How will it be built? How will it be deployed? How much infrastructure should developers need to understand? Where should platforms remove complexity? And what changes when AI agents become part of the development workflow?

Application developers will find those questions throughout KubeCon + CloudNativeCon North America 2026.

The Application Development + Delivery track is an obvious starting point, but your path may also take you through Platform Engineering, Security, Observability, AI Inference + Agentic, Maintainer Track sessions, and CNCF project communities. View the tracks + schedule.

Why application developers should attend

Cloud native is increasingly shaping what happens before and after code is written. KubeCon + CloudNativeCon gives application developers a chance to see how developer platforms, delivery workflows, observability, security, and AI-assisted tooling are changing the way applications move from a laptop into production.

It is also a chance to meet the communities building the tools developers use every day, understand the infrastructure decisions happening behind the abstractions, and bring a developer perspective into conversations about what the next generation of cloud native platforms should look like.

Come with your biggest developer-friction point

You do not need to arrive with a perfectly defined cloud native strategy. Start with the part of building and shipping software that feels harder than it should, then use the conference to explore how other teams are approaching it.

Ask yourself:

Where does your inner development loop slow down?
How much infrastructure knowledge do developers on your team need just to ship safely?
Where do platform abstractions help, and where do they create new friction?
What changes when AI-assisted development or agentic workflows become part of the application lifecycle?

Choose your Monday based on what you want to build

If you want to attend Monday’s CNCF-hosted co-located events, choose an All-Access Pass; the KubeCon + CloudNativeCon Only Pass does not include them.

For developers focused on developer experience and internal tooling, BackstageCon offers a full day centered on the open source framework for building developer portals.

If AI-assisted development is becoming part of your work, Agentics Day: MCP + Agents explores real-world implementations and emerging patterns around agents, tools, data, and Model Context Protocol.

And if you spend a lot of time thinking about the relationship between developers and the platforms beneath them, Platform Engineering Day provides another path into developer experience, self-service, and internal developer platforms.

There is no single correct choice. Start with the problem closest to your work.

Follow the application from laptop to production

Across the main conference, developers can explore everything from application architecture and deployment to feature delivery and AI-assisted workflows. Sessions to consider include:

The common thread is not simply Kubernetes. It is how developers can build, ship, test, and operate software more effectively using cloud native technologies.

Meet the projects shaping the developer experience

Build time into your agenda to explore projects including Backstage, Argo, OpenFeature, OpenTelemetry, Podman, and other projects connected to the way your applications are developed and delivered.

Maintainer Track sessions can help you understand where projects are heading, while the Project Pavilion gives you an opportunity to ask questions directly.

And if you are interested in doing more than using a project, look for a ContribFest.

For example, From Curious to Contributor: Ship Your First Backstage PR is designed to help attendees contribute directly with project maintainers. Podman also has a ContribFest welcoming first-time contributors.

Look beyond the application

One of the benefits of attending KubeCon + CloudNativeCon as an application developer is understanding what happens after your code leaves your laptop.

Spend some time in an Observability session. Sit in on a Platform Engineering talk. Learn how infrastructure teams think about reliability, security, resource management, and scale.

The better developers and infrastructure teams understand one another, the easier it becomes to build platforms and applications that work well together.

Don’t schedule every minute

The sessions will give you ideas, but the in-person value is being able to keep the conversation going. Talk with the maintainers behind the tools you use, compare developer experience with other application teams, and spend time with platform engineers to understand why the systems beneath your applications work the way they do.

Make room for the Project Pavilion, Solutions Showcase, and informal conversations between sessions. Sometimes the most useful takeaway is not a new tool; it is finding someone who has already solved the developer problem your team is facing.

What you can take back to your team

A good developer journey should leave you with more than a few new project names. It should give you concrete ideas for making the path from code to production simpler, safer, and easier to understand.

You may leave with:

Patterns for improving developer experience and reducing cognitive load
New approaches to the inner loop, delivery workflows, feature management, and application observability
A better understanding of how platform and infrastructure decisions affect developers
Projects, maintainers, and communities you can turn to when you want to go deeper

Where does the developer journey go next?

If you find yourself especially interested in platforms, automation, or developer self-service, explore our Infrastructure Engineer Journey.

If you are increasingly interested in the open source projects behind your developer experience, the Maintainer Journey can show you how to move from user to contributor and potentially become part of the community shaping those technologies.

Join us in Salt Lake City and build your own path through KubeCon + CloudNativeCon North America.

Don’t forget to register for KubeCon + CloudNativeCon North America 2026.

KubeCon + CloudNativeCon North America 2026: Build your SRE journey

Wed, 30 Sep 2026 16:00:00 GMT

If your day starts with questions about reliability, performance, observability, incidents, scaling, or what is going to break next, there is no shortage of places to spend your time at KubeCon + CloudNativeCon North America 2026.

But the SRE journey does not live inside a single conference track.

Operations + Performance may be the obvious starting point, but SREs will also find relevant conversations across Observability, Connectivity, Security, Platform Engineering, Maintainer Track sessions, and the CNCF project communities themselves. View the tracks + schedule.

Here is one way to build your week. Register Today!

Why SREs should attend

Production reliability rarely fits inside one tool or team. KubeCon + CloudNativeCon puts the people building and operating the cloud native stack in the same place, making it possible to compare approaches to observability, incident response, autoscaling, networking, security, and performance with teams facing many of the same challenges.

Beyond the sessions, SREs can bring real production questions directly to maintainers and other practitioners, learn from failure stories and operational lessons, and come away with ideas they can apply to systems they are responsible for today.

Come with a production problem to solve

The strongest SRE journey starts with a problem you already care about. Before you arrive, think about the recurring incident, source of operational toil, or reliability constraint your team keeps running into. Use that problem as a filter for the week rather than trying to attend everything.

A few questions can help shape your agenda:

Where does your team lose the most time during incidents?
Which part of your stack creates the most noise, toil, or uncertainty?
What is hardest to scale or optimize without putting reliability at risk?
Which project or dependency would you most like to understand directly from its maintainers?

Start Monday by going deep

For attendees with an All-Access Pass, Monday’s CNCF-hosted co-located events offer an opportunity to spend a full or half day focused on one area.

Observability Day is a natural starting point for SREs, bringing together maintainers, operators, and end users working with technologies including OpenTelemetry, Prometheus, Fluentd, Fluent Bit, Jaeger, Thanos, Cortex, Perses, Pixie, Kepler, and Inspektor Gadget.

Depending on your day-to-day responsibilities, CiliumCon may be another option for deeper conversations around networking, security, performance, and observability, while FluxCon offers a closer look at GitOps and continuous delivery in production.

Think about the problem you spend the most time solving today and use Monday to go deeper.

Put reliability at the center of your schedule

As the main conference begins, look beyond one track when building your agenda. Sessions SREs may want to put on their radar include:

These sessions span operations, observability, networking, autoscaling, GitOps, and incident recovery, reflecting how reliability work actually happens in production.

Follow the projects behind the systems you operate

KubeCon + CloudNativeCon is also an opportunity to go directly to the communities building the technologies you depend on.

Look for Maintainer Track sessions and Project Pavilion conversations around projects such as OpenTelemetry, Prometheus, KEDA, Flux, Cilium, etcd, OpenCost, Linkerd, and other technologies in your production stack.

Maintainer Track sessions are particularly useful if you want to understand upcoming changes, implementation considerations, current technical challenges, or project roadmaps directly from the people building the software.

Leave room for troubleshooting conversations

Not every useful SRE conversation will happen in a scheduled session.

Visit the Project Pavilion. Talk to maintainers. Compare architectures with other operators. Ask someone else how they solved the problem your team has been fighting for six months.

That direct exchange is one of the biggest differences between watching a recording later and participating in KubeCon + CloudNativeCon in person.

Don’t schedule every minute

Most breakout and keynote recordings will be available after the event. The reason to be in Salt Lake City is what you can do with a session while you are there: ask the speaker about the edge case that looks like your environment, compare an incident pattern with another operator, or take a question directly to a project maintainer.

Leave some white space in your agenda for the Project Pavilion, Solutions Showcase, and the conversations that start when someone says, “We ran into that too.” Those exchanges can be just as useful as the session you originally planned to attend.

What you can take back to your team

The goal is not to leave with a list of tools to buy. It is to leave with stronger patterns, better questions, and a clearer sense of what is working for other teams operating cloud native systems in production.

By the end of the week, you may have:

New approaches to observability, incident response, performance, and telemetry cost
Ideas to test around autoscaling, networking, GitOps, and production resilience
Direct context from maintainers about project roadmaps and implementation tradeoffs
Peers you can continue comparing notes with after the conference

Where does the SRE journey go next?

SREs will probably see significant overlap with our Infrastructure Engineer Journey, especially around platform engineering, networking, scaling, and cluster operations.

And if you have ever filed an issue, helped debug a project, contributed documentation, or wanted more influence over a technology you rely on, take a look at the Maintainer Journey too.

Your next step in cloud native might be closer than you think.

Join us in Salt Lake City and start building your KubeCon + CloudNativeCon North America journey.

Don’t forget to register for KubeCon + CloudNativeCon North America 2026.

When Kubernetes restarts your pod — And when it doesn’t

Mon, 16 Mar 2026 16:00:00 GMT

A production internals guide verified against Kubernetes 1.35 GA
Companion repository: github.com/opscart/k8s-pod-restart-mechanics

The terminology problem

Engineers say “the pod restarted” when they mean four different things. Getting this wrong leads to flawed runbooks and bad on-call decisions.

Term	Pod UID Changes?	Pod IP Changes?	Restart Count
Container restart (process restart inside the same pod)	No	No	+1
Pod recreation (rolling update, drain)	Yes	Yes	Resets to 0
In-place resize (1.35 GA) — CPU	No	No	0
In-place resize (1.35 GA) — memory (RestartContainer policy)	No	No	+1

The practical test: Did the pod UID change? If yes — that is recreation, not a container restart. Restart count resets to zero. If no — same pod object, container process restarted inside it.

The core insight: What kubelet actually watches

kubelet watches the pod spec — not ConfigMaps, not Secrets, not Istio CRDs. If the pod spec didn’t change, kubelet never fires. This single fact explains the majority of “why didn’t my config update?” investigations in production.

Mutating admission webhooks can change the pod spec at creation time, but never after admission — they cannot trigger container restarts post-creation.

Decision matrix

Change	Container Restart?	Pod Recreated?	Automatic?
Container image	Yes	Yes	Yes — Deployment controller
Env var (any source)	Yes	No	Manual rollout
ConfigMap — volume mount	App decides	No	Partial — app must watch inotify
ConfigMap — envFrom	Yes	No	Manual rollout
Secret — volume mount	App decides	No	Partial — app must watch inotify
Secret — envFrom	Yes	No	Manual rollout
Projected ServiceAccount token	Never	No	Yes — kubelet auto-rotates
CPU resize (K8s 1.35+)	Never	No	Manual patch
Memory resize (K8s 1.35+)	Per resizePolicy	No	Manual patch
Istio VirtualService / DestinationRule	Never	No	Yes — xDS push
NetworkPolicy	Never	No	Yes — CNI agent
Service ports	Never	No	Yes — kube-proxy
RBAC	Never	No	Yes — API server
Node drain / eviction	Yes	Yes	Yes — automatic

The flowchart below translates the same matrix into a decision path you can walk at 2am during an incident.

Diagram 1: Complete decision flowchart — does this change require a pod restart?

Scenario 1: ConfigMap — Why the same change has two behaviors

[Diagram 1: ConfigMap env var vs volume mount — env var pod frozen, volume pod auto-synced via kubelet symlink swap]

Env var mode (envFrom / valueFrom): The kernel copies env vars into /proc/<pid>/environ at execve(). That memory is owned by the process — no external system can modify it. Update the ConfigMap and kubelet sees no pod spec change, does nothing. The process keeps old values indefinitely.

Volume mount mode: kubelet syncs via an atomic symlink swap, not a file write:

/etc/config/
├── ..2025_12_19_11_30_00/   ← NEW data dir (kubelet creates this)
│   └── APP_COLOR            ← "red"
├── ..data ─────────────────▶ ..2025_12_19_11_30_00/  ← symlink SWAPPED atomically
└── APP_COLOR ──────────────▶ ..data/APP_COLOR

The symlink swap generates IN_CREATE on ..data — NOT IN_MODIFY on the file. Applications watching IN_MODIFY on an open file descriptor miss this entirely. This is why nginx does not auto-reload on ConfigMap changes without explicit inotify handling.

Lab Evidence (01-configmap/ in companion repo)

ConfigMap updated: APP_COLOR blue → red


Pod A (env var):      APP_COLOR=blue  ← frozen, restart count: 0
Pod B (volume mount): APP_COLOR=red   ← auto-synced, restart count: 0

Correct inotify pattern — watch the directory, not the file

watcher.Add(filepath.Dir(configPath))  // watches /etc/config/ — catches IN_CREATE
// watcher.Add(configPath)             // misses symlink swap entirely


for event := range watcher.Events {
    if event.Op&fsnotify.Create == fsnotify.Create {
        reloadConfig()
    }
}

Scenario 2: Image updates — Recreation vs container restart vs CrashLoop

These three scenarios look similar but are fundamentally different:

Successful image update — pod is recreated

BEFORE: Pod UID aaa-bbb, IP 10.244.1.5, nginx:1.25, restarts: 0
AFTER:  Pod UID xxx-yyy, IP 10.244.1.6, nginx:1.27, restarts: 0
                ↑ UID changed — RECREATION, not container restart

Diagram 3: Rolling update flow showing new ReplicaSet creation, pod recreation, and old RS retained for rollback.

ImagePullBackOff — old pod stays protected

Old pod: Running           ← Kubernetes keeps it alive
New pod: ImagePullBackOff  ← stuck, old pod never killed until new one is healthy

CrashLoopBackOff — same pod, restart count climbs

Pod UID:       aaa-bbb  ← UNCHANGED
Restart count: 0 → 1 → 2 → 3  ← same pod object, container crashing

Diagnostic rule: Climbing restart count with unchanged UID = crash loop. Zero restart count with new UID = rolling update.

StatefulSet note: StatefulSet pods are also recreated on image change, but ordinal identity (pod-0, pod-1) and PVC bindings are preserved. Container restart semantics are identical to Deployments — identity persistence does not imply in-place container restart.

Scenario 3: In-place resource resize (K8s 1.35 GA)

K8s 1.35 made in-place pod resize generally available (kubernetes.io/blog/2025/12/19/kubernetes-v1-35-in-place-pod-resize-ga). Both CPU and memory can be resized without pod recreation — UID and IP never change. In-place resize availability depends on CRI and node OS support; verified on containerd 1.7+ with Linux cgroups v2.

What happens to the container depends on resizePolicy, which you define explicitly:

resizePolicy:
- resourceName: cpu
  restartPolicy: NotRequired      # cgroup quota updated, process untouched
- resourceName: memory
  restartPolicy: RestartContainer # container restarts inside same pod

Lab Evidence (05-resource-resize/ — requires K8s 1.35+)

CPU resize 200m → 500m (NotRequired):
  UID: d7c99204  IP: 10.244.0.7  Restarts: 0  ← all unchanged


Memory resize 256Mi → 512Mi (RestartContainer):
  UID: d7c99204  IP: 10.244.0.7  Restarts: 1
       ↑ same pod      ↑ same IP    ↑ our policy choice, not K8s forcing it

Important: The default resizePolicy for memory is NotRequired. If you omit it, memory resize silently updates the cgroup without restarting the container — and your JVM heap stays at the old size. Always define resizePolicy explicitly for memory.

How to apply

kubectl patch pod my-pod -n my-namespace \
  --subresource resize \
  -p '{"spec":{"containers":[{"name":"app","resources":{
    "requests":{"cpu":"250m","memory":"128Mi"},
    "limits":{"cpu":"500m","memory":"256Mi"}
  }}]}}'
# Note: omit --type=merge — causes a validation error with --subresource resize

Scenario 4: Istio routing — Zero restarts via xDS

Istio VirtualService and DestinationRule changes never trigger container restarts. Istiod maintains a persistent bidirectional gRPC stream to each Envoy sidecar — routing updates are pushed in milliseconds, in-memory swap, no pod touched, no file written.

Lab Evidence (04-istio-routing/ in companion repo)

Four pods. Three routing changes:
  100% v1  →  80/20 canary  →  100% v2


Restart counts: BEFORE 0 0 0 0 / AFTER 0 0 0 0
Pod ages: unchanged throughout all three changes.

Scenario 5: Stakater reloader — Automating the manual step

When apps consume ConfigMaps via env vars, someone must run kubectl rollout restart after every update. Reloader automates this using Kubernetes watch events — detection is near-instant, not polling.

metadata:
  annotations:
    reloader.stakater.com/auto: "true"

Production gotcha: The default Helm install uses watchGlobally=false — Reloader only watches its own namespace. Annotated Deployments in other namespaces are silently ignored, no error thrown. Always install with watchGlobally=true.

helm install reloader stakater/reloader \
  --namespace reloader \
  --set reloader.watchGlobally=true

Lab Evidence (07-stakater-reloader/ in companion repo)

ConfigMap updated. No kubectl rollout restart run.


New pod APP_MESSAGE: "Hello from OpsCart v2 — auto reloaded!"
Rolling container restart triggered automatically.

When hot-reload goes wrong

Hot-reload is not always safer than a container restart. Two failure modes worth knowing:

Semantically invalid config accepted silently

The file updates, the inotify handler fires, no error is thrown — but the new config has a logic error. The pod passes health checks and runs broken for hours. A container restart with a bad config fails immediately and loudly. Hot-reload with bad config fails quietly and late.

Mitigation: Validate config before swapping atomically.

Envoy rejects xDS push silently

Istiod pushes a RouteConfiguration referencing a cluster not yet propagated. Envoy rejects it and continues with old routing rules. No pod event fires. Mitigation: Monitor pilot_xds_push_errors and use istioctl proxy-status.

Observability: Three commands every operator should know

# 1. Container restart or pod recreation? Check UID change
kubectl get pod <pod> -o custom-columns=\
  "NAME:.metadata.name,UID:.metadata.uid,IP:.status.podIP,RESTARTS:.status.containerStatuses[0].restartCount"


# 2. Events on the pod
kubectl describe pod <pod> | grep -A 20 "Events:"


# 3. In-place resize status
kubectl get pod <pod> -o jsonpath='{.status.resize}'

Conclusion

A container restart is disruptive but honest — failure modes are immediate and visible. Hot-reload optimizes for availability but shifts failures to be delayed and subtle. Both are valid strategies. The choice should be conscious.

The goal is not to automate restarts faster. It is to understand deeply enough that you trigger a container restart only when the process genuinely needs to die — and use every other mechanism when it does not.

Companion repository: github.com/opscart/k8s-pod-restart-mechanics

Author: Shamsher Khan — Senior DevOps Engineer, IEEE Senior Member. https://OpsCart.com

This article is adapted from an earlier version published on OpsCart.com. Republished here with permission.

CNCF - Blog

Deploying, maintaining, and migrating production n8n on EKS

Why we started with the community chart

How we ran n8n

What changed with the official chart

Moving traffic safely

Putting Nepal on the CNCF Golden Kubestronaut map

Golden Kubestronaut: Basic anatomy of this program

Kubestronaut journey: A bit of history

From Kubestronaut to Golden Kubestronaut

Why I decided to pursue Golden Kubestronaut

What comes next

Carbon as a first-class signal: Bringing software carbon intensity to OpenTelemetry

The missing signal in observability

Understanding Software Carbon Intensity

Why OpenTelemetry matters

The difficult part: attribution and context

From scope to semantic conventions: where the work stands today

1. Helping engineers understand the impact of their changes

2. Comparing carbon impact across services and providers

3. Supporting automated carbon-aware decisions

What these decisions mean for the next phases

Community participation and diversity

Get involved in the next phases

From carbon measurements to engineering decisions

An opportunity for the cloud-native community

Making carbon a first-class operational signal

Extending Kubernetes policy enforcement to runtime security for AI workloads

Below the workload, not beside it

Reusing Kubernetes policy concepts for runtime enforcement

The same guarantee for AI agents in your cluster

Observability that says why, not just that

What’s next

From zero to agent builder: Running kagent on your laptop

Why kagent makes this almost boring

Before you start: What this lab is not

Step 1: Pick a local model that can actually do it

Step 2: Install kagent with a tool server that can only write Agents

Step 3: The builder is just an agent

Step 4: Ask for an agent

Step 5: The same thing without the UI

Why three guardrails and not one

The part this lab leaves open

Clean up

Where to go from here

Join Platform Engineering Day at KubeCon + CloudNativeCon North America 2026

Created by the community, for practitioners

Learning from real-world platform engineering

A full day for every stage of the platform engineering journey

Where platform engineering is heading

Making the Golden Kubestronaut a little more golden

My Journey to Golden Kubestronaut

More Than a Gold Necklace

Who owns NIS2 and DORA on a Kubernetes platform team

What happens when security owns the regulation

What NIS2 Article 20 and DORA Article 5 actually assign

Put artifacts on the board, not regulation names

The cloud native projects that produce the evidence

A RACI for a Kubernetes platform

Detection is not notification

Exceptions belong to the requesting engineering manager

Capacity is the part that gets skipped

Look at the engineering manager first

References

Don’t give AI agents root. Make them propose the next system state.

The uncomfortable question

Root is not an API

Containers changed the boundary

Immutability is a means to an end

Immutability is not a sandbox

The agent is a workload too

Close the loop without opening the host

Trust the boundaries, not the model

CiliumCon is back at KubeCon + CloudNativeCon North America 2026

Why now?

Where it all began

Why CiliumCon?

What’s on the agenda

Cloud native themes

What you’ll walk away with

Observability that says ***why, not just that***