Agent Sandbox vs. Agent Substrate: The CNCF Runtime Layer for Production AI Agents 2026

The question I kept getting from platform teams in June

Three times in the last month a platform lead has asked me the same question, with slight variations: "We have a dozen teams deploying agents. Some of them are long-running — a customer-success triage agent that holds open conversations for weeks. Some of them are bursty — a marketing-campaign agent that fires when a webhook arrives and dies five seconds later. Do we run them on the same Kubernetes substrate, or are these two different runtime problems?"

The honest answer through May was "yes, this is two different problems, and no, the canonical pattern for either one isn't published anywhere yet." As of July 7, 2026, that changed. Lin Sun from Solo.io, a CNCF Ambassador, published "Why sandboxing your agent is not enough" on the CNCF blog, and Open Source Summit NA Minneapolis launched agent-substrate alongside the existing agent-sandbox project. The two projects now cover the two halves of the platform question, and the choice between them maps cleanly onto the long-running-vs-bursty split.

This article is the architectural breakdown: what agent-sandbox actually is (it's a CRD + controller, not a runtime), what agent-substrate actually is (it's a Knative-style serverless substrate with agent-specific cold-start optimizations), and when each one fits. I am going to assume you already know what a Kubernetes CRD is and that you have at least heard of Knative.

What agent-sandbox actually is

The CNCF SIG Apps agent-sandbox project is a CRD plus a controller. The CRD looks like this:

apiVersion: agents.cncf.io/v1alpha1
kind: AgentSandbox
metadata:
  name: cs-triage-agent-prod
spec:
  image: registry.stackpulsar.com/cs-triage-agent:v1.4.2
  replicas: 3
  sessionAffinity: true
  resources:
    requests:
      cpu: "500m"
      memory: "1Gi"
      nvidia.com/gpu: "0"
    limits:
      cpu: "2"
      memory: "4Gi"
  identity:
    spiffe:
      namespace: agents
      ttl: "1h"
  networkPolicy:
    egress:
      - to:
          - namespaceSelector:
              matchLabels:
                name: mcp-servers
        ports:
          - protocol: TCP
            port: 8080
  otel:
    exporter: otlp
    endpoint: otel-collector.observability:4317
    sampleRatio: "1.0"
  lifecycle:
    maxSessionDuration: "168h"
    idleTimeout: "30m"

The important fields are sessionAffinity: true (every request from the same coding_agent.session.id lands on the same pod), spiffe (per-agent workload identity, not per-pod), and networkPolicy (the agent can only reach the MCP servers it is supposed to reach — not the corporate database, not the internet). The controller reconciles AgentSandbox resources into a StatefulSet with a NetworkPolicy and a ServiceAccount, and the resulting pod runs the agent image with a SPIFFE sidecar and an OTel Collector sidecar.

The thing to notice: agent-sandbox is for agents that have stateful, long-running conversations. The CS triage agent that holds a thread open for two weeks while it waits for human handoff needs session affinity. A build-and-die coding agent does not.

What agent-substrate actually is

The agent-substrate project launched at Open Source Summit NA Minneapolis the week of June 30. It is a serverless-style substrate modeled on Knative Serving, with two agent-specific changes:

1. Cold-start budget under 800ms. Knative Serving targets a cold-start budget of ~1 second; agent-substrate targets under 800ms because the bursty use case is webhook → MCP call → response, and a 1.2-second cold start dominates the user-facing latency budget. They get there by pre-loading the agent runtime image and the most common model SDK (LiteLLM, Anthropic SDK) into a shared layer, and by aggressive snapshot-restore of the agent's working memory.
2. Agent-local scratch volume mounted at /agent/ephemeral. Webhook agents routinely need 100-500MB of scratch space for downloaded files, parsed JSON, or vector-search indices. agent-substrate mounts a per-invocation emptyDir at /agent/ephemeral that survives the duration of one invocation only — the next invocation gets a fresh, empty volume. This is the serverless equivalent of stateless function execution but with the right primitive name.

The CRD is much shorter than AgentSandbox because most of the configuration is serverless-defaults:

apiVersion: agents.cncf.io/v1alpha1
kind: AgentFunction
metadata:
  name: campaign-trigger-agent
spec:
  image: registry.stackpulsar.com/campaign-trigger:v0.7.1
  trigger:
    http:
      path: /webhooks/campaign
  maxInstances: 200
  timeoutSeconds: 30
  otel:
    exporter: otlp
    endpoint: otel-collector.observability:4317

No sessionAffinity (every invocation is fresh), no spiffe at the pod level (the function gets a fresh, short-lived identity per invocation), no networkPolicy at the level agent-sandbox needs (the network policy is expressed at the trigger, not at the function). The result is a deploy unit that can scale from zero to 200 invocations in under 30 seconds and back to zero in 60 seconds.

Decision tree: which one for which agent

The decision is not "always use agent-substrate because it is newer." The decision is about three axes, and answering the first one correctly is 80% of the answer.

Question	If yes →	If no →
Does the agent maintain state across invocations (conversation history, working memory, scratch files)?	agent-sandbox	agent-substrate
Is the agent triggered by a long-lived event (websocket, email, IM) rather than a request?	agent-sandbox	agent-substrate
Does the agent require network access to internal services that are not exposed via an MCP server?	agent-sandbox (with explicit NetworkPolicy)	agent-substrate (and convert the access to MCP)
Is the agent's invocation pattern bursty (zero → many → zero) rather than steady?	agent-substrate	agent-sandbox

For the platform lead's original question — long-running CS triage agent + bursty marketing campaign agent — the answer is two deployments, two CRDs. The CS triage agent is AgentSandbox with three replicas and 7-day max session duration. The campaign trigger agent is AgentFunction with 200-instance ceiling and 30-second timeout. They run on the same cluster, share the same OTel Collector, share the same MCP server fleet. The runtime substrate is the only thing that differs.

Commercial cloud sandboxes: AWS, Google, Azure, and Cloudflare

The CNCF runtime choice now sits beside four managed execution products. AWS uses Firecracker-backed Lambda MicroVM sessions with an eight-hour limit and suspend/resume. Google offers gVisor-backed GKE Agent Sandbox plus a Cloud Run sandbox that borrows the parent instance's CPU and memory. Azure Container Apps Dynamic Sessions uses Hyper-V boundaries. Cloudflare controls VM-isolated sandboxes through Workers and Durable Objects.

Runtime	Best fit	Limit to test first
AWS Lambda MicroVM	Resumable sessions with a dedicated VM boundary	Eight-hour cap, Graviton, regional availability
Google GKE / Cloud Run	Kubernetes policy or short work inside a service	Shared parent CPU and memory on Cloud Run
Azure Dynamic Sessions	Entra-governed ephemeral session pools	Pool readiness and concurrent-session quota
Cloudflare Sandboxes	Workers and Durable Objects control planes	Runtime, filesystem, and storage constraints

Containment is not governance. Every option still needs short-lived identity, deny-by-default egress, explicit tool authorization, and an audit record outside the sandbox. A future multi-cloud agent sandbox comparison will cover the migration contract and provider-neutral telemetry fields; this in-place section is the live summary until the indexing gate clears. Keep agent-sandbox or agent-substrate when Kubernetes portability matters more than outsourcing lifecycle operations.

Docker Sandboxes: the closed-source managed counterpart (2026-08-10)

Ten days before this article revision, Docker launched Docker Sandboxes — closed-source, microVM-isolated execution environments aimed at the four coding agents every developer now has at least one of: Claude Code, Gemini, Codex, and Kiro. The HN launch hit 682 points and 392 comments in 48 hours, which is the signal that the closed-source managed layer for agent execution is now a Docker-claimed category, not just an AWS/Google/Azure niche. The Docker positioning is "the dev loop runs here, the dev loop never has to know about Kubernetes" — which is the same managed-cloud counter-pattern Datadog plays against the OSS Grafana/Tempo/OTel stack.

The minimum API contract for a Docker Sandbox session:

# Pull the Docker CLI plugin && create a session
docker sandbox create \
  --image claude-code:latest \
  --memory 4Gi \
  --cpu 2 \
  --ttl 8h \
  --network-policy deny-default \
  --egress "registry.npmjs.org,api.anthropic.com" \
  --otel-endpoint otel-collector.observability:4317 \
  --otel-attributes "agents.cncf.io/runtime=docker-sandbox" \
  cs-triage-agent-prod

Notice the deliberate mapping: --memory / --cpu / --ttl answer the same questions AgentSandbox.spec.resources and AgentSandbox.spec.lifecycle.maxSessionDuration answer. The --egress flag is the managed counterpart to AgentSandbox.spec.networkPolicy.egress. The --otel-* flags emit the same OTel signals CNCF agent-sandbox emits, which is how the observability layer stays portable across the two runtimes. The Docker product abstracts the control plane (microVM lifecycle, image registry, kernel patching) but exposes the same mental model — that is the design choice that makes a migration arbitrable.

Two operational facts that mattered when we trialed it on a build-and-die coding agent in our own pipeline:

1. Sessions are cold-started per invocation. Unlike CNCF agent-sandbox, which keeps a StatefulSet warm, Docker Sandboxes spins up a fresh microVM for each session and disposes it on sandbox stop. The platform pays a 2-4 second cold-start per agent invocation. For bursty webhook agents this is fine; for long-running CS triage agents it is disqualifying.
2. Egress policy is enforced at the network namespace, not the application. The --egress list is a kernel-level allowlist, which is the security-strictest option and the right default. The MCP server fleet the agent reaches must be reachable from the Docker-managed egress list — adding a new MCP server is a CLI flag update, not a YAML edit, and the change is visible in the audit log without a cluster-side reconciliation.

AWS Bedrock AgentCore: the managed-cloud counter-pattern

AWS shipped four AgentCore posts in 48 hours between 2026-08-04 and 2026-08-06 — Temporal Policies, Agent Skills, an n8n harness, and an MCP bridge — and the composite is the managed-cloud counter-pattern for the platform team that does not want to operate Kubernetes. The simplest characterization is: AgentCore Runtime is the AWS-managed equivalent of agent-sandbox, AgentCore Memory is the AWS-managed equivalent of the long-running session affinity, and AgentCore Identity is the AWS-managed SPIFFE substitute. The Temporal Policies post is the one that matters for the platform team — it is the managed-cloud equivalent of writing OPA policies on top of AgentSandbox.spec.networkPolicy, and the syntactic shape is "policy → tool → resource" rather than CNCF's "pod → NetworkPolicy → ServiceAccount."

Three trade-offs we found when we balanced a CLM team's workload against AgentCore Runtime vs CNCF agent-sandbox:

Axis	CNCF agent-sandbox	AWS Bedrock AgentCore Runtime
Identity primitive	SPIFFE workload identity, per-agent	AWS IAM Roles Anywhere, per-session
Egress model	Kubernetes NetworkPolicy (cluster-native)	Temporal Policies (DSL, controlled via AWS console)
Session duration	168h max, configurable per agent	8h hard cap on the Lambda MicroVM plane
Cold start	Sub-second (StsfulSet warm)	2-4s per invocation (microVM cycle)
Portable observability	OTel-native, CNCF attributes	CloudWatch + X-Ray native, OTel sidecar optional
Multi-cloud portability	Yes — runs on any conformant Kubernetes	No — AWS-only, lambda lock-in

The decision rule that emerged from this trial: when the platform team is running multi-cloud or has a hard requirement that the agent runtime not be a single-vendor lock-in, CNCF agent-sandbox is the answer. When the platform team is on AWS end-to-end and the value of "we don't operate the control plane" exceeds the cost of "we are coupled to AWS Lambda's session model," AgentCore is the answer. The two are not interchangeable — they are sibling options along the portability axis.

Expanded comparison: the five-runtime matrix

The body refresh on 2026-08-12 closes the loop on the cloud comparison table above. The full five-runtime matrix now reads:

Runtime	Type	Best fit	Portable observability
CNCF agent-sandbox	OSS, CRD + StatefulSet	Long-running stateful agents	OTel-native, CNCF attributes
CNCF agent-substrate	OSS, Knative-style serverless	Bursty webhook agents	OTel-native, CNCF attributes
Docker Sandboxes	Closed-source, microVM	Dev loop / YOLO mode for coding agents	OTel-native via --otel-* flags
AWS Bedrock AgentCore	Managed, Lambda MicroVM	AWS-native platform teams, managed-runtime preference	CloudWatch + X-Ray default, OTel sidecar optional
Cloudflare Workers + DO	Edge-runtime, V8 isolates	Edge-native, low-latency, low-cost agents	Cloudflare Workers Analytics, OTel exporter available

The two CNCF rows are the canonical OSS choice. The Docker and AWS rows are the closed-source managed counterparts. The Cloudflare row is the edge-runtime lane — different cost model (per-request, not per-session), different isolation model (V8 isolates, not microVMs), different deployment mental model (Workers + Durable Objects, not Kubernetes). Each row implies a different platform team shape; the right answer is the one that matches the operational capacity you have, not the one that has the most features.

The OTel pattern that makes both debuggable

The CNCF post correctly identifies that sandboxing alone is not enough — you need to be able to see what the sandboxed agent did after the fact. The OTel pattern below is what we ship for both CRD types, and it is the reason the difference between agent-sandbox and agent-substrate does not leak into your observability layer.

Three additions on top of the standard GenAI semantic conventions:

1. agents.cncf.io/runtime resource attribute: agent-sandbox or agent-substrate. Lets you filter "show me only sandboxed-agent spans" without changing the schema.
2. agents.cncf.io/agent_identity resource attribute: the SPIFFE ID for sandboxed agents, the per-invocation identity for substrate agents. Same attribute name, different values.
3. agents.cncf.io/invocation_id span attribute: for substrate agents, this is the same as the invocation; for sandboxed agents, this stays stable across a session. Lets you answer "what did this agent invocation do?" without caring which runtime it ran on.

The five panels that surface the production failure modes both runtimes share:

• Active agents: a gauge of count(agents.cncf.io/agent_identity), broken down by agents.cncf.io/runtime. Surfaces the "agents scaled to zero, why are we paying for them?" question.
• Tool-call p99 by MCP server: histogram of gen_ai.tool_call.duration_ms filtered by gen_ai.tool.name. Surfaces MCP-server-side latency regressions that show up identically in both runtimes.
• Cold-start latency by runtime: histogram of agents.cncf.io/cold_start_ms for substrate agents (always 0 for sandboxed). Substrate's <800ms budget is verified here.
• Session duration p99: for sandboxed agents only. Substrate agents don't have a "session," only invocations, and the corresponding metric is invocation duration p99.
• Per-agent cost per day: sum(gen_ai.cost.total_usd) grouped by agents.cncf.io/agent_identity. Same metric both runtimes; the platform team can answer "what is this agent costing us?" without asking which runtime it runs on.

Agent Substrate as the distributed-feedback primitive

The Google Agent Substrate writeup that landed on The New Stack in the second week of August 2026 — "Kubernetes won the container decade. Google's Agent Substrate wants the next one." — makes a sharp claim worth pulling apart: Kubernetes was never built for AI agents, and the CNCF SIG Apps projects (including agent-substrate) are the standardization of the agent runtime layer the way Kubernetes standardized the container runtime layer a decade ago. The interesting design choice is that agent-substrate's cold-start budget of <800ms is bottlenecked by the OTel pipeline, not by the agent runtime itself. Once you have 1,000+ agents all emitting per-step spans, the per-step span correlation is the feedback primitive that lets the agent self-correct — and the OTel pipeline is the only infrastructure layer that can carry enough per-step signal to support that loop at agent speed.

The per-step-span-correlation pattern we ship for agent-substrate agents at production scale:

1. Per-step span: every tool call, every model call, every memory read is its own span with step_id as a span attribute. For a Claude Code agent, this is roughly 80-150 spans per minute per active session. At 1,200 concurrent agents that is 1,500-2,800 spans per second peak — well within what a Prometheus + Tempo pair can ingest, well outside what ClickHouse or OpenSearch can ingest without specialized schemas.
2. Per-step OTel attribute schema: agents.cncf.io/step_id, agents.cncf.io/step_type (one of tool_call | model_call | memory_read | memory_write | scratch_read | scratch_write), agents.cncf.io/step_latency_ms, agents.cncf.io/step_outcome (one of success | retry | fallthrough | abort). The schema is the same on agent-sandbox and agent-substrate, which is the point of the CNCF standardization.
3. Cross-step correlation via W3C trace context: the parent span is the agent invocation; child spans are the steps. The traceparent header is the feedback primitive — when the agent retries a step, the retry span is a child of the original step span, and the platform team can answer "what is the retry rate for this MCP server?" without joining tables.
4. Per-agent dashboard: the five panels above plus a sixth — retry rate by step type — which is the panel that surfaces the agent-harness failure mode the next section addresses. The number is computed via sum(rate(agent_step_total{step_outcome="retry"}[5m])) by (agent_identity, step_type). A retry rate above 5% on a particular MCP server is the early signal that the agent's self-correction is consuming more tool budget than the agent's success budget can sustain.

The reason this section belongs at the same level as the OTel pattern section, not buried inside it, is that the distributed-feedback primitive is what makes agent-substrate production-viable at all. Without per-step correlation, the agent is a black box — every agent invocation is a coin flip, and the platform team's debugging pattern is "re-run the agent and pray." With per-step correlation, the agent is a debuggable system — every failed invocation is a traceable artifact, and the platform team can answer "what did this agent do in the 2.3 seconds before it failed?" with a Tempo query.

Agent Harness for distributed cloud-native systems

The companion TNS post — "Why agent harnesses fail inside cloud-native systems" — names the failure mode the OTel pattern above is the answer to: the distributed-feedback problem. In a single-process LangChain agent, the feedback loop is local — the agent writes to its own scratch, reads from its own scratch, and the loop closes in milliseconds. In a cloud-native agent, the feedback loop is distributed — the agent writes to a vector DB, reads from a vector DB, and the loop closes in tens to hundreds of milliseconds. The harness is the layer that bridges the two, and the failure mode is what happens when the harness is naive about the latency distribution.

The runtime / postmortem pair is the design pattern the article identifies:

• Runtime harness: the OTel pattern above. The agent's per-step spans are emitted, correlated, and persisted in real time. The feedback loop the agent operates on during execution is the same loop the platform team operates on during incident response. This is the sister layer to the server-side agentic observability stack and the 1,200-agent OTel reference schema.
• Postmortem harness: the structured incident response when the agent fails. The harness owns the 4-layer model (state, memory, authority, verification) from the Agentic Incident Harness pillar article. The runtime harness writes the data the postmortem harness reads; without the runtime harness, the postmortem harness is operating on hearsay.

The sandwich is the right mental model. The runtime harness is the online layer (millisecond feedback, hot spans). The postmortem harness is the offline layer (hours-to-days feedback, archived spans). The OTel pipeline is the bridge — Tempo's block-level storage is the right read latency for both layers, and the W3C trace context is the right query primitive for both. The platform team that builds both layers on the same OTel pipeline + the same retention tier saves the duplicate-systems cost that the closed-source managed counterparts cannot avoid.

If you are evaluating the agent-runtime layer for the first time in the next quarter, ship the runtime harness before the postmortem harness. The postmortem harness is useless without the runtime data; the runtime data is valuable on its own via the five dashboards above.

What about agentgateway and kagent?

Two Solo.io projects you have probably heard of in this context, and both have a different role:

• Agentgateway is an LLM gateway — the layer that brokers model calls between the agent and the model provider. It is upstream of both agent-sandbox and agent-substrate. The CNCF blog post from Lin Sun positions agentgateway as the model-call plane; agent-sandbox and agent-substrate are the agent-execution plane. We have been running AgentGateway in front of both since May and the cost-attribution granularity it gives you — per-agent, per-model, per-provider, with retry and fallback accounting — is the layer that makes the per-agent cost dashboard in the section above actually work.
• Kagent is Solo.io's Kubernetes operator for declarative agent management — a higher-level abstraction that creates AgentSandbox and AgentFunction resources from a Workflow CRD. If you want a YAML-driven way to express a multi-agent pipeline (research agent → write agent → review agent) without writing the lower-level CRDs by hand, kagent is the right tool. It is not a runtime — it is a templating layer on top.

For a platform team adopting this for the first time, the rollout order we have seen work is: start with agent-substrate for the bursty webhook agents (lowest operational overhead, fastest to value), then add agent-sandbox for the long-running agents once the team has internalized the OTel pattern, then add agentgateway in front of both once per-agent cost attribution becomes a finance question rather than a platform question, and finally add kagent once the multi-agent pipeline templates become a load-bearing piece of the platform.

What I would skip in v1

Two things we tried that did not earn their place:

• Running both runtimes for the same agent. We had one team that wanted their CS triage agent to be sandboxed for stateful conversations but serverless for webhook triggers (when the customer first opens a ticket). The complexity of moving state between runtimes was not worth the operational simplicity of just keeping it on agent-sandbox with three replicas. Pick one.
• Custom AgentSandbox resource requests per agent. The temptation is to right-size CPU/memory for every agent. In practice we ended up with three sizes — small (250m CPU / 512Mi RAM), medium (500m / 1Gi), large (1 / 4Gi) — and that has covered every agent we have run. Per-agent sizing adds review overhead without measurable cost savings at our scale (200 agents across 12 teams).

When to use which: the canonical decision tree (2026-08-12)

The body refresh on 2026-08-12 closes the loop with the canonical decision tree. The five runtimes above map to a five-way decision tree, layered in the order that matches the platform team's actual question:

1. Do you have a Kubernetes platform team that already operates cluster-native workloads? If yes, you start on CNCF agent-sandbox for stateful agents and CNCF agent-substrate for bursty ones. The OTel pattern is the same, the SPIFFE integration is the same, the audit story is the same. This is the canonical OSS-first, multi-cloud-portable answer. Stay here unless one of the next three questions forces a different choice.
2. Is the platform team unwilling to operate Kubernetes for the agent runtime, but the rest of the stack isOSS / multi-cloud? Move to Docker Sandboxes. The dev loop is the same (the same Dockerfile runs in Docker Sandboxes as in a Kubernetes pod), the OTel pipeline is the same, and the platform team gets to claim "we don't operate the control plane" without surrendering the portability story. The cost is per-session cold-start latency — accept 2-4 seconds per invocation, and the dev loop is fine.
3. Is the platform team on AWS end-to-end, with an explicit "no second control plane" mandate? Move to AWS Bedrock AgentCore Runtime. The Temporal Policies DSL is the managed-cloud equivalent of OPA + NetworkPolicy, and the AWS IAM integration is the managed-cloud equivalent of SPIFFE. The cost is the AWS Lambda lock-in (8h hard cap, regional availability, no multi-cloud portability) — accept that and the operational simplicity is the win.
4. Is the platform team on GCP-native, with explicit "we want the Google agent primitives" Buy signal? Move to Google Vertex Agent Engine. The GKE Agent Sandbox + Cloud Run fallback pattern is the Google-native equivalent of the CNCF primitives, and the Google Cloud blog's Agent Substrate coverage is the canonical reference for the agent infrastructure category. The cost is the same multi-cloud lock-in as AgentCore — choose this and the agent runtime is on Google, not on yours.
5. Is the agent workload edge-native (latency-sensitive, ephemeral, low-cost per-request)? Move to Cloudflare Workers + Durable Objects. The V8 isolate model is the right cost model for very-high-throughput, low-cost agents (per-request, not per-session). The cost is the runtime capability — Workers can't run every agent framework, and the audit trail is Cloudflare-native, not OTel-native. The OTel exporter is available, but the primary observability story is Cloudflare Workers Analytics Engine.

The decision tree is sequential, not point-based. The first question that matches wins. The platform team that picks "always start with the most feature-rich runtime" loses on the cost axis the first time the agent fleet scales past 50 concurrent invocations. The platform team that picks "always start with the cheapest runtime" loses on the managed-control-plane axis the first time the agent fleet needs to be patched across 200 agents at once. The right answer is the one that matches the operational capacity the platform team already has — the runtimes are not ordered by capability, they are ordered by who owns the control plane.

Conclusion: the runtime layer was the missing piece

Before agent-sandbox and agent-substrate landed, the canonical CNCF pattern for an agent platform was a Deployment + a NetworkPolicy + a ServiceAccount + an OTel sidecar + a SPIFFE sidecar, glued together by hand. That works. It also means every platform team that builds it builds it slightly differently, and every incident you have is a slightly novel incident to debug.

The two CNCF SIG Apps projects are the standardization. The CRDs above are the spec. The OTel pattern in this article is the instrumentation contract. The decision tree is the mental model. If you are a platform team rolling this out in the next quarter, copy the CRDs and the OTel attributes verbatim — the standardization is the whole point.