Agent Sandbox vs. Agent Substrate: The CNCF Runtime Layer for Production AI Agents 2026

The question I kept getting from platform teams in June

Three times in the last month a platform lead has asked me the same question, with slight variations: "We have a dozen teams deploying agents. Some of them are long-running — a customer-success triage agent that holds open conversations for weeks. Some of them are bursty — a marketing-campaign agent that fires when a webhook arrives and dies five seconds later. Do we run them on the same Kubernetes substrate, or are these two different runtime problems?"

The honest answer through May was "yes, this is two different problems, and no, the canonical pattern for either one isn't published anywhere yet." As of July 7, 2026, that changed. Lin Sun from Solo.io, a CNCF Ambassador, published "Why sandboxing your agent is not enough" on the CNCF blog, and Open Source Summit NA Minneapolis launched agent-substrate alongside the existing agent-sandbox project. The two projects now cover the two halves of the platform question, and the choice between them maps cleanly onto the long-running-vs-bursty split.

This article is the architectural breakdown: what agent-sandbox actually is (it's a CRD + controller, not a runtime), what agent-substrate actually is (it's a Knative-style serverless substrate with agent-specific cold-start optimizations), and when each one fits. I am going to assume you already know what a Kubernetes CRD is and that you have at least heard of Knative.

What agent-sandbox actually is

The CNCF SIG Apps agent-sandbox project is a CRD plus a controller. The CRD looks like this:

apiVersion: agents.cncf.io/v1alpha1
kind: AgentSandbox
metadata:
  name: cs-triage-agent-prod
spec:
  image: registry.stackpulsar.com/cs-triage-agent:v1.4.2
  replicas: 3
  sessionAffinity: true
  resources:
    requests:
      cpu: "500m"
      memory: "1Gi"
      nvidia.com/gpu: "0"
    limits:
      cpu: "2"
      memory: "4Gi"
  identity:
    spiffe:
      namespace: agents
      ttl: "1h"
  networkPolicy:
    egress:
      - to:
          - namespaceSelector:
              matchLabels:
                name: mcp-servers
        ports:
          - protocol: TCP
            port: 8080
  otel:
    exporter: otlp
    endpoint: otel-collector.observability:4317
    sampleRatio: "1.0"
  lifecycle:
    maxSessionDuration: "168h"
    idleTimeout: "30m"

The important fields are sessionAffinity: true (every request from the same coding_agent.session.id lands on the same pod), spiffe (per-agent workload identity, not per-pod), and networkPolicy (the agent can only reach the MCP servers it is supposed to reach — not the corporate database, not the internet). The controller reconciles AgentSandbox resources into a StatefulSet with a NetworkPolicy and a ServiceAccount, and the resulting pod runs the agent image with a SPIFFE sidecar and an OTel Collector sidecar.

The thing to notice: agent-sandbox is for agents that have stateful, long-running conversations. The CS triage agent that holds a thread open for two weeks while it waits for human handoff needs session affinity. A build-and-die coding agent does not.

What agent-substrate actually is

The agent-substrate project launched at Open Source Summit NA Minneapolis the week of June 30. It is a serverless-style substrate modeled on Knative Serving, with two agent-specific changes:

1. Cold-start budget under 800ms. Knative Serving targets a cold-start budget of ~1 second; agent-substrate targets under 800ms because the bursty use case is webhook → MCP call → response, and a 1.2-second cold start dominates the user-facing latency budget. They get there by pre-loading the agent runtime image and the most common model SDK (LiteLLM, Anthropic SDK) into a shared layer, and by aggressive snapshot-restore of the agent's working memory.
2. Agent-local scratch volume mounted at /agent/ephemeral. Webhook agents routinely need 100-500MB of scratch space for downloaded files, parsed JSON, or vector-search indices. agent-substrate mounts a per-invocation emptyDir at /agent/ephemeral that survives the duration of one invocation only — the next invocation gets a fresh, empty volume. This is the serverless equivalent of stateless function execution but with the right primitive name.

The CRD is much shorter than AgentSandbox because most of the configuration is serverless-defaults:

apiVersion: agents.cncf.io/v1alpha1
kind: AgentFunction
metadata:
  name: campaign-trigger-agent
spec:
  image: registry.stackpulsar.com/campaign-trigger:v0.7.1
  trigger:
    http:
      path: /webhooks/campaign
  maxInstances: 200
  timeoutSeconds: 30
  otel:
    exporter: otlp
    endpoint: otel-collector.observability:4317

No sessionAffinity (every invocation is fresh), no spiffe at the pod level (the function gets a fresh, short-lived identity per invocation), no networkPolicy at the level agent-sandbox needs (the network policy is expressed at the trigger, not at the function). The result is a deploy unit that can scale from zero to 200 invocations in under 30 seconds and back to zero in 60 seconds.

Decision tree: which one for which agent

The decision is not "always use agent-substrate because it is newer." The decision is about three axes, and answering the first one correctly is 80% of the answer.

Question	If yes →	If no →
Does the agent maintain state across invocations (conversation history, working memory, scratch files)?	agent-sandbox	agent-substrate
Is the agent triggered by a long-lived event (websocket, email, IM) rather than a request?	agent-sandbox	agent-substrate
Does the agent require network access to internal services that are not exposed via an MCP server?	agent-sandbox (with explicit NetworkPolicy)	agent-substrate (and convert the access to MCP)
Is the agent's invocation pattern bursty (zero → many → zero) rather than steady?	agent-substrate	agent-sandbox

For the platform lead's original question — long-running CS triage agent + bursty marketing campaign agent — the answer is two deployments, two CRDs. The CS triage agent is AgentSandbox with three replicas and 7-day max session duration. The campaign trigger agent is AgentFunction with 200-instance ceiling and 30-second timeout. They run on the same cluster, share the same OTel Collector, share the same MCP server fleet. The runtime substrate is the only thing that differs.

The OTel pattern that makes both debuggable

The CNCF post correctly identifies that sandboxing alone is not enough — you need to be able to see what the sandboxed agent did after the fact. The OTel pattern below is what we ship for both CRD types, and it is the reason the difference between agent-sandbox and agent-substrate does not leak into your observability layer.

Three additions on top of the standard GenAI semantic conventions:

1. agents.cncf.io/runtime resource attribute: agent-sandbox or agent-substrate. Lets you filter "show me only sandboxed-agent spans" without changing the schema.
2. agents.cncf.io/agent_identity resource attribute: the SPIFFE ID for sandboxed agents, the per-invocation identity for substrate agents. Same attribute name, different values.
3. agents.cncf.io/invocation_id span attribute: for substrate agents, this is the same as the invocation; for sandboxed agents, this stays stable across a session. Lets you answer "what did this agent invocation do?" without caring which runtime it ran on.

The five panels that surface the production failure modes both runtimes share:

• Active agents: a gauge of count(agents.cncf.io/agent_identity), broken down by agents.cncf.io/runtime. Surfaces the "agents scaled to zero, why are we paying for them?" question.
• Tool-call p99 by MCP server: histogram of gen_ai.tool_call.duration_ms filtered by gen_ai.tool.name. Surfaces MCP-server-side latency regressions that show up identically in both runtimes.
• Cold-start latency by runtime: histogram of agents.cncf.io/cold_start_ms for substrate agents (always 0 for sandboxed). Substrate's <800ms budget is verified here.
• Session duration p99: for sandboxed agents only. Substrate agents don't have a "session," only invocations, and the corresponding metric is invocation duration p99.
• Per-agent cost per day: sum(gen_ai.cost.total_usd) grouped by agents.cncf.io/agent_identity. Same metric both runtimes; the platform team can answer "what is this agent costing us?" without asking which runtime it runs on.

What about agentgateway and kagent?

Two Solo.io projects you have probably heard of in this context, and both have a different role:

• Agentgateway is an LLM gateway — the layer that brokers model calls between the agent and the model provider. It is upstream of both agent-sandbox and agent-substrate. The CNCF blog post from Lin Sun positions agentgateway as the model-call plane; agent-sandbox and agent-substrate are the agent-execution plane. We have been running AgentGateway in front of both since May and the cost-attribution granularity it gives you — per-agent, per-model, per-provider, with retry and fallback accounting — is the layer that makes the per-agent cost dashboard in the section above actually work.
• Kagent is Solo.io's Kubernetes operator for declarative agent management — a higher-level abstraction that creates AgentSandbox and AgentFunction resources from a Workflow CRD. If you want a YAML-driven way to express a multi-agent pipeline (research agent → write agent → review agent) without writing the lower-level CRDs by hand, kagent is the right tool. It is not a runtime — it is a templating layer on top.

For a platform team adopting this for the first time, the rollout order we have seen work is: start with agent-substrate for the bursty webhook agents (lowest operational overhead, fastest to value), then add agent-sandbox for the long-running agents once the team has internalized the OTel pattern, then add agentgateway in front of both once per-agent cost attribution becomes a finance question rather than a platform question, and finally add kagent once the multi-agent pipeline templates become a load-bearing piece of the platform.

What I would skip in v1

Two things we tried that did not earn their place:

• Running both runtimes for the same agent. We had one team that wanted their CS triage agent to be sandboxed for stateful conversations but serverless for webhook triggers (when the customer first opens a ticket). The complexity of moving state between runtimes was not worth the operational simplicity of just keeping it on agent-sandbox with three replicas. Pick one.
• Custom AgentSandbox resource requests per agent. The temptation is to right-size CPU/memory for every agent. In practice we ended up with three sizes — small (250m CPU / 512Mi RAM), medium (500m / 1Gi), large (1 / 4Gi) — and that has covered every agent we have run. Per-agent sizing adds review overhead without measurable cost savings at our scale (200 agents across 12 teams).

Conclusion: the runtime layer was the missing piece

Before agent-sandbox and agent-substrate landed, the canonical CNCF pattern for an agent platform was a Deployment + a NetworkPolicy + a ServiceAccount + an OTel sidecar + a SPIFFE sidecar, glued together by hand. That works. It also means every platform team that builds it builds it slightly differently, and every incident you have is a slightly novel incident to debug.

The two CNCF SIG Apps projects are the standardization. The CRDs above are the spec. The OTel pattern in this article is the instrumentation contract. The decision tree is the mental model. If you are a platform team rolling this out in the next quarter, copy the CRDs and the OTel attributes verbatim — the standardization is the whole point.