Agent Observability at 1,200+ Agents: The OpenTelemetry Native Stack 2026 (AWS AgentCore vs. OSS, the Coinbase Pattern)

Why the AWS pattern matters even if you never touch AWS

On July 7, 2026, The New Stack published two articles back to back that, taken together, set the agent-observability reference architecture for the second half of the year. The first, "Watch AWS engineers troubleshoot agentic AI with OpenTelemetry and OpenSearch", walks through Amazon's Bedrock AgentCore Observability team's pattern: OTel gen_ai.* semantic conventions, OpenSearch as the trace store, and a Quality/Reliability/Efficiency taxonomy that surfaces the failure modes that ship in agent loops. The second, "Coinbase runs 1,200 agents and just slashed its AI bill in half", walks through Coinbase's production agent fleet and the cost-control stack that cut their bill from $800K/month to $400K/month.

Both pieces are AWS-adjacent. Neither one is required to be implemented on AWS — the OTel gen_ai.* conventions are vendor-neutral by design, and Coinbase's stack is open-source Tempo + Grafana + a custom cost-control layer. But the pattern they describe — semantic-convention-native instrumentation at the agent-loop level, trace-store choice driven by retention math, and a Quality/Reliability/Efficiency taxonomy that maps onto the dashboards — is the canonical reference architecture for the rest of 2026. If StackPulsar doesn't publish the OTel-native, vendor-neutral counter-pattern now, the SERP for "agent observability" will be owned by AWS's own posts.

The OTel gen_ai.* conventions for agent loops

The GenAI semantic conventions stabilized mid-2026, and the agent-loop conventions — distinct from the inference conventions I covered in OpenTelemetry AI Inference Tracing — are what most teams have wrong. The conventions for an agent loop look like this:

# Agent loop root span — wraps the entire agent execution
gen_ai.agent.name: cs-triage-agent
gen_ai.agent.id: spiffe://stackpulsar/agents/cs/triage-prod
gen_ai.agent.loop.iteration: 3              # 0-indexed iteration counter
gen_ai.agent.loop.max_iterations: 12

# Each agent step emits a child span
gen_ai.operation.name: tool_call | reasoning | planning | response
gen_ai.tool.name: edit_file | search_code | run_command | lookup_order
gen_ai.tool.call.id: 9a1f2b3c-...            # unique per tool call
gen_ai.tool.call.duration_ms: 1240

# Model call inside the reasoning step — nested span
gen_ai.request.model: claude-opus-4-1
gen_ai.request.temperature: 0.7
gen_ai.usage.input_tokens: 1284
gen_ai.usage.output_tokens: 412
gen_ai.response.finish_reason: end_turn | tool_use | max_tokens | stop_sequence

The convention I see most teams getting wrong is the gen_ai.tool.call.id. It must be unique per tool call, not per tool invocation target — a single MCP call to lookup_order at iteration 3 is a different tool call than the same MCP call at iteration 4, even if the parameters are identical. Without per-call uniqueness you cannot debug "the same tool call returned different things at different iterations" — which is one of the most common agent failure modes.

The second thing teams get wrong is gen_ai.response.finish_reason. For an agent loop, the meaningful finish reasons are:

• tool_use: the model wants to make another tool call. Continue the loop.
• end_turn: the model is done. Return the final response.
• max_tokens: the model hit the per-call token limit. Usually a planning failure on the previous iteration; needs alerting.
• stop_sequence: a configured stop sequence fired. Usually intentional.

The distinction between tool_use and end_turn is the loop-control signal. If you don't capture it on every iteration you cannot answer "why did the agent loop end?" after the fact.

The Coinbase pattern, in detail

Coinbase's 1,200-agent fleet runs on what Coinbase calls the "Three-Pane Agent Telemetry" pattern: traces, metrics, and logs in a single OTel-native pipeline, with the per-agent cost dashboard being the panel the finance team looks at. The architecture they published in July:

1. Instrumentation: OTel SDK in the agent runtime (Coinbase uses a custom fork of LangGraph), with auto-instrumentation for Anthropic and OpenAI SDKs and manual instrumentation for the agent-loop control plane. Every agent step emits a span; every model call inside the step is a child span; every tool call is a grandchild span with the MCP server as a resource attribute.
2. Collection: a fleet of OTel Collectors (one per 100 agents, autoscaled) with a tail-sampling processor that keeps all error-path spans and 5% of happy-path spans. Tail sampling is the right pattern at this scale — head sampling loses the errors you actually need; full sampling is unaffordable.
3. Trace storage: Tempo with a 30-day retention for hot traces, plus an S3 export for 90-day cold retention. Coinbase's choice of Tempo over OpenSearch was driven by the retention math: at 1,200 agents × ~50 spans per session × ~3 sessions per agent per day, Tempo was ~$8K/month for the hot tier; OpenSearch at the same ingest was ~$24K/month.
4. Cost control: a custom cost-control layer that joins per-agent gen_ai.cost.total_usd spans to a per-team budget manifest, and emits a soft-then-hard throttle when a team exceeds its daily budget. The throttling is what cut the bill from $800K to $400K/month — not better cost attribution, but actual enforcement.

Tempo vs. ClickHouse vs. OpenSearch at 1,200 agents

The trace-store decision at the 1,200-agent scale is the one AWS gets wrong by default (they ship OpenSearch because they own it, not because it is the best fit), and the one Coinbase gets right (they chose Tempo for the retention math). The three-way comparison, with real numbers from July 2026 price sheets:

Dimension	Tempo	ClickHouse (self-hosted)	OpenSearch
Cost @ 1,200 agents, 30d retention	~$8K/mo	~$3K/mo (single r6i.2xlarge)	~$24K/mo (3-node cluster)
p99 trace search latency	1.8s	280ms	640ms
Schema flexibility for new OTel attributes	High (columnar JSON)	Highest (DDL on demand)	Medium (mappings, slower rollouts)
Operational complexity	Low (Grafana-managed)	High (you own backups, upgrades, replication)	Medium (managed options exist)
Best for	Teams already on Grafana, <90d retention	Cost-sensitive, >90d retention, want query power	AWS-native shops, full-text search across prompts

The Coinbase choice of Tempo makes sense because their use case is "show me the agent's trace for this user session right now" with 30-day retention — Tempo's exactly that workload. If you need prompt-body full-text search ("which agents were jailbroken via this prompt pattern?"), OpenSearch is the better fit. If you need 90+ day retention at the lowest possible cost, ClickHouse wins on both axes.

The Quality/Reliability/Efficiency taxonomy that AWS publishes

The taxonomy that surfaces the right alerts for the right failure modes — the one Amazon's AgentCore team uses, and the one Coinbase adopted after seeing the July 7 article — has three buckets:

Quality

• Agent-output evaluation score (LLM-as-judge or human-spot-check) below threshold
• Hallucination rate per agent above threshold
• Tool-call success rate below threshold (quality of the MCP server's response, not the call succeeding at the network level)

Reliability

• Loop iteration count above threshold (the agent is re-planning without making progress)
• Tool-call timeout rate per MCP server above threshold
• Context-window overflow rate (the agent hit the model's context limit mid-iteration)
• Cold-start p99 above threshold (for serverless-style agents on agent-substrate)

Efficiency

• Per-agent cost per day above budget
• Token-discard rate above threshold (the agent generated tokens that were thrown away by the loop control)
• Cache hit rate below threshold (the prompt cache is misconfigured, costing real money)

For a 1,200-agent fleet, the alert routing is what makes the difference between an actionable monitoring system and an alert storm. The Coinbase pattern is to route by agent identity: the agent's team lead gets the Quality and Reliability alerts; the platform team gets the Efficiency alerts. The Quality and Reliability alerts are about what the agent did, which the team lead has context for. The Efficiency alerts are about how much it cost, which is a platform-level concern.

What AWS AgentCore does that OSS doesn't

Bedrock AgentCore Observability is the AWS-managed version of the OTel-native pattern, and three things it adds that the OSS stack does not have:

1. Built-in prompt-body capture with PII redaction. AgentCore applies a managed redaction pipeline before the prompt is sent to the trace store. The OSS equivalent is a Collector processor that uses a regex set against the most common PII patterns; AgentCore's is a managed ML model. The cost difference is meaningful: a managed PII-redaction model is ~$2K/month at the 1,200-agent scale; the OSS regex set misses ~15% of PII patterns per Coinbase's internal measurement.
2. Cross-agent attribution via the AgentCore ID. When two agents on the same AgentCore deployment invoke each other, the platform automatically threads the parent-agent identity through the child-agent trace. The OSS equivalent requires explicit W3C trace-context propagation between the two agent runtimes, which works but is easy to get wrong (one missed header and you lose the cross-agent attribution).
3. Cost dashboards pre-built against Bedrock model pricing. If every model in your fleet is a Bedrock model, the per-agent cost dashboard is pre-wired; if you run Anthropic direct + Bedrock + OpenAI, you wire it yourself. Coinbase's pattern handles the multi-vendor case because they have to.

The interesting design question for a platform team in mid-2026 is whether to go all-in on Bedrock AgentCore or to run the OSS OTel-native stack with a paid PII-redaction SaaS in front of it. For teams that are already on Bedrock, the AgentCore answer is almost certainly cheaper — the cost of the managed PII model plus the cross-agent attribution is more than the operational cost of running the OSS equivalent. For teams that have made a multi-vendor commitment (which is most platform teams I talk to), the OSS stack is the right answer because the AgentCore pre-built cost dashboards do not work outside Bedrock.

What I would skip

Two things we tried at the 800-agent mark that did not earn their place:

• Per-step LLM-as-judge evaluation in production. The temptation is to evaluate every agent step against a rubric; in practice the latency cost of the judge call is larger than the step itself, and the false-positive rate on a single step is too high to alert on. Evaluations belong in CI/CD and on a sampled 1% of production steps, not on every step.
• Cross-agent attribution for short-lived (<5s) invocations. The propagation works, but the resulting trace graph is so noisy that it does not produce useful alerts. For short-lived invocations, the per-agent dashboard is more useful than the cross-agent graph.

Conclusion: the reference architecture is set

The July 7 articles from The New Stack are the closing of the agent-observability reference architecture for the second half of 2026. OTel gen_ai.* for agent loops, with the per-step tool-call + finish-reason attributes, is the schema. Tempo or ClickHouse (not OpenSearch, unless you need prompt-body full-text search) is the trace store. The Quality/Reliability/Efficiency taxonomy is the alert routing. The Coinbase pattern of tail sampling plus a per-team cost-control layer is the operational discipline.

For a platform team rolling out agent observability in the next quarter, the actionable stack is: copy the OTel attributes verbatim, pick Tempo or ClickHouse based on the retention math, build the three-bucket alert taxonomy, and wire the per-team cost-control layer before the bill surprises you. The AWS pattern is the canonical reference. The Coinbase pattern is the canonical implementation. The OSS stack in this article is the canonical vendor-neutral path.