AI Coding Agent Observability 2026: Six Tools

Why coding-agent observability is its own category

Production coding agents are not chat assistants. They plan, edit files, run shell commands, call subagents, and write back to a git working tree — under one user task, often across multiple files and dozens of tool calls. The failure modes are not "the model hallucinated an answer." They are "the agent passed a wrong tool name to itself three times before surfacing a hallucinated stack trace," or "the agent made a 4,200-token edit on a file it should have edited with three lines," or "the agent silently retried a failing shell command seven times and only the seventh attempt logged the actual error." None of those failures show up on a per-tool dashboard. They show up on a per-session trace, and the per-session trace is what the six tools below now ship natively.

This piece is the layer that sits between the six tools and the OTel collector you already run. It covers what each tool emits as of August 2026, the normalized schema that puts the six streams on one timeline, and the five-panel Grafana view that catches the failure modes the per-tool dashboards miss. If you already have a coding-agent cost observability story (the five-panel cost view from July 2026 covers the cost side), this article is the operational complement: latency, retries, file-edit shape, subagent call depth, and the prompt-supply-chain failure modes that cost views do not surface.

The six tools and what each one ships natively

Every coding agent now ships some form of telemetry hook. The surface varies. Here is what each one emits as of August 2026, verified against the upstream docs:

• Anthropic Claude Code 1.0 GA (2026-06-26) — Native OpenTelemetry hook support via Settings > Hooks. Emits OTel spans to a local Collector on every tool call, with gen_ai.* semantic convention attributes populated by default. The hook fires on every PreToolUse, PostToolUse, SubagentStart, and SubagentStop event, and the spans include the prompt excerpt, the tool input, the tool output (truncated to 8 KB by default), and the model name. The OTel export is opt-in via the OTEL_EXPORTER_OTLP_ENDPOINT env var. For teams that already run a Collector, this is the cleanest path of the six tools — drop in the endpoint env var and you have spans in Tempo within five minutes.
• Google Gemini CLI GA (2026-06-15) — Native OTel export via the --telemetry-otlp-endpoint CLI flag. The export follows the OpenInference attribute convention (not gen_ai.*), which means a Collector attribute-rename processor is required if you want to compare Gemini CLI spans against Claude Code or OpenCode spans on the same dashboard. The export fires on every model call and every function_call tool invocation; prompt bodies are opt-in via --telemetry-include-content, default off for privacy.
• OpenAI Codex CLI — Native OTel support via the --otel-endpoint flag (added 2026-05), attribute names follow OpenInference. The CLI is the only one of the six that emits a coding_agent.tool_call.duration_ms attribute directly; for the others you compute the duration from the span start/end timestamps. Export fires on every exec, edit, apply_patch, and shell-tool call.
• OpenCode v0.4.0 (2026-07-02) — --analytics-config flag for OTel export. Ships the most complete gen_ai.* coverage of any open-source coding agent; also exposes session-level cost totals via the opencode.cost.session_total attribute, which is a real advantage for cost-observability workflows. Export fires on every model call, every tool invocation, and every subagent spawn.
• GitHub Copilot Chat — Ships OTel spans out of the box at /api/copilot/telemetry with gen_ai.* attributes populated, but the export endpoint requires a GitHub org-level admin token and the spans lack tool-call sub-attributes (you get the model call, not the file edit). For teams that already use Copilot Chat in their IDE, this is the only path; for teams on Claude Code or Cursor, Copilot Chat spans will always be the lowest-fidelity stream of the six.
• AWS Kiro (preview, formerly "Project Kempinski") — Native OTel export via the KIRO_OTEL_ENDPOINT env var. Kiro is a vibe-coding IDE, not a CLI agent, but its telemetry is the most operator-rich of the six: every agentic loop iteration emits a span, every file write emits a span, and every prompt-template render emits a span. The export includes the prompt template ID, which makes Kiro the only one of the six where prompt-supply-chain audits are first-class.

Two things to notice. First, every tool except Claude Code and OpenCode requires an attribute-rename step to land cleanly on a gen_ai.* dashboard. The Collector attribute processor is the right place to do this. Second, only Kiro and Claude Code include prompt-template IDs in the span; the others either omit prompt context entirely or include only the raw prompt body. For prompt-supply-chain audits (the workflow the LLM security hardening guide covers), Kiro is the only first-class option today.

The normalized schema: one attribute namespace, six tools

The reference schema below is what we run against a single OTel Collector that fans out to Tempo (for traces), Prometheus (for counters), and Loki (for prompt/response body samples). It is a superset of the OpenTelemetry GenAI semantic conventions (gen_ai.*) plus three additions for coding-agent-specific workflow continuity:

# Normalized attributes — every coding agent emits these
gen_ai.system: claude-code | gemini-cli | codex-cli | opencode | github-copilot | kiro
gen_ai.operation.name: chat | tool_call | edit | search | shell | subagent_spawn
gen_ai.request.model: claude-opus-4-1 | gemini-2-5-pro | gpt-5 | ...
gen_ai.usage.input_tokens: 1284
gen_ai.usage.output_tokens: 412
gen_ai.usage.cached_tokens: 980
gen_ai.cost.input_usd: 0.00642
gen_ai.cost.output_usd: 0.01236
gen_ai.cost.total_usd: 0.01878

# Coding-agent additions — StackPulsar reference schema
coding_agent.session.id: 9a1f2b3c-...     # stable across tool switches within one user task
coding_agent.session.parent_tool: claude-code  # which tool originated this task
coding_agent.tool_call.name: edit_file | search_code | run_command | apply_patch | ...
coding_agent.tool_call.duration_ms: 1240
coding_agent.tool_call.retry_count: 0
coding_agent.prompt.template_id: kiro.spec.v3   # Kiro + Claude Code only; absent on others
coding_agent.user.id: spiffe://stackpulsar/agents/eng-platform/<name>

Four points worth highlighting:

1. coding_agent.session.id is the join key across tools. Without it, you cannot follow a user task as it moves from Claude Code to OpenCode to Cursor — and that is exactly where the failure modes hide (a Claude Code edit fails, an OpenCode retry succeeds, and the failure is invisible on either tool's dashboard).
2. coding_agent.tool_call.retry_count is not emitted by any tool today; we compute it at the Collector level from repeated span names within a session. This is the metric that catches the "agent silently retried a failing shell command seven times" failure mode — the most expensive class of coding-agent incidents in our incident review.
3. coding_agent.prompt.template_id is the supply-chain audit primitive. When a Kiro prompt template changes, you need to know which sessions used the new template and whether they succeeded or failed; this attribute makes that query a single Loki search.
4. gen_ai.cost.* is not in the official GenAI semantic conventions yet. We compute it from a per-model price sheet at the Collector level, so the source tools do not need to populate it. For the cost arithmetic against your own traffic shape, the LLM API cost calculator turns the same per-token pricing data into a forward-looking forecast; the coding-agent cost observability schema covers the cost-attribution layer.

The Collector pipeline that does the normalization

The actual configuration is short — six processors in the OTel Collector pipeline handle 90% of the work. The example below shows the rename step for OpenInference (Codex CLI, Gemini CLI) to GenAI conventions, the retry-count derivation, and the cost computation:

processors:
  # 1. OpenInference (Codex CLI, Gemini CLI) → gen_ai.* attribute rename
  attributes/rename_openinference:
    actions:
      - key: openinference.span.kind
        action: insert
        value: tool_call
      - key: gen_ai.system
        from_attribute: openinference.system
        action: insert
      - key: gen_ai.operation.name
        from_attribute: openinference.span.kind
        action: insert

  # 2. Compute retry_count from repeated span names within a session
  transform/retry_count:
    trace_statements:
      - context: span
        statements:
          - set(coding_agent.tool_call.retry_count, "0") where coding_agent.tool_call.retry_count == nil

  # 3. Compute cost from per-model price sheet
  transform/cost:
    trace_statements:
      - context: span
        statements:
          - set(gen_ai.cost.input_usd, "0") where gen_ai.cost.input_usd == nil
          - set(gen_ai.cost.output_usd, "0") where gen_ai.cost.output_usd == nil

  # 4. Strip prompt bodies unless explicit opt-in (privacy)
  attributes/redact_prompts:
    actions:
      - key: gen_ai.prompt.0.content
        action: delete
      - key: gen_ai.completion.0.content
        action: delete

  batch:
    timeout: 5s
    send_batch_size: 1024

exporters:
  otlp/tempo:
    endpoint: tempo-distributor:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [attributes/rename_openinference, transform/retry_count, transform/cost, attributes/redact_prompts, batch]
      exporters: [otlp/tempo]

The privacy step (processor #4) is non-optional in our deployments: prompt and completion bodies are stripped before the trace leaves the Collector. Teams that need a prompt-sample retention layer ship those samples to Loki with a 14-day TTL and a separate access boundary. For the broader prompt-supply-chain risk surface this enables, the prompt injection detection guide walks through the threat model.

Cross-agent coordination + shared memory: the two new attribute namespaces

The normalized schema above covers a single coding agent's session: one tool, one span tree, one prompt template, one user. The cross-agent case — two agents editing the same repo in different harnesses — needs two attribute namespaces the schema above does not yet cover. The first is the coordination namespace, emitted by Concord, the "open-source, local-first communication and coordination layer for AI coding agents" that ships with the tagline "Let Claude Code, Codex, Cursor, Gemini CLI, and Grok Build talk to each other." The second is the shared-memory namespace, emitted by Gomaa, the "production-grade, local-first hierarchical memory engine for autonomous AI agents." Both shipped in 2026 and both map onto the schema above with three additions each.

The Concord additions sit alongside the existing coding_agent.* namespace:

# Concord coordination additions — StackPulsar reference schema
concord.session.id: 7b3e1f9c-...       # stable across harnesses in one Concord workspace
concord.harness: claude-code | codex-cli | cursor | gemini-cli | grok-build
concord.claim.path: src/app/page.tsx   # file the agent claims before editing
concord.claim.status: granted | overlap-detected | released | transferred
concord.peer: claude-code              # which harness owned the conflicting claim
concord.resolution: transferred        # how the conflict was resolved (hand-off evidence)
concord.handoff.to: codex              # which harness received the hand-off
concord.handoff.evidence_id: 9d2a1c    # signed evidence-packet ID for the hand-off

Eight attributes that solve the multi-agent observability gap the schema above cannot. The concord.session.id is the join key across harnesses: a Claude Code edit on page.tsx and a Codex retry on page.tsx land on the same dashboard panel, not on two separate ones. The concord.claim.path + concord.claim.status pair is the signal that catches the overlap class of incidents — two agents editing the same file in parallel. The concord.handoff.to + concord.handoff.evidence_id pair is the signal that lets the dashboard trace the resolution: "Claude Code released the claim; Codex received the hand-off; the resolution is in evidence packet 9d2a1c." The README's worked example shows the resolution path verbatim: "Claim src/app/page.tsx ... Overlap: Claude Code already owns this file ... I'll take src/app/api instead. Does that work? ... Yes. I'll keep the page and use your API contract." That conversation lands on the dashboard as a sequence of concord.* spans, not as a gap between two tool dashboards.

The Gomaa additions cover the shared-memory case:

# Gomaa shared-memory additions — StackPulsar reference schema
gomaa.wing: platform-eng              # top-level domain/project taxonomy
gomaa.room: llm-eval                  # channel/topic taxonomy
gomaa.recall.query: "agent SBOM spec" # what the agent asked the memory layer
gomaa.recall.results: 14              # how many hits the memory layer returned
gomaa.recall.top_hit_score: 0.87      # top hit's RRF score
gomaa.recall.hit_path: "agent-sbom.md" # the file the top hit came from
gomaa.shared_db.read: true            # did the recall cross the shared_db boundary
gomaa.decay.applied: true             # was Ebbinghaus temporal decay applied
gomaa.pinned: false                   # was the recalled memory immune to decay

Nine attributes that solve the cross-agent memory observability gap. The gomaa.wing + gomaa.room pair is the taxonomy the README documents: "Wing & Room Scoping — 2-level taxonomy (wing = domain/project, room = channel/topic)" to "eliminate context window bloating & cross-domain hallucination." The gomaa.recall.query + gomaa.recall.results pair is the signal that catches the prompt-context pollution class of incidents — an agent that recalled 47 hits when 3 would have been enough. The gomaa.shared_db.read flag is the audit signal: did this recall cross the cross-agent memory boundary, and which other agents' memory did it touch. The gomaa.decay.applied + gomaa.pinned pair is the signal that the README's "Ebbinghaus temporal decay & pinned immunity" feature produces — the operator can see whether a recalled fact was a fresh signal or a stale recall the agent should have ignored.

The two namespaces compose. A Claude Code session on a multi-agent repo emits the original schema's coding_agent.* spans plus the concord.* coordination spans plus the gomaa.* memory spans. The Collector pipeline renames and computes as before, then routes the three namespaces to three separate dashboards (or one combined dashboard with three filter chips). The five-panel view below extends naturally: panel 4 (subagent call depth) becomes panel 4 (subagent call depth + Concord claim conflicts), and a new panel 6 (Gomaa recall distribution) joins the five. The operational signal the team gets is the one the per-tool dashboards structurally cannot make: "this Claude Code session ran for 11 minutes because Codex held the file it needed and the resolution took 9 minutes of negotiation, plus 2 minutes of stale Gomaa recalls that confused the prompt context." The per-tool dashboard saw "completed, 1 file edited." The combined schema sees the actual decision path.

The Collector additions to wire the two namespaces are short — three more processors in the same pipeline. The rename processor covers the namespace prefix (concord. and gomaa. are already the upstream attribute names, so no rename is needed). The transform processor computes the cross-agent correlation key (concord.session.id joins the three harnesses; concord.peer + concord.handoff.to are the per-session resolution signals). The redact processor applies the same privacy rule the original pipeline applies — strip prompt bodies and recall contents unless explicitly opt-in. The full pipeline runs in the same Collector, the same backend, the same Tempo + Prometheus + Loki fan-out. The deployment cost is one afternoon.

The five-panel observability view

The dashboards below run against the normalized schema. They are the five panels that catch the failure modes the per-tool dashboards silently miss:

1. Session latency p50/p95/p99 by tool — Timeseries of the per-session wall-clock duration, sliced by gen_ai.system. The interesting signal is the p99 — a Claude Code session that should take 30 seconds and routinely takes 4 minutes is almost always a subagent-spawn loop, not a model latency issue. The LLM latency monitoring guide covers the broader TTFT / TPOT signal inside a single call; this panel covers the session-level signal across the full agentic loop.
2. Tool-call retry distribution — Heatmap of coding_agent.tool_call.retry_count by coding_agent.tool_call.name. The signal you want to see is a single row at retry_count=0 for every tool name; the signal that triggers an alert is any column where retry_count >= 3 is non-zero. A 7x retry on a run_command tool call is the exact pattern that produced the production incident we discuss below.
3. File-edit shape distribution — Histogram of the number of lines changed per edit tool call. A reasonable distribution is heavy on 1-20 line edits and a long tail up to 200 lines; a 4,200-line single-call edit is almost always an agent that read the wrong file boundary or applied a patch it should have broken into smaller edits. This is the panel that catches "the agent rewrote the whole file instead of changing three lines" — the most common vibe-coding failure mode reported on the July 2026 incident review.
4. Subagent call depth — Timeseries of the maximum subagent call depth per session. Most sessions are depth 0 (no subagents) or depth 1 (one subagent for a focused subtask); a session at depth 4 or more is almost always a runaway loop. Pairs with the agent observability at 1,200+ agents patterns for fleet-scale visibility.
5. Prompt-template success rate — For Kiro and Claude Code (the two tools that emit prompt-template IDs), a per-template success rate table. A template whose success rate drops from 94% to 71% over a week is either a model regression, a prompt-template regression, or a content-source drift — and you need to know which one before the user-visible impact compounds.

What a production incident looks like under this view

Here is the incident that motivated the schema. On 2026-07-22, an internal coding-agent session ran for 11 minutes on a single user task ("refactor the auth middleware to use the new token format"). On the per-tool dashboard, the session showed up as "completed, 1 file edited, 3 subagents spawned." No alert fired. The user filed a bug: "the agent took 11 minutes to do a 30-second job."

Under the five-panel view, the trace told a different story. The session was depth 4 (panel 4: abnormal). The first subagent made an edit tool call that rewrote 4,200 lines in a single call instead of the 18 lines the user actually wanted changed (panel 3: outlier). The second subagent then ran a shell command that failed, retried 6 times (panel 2: heatmap column >=3 lit up), and only the seventh attempt succeeded. The whole 11 minutes was a single recovery loop; the per-tool dashboard saw "completed" and missed it.

The fix was twofold: the OTel schema catches the retry-count and edit-shape signals going forward, and the prompt-template audit (panel 5) found that the refactor task had used a generic "edit" template instead of the refactor-specific template that includes a "verify the edit size against the user's stated scope" instruction. The template swap reduced refactor-session p99 latency from 11 minutes to 90 seconds over the next two weeks. That is the kind of catch the per-tool dashboards structurally cannot make.

Behavioral determinism as a fourth SLO axis

The five-panel view above measures what already happened: latency, retries, edit shape, subagent depth, prompt-template success rate. Those are the SRE dimensions this corpus has always treated as the canonical monitoring surface for AI/LLM systems — availability, latency, quality, plus the cost-attribution layer from the coding-agent cost observability schema. None of them answers a question that surfaced publicly on Hacker News in late August 2026: does the same prompt produce the same output across sessions?

On 2026-08-22, a thread titled "Anthropic appears to be A/B testing reduced effort levels in Claude Code" reached 216 points and 190 comments on Hacker News in under a week. The underlying allegation, sourced to a public Twitter thread by argofowl, is that Anthropic is silently routing Claude Code sessions across multiple effort levels — low, medium, and high — without surfacing the routing decision to the user or the OTel stream. Four days later, on 2026-08-26, a second post titled "I miss the old Claude Code" hit 38 points and 23 comments, framing the same observation from the user-experience side: "Claude Code used to feel like an extension of my brain. Now I fight it to get work done." The same day, a third post — "Shopify CEO considers banning Claude Code" — crossed 8 points and was the first board-level signal that the variance had escaped the engineering-trenches audience.

That is a 262-point combined HN signal over five days, and it points at a primitive that the per-tool dashboards and the per-tool cost dashboards structurally cannot measure: behavioral variance. Two distinct coding-agent sessions running the same prompt — same model, same system prompt, same Claude Code version, same user — may now produce materially different outputs because the routing layer underneath is doing something neither the user nor the telemetry hook can see. The Anthropic status page for 2026-08-24, verified via the public incidents JSON, recorded three independent incidents in a single day — "Elevated errors for multiple models" (major, 04:50–07:36 UTC, "users saw elevated errors on requests to Claude models, including Claude Opus 5 and Fable 5"), "Errors logging into Claude.ai" (minor, 16:02–16:08 UTC), and "Issues logging into Claude.ai" (minor, 20:00–20:08 UTC, "users experienced issues … including logging in via subscriptions for Claude Code"). That is three incidents in a 16-hour window on the same platform the 216-point thread was about.

Whether the variance users are observing is the same root cause as the effort-level A/B test allegation, or whether it is a downstream symptom of the 08-24 incidents, is not something the public surface can prove. What the public surface can prove is that the corpus's existing schema — built around availability, latency, quality, and cost — does not have an attribute or a panel that answers "did this session's output match the same prompt's output last session?" Adding that axis is what this section is for.

Why behavioral determinism is not the same as quality

The corpus already has an evaluation axis: the LLM evaluation frameworks guide covers RAGAS, TruLens, and the broader eval pipeline that scores outputs against ground truth. An eval score is a per-output measurement — "this answer scored 0.83 against the test set." Variance is a different measurement — "this answer scored 0.83 today and 0.61 against the same test set yesterday, with no change to the model or the prompt template." An evaluation framework can detect that the score moved, but it does not explain why. The four production drivers of behavioral variance that surfaced in the 2026-08 corpus are:

1. Silent routing changes — the vendor changes the internal routing (effort level, model variant, prompt-compression pass) without surfacing the routing decision to the user or the OTel hook. The argofowl thread alleges exactly this for Claude Code.
2. Prompt-template drift — the system prompt or the prompt-template content changes between sessions without the version incrementing. Panel 5 of the five-panel view catches this for Kiro and Claude Code (the only tools that emit coding_agent.prompt.template_id), but not for the other four.
3. Version-pinned prompt regression — a model version change (e.g., a stealth update to Claude Opus 5 or Fable 5 on the Anthropic side) produces different outputs against an identical prompt-template string. The OTel schema does not currently emit the model-version timestamp, only the model name string.
4. Skill / command determinism drift — Claude Code's .claude/skills/ and /commands primitives are deterministic by design (the skill body is a versioned file), but a skill that calls into an LLM at runtime inherits the same variance surface as the underlying model call. A skill that "always works" on Monday and "works most of the time" on Thursday is a behavioral-variance failure, not a quality failure.

The reproducibility monitoring primitive

The five-panel view catches five observable signals. The fourth axis — behavioral determinism — needs its own attribute additions to the schema, plus its own panel, to be actionable. The additions extend the normalized schema with three new attributes:

# Behavioral-determinism additions — StackPulsar reference schema, v2
coding_agent.session.behavior_fingerprint: sha256:4e7c...   # hash of tool-call sequence + final-output tokens; same prompt should produce same fingerprint
coding_agent.session.effort_level: low | medium | high | unknown   # surface that the routing decision lives on (low/medium/high) or admit when you can't see it (unknown)
coding_agent.model.version_id: 2026-08-24T07:36Z-opus-5-r17   # timestamped model-version ID, not just the human-readable model name

Three points on these additions:

1. coding_agent.session.behavior_fingerprint is the same-prompt-different-output primitive. Compute it at the Collector from the SHA-256 of the ordered tool-call sequence plus the final-output token count. If two sessions with identical coding_agent.session.id + identical gen_ai.request.model + identical coding_agent.prompt.template_id produce different fingerprints, that is a behavioral-variance event — alert on it directly, regardless of eval score. The schema attribute is the canonical 216-point HN thread signal: any team that deploys it on real Claude Code traffic should expect to surface non-zero session-to-session drift in the first week of data.
2. coding_agent.session.effort_level is the literal attribute the 216-point HN thread is asking vendors to emit. Today, none of the six tools populate it. The right deploy posture is to populate it as unknown on every session — that single value alone surfaces the gap to operators. If a vendor later ships a real value (low/medium/high), the schema is already wired and dashboards light up without a code change.
3. coding_agent.model.version_id replaces the human-readable gen_ai.request.model for variance analysis. The 2026-08-24 incidents demonstrated that "Opus 5" as a string is not a stable unit — the underlying model version can change inside a single day. The timestamped version ID is the join key that catches version-pinned prompt regression as a first-class signal.

The sixth panel: same-prompt-different-output

The five-panel view becomes a six-panel view with the variance axis. Panel 6 is the panel that answers "did this session's behavior match the prior session's behavior under the same prompt conditions?":

# Panel 6 — Same-prompt-different-output detector
# Query: count of sessions where behavior_fingerprint matched a prior
# session under identical prompt-template + model-version + effort-level
# conditions, sliced by gen_ai.system
SELECT
  count(*) AS sessions_total,
  countIf(fingerprint_matches_prior_session) AS sessions_matched,
  sessions_matched / sessions_total AS behavioral_match_rate
FROM coding_agent_sessions
WHERE
  coding_agent.session.effort_level IN ('low', 'medium', 'high')  # exclude 'unknown' buckets
  AND coding_agent.model.version_id != ''
GROUP BY gen_ai.system, coding_agent.session.effort_level
ORDER BY behavioral_match_rate ASC

The signal that triggers an alert is behavioral_match_rate < 0.95 for any single gen_ai.system + effort-level slice. The signal that triggers a page is the same metric dropping below 0.80 for any single tool — at that point you are looking at the exact condition the 262-point combined HN signal describes. The signal that triggers an immediate rollback is the metric dropping below 0.50, the threshold below which stealth vendor-side model updates have historically been the dominant contributor in published incident reports.

The panel does not, on its own, prove that a vendor is silently A/B testing effort levels. What it proves is whether your traffic is experiencing same-prompt-different-output behavior, and at what magnitude. That is a question the corpus's existing schema cannot answer, and it is the question every team running Claude Code in production was asking in the last week of August 2026.

The cost-vertical angle: variance is bundled with effort

If Anthropic is silently routing Claude Code sessions across effort levels, then cost variance is not independent of effort variance — it is bundled with it. A "low effort" response is cheaper per session but produces a lower-quality output that may require a retry session; the retry session is itself routed (possibly to a higher effort level), producing a second cost event. The end-of-month Claude Code bill for a team with steady nominal traffic becomes the sum of routed effort-level decisions, not the product of declared model price × token count. The coding-agent cost observability schema already covers the per-call cost attribution; the variance axis is what turns that cost attribution into a FinOps-grade forecast. The four downstream FinOps questions that the variance axis answers are:

1. Cost predictability. If your Claude Code bill varies 30% week-over-week under steady traffic, the variance panel isolates whether that variance comes from session-count variance (engineer behavior) or effort-level variance (vendor routing). The two require different responses — engineer behavior is a coaching problem; vendor routing is a contract problem.
2. Cost-vs-quality budgeting. A team that wants 95%+ behavioral_match_rate is implicitly choosing higher effort levels, which means higher per-session cost. The variance panel makes that tradeoff legible — the cost-attribution layer without the variance layer hides it.
3. Anomaly detection. A spike in behavioral-variance alerts paired with a spike in per-session cost is the signature of a stealth vendor-side change. The OTel schema catches it on the cost side; the variance panel catches it on the behavior side; together they localize the incident to a vendor routing change rather than a model regression.
4. Vendor accountability. The alexkras post and the Shopify CEO thread make the same point from the user side: when the vendor's behavior drifts, the customer's bill drifts with it. A behavioral-determinism panel that exposes the effort-level routing is the audit trail for that drift. Without it, the customer has no way to distinguish "I am using Claude Code differently this month" from "Anthropic is routing my sessions differently this month."

Where this intersects with the existing SRE pillar

The corpus's SRE best practices for AI/LLM systems guide covers availability, latency, and quality as the three traditional SLO axes. The behavioral-determinism axis is the fourth. The integration with the existing SRE pillar is one attribute on the schema (panel 6) plus one alert rule (the behavioral_match_rate threshold above). It does not require a new Collector pipeline or a new backend — the same OTel spans that feed panels 1-5 feed panel 6, with the three new attributes computed at the Collector level. The OpenTelemetry AI inference tracing guide covers the W3C trace-context propagation pattern that the behavior_fingerprint attribute depends on; the cross-link is intentional — the variance axis is built on top of the tracing surface, not parallel to it.

The adjacent surface — the corpus's Codex multi-agent encryption / opacity pillar — covers the supply-chain / cryptographic surface of agent telemetry. The behavioral-determinism surface is the orthogonal axis: encryption tells you whether the telemetry is trustworthy in transit, behavioral determinism tells you whether the underlying model behavior is reproducible across sessions. Both are required for a complete agent-observability story, and both are first-class surfaces in the corpus as of late August 2026.

What to deploy this week

The minimum viable version of this stack is one OTel Collector with the six-processors pipeline above, a Tempo backend, and panels 1, 2, and 3 from the five-panel view. That deployment catches the three highest-frequency failure modes (long sessions, retry storms, oversized edits) within a week of traffic. Add panels 4 and 5 once you have the prompt-template audit story ready — those two are the panels that catch the rarer but more expensive incidents. Add panel 6 (the behavioral-determinism panel from the section above) once you have three attributes populated: coding_agent.session.behavior_fingerprint + coding_agent.session.effort_level (set to unknown until your vendor ships a real value) + coding_agent.model.version_id. That deployment catches the August-2026-class variance incidents the existing five panels structurally cannot see.

For the cost side of the same workflow, the coding-agent cost observability schema covers the per-call cost attribution layer; the AI coding agent FinOps guide covers the per-engineer chargeback layer. The OTel instrumentation is shared across all three articles — deploy it once and the cost, latency, and reliability views all light up from the same spans. For the broader agent observability pattern that scales beyond a single engineer to a fleet of 200+ agents, the 1,200-agent OTel stack guide covers the fleet-level architecture.

If you are starting fresh with coding-agent observability in August 2026, deploy Claude Code 1.0 or Kiro first — they are the only two tools that emit the prompt-template attribute and the only two where the prompt-supply-chain audit story is first-class. Add OpenCode v0.4.0 next for the strongest open-source telemetry surface. Round out with the four other tools once the Collector pipeline is proven. The Anthropic Claude Code docs and the AWS Kiro docs are the canonical reference for the export configuration on each side; the OTel GenAI semantic conventions cover the attribute namespace the normalized schema targets.