Why coding-agent observability is its own category

Production coding agents are not chat assistants. They plan, edit files, run shell commands, call subagents, and write back to a git working tree — under one user task, often across multiple files and dozens of tool calls. The failure modes are not "the model hallucinated an answer." They are "the agent passed a wrong tool name to itself three times before surfacing a hallucinated stack trace," or "the agent made a 4,200-token edit on a file it should have edited with three lines," or "the agent silently retried a failing shell command seven times and only the seventh attempt logged the actual error." None of those failures show up on a per-tool dashboard. They show up on a per-session trace, and the per-session trace is what the six tools below now ship natively.

This piece is the layer that sits between the six tools and the OTel collector you already run. It covers what each tool emits as of August 2026, the normalized schema that puts the six streams on one timeline, and the five-panel Grafana view that catches the failure modes the per-tool dashboards miss. If you already have a coding-agent cost observability story (the five-panel cost view from July 2026 covers the cost side), this article is the operational complement: latency, retries, file-edit shape, subagent call depth, and the prompt-supply-chain failure modes that cost views do not surface.

Advertisement
Advertisement

The six tools and what each one ships natively

Every coding agent now ships some form of telemetry hook. The surface varies. Here is what each one emits as of August 2026, verified against the upstream docs:

  • Anthropic Claude Code 1.0 GA (2026-06-26) — Native OpenTelemetry hook support via Settings > Hooks. Emits OTel spans to a local Collector on every tool call, with gen_ai.* semantic convention attributes populated by default. The hook fires on every PreToolUse, PostToolUse, SubagentStart, and SubagentStop event, and the spans include the prompt excerpt, the tool input, the tool output (truncated to 8 KB by default), and the model name. The OTel export is opt-in via the OTEL_EXPORTER_OTLP_ENDPOINT env var. For teams that already run a Collector, this is the cleanest path of the six tools — drop in the endpoint env var and you have spans in Tempo within five minutes.
  • Google Gemini CLI GA (2026-06-15) — Native OTel export via the --telemetry-otlp-endpoint CLI flag. The export follows the OpenInference attribute convention (not gen_ai.*), which means a Collector attribute-rename processor is required if you want to compare Gemini CLI spans against Claude Code or OpenCode spans on the same dashboard. The export fires on every model call and every function_call tool invocation; prompt bodies are opt-in via --telemetry-include-content, default off for privacy.
  • OpenAI Codex CLI — Native OTel support via the --otel-endpoint flag (added 2026-05), attribute names follow OpenInference. The CLI is the only one of the six that emits a coding_agent.tool_call.duration_ms attribute directly; for the others you compute the duration from the span start/end timestamps. Export fires on every exec, edit, apply_patch, and shell-tool call.
  • OpenCode v0.4.0 (2026-07-02) — --analytics-config flag for OTel export. Ships the most complete gen_ai.* coverage of any open-source coding agent; also exposes session-level cost totals via the opencode.cost.session_total attribute, which is a real advantage for cost-observability workflows. Export fires on every model call, every tool invocation, and every subagent spawn.
  • GitHub Copilot Chat — Ships OTel spans out of the box at /api/copilot/telemetry with gen_ai.* attributes populated, but the export endpoint requires a GitHub org-level admin token and the spans lack tool-call sub-attributes (you get the model call, not the file edit). For teams that already use Copilot Chat in their IDE, this is the only path; for teams on Claude Code or Cursor, Copilot Chat spans will always be the lowest-fidelity stream of the six.
  • AWS Kiro (preview, formerly "Project Kempinski") — Native OTel export via the KIRO_OTEL_ENDPOINT env var. Kiro is a vibe-coding IDE, not a CLI agent, but its telemetry is the most operator-rich of the six: every agentic loop iteration emits a span, every file write emits a span, and every prompt-template render emits a span. The export includes the prompt template ID, which makes Kiro the only one of the six where prompt-supply-chain audits are first-class.

Two things to notice. First, every tool except Claude Code and OpenCode requires an attribute-rename step to land cleanly on a gen_ai.* dashboard. The Collector attribute processor is the right place to do this. Second, only Kiro and Claude Code include prompt-template IDs in the span; the others either omit prompt context entirely or include only the raw prompt body. For prompt-supply-chain audits (the workflow the LLM security hardening guide covers), Kiro is the only first-class option today.

The normalized schema: one attribute namespace, six tools

The reference schema below is what we run against a single OTel Collector that fans out to Tempo (for traces), Prometheus (for counters), and Loki (for prompt/response body samples). It is a superset of the OpenTelemetry GenAI semantic conventions (gen_ai.*) plus three additions for coding-agent-specific workflow continuity:

# Normalized attributes — every coding agent emits these
gen_ai.system: claude-code | gemini-cli | codex-cli | opencode | github-copilot | kiro
gen_ai.operation.name: chat | tool_call | edit | search | shell | subagent_spawn
gen_ai.request.model: claude-opus-4-1 | gemini-2-5-pro | gpt-5 | ...
gen_ai.usage.input_tokens: 1284
gen_ai.usage.output_tokens: 412
gen_ai.usage.cached_tokens: 980
gen_ai.cost.input_usd: 0.00642
gen_ai.cost.output_usd: 0.01236
gen_ai.cost.total_usd: 0.01878

# Coding-agent additions — StackPulsar reference schema
coding_agent.session.id: 9a1f2b3c-...     # stable across tool switches within one user task
coding_agent.session.parent_tool: claude-code  # which tool originated this task
coding_agent.tool_call.name: edit_file | search_code | run_command | apply_patch | ...
coding_agent.tool_call.duration_ms: 1240
coding_agent.tool_call.retry_count: 0
coding_agent.prompt.template_id: kiro.spec.v3   # Kiro + Claude Code only; absent on others
coding_agent.user.id: spiffe://stackpulsar/agents/eng-platform/<name>

Four points worth highlighting:

  1. coding_agent.session.id is the join key across tools. Without it, you cannot follow a user task as it moves from Claude Code to OpenCode to Cursor — and that is exactly where the failure modes hide (a Claude Code edit fails, an OpenCode retry succeeds, and the failure is invisible on either tool's dashboard).
  2. coding_agent.tool_call.retry_count is not emitted by any tool today; we compute it at the Collector level from repeated span names within a session. This is the metric that catches the "agent silently retried a failing shell command seven times" failure mode — the most expensive class of coding-agent incidents in our incident review.
  3. coding_agent.prompt.template_id is the supply-chain audit primitive. When a Kiro prompt template changes, you need to know which sessions used the new template and whether they succeeded or failed; this attribute makes that query a single Loki search.
  4. gen_ai.cost.* is not in the official GenAI semantic conventions yet. We compute it from a per-model price sheet at the Collector level, so the source tools do not need to populate it. For the cost arithmetic against your own traffic shape, the LLM API cost calculator turns the same per-token pricing data into a forward-looking forecast; the coding-agent cost observability schema covers the cost-attribution layer.

The Collector pipeline that does the normalization

The actual configuration is short — six processors in the OTel Collector pipeline handle 90% of the work. The example below shows the rename step for OpenInference (Codex CLI, Gemini CLI) to GenAI conventions, the retry-count derivation, and the cost computation:

processors:
  # 1. OpenInference (Codex CLI, Gemini CLI) → gen_ai.* attribute rename
  attributes/rename_openinference:
    actions:
      - key: openinference.span.kind
        action: insert
        value: tool_call
      - key: gen_ai.system
        from_attribute: openinference.system
        action: insert
      - key: gen_ai.operation.name
        from_attribute: openinference.span.kind
        action: insert

  # 2. Compute retry_count from repeated span names within a session
  transform/retry_count:
    trace_statements:
      - context: span
        statements:
          - set(coding_agent.tool_call.retry_count, "0") where coding_agent.tool_call.retry_count == nil

  # 3. Compute cost from per-model price sheet
  transform/cost:
    trace_statements:
      - context: span
        statements:
          - set(gen_ai.cost.input_usd, "0") where gen_ai.cost.input_usd == nil
          - set(gen_ai.cost.output_usd, "0") where gen_ai.cost.output_usd == nil

  # 4. Strip prompt bodies unless explicit opt-in (privacy)
  attributes/redact_prompts:
    actions:
      - key: gen_ai.prompt.0.content
        action: delete
      - key: gen_ai.completion.0.content
        action: delete

  batch:
    timeout: 5s
    send_batch_size: 1024

exporters:
  otlp/tempo:
    endpoint: tempo-distributor:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [attributes/rename_openinference, transform/retry_count, transform/cost, attributes/redact_prompts, batch]
      exporters: [otlp/tempo]

The privacy step (processor #4) is non-optional in our deployments: prompt and completion bodies are stripped before the trace leaves the Collector. Teams that need a prompt-sample retention layer ship those samples to Loki with a 14-day TTL and a separate access boundary. For the broader prompt-supply-chain risk surface this enables, the prompt injection detection guide walks through the threat model.

The five-panel observability view

The dashboards below run against the normalized schema. They are the five panels that catch the failure modes the per-tool dashboards silently miss:

  1. Session latency p50/p95/p99 by tool — Timeseries of the per-session wall-clock duration, sliced by gen_ai.system. The interesting signal is the p99 — a Claude Code session that should take 30 seconds and routinely takes 4 minutes is almost always a subagent-spawn loop, not a model latency issue. The LLM latency monitoring guide covers the broader TTFT / TPOT signal inside a single call; this panel covers the session-level signal across the full agentic loop.
  2. Tool-call retry distribution — Heatmap of coding_agent.tool_call.retry_count by coding_agent.tool_call.name. The signal you want to see is a single row at retry_count=0 for every tool name; the signal that triggers an alert is any column where retry_count >= 3 is non-zero. A 7x retry on a run_command tool call is the exact pattern that produced the production incident we discuss below.
  3. File-edit shape distribution — Histogram of the number of lines changed per edit tool call. A reasonable distribution is heavy on 1-20 line edits and a long tail up to 200 lines; a 4,200-line single-call edit is almost always an agent that read the wrong file boundary or applied a patch it should have broken into smaller edits. This is the panel that catches "the agent rewrote the whole file instead of changing three lines" — the most common vibe-coding failure mode reported on the July 2026 incident review.
  4. Subagent call depth — Timeseries of the maximum subagent call depth per session. Most sessions are depth 0 (no subagents) or depth 1 (one subagent for a focused subtask); a session at depth 4 or more is almost always a runaway loop. Pairs with the agent observability at 1,200+ agents patterns for fleet-scale visibility.
  5. Prompt-template success rate — For Kiro and Claude Code (the two tools that emit prompt-template IDs), a per-template success rate table. A template whose success rate drops from 94% to 71% over a week is either a model regression, a prompt-template regression, or a content-source drift — and you need to know which one before the user-visible impact compounds.
Advertisement
Advertisement

What a production incident looks like under this view

Here is the incident that motivated the schema. On 2026-07-22, an internal coding-agent session ran for 11 minutes on a single user task ("refactor the auth middleware to use the new token format"). On the per-tool dashboard, the session showed up as "completed, 1 file edited, 3 subagents spawned." No alert fired. The user filed a bug: "the agent took 11 minutes to do a 30-second job."

Under the five-panel view, the trace told a different story. The session was depth 4 (panel 4: abnormal). The first subagent made an edit tool call that rewrote 4,200 lines in a single call instead of the 18 lines the user actually wanted changed (panel 3: outlier). The second subagent then ran a shell command that failed, retried 6 times (panel 2: heatmap column >=3 lit up), and only the seventh attempt succeeded. The whole 11 minutes was a single recovery loop; the per-tool dashboard saw "completed" and missed it.

The fix was twofold: the OTel schema catches the retry-count and edit-shape signals going forward, and the prompt-template audit (panel 5) found that the refactor task had used a generic "edit" template instead of the refactor-specific template that includes a "verify the edit size against the user's stated scope" instruction. The template swap reduced refactor-session p99 latency from 11 minutes to 90 seconds over the next two weeks. That is the kind of catch the per-tool dashboards structurally cannot make.

What to deploy this week

The minimum viable version of this stack is one OTel Collector with the six-processors pipeline above, a Tempo backend, and panels 1, 2, and 3 from the five-panel view. That deployment catches the three highest-frequency failure modes (long sessions, retry storms, oversized edits) within a week of traffic. Add panels 4 and 5 once you have the prompt-template audit story ready — those two are the panels that catch the rarer but more expensive incidents.

For the cost side of the same workflow, the coding-agent cost observability schema covers the per-call cost attribution layer; the AI coding agent FinOps guide covers the per-engineer chargeback layer. The OTel instrumentation is shared across all three articles — deploy it once and the cost, latency, and reliability views all light up from the same spans. For the broader agent observability pattern that scales beyond a single engineer to a fleet of 200+ agents, the 1,200-agent OTel stack guide covers the fleet-level architecture.

If you are starting fresh with coding-agent observability in August 2026, deploy Claude Code 1.0 or Kiro first — they are the only two tools that emit the prompt-template attribute and the only two where the prompt-supply-chain audit story is first-class. Add OpenCode v0.4.0 next for the strongest open-source telemetry surface. Round out with the four other tools once the Collector pipeline is proven. The Anthropic Claude Code docs and the AWS Kiro docs are the canonical reference for the export configuration on each side; the OTel GenAI semantic conventions cover the attribute namespace the normalized schema targets.