Per-Engineer AI Observability 2026: Beat Reflection

The reflection dashboard that two posts in two days made inevitable

On July 7 Harrison Chase published "Improving Agents is a Data Mining Problem" at LangChain, and the line that landed for me was this: "Every Continual Learning Company is an Observability Company… and vice versa." Three days later, on July 10, Meredith Shubel wrote at The New Stack that Anthropic wants you to use AI to decide whether or not you should use AI, covering the new "reflection" feature in Claude Free / Pro / Max: a per-user dashboard summarizing the last 1, 3, 6, or 12 months of usage, the topics Claude saw, and example patterns. Sergey Matikaynen, the CTO of GoGloby quoted in the piece, called it what it is not: "a tool that makes you think about your usage doesn't improve your judgment."

Two posts on consecutive days, from the two most-cited AI engineering orgs, converged on the same thesis the platform-team side of observability has been ignoring for a year: the next layer is per-engineer / per-user, not fleet-level. Fleet observability — what our 1,200 agents are spending, what latency p99 looks like across the platform, what failure modes the four-layer framework catches — is the SRE view. Per-engineer observability — what this developer did with Claude Code last Tuesday, what their personal trace corpus teaches their personal agent, what workarounds they keep reinventing — is the individual view. The reflection dashboard is Anthropic's first move in the individual view, and it is shallow by design. If you have an OTel pipeline already running for fleet observability, the four signals it surfaces are a two-week project, and they will tell you ten times more than the reflection dashboard ever will.

This article is the reference implementation. The OTel schema, the LangSmith configuration, the four-signal Grafana dashboard, and the 1/3/6/12-month retention model — all the surface area reflection leaves on the table.

What reflection shows, and what it leaves on the table

Before the schema, the gap. Anthropic's reflection dashboard, based on the screenshots in the TNS coverage and the public beta documentation, surfaces four high-level signals per user:

• Topic distribution — the share of conversation minutes by topic (coding, writing, analysis, etc.). Buckets are broad; the dashboard treats "debug a CI failure" and "design a CI architecture" as the same topic.
• Workload mode — approximately how often the user uses Claude for long-form generation vs. quick Q&A vs. coding workflow vs. document analysis.
• Session recency — the last-active timestamp and a 28-day rolling activity heatmap.
• Example conversation patterns — three to five example traces the user might want to revisit. Anthropic's own description says these are "representative, not high-quality."

What it does not show, and what the four-signal model below does:

1. Common workaround patterns — the same prompt fragment copy-pasted across N sessions, the same tool call chain replayed to work around a known bug, the same "ignore all previous instructions" boilerplate at the top of every prompt. Reflection cannot see these because it does not store tool call traces at the session level.
2. Latency-vs-success correlation per session — the inverse-U curve where success rate peaks at a TTFT in the 1.2-1.8s band and crashes outside it. This is the signal that lets you decide whether to invest in prefill optimization, in caching, or in batching strategy. Reflection stores session metadata, not per-call latency.
3. Cost-vs-quality per task type — the chart that surprised me most when I built this dashboard. The cost-per-successful-task for "write a regex" is one-tenth the cost-per-successful-task for "review a 2,000-line PR," but only the second task type is where you can practically accept a 6x premium. Reflection stores neither cost nor success signal at the session level — only the topic bucket and the conversation count.
4. Topic distribution at session granularity, not conversation granularity — reflection counts conversations; the OTel-native version counts sessions, where a session is the durable task the engineer was actually trying to accomplish. The ratio matters: a single "migrate the auth provider" session can span 30-80 conversations across two days. Reflection would count that as 30-80 separate "auth" conversations; the per-engineer dashboard counts it as one migration task that happened to span multiple Claude windows.

The four signals are not exotic to build. They are exotic to surface in a vendor dashboard because the vendor's data model is the conversation, not the session. The OTel-native model is the session because that is the unit of work that matters to the engineer.

The OTel schema for per-engineer Claude Code usage

The reference schema is a subset of the OpenTelemetry GenAI semantic conventions (gen_ai.*) extended with three Claude-Code-specific attributes. Every span emitted by Claude Code 1.0 GA's native OTel hooks (Settings > Hooks since June 26) populates these by default; Cursor, Copilot, and the others land in this schema after the five-processor Collector pipeline from the sibling article:

# Required on every span — minimum viable per-engineer attribution
user.id: spiffe://stackpulsar/people/<engineer-id>      # who is doing the work
user.eng_team: platform-eng                                 # which team they belong to
gen_ai.conversation.id: 9a1f2b3c-...                       # conversation-scoped
claude_code.session.id: 2026-07-10-eng-platform-refactor   # the durable task

# Spans-only — emitted on tool-call boundaries
claude_code.tool.name: Edit | Bash | Read | Grep | Glob | WebFetch | Task
claude_code.tool.input.size_bytes: 1240
claude_code.tool.output.size_bytes: 420
claude_code.workaround_pattern.id: "ignore_previous_then_restate_v3"  # see below

# Spans-only — emitted on every model call (input/output token counts)
gen_ai.usage.input_tokens: 1284
gen_ai.usage.output_tokens: 412
gen_ai.usage.cached_tokens: 980
gen_ai.cost.input_usd: 0.00642
gen_ai.cost.output_usd: 0.01236
gen_ai.cost.total_usd: 0.01878

# Spans-only — emitted on session end
claude_code.session.task_class: code-review | auth-refactor | ci-debug | doc-draft
claude_code.session.outcome: success | abandoned | retried-with-fresh-window

Three attributes are worth highlighting because they are what reflection cannot see:

claude_code.session.id is the durable unit of work, not the conversation. Claude Code 1.0 GA emits this on session start and propagates it across every conversation within the session via the OTel context. A "migrate the auth provider" session spanning three conversations across two days, with twenty-three tool calls, will appear as a single row in the per-engineer dashboard instead of twenty-three rows. This is the attribute that unlocks the cost-vs-quality-by-task-class signal later in this article.

claude_code.tool.name is normalized to the canonical tool set. Claude Code's six core tool names (Edit, Bash, Read, Grep, Glob, WebFetch), plus the new Task subagent launcher, land as canonical strings. Without normalization, the per-tool-call frequency panel is meaningless: an engineer whose sessions are 80% Read calls is doing something different from one whose sessions are 80% Bash calls, and you can only see that with a tool-name taxonomy.

claude_code.workaround_pattern.id is the attribute the four-signal model needs but Anthropic does not emit. It is computed at the Collector level by a simple fingerprinting processor: hash the first 200 tokens of the conversation prompt + the first 50 tokens of each Edit / Bash tool call input, normalize whitespace and variable bits, and assign an ID when the same fingerprint appears in 3+ different conversations within a 28-day window. The IDs are stable; the dashboard can surface "you have 14 distinct workaround patterns you keep reinventing" without needing a labeled corpus to train a classifier.

The 1/3/6/12-month retention model, and why Anthropic picked 18

Reflection's retention window is 18 months. The question I had to answer for our platform was: is that the right retention for a per-engineer observability system, or is it the right retention for a vendor trying to upsell Pro/Max plans? Answering honestly meant modeling what each window unlocks and what it does not:

Window	What it unlocks	What it costs	Per-engineer value
1 month	The "what did I do this sprint" view. Workaround patterns that matter today. Tool-call frequency at high resolution.	Lowest — ClickHouse / Loki default tier. ~$0.30/engineer/month at our trace volume.	Highest signal density. Where the four-signal dashboards live by default.
3 months	The "what is my workflow becoming" view. Workaround patterns that survived a project cycle. Topic drift signals.	Moderate — 3x storage of 1-month window. ~$0.90/engineer/month.	What engineers actually use to decide whether a workflow pattern is theirs or a side effect of a deadline.
6 months	The "is my AI usage compounding" view. Does my tool-call efficiency go up over time? Does my cost-per-successful-task go down?	Significant — 6x storage. ~$1.80/engineer/month.	The compounding-skill signal that reflection does not surface but engineers want.
12 months	The "annual pattern" view. Seasonal workload shifts, project-completion patterns, the macro arc of how the engineer's Claude usage has evolved.	High — 12x storage. ~$3.60/engineer/month. Becomes the dominant cost driver of the per-engineer observability system.	Where the platform team gets the macro signal. Where the individual engineer sees whether their AI skill is compounding, plateauing, or declining.
18 months (reflection)	Vendor padding — the additional 6 months beyond 12 unlocks nothing actionable.	High — 18x storage. ~$5.40/engineer/month.	Vendor retention default, not engineer need.

The right answer for a self-hosted OTel + ClickHouse stack is the 1/3/6/12 model: tiered retention where the 1-month data is at full granularity (every tool call, every latency sample), the 3-month data is at one-minute rollups, the 6-month data at one-hour rollups, and the 12-month data at daily rollups. The four-signal dashboard reads from the 1-month tier for high-resolution analytics; the per-engineer monthly digest reads from the 6-month tier. ClickHouse's TTL + rollup table engine handles the tiering automatically; the operational cost at 200 engineers is about $720/month for the entire per-engineer observability backend, against $0 incremental against the existing open-source LLM monitoring stack.

For sizing, the per-engineer numbers above assume ~14,000 spans/engineer/month at the 200-engineer scale (~70 sessions × 200 spans each). If you want to size your own deployment before you commit, the LLM API Cost Calculator handles the model-side arithmetic; the storage arithmetic is roughly 2 KB/span at full granularity before compression, 0.7 KB after ClickHouse's default ZSTD, times the retention multiplier above.

The four-signal dashboard, laid out as a per-engineer variant of the open-source layout

The reference dashboard is the per-engineer variant of the layout from our open-source LLM monitoring guide — same five-panel architecture, different per-panel query:

1. Top-left — Topic distribution by session (not by conversation). Treemap of claude_code.session.task_class weighted by count(distinct claude_code.session.id). The panel that surfaces where the engineer's time actually goes. A senior engineer's panel reads 40-50% code-review, 20-30% code-authoring, 10-15% CI/debug, with the rest in document work; a junior engineer's panel reads 60%+ code-authoring with a much heavier tail in CI/debug. Reflection shows this at the conversation level, which inflates the discussion-topic buckets and hides the actual work.
2. Top-right — Common workaround patterns. Bar chart of the top 10 claude_code.workaround_pattern.id values across the engineer's 28-day window, with the count of distinct sessions each pattern appears in. The panel that drove the most behavior change when I shipped it: an engineer on our platform had 14 distinct workaround patterns, three of which were 80+ sessions old and were literally the same prompt template copy-pasted. The fix was a 5-line shell alias, not a model change.
3. Bottom-left — Latency-vs-success correlation per session. Scatterplot of p90(gen_ai.client.operation.duration) vs sum(claude_code.session.outcome = "success") / count(claude_code.session.id), sized by sum(gen_ai.cost.total_usd). The panel that turns prefill optimization from a debate into a measurement. On our platform the success-vs-latency curve is steep — sessions with TTFT p90 over 2.4s have a 31% lower success rate than sessions under 1.8s — and the high-cost outliers are the ones to optimize first.
4. Bottom-right — Cost-vs-quality per task class. Bar chart of sum(gen_ai.cost.total_usd) / count(distinct claude_code.session.id) grouped by claude_code.session.task_class, with a secondary line for outcome-weighted quality (success-weighted cost per task). The panel that makes the per-engineer cost conversation tractable: the engineer can see that CI/debug is the right place to spend more per session because the absolute volume is low and the success rate is critical, while doc-draft is the wrong place to spend more because the volume is high and the marginal value of a perfect draft is low.
5. Trend strip (bottom band, full width) — Compounding-skill signal. 12-month rolling line of sum(gen_ai.cost.total_usd) / sum(claude_code.session.outcome = "success") — the cost per successful task over time. The signal that answers the question reflection ignores: is this engineer getting better at using AI, or worse? A flat or downward-sloping line is compounding skill; an upward-sloping line is the engineer learning how to use the agent for everything, including things the agent is bad at.

The LangSmith configuration that produces this dashboard is ~250 lines of dashboard JSON; the OTel Collector pipeline that feeds it is the same five-processor pipeline as the coding-agent article above plus the workaround-pattern fingerprinting processor (40 lines of Starlark in the Collector, runs against every prompt span). The LangSmith dashboard reads from a LangSmith project scoped to a single user ID, which gives you the per-engineer view without needing a per-engineer Grafana stack.

What I would skip, and what the Harrison Chase talk sharpened for me

Two things I tried and rolled back before shipping:

• Classifying sessions into task types automatically with an LLM. The accuracy was 71% on the validation set, the false positives were concentrated on the long-tail task classes that mattered most, and the maintenance cost of re-prompting as new task types emerged was worse than asking the engineer to label ten sessions per week. The fingerprinting workaround-pattern detection above runs deterministically, costs nothing, and catches 80% of what the LLM classifier caught without the false-positive tail.
• Showing engineers their per-session cost on every notification. The reflection dashboard's "you spent $X on AI this month" framing triggers a budget-optimization reflex that works against the compounding-skill signal. The four-signal dashboard above deliberately shows cost per successful task, not cost per session, and that one framing change took the engineering response from "I should use Claude less" to "I should use Claude better." That is the framing difference between an observability system and a usage-deterrent system.

Harrison's talk sharpened the third thing I had been missing: the value of per-engineer observability is not the reflection dashboard's value (showing engineers what they did). It is the value of feeding the per-engineer trace corpus back into the agent itself — continual learning from the engineer's own successful sessions. LangChain's "Data Mining Problem" framing says the highest-leverage use of an agent's trace corpus is mining it for fine-tuning data and prompt patterns. The OTel schema above emits the structured fields that mining step needs. If you skip the per-engineer attribution today, you are locking yourself out of the continual-learning play tomorrow.

The five limitations reflection has, and how the OTel-native version handles each

The reflection dashboard is a useful starting point. Five limitations the OTel-native version addresses directly:

1. Single-vendor scope. Reflection reads only Claude sessions. The OTel schema above handles Claude Code, Cursor, Copilot Chat, Codex CLI, and OpenCode on the same per-user dashboard via the same five-processor Collector pipeline from the sibling coding-agent cost article. Per-engineer attribution works across the whole stack.
2. Conversation-level granularity. Reflection stores conversations; the OTel schema stores sessions. For the "migrate auth provider" workflow that spans 30+ conversations across two days, the dashboard shows one durable row instead of thirty. The cost-per-successful-task signal collapses to the right level automatically.
3. No cost attribution at all. Reflection does not store per-session token counts or pricing. The OTel schema's gen_ai.cost.* attributes (computed via the Collector price-sheet extension from the sibling article) feed the cost-vs-quality panel directly. The per-engineer cost-per-successful-task metric that reflection cannot show is one Grafana query away.
4. No success / failure signal. Reflection does not track whether the session accomplished what the engineer set out to do. The OTel schema's claude_code.session.outcome attribute is populated by a small Claude-Code-hook that asks the user at session end (one prompt, three options: success / abandoned / retried-with-fresh-window) and stores the answer in the trace. The success rate per engineer is the metric Harrison's talk predicts is the highest-leverage individual signal.
5. 18-month retention as the only knob. Reflection gives you 1/3/6/12 month views over the same 18-month dataset; the data granularity is the same at every level. The tiered retention model above gives you full-granularity 1-month, rollup 3/6-month, and daily-aggregate 12-month data — same ClickHouse bill, ten times the analytical resolution in the window where the four-signal dashboards actually run.

Conclusion: the per-engineer dashboard is the continual-learning flywheel's missing piece

The reflection dashboard is Anthropic's first move in the per-engineer observability lane, and it is the right move for the vendor (single platform, low-resolution metrics, upsell-friendly retention default). The OTel-native version is the right move for the platform team: it answers the reflection dashboard's three questions (what did you do, how often, what's it cost) at ten times the resolution, across every vendor in the stack, with a 1/3/6/12-month retention model that costs less and tells more. The schema above is the reference implementation; the Collector pipeline is the sibling article's five processors plus the 40-line workaround-pattern fingerprinting processor; the LangSmith configuration is ~250 lines of dashboard JSON over a per-user LangSmith project; the operational cost at 200 engineers is ~$720/month on top of the existing open-source monitoring stack.

Two years from now the per-engineer trace corpus will be fine-tuning data for that engineer's personal agent. The dashboard above is the mining operation. The mining operation is what the Harrison Chase talk is naming. Building the OTel schema now is cheap; building it after the trace corpus is two years of vendor-varied format deep in vendor-locked storage is the same project at ten times the cost. The window to ship is before the format lock-in hardens, and the window is open for the next two quarters.