Agentic Observability: Monitoring Multi-Agent LLM Systems in Production

Introduction: The Monitoring Gap No One Is Talking About

Your LLM application worked fine until it started using tools.

Before agents, monitoring was straightforward: latency, error rate, token count, cost per request. You had a prompt in, you had a completion out, and you could measure every dimension of that interaction. Then your system started calling APIs, executing code, querying databases, spawning sub-agents, and making decisions across multiple reasoning steps. Now your most critical failure modes are invisible to your existing dashboards.

An agentic system doesn't fail the way a web service fails. It doesn't return a 500 error. It returns a subtly wrong answer after taking a hundred tool calls and burning $40 in GPU time. The API was healthy. The model was responsive. The failure was in the reasoning chain — and you had no visibility into it.

This is the agentic observability gap. Closing it requires rethinking what "observability" means for systems that reason, plan, and act. This article is a practical guide to doing that.

Why Standard LLM Monitoring Falls Short

Before building an observability strategy for agentic systems, it's worth understanding precisely what's missing from conventional LLM monitoring.

Token-level metrics miss reasoning quality. Standard LLM dashboards track tokens per minute, time to first token, and cost per 1K tokens. These tell you how efficiently the model is generating text. They tell you nothing about whether the reasoning that preceded that text was sound. An agent can produce coherent, token-efficient output while making a catastrophic planning error three steps upstream.

Single-request tracing misses multi-turn state. Conventional tracing follows a single prompt-completion pair. Agentic systems maintain state across arbitrarily long interaction windows — working memory that persists across steps, tool results that inform subsequent reasoning, shared context across parallel sub-agents. If you're only tracing at the request level, you're blind to failures in inter-step state management.

Uptime monitoring misses reliability degradation. An agentic system can be technically "up" — responding to requests — while producing increasingly unreliable outputs as retrieval quality degrades, tool APIs drift, or model behavior shifts. Traditional uptime metrics don't capture this kind of graceful degradation.

Cost dashboards miss loop risk. Standard cost monitoring tracks spend at the transaction level. Agentic systems introduce unbounded cost risk: a misconfigured agent can enter a reasoning loop, burning tokens with each iteration until it hits a hard limit or your budget runs out. By the time your cost alert fires, you've already paid for the damage.

The core problem is that agentic systems introduce a new class of failure modes — reasoning failures, planning failures, tool-call failures, state corruption — that exist at a different level than the metrics conventional monitoring captures. You need a monitoring architecture designed for this specific failure mode topology.

The Four-Layer Agentic Observability Stack

A production-ready observability strategy for agentic systems covers four distinct layers, each requiring different instrumentation and tooling.

Layer 1: Trace Observability — Following the Reasoning Chain

The foundation of agentic observability is step-level tracing — capturing every reasoning step, tool call, and state transition in the execution graph.

This goes beyond standard OpenTelemetry tracing in one critical way: the semantics of the span matter, not just its timing. A span representing a tool call to a search API is fundamentally different from a span representing an internal reflection step. Your observability layer needs to capture not just that a step happened, but what type of step it was, what inputs it received, and what outputs it produced.

What to instrument:

• Step type taxonomy — Classify every step in the agent loop: reasoning, tool_call, tool_result, plan_update, sub_agent_spawn, sub_agent_result, artifact_generation, error, circuit_breaker_triggered. This taxonomy is the foundation for everything else.
• Step inputs and outputs — Log the full input/output of each step, not just its duration. For reasoning steps, capture the prompt and completion. For tool calls, capture the full request and response. This data is essential for debugging failures.
• Step-level latency — Track latency for each step independently, including time-to-first-token for reasoning steps and wall-clock time for tool calls. Correlate step latency with total end-to-end latency to identify bottlenecks.
• Branching and parallelism — When an agent spawns parallel sub-agents or makes parallel tool calls, trace each branch independently and capture the dependency graph. Total latency is the critical path through this graph, not the sum of all branches.

Recommended tooling: OpenTelemetry with custom span attributes is the minimum viable foundation. For higher-level abstraction, LangSmith, Phoenix (by Arize), or Weights & Biases Weave give you purpose-built agent tracing with reasoning step reconstruction. For production-scale systems with custom agent runtimes, building on top of OpenTelemetry's SDK with a custom collector pipeline gives you the most flexibility.

Layer 2: Cost Observability — Tracking the Token Meter

Agentic systems make cost management genuinely hard. A single user request can trigger dozens of LLM calls — the initial reasoning, tool-call summaries, sub-agent summaries, final synthesis. Multiply this by concurrent users and you have a cost structure that's nearly impossible to reason about without instrumentation.

Per-step token accounting. Track tokens at the step level, not just the request level. This requires instrumenting your LLM calls to log input and output token counts per step, then aggregating by step type, tool, and sub-agent. The goal: for any running session, you can answer "where is the token spend happening?"

Cost attribution by task type. Not all tasks cost the same. A reasoning-heavy step that uses a frontier model costs orders of magnitude more than a tool-call summary step handled by a smaller model. Tag each LLM call with the task type and model used, then build cost dashboards that show spend by task type. This is the data you need to make routing decisions.

Loop detection and circuit breakers. The most dangerous cost risk in agentic systems is the unbounded loop — a reasoning chain that never terminates, either because the agent keeps re-planning without converging or because a tool call keeps returning partial results that trigger retries. Instrument a "steps in current reasoning cycle" counter that increments with each LLM call and resets when a user-visible output is produced. Set alerts at configurable thresholds. Implement hard circuit breakers: if the counter exceeds N steps, halt execution and return an error rather than continuing to burn budget.

Cost-per-outcome tracking. This is the FinOps metric that matters most for agentic systems: not cost per token, but cost per successful task completion. Measure this at the session level and track it over time. If cost-per-outcome is trending up without a corresponding improvement in outcome quality, you have an efficiency problem that cost-per-token metrics would miss.

Layer 3: Reliability Observability — Detecting Reasoning Failures

This is the hardest layer to instrument, because it requires evaluating the quality of the agent's reasoning, not just its efficiency.

Task completion rate. Track whether the agent successfully completed the task it was given, across sessions. This is a coarse metric but essential: if 30% of your agentic sessions are ending in failure, that's a signal that needs investigation regardless of what your latency or cost dashboards show.

Tool-call failure rate and categorization. Instrument every tool call to capture: whether it succeeded, failed, or returned a partial result; the error type if it failed; and whether the failure was transient (network timeout) or deterministic (authentication error, invalid input). Aggregate these by tool to identify unreliable integrations. A tool with a 15% failure rate is a production reliability problem regardless of how fast it is when it works.

Retrieval quality drift. For agents that depend on retrieval (RAG agents, agents with knowledge bases), monitor retrieval quality over time. Track metrics like the fraction of retrieved context chunks that are actually referenced in the agent's reasoning steps — if this ratio drops, it suggests the retriever is returning increasingly irrelevant context. Embedding drift detection (comparing query embedding distributions over time) can serve as an early warning system.

Hallucination and fabrication detection. The agentic equivalent of standard LLM hallucination detection — but compounded by the fact that the agent may be building on false tool-call results or corrupted intermediate state. Techniques include semantic similarity between agent claims and retrieved context, cross-validation against known-ground-truth queries, and LLM-as-judge evaluation on a sample of production outputs.

Layer 4: Security Observability — Watching for Agentic Attack Surfaces

Agentic systems introduce a new class of security concerns that compound traditional LLM vulnerabilities.

Tool-call anomaly detection. An agent that suddenly starts calling tools it hasn't called before — or calling familiar tools with unexpected parameters — may be exhibiting the early signs of a prompt injection attack. Instrument a baseline of normal tool-call patterns (which tools, which parameter patterns, which sequences) and alert on statistically significant deviations.

Resource consumption monitoring. Agentic systems can consume resources in unexpected ways — writing large files, making excessive API calls, spawning unbounded sub-agents. Monitor process-level resource usage (CPU, memory, file descriptors, network connections) per agent session. Set baseline profiles and alert on deviations.

State isolation verification. For agents handling multiple concurrent sessions, verify that session state is properly isolated — that one session's memory and tool access can't bleed into another. This requires instrumentation at the session management layer, not just the agent layer.

Layer 5: Data-Factory Observability — The Constraint Below the Code

The four layers above all live inside the agent runtime. But the August 2026 IBM TLS Agentic Platform postmortem surfaces a fifth constraint that sits below them: data access. Tony Erwin, Chief Architect for the TLS Agentic Platform inside IBM Infrastructure AI CoE, put it this way in the postmortem on production agentic AI at IBM:

"The real constraint was never agent code — it was data access. Everyone wants an agent factory, a team stamping out new agents; we needed a data factory — finding the systems of record, negotiating access, building the tool layer."

Once you internalize this, the four-layer framework needs a Layer 0 that precedes it. You cannot meaningfully observe a layer-1 trace through an MCP server that does not yet exist because the enterprise has not yet granted access to the system of record behind it. The instrument-before-you-build principle Erwin emphasizes — "Instrument before you build — evals, traces, business metrics — with tools that exist today. We flew blind too long" — applies to the data layer with the same force as the runtime layer.

What to instrument at Layer 0:

• Systems-of-record registry — for every agent in your platform, record which system of record actually holds the data the agent needs to answer. The answer is rarely "the database" — it is usually "the Salesforce proxy API" or "the internal RAG API" or "the asset data layer owned by another team." Erwin's team discovered this distinction the hard way: "figuring out which system of record actually held the data a given use case needed, whether that data was accessible at all or locked behind a team that had never been asked to expose it" was the work that consumed most of the project schedule.
• Authorization negotiation ledger — track which teams have granted read access, which have not, and the latency between request and grant. Token-exchange boundaries (e.g. IBM Verify performing the OAuth 2.0 exchange at the platform boundary) need explicit spans. Erwin's rule: "Carry the user's identity through every hop — propagate or exchange, never substitute a service account. The moment one delegation drops the caller's identity, authorization quietly leaves the enterprise's model for yours."
• Mocked-vs-real data coverage — the IBM team explicitly calls out that early prototypes mock MCP endpoints and "there's a hard limit on how far a mocked data layer takes you. Nothing is real until the actual data is flowing." Instrument the fraction of agent capabilities running on mocked vs real data per agent, and alert when production traffic is still hitting mocks — the most common reason agent demos "work in staging and fail in production."

The data-factory primitive is the prerequisite for Layers 1–4. Without it you are observing a closed-loop fiction. The runtime harness, the per-step spans, the per-agent cost dashboards — all of it depends on real data flowing through real MCP servers with real authorization boundaries. Building the data factory is the work that determines whether the agent platform delivers value at all.

Enterprise case study: IBM TLS Agentic Platform (Tony Erwin, Aug 2026)

The corpus has been missing a first-person enterprise-scale agent-observability case study. Tony Erwin's August 2026 postmortem on the TLS Agentic Platform — a multi-agent system IBM's Technology Lifecycle Services organization has been running in production since fall 2025, built by Erwin as Chief Architect for the platform inside IBM Infrastructure AI CoE — fills that gap. Three architectural primitives from that post are directly relevant to the framework above.

Primitive 1: A2A for every agent boundary, MCP for every tool boundary. Erwin's deployment rule for anyone starting a similar project: "Use A2A for every agent boundary, MCP for every tool boundary. That uniformity is what let five of our six specialist agents be built by separate teams, one in an entirely separate environment." The Asset Agent runs in a completely separate environment operated by a different team and is connected purely via A2A — deployment isolation is a production requirement, not an optimization. For observability, the implication is that the supervisor-to-specialist hop is always an HTTP boundary with a self-describing AgentCard, which means every agent boundary is a span boundary with no special instrumentation required.

Primitive 2: Identity propagation across every hop. The user's authorization token is propagated the full length of a request: BFF to Supervisor, Supervisor to specialist agent, agent to MCP server, MCP server to the underlying API. "No hop substitutes a service account, and where the request crosses into a system with its own identity provider — Support Insights for asset data, or an enterprise backend like Salesforce — the token is exchanged rather than dropped. IBM Verify performs the OAuth 2.0 token exchange, so the same user arrives on the other side. Propagate or exchange, never substitute." For trace correlation this matters because the span attribute that ties the request together is the user identity, not a session ID — and that identity has to survive token exchanges without being substituted by a service-account identity, which would silently break authorization without breaking the trace.

Primitive 3: Factory pattern as the single observability injection point. The AI CoE Agent Utils library — the shared Python glue code Erwin's team built — centralizes observability at one place: "Central to the library is the factory pattern: a factory class that constructs an agent with all its dependencies (LLM, tools, tracing, MCP connections) injected at construction time rather than scattered through application code. The factory is also the single injection point for observability. Wire Langfuse tracing into the factory once, and every agent created by any factory automatically produces traces. Without that clean injection point, adding tracing post-hoc becomes a distributed find-and-update exercise across every agent repo." The lesson for cross-team agent platforms: per-agent observability is the wrong default; per-factory observability is the right one.

Erwin's broader point — "be wary of taking a hard dependency on a platform that is still a roadmap item, however credible the roadmap. Pick something proven and available today." — applies to Langfuse, OpenTelemetry, and the entire observability stack above. The instrumentation you can ship this week is more valuable than the instrumentation you are waiting for. Trace the data-access work with the same rigor as the runtime work; the data factory is the layer that determines whether the agent platform delivers value.

For the corpus-wide implications of this postmortem — including how it changes the data-factory precondition for any agent sandbox, the team-alignment primitive for multi-team reliability, and the per-step correlation harness that ties them together — see the related coverage on agent-sandbox-vs-agent-substrate-2026, ai-agent-reliability-monitoring, opentelemetry-ai-inference-tracing, agentic-incident-harness, and agent-evidence-packet-analytics-2026. Tony Erwin's full post is at tonyerwin.com/2026/08/building-production-agentic-ai-at-ibm.html.

Practical Implementation: Building the Instrumentation

Now for the implementation question: how do you actually instrument a production agentic system? The answer depends on your agent framework.

If you're using LangChain: LangSmith provides built-in step-level tracing with minimal code changes. You get reasoning step reconstruction, token accounting, and tool-call tracing out of the box. The trade-off is vendor lock-in and cost at scale.

If you're using LangGraph, AutoGen, or a custom agent runtime: You'll likely need to build on top of OpenTelemetry. The key is designing a tracing context that persists across the full agent session — not just a single LLM call. This requires threading a context object through your agent loop and using OpenTelemetry's context propagation to maintain continuity across async boundaries and sub-agent calls.

The custom instrumentation checklist:

1. Define your step type taxonomy before writing any instrumentation code. Every step in your agent loop should map to one type in your taxonomy.
2. Instrument at the agent loop level, not inside individual tools. Tools should be dumb — the agent loop is where you capture the step metadata that makes observability meaningful.
3. Emit spans for every step with at minimum: step type, step number in the reasoning chain, parent step (for nested reasoning), input tokens, output tokens, latency, and a summary of the step output.
4. Track a session-level context object that persists across all steps: session ID, user ID, task description, accumulated cost, step counter, and circuit breaker state.
5. Build aggregation queries first. The raw trace data is only as useful as your ability to query it. Design the queries you want to run — cost by step type, latency by tool, failure rate by session — before finalizing your instrumentation schema.

The Minimum Viable Agentic Monitoring Stack

If you're building this for the first time and need to ship something production-ready without a six-month observability project, here's the minimum stack:

Step 1 — Capture step-level traces. Use OpenTelemetry or LangSmith to log every reasoning step, tool call, and sub-agent result with input/output summaries and token counts. This is non-negotiable.

Step 2 — Build a cost-per-session dashboard. Take your per-step token data and aggregate it at the session level. Show: total tokens, total cost, cost by step type, and cost by model used. Add an alert when cost-per-session exceeds a threshold you define based on observed baselines.

Step 3 — Implement circuit breakers. Set a maximum step count per session (start conservative — 50 is reasonable for most use cases). When the counter hits the limit, halt execution, log the failure, and return an error. Track how often circuit breakers fire — a rising rate is an early warning sign.

Step 4 — Add tool-call reliability metrics. Track success/failure rates per tool, categorized by error type. Alert on tools whose failure rate exceeds your threshold. This alone will catch most of your agentic reliability problems.

Step 5 — Instrument loop detection. Track when the agent makes the same tool call with similar parameters within a configurable lookback window. This catches reasoning loops before they burn through your budget. Alert and halt when detected.

This stack won't catch every failure mode. But it will give you visibility into the three most common and most expensive ones: cost overruns from unbounded reasoning, reliability failures from flaky tool integrations, and reasoning degradation from retrieval quality problems.

Conclusion: Observability Is the Infrastructure

The uncomfortable truth about agentic systems is that they fail in ways that are qualitatively different from the failures we've learned to monitor in traditional software. A reasoning chain that takes a hundred steps to complete an objective can fail at any of those hundred steps — and often the failure is invisible until it produces a wrong answer.

The teams that will operate agentic systems successfully in production are the ones that treat observability as a first-class infrastructure concern, not an afterthought. That means instrumenting the reasoning chain, tracking cost at the step level, monitoring tool reliability, and building circuit breakers before the first production deployment — not after the first $10,000 bill from an unbounded loop.

The good news: the primitives exist. OpenTelemetry, purpose-built agent tracing tools, and the standard observability stack you're already running can be extended to cover agentic workloads. The hard part is designing the instrumentation taxonomy that makes the trace data actually useful for debugging agentic failures — and that's a design problem, not a tooling problem.

The monitoring stack for the agentic era is being built right now. The teams that build it right will be the ones who understand that the most important metric for an agentic system isn't how fast it responds — it's whether it was thinking correctly.