LiteLLM Production Monitoring 2026: Gateway + Cost Tracking

What's New in LiteLLM 1.93.0-dev.1

LiteLLM 1.93.0-dev.1 shipped 2026-07-08 as the first development tag on the 1.93 line. The 1.93.0 stable tag has not shipped yet, so treat 1.93.0-dev.1 as the validation target and 1.92.0-rc.1 as the recommended production pin until 1.93.0 stable lands. The release body lists 60+ PRs across security hardening, MCP OBO extensions, cost accuracy, UI polish, and a long tail of correctness fixes. The dev tag is also the first 1.93-line image that ships with the same cosign signing key as the prior 1.92 cycle — verify the same way the 1.92.0-rc.1 section walks through, just swap the image tag to v1.93.0-dev.1. The headline changes from the 1.93.0-dev.1 release notes:

Security: hash Bearer-prefixed API keys in spend logs (#31799) — Until 1.93.0-dev.1, the spend-log writer emitted Bearer-prefixed virtual keys in plaintext (the same plaintext that the database stored before 1.92.0-dev.1's AES-256-GCM at-rest encryption). On a database export, a raw sk-... key was directly readable from the spend-log line. 1.93.0-dev.1 hashes the Bearer-prefixed API key at spend-log emission so even an unencrypted database export yields only the hash, not the key. For teams subject to SOC 2, HIPAA, or internal data-handling audits, this is the release that closes the spend-log credential leak on top of the at-rest encryption that 1.92.0-dev.1 added. Pairs with the supply chain security for DevOps 2026 guide for the broader defense-in-depth surface.
Security: stop CacheCodec dropping null fields on cache round-trip (#32207) + fix streaming SSE in-body error payloads on OpenAI-compatible streams (#32237) + forward verbosity param to chat completion providers (#32254) — Three correctness fixes on the request and cache path. The cache-codec fix ensures that null fields round-trip correctly (previously they were dropped, which silently changed the cached payload's shape on read). The streaming SSE fix surfaces in-body error payloads on OpenAI-compatible streams so a downstream client can react to mid-stream failures instead of treating them as end-of-stream. The verbosity-forwarding fix passes the OpenAI verbosity param through to the chat completion provider so it actually takes effect. For teams running LiteLLM as the central gateway in front of multiple providers (the pattern the inference API gateways guide covers), 1.93.0-dev.1 is the line where cache shape fidelity, mid-stream error visibility, and provider-param forwarding all move from "almost right" to correct.
MCP: add entra_obo profile to the token_exchange (OBO) arm (#31983) — A new OAuth profile for the MCP token_exchange (OBO) arm, specifically targeting Microsoft Entra ID. Until 1.93.0-dev.1, the proxy supported OBO against generic OIDC IdPs (the path 1.92.0-rc.1 made production-ready) but Entra's specific discovery quirks (issuer URL canonicalization, the Entra-specific azp claim, the Entra audience format) had to be hand-wired. 1.93.0-dev.1 adds a first-class entra_obo profile that wires the Entra-specific configuration automatically. For teams running MCP servers behind Entra-issued tokens (a common pattern in Microsoft-shop enterprise deployments), 1.93.0-dev.1 is the line where the OBO path stops requiring per-deployment hand-edits. Pairs with the MCP enterprise authorization 2026 guide for the broader OBO architecture.
Router: separate ITPM/OTPM deployment rate limits (#31952) — The router previously applied a single rate limit per deployment regardless of whether the request was an ITPM (input tokens per minute) or OTPM (output tokens per minute) budget. For workloads with asymmetric token profiles (long context window, short answer — the typical RAG pattern), the single-limit approach was both over-permissive on the input side and under-permissive on the output side. 1.93.0-dev.1 splits the two limits so each can be tuned independently. For teams running cost-attribution middleware that keys to per-token cost, the ITPM/OTPM split is the lever that lets you cap a long-context workload by input volume without throttling short-form traffic. Pairs with the LLM cost monitoring tools 2026 roundup for the broader cost dashboard pattern.
Guardrails: send only new messages since last assistant turn to CrowdStrike AIDR (#31974) + Headroom guardrail: log real token/compression stats instead of "allow" (#32158) — Two guardrail-side fixes. The CrowdStrike AIDR fix reduces the token payload sent to the guardrail to only the new messages since the last assistant turn (until 1.93.0-dev.1, the entire conversation was sent on every guardrail call, which inflated both latency and cost). The Headroom guardrail fix replaces the opaque "allow" log line with the real token/compression statistics the guardrail actually computed. For teams running CrowdStrike AIDR as the primary guardrail on a long-conversation workload (customer-support tools, agent loops), 1.93.0-dev.1 is the line where the guardrail cost-per-call stops scaling with conversation length. The LLM security hardening guide covers the broader guardrail-cost-control pattern.
Spend accuracy: filter /global/spend/report by team_id when group_by=team (#32170) — A long-standing cost-dashboard correctness bug. Until 1.93.0-dev.1, requesting /global/spend/report?group_by=team returned a global rollup that ignored the team_id filter, which meant the per-team rows were silently the same as the global total. 1.93.0-dev.1 correctly scopes the report to the requested team_id. For teams using the spend report as the chargeback source-of-truth, this is the fix that finally makes per-team cost reports accurate. Pairs with the LLM FinOps 2026 playbook and the LLM API cost calculator for forecasting the corrected numbers.
UI: cost optimization feedback banner on the models page (#32174) + UI: reflect persisted "Store Prompts in Spend Logs" toggle on load (#32145) + UI: flag experimental dashboard pages on the draft deprecation list (#32132) — Three UI improvements. The cost optimization banner surfaces suggestions for routing high-volume workloads to cheaper models (the pattern the cost optimization work in 1.92.0-dev.1 also walked through). The toggle-load fix ensures the "Store Prompts in Spend Logs" toggle reflects its actual persisted state on first load (previously the UI defaulted to off regardless of the stored value). The experimental-page deprecation list adds a visible banner to dashboard pages that are slated for removal, so operators are not surprised by a future breaking UI removal.
Proxy: resolve os.environ/ refs for all AWS auth params in DB-sourced models (#32256) + Helm: support user-defined volumes and volumeMounts in the microservices chart (#32233) — Two deployment-shape fixes. The AWS auth fix ensures that all AWS auth parameters (not just the ones in the original config file) correctly resolve os.environ/NAME references when the model config is sourced from the database. The Helm fix lets teams mount their own volumes (custom CA certs, secrets stores, model weights caches) into the LiteLLM microservices chart, which the previous chart structure did not expose. For teams running LiteLLM in air-gapped or restricted-network Kubernetes clusters (the pattern the supply-chain guide walks through), 1.93.0-dev.1 is the line where the deployment shape finally fits.
Bedrock: emit SSE error event when invoke Messages stream ends without message_stop (#32159) + Anthropic messages: forward provider response headers on streaming /v1/messages responses (#32160) + Azure: build responses input_items URL with path before query string (#32270) + Complexity router: add custom_technical_keywords config (#32262) + Merge websearch tool params (#32162) — Five provider-side correctness fixes. The Bedrock fix ensures an early-stream-end surfaces as an SSE error event rather than a silently truncated stream. The Anthropic fix forwards the provider's response headers (rate-limit info, request ID) on streaming responses, which previously dropped them. The Azure fix correctly builds the input_items URL with the query string after the path, which was previously a 4xx on a representative subset of Azure deployments. The complexity-router fix lets teams add their own technical-keyword list to the heuristic that decides whether a request is "complex enough" to route to a stronger model — useful for domain-specific workloads where the default keyword set misses. The websearch merge fix ensures multi-source websearch tool parameters combine correctly. Pairs with the inference API gateways guide for the multi-provider routing surface.

Upgrade via pip install litellm==1.93.0.dev1 or docker pull ghcr.io/berriai/litellm:v1.93.0-dev.1. The Docker image is cosign-signed with the same key as the 1.92 cycle — verify before deploying. There are no breaking changes in 1.93.0-dev.1. The 1.93.0 stable tag has not shipped at the time of this writing; treat 1.92.0-rc.1 as the validated production pin and 1.93.0-dev.1 as the preview where the next wave of security hardening (Bearer-key hashing in spend logs, cache-codec null-field fidelity, streaming SSE error visibility, Entra OBO profile) and the long tail of provider-side correctness fixes get their first real-world shakedown. The 1.91.1 stable tag (released 2026-07-08, the same day as 1.93.0-dev.1) is the recommended production pin for teams that do not want to live on the 1.92 RC line yet.

What's New in LiteLLM 1.92.0-rc.1

LiteLLM 1.92.0-rc.1 shipped 2026-07-05 as the first release candidate on the 1.92 line — cut from main via the internal staging promotion PR #32156. The 1.92.0 stable tag has not shipped yet, so treat 1.92.0-rc.1 as the validation target and 1.91.0 stable as the recommended production pin until 1.92.0 stable lands. The RC is also the first LiteLLM release that ships with cosign-signed Docker images by default — every image is signed with the same key introduced in commit 0112e53, so the verification workflow the supply chain security for DevOps 2026 guide walks through now applies to LiteLLM directly. Verify with the pinned commit hash (the strongest check):

cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \
  ghcr.io/berriai/litellm:v1.92.0-rc.1

The release body lists 30 PRs. The headline changes from the 1.92.0-rc.1 release notes:

Docker images are now cosign-signed by default — Until 1.92.0-rc.1, LiteLLM images were unsigned and teams that cared about supply-chain integrity had to fork the image, sign it themselves, and run a private registry mirror. 1.92.0-rc.1 ships the signing material in-tree, so the upstream image can be verified directly against the key pinned at commit 0112e53. For teams that already have a cosign verification gate in their admission controller (the workflow the supply-chain guide walks through), 1.92.0-rc.1 is the line where that gate passes on LiteLLM without a private-mirror detour. The release adds both verification paths — pinned-commit (recommended, cryptographically immutable) and tag-based (convenience, depends on tag protection rules).
MCP: all-proxy-mcpservers sentinel grants every MCP server to a team (#32012) — A new sentinel value for the team MCP allowlist that grants access to every MCP server registered to the proxy in one line of config, instead of enumerating them per-team. For teams running a small number of MCP servers (the typical pattern), all-proxy-mcpservers replaces a config that drifted every time a new server was added. Pairs with the MCP monitoring guide for teams that fan a single MCP allowlist across many teams.
MCP: token_exchange (OBO) production-ready — discovery threading + audit hardening + RFC 9728 challenge (#31622, #31762) + v2 resolver migration (#31526) — Three PRs that harden the OAuth On-Behalf-Of path for MCP. Until 1.92.0-rc.1, the token_exchange arm was treated as an experimental path that required manual IdP-endpoint configuration. 1.92.0-rc.1 makes it production-ready: discovery threads through RFC 9728 challenge handling, RFC 8414 metadata parsing (no IdP guessing), audit log hardening, and the v2 resolver migration. For teams running MCP servers behind an OIDC IdP that requires OBO token exchange (the pattern the MCP enterprise authorization 2026 guide walks through), 1.92.0-rc.1 is the line where the proxy stops requiring a custom callback for OBO and starts handling the discovery and audit logging natively.
Cost: store cost breakdown for /v1/realtime sessions (#30069) — Realtime session cost was previously aggregated into a single line at session close, losing the per-token-type breakdown the rest of the proxy emits. 1.92.0-rc.1 stores the breakdown so voice-agent costs roll into the same per-token-type cost dashboard that the 1.92.0-dev.1 cost PR landed. Pairs with the LLM cost monitoring tools 2026 roundup for teams wiring realtime spend into the same chargeback view as the synchronous traffic.
Prometheus: api_provider label on token, latency, request, and cache metrics (#32126) — The Prometheus exporter now tags the upstream provider on the metrics that previously only carried the resolved model name. For teams running multi-provider workloads where the same model name routes to different providers (e.g. gpt-4o via OpenAI direct, Azure, and a self-hosted vLLM), the new label makes provider-level slicing possible without resorting to model-name heuristics. The inference API gateways guide walks through how this label slots into the routing-fallback dashboard.
Anthropic: keep context_management working when drop_params is enabled (#32020) + require caller api_key and SSRF-validate api_base in advisor tool (#32093) + bill streaming 1h prompt-cache writes at the 1h rate (#32073) — Three Anthropic-side fixes. The context_management fix preserves the long-context window when drop_params is enabled (the two settings used to conflict on Anthropic). The advisor tool now requires a caller api_key and validates the api_base against SSRF, closing a real attack surface. The cache-billing fix correctly bills streaming 1h prompt-cache writes at the 1h rate rather than the 5m rate, which on a long-context streaming workload was under-billing by a factor of 12. Worth verifying on the cost dashboard if your chargeback keys to per-token cost.
Router: mask provider credentials embedded in fallback error messages (#32083) + proxy: stop leaking master_key and database_url in startup DEBUG logs (#31944) — Two security hardening PRs. Until 1.92.0-rc.1, a fallback error that included the provider's error body could leak the provider API key in the message text. The fix scrubs credentials before the error is surfaced. The startup-log fix removes the master_key and database_url from the DEBUG log line — both real credentials that an attacker reading the log could lift. For SOC 2 / HIPAA audits, 1.92.0-rc.1 is the line where both surfaces stop being credential leaks.
Realtime: route HTTP endpoints through router for credential resolution (#32077) — Realtime HTTP endpoints (the control plane alongside the WS stream) previously bypassed the router's credential resolution path, which meant a per-team virtual key could not be used to authenticate against them. 1.92.0-rc.1 routes through the router so the same credential machinery that gates /v1/chat/completions applies to realtime. Pairs with the LLM security hardening guide for teams that have standardized on virtual-key-based access control.
Proxy: keep serving reads from the read replica when the primary DB is down at startup (#31951) — Until 1.92.0-rc.1, a primary-DB outage at startup kept the proxy from serving reads even when the read replica was healthy. The fix lets the proxy start in a degraded-reads-from-replica mode. For teams that run proxy behind a DB HA setup, this is the release that closes the "primary DB restart = proxy refuses to start" gap.
UI: budget fallbacks configuration on key create/edit forms (#32072) + migrate chat UI from antd to shadcn/ui (#32074) — Two UI changes. The budget-fallbacks form field surfaces the budget_fallbacks config that previously required a config-file edit. The antd → shadcn/ui migration is a long-running UI rewrite that 1.92.0-rc.1 completes for the chat surface; the chat now uses the same component library as the rest of the dashboard.
MCP: prevent duplicate budget alert emails on concurrent threshold crossings (#32011) — A real noise-reduction fix. Until 1.92.0-rc.1, a budget threshold that was crossed concurrently by multiple request paths could trigger multiple identical budget-alert emails. The fix deduplicates at the alert-emission boundary. Pairs with the budget-alert path the LLM FinOps 2026 playbook recommends setting up.
License seat limit: count only active users (#31227) + enterprise dep bump 0.1.46 → 0.1.47 (#32150) — Two license-side changes. The seat-count fix lands as part of the 1.92.0-rc.1 dependency bump.

Upgrade via pip install litellm==1.92.0rc1 or docker pull ghcr.io/berriai/litellm:v1.92.0-rc.1. The Docker image is cosign-signed — verify before deploying, especially if you are pulling from a mirror that may not preserve signatures. There are no breaking changes in the RC, but the cosign signing path is new — teams that pin latest should verify their admission controller accepts signed images on the day of the upgrade. The 1.92.0 stable tag has not shipped at the time of this writing; treat 1.91.0 as the validated production pin and 1.92.0-rc.1 as the preview where the structural changes (cosign signing, MCP OBO production-ready, realtime cost breakdown, the SSRF-validated advisor, the credential-leak fix) get their first real-world shakedown.

What's New in LiteLLM 1.92.0-dev.1

LiteLLM 1.92.0-dev.1 shipped 2026-06-30 as the first development tag on the 1.92 line — the bump PR rolls the public version to 1.92.0 and the enterprise dep to 0.1.45 in lockstep. This is a development release, not the stable tag — treat 1.91.0-rc.1 as the last-validated build and 1.92.0-dev.1 as the preview where the bigger structural changes (A2A agent routing, AES-256-GCM credential encryption, Model Armor file scanning, the new tools/list OTel client span) land before the stable cut. The release body lists 50+ PRs spanning security hardening, agent-protocol support, spend-log durability, and the long-promised explicit requested_model cost-card wire. The headline changes from the 1.92.0-dev.1 release notes:

Proxy: AES-256-GCM at-rest credential encryption with versioned format and re-encryption migration (#31215) — The biggest security hardening in the 1.92 cycle. Until 1.92.0-dev.1, the proxy stored virtual-key and provider credentials in the backing database in plaintext (or with a thin, undocumented encoding). 1.92.0-dev.1 adds AES-256-GCM at-rest encryption with a versioned envelope, so a database dump no longer yields usable credentials. A migration path is included that re-encrypts existing rows on first read — no operator action required, but the migration is forward-only. For teams subject to SOC 2, HIPAA, or internal data-handling audits, this is the release that closes the "credentials at rest" control. Pairs with the supply chain security for DevOps 2026 guide — at-rest encryption is the other half of the defense-in-depth story that image-signing alone cannot cover.
Realtime: stop second Gemini Live setup, retry hung handshake, close guardrail bypass (#31519) — A three-birds-one-stone fix on the realtime streaming path. The Gemini Live provider was being initialized twice per session (once for the WS handshake, once for the post-connect stream), which on a flaky network meant a hung handshake that never errored to the caller — the user saw a silent stall. The second-init was also a guardrail bypass, since the guardrail pipeline only saw the first init. 1.92.0-dev.1 collapses to a single init, retries the handshake on transient failure, and ensures the guardrail hooks are attached on the only init that runs. If you wire Gemini Live into a voice agent (the pattern the LLM latency monitoring 2026 guide covers), this is the fix for "my agent stops responding after the first reconnect."
A2A: support a2a-sdk 1.x proxy routing for 0.3 and 1.0 agents (#30950) — A2A (Agent-to-Agent) protocol support lands in the proxy. Teams running multi-agent stacks where one agent calls another across the network can now route both 0.3 and 1.0 a2a-sdk versions through the same proxy with unified cost and latency tracking. Until 1.92.0-dev.1, A2A traffic either bypassed the proxy entirely (losing cost attribution) or hit a 4xx because the proxy did not know the a2a-sdk wire format. For teams that have standardized on the A2A protocol alongside MCP for inter-agent calls, 1.92.0-dev.1 is the line where the agent-call cost rolls into the same LiteLLM cost ledger as direct model calls.
Cost: log per-token-type reasoning and cache cost breakdown (#31686) — The cost callback now emits separate per-token-type entries for reasoning tokens, cache read tokens, and cache write tokens, rather than folding them all into a single "output_tokens" line. For teams using Anthropic's extended-thinking or OpenAI's o-series with reasoning effort, this is the change that finally separates the bill into "thinking cost" and "answer cost" on the cost dashboard. The LLM cost monitoring tools 2026 roundup walks through how to build the per-token-type rollup panel once the new entries are flowing.
Guardrails: scan file and document attachments with Model Armor (#31655) — Model Armor (Google's managed prompt-injection / PII guardrail) now inspects file and document attachments, not just the text prompt. Until 1.92.0-dev.1, an attacker could bypass the guardrail by stuffing the payload into an attached PDF or image, since the model-armor call only saw the text wrapper. 1.92.0-dev.1 extracts attachment content and routes it through the same model-armor policy. For teams that accept user-uploaded files (the RAG ingestion path, the customer-support tool the RAG observability guide walks through), this is the fix that closes the attachment-bypass gap.
Guardrails: expose streaming knobs on generic_guardrail_api (#31730) — The generic guardrail API now exposes streaming configuration — per-chunk-vs-full-response, buffer window, and the call to invoke the guardrail on partial token streams. Until 1.92.0-dev.1, custom guardrails were effectively batch-only on streaming responses, which either added end-of-stream latency (waiting for the full response) or skipped guardrail calls on streamed chunks. 1.92.0-dev.1 lets custom guardrail authors pick the tradeoff that fits their model.
Prometheus: litellm_overhead_with_guardrails_latency_metric (#31593) — A new histogram that captures proxy-side latency attributable to guardrail evaluation, separately from the provider TTFT. Until 1.92.0-dev.1, the latency you saw in litellm_request_duration_seconds mixed provider latency and guardrail latency, so an alert on p99 latency could fire from either source. 1.92.0-dev.1 lets you alert on "guardrail overhead" specifically — useful for catching a guardrail model regression before it pollutes the whole latency dashboard.
MCP: resolve per-user OAuth identity authoritatively at the token endpoint (#31657) — The MCP OAuth token endpoint was looking up the per-user identity via a cached claim, which on a stale cache could resolve to a different user than the one currently authenticating. 1.92.0-dev.1 forces a fresh, authoritative identity resolution at token-issuance time, so the MCP server always sees the identity the OIDC provider just attested. Pairs with the MCP enterprise authorization 2026 guide for teams running per-user-scoped MCP tools.
MCP: support client_secret_basic for upstream OAuth token endpoints (#31635) — Until 1.92.0-dev.1, the proxy only supported client_secret_post for upstream OAuth client authentication, which some strict IdPs reject. 1.92.0-dev.1 adds client_secret_basic support, so the proxy can integrate with IdPs that require HTTP Basic credentials on the token endpoint.
MCP: stop one unauthenticated server from emptying the aggregate tools/list (#31684) — When an MCP server rejected the auth challenge and returned an empty tools list, the proxy was collapsing the entire aggregate tools/list response to empty for that key, even though other MCP servers on the same key had working tools. 1.92.0-dev.1 isolates the unauthenticated server's empty list, so the aggregate response contains the tools from the servers that did authenticate. The fix matters most for teams that fan a single API key across multiple MCP servers — until 1.92.0-dev.1, one misconfigured server could blank the whole tool surface for that key.
OTel: emit a tools/list CLIENT span for MCP discovery under otel_v2 (#31525) — The OTel v2 exporter now emits a CLIENT span for the tools/list discovery call (previously the call was a server-side operation with no outgoing span). 1.92.0-dev.1 surfaces the discovery round-trip in the trace, so a tool-not-listed issue can be traced to either the proxy's own enumeration or a slow upstream MCP server. The MCP tool-call visibility that the MCP monitoring pattern depends on now extends to the discovery phase.
Spend-log durability: isolate poison rows, gather independent increments, re-establish Redis cluster after restart (#31705, #31578, #31577) — Three PRs that harden the spend-log path. The poison-row isolation PR catches a single malformed spend-log row (a missing required field, a type mismatch) and quarantines it instead of failing the whole batch. The per-scope gather PR moves the cost-counter increments off the request event loop. The Redis cluster re-establishment PR reconnects to a Redis cluster after a node restart, where before 1.92.0-dev.1 the proxy would log a connection error every request until a manual restart. Together these are the "we stopped losing spend lines on busy proxies" PRs that the cost dashboard has been waiting for.
Databricks: split parallel tool calls so each tool message follows tool_calls (#31633) — A correctness fix for the Databricks provider: parallel tool calls were being collapsed into a single message, which broke the Databricks wire format and returned 4xx. 1.92.0-dev.1 preserves the per-call message structure.
Messages: passthrough /v1/messages to native endpoints via supported_endpoints (#31685) — The /v1/messages route can now be configured to passthrough to a native provider endpoint (Anthropic's first-party /v1/messages, for example) instead of going through the LiteLLM translation layer. Lower latency, no translation overhead, and the cost ledger still records the call.
Proxy: emit x-litellm-response-cost header on /messages and /generateContent (#31675) — The cost header is now emitted on the Messages and Gemini-compatible endpoints (it was already on the OpenAI-compatible /v1/chat/completions). 1.92.0-dev.1 closes the gap so cost-attribution middleware can read the cost from any of the three main wire surfaces.
Anthropic: add Claude Sonnet 5 (#31740) — Claude Sonnet 5 is now a first-class model on Anthropic and Anthropic-compatible backends. Pricing entries are in the cost map; if you are running the LiteLLM cost dashboard, expect a new row on the day the upstream Anthropic price card goes live.
Router: tag routing denylist support via ! prefix (#31728) — A new config syntax for the router denylist. Until 1.92.0-dev.1, denying a model required a separate denied_models list. 1.92.0-dev.1 lets you inline-deny with a ! prefix on the model name in the allowlist (e.g. !gpt-4o, claude-3-5-sonnet, !claude-3-haiku), which is the same syntax the allowlist already used. Small UX win, real ergonomics for teams with long allow/deny lists.
Declarative fallback generalizations for unknown models (#29718) — A long-standing request: until 1.92.0-dev.1, a fallback chain would fail with a "model not found" error if the primary model name was not in LiteLLM's registry. 1.92.0-dev.1 generalizes the fallback path so unknown models are routed through the first matching provider in the chain, rather than erroring on lookup.
Proxy: gate non-admin /key/generate budget_limits and permissions (#31469), reject non-finite budget_limits windows (#31630), hard-reject CLI session token personal-key budget_limits (#31631), reject team-scoped object_permission on personal keys for non-admins (#31471) — Four PRs that tighten the API surface on key generation: non-admins can no longer set budget limits, budget windows with NaN / infinity values are rejected at the API boundary, CLI-issued session tokens cannot carry per-key budget overrides, and non-admins cannot grant team-scoped object permissions on a personal key. These are the "least-privilege key" enforcement that the AI operational debt guide recommends; 1.92.0-dev.1 is the line where the enforcement is finally consistent across all four paths.
License seat limit: count only active users (#31227) — The enterprise license seat count was including deactivated user records, which caused spurious "license exceeded" alerts. 1.92.0-dev.1 filters to active users only.
Email: apply EMAIL_SIGNATURE to budget alert emails (#31712) — Until 1.92.0-dev.1, the EMAIL_SIGNATURE env var was only applied to a subset of email templates, leaving budget alerts without the operator-configured signature line. 1.92.0-dev.1 applies the signature to budget alerts too.
UI: stop Request Logs page from overflowing horizontally and size its columns (#31426), keep virtual-keys filters across delete and refresh (#31533), load virtual-keys team filter from the fast v2 endpoint (#31638), allow any git host on the skills add form (#31652) — Four UI-side polish fixes. The Request Logs page was overflowing on wide columns (a real problem on mobile and tablet), the virtual-keys filter state was lost on a delete or refresh, the team filter was hitting a slow endpoint, and the skills form rejected non-GitHub git hosts. Each is small; together they remove a class of operator-friction tickets that show up in the first week of a 1.92.0 stable deployment.
Performance: pre-warm upstream realtime connection pool + memoize per-request lazy import of otel runtime hooks + move cost-callback payload deepcopy off the request event loop (#31579, #31707, plus the realtime pool work from 1.91.x carrying forward) — Three latency wins. The OTel lazy-import memoization cuts per-request import overhead. The cost-callback deepcopy off-loop unblocks the event loop on heavy spend-log payloads. The realtime pool pre-warm cuts session-establishment latency on voice agents. For teams that have standardized on LiteLLM as their inference API gateway and are comparing the 1.92 line against alternative gateways in a procurement, the three perf wins change the relative cost of running LiteLLM at the edge of a voice-agent fleet versus a pure Cloudflare AI Gateway or Portkey setup. If you are tracking how the gateway landscape evolves alongside the LLMOps surface, the LLMOps platform comparison 2026 maps the 1.92.0-dev.1 changes against the eval / observability features that ship on the same cadence from the LLMOps platform vendors.

Upgrade via pip install litellm==1.92.0.dev1 (note the .dev1 suffix — PyPI uses dots, not dashes, for the dev tag) or docker pull ghcr.io/berriai/litellm:v1.92.0-dev.1. The 1.92.0-dev.1 image is signed with the same cosign key as 1.91.0-rc.1 (the pinned commit 0112e53), so the verification workflow is unchanged. There are no breaking changes in the dev tag, but two are worth flagging on the way to stable: the AES-256-GCM credential migration is forward-only (a backup of the database before the upgrade is the right posture), and the per-key budget enforcement rejection may surface a class of "non-admin tried to set a budget limit" 4xx errors that were previously silently accepted. The 1.92.0 stable tag has not shipped at the time of this writing; treat 1.91.0-rc.1 as the validated production pin and 1.92.0-dev.1 as the preview where the structural changes (credential encryption, Model Armor on attachments, per-token-type cost breakdown, A2A routing) get their first real-world shakedown. The LLMOps observability blueprint walks through how the new litellm_overhead_with_guardrails_latency_metric and the per-token-type cost entries fold into the same reconciliation story as the rest of the LiteLLM metrics — three new signals that together close the loop on cost attribution accuracy and guardrail-overhead alerting, and all worth adding to the dashboard before the 1.92.0 stable tag lands.

What's New in LiteLLM 1.91.0-rc.1

LiteLLM 1.91.0-rc.1 shipped 2026-06-28 as the first release candidate on the 1.91 line, cut from main via the internal staging promotion PR #31542. The stable tag has not shipped yet — 1.90.0 remains the recommended production pin until 1.91.0 stable lands — but the RC is the right place to validate the new surface if you are running a CI gate that mirrors a real eval suite. The headline changes from the 1.91.0-rc.1 release notes:

Prometheus: requested_model label on spend and requests metrics (#31410) — Until 1.91.0-rc.1, the Prometheus exporter carried the resolved (routed) model label but not the user-requested model, which made it impossible to distinguish "client asked for gpt-4o, we fell back to gpt-4o-mini" from "client asked for gpt-4o-mini directly" in a single dashboard. 1.91.0-rc.1 adds requested_model as a label on litellm_spend_total and litellm_requests_total, so the fall-back-rate chart finally has the data it has always needed. For teams that budget by requested model (the pattern the LLM FinOps strategies guide walks through), this is the line that closes the loop on attribution accuracy. The 1.91.0-rc.1 line also adds a per-team litellm_team_members_metric gauge (#31506) so chargeback dashboards can divide cost by active-seat count without scraping team-config endpoints.
MCP: shared OAuth token foundation + cross-replica single-flight refresh (#31275, #31493) — Two PRs that together replace the per-request OAuth dance with a shared token foundation: challenge, store seam, expiry-aware cache, single-flight refresh, and (on the v2 per-user OAuth store) cross-replica single-flight refresh. In practice this is a latency and reliability win for teams running the proxy behind a load balancer — until 1.91.0-rc.1, two replicas refreshing the same MCP user token would each fire their own refresh, which on a strict-rate-limited IdP returned 429 storms. The shared foundation collapses the per-replica refresh to a single flight, and the cross-replica v2 store extends the same protection to the authorization_code path (#31473). The 1.91.0-rc.1 cycle also stops logging MCP tool-call input in the MCP client (#31393) — a real security hardening move for teams that have been redacting tool inputs in their own exporters because the default log path was capturing them.
Performance: bound event-loop blocking from oversized requests (#31497) + proxy: cancel upstream LLM stream when client disconnects during time-to-first-token (#31499) — Two PRs that together fix two long-standing latency-tail sources. The event-loop bounding caps the work a single oversized request can push onto the loop, which previously caused head-of-line blocking across all other in-flight requests on a busy proxy. The TTFT-disconnect cancellation is the more visible fix: until 1.91.0-rc.1, a client that disconnected after sending the request but before the first token arrived would still keep the upstream provider call alive for its full TTL, leaking provider spend. The cancellation path propagates the disconnect to the upstream call so the spend stops accruing at the same instant. If you have ever seen your provider bill include requests whose clients had long since given up, this is the fix.
Bedrock: normalize Messages system role and adaptive-thinking for Claude Invoke (#31364) + Vertex AI: append rawPredict suffix for custom api_base on /v1/models (#31529) — Two provider-correctness fixes. The Bedrock path was double-normalizing the system role and dropping adaptive-thinking metadata on the Claude Invoke API; 1.91.0-rc.1 routes the call through the correct normalization. The Vertex AI fix is a smaller but real path-completion fix: teams using a custom api_base against Vertex were getting 404s on the raw-predict endpoint because the suffix was not being appended. Both are pure correctness — no behavior changes, just less 4xx noise on the provider dashboards. The Bedrock change pairs with the inference-backend context in the AWS Trainium and Inferentia guide for teams routing Anthropic-style tool definitions through Bedrock; the Vertex change matters most for teams running Gemini side-by-side with Anthropic on the same proxy.
Pass-through: stop registry growth on every reload (PERF-13) (#31313) + pass-through: forward all multipart files with repeated field names (#31391) — A memory-leak fix and a correctness fix. The pass-through route registry used to accumulate entries on every config reload, which on a long-running proxy that reloaded weekly (the hot-reload pattern) meant the registry grew unbounded and eventually OOM'd the process. 1.91.0-rc.1 makes the registry swap-and-discard on reload, so memory stays flat across the lifetime of the proxy. The multipart fix is the smaller one: pass-through endpoints that take repeated field names (the form-upload pattern that the OpenAI Files API and several Anthropic-compatible providers use) used to drop all but the first occurrence. 1.91.0-rc.1 forwards all of them. If you have ever seen a file-upload call silently lose files after the second one, this is the fix.
Pass-through: durable spend logging via the logging worker (#31485) + cost: preserve Anthropic server_tool_use web search usage in cost tracking (#31355) — Two changes that tighten the cost-attribution story. The pass-through logging worker means spend lines for pass-through calls now survive a proxy restart, where they used to be lost if the in-memory buffer had not been flushed yet. The Anthropic cost-tracking fix preserves the server_tool_use web-search usage block (the billable web-search token count) in the cost ledger, where it was being dropped. If your cost dashboard has been under-counting Anthropic traffic that exercised the web-search tool, 1.91.0-rc.1 brings it into line. The cost-dashboard patterns in the LLM cost monitoring tools 2026 roundup are the right place to start if you are wiring the new requested_model label and the litellm_team_members_metric gauge into a chargeback workflow.
Guardrails: headroom guardrail for message compression (#31407) + guardrails: match policy-pipeline block response to direct guardrail attachment (#31421) — Two guardrail-side changes. The headroom guardrail is a new opt-in guardrail that fires when message compression is about to drop below a configured token budget, useful for catching the "we are about to lose context because the conversation got long" case before it actually drops content. The policy-pipeline block response change aligns the response shape a guardrail emits when invoked via the policy pipeline with the shape it emits when attached directly, which removes a class of client-side parsers that were treating the two paths as different error types. For teams that wire the guardrail pipeline into a structured logging backend, the change closes the "is this an error or a block" ambiguity in the access log.
UI / agent fixes: show an agent's attached virtual key in the UI (#29619), router: persist global retry_policy via /config/update (#29540), OTel: point AgentOps OTLP exporter at otlp.agentops.ai (#31490), websearch: wrap agentic loop response in fake stream for streaming requests (#31484) — A small bag of polish fixes. The agent virtual-key UI add makes it possible to see which virtual key an agent is currently using from the agent detail page, useful for debugging "which key is this agent actually authenticating as" tickets. The router retry_policy persistence closes a long-standing gap where a global retry policy set via /config/update was not being written to the backing store, so a config reload reverted it. The AgentOps OTLP endpoint correction is a supply-chain fix — the exporter was pointing at a generic OTel collector URL, and the AgentOps SaaS expects a vendor-specific endpoint. The websearch streaming fix prevents an exception when a web-search tool returns an agentic-loop response to a streaming caller. None of these are breaking, all are pure correctness, and together they take the rough edges off the 1.90.0 stable line that teams will hit when they adopt the 1.91 cycle.

Upgrade via pip install litellm==1.91.0rc1 or docker pull ghcr.io/berriai/litellm:v1.91.0-rc.1. The 1.91.0-rc.1 image is signed with the same cosign key as 1.90.0 (the pinned commit 0112e53), so the verification workflow documented in 1.89.1 works as-is — just swap the version tag. There are no breaking changes in the RC; the new requested_model Prometheus label, the per-team litellm_team_members_metric gauge, the shared MCP OAuth token foundation, the event-loop bounding, the upstream-stream cancellation, the pass-through registry swap, the durable pass-through spend logging, and the guardrail headroom check are all additive on the 1.90 line. If you operate a hot-reload config pattern, the pass-through registry fix is the one to verify on the way up — confirm the proxy memory stays flat across a few reload cycles after the upgrade. The OTel, websearch, and router retry_policy fixes are silent wins; you should not notice any change beyond the new metrics being available and the old edge-case 4xx errors disappearing. The 1.91.0 stable tag has not shipped at the time of this writing; treat 1.90.0 as the production pin and 1.91.0-rc.1 as the validation target. For teams standardizing on the broader agent-eval pipeline that the RC is part of, the LLMOps observability blueprint shows how the new requested_model label, the litellm_team_members_metric gauge, and the durable pass-through spend logging fit into the same reconciliation story as the rest of the LiteLLM metrics — three new signals that all close the loop on cost attribution, and all worth adding to the dashboard before the 1.91.0 stable tag lands.

What is New in LiteLLM 1.90.0

LiteLLM 1.90.0 shipped 2026-06-27 as the stable line bump from 1.89.x and the release where the gateway story hardens across three surfaces: rate-limit error structure, MCP OBO token handling, and Bedrock Converse tooling. There is also a release candidate line following it — v1.91.0-rc.1 shipped 2026-06-28 with cosign signing on the same pinned key — so teams that want to test the RC against their own eval suite can pull ghcr.io/berriai/litellm:v1.91.0-rc.1 while the stable 1.90.0 line catches up via patches. The headline changes from the 1.90.0 release notes:

Rate-limit errors: standardized envelope with category, rate_limit_type, model, and llm_provider fields (#27687) — Until 1.90.0, rate-limit errors from different providers landed in your logs with provider-specific shapes, which meant downstream alerting rules had to special-case each provider. 1.90.0 emits a unified envelope so a single PromQL alert can key off category="rate_limit_error" across OpenAI, Anthropic, and Bedrock. The change pairs with the LLM latency monitoring 2026 guide — the alert that catches "we are about to be rate-limited on a model the rest of the stack depends on" finally works the same way regardless of provider.
MCP OBO: mirror upstream token lifetime instead of forcing a 1h expiry (#29951) — The on-behalf-of (OBO) auth flow for MCP servers used to cap the token at 1 hour regardless of what the upstream identity provider returned, which made long-running MCP tool calls hit a hard expiry boundary mid-execution. 1.90.0 mirrors the upstream token's actual lifetime. For teams using OBO to scope an MCP tool to a per-user identity (the pattern the MCP enterprise authorization guide documents), 1.90.0 is the line where long tool invocations stop hitting the artificial 1h cliff.
MCP: let non-creator users OAuth into OBO-mode MCP servers from the Tools page (#29867) — Non-admin team members used to hit a dead end when trying to OAuth into an OBO-mode MCP server — the UI assumed admin-only access. 1.90.0 lets any user with access-group membership complete the OAuth dance. For multi-tenant deployments where each user needs their own MCP identity, this is the change that makes the "Tools" page usable for non-admins.
MCP: load MCP tool config tools via the OBO/passthrough-aware GET path (#29960) — A correctness fix for the tool-listing endpoint that was using a non-OBO-aware path, meaning an OBO-scoped key would either fail to enumerate tools or enumerate the wrong set. 1.90.0 routes the GET path through the same OBO/passthrough logic the call path uses, so the tools/list response matches the tools the key can actually call.
MCP: highlight MCP cards red when the logged-in user is missing per-user env vars (#29856) — A UI affordance fix. When an MCP server requires per-user env vars that the operator has not provided, the MCP card on the dashboard now renders red instead of silently appearing available. For teams running a per-user MCP allow-list, the red-card state is the new "do not click this" signal.
Bedrock: forward strict and additionalProperties to Converse toolSpec (#29814) + Bedrock: Converse toolSpec via BedrockToolSpec dict subclass (#29869) — Two PRs that together bring Bedrock's Converse tool-spec path into alignment with the strict-schema validation other providers already use. If you have agents that build tool schemas programmatically, fewer requests will now 400 on Bedrock for missing strict or extra additionalProperties. For teams that route Anthropic-style tool definitions through Bedrock, the AWS Trainium and Inferentia guide is the broader inference-backend context for this kind of parity work.
Anthropic: Claude Fable 5 model added across Anthropic, Bedrock, Vertex AI, and Azure AI (#30064) + cost-map data-only hotfix for the hosted map (#30076) — Claude Fable 5 is now a first-class model on all four Anthropic-compatible backends, with cost-map entries in place. The cost-map data-only hotfix is the part to watch for FinOps teams — if you query LiteLLM's hosted cost map, the Fable 5 price card now exists end-to-end.
JWT auth: opt-in fallback to DB team on unresolved JWT claim (#28913) — A long-standing edge case in the JWT auth path: when a token arrives with a team claim that the proxy cannot resolve, the request used to 401. 1.90.0 adds an opt-in fallback that looks the team up in the DB. Useful for teams migrating from a static JWT issuer to a dynamic one (or operating a hybrid where some tokens come from a different IDP than the one in team_alias_map).
Caching: restore stored prompt_tokens on embedding cache hits (#30046) — Embedding cache hits used to recompute prompt_tokens from scratch on every hit, which inflated the cost line on cached calls. 1.90.0 restores the stored value. The cost dashboard numbers come into line on day one of the upgrade for any team relying on embedding caching.
Azure AI: MAI-Image-2.5 image generation support (#29688) — A new image generation model on the Azure AI backend. If you have been holding off on Azure AI image generation because the model picker was missing the entry, 1.90.0 unblocks that without an env-var workaround.
Proxy: self-heal startup/reload Prisma reads on engine disconnect (#28803) — A real production hardening fix. Prisma engine disconnects on proxy startup or config reload used to surface as 500s on the first request after the reload. 1.90.0 detects the disconnect, re-establishes the engine, and retries — meaning a config-reload or pod-restart no longer costs you the first batch of user requests. The change is particularly visible to teams running hot-reload config patterns in Kubernetes.
UI: budget duration on edit team member form (#29717) — A small but useful UI add. The team-member edit form now exposes budget duration (daily, weekly, monthly) inline, so per-member budget tuning does not require a separate config-file edit.
UI: admin flag to disable in-product UI nudges for everyone (#29796) — For teams operating LiteLLM as a shared instance across a large user base, the in-product UI nudges (the "try this feature" cards) are a noise tax. 1.90.0 ships an admin flag to disable them globally.

Upgrade via pip install litellm==1.90.0 or docker pull ghcr.io/berriai/litellm:v1.90.0. The 1.90.0 image is signed with the same cosign key as the 1.89 line (the pinned commit 0112e53), so the verification workflow documented below works as-is — just swap the version tag. No relay-proxy config changes are required; the JWT fallback is opt-in, the MCP changes are additive, and the rate-limit error envelope is the only behavior change that requires downstream alerting rules to be re-tuned. If you want to test the upcoming 1.91.0 cycle, ghcr.io/berriai/litellm:v1.91.0-rc.1 is the RC image to pull; it is signed with the same key and the verification command is the same shape with the RC tag substituted. The LLM observability guide covers the broader pattern for how the rate-limit envelope change and the Prisma self-heal move the needle on operational posture.

What is New in LiteLLM 1.89.4

LiteLLM 1.89.4 shipped 2026-06-25 as the third patch on the stable 1.89.x line, cut from stable/1.89.x via the backport rollup PR #31168. The release is small in surface area — no new features for end users, no breaking changes — but it backports the MCP "no-mcp-servers" sentinel from PR #31029, which closes a long-standing gap in how team-scoped API keys interact with the MCP server allow-list, and it ships a refreshed UI dashboard build alongside an enterprise dependency bump. If you are running the proxy with per-team MCP allow-lists — the pattern the MCP enterprise authorization guide documents — 1.89.4 is the patch that lets you explicitly scope a key to zero MCP servers without falling back to the team default.

MCP: scope a key to zero MCP servers with the no-mcp-servers sentinel (#31029) — Until 1.89.4, a key under a team that had MCP servers (e.g. [x, y, z]) had no way to be scoped to zero MCP servers. An empty mcp_servers list has always meant "inherit the team", so there was no value an admin could set to say "this key gets none." 1.89.4 introduces a no-mcp-servers sentinel (mirroring the existing no-default-models sentinel for model scoping) stored on the key's object_permission.mcp_servers. An explicit empty result overrides team inheritance, additive access-group grants, and allow_all_keys servers — the key's tools/list response is empty regardless of what the team or access-group policy says. Concretely, you create the key in the UI under Create Key → MCP Servers / Access Groups → "No MCP Servers", and a curl against /mcp with that key returns an empty tools list, while a sibling key with the MCP field left blank still inherits the team's full set. For teams using per-key MCP isolation as a defense-in-depth against tool-exfil patterns (the exact threat model the broader agentic incident harness guide uses as its worked example), 1.89.4 is the patch that makes the "this key gets no tools" policy enforceable without resorting to a separate proxy.
UI: rebuilt dashboard artifacts on stable/1.89.x (#31170) — A dashboard rebuild that ships with 1.89.4 to keep the UI assets in lockstep with the proxy release cadence. If you pin the proxy image and the dashboard separately (e.g. you serve the dashboard behind a CDN while the proxy runs in-cluster), confirm the dashboard artifact version lines up with the proxy version after the upgrade — 1.89.4 is the line where the UI rebuild matters most for the new MCP-sentinel surface, because the No MCP Servers picker ships in this dashboard build.
Enterprise: litellm-enterprise bumped to 0.1.42.post2 (#31259) — Enterprise-only bump that rolls up PRs #30787, #30788, #31035, #31036, #31122, and #31133. These are enterprise-tier fixes that ship with the public image but are only active under an enterprise license — for the open-source proxy deployment the bump is a no-op, but it does mean the image hash for 1.89.4 differs from a hypothetical pure-OSS build. Worth noting if you hash-pin your deploys and watch image diffs in CI; the hash will change even on a pure-OSS install.

Upgrade via pip install litellm==1.89.4 or docker pull ghcr.io/berriai/litellm:v1.89.4. The 1.89.4 image is signed with the same cosign key as the rest of the 1.89 line (the pinned commit 0112e53), so the verification workflow documented in 1.89.1 works as-is — just swap the version tag. No relay-proxy configuration changes are required for the OSS path; if you operate a per-team MCP allow-list, the only behavioral change to verify on the way up is the new no-mcp-servers sentinel — confirm any keys you previously created with an explicitly-empty mcp_servers field are still inheriting the team default (the sentinel is opt-in via the UI picker or the API, not auto-applied to existing keys). The AI operational debt guide uses 1.89.4's MCP-sentinel feature as the canonical "least-privilege key scoping" example because it is the first release where the key-level explicit-empty override is durable across proxy restarts and access-group changes.

What is New in LiteLLM 1.89.2

LiteLLM 1.89.2 shipped 2026-06-18 as the second patch on the stable 1.89.x line, cut from stable/1.89.x via the backport rollup PR #30681. There are no new features and no breaking changes — 1.89.2 is a pure stability release, but the seven backports in the rollup are exactly the kind of issues that show up as "this number on the cost dashboard looks wrong" or "this model is not appearing in the list endpoint" tickets on the on-call queue. If you are already on 1.89.1, the upgrade is a no-brainer; if you are still on 1.88.2 wondering whether the 1.89 line is safe, this is the patch that closes the remaining teeth-grinding edge cases.

OTel v2: record the full error message on the standard exception event (#30380) — The OpenTelemetry v2 exporter used to emit a generic exception event for failed requests without the underlying error string. On a busy proxy, that meant tracing a 500 back to the actual model error meant correlating against the access log, not the trace itself. 1.89.2 fixes the exporter to record the full error message on the standard OTel exception event. If you wired LiteLLM traces into Grafana Tempo or Honeycomb, the failed-request view now shows the real reason inline rather than a blank exception type.
Proxy: internal roles can access vector store CRUD routes (#30503) — The vector store CRUD endpoints (create / read / update / delete on managed vector stores) were restricted to admin roles, which blocked internal services that legitimately need to write to vector stores through the proxy. 1.89.2 corrects the role check so internal roles (proxy admin, internal-user) can hit the CRUD routes. This is the fix for the "my internal RAG indexer is getting 403" ticket.
Anthropic: surface service_tier in cost tracking (#30558) — Anthropic's pricing depends on the service_tier field (default vs. priority), and 1.89.2 finally surfaces that field in the LiteLLM cost ledger. If you have been seeing Anthropic cost deltas of a few percent per million tokens that did not line up with the published price cards, this is why — priority-tier traffic was being billed at the default-tier rate. The cost dashboard now reflects the real Anthropic bill.
Feature: opt-in healthy_only filter on GET /v1/models (#30130) — GET /v1/models now accepts a healthy_only=true query parameter that excludes models whose backing provider is currently failing health checks. Useful for client-side model pickers that were showing dead models to users; pair it with the LLM monitoring dashboard templates to see which providers the proxy thinks are healthy in real time.
Proxy: list the public team model name in /v1/models (#30588) — Teams that publish models under a public-facing alias (e.g. internal-claude -> claude-3-5-sonnet) saw the upstream model name leak in /v1/models, which broke the abstraction for client code. 1.89.2 fixes the response to return the public alias, not the upstream name. The change matters most for teams that sell API access to downstream customers and do not want to expose their provider relationships.
Proxy: resolve list-files credentials from team BYOK deployments (#30495) — Teams with bring-your-own-key (BYOK) deployments could not list files attached to their completions because the proxy was resolving credentials from the global config, not the team's BYOK bucket. 1.89.2 fixes the credential resolution to look up the team's BYOK config first. If you are running a multi-tenant deployment with per-team credential isolation, this is the patch that makes file-attach work end-to-end for non-admin teams.
Cost tracking: stop dropping cost on non-string service_tier (#30690) — When a provider returned service_tier as a non-string value (e.g. an enum or null), the cost-tracking code path would silently skip the entry rather than fall back to the default rate. 1.89.2 adds the type coercion so the cost line is always written. Pairs with the LLM cost monitoring tools 2026 roundup — if you are using a third-party cost dashboard, you may notice the totals come into line on the first day after the upgrade.

Upgrade via pip install litellm==1.89.2 or docker pull ghcr.io/berriai/litellm:v1.89.2. The 1.89.2 image is signed with the same cosign key as 1.89.1 (the pinned commit 0112e53), so the verification workflow documented in 1.89.1 works as-is — just swap the version tag. No relay-proxy configuration changes are required; the backports are pure additions / corrections on the 1.89 line. If you are on 1.88.2, 1.89.2 is the recommended upgrade target — it is the line where the OTel v2 fixes, the service_tier cost ledger, and the healthy_only filter all ship together.

What is New in LiteLLM 1.89.1

LiteLLM 1.89.1 shipped 2026-06-16 as the first patch on the stable 1.89.x line, cut from the new stable/1.89.x branch that 1.89.0's CI change introduced. It is a backport-and-fixes release: the 1.84.8 patch set is backported to 1.89.x, plus a batch of MCP, model-info, and database fixes rolled up under PR #30502. The bigger story for production teams is that 1.89.1 also formally documents the new cosign-based Docker image signature verification workflow that 1.87.x / 1.88.x backported earlier — release tags are now protected and resolve to the same signing key introduced in commit 0112e53, so the two verification paths (pinned commit hash and release tag) are equivalent in trust posture. The signing key itself is unchanged.

Backport: 1.84.8 patch set to 1.89.x — The 1.84.8 fixes are now also on 1.89.x, so teams on the new 1.89 line do not have to backport them manually. If you are riding 1.89.0-rc.x and were waiting on stable to pick up the 1.84.8 stability fixes, 1.89.1 is the line that ships them.
Backport: MCP / model-info / database fixes (PR #30502) — A rollup of fixes across the MCP integration (the Anthropic-compatible tool-call surface the MCP monitoring pattern builds on), the model-info registry, and the database layer. For teams running LiteLLM with the proxy pattern against Anthropic / MCP-backed agents, the MCP fixes are the ones most likely to show up as fewer spurious 4xx errors on tool calls. Treat this as a stability release — none of the backports introduce new behavior, they close gaps that were already there.
New: cosign Docker image signature verification workflow — 1.89.1 is the first release where the cosign verification story is documented as a feature, not just a supply-chain guardrail. You now have two equivalent paths to verify the image at deploy time. The pinned-commit-hash path is the strongest (a commit hash is cryptographically immutable, so you are verifying against the exact key the maintainers used at signing time):

cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \
  ghcr.io/berriai/litellm:v1.89.1

The release-tag path is the convenience path — useful when you are scripting a CI job that needs to verify whatever tag you are about to deploy without hardcoding a commit hash:

cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/v1.89.1/cosign.pub \
  ghcr.io/berriai/litellm:v1.89.1

Both paths resolve to the same key, so for the cost of one extra cosign verify step in your deploy pipeline you get a hard guarantee that the image you are pulling is the one the LiteLLM maintainers actually signed. For teams that already pin by SHA in docker pull, the cosign check is a defense-in-depth — it catches the case where a registry-side redirect swaps the image under a pinned tag. The two checks together are the right posture for a regulated deployment.

Upgrade via pip install litellm==1.89.1 or docker pull ghcr.io/berriai/litellm:v1.89.1. No relay-proxy config changes are required — the backports are pure additions on the 1.89 line. If you are on 1.88.2, the 1.89.1 backports are a drop-in upgrade; if you are on 1.89.0-rc.x, you already have the equivalent fixes plus the OTel/metrics work that defines the 1.89 release. The 1.89 stable line is the recommended production pin going forward, with 1.88.2 as the conservative fall-back for teams that want one more cycle on the older line before adopting 1.89. The inference API gateway comparison covers the broader gateway landscape if you are evaluating LiteLLM against alternatives like Portkey, OpenRouter, and Cloudflare AI Gateway — the cosign workflow here is part of why LiteLLM is the strongest choice for teams with hard supply-chain requirements.

What is New in LiteLLM 1.88.2

LiteLLM 1.88.2 shipped 2026-06-14 as the latest stable release on the 1.88.x line and the recommended production pin for teams not yet ready to ride the 1.89.0-rc cycle. Two release-pr chore PRs landed back-to-back (#30144 and #30408) and pulled the same five high-value backports that 1.87.2 promoted one cycle earlier — Fable 5 model support, batch-file auth, CrowdStrike AIDR integration, Mantle Responses SigV4, and a NetApp streaming-cost fix — but they now sit on the newer 1.88 line. Cosign image signing on the pinned commit key is also enforced, so the supply-chain posture is unchanged from 1.87.2.

Fable 5 model backport — Anthropic's latest snapshot (Fable 5) is now supported on the 1.88 line. If you have been holding off on the 1.89 RC specifically to wait for Fable 5, 1.88.2 unblocks that without forcing an upgrade into the RC line.
Batch-file auth support — Auth credentials can now be loaded from a batch file when running bulk operations against multiple provider endpoints. The previous per-request credential model became a bottleneck for teams running scheduled re-scoring jobs against hundreds of thousands of stored completions; if you are integrating LiteLLM with a job runner like Dagster or Airflow for nightly eval batches, 1.88.2 is the line that handles that flow natively.
CrowdStrike AIDR integration — A new managed-detection callback ships in 1.88.2, letting teams forward LiteLLM request metadata to CrowdStrike's AI Detection & Response service without writing a custom exporter. Useful for regulated industries that already standardize on Falcon log ingestion. Pairs cleanly with an OTel-based audit trail in Grafana Tempo if you are also wiring the MCP monitoring pattern for tool calls in the same environment.
Mantle Responses SigV4 auth — Mantle (an Anthropic-compatible provider) gained first-class SigV4 signing on the Responses API path. If you route Mantle traffic through LiteLLM, the 1.88 line now handles auth natively instead of requiring a custom callback.
NetApp streaming-cost fix — A backport that addresses cost-attribution drift when LiteLLM streams completions to NetApp-backed object storage. If your cost dashboard had unexplained per-token deltas on streamed responses to NetApp, this is the fix.

cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \
  ghcr.io/berriai/litellm:v1.88.2

Upgrade via pip install litellm==1.88.2 or docker pull ghcr.io/berriai/litellm:v1.88.2. No relay-proxy config changes are required — these are pure additions on the 1.88 line. If you are on 1.87.x, the 1.88.2 backports are a drop-in upgrade; if you are on 1.89.0-rc.x, you already have them. For teams running an SLO-based posture on cost and TTFT, this is the right stable pin while 1.89 stabilizes; the AI SLO/SLA contracts guide shows how to write the gate that protects against a forced rollback if 1.89 ships with a regression.

What is New in LiteLLM 1.89.0

LiteLLM 1.89.0 is currently shipping as release candidate v1.89.0-rc.1 with the stable tag expected in the next few days. The release leans into two themes that matter for production observability: tighter OTel/metrics integration and fewer silent failure modes in the proxy layer. The headline changes from the 1.89.0 release notes:

OpenTelemetry: team_metadata sub-keys now propagatable via W3C baggage (#29442) — The OTel exporter learned to selectively promote specific team_metadata sub-keys into baggage headers. For teams running OpenTelemetry distributed tracing across multi-service AI stacks, this means a team identifier set in one service's team_metadata now shows up in the trace context of downstream services without custom propagation code.
Datadog exporter: split oversized batches on 413 instead of re-queueing forever (#29444) — A real production bug fix. When the Datadog logs intake returned HTTP 413 (payload too large), the exporter used to re-enqueue the batch at the same size, hitting 413 in an infinite loop and silently dropping the batch. 1.89.0 splits the batch on 413 and retries with smaller chunks. If you run LiteLLM with the Datadog log callback, you were losing log volume on bursts; the fix is non-trivial for cost dashboards that looked suspiciously low on heavy-traffic days.
Release process: stable/X.Y.x branches auto-created on X.Y.0 tags (#29457) — A CI change that affects how the project cuts patch releases. Not user-facing for downstream consumers, but it means hotfixes will land on a stable branch and ship as 1.89.1, 1.89.2, etc. without waiting for a full release cycle.
Anthropic + Fireworks: inline legacy $ref defs in tool schemas (#28646) — A long-standing compatibility fix for tool-use JSON schemas that reference older $ref shapes. If your agents build tool schemas programmatically with shared component definitions, fewer requests will now 400 on schema validation.
Cosign image signing continues — Every 1.89.0-rc.x image is signed with the same pinned key (commit 0112e53). Verify the RC image with:

cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \
  ghcr.io/berriai/litellm:v1.89.0-rc.1

The monitoring improvements from 1.87.0/1.88.0 — the litellm_budget_resets_total metric, per-provider status in the relay proxy /health endpoint, and corrected cost attribution for batched cached responses — all carry forward unchanged. Upgrade via pip install litellm==1.89.0 once the stable tag ships, or pull ghcr.io/berriai/litellm:v1.89.0-rc.1 now to validate against the RC. No relay-proxy configuration changes are required.

What is New in LiteLLM 1.87.2

LiteLLM 1.87.2 shipped 2026-06-11 as the previous stable patch on the 1.87 line. With 1.88.2 now the recommended production pin, 1.87.2 is the older stable line that continues to receive backports for teams pinned to it for compatibility reasons. 1.87.2 introduced the cosign-signing workflow that protects the supply chain, and the same five backports (Fable 5, batch-file auth, CrowdStrike AIDR, Mantle SigV4, NetApp streaming-cost fix) originally landed here before being promoted to 1.88.2:

Cosign image signing is now mandatory for every release — Every LiteLLM Docker image (including the 1.87.2 patch) is signed with cosign using the pinned key from commit 0112e53. A commit hash is cryptographically immutable, so this is the strongest way to confirm you are pulling the original signing key. To verify the 1.87.2 image at deploy time:

cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \
  ghcr.io/berriai/litellm:v1.87.2

Fable 5 model backport — Anthropic's latest snapshot (Fable 5) is now supported on the 1.87 line. If you are pinned to 1.87.x and waiting on 1.89 before adopting new models, 1.87.2 unblocks that without forcing an upgrade into the RC line.
Batch-file auth support — Auth credentials can now be loaded from a batch file when running bulk operations against multiple provider endpoints. The previous per-request credential model became a bottleneck for teams running scheduled re-scoring jobs against hundreds of thousands of stored completions.
CrowdStrike AIDR integration — A new managed-detection callback ships in 1.87.2, letting teams forward LiteLLM request metadata to CrowdStrike's AI Detection & Response service without writing a custom exporter. Useful for regulated industries that already standardize on Falcon log ingestion.
Mantle Responses SigV4 auth — Mantle (an Anthropic-compatible provider) gained first-class SigV4 signing on the Responses API path. If you route Mantle traffic through LiteLLM, the 1.87 line now handles auth natively instead of requiring a custom callback.

Upgrade via pip install litellm==1.87.2 or docker pull ghcr.io/berriai/litellm:v1.87.2. No relay-proxy config changes required — the backports are pure additions on the 1.87 line. If you are already on 1.89.0-rc.1, the 1.87.2 backports are already in 1.89 main; you do not need to roll back. The 1.87 stable line continues to receive security and auth backports independently of the 1.89 RC work, which is the right posture for production teams that want a slow-moving pin.

What LiteLLM Does in Your Stack

LiteLLM has become the standard API gateway layer for teams running multiple LLM providers. Instead of managing separate SDK integrations for OpenAI, Anthropic, Azure, vLLM, Ollama, and a dozen other providers, teams standardize on LiteLLM's single interface — and get unified logging, cost tracking, and rate limiting as a byproduct.

But that ubiquity creates a new monitoring challenge: LiteLLM is now a critical path component. When it goes down, every AI feature in your stack goes down. And because LiteLLM sits at the aggregation point for all model calls, the metrics it emits are the most complete view of your AI infrastructure health you can get.

This guide covers what to monitor in a LiteLLM production deployment — from the core proxy metrics to spend tracking, latency percentiles, and error classification across providers.

LiteLLM runs as a proxy server between your application code and your LLM providers. Your code calls litellm.completion() or sends requests to the proxy endpoint, and LiteLLM handles:

Provider routing — sending requests to OpenAI, Anthropic, Azure, or self-hosted endpoints based on model name or cost
Unified interface — same request/response format regardless of which provider backs it
Cost normalization — converting provider-specific pricing into a unified cost ledger
Rate limiting — enforcing per-model, per-team, and per-API-key rate limits
Retries and fallbacks — automatically retrying on provider errors and falling back to cheaper models when limits are hit

LiteLLM also ships a relay proxy (litellm --port 8000 --detailed_settings) that runs as a dedicated service with its own metrics endpoint and separate configuration from the SDK usage pattern. Because LiteLLM sits at the aggregation point for every model call in the stack, the seven metrics below are also the spine of the broader AI/ML pipeline observability surface — the AI/ML pipeline observability guide shows how to correlate these LiteLLM-specific metrics with upstream embedding jobs, downstream eval runs, and the feature-store metrics that touch the same request path, so an anomaly in one place shows up as a single coherent incident rather than three disconnected alerts.

What metrics should you monitor in a LiteLLM production deployment?

1. Request Volume and Throughput

Track total requests and tokens processed per minute. LiteLLM exposes these via its /metrics endpoint in Prometheus format:

# HELP litellm_requests_total Total number of calls to litellm
litellm_requests_total{model="gpt-4o", team="backend"} 4821
litellm_requests_total{model="claude-3-5-sonnet", team="frontend"} 1204

Slice by model and team labels to understand which parts of your product are driving volume. A sudden spike in team="backend" requests while team="frontend" stays flat tells you something specific changed in your backend workflows, not your user-facing features.

Track rate(litellm_requests_total[5m]) to get requests per second.

2. Token Throughput

Both input and output tokens:

litellm_tokens_used_total{model="gpt-4o", token_type="input"} 2847392
litellm_tokens_used_total{model="gpt-4o", token_type="output"} 1847293

Use these to calculate cost. For OpenAI models, input tokens are charged at $2.50-15.00 per million tokens and output at $10-60 per million. Calculate your per-token cost:

cost_per_request = (input_tokens * input_rate + output_tokens * output_rate) / 1_000_000

Track this in Grafana with a PromQL query like sum by (model) (rate(litellm_tokens_used_total[1h])).

3. Latency Percentiles

LiteLLM latency has three distinct components:

Time to first token (TTFT) — dominated by provider API latency and your model choice
Time per output token (TPOT) — determined by provider throughput
End-to-end latency — from your application sending the request to receiving the full response

Track all three via the litellm latency histogram. Separate p50 from p95 and p99. A healthy p50 with a spiking p99 means you have tail latency outliers — often a specific model or provider behaving badly under load. For the deeper latency story — how TTFT, TPOT, and end-to-end each decompose into provider, network, and queue time, plus the PromQL that actually catches a regression before users do — the LLM latency monitoring 2026 guide walks through the metric definitions, alert thresholds, and dashboard patterns that work in production.

4. Error Rate by Provider

LiteLLM surfaces errors with provider-specific labels:

litellm_errors_total{error_type="rate_limit_error", provider="openai"}
litellm_errors_total{error_type="authentication_error", provider="anthropic"}
litellm_errors_total{error_type="timeout", provider="azure"}

Track error rate as a percentage of total requests:

sum(rate(litellm_errors_total[5m])) by (provider) /
sum(rate(litellm_requests_total[5m])) by (provider)

A rising rate_limit_error rate on OpenAI is a leading indicator you need to either switch to a fallback model or negotiate higher limits. Don't wait for the error rate to hit 100% — alert at 5%. For teams that have moved beyond hard thresholds into distribution-aware alerting (p95 shifts on error-rate distributions, quantile-based SLOs, etc.), the probabilistic observability guide shows how to swap the > 0.05 rule above for a CUSUM or Bayesian-shift detector that catches a slow drift in error rate before any single bucket crosses the threshold.

5. Cost Tracking

LiteLLM has built-in spend tracking that logs every request with its calculated cost. The spend logs table (if you're using the database adapter) includes:

field	description
`model`	e.g. `gpt-4o`, `claude-3-5-sonnet-20241022`
`total_cost`	Calculated cost in USD
`total_tokens`	Input + output tokens
`response_ms`	Response time in milliseconds
`user`	Team or API key identifier
`metadata`	Custom metadata passed at call time

Query this to build a cost dashboard. Track spend by user (team) to allocate AI costs back to product teams. This is essential for FinOps reporting and for detecting runaway experiments. For teams that want the broader pattern — the cost-by-workflow view that maps LiteLLM token usage to a feature surface and surfaces where the spend is actually coming from — the token cost by workflow guide walks through the labeling, slicing, and alerting that turns per-team attribution into actionable chargeback.

6. Cache Hit Rate

LiteLLM supports semantic caching via Redis. When enabled, semantically similar requests return cached responses instead of calling the provider:

litellm_cache_hits_total{model="gpt-4o"}
litellm_cache_misses_total{model="gpt-4o"}

Cache hit rate directly reduces your cost:

litellm_cache_hits_total / (litellm_cache_hits_total + litellm_cache_misses_total)

A low cache hit rate (below 30%) might mean your cache TTL is too short or your request patterns are too diverse. Tune with the caching_budget parameter at call time.

7. Fallback Success Rate

When LiteLLM falls back from a primary to a secondary model, track whether the fallback succeeds:

litellm_fallbacks_total{from_model="gpt-4o", to_model="gpt-4o-mini", success="true"}
litellm_fallbacks_total{from_model="gpt-4o", to_model="gpt-4o-mini", success="false"}

A fallback that fails 100% of the time is worse than no fallback — it's adding latency with no reliability benefit. A 50% success rate on gpt-4o-mini might be acceptable for non-critical features but unacceptable for customer-facing chat.

If you are designing the fallback layer as a first-class piece of routing logic rather than a config switch, the multi-LLM routing and ensembling guide walks through how production teams combine LiteLLM with cost-plus-latency routing across GPT-4o, Claude, and Gemini to cut spend 40%+ while keeping fallback paths coherent. For the broader FinOps posture that wraps around that routing — tiered model selection, prompt-level optimization, and the cost-attribution math that has to reconcile against the provider invoice — the LLM FinOps 2026 playbook is the next read; the LiteLLM cost ledger is the data source that strategy is built on.

How do you set up Prometheus monitoring for LiteLLM?

LiteLLM's proxy endpoint exposes a Prometheus-compatible /metrics endpoint. Add it to your Prometheus scrape config:

scrape_configs:
  - job_name: 'litellm'
    static_configs:
      - targets: ['litellm-proxy:4000']
    metrics_path: '/metrics'
    scrape_interval: 15s

The /metrics endpoint includes all the metrics above plus several more: litellm_requests_remaining (rate limit headroom), litellm_current_load (concurrent requests), and model-specific pricing labels.

Grafana Dashboard

Import the official LiteLLM Grafana dashboard (ID: 19253) or build your own. Key panels:

Request volume — requests per second by model, colored by provider
Token throughput — input and output tokens per hour
Cost accumulator — cumulative daily spend by model
Latency percentiles — p50/p95/p99 over time, faceted by model
Error rate — errors per 100 requests by provider
Cache efficiency — hit rate as a gauge with 30% and 70% thresholds
Fallback heatmap — when fallbacks fire and whether they succeed

Recommended Tool Grafana Cloud

Full-stack observability for LiteLLM — Prometheus metrics, Grafana dashboards, and managed alerting. 14-day retention free.

What should you alert on in a LiteLLM deployment?

At minimum, alert on:

groups:
  - name: litellm_alerts
    rules:
      - alert: LitellmHighErrorRate
        expr: |
          sum(rate(litellm_errors_total[5m])) by (provider)
          / sum(rate(litellm_requests_total[5m])) by (provider)
          > 0.05
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "LiteLLM error rate above 5% for provider {{ $labels.provider }}"

      - alert: LitellmRateLimitHeadroom
        expr: litellm_requests_remaining / litellm_max_parallel_requests < 0.2
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "LiteLLM rate limit headroom below 20%"

      - alert: LitellmHighP99Latency
        expr: histogram_quantile(0.99, rate(litellm_request_duration_bucket[5m])) > 30
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "LiteLLM p99 latency above 30s"

Recommended Tool Helicone

AI observability for LiteLLM — logging, analytics, and caching with a 5-minute setup. Tracks cost, latency, and usage across all providers.

Cost Allocation by Team

Pass user and metadata at call time to get per-team cost attribution:

response = litellm.completion(
    model="gpt-4o",
    messages=[{"role": "user", "content": "analyze this data"}],
    user="data-team",
    metadata={
        "feature": "analytics-dashboard",
        "environment": "production",
        "request_id": "req_abc123"
    }
)

These fields appear in spend logs and allow Grafana to partition costs by team, feature, or environment. For chargeback reporting, group by user to get a per-team AI spend report. If you are building the chargeback model from scratch, the LLM API cost calculator turns the same per-token pricing data that LiteLLM stores in the spend log into a forward-looking cost forecast — useful for budgeting the next quarter against the spend you are seeing today, and for answering the "what would happen if we doubled the team" question without rerunning the model.

LiteLLM vs Running Provider SDKs Directly

The monitoring argument for LiteLLM is strong: instead of five different logging interfaces from five different providers, you get one metrics endpoint with unified cost and latency tracking. The tradeoff is added infrastructure — LiteLLM itself becomes a dependency — and a small latency overhead (typically 5-15ms for the proxy routing layer).

For teams running two or more providers, the unified observability and cost normalization justify the overhead. For single-provider teams, the operational complexity might outweigh the benefits until you scale. If you are still weighing whether you need an LLM-specific monitoring layer versus extending your existing APM, the AI model monitoring vs APM comparison walks through the scenarios where each path makes more sense — including the specific metrics LiteLLM surfaces that traditional APM tools do not, like per-token cost attribution and provider-fallback success rates. And if your stack mixes self-hosted fine-tuned models with hosted APIs, the same proxy is the right place to route both — the fine-tuning in production guide covers the observability gaps that appear when a self-hosted LoRA adapter on vLLM is sitting behind the same gateway as your hosted traffic.

Recommended Tool Portkey AI

AI gateway with LiteLLM compatibility — full observability, cost management, and multi-provider routing. 50+ models supported.

Conclusion

LiteLLM's value as an API gateway is matched by its value as a monitoring aggregation point. The /metrics endpoint gives you request volume, token throughput, cost tracking, error classification, cache efficiency, and fallback behavior — everything you need to operate a multi-provider LLM stack with confidence.

The key panels to have in Grafana: latency percentiles by model, cost accumulator by team, error rate by provider, and fallback success rate. Alert when error rate exceeds 5%, when p99 latency crosses 30 seconds, or when rate limit headroom drops below 20%.

Combined with LiteLLM's built-in spend logs, you get the full picture: how much you're spending, where, and why — which is the foundation of any serious LLM FinOps practice. If you are also routing self-hosted inference (SGLang, vLLM, Ollama, TRT-LLM) through the same proxy, the SGLang production monitoring guide covers the RadixAttention-aware metrics that the stock LiteLLM panel does not surface out of the box, and the state of observability 2026 piece is the broader read on where the agentic-AI observability stack is heading next. For the FinOps side of the same picture — what happens when a cost anomaly surfaces and an engineer has to investigate it before it lands on the provider invoice — the AWS FinOps Agent 2026 frontier-agent pattern walks through how a frontier model investigates cost anomalies in plain English inside Slack and Jira, which pairs naturally with the per-token attribution LiteLLM provides. And if the LiteLLM gateway is sitting in front of a multi-thousand-dollar-per-month observability bill (the typical Datadog shape), the Datadog migration playbook walks through three Grafana + Prometheus + Tempo migrations that averaged 80% savings on the same LiteLLM-aggregated metrics — the playbook shows the exact sequence and the three dashboards you need first.