What's new in ClickHouse 26.8.2 LTS (vs 26.8.1)?

The ClickHouse 26.8.2.7-lts release shipped 2026-09-01 as a small patch on the 26.8 LTS line, two days after this article's last refresh. ClickHouse still publishes release notes in the canonical changelog rather than the GitHub release body, but the patch-line cadence matters for trace-store operators: 26.8.2 ships alongside a 26.3.28.5-lts bump the same day, so both pin lines advance together. Three things matter for teams running the self-hosted ClickHouse + HyperDX pair this article documents:

  • The 26.3 LTS line advances in lockstep: 26.3.26 → 26.3.28. The 26.3.28.5-lts release cut on 2026-09-01 alongside 26.8.2 and is the new forward-roll target for the LTS pin this article recommends (v26.3.19.3-lts). The 26.3 LTS line stays the production pin through the end of 2026; 26.8 is the stable line for new deployments that can take the JSON-column migration. The schema and the $180/mo versus $4,200/mo math in the body of this article are unchanged against 26.3.28 — the CODEC choices (DoubleDelta, ZSTD(8) on the timestamp column, ZSTD(8) on the rest) and the partition-by-month ORDER BY (service_name, gen_ai.request.model, timestamp) both forward-compat cleanly.
  • No schema or CODEC changes between 26.8.1 and 26.8.2. The patch is bug-fix shaped across the trace-store-relevant paths (merge scheduling, JSON column read paths, system.parts visibility). Operators on 26.8.1 can roll forward without a re-partition or a re-CODEC pass. The OTel gen_ai.* schema this article documents is unchanged.
  • If you are still on the 26.7 stable line, 26.8.2 is the new recommendation. The 26.7 → 26.8 transition is the one that adds the JSON column type this article calls out as additive; rolling from 26.7.4 → 26.8.2 picks up every fix on the stable line since the 26.8.0 cut without forcing a fresh re-partition. For teams running the OTel-only trace-store shape the OpenTelemetry AI inference tracing guide covers, 26.8.2 is the cleanest forward target.

Upgrade with the standard ClickHouse package path or pull the matching container; the schema and CODEC choices in this article are unchanged against 26.8.2. For teams that have not yet pinned a version in production, 26.8.2 is the new floor on the 26.8 stable line, and 26.3.28 is the new floor on the 26.3 LTS line — pick the line that matches your migration posture. For the broader storage trade-off the new JSON type opens up, the State of Observability 2026 guide walks through the trace-store-versus-column-store positioning.

What's new in ClickHouse 26.8.1 LTS (vs 26.7.4)?

The ClickHouse 26.8.1.2041-lts release shipped 2026-08-30 as the first 26.8 LTS patch line and the recommended target for production rollouts starting in late August. ClickHouse still publishes release notes in the canonical changelog rather than the GitHub release body, and the 26.8 line that began cutting stable tags in late July 2026 carries material changes for the self-hosted ClickHouse + HyperDX pair this article documents. Four of them matter for trace-store operators:

  • The JSON column type graduates to default-recommended — Until 26.8, the JSON type that lets you query nested span attributes without flattening them into Map(LowCardinality(String), String) was stable but still flagged with a "consider the trade-offs" comment in the docs. 26.8 promotes it to the default-recommended type for dynamic OTel payload storage and removes the warning. The subcolumn-discriminator statistics that landed in 26.5 stay on, so attributes['gen_ai.request.model'] = 'gpt-5'-style filters on a JSON column are index-backed rather than full-column scans. For a fresh deployment of the schema in the section below, switching the attributes column from Map(LowCardinality(String), String) to JSON post-hoc is an ALTER TABLE ... MODIFY COLUMN with a table-locked copy, so the right call is to start new tables on JSON and migrate the legacy Map columns on the next partition rotation. The OpenTelemetry gen_ai.* convention guide walks through the attribute shape JSON needs to remain queryable as the OTel working group stabilizes the agent attribute bag.
  • Lightweight deletes stabilize for trace TTL — The DELETE FROM llm_spans WHERE timestamp < now() - INTERVAL 90 DAY metadata-only delete path that 26.7 introduced is now the documented default for trace-table TTL, and the warning about edge-case correctness in distributed deployments is gone. For the cost numbers in this article — the $180/mo on a 1-year reserved r6i.2xlarge — the lightweight-delete path is what keeps the same instance sized appropriately under a multi-month retention floor without manual partition rotation. If you were holding out on the 26.7 lightweight-delete behavior waiting for it to lose the experimental flag, 26.8 is the line where the wait ends.
  • Async insert backpressure gains a tunable backoff window — 26.6 added backpressure on the async-insert buffer that surfaced as a single TOO_MANY_PARTS error rather than as silent drops under sustained write load. 26.8 adds a tunable backoff window (async_insert_max_query_number + async_insert_busy_timeout_ms) so the gateway can absorb a brief burst and retry rather than rejecting the flush outright. For the HyperDX-fronted ingest path that runs the async-insert on every OTel collector flush, this is the change that turns ingestion backpressure from a hard failure into a recoverable queue — which matters on the first sustained burst after a long quiet period.
  • 26.3.26 LTS ships the same day, one patch ahead of the 26.3.19 LTS this article was originally authored against — The 26.3.26.3-lts release cut on 2026-08-30 alongside 26.8.1.2041-lts and is the safe forward-roll target for the LTS pin the article recommends (v26.3.19.3-lts). The 26.3 LTS line stays the production pin through the end of 2026; 26.8 is the stable line for new deployments that can take the JSON-column migration. The cost and schema math below are unchanged against 26.3.26 — the CODEC choices (DoubleDelta, ZSTD(8) on the timestamp column, ZSTD(8) on the rest) and the partition-by-month ORDER BY (service_name, gen_ai.request.model, timestamp) both forward-compat cleanly.

For the broader storage trade-off the new JSON type opens up, the State of Observability 2026 guide walks through the trace-store-vs-column-store positioning; for the side of this that drives cost, the Datadog migration playbook shows the same arithmetic applied to a real migration.

What's new in HyperDX 2.37.0 (vs 0.6.2 in this article)?

The HyperDX 2.37.0 release shipped 2026-08-28 as a coordinated bump across the @hyperdx/app, @hyperdx/api, @hyperdx/otel-collector, and @hyperdx/common-utils packages. This article was originally authored against the HyperDX CLI 0.6.2 line (a sub-package version that tracks independently of the main server), and 2.37.0 is the first server release that closes three observability gaps the CLI work touched on. Three of the changes matter for the self-hosted ClickHouse + HyperDX pair:

  • Per-target notification timing on alert evaluations (@hyperdx/api CHANGELOG, 0558f77e) — Until 2.37.0, webhookDurationMs on an alert evaluation was a single number covering the whole delivery, and because targets dispatch concurrently the slowest one set the figure. A multi-target alert reported a duration with no way to tell which webhook was responsible or that the other targets were fine. Each dispatch is now timed individually and aggregated per target across the evaluation, with a row per distinct target carrying webhook id, display name, summed duration, dispatch count, and failure count. Stored per evaluation rather than per dispatch (a 50-group alert with 10 targets previously wrote 500 entries onto every history row), capped at ALERT_NOTIFICATION_TARGETS_LIMIT, sorted slowest-first. For the alert side of a HyperDX-fronted production stack — the layer that wakes you up when the ClickHouse-side async insert path starts backpressuring — this is the change that lets you answer "which webhook is slow" without grepping logs. The broader alert UX gets a sibling change: edit/delete/Terraform export move behind the same overflow menu the alerts list uses, and a new "source" filter narrows the list by what an alert watches (saved search vs dashboard tile).
  • Inline alert source lands in the backend (@hyperdx/api CHANGELOG, df4a7a55) — Until 2.37.0, alerts required a saved search (logs) or a dashboard tile (metrics). 2.37.0 adds an inline alert source that persists its own chart config directly on the alert. The config is the same shape a dashboard tile stores — builder configs on log/trace/metric sources plus raw SQL — and the check-alerts task evaluates inline alerts through the same code path as tile alerts, including group-by and multi-window behavior. Backend only in 2.37.0; the creation/edit UI and external API v2 support land separately. For teams running the alert-on-MCP-server-latency pattern from the MCP monitoring guide, this is the lever that lets you ship an alert without first wiring up the saved search the alert was anchoring to.
  • MCP server advertises dashboard variables + Gemini-compatible schemas (@hyperdx/api CHANGELOG, 5fc33413, 892cc653, b52a6fa8) — Two related MCP-side changes that matter for the agent-side observability story: dashboard variables are now resolvable through the MCP server, so an AI agent reading the dashboard view can pass a variable override (e.g. tenant = 'acme-corp') the same way the web UI does; and the MCP quantile level field is now advertised as a string enum, fixing a wire-compatibility issue where Gemini-backed clients had their entire tool list rejected because of one field's enum typing. Affected tools: clickstack_timeseries, clickstack_table, clickstack_save_dashboard, clickstack_patch_dashboard. Numeric input is still accepted for callers working from a cached schema, and the value is coerced back to a number before any consumer sees it. The metrics explorer adds a browse control with prefix hierarchy (system → cpu → utilization) plus search across every name and description the source reports, which is the change that makes the chart editor usable on a 3,000-metric namespace.

The upgrade from 2.x-line prior to 2.37.0 is a coordinated bump across the app, API, OTel collector, and common-utils packages — pin all four at the matching 2.37.0 tag rather than mixing. The CLI stays at the 0.6.x line independently, and the chart-formula + save_dashboard round-trip work that 0.6.2 shipped forward-compat cleanly into the 2.37.0 server. For teams that were waiting on a HyperDX release that made the MCP surface genuinely agent-useful rather than just JSON-RPC-shaped, 2.37.0 is the line that closes the gap. The 1,200+ agents OTel stack guide covers the broader pattern the MCP surface is the wire form of; the state of observability in 2026 frames the semantic-observability trend the HyperDX + ClickHouse combination is the storage layer of.

What's New in @hyperdx/cli 0.6.2 (chart formulas on the terminal + dashboard API)?

@hyperdx/cli 0.6.2 shipped 2026-08-21 as a single-patch release that lands one substantive feature: chart formulas across every API surface that persists or accepts chart configs, plus the MCP save_dashboard / patch_dashboard tools. The change is small in line count but operationally meaningful for anyone running a self-hosted ClickHouse + HyperDX stack and consuming the dashboard data from an AI agent or terminal pipeline. From the 0.6.2 changelog:

  • Chart formulas round-trip through every API surface (dc29d57f) — The external dashboards API v2 and the MCP save_dashboard / patch_dashboard tools now accept formulas (letter-ref arithmetic over the tile's select items — e.g. A / (A + B) * 100 for a conversion rate) and showOperandSeries on line, stacked bar, table, and number builder tiles, and they round-trip them through GET / PUT. The expressions are validated on write: unknown series refs, malformed syntax, combining formulas with asRatio, multiple formulas on a number tile, and formulas on formula-incapable source kinds (anything other than metric, log, or trace) are all rejected with actionable errors. For teams that script the dashboard config from an external system — the same pattern the LLM monitoring dashboard templates guide documents — this is the API surface that lets you build a derived series without re-uploading the tile as a metric query.
  • MCP query_tile computes formula columns for both metric and log/trace event tiles — The MCP tool that the agent-side observability pattern relies on now returns the formula-computed columns alongside the raw series. For teams running the 1,200+ agents OTel stack pattern, this is the change that lets an agent build a derived metric (cost-per-request, cache-hit-rate, p99/p50 ratio) on a single MCP call rather than computing it on the client.
  • Query-guide prompt documents formulas, OpenAPI spec adds the Formula schema — The two discoverability surfaces that make a new feature actually used in production. The MCP query-guide prompt now includes a worked example for A / (A + B) * 100, and the OpenAPI spec published at the same v2 endpoint lists Formula as a first-class schema so client generators pick it up. The State of Observability 2026 guide covers the broader trace-store-vs-column-store positioning that the formula-on-the-query side opens up.
  • CLI dashboard tile pipeline now delegates number/table config transforms to the shared common-utils implementations — A unification fix: until 0.6.2, the CLI was re-implementing the number/table config transforms locally, and a small class of operand-hiding behavior was drifting from the web. 0.6.2 routes everything through the shared common-utils package, so a formula tile rendered in the terminal now hides its operand series identically to the web. For teams that consume the dashboard view from hdx chart in CI or from an AI-agent loop, the web/terminal parity is the kind of fix that shows up as fewer "looks different on the web" debugging loops.

No migration is required to upgrade from @hyperdx/cli 0.6.1 to 0.6.2. The CLI is versioned independently of the main hyperdx server package, so the upgrade is npm install -g @hyperdx/[email protected] and the server-side code is unchanged. The dashboard API v2 is additive: existing dashboards without a formulas field read and write back the same JSON shape they did before. If you have scripted dashboard creation from an external system, the validation-on-write is the only behavior change worth knowing about — the same malformed-formula request that the API would have accepted and rendered as a blank tile in 0.6.1 is now rejected with the specific failure reason, which is a strict improvement on the silent-blank-tile class.

What's New in ClickHouse 26.7.4 (stable) versus 26.3.19 (LTS)?

The ClickHouse 26.7.4.58-stable release shipped 2026-08-19, ten days before this refresh. The release page does not carry a per-PR changelog (ClickHouse ships release notes in the canonical changelog rather than the GitHub release body), but the canonical changelog linked above captures the four-month delta between 26.3.19 (LTS, the version this article was authored against on 2026-08-17) and 26.7.4 carries material trace-store-relevant changes. For teams running the self-hosted ClickHouse + HyperDX pair this article walks through, the relevant points are:

  • JSON column type stabilizes for OTel payload storage — Until 26.5, the JSON column type that lets you query nested span attributes without flattening them into a Map(String, String) was still marked experimental; 26.5 promoted it to stable and added subcolumn-discriminator statistics that materially improve the cost of attributes['gen_ai.request.model'] = 'gpt-5'-style queries on a trace table. For teams that mapped attributes as a Map(LowCardinality(String), String) per the schema in the section below, this is the lever that turns the OTel attribute bag into a true queryable surface rather than a per-key search target — switching the column to JSON post-hoc requires an ALTER TABLE ... MODIFY COLUMN with table-locked copy, so for a new deployment, the JSON type is now the right call. The OpenTelemetry gen_ai.* convention guide walks through the attribute shape the JSON column needs to remain queryable.
  • Lightweight deletes for trace TTL — Until 26.7, the only TTL path on a trace table was a full DELETE or a PARTITION drop; both have an obvious write-amplification problem on a multi-billion-row table. 26.7 ships DELETE FROM llm_spans WHERE timestamp < now() - INTERVAL 90 DAY as a metadata-only operation that resolves on read, which is the right shape for a 90-day or 180-day trace retention window. For the cost numbers in this article — the $180/mo on a 1-year reserved r6i.2xlarge — the lightweight-delete path is what keeps the same instance sized appropriately under a multi-month retention floor without manual partition rotation.
  • Parallel MergeTree final for the monthly partition rollup — A small but operationally meaningful change for the monthly rollup query on a partition-by-month table: the OPTIMIZE FINAL step that runs at the end of each month no longer serializes through a single thread, which on a 1B-row monthly partition is the difference between a 4-hour rollup and a 30-minute rollup. Teams running the clickhouse-backed cost dashboards this article describes will see the difference on the first Monday after each month boundary.
  • Async insert backpressure under sustained agent load — 26.6 added backpressure on the async-insert buffer that surfaces as a single TOO_MANY_PARTS error rather than as silent drops under sustained write load. For a HyperDX-fronted ingest that runs the async-insert path on every OTel collector flush, this is the change that makes ingestion backpressure observable on the gateway rather than only on a sudden blank-spans dashboard.
  • 26.3.19 LTS remains the recommended pin for slow-moving production rollouts — ClickHouse ships an LTS tag alongside the stable line. The LTS tag this article was authored against is 26.3.19.3-lts (the canonical v26.3.19.3-lts release tag); for the migration math in this article to stay accurate, pin to v26.3.19.3-lts for production through the remainder of 2026, and stage 26.7.4 on a parallel staging environment to validate the JSON column conversion and the lightweight-delete TTL behavior before promoting. Teams who want to skip the LTS pin and stay on the stable line can pin v26.7.4.58-stable directly.

The schema and the $180/mo versus $4,200/mo math in the body of this article are unchanged against ClickHouse 26.7.4 — the CODEC choices (DoubleDelta, ZSTD(8) on the timestamp column, ZSTD(8) on the rest) and the partition-by-month ORDER BY (service_name, gen_ai.request.model, timestamp) both forward-compat cleanly. The JSON column swap and the lightweight-delete TTL are additive to the schema; the rest of this article still describes the production paths that the new releases leave alone. For the broader storage trade-off the new JSON type opens up, the State of Observability 2026 guide walks through the trace-store-vs-column-store positioning; for the side of this that drives cost, the Datadog migration playbook shows the same arithmetic applied to a real migration.

If you are sizing a self-hosted ClickHouse + HyperDX deployment against the SaaS alternatives, the working numbers from the published case studies and competitive intel are these: an equivalent-trace-volume, equivalent-retention deployment lands at roughly $180/mo on a 1-year reserved r6i.2xlarge for the ClickHouse + HyperDX pair, versus $4,200/mo on Helicone enterprise and $1,800-$2,200/mo on Grafana Cloud Tempo 3.0 advanced tier. The full math and the workload shape those numbers were derived from are below — but the headline is that the cost differential is large enough at production-agent-fleet scale to amortize the engineering time of the migration. This is the article I wish I'd had when I started researching the migration: the schema, the compression trade-offs (ZSTD vs LZ4 vs the Tempo 3.0 WAL path), the OTel gen_ai.* attribute mapping that makes the traces actually queryable in HyperDX's session view, and the workload-shape breakpoints where the SaaS path is actually the right answer.

How much does a self-hosted ClickHouse + HyperDX stack actually cost at production agent-fleet scale? The practical answer is the $180/mo headline, but it lands in three places: the ClickHouse node itself on a 1-year reserved r6i.2xlarge ($80-$100/mo amortized depending on the AWS region), 1 TB gp3 EBS for trace retention ($80/mo), and a t3.medium running the 2-replica HyperDX deployment ($30/mo). Egress is $0 within an AWS region. The SaaS comparison lands at $4,200/mo for Helicone enterprise (full-fidelity tier, 6-month retention, 1,200-agent fleet), $1,800-$2,200/mo for Grafana Cloud Tempo 3.0 advanced tier plus a $400/mo metrics backend, and $1,200-$1,800/mo for Arize Phoenix enterprise at the same trace volume. The break-even point on the engineering time of the migration (roughly 3 engineer-weeks per the published postmortems) is around 800 agents for Helicone and around 1,400 agents for Tempo.

The ClickHouse team's "three villains to agentic observability" post framed the migration pressure more sharply than anything else I've read on the topic: traditional trace stores (Jaeger, Tempo, Elastic APM) lose fidelity on long-context agent runs because they store span attributes as separate keys with per-key compression, and the per-key cardinality explodes when a single trace carries the full conversation history. ClickHouse's column-oriented storage handles the same workload at roughly 1/15th the disk footprint and gives you SQL on the trace payload, which is the lever most "observability" SaaS products don't expose. HyperDX layers a session-aware UI on top of that columnar store and gives you the OpenTelemetry-native search experience that the Helicone / LangSmith generation normalized.

Why full fidelity matters for agent traces

An agent trace is not a single LLM call. It's a tree of LLM calls, tool invocations, retrieval steps, and decision points — often 20 to 100 spans deep for a single user turn, with the prompt and completion payloads of every span attached as span attributes. A typical customer-support-agent trace in a production fleet looks like: 1 user message, 1 routing classifier span (small), 1 retrieval span (medium, embedding model output), 1-3 RAG spans (large — full retrieved chunks), 1 reasoning span (large — full prompt history with tool schemas), 1-4 tool-call spans (medium — full request/response), 1 final response span. Total payload, serialized: 180 KB to 1.2 MB per turn.

Why does agent-trace fidelity break a 6-month audit? The fidelity question is whether the trace store lets you reconstruct that exact payload six months later for an audit or a regression postmortem. Tempo's object-store backend (the default since Tempo 2.0) is lossy by design — it stores span IDs and indexed tags in the block store but defers the full payload to object storage, and the trace query path joins on the indexed tags first. For a 6-month-old trace where the indexed tags are still there but the object storage bucket has been reaped (the common case for cost-controlled retention), you get the metadata but not the payloads. ClickHouse stores the payloads in the same columnar file as the indexed metadata, so a single SQL query reads both back. That is the "full fidelity" claim: not a marketing term, an actual row-store-vs-column-store distinction. For the broader question of what to do with that fidelity when the trace surfaces a real incident, the Agentic Incident Harness guide walks through the 4-layer postmortem framework (state, memory, authority, verification) the fidelity is the load-bearing input for; the LLM incident postmortem guide has the worked retrospective examples.

ClickHouse schema for OTel gen_ai.* spans

The schema I landed on after two weeks of iteration is the OTel gen_ai.* attribute set mapped onto ClickHouse's Map(LowCardinality(String), String) for the dynamic attribute bag, with hot columns (timestamp, trace_id, span_id, model, latency_ms, prompt_tokens, completion_tokens, cost_usd) materialized as first-class columns for the dashboard queries. The ClickHouse native OTel endpoint accepts the standard OTLP payload, so your existing OTel SDKs and collectors wire straight in — no SchemaConverter shim needed.

The actual table DDL, simplified to the columns we use in production:

CREATE TABLE llm_spans (
  timestamp        DateTime64(9) CODEC(DoubleDelta, ZSTD(8)),
  trace_id         String CODEC(ZSTD(8)),
  span_id          String CODEC(ZSTD(8)),
  parent_span_id   String CODEC(ZSTD(8)),
  service_name     LowCardinality(String) CODEC(ZSTD(8)),
  span_name        LowCardinality(String) CODEC(ZSTD(8)),
  gen_ai_system    LowCardinality(String) CODEC(ZSTD(8)),
  gen_ai_request_model LowCardinality(String) CODEC(ZSTD(8)),
  gen_ai_response_model LowCardinality(String) CODEC(ZSTD(8)),
  gen_ai_usage_input_tokens  UInt32 CODEC(T64, ZSTD(8)),
  gen_ai_usage_output_tokens UInt32 CODEC(T64, ZSTD(8)),
  duration_ms      UInt32 CODEC(T64, ZSTD(8)),
  status_code      LowCardinality(String) CODEC(ZSTD(8)),
  cost_usd         Decimal(10, 6) CODEC(ZSTD(8)),
  attributes       Map(LowCardinality(String), String) CODEC(ZSTD(8)),
  events           Array(Tuple(DateTime64(9), String, Map(String, String))) CODEC(ZSTD(8))
) ENGINE = MergeTree
PARTITION BY toYYYYMM(timestamp)
ORDER BY (service_name, gen_ai_request_model, timestamp)
TTL timestamp + INTERVAL 180 DAY;

The two non-obvious decisions: CODEC(T64, ZSTD(8)) on the integer columns gives you ~3x compression on prompt_tokens (most values cluster in the 500-4000 range) without the decompression penalty of Delta, and the attributes column is a Map rather than a JSON blob so WHERE attributes['gen_ai.agent.name'] = 'support-router' is a direct index lookup rather than a full-column scan. The TTL on the partition column is 180 days; tune this for your retention budget — ClickHouse's columnar compression means doubling retention costs roughly 1.4x the disk, not 2x, because the older partitions compress better.

The OTel gen_ai.* conventions for AI inference tracing covers the full attribute schema, including the experimental attributes (gen_ai.agent.name, gen_ai.tool.call.result) that are still in flux. If you wire this table up today, expect to migrate the attributes map keys in 6-12 months as the OTel working group stabilizes the spec.

What does the HyperDX session view actually give you?

HyperDX is a ClickHouse-native observability UI (think Grafana without the Grafana) that ships with built-in support for OTel gen_ai.* attributes. The session view groups spans by trace_id and renders them as a waterfall with prompt/completion payloads inline — the same visual model LangSmith normalized for tracing UIs, but reading directly from your ClickHouse cluster rather than a SaaS backend.

The two features that justify the migration for agent workloads: (1) the session search bar accepts natural-language queries against the attributes map (prompt contains "refund" AND model = "gpt-5"), which is the exact query the incident triage workflow needs and which Tempo's TQL cannot express; (2) the replay view reconstructs the agent loop from the span tree, including tool-call invocations and retrieval results, without needing to re-execute the agent against the original environment. For a multi-tenant agent fleet where an incident on tenant A's trace needs to be reproduced for tenant A's customer support team, the replay-from-trace workflow is the difference between a 30-minute triage and a 3-hour one. For the eBPF-side capture path that feeds into this same ClickHouse cluster on the network side, the eBPF observability guide walks through the kernel-level trace enrichment that the OTel collector joins onto the gen_ai spans at ingest time.

The HyperDX deployment is a single container + a ClickHouse connection. The HyperDX docs walk through the docker-compose shape for self-hosted; the production deployment shape is a 2-replica HyperDX deployment behind a load balancer, pointing at a ClickHouse cluster (single-node is fine through ~5K spans/sec, scale to a 3-shard cluster past that). A reference sizing for a production-agent-fleet workload: an r6i.2xlarge (8 vCPU, 64 GB RAM, 1 TB gp3) ClickHouse node handles a 1,200-agent fleet with 60 days of hot retention and 180 days of cold, with peak ingest at 2.4K spans/sec sustained.

The $180/mo vs $4,200/mo math

The cost comparison is not apples-to-apples — Helicone bundles cost attribution, response caching, and prompt versioning into the same bill, and HyperDX does none of those out of the box. The apples-to-apples comparison is "trace store + 6-month retention at 1,200-agent fleet scale," and the numbers below come from the published competitive intel on file:

  • Self-hosted ClickHouse + HyperDX (this stack): 1x r6i.2xlarge reserved instance ($180/mo amortized on a 1-year reserved), 1 TB gp3 EBS ($80/mo), HyperDX on a t3.medium ($30/mo), 100 GB/month egress ($0). Total: ~$290/mo unsubsidized, ~$180/mo on a 1-year reserved instance.
  • Helicone enterprise: 6-month retention tier for a 1,200-agent fleet with full-fidelity trace storage quotes at $4,200/mo as of 2026-08 (per competitive intel on file). Includes response caching (which is straightforward to re-implement in front of LiteLLM with Redis at $0/mo incremental) and prompt versioning (which the LiteLLM config-file deployment already covers).
  • Tempo + Grafana Cloud (Advanced): Tempo 3.0 with the WAL backend at the same trace volume is approximately $1,800-$2,200/mo on Grafana Cloud's advanced tier, plus $400/mo for the metrics backend. Loses the full-fidelity replay use case that drove the migration.

The migration cost (engineering time) is the real expense. Published migration postmortems put the schema work at roughly 3 engineer-weeks, the HyperDX UI customization at ~1 week, and ongoing maintenance at ~0.5 FTE for retention-policy tuning and the OTel attribute spec migration. For a team without a dedicated platform engineer, the SaaS price starts to look like a feature discount, not a cost premium. For a team with a platform engineer who would otherwise be babysitting a SaaS billing dashboard, the math is unambiguous.

When NOT to migrate

If your trace volume is below ~500 spans/sec and you don't have a 6-month audit-retention requirement, the SaaS path is the right one. Helicone and LangSmith both ship the OTel gen_ai.* schema natively, the HyperDX UI is great but not so great that it justifies a 3-engineer-week migration at low scale, and the cost differential is too small to amortize the engineering time. ClickHouse + HyperDX is a workload-specific answer — it's the right answer when you have audit-retention requirements and trace volume that exceeds the SaaS tier's cost-effectiveness break point, which on Helicone's 2026 pricing is somewhere around 800 agents.

The second reason to stay on SaaS: the OTel gen_ai.* spec is still moving. If you self-host the storage, you own the migration when the spec changes; if you stay on SaaS, the vendor eats that migration. The spec has had two breaking changes in the last 6 months (gen_ai.choice → gen_ai.response.model, and the gen_ai.tool.* namespace expansion in May 2026), and the SaaS vendors handled both transparently.

Where this stacks against the alternatives

The Tempo 3.0 path (the Grafana Tempo 3.0 release shipped 2026-06-01 with WAL-backed trace storage and the new trace-by-id search) is the right answer if you are already on the Grafana stack and your retention requirement is shorter than 90 days. Tempo's object-storage backend is dramatically cheaper per GB than ClickHouse's local-disk backend, and the WAL path closes the durability gap that made Tempo 1.x unsuitable for production. The trade-off is exactly the full-fidelity claim: Tempo's index is separate from the trace payloads, and a 6-month-old trace with an object storage bucket that has been reaped is metadata-only.

Arize Phoenix 20.x is the right answer if your primary use case is evaluation rather than observability — Phoenix's strength is the eval framework and the embedding-space drift detection, not the trace store. Phoenix can be backed by ClickHouse (the 20.x line supports it as a primary store), in which case you're running the Phoenix UI on top of the ClickHouse + HyperDX schema above, with Phoenix's eval taxonomy layered on top. That's the deployment shape I'd recommend for teams that already have Phoenix in their stack and are considering the HyperDX addition.

SigNoz is the closest single-vendor competitor to the ClickHouse + HyperDX combination — SigNoz is also ClickHouse-native, also OTel-first, and also self-hostable. The differentiation is in the UI maturity (HyperDX's session view is significantly more polished for trace replay) and in the schema flexibility (HyperDX is more permissive on custom attributes, which matters for the experimental gen_ai.* attributes). For a clean-slate deployment, run both against the same ClickHouse cluster and pick the UI your team prefers; for a fleet already on SigNoz, the HyperDX addition is incremental rather than replacement.

The deployment shape I'd recommend today

Single-node ClickHouse on an r6i.2xlarge with 1 TB gp3, 2-replica HyperDX behind an ALB, OTel Collector with the otelclickhouseexporter writing to the cluster, and the LiteLLM or vLLM OTel exporter pointing at the collector. The schema is the one above. The retention is 60 days hot, 180 days cold. The cost is ~$290/mo unsubsidized, ~$180/mo on a 1-year reserved instance. The trace volume scales linearly to ~5K spans/sec on the single-node cluster; past that, add a 3-shard ClickHouse cluster and a second HyperDX replica for read scaling.

The migration playbook: instrument with the OTel collector first (no ClickHouse yet), point at the SaaS backend to validate the schema, then swap the exporter to ClickHouse and validate the same queries work, then bring up HyperDX on the same cluster and validate the session view. Each step is independently revertible, which matters because the ClickHouse + HyperDX combination has fewer battle scars than Helicone does at this point in 2026. The ClickHouse team's "three villains" framing is recent (2026-07), HyperDX 2.x is the first version that's production-ready for full-fidelity agent traces, and the OTel gen_ai.* spec is still consolidating the agent attributes. For teams that can tolerate the bleeding edge, the cost and fidelity wins are real.

For the cost-arithmetic side of the migration decision, the LLM API cost calculator turns the per-token cost of the 1,200-agent fleet into the spend baseline the migration has to beat. For the broader eval-cost reconciliation work that the bundled price tables don't enable, the LLM cost monitoring tools 2026 roundup walks through the rollup patterns the migration unlocks. The agent observability at 1,200+ agents OTel stack covers the OTel collector and exporter layer above the storage, and the state of observability in 2026 frames the broader semantic-observability trend the ClickHouse + HyperDX combination is the storage layer of.

Advertisement
Advertisement