What's New in @hyperdx/cli 0.6.2 (chart formulas on the terminal + dashboard API)?
@hyperdx/cli 0.6.2 shipped 2026-08-21 as a single-patch release that lands one substantive feature: chart formulas across every API surface that persists or accepts chart configs, plus the MCP save_dashboard / patch_dashboard tools. The change is small in line count but operationally meaningful for anyone running a self-hosted ClickHouse + HyperDX stack and consuming the dashboard data from an AI agent or terminal pipeline. From the 0.6.2 changelog:
- Chart formulas round-trip through every API surface (
dc29d57f) — The external dashboards API v2 and the MCPsave_dashboard/patch_dashboardtools now acceptformulas(letter-ref arithmetic over the tile's select items — e.g.A / (A + B) * 100for a conversion rate) andshowOperandSerieson line, stacked bar, table, and number builder tiles, and they round-trip them through GET / PUT. The expressions are validated on write: unknown series refs, malformed syntax, combining formulas withasRatio, multiple formulas on a number tile, and formulas on formula-incapable source kinds (anything other than metric, log, or trace) are all rejected with actionable errors. For teams that script the dashboard config from an external system — the same pattern the LLM monitoring dashboard templates guide documents — this is the API surface that lets you build a derived series without re-uploading the tile as a metric query. - MCP
query_tilecomputes formula columns for both metric and log/trace event tiles — The MCP tool that the agent-side observability pattern relies on now returns the formula-computed columns alongside the raw series. For teams running the 1,200+ agents OTel stack pattern, this is the change that lets an agent build a derived metric (cost-per-request, cache-hit-rate, p99/p50 ratio) on a single MCP call rather than computing it on the client. - Query-guide prompt documents formulas, OpenAPI spec adds the
Formulaschema — The two discoverability surfaces that make a new feature actually used in production. The MCPquery-guideprompt now includes a worked example forA / (A + B) * 100, and the OpenAPI spec published at the same v2 endpoint listsFormulaas a first-class schema so client generators pick it up. The State of Observability 2026 guide covers the broader trace-store-vs-column-store positioning that the formula-on-the-query side opens up. - CLI dashboard tile pipeline now delegates number/table config transforms to the shared
common-utilsimplementations — A unification fix: until 0.6.2, the CLI was re-implementing the number/table config transforms locally, and a small class of operand-hiding behavior was drifting from the web. 0.6.2 routes everything through the sharedcommon-utilspackage, so a formula tile rendered in the terminal now hides its operand series identically to the web. For teams that consume the dashboard view fromhdx chartin CI or from an AI-agent loop, the web/terminal parity is the kind of fix that shows up as fewer "looks different on the web" debugging loops.
No migration is required to upgrade from @hyperdx/cli 0.6.1 to 0.6.2. The CLI is versioned independently of the main hyperdx server package, so the upgrade is npm install -g @hyperdx/[email protected] and the server-side code is unchanged. The dashboard API v2 is additive: existing dashboards without a formulas field read and write back the same JSON shape they did before. If you have scripted dashboard creation from an external system, the validation-on-write is the only behavior change worth knowing about — the same malformed-formula request that the API would have accepted and rendered as a blank tile in 0.6.1 is now rejected with the specific failure reason, which is a strict improvement on the silent-blank-tile class.
What's New in ClickHouse 26.7.4 (stable) versus 26.3.19 (LTS)?
The ClickHouse 26.7.4.58-stable release shipped 2026-08-19, ten days before this refresh. The release page does not carry a per-PR changelog (ClickHouse ships release notes in the canonical changelog rather than the GitHub release body), but the canonical changelog linked above captures the four-month delta between 26.3.19 (LTS, the version this article was authored against on 2026-08-17) and 26.7.4 carries material trace-store-relevant changes. For teams running the self-hosted ClickHouse + HyperDX pair this article walks through, the relevant points are:
- JSON column type stabilizes for OTel payload storage — Until 26.5, the
JSONcolumn type that lets you query nested span attributes without flattening them into aMap(String, String)was still marked experimental; 26.5 promoted it to stable and added subcolumn-discriminator statistics that materially improve the cost ofattributes['gen_ai.request.model'] = 'gpt-5'-style queries on a trace table. For teams that mappedattributesas aMap(LowCardinality(String), String)per the schema in the section below, this is the lever that turns the OTel attribute bag into a true queryable surface rather than a per-key search target — switching the column toJSONpost-hoc requires anALTER TABLE ... MODIFY COLUMNwith table-locked copy, so for a new deployment, the JSON type is now the right call. The OpenTelemetry gen_ai.* convention guide walks through the attribute shape the JSON column needs to remain queryable. - Lightweight deletes for trace TTL — Until 26.7, the only TTL path on a trace table was a full
DELETEor aPARTITIONdrop; both have an obvious write-amplification problem on a multi-billion-row table. 26.7 shipsDELETE FROM llm_spans WHERE timestamp < now() - INTERVAL 90 DAYas a metadata-only operation that resolves on read, which is the right shape for a 90-day or 180-day trace retention window. For the cost numbers in this article — the $180/mo on a 1-year reservedr6i.2xlarge— the lightweight-delete path is what keeps the same instance sized appropriately under a multi-month retention floor without manual partition rotation. - Parallel
MergeTreefinal for the monthly partition rollup — A small but operationally meaningful change for the monthly rollup query on a partition-by-month table: theOPTIMIZE FINALstep that runs at the end of each month no longer serializes through a single thread, which on a 1B-row monthly partition is the difference between a 4-hour rollup and a 30-minute rollup. Teams running the clickhouse-backed cost dashboards this article describes will see the difference on the first Monday after each month boundary. - Async insert backpressure under sustained agent load — 26.6 added backpressure on the async-insert buffer that surfaces as a single
TOO_MANY_PARTSerror rather than as silent drops under sustained write load. For a HyperDX-fronted ingest that runs the async-insert path on every OTel collector flush, this is the change that makes ingestion backpressure observable on the gateway rather than only on a sudden blank-spans dashboard. - 26.3.19 LTS remains the recommended pin for slow-moving production rollouts — ClickHouse ships an LTS tag alongside the stable line. The LTS tag this article was authored against is 26.3.19.3-lts (the canonical
v26.3.19.3-ltsrelease tag); for the migration math in this article to stay accurate, pin tov26.3.19.3-ltsfor production through the remainder of 2026, and stage 26.7.4 on a parallel staging environment to validate the JSON column conversion and the lightweight-delete TTL behavior before promoting. Teams who want to skip the LTS pin and stay on the stable line can pinv26.7.4.58-stabledirectly.
The schema and the $180/mo versus $4,200/mo math in the body of this article are unchanged against ClickHouse 26.7.4 — the CODEC choices (DoubleDelta, ZSTD(8) on the timestamp column, ZSTD(8) on the rest) and the partition-by-month ORDER BY (service_name, gen_ai.request.model, timestamp) both forward-compat cleanly. The JSON column swap and the lightweight-delete TTL are additive to the schema; the rest of this article still describes the production paths that the new releases leave alone. For the broader storage trade-off the new JSON type opens up, the State of Observability 2026 guide walks through the trace-store-vs-column-store positioning; for the side of this that drives cost, the Datadog migration playbook shows the same arithmetic applied to a real migration.
If you are sizing a self-hosted ClickHouse + HyperDX deployment against the SaaS alternatives, the working numbers from the published case studies and competitive intel are these: an equivalent-trace-volume, equivalent-retention deployment lands at roughly $180/mo on a 1-year reserved r6i.2xlarge for the ClickHouse + HyperDX pair, versus $4,200/mo on Helicone enterprise and $1,800-$2,200/mo on Grafana Cloud Tempo 3.0 advanced tier. The full math and the workload shape those numbers were derived from are below — but the headline is that the cost differential is large enough at production-agent-fleet scale to amortize the engineering time of the migration. This is the article I wish I'd had when I started researching the migration: the schema, the compression trade-offs (ZSTD vs LZ4 vs the Tempo 3.0 WAL path), the OTel gen_ai.* attribute mapping that makes the traces actually queryable in HyperDX's session view, and the workload-shape breakpoints where the SaaS path is actually the right answer.
The ClickHouse team's "three villains to agentic observability" post framed the migration pressure more sharply than anything else I've read on the topic: traditional trace stores (Jaeger, Tempo, Elastic APM) lose fidelity on long-context agent runs because they store span attributes as separate keys with per-key compression, and the per-key cardinality explodes when a single trace carries the full conversation history. ClickHouse's column-oriented storage handles the same workload at roughly 1/15th the disk footprint and gives you SQL on the trace payload, which is the lever most "observability" SaaS products don't expose. HyperDX layers a session-aware UI on top of that columnar store and gives you the OpenTelemetry-native search experience that the Helicone / LangSmith generation normalized.
Why full fidelity matters for agent traces
An agent trace is not a single LLM call. It's a tree of LLM calls, tool invocations, retrieval steps, and decision points — often 20 to 100 spans deep for a single user turn, with the prompt and completion payloads of every span attached as span attributes. A typical customer-support-agent trace in a production fleet looks like: 1 user message, 1 routing classifier span (small), 1 retrieval span (medium, embedding model output), 1-3 RAG spans (large — full retrieved chunks), 1 reasoning span (large — full prompt history with tool schemas), 1-4 tool-call spans (medium — full request/response), 1 final response span. Total payload, serialized: 180 KB to 1.2 MB per turn.
The "fidelity" question is whether the trace store lets you reconstruct that exact payload six months later for an audit or a regression postmortem. Tempo's object-store backend (the default since Tempo 2.0) is lossy by design — it stores span IDs and indexed tags in the block store but defers the full payload to object storage, and the trace query path joins on the indexed tags first. For a 6-month-old trace where the indexed tags are still there but the object storage bucket has been reaped (the common case for cost-controlled retention), you get the metadata but not the payloads. ClickHouse stores the payloads in the same columnar file as the indexed metadata, so a single SQL query reads both back. That is the "full fidelity" claim: not a marketing term, an actual row-store-vs-column-store distinction.
ClickHouse schema for OTel gen_ai.* spans
The schema I landed on after two weeks of iteration is the OTel gen_ai.* attribute set mapped onto ClickHouse's Map(LowCardinality(String), String) for the dynamic attribute bag, with hot columns (timestamp, trace_id, span_id, model, latency_ms, prompt_tokens, completion_tokens, cost_usd) materialized as first-class columns for the dashboard queries. The ClickHouse native OTel endpoint accepts the standard OTLP payload, so your existing OTel SDKs and collectors wire straight in — no SchemaConverter shim needed.
The actual table DDL, simplified to the columns we use in production:
CREATE TABLE llm_spans (
timestamp DateTime64(9) CODEC(DoubleDelta, ZSTD(8)),
trace_id String CODEC(ZSTD(8)),
span_id String CODEC(ZSTD(8)),
parent_span_id String CODEC(ZSTD(8)),
service_name LowCardinality(String) CODEC(ZSTD(8)),
span_name LowCardinality(String) CODEC(ZSTD(8)),
gen_ai_system LowCardinality(String) CODEC(ZSTD(8)),
gen_ai_request_model LowCardinality(String) CODEC(ZSTD(8)),
gen_ai_response_model LowCardinality(String) CODEC(ZSTD(8)),
gen_ai_usage_input_tokens UInt32 CODEC(T64, ZSTD(8)),
gen_ai_usage_output_tokens UInt32 CODEC(T64, ZSTD(8)),
duration_ms UInt32 CODEC(T64, ZSTD(8)),
status_code LowCardinality(String) CODEC(ZSTD(8)),
cost_usd Decimal(10, 6) CODEC(ZSTD(8)),
attributes Map(LowCardinality(String), String) CODEC(ZSTD(8)),
events Array(Tuple(DateTime64(9), String, Map(String, String))) CODEC(ZSTD(8))
) ENGINE = MergeTree
PARTITION BY toYYYYMM(timestamp)
ORDER BY (service_name, gen_ai_request_model, timestamp)
TTL timestamp + INTERVAL 180 DAY;
The two non-obvious decisions: CODEC(T64, ZSTD(8)) on the integer columns gives you ~3x compression on prompt_tokens (most values cluster in the 500-4000 range) without the decompression penalty of Delta, and the attributes column is a Map rather than a JSON blob so WHERE attributes['gen_ai.agent.name'] = 'support-router' is a direct index lookup rather than a full-column scan. The TTL on the partition column is 180 days; tune this for your retention budget — ClickHouse's columnar compression means doubling retention costs roughly 1.4x the disk, not 2x, because the older partitions compress better.
The OTel gen_ai.* conventions for AI inference tracing covers the full attribute schema, including the experimental attributes (gen_ai.agent.name, gen_ai.tool.call.result) that are still in flux. If you wire this table up today, expect to migrate the attributes map keys in 6-12 months as the OTel working group stabilizes the spec.
What does the HyperDX session view actually give you?
HyperDX is a ClickHouse-native observability UI (think Grafana without the Grafana) that ships with built-in support for OTel gen_ai.* attributes. The session view groups spans by trace_id and renders them as a waterfall with prompt/completion payloads inline — the same visual model LangSmith normalized for tracing UIs, but reading directly from your ClickHouse cluster rather than a SaaS backend.
The two features that justify the migration for agent workloads: (1) the session search bar accepts natural-language queries against the attributes map (prompt contains "refund" AND model = "gpt-5"), which is the exact query the incident triage workflow needs and which Tempo's TQL cannot express; (2) the replay view reconstructs the agent loop from the span tree, including tool-call invocations and retrieval results, without needing to re-execute the agent against the original environment. For a multi-tenant agent fleet where an incident on tenant A's trace needs to be reproduced for tenant A's customer support team, the replay-from-trace workflow is the difference between a 30-minute triage and a 3-hour one. For the eBPF-side capture path that feeds into this same ClickHouse cluster on the network side, the eBPF observability guide walks through the kernel-level trace enrichment that the OTel collector joins onto the gen_ai spans at ingest time.
The HyperDX deployment is a single container + a ClickHouse connection. The HyperDX docs walk through the docker-compose shape for self-hosted; the production deployment shape is a 2-replica HyperDX deployment behind a load balancer, pointing at a ClickHouse cluster (single-node is fine through ~5K spans/sec, scale to a 3-shard cluster past that). A reference sizing for a production-agent-fleet workload: an r6i.2xlarge (8 vCPU, 64 GB RAM, 1 TB gp3) ClickHouse node handles a 1,200-agent fleet with 60 days of hot retention and 180 days of cold, with peak ingest at 2.4K spans/sec sustained.
The $180/mo vs $4,200/mo math
The cost comparison is not apples-to-apples — Helicone bundles cost attribution, response caching, and prompt versioning into the same bill, and HyperDX does none of those out of the box. The apples-to-apples comparison is "trace store + 6-month retention at 1,200-agent fleet scale," and the numbers below come from the published competitive intel on file:
- Self-hosted ClickHouse + HyperDX (this stack): 1x r6i.2xlarge reserved instance ($180/mo amortized on a 1-year reserved), 1 TB gp3 EBS ($80/mo), HyperDX on a t3.medium ($30/mo), 100 GB/month egress ($0). Total: ~$290/mo unsubsidized, ~$180/mo on a 1-year reserved instance.
- Helicone enterprise: 6-month retention tier for a 1,200-agent fleet with full-fidelity trace storage quotes at $4,200/mo as of 2026-08 (per competitive intel on file). Includes response caching (which is straightforward to re-implement in front of LiteLLM with Redis at $0/mo incremental) and prompt versioning (which the LiteLLM config-file deployment already covers).
- Tempo + Grafana Cloud (Advanced): Tempo 3.0 with the WAL backend at the same trace volume is approximately $1,800-$2,200/mo on Grafana Cloud's advanced tier, plus $400/mo for the metrics backend. Loses the full-fidelity replay use case that drove the migration.
The migration cost (engineering time) is the real expense. Published migration postmortems put the schema work at roughly 3 engineer-weeks, the HyperDX UI customization at ~1 week, and ongoing maintenance at ~0.5 FTE for retention-policy tuning and the OTel attribute spec migration. For a team without a dedicated platform engineer, the SaaS price starts to look like a feature discount, not a cost premium. For a team with a platform engineer who would otherwise be babysitting a SaaS billing dashboard, the math is unambiguous.
When NOT to migrate
If your trace volume is below ~500 spans/sec and you don't have a 6-month audit-retention requirement, the SaaS path is the right one. Helicone and LangSmith both ship the OTel gen_ai.* schema natively, the HyperDX UI is great but not so great that it justifies a 3-engineer-week migration at low scale, and the cost differential is too small to amortize the engineering time. ClickHouse + HyperDX is a workload-specific answer — it's the right answer when you have audit-retention requirements and trace volume that exceeds the SaaS tier's cost-effectiveness break point, which on Helicone's 2026 pricing is somewhere around 800 agents.
The second reason to stay on SaaS: the OTel gen_ai.* spec is still moving. If you self-host the storage, you own the migration when the spec changes; if you stay on SaaS, the vendor eats that migration. The spec has had two breaking changes in the last 6 months (gen_ai.choice → gen_ai.response.model, and the gen_ai.tool.* namespace expansion in May 2026), and the SaaS vendors handled both transparently.
Where this stacks against the alternatives
The Tempo 3.0 path (the Grafana Tempo 3.0 release shipped 2026-06-01 with WAL-backed trace storage and the new trace-by-id search) is the right answer if you are already on the Grafana stack and your retention requirement is shorter than 90 days. Tempo's object-storage backend is dramatically cheaper per GB than ClickHouse's local-disk backend, and the WAL path closes the durability gap that made Tempo 1.x unsuitable for production. The trade-off is exactly the full-fidelity claim: Tempo's index is separate from the trace payloads, and a 6-month-old trace with an object storage bucket that has been reaped is metadata-only.
Arize Phoenix 20.x is the right answer if your primary use case is evaluation rather than observability — Phoenix's strength is the eval framework and the embedding-space drift detection, not the trace store. Phoenix can be backed by ClickHouse (the 20.x line supports it as a primary store), in which case you're running the Phoenix UI on top of the ClickHouse + HyperDX schema above, with Phoenix's eval taxonomy layered on top. That's the deployment shape I'd recommend for teams that already have Phoenix in their stack and are considering the HyperDX addition.
SigNoz is the closest single-vendor competitor to the ClickHouse + HyperDX combination — SigNoz is also ClickHouse-native, also OTel-first, and also self-hostable. The differentiation is in the UI maturity (HyperDX's session view is significantly more polished for trace replay) and in the schema flexibility (HyperDX is more permissive on custom attributes, which matters for the experimental gen_ai.* attributes). For a clean-slate deployment, run both against the same ClickHouse cluster and pick the UI your team prefers; for a fleet already on SigNoz, the HyperDX addition is incremental rather than replacement.
The deployment shape I'd recommend today
Single-node ClickHouse on an r6i.2xlarge with 1 TB gp3, 2-replica HyperDX behind an ALB, OTel Collector with the otelclickhouseexporter writing to the cluster, and the LiteLLM or vLLM OTel exporter pointing at the collector. The schema is the one above. The retention is 60 days hot, 180 days cold. The cost is ~$290/mo unsubsidized, ~$180/mo on a 1-year reserved instance. The trace volume scales linearly to ~5K spans/sec on the single-node cluster; past that, add a 3-shard ClickHouse cluster and a second HyperDX replica for read scaling.
The migration playbook: instrument with the OTel collector first (no ClickHouse yet), point at the SaaS backend to validate the schema, then swap the exporter to ClickHouse and validate the same queries work, then bring up HyperDX on the same cluster and validate the session view. Each step is independently revertible, which matters because the ClickHouse + HyperDX combination has fewer battle scars than Helicone does at this point in 2026. The ClickHouse team's "three villains" framing is recent (2026-07), HyperDX 2.x is the first version that's production-ready for full-fidelity agent traces, and the OTel gen_ai.* spec is still consolidating the agent attributes. For teams that can tolerate the bleeding edge, the cost and fidelity wins are real.
For the cost-arithmetic side of the migration decision, the LLM API cost calculator turns the per-token cost of the 1,200-agent fleet into the spend baseline the migration has to beat. For the broader eval-cost reconciliation work that the bundled price tables don't enable, the LLM cost monitoring tools 2026 roundup walks through the rollup patterns the migration unlocks. The agent observability at 1,200+ agents OTel stack covers the OTel collector and exporter layer above the storage, and the state of observability in 2026 frames the broader semantic-observability trend the ClickHouse + HyperDX combination is the storage layer of.