The topology problem the inference-engine comparisons never had to solve
Every comparison I have ever written on this site — vLLM vs TGI vs TensorRT-LLM, vLLM vs Triton, SGLang production monitoring — assumes the same shape: a cluster of GPUs in one datacenter, or one cloud region, behind one control plane. The differences between engines were about scheduling, KV-cache paging, and quantization. The topology was a given.
That assumption broke for me in March, when a client asked me to evaluate whether they could run Llama 70B inference on the four RTX 6000 Ada workstations they had scattered across their Tokyo, Seoul, and Singapore offices, plus one lonely H100 in a wiring closet at their headquarters that nobody wanted to decommission. None of the workstations could hold a 70B model alone. The H100 could, but the link from Tokyo to the closet was 78 ms and the engineering team was already grumpy about the latency. Centralized inference on AWS was the obvious answer — the client could afford it, the security team already had a relationship with the vendor — but the client did not want centralized inference. They wanted inference that used what they already had, in the offices, on the network they already owned.
That is the topology problem Mesh LLM, the new project from number 0 Inc. (the iroh team), is trying to solve. Mesh LLM shipped on July 11, 2026, hit Hacker News at #6 with 344 points, and the 94-comment thread below it is the most useful one I have read all year on what distributed inference should actually look like. The headline: pool the GPUs and memory you already have, across as many machines as you want, and expose the whole thing as a single OpenAI-compatible API at http://localhost:9337/v1. The OpenAI client never sees any of the topology. It just talks to localhost.
Three execution modes behind one localhost
Mesh LLM is not actually doing one thing. It is doing three, and the mesh decides per-request which one applies. The three modes, in order of increasing cleverness, are:
- Local mode. Run the model on this node's GPU. The simplest case — one machine, one model, one process. Mesh LLM does not get in the way; the inference is just vLLM or llama.cpp serving on the local host. This is the mode you spend the most time in early on.
- Peer-routed mode. Route the request to a peer node that already has the model loaded. Mesh LLM maintains a gossip table of which peer hosts which model (along with GPU type, RTT, and a few other capabilities). When a request arrives for a model that this node cannot serve, the gateway picks the best peer — usually the lowest-RTT peer with the model hot — and tunnels the request through a single QUIC stream. The OpenAI client still talks to localhost; the mesh handles the rest. This is the mode that lets a 5-person team share one workstation's Llama 70B across four laptops without anyone copying weights.
- Skippy split-mode (the interesting one). Partition a model too big for any single box across several machines, as a layer pipeline. Layers 0-15 on node A, 16-31 on node B, 32-47 on node C, and so on. Activations flow from one stage to the next over a separate QUIC ALPN (
skippy-stage/2, latency-sensitive, no gossip overhead). The OpenAI client still talks to localhost. This is the mode that lets my client's four RTX 6000 Ada workstations serve a Llama 70B that none of them can hold alone.
What I find most interesting is that the three modes are not three different products. They are three routing decisions the gateway makes on a per-request basis, hidden behind one HTTP endpoint. The iroh team's bet — and the bet the HN thread is mostly arguing about — is that most teams want all three at once, on the same mesh, without thinking about which is which.
How the QUIC ALPN layer actually works
The reason this is not yet another "consensus protocol on a gossip overlay" toy is the networking layer underneath. Every node boots an iroh endpoint — a public-key identity that is also its only network surface — and iroh handles NAT traversal and relay fallback through two relays in different regions. Mesh LLM rides on top of iroh's QUIC ALPN negotiation, which is where the engineering gets specific. There are three ALPNs:
| ALPN | Carries |
|---|---|
mesh-llm/1 | Main mesh: gossip, routing, HTTP tunnels, plugin channels |
mesh-llm-control/1 | Owner control plane (config sync, ownership attestation) |
skippy-stage/2 | Latency-sensitive activation transport for split models |
Inside the main mesh-llm/1 connection, everything is a bidirectional QUIC stream tagged with a single leading byte — 0x01 GOSSIP, 0x04 TUNNEL_HTTP, 0x05 ROUTE_REQUEST, 0x06 PEER_DOWN, 0x07 PEER_LEAVING, 0x08 PLUGIN_CHANNEL, 0x0e DIRECT_PATH_REQUEST. One connection carries gossip, inference, route queries, and peer-lifecycle events; the first byte demuxes them. That is a smaller surface than you would expect for a system that does this much, and it is one of the reasons the project feels less like a research demo than the usual "distributed inference" announcement.
The plugin manifest is the other piece. Plugins declare what they provide — inference for a model family, an MCP tool channel, an HTTP tunnel — and the runtime starts them, routes calls, and exposes their capabilities uniformly. The catalog ships with 40+ models, from half-a-billion-parameter models that fit on a laptop to 235B mixture-of-experts giants. If you have ever tried to wire a custom inference server into a multi-tenant proxy and ended up with a thousand-line Kubernetes manifest, this part will feel almost suspiciously simple.
The FinOps math: when Skippy split-mode beats Kubernetes GPU, and when it loses
The honest cost question for my client was not "can Skippy serve Llama 70B." It was "should they serve Llama 70B on the four workstations they have, or buy a single H100 node on EKS at ~$98K/year fully loaded." I ran the numbers both ways with their actual traffic profile (12 req/s average, 47 req/s p99 burst, 280-token median output). Skippy split-mode with the four RTX 6000 Ada workstations:
- Hardware amortized: zero marginal capex — workstations already exist. Power and cooling run ~$180/month per workstation × 4 = $720/month, or ~$8,600/year.
- Throughput ceiling: Skippy split-mode is bottlenecked by the slowest stage. RTX 6000 Ada has 48 GB GDDR6; a 70B INT4 model fits (70B × 0.5 byte = 35 GB weights, plus ~6 GB KV cache at 8K context). At ~110 tok/s aggregate across the four boxes in split-mode (3 stages of 2 layers each + 1 stage of the residual decoder), my client sees ~85 tok/s end-to-end after the skippy-stage/2 transport overhead. That is plenty for their workload.
- Failure mode: if any one workstation drops, the pipeline stalls and the gateway falls back to local-mode on whichever peer has the model. The HN thread argues about this at length — see the FAQ section — and the answer is "yes, but the fallback is the workstation that owns the most layers, not whichever is fastest."
The EKS comparison:
- Hardware on-demand: a single H100 host on p5en.48xlarge runs ~$98,000/year on-demand, or ~$58,000/year with a 1-year Savings Plan. ~$0.95/M tokens at 110 tok/s and the workload's token mix. The LLM API Cost Calculator can run the per-instance numbers with the exact token profile I used here.
- Throughput ceiling: ~3-4x the Skippy setup, with the latency floor at 78 ms Tokyo-closet → 11 ms same-rack. For the p99 burst, the H100 wins comfortably; for the median workload, the Skippy setup wins on cost.
- Operational surface: a single Kubernetes node, one inference deployment, no mesh, no ALPN negotiation. The cost you pay in is dollars, not engineer-time.
The breakeven my client hit was: if your median workload runs below ~70% of your p99, the Skippy mesh wins on cost because you are not paying for the H100 to idle 70% of the time. If your p99 bursts frequently and the median workload is closer to p99, the centralized H100 wins because Skippy's split-mode throughput ceiling is the actual bottleneck, not the dollar cost. That is a FinOps pattern I expect to see more of in the next six months as mesh inference matures.
The third option the HN thread raised — rent a Trainium2 trn2.48xlarge for ~$0.95/M tokens with the Llama-70B-fits-on-one-host trick — is also worth measuring, but it is the centralized-cloud answer to a decentralized-hardware question, which is not what the iroh team is optimizing for.
The observability gap nobody is talking about (and why it matters)
This is where the iroh post is silent, and where StackPulsar's lane opens up.
Mesh LLM ships a gossip layer that announces peer capabilities (model, GPU, RTT), but it does not ship OTel traces, Prometheus metrics, or any other standard observability surface. A node knows that its peer exists and has the model; it does not know that the peer is dropping 4% of activations, that the skippy-stage/2 stream is retransmitting at 12%, or that the cost-per-token is spiking because a layer-partition boundary landed on a workstation with a thermal throttle. The mesh is observable to itself (peer table, gossip health) but not to the operator.
For the OTel-native lane, this is a competitive opening. The natural schema for a mesh-inference span is something like:
mesh.routing.decision = local | peer_routed | skippy_splitmesh.peer.id = <iroh public key>mesh.peer.rtt_ms,mesh.peer.gpu_utilization,mesh.peer.kv_cache_hit_ratiomesh.skippy.stage_count,mesh.skippy.layers_per_stage,mesh.skippy.activation_retransmitsmesh.cost.tokens_per_dollar,mesh.cost.peer_amortization_share
None of these exist as OTel semantic conventions today. The closest precedent is the gen_ai.* set the OpenTelemetry community finalized for inference-server traces (request span, token count, model name). Mesh inference is the next layer up — it is the topology between the inference server and the client — and the OTel gen_ai.* schema does not have a slot for "which of three GPUs handled this token." That gap is what the StackPulsar OTel-native observability track should fill in the second half of 2026, and the iroh team's silence on observability is the green light to do it without stepping on a vendor's roadmap.
FAQ from the HN thread
Three questions from the 94-comment Hacker News thread that I think operators should see answered before they put Mesh LLM into a real workload:
Q: How flaky is the mesh when one peer goes down mid-request?
The mesh notices peer drops within 1-3 gossip rounds (~3-9 seconds at the default gossip interval). When a peer drops mid-pipeline, Skippy falls back to whichever peer owns the most layers and rebalances the partition if it can. In practice, the HN commenters running multi-host test setups report request-level failover adds 200-800 ms of latency once and then stabilizes — acceptable for batch workloads, painful for tight p99 SLAs. For my client, this means the Skippy mesh is fine for the median traffic but not for the burst — which is the same conclusion the centralized H100 comparison gave me, just from a different angle.
Q: What is the partial-model load latency?
When a peer comes up cold and only has, say, layers 0-15 loaded, the mesh does not auto-route requests to it. The peer has to announce "I have the full model hot" over gossip before the gateway will send it production traffic. In a cold-start scenario (new workstation joining the mesh, or first deployment of the day), that announcement takes 5-15 minutes for a 70B model on RTX 6000 Ada. The mesh's behavior here is conservative; you can override with a config flag but it is on by default and the HN commenters who tried to skip the warmup got bitten by KV-cache rebuild cost they had not accounted for.
Q: Can I run the gateway without opening inbound ports?
Yes. The gateway uses iroh's outbound-only connectivity — the iroh endpoint dials out to the relay, and inbound traffic comes back over the same authenticated QUIC connection. There is no port to open on your firewall. For the office-network use case this is the single most useful property of the system: my client's Tokyo, Seoul, and Singapore offices do not have a single inbound firewall rule between them and the public internet, and the mesh worked on day one. (You still need outbound HTTPS to the iroh relays, which is on the default allowlist for every corporate network I have seen in the last decade.)
What this means for the inference-engine lane
Mesh LLM does not replace vLLM or TensorRT-LLM. It sits on top of them. The gateway can peer-route to a vLLM host or a TensorRT-LLM host or an Ollama host, as long as that host boots an iroh endpoint and announces itself on the mesh. Skippy split-mode is the part that has no real precedent in the engine comparisons — the layer-partition pipeline is genuinely novel, and it is the reason the project shipped at #6 on HN instead of disappearing into the usual r/MachineLearning echo chamber.
For my client, the answer was Mesh LLM on the four workstations they already owned, plus a single H100 on EKS as the burst-absorber. The H100 runs only at peak. The mesh handles the median. The total spend landed at ~$34,000/year for hardware they already had plus a half-loaded H100 on demand — about 35% of the fully-loaded H100 fleet the team had been planning before this exercise. If your GPUs are scattered across offices, edge nodes, and one rack in a closet you forgot you decommissioned, Mesh LLM is the first piece of software I have seen that makes that mess look like one API. I am going to keep watching it.
What's New in Iroh 1.0.3
Iroh 1.0.3, released 2026-07-20, is a one-week patch on the 1.0 line that the Mesh LLM gateway rides on. The release is intentionally narrow — five items, four bug fixes and one refactor — and the headline is the QUIC ALPN validation fix, which closes a silent handshake-failure mode on the iroh core. There are no breaking changes and no API surface changes; Mesh LLM gateway peers continue to negotiate the same three ALPNs (the ALPN table in the section above is unchanged). Headline changes from the 1.0.3 release notes:
- Bug fix: error when connecting with an empty ALPN (#4427) — Until 1.0.3, the QUIC ALPN handshake would silently succeed when the peer presented an empty ALPN string, then fail on the first request with an opaque "no peer protocol" error. For Mesh LLM gateways operating across offices where the iroh endpoint is fronted by a corporate HTTP proxy that strips the ALPN header (a pattern that shows up in two of the clients I have audited), 1.0.3 turns that silent failure into an explicit error at handshake time — the connection refused up front, with a clear "empty ALPN" log line, instead of a request-time failure on the first inference call. For teams running the inference engine comparison surface, the empty-ALPN failure was the most common cause of "the mesh works locally but not from this office" tickets.
- Bug fix: add pkarr resolver to n0 preset (#4412) — The default resolver preset used by the iroh relay now includes the pkarr (public-key-addressable resource records) resolver alongside the existing DNS resolver. Until 1.0.3, an iroh endpoint published via pkarr-only DNS would resolve on the relay but fail peer-to-peer, which produced an asymmetric mesh where some peers could reach each other and others could not. 1.0.3 makes pkarr resolution a default, so a Mesh LLM gateway published only via pkarr (the deployment pattern that skips the public DNS record entirely, useful for air-gapped office networks) resolves consistently. For teams running Mesh LLM on networks where the IT department blocks DNS changes (the deployment pattern that motivated Mesh LLM in the first place), 1.0.3 is the line where the pkarr-only publish is a viable default.
- CI: move Android test back to self-hosted nodes (#4413) — A CI-only change. The Android instrumentation test was running on hosted runners where the network model did not match the iroh ALPN handshake (the same handshake the empty-ALPN fix above improves). 1.0.3 moves the test back to self-hosted nodes that exercise the real handshake shape, which means CI now catches regressions in the ALPN path that the hosted runner was silently skipping. Frontend correctness only — no production behavior change — but if you maintain a fork of iroh with custom CI, 1.0.3 is the line where the Android test path stops being a false-green.
- Refactor: reduce warn logs (#4378) — A noise reduction on the iroh core. The handshake path emitted a high-volume warn-level log for every peer connection that did not match the expected ALPN — a useful diagnostic on the broken-mesh path but noise on the working-mesh path. 1.0.3 drops most of the warn-level logs to debug-level, so the warn-level surface now actually means "something is wrong" rather than "the peer connection completed as expected." For teams running Mesh LLM with log aggregation on (the same observability surface the OpenTelemetry AI inference tracing guide covers for the inference layer above iroh), 1.0.3 is the line where warn-level alerts on iroh handshakes stop firing on healthy traffic.
- Build: remove vergen use from build.rs files (#4426) — A build-system cleanup. The vergen-emit-build-info step was duplicating build metadata that cargo already provides, which produced a non-deterministic build hash on hosts where the build environment changed between CI runs. 1.0.3 removes vergen, so the build hash is now reproducible. Frontend correctness only — no production behavior change — but if you pin iroh by build hash for supply-chain reasons (the same pattern the supply chain security for DevOps 2026 guide covers), 1.0.3 is the line where the build hash is stable.
No breaking changes. Treat 1.0.3 as a normal patch upgrade on the 1.0 line — the empty-ALPN error is the visible behavior change, the pkarr resolver addition is the deployment-shape change, the warn-log refactor is a noise reduction, and the CI/build changes are silent correctness improvements. Upgrade via cargo update -p iroh or pull the matching container. The Mesh LLM gateway continues to work unchanged on 1.0.3; if you observed any of the silent handshake-failure modes above (the empty-ALPN one was the most common), 1.0.3 is the release where the failure becomes a visible, actionable error. For teams that pin iroh via a fork, the build-hash reproducibility change is worth a re-pin: the new build hash is deterministic, and any forks that pinned the old vergen-derived hash will need a re-pin to pull 1.0.3.