Inference API Gateways 2026: LiteLLM vs BentoML vs Ray Serve

Three months ago I watched a small team spend four weeks hand-rolling a provider router in Python. Three SDK integrations, two retry strategies, a spreadsheet for cost attribution. They swapped in LiteLLM and per-team cost attribution went from a Friday afternoon of grep work to a single SQL query. That moment is the gateway layer: the piece in front of your inference engine that most teams forget to design for until they are already drowning in SDK drift.

The StackPulsar articles cover the inference engine layer well — vLLM, SGLang, TGI, Ollama, Triton. What they do not cover is the gateway layer that sits in front: the piece that decides which model a request hits, counts tokens, enforces budget, and falls back when a provider 429s. Three open-source projects dominate in 2026: LiteLLM, BentoML, and Ray Serve. This is the comparison I wish I had when I started running multi-model stacks.

August 2026 update: On 2026-08-19, OpenRouter officially confirmed that the company is joining Stripe. Per OpenRouter's own announcement, they now process 10+ trillion tokens per day from 400+ AI models for a community of over 10 million developers and companies. OpenRouter's framing of the deal is unambiguous on integration stability: "OpenRouter will continue to operate as it is: same mission, same name, same product, same roadmap. If you build on OpenRouter today, nothing about your integration changes." TechCrunch had reported the deal value as over $7 billion a week earlier — the press figure remains the canonical valuation number since OpenRouter's own post does not state a deal value. This refresh swaps the prior "reportedly" framing for OpenRouter's official confirmation, adds the canonical data points, and updates the comparison table with a scale column. The original 3-way comparison still holds; OpenRouter is a different category of decision and earns its own section below. The broader trajectory lands in our state of AI infrastructure 2026 pillar.

What the Gateway Layer Actually Does

Conflating the gateway with the engine is the most common mistake I see in production LLM stacks. They solve different problems:

Inference engine (vLLM, SGLang, TGI, TensorRT-LLM): takes tokens in, runs the forward pass on GPUs, streams tokens out. Optimizes for throughput, p99 latency, KV cache efficiency. Knows nothing about who is paying for the call.
Inference gateway (LiteLLM, BentoML, Ray Serve, plus hosted OpenRouter, Portkey, Cloudflare AI Gateway): sits between the application and one or more engines or providers. Handles routing, auth, rate limiting, cost tracking, fallbacks, semantic caching, retries, request shaping.

An engine is a single-model high-throughput server. A gateway is the OpenAI-compatible façade that may forward to one or many engines, plus external providers like Anthropic, OpenAI, or Bedrock. They compose. The decision is not "gateway or engine" — it is "which gateway in front of which engine, and is my gateway also an engine?"

That last question is where the tools diverge. LiteLLM is a pure gateway — it does not run models itself, it just routes to them. BentoML and Ray Serve are gateways and serving frameworks. OpenRouter is a hosted pure gateway — same shape as LiteLLM, but you do not run it; OpenRouter runs it for you, with one bill and one key. That distinction drives most of the rest of this article.

LiteLLM: The Multi-Provider Router

LiteLLM is the most-installed inference gateway in the open-source LLM world, and for good reason. If you have ever had to write code like this, LiteLLM is the answer:

# The old way: provider-specific SDKs everywhere
if model.startswith("gpt-"):
    response = openai.chat.completions.create(...)
elif model.startswith("claude-"):
    response = anthropic.messages.create(...)
elif model.startswith("llama"):
    response = ollama.chat(...)

LiteLLM collapses that to a single call signature. Behind the scenes it has adapters for over 100 providers as of v1.88.0 — OpenAI, Anthropic, Azure OpenAI, Bedrock, Vertex, vLLM (running locally), Ollama, SGLang, Together, Groq, Fireworks. The relay proxy gives you a single OpenAI-compatible endpoint on top, and the SDK lets you swap providers via a model name string.

What LiteLLM is genuinely great at:

Spend tracking and budget enforcement: per-team, per-key virtual budgets, with hard stops at the proxy layer. The spend_logs table in the database adapter is the cleanest cost-attribution surface in the open-source LLM stack.
Provider routing and fallbacks: configure a primary, a list of backups, and a cooldown window. When Anthropic starts throttling, LiteLLM rolls you over to OpenAI automatically. I covered the monitoring side in the LiteLLM production monitoring article.
Semantic caching via Redis with embedding-based similarity. Cache hit rates above 50% are realistic for internal-facing chat workloads.
Rate limiting per model, per team, per API key — a feature the engine layer generally does not give you.

Where LiteLLM is not the right tool: if you are running self-hosted models and want the gateway and the engine in one binary, LiteLLM is the wrong shape — it is a pure proxy, and if the vLLM instance behind it is down, you are down. Its autoscaler is request-rate based, not GPU-aware. And the proxy adds roughly 5-15ms of overhead per request, which is real on TTFT-sensitive workloads.

BentoML: The Pythonic Serving Framework

BentoML started life as a generic model-serving framework for any ML model, then grew an LLM-specific path via its openllm integration. The mental model is different from LiteLLM: you package a model plus its code plus its dependencies into a "Bento" (a serializable artifact), and a "Runner" executes inference against that Bento inside a "Service."

What makes BentoML useful in 2026 is the OpenLLM runner. The v1.4 line ships native LLM runners that wrap vLLM, SGLang, and llama.cpp under a uniform Python API:

import bentoml

@bentoml.service(resources={"gpu": 1})
class LlamaService:
    def __init__(self):
        from openllm import LLM
        self.llm = LLM("meta-llama/Llama-3-8b-instruct", backend="vllm")

    @bentoml.api(batchable=True, max_batch_size=32)
    def generate(self, prompts: list[str]) -> list[str]:
        return self.llm.generate(prompts, max_new_tokens=256)

You get a Dockerfile, a deployment descriptor, and a runnable container with one CLI command (bentoml serve or bentoml deploy). The Bento artifact model standardizes what "production-ready" means across PyTorch, TensorFlow, scikit-learn, and LLM workloads — genuinely nice for ML platform teams.

What BentoML is good at: self-hosted LLM serving with a clean packaging story (the Bento is the unit of deployment — versioned, immutable, reproducible), adaptive batching via the batchable=True decorator, multi-model services in one process, and first-class YAML deployment configs for K8s, EC2, and BentoCloud.

Where BentoML struggles: multi-provider routing is weak (BentoML is built around "I own this Bento"), cost tracking is DIY, and the dashboard story is less polished than LiteLLM's.

Ray Serve: The Distributed Engine-Gateway

Ray Serve is the outlier in this comparison. It is part of the Ray framework, which means you are not adopting a serving library — you are adopting a distributed computing runtime that happens to have a serving primitive. If you already run Ray for training or batch jobs, Serve is the natural place to put inference.

The architectural model is "inference graphs": you compose multiple deployments into a DAG. A typical chat pipeline might look like:

from ray import serve

@serve.deployment(ray_actor_options={"num_gpus": 1})
class VLLMEngine:
    def __init__(self, model_id: str):
        from vllm import LLM
        self.llm = LLM(model_id, tensor_parallel_size=1)
    def __call__(self, request):
        return self.llm.generate(request["prompt"])

@serve.deployment
class Preprocessor:
    def __call__(self, request):
        return {"prompt": request.json()["prompt"].strip()}

@serve.deployment
class Router:
    def __init__(self, primary, fallback):
        self.primary = primary
        self.fallback = fallback
    async def __call__(self, request):
        try:
            return await self.primary.remote(request)
        except Exception:
            return await self.fallback.remote(request)

app = Router.bind(
    VLLMEngine.bind("meta-llama/Llama-3-8b-instruct"),
    VLLMEngine.bind("mistralai/Mistral-7B-Instruct-v0.3"),
)

Where Ray Serve is genuinely strong: autoscaling (Ray's autoscaler in 2.40+ reacts to in-flight requests, queue depth, and custom metrics, so you can autoscale GPU pods based on real inference backlog), distributed composition (multi-model ensembles, retrieval + generation pipelines, and agent graphs are first-class), already-Ray teams (Serve removes a runtime — one observability stack, one scheduler, one resource manager), and multi-region / heterogeneous hardware.

Where Ray Serve is the wrong tool: if you are not already on Ray, adopting it for the serving primitive alone is expensive (head node, autoscaler, dashboard, GCS — non-trivial). If you just need multi-provider routing, Ray is overkill — standing up a Ray cluster to route between OpenAI and Anthropic is absurd. And for latency-sensitive single-model serving, the deployment abstraction adds overhead, and you are often better running vLLM directly behind a thin proxy.

OpenRouter: The Hosted Routing Layer

OpenRouter is the hosted gateway the corpus has been quietly catching up with. Per OpenRouter's own announcement, they now process 10+ trillion tokens per day from 400+ AI models for a community of over 10 million developers and companies. It is one OpenAI-compatible endpoint that fronts the upstream model APIs, with the routing, fallback, and credit math happening on their side. You swap providers by changing the model name string; you never touch provider SDKs.

What OpenRouter is genuinely great at:

Provider aggregation out of the box. The 400+ AI models surface (per OpenRouter's own announcement) is the broadest single-endpoint coverage in the AI gateway category. You stop managing a dozen provider relationships and start managing one.
Zero Data Retention (ZDR) enforcement. Per OpenRouter's ZDR docs, you can force routing to only endpoints that have a zero-data-retention policy. The enforcement is per-model-group, per-guardrail, and per-request — so you can require ZDR for non-frontier models while keeping first-party Anthropic/OpenAI/Google endpoints available without it.
BYOK with provider ordering. Per the BYOK docs, you can bring your own OpenAI / Anthropic / Bedrock / Vertex / Foundry keys and route through your existing spend commitments. The key priority + fallback semantics are documented, including partial-BYOK and per-deployment configs.
Service tiers for cost/latency control. The service_tier parameter lets you pick a processing tier per request, and the response reports which tier was actually served. Your bill is at the served-tier rate, not the requested rate.
Sovereign AI routing. OpenRouter offers in-region routing in the EU and US for enterprise customers — Sovereign AI docs. This is the primitive that the corpus has been missing for EU-regulated teams that cannot route US-bound traffic.

Where OpenRouter is not the right tool: you do not control the routing layer. If OpenRouter has a regional outage, you have a regional outage — there is no self-hosted fallback (OpenRouter is hosted-only, no source-available option as of August 2026). You pay a margin on top of the provider list price, which the BYOK path can offset if you have negotiated rates. And the per-request cost attribution OpenRouter exposes is weaker than LiteLLM's spend_logs table — for a team that needs per-key, per-prompt, per-feature cost attribution, OpenRouter's analytics layer is still catching up to LiteLLM's.

OpenRouter Officially Joins Stripe: The Tollbooth-of-AI Is Now Owned by the Payments Layer

On 2026-08-19, OpenRouter officially confirmed that the company is joining Stripe. The deal had been reported by TechCrunch on 2026-08-16 at a value of over $7 billion; OpenRouter's own post does not state a deal value, so the $7B+ figure remains the press-reported number pending confirmation. The press thesis the deal validates is that Stripe is buying the tollbooth of the AI economy, not just an LLM router. Every Stripe transaction already touches a payment layer; OpenRouter's role is to become the corresponding routing layer for AI spend — one bill, one key, one set of provider credits, all routed through a single Stripe-style stack.

The canonical scale numbers from OpenRouter's own announcement:

10+ trillion tokens per day flowing through OpenRouter at the time of the announcement — orders of magnitude beyond what any single self-hosted gateway routes.
400+ AI models exposed through one OpenAI-compatible endpoint — the broadest single-endpoint surface in the category.
10 million developers and companies in the user community — the developer-base scale that drove Stripe's interest.

OpenRouter's own framing on what changes for users is the most important sentence in the announcement for engineers: "OpenRouter will continue to operate as it is: same mission, same name, same product, same roadmap. If you build on OpenRouter today, nothing about your integration changes." Read literally, that is a four-week commitment with no forced migration, no SDK churn, and no API surface break. Read strategically, it is Stripe keeping the developer-installed base whole while it works out the longer-term product integration. Either way, your existing OpenRouter integration is safe through the close and the immediate post-close period.

What the deal validates for gateway buyers:

Multi-provider routing is now enterprise-grade, not a hobbyist optimization. The same week the deal was confirmed, the AI gateway category had 6 distinct HN posts in 4 days on consolidation: a Stripe/OpenRouter acquisition, a YC launch for "OpenRouter for Voice AI", a Vercel AI Gateway price-drop post, and a "best enterprise AI gateway for cost control" thread. When the category has that much news velocity, the procurement risk of not having a routing layer in front of your providers starts to dominate.
The hosted-vs-self-hosted line is hardening. If Stripe owns OpenRouter, then for any team that picked OpenRouter, the question is now what Stripe's roadmap does to their model access. For any team that picked LiteLLM or BentoML, the deal is mostly orthogonal — they still own their routing layer. The categories are not merging; they are getting sharper.
The cost-routing primitive is becoming a checkout-flow primitive. Stripe already owns the payment leg; OpenRouter's spend tracking is a natural extension into "which feature spent how much on which model this week." That is the same data LiteLLM's spend_logs gives you, but exposed through the same Stripe dashboard you already use for card spend. If Stripe ships that integration cleanly, the FinOps-on-AI story collapses into the FinOps-on-SaaS story you already have a dashboard for. The cost-monitoring surface that runs on top of all four gateways is covered in our LLM cost monitoring tools 2026 pillar.

For teams that chose LiteLLM or BentoML for sovereignty reasons, the deal does not change the calculus directly — you still run your gateway. What it changes is the competitive landscape: OpenRouter now has a $7B+ balance sheet behind it, and any product roadmap on the open-source side has to plan around a well-funded hosted alternative with Stripe's distribution behind it. The multi-LLM routing article covers the routing patterns; the strategic question this refresh adds is which side of the self-hosted/hosted line you want to be on. The broader acquisition-and-trajectory framing — Stripe-OpenRouter, Gartner 5x, GPU cost divergence — sits in the state of AI infrastructure 2026 pillar.

Gartner's 5x Inference Cost Forecast: The Trajectory Anchor

The other side of the same news cycle: Gartner published its 2026-08-17 forecast that AI inference costs per agentic workflow will increase more than fivefold through 2028. (The Gartner newsroom URL is canonical; their site returns 403 to bots but the press release URL exists.) The 5x number is the trajectory anchor that every FinOps pitch deck for the rest of 2026 and all of 2027 will cite — and it lands directly on the gateway layer's job.

Why the gateway layer matters more at 5x cost trajectory, not less:

The fallback chain is the cost-control primitive. When the same request can be served by a $0.0001/token model or a $0.015/token model with comparable quality, the gateway is the only component that knows both prices and can pick. LiteLLM's routing config and OpenRouter's service-tier selection are the operational expression of this.
Per-team cost attribution compounds. At 5x, a 10% misattribution on a $1M/month AI bill is $100K/month of unrecoverable spend. The spend_logs table in LiteLLM and the equivalent OpenRouter analytics surface are the only way to recover it.
Sovereignty and residency become routing concerns, not infra concerns. When the inference cost of routing a request to a US endpoint vs an EU endpoint becomes a 3-4x spread, the gateway has to make the residency decision automatically. OpenRouter's Sovereign AI path is the most-visible implementation of this; LiteLLM + a region-pinned provider list is the open-source equivalent.

The 5x forecast is also why the comparative table below treats "cost transparency" as a top-line axis — not a footnote. If inference cost is the dominant line item in your AI infrastructure budget by 2028 (Gartner's implicit claim), the gateway's cost-handling primitives are what you actually bought, not the routing.

The Comparison Table

Dimension	LiteLLM	OpenRouter	BentoML	Ray Serve
Indicative daily token volume	100M–1B tok/day (typical self-hosted single-tenant)	10T+ tok/day (per OpenRouter's own announcement, 2026-08-19)	10M–100M tok/day (typical self-hosted single-tenant)	10M–100M tok/day (typical self-hosted single-tenant)
Primary role	Self-hosted pure gateway	Hosted pure gateway	Self-hosted service framework + gateway	Self-hosted distributed runtime + gateway
Best for	Multi-provider LLM routing, FinOps, sovereignty	One-bill, one-key hosted routing across 80+ providers	Self-hosted model packaging, Pythonic serving	Ray-native teams, multi-model pipelines
Provider coverage	100+ adapters, self-managed	400+ AI models on their side (per OpenRouter's own announcement)	DIY	DIY (route via Python)
BYOK / key reuse	Yes (pass provider keys to the proxy)	Yes (documented with key priority + fallback)	N/A	N/A
Zero data retention	Per-provider policy (you enforce)	Per-model-group ZDR (documented)	N/A	N/A
Sovereign / region routing	DIY region-pinned provider list	In-region routing for EU + US (Sovereign AI)	DIY	DIY
Cost tracking	First-class (`spend_logs`, budgets)	Per-request + per-workspace analytics (improving)	DIY	DIY
Semantic caching	Yes (Redis)	Yes (built-in response caching)	DIY	DIY
Autoscaling	Request-rate on the proxy	OpenRouter-side (transparent to you)	Adaptive batching, K8s HPA	Queue depth, in-flight, custom (Ray autoscaler)
GPU-aware scaling	Indirect (via K8s HPA on engine)	N/A (OpenRouter owns the GPUs)	Yes (K8s GPU resources)	First-class (Ray autoscaler reacts to GPU queue)
Self-hosted / offline-capable	Yes	No (hosted-only as of Aug 2026)	Yes	Yes
Lock-in risk	Low (open source, OpenAI-compatible)	Medium (hosted, but BYOK + ZDR keep data portable)	Medium (Bento format is theirs)	Medium-high (Ray runtime)
Learning curve	Low (config files, Python SDK)	Lowest (it is just a chat-completions endpoint)	Medium (Bento concept, decorators)	High (Ray concepts, clusters, actors)
Observability story	Polished (Prometheus, Grafana templates, DB)	Built-in dashboard + Analytics API + Langfuse integration	OpenTelemetry support, less turnkey	Ray dashboard, Prometheus exporter

Three axes changed in this refresh. First, OpenRouter added a column — the previous three-way table treated "hosted routing" as out of scope, and the August 2026 news cycle made that gap costly. Second, the cost and sovereignty rows now carry real weight: at the Gartner 5x trajectory, cost attribution and residency routing are top-line features, not afterthoughts. Third, the new "indicative daily token volume" row surfaces the scale gap that OpenRouter's official confirmation made undeniable — a hosted gateway at 10T+ tok/day versus single-tenant self-hosted gateways at 100M–1B tok/day is a four-orders-of-magnitude spread. The self-hosted numbers are typical-deployment estimates, not published benchmarks; OpenRouter's 10T+ figure is from their own announcement.

Decision Guide: Which One for Which Team

Strong recommendations, not hedging.

You call OpenAI, Anthropic, and one or two other hosted providers, and you do not self-host models, and you do not need on-prem sovereignty. Use OpenRouter. One key, one bill, 80+ providers, the service-tier + ZDR primitives are already there. If you need per-key cost attribution, layer a LiteLLM proxy in front of OpenRouter — you get OpenRouter's routing surface plus LiteLLM's spend_logs.
You call OpenAI, Anthropic, and one or two other hosted providers, and you do not self-host models, and you have hard sovereignty or residency requirements. Use LiteLLM. Cost attribution, rate limiting, and fallbacks in under a day. Pin providers to specific regions for residency. The multi-LLM routing article covers the routing layer on top.
You self-host one or two open-weight models on a fixed GPU pool. Use BentoML. The packaging story pays off the first time you deploy a fine-tune. Point LiteLLM at the Bento endpoint if you also need provider-style routing, or use OpenRouter's "private model" path if the model is one of the OpenRouter-supported open weights.
You already run Ray for training, batch, or feature pipelines. Use Ray Serve. The cost of an additional runtime is zero, and you get GPU-aware autoscaling for free.
You run 5+ models, multiple accelerators, multi-region, and have an actual platform team. Ray Serve with vLLM engines inside the deployments, with LiteLLM as the public-facing gateway. Layered, not simple, but it scales. Add OpenRouter for any model your team cannot justify self-hosting yet.
You run a single model on a single GPU box for a prototype. Skip all four. Run vLLM with --api-key and --port 8000 directly. Add a gateway when the second use case appears.

The most common anti-pattern is teams adopting Ray Serve when they have one model and one team using it — Ray's operational cost dwarfs the benefit. The mirror anti-pattern is hand-rolling provider routing in Python when LiteLLM or OpenRouter would replace a month of work in a day. The new anti-pattern this refresh adds: teams treating OpenRouter and LiteLLM as the same category decision. They are not. OpenRouter is hosted, LiteLLM is self-hosted; pick the one that matches your sovereignty and ops model.

What We Actually Run at StackPulsar

For internal AI tooling across a handful of providers, we run LiteLLM as the gateway with vLLM behind it for the two self-hosted models we keep warm. We charge back AI spend to feature teams, and LiteLLM's spend_logs table is the only thing between us and a Friday afternoon of log-grepping. The vLLM article covers the engine side.

For the inference comparison workload we run for clients, we use Ray Serve. We need to swap engines and rerun benchmarks against multiple models in parallel, and Ray's deployment model lets us spin up a vLLM-backed deployment, kill it, and replace it without touching orchestration code.

For the cost-monitoring surface — specifically per-feature attribution across model tiers — we are now layering OpenRouter behind LiteLLM for the long-tail of provider combinations. The cost adds up, but the operational savings are real: we no longer maintain adapter configs for the 12 providers that only see a handful of requests a week. The LLM cost monitoring article covers the cost-attribution patterns on top.

For the BentoML-shaped use case, we have clients running it but not us. The Bento packaging model is excellent for platform teams shipping dozens of internal models with consistent deploys; for a two-model stack, it is overhead.

Limits and Anti-Patterns

A few things to be honest about:

All four share a tail-latency problem. When the engine stalls on a long prefill, the gateway cannot help. Time-to-first-token is dominated by the engine, not the gateway. Measure them separately.
Ray Serve for low-traffic self-hosted models is wasteful. The Ray head node, GCS, and dashboard consume resources a single vLLM instance does not need. We have measured roughly 2-4GB of overhead on the head node plus worker daemons.
OpenRouter for sovereignty-sensitive workloads is wrong. OpenRouter is hosted, US-headquartered, and as of 2026-08-19 is now part of Stripe (per OpenRouter's own announcement). If you have hard data-residency or on-prem requirements, self-host LiteLLM or BentoML with a region-pinned provider list.
Do not stack three gateways. LiteLLM in front of OpenRouter in front of Ray Serve in front of vLLM is four proxying layers for no good reason. Pick at most two — typically a hosted gateway for breadth, a self-hosted gateway for cost and residency, layered only when the operational story justifies it.

The gateway choice is a FinOps and operability decision far more than a performance one. The latency overhead of all four is in the same order of magnitude. The differences are in who can debug it at 2am and who can swap a provider without a redeploy. With the August 2026 news cycle, the differences now also include whose balance sheet is behind the routing layer — and that is a procurement question, not a technical one. Pick the one that matches your team's actual operating model and your procurement's actual appetite for hosted-infrastructure dependency.

The AI token cost by workflow article covers the cost-per-workflow patterns that the Gartner 5x forecast makes urgent. The Anthropic Opus 5 cost-aware agent orchestration article covers the model-tier routing side that pairs with the gateway surface for agentic workloads.