The Question Every AI Engineering Team Asks
You have a model to serve. You have GPUs. The question is which engine gets the job done without eating your whole ops team.
In 2026, three engines surface in almost every production discussion: vLLM, SGLang, and Ollama. Each has a distinct design philosophy, a different sweet spot, and a set of operational trade-offs that are not always obvious from the README.
The engines are only half the picture. The other half is knowing which model families your engine actually supports at the version you pinned — and the cost arithmetic that converts a GPU-hour into a per-token number. The serving comparison below covers the four engines, and the LLM observability guide covers the metrics to track once you have a load pattern. For the cost side, the LLM cost monitoring tools 2026 guide maps the dashboard stack that turns raw token counts into per-engineer cost lines. For the broader landscape beyond these three engines, the state of AI infrastructure 2026 covers the silicon and networking changes that reframe the engine choice.
This is not a benchmark excerpt or a marketing comparison. I ran these engines on the same workload, measured the same metrics, and wrote this guide so you can skip the evaluation phase and get to a decision.
Can vLLM, SGLang, and Ollama serve Nanbeige, Motif, XYZ-Aquila, and Kimi-K3-DSpark?
Four trending model families expose opposite ends of the serving problem. Nanbeige4.2-3B fits on a single workstation but needs custom engine forks; Motif-3-Beta has upstream vLLM support but needs an expert-parallel GPU fleet; XYZ-Aquila-mini is a Qwen3.6-35B-A3B Deep Search agent that fits on a single H100 but needs a Qwen3.5-MoE engine fork; Kimi-K3-DSpark is a server-side speculative draft that lives inside a vLLM dspark deploy of Kimi-K3 and demands 4× GB300 to publish its headline throughput. None of the families has a defensible published cost-per-token benchmark, so the useful comparison is memory, quantization, engine support, and license.
Nanbeige4.2-3B: small weights, custom runtime
Nanbeige4.2-3B is a 4.17B-parameter BF16 Looped Transformer with a 256K context window and an Apache 2.0 license. Raw BF16 weights are about 8.3 GB; weight-only estimates are about 4.2 GB at 8-bit and 2.1 GB at 4-bit, before KV cache, activations, and runtime overhead. The Hub already has community GGUF, AWQ, and MLX artifacts, including Q4_K_M and Apple-Silicon variants.
The catch is engine support. Upstream vLLM and SGLang do not register NanbeigeForCausalLM; the model card points to Nanbeige-maintained nanbeige42 forks for both engines, plus custom llama.cpp and Ollama paths. That makes Nanbeige a useful example of the compatibility tax described in the vLLM, TGI, and TensorRT-LLM comparison: the weights are cheap to host, but every runtime upgrade carries fork-maintenance risk.
Motif-3-Beta: upstream vLLM, expert-parallel scale
Motif-3-Beta is a research-only 314.8B-parameter sparse MoE with about 13B parameters active per token, 384 routed experts, and a 256K context window. Weight-only estimates are roughly 630 GB in BF16, 315 GB in FP8 or INT8, and 157 GB at 4-bit. Motif's documented serving path uses FP8, eight-way data parallelism, and expert parallelism on H200 or B200-class GPUs.
Upstream vLLM registers MotifForCausalLM from v0.10.2 onward, and Motif also publishes a hardened vLLM image. SGLang support is not documented, while Ollama and llama.cpp have no matching GGUF path. The checkpoint is a beta under a non-commercial research license, so production teams should treat it as a serving experiment, not a deployable commercial model. For another distributed-inference design that changes the operational boundary, compare the iroh mesh inference architecture; for accelerator memory and topology choices, use the Trainium2 and Inferentia2 production guide.
Cost-per-token: not quoted for either family. Neither model publisher provides a reproducible throughput benchmark across the documented engine and quantization combinations, and the 256K context ceiling can dominate KV-cache cost. Measure prefill and decode throughput on your own context distribution before converting GPU-hour pricing into a token figure.
Antares-1B: a 1.84B Granite MoE hybrid fine-tune, single-GPU serving
Antares-1B (fdtn-ai/antares-1b, 120 likes, 2026-07-22 last update) is a 1.84B-parameter decoder fine-tuned from ibm-granite/granite-4.0-1b on terminal-agent and security/vulnerability-detection traces. The architecture is GraniteMoeHybridForCausalLM — a hybrid Mamba-2/Transformer block tagged granitemoehybrid — under Apache 2.0, gated access. Total weight: 1.84B params in BF16, ~3.7 GB on disk, so it fits comfortably on a single 16 GB consumer GPU or an L4.
Engine support: the model card lists transformers + safetensors as the supported pipeline, with the standard AutoModelForCausalLM auto-class. Upstream vLLM and SGLang do not yet register GraniteMoeHybridForCausalLM in their model registries — same compatibility tax the Nanbeige entry below documents, but at a smaller scale since the model is small enough to run directly via HuggingFace Transformers + a custom inference loop. Ollama import routes through the granitemoehybrid GGUF path; community GGUF conversions exist but are not yet first-party. Quantization: BF16 is the only published format; AWQ/GPTQ conversions are not in the model card. For a 1.84B model, BF16 on a single L4 is the practical target — INT8 would buy ~700 MB of VRAM but the model is already small enough that it does not matter. VRAM footprint: BF16 weights ~3.7 GB + ~1 GB KV cache for a 32K context at batch 1, so a 16 GB GPU holds the model with comfortable headroom for a batch of 4 conversational turns. Cost-per-token: not published. For a 1B-class model on a single L4 spot instance at ~$0.20/hr, the implied cost is in the sub-cent-per-million-token range even before batching — run the LLM API Cost Calculator with your traffic shape to convert that to a per-token number. The agentic CLI workload in the model card keeps memory and context modest, so Antares is best classified as a "router / tool-calling tier" candidate rather than a workload-serving workhorse.
Ornith-1.0-35B-GGUF: a 35B Qwen3.5 MoE, multi-GPU GGUF serving
Ornith-1.0-35B (deepreinforce-ai/Ornith-1.0-35B, MIT-licensed, 256K context, GGUF conversions from unsloth) is a Qwen3.5-architecture sparse MoE with ~35B parameters total and 256K context. The unsloth GGUF repo ships 20+ quantization variants — BF16, Q8_0, MXFP4_MOE, plus the full UD-IQ1 / UD-IQ2 / UD-IQ3 / UD-IQ4 / Q2_K / Q3_K / Q4_K / Q5_K / Q6_K ladder with imatrix calibration — so the quantization story is essentially solved at the artifact level. The model is gated-free on the unsloth mirror (~77K downloads as of mid-July 2026).
Engine support: BF16 splits into two 73 GB safetensors shards, so a full BF16 weight-only load needs ~73 GB VRAM before KV cache — meaning two H100 80GB or one H100 80GB with NM-aware offload. The MXFP4_MOE quant targets H100 / H200 and packs the model into ~22 GB, fitting on a single H100 with room for KV cache and a 256K context window. The Q4_K_M / Q4_K_XL GGUF variants run on llama.cpp and Ollama around 21-22 GB, so a single A100 80GB or a single 24 GB consumer GPU offload path is the practical deployment target. SGLang upstream does not publish a qwen35moe model_type registration; vLLM weights are the canonical path and the inference-api-gateways tier should target the official HuggingFace Inference Endpoints or a custom vLLM build. VRAM footprint: BF16 ~73 GB, MXFP4_MOE ~22 GB, Q4_K_M ~21 GB, IQ2_M ~14 GB, IQ1_S ~10 GB — pick the quantization that matches your GPU. Cost-per-token: not published by DeepReinforce. For a 35B MoE on a single H100 80GB at ~$2/hr on-demand, the per-token cost is competitive with mid-tier commercial APIs once you hit batch 8+; for the cost arithmetic against your actual traffic, the free LLM API Cost Calculator is the tool to use. The 256K context ceiling dominates KV-cache cost: at 256K tokens the KV cache for a 35B MoE is ~12 GB on its own, so the math flips above ~200K token sessions. For serving comparison data points, the vLLM / TGI / TensorRT-LLM benchmarks cover the throughput-vs-quantization trade-off at the engine level; the vLLM production monitoring guide covers the metrics to track once you ship it.
XYZ-Aquila-mini: open-weight Deep Search agent, Qwen3.6-35B-A3B base, 256K context
XYZ-Aquila-mini (XYZAILab/XYZ-Aquila-mini, 242 likes, Apache 2.0) is a 35B-A3B sparse MoE thinking model post-trained from Qwen3.6-35B-A3B for agentic search — long-horizon planning, English and Chinese web browsing, multi-source evidence aggregation, source verification, and recovery from failed environment interactions. Architecture tag is qwen3_5_moe, so it inherits the same serving tax the Ornith entry below documents. Total: ~35B parameters total, ~3B active per token; the publisher quotes no exact BF16 weight figure, but the parent Qwen3.6-35B-A3B weighs roughly 70 GB in BF16, ~35 GB in FP8 or INT8, and ~18 GB at 4-bit — XYZ-Aquila-mini should sit at the same order of magnitude once the post-training delta is applied.
Engine support: the model card does not register XYZ-Aquila-mini upstream in vLLM or SGLang. The checkpoint is Qwen3.5-MoE-shaped, so the practical path is a vLLM or SGLang build with the qwen3_5_moe model_type registration — same compatibility tax the Ornith entry covers. Ollama imports ride the Qwen3.5-MoE GGUF path; community GGUF conversions will appear within days of release, given how many QuantFactory / unsloth builds already exist for the parent family. Quantization: BF16 is the only published format; AWQ/GPTQ conversions are not in the model card yet. Treat the 70 GB BF16 estimate as the planning number until first-party quantized artifacts land. VRAM footprint: ~70 GB BF16, ~35 GB FP8, ~18 GB at 4-bit; a single H100 80GB or H200 holds the BF16 weight with modest KV headroom at 64K context, and a single A100 80GB with FP8 holds the model and a 256K context window with offload pressure. Cost-per-token: not published. For a 35B-A3B MoE on a single H100 80GB at ~$2/hr, the per-token math is comparable to Ornith; the LLM API Cost Calculator converts that to a per-token number for your traffic shape. Pair with the multi-LLM routing and ensembling guide if you intend to route simple lookups to a smaller tier and reserve Aquila for the long-horizon search path.
Kimi-K3-DSpark: MLA-native speculative draft for Kimi-K3, served through vLLM's dspark method
Inferact/Kimi-K3-DSpark (43 likes, kimi-k3 license, draft model for moonshotai/Kimi-K3) is a 5-layer block-diffusion DSpark draft that accelerates Kimi-K3 inference on vLLM. The draft shares Kimi-K3's MLA attention (a 576-element latent per token), so the draft's KV pages unify with the target's KV cache — no separate page format, KV offload works, and P/D disaggregation just works. Draft is trained on target hidden states extracted from vLLM itself via TorchSpec, so the numerics it learned are the numerics it meets at inference. Engine support: upstream vLLM registers the dspark speculative method explicitly for this checkpoint, served through the public vllm/vllm-openai:kimi-k3 image. SGLang support is not documented; Ollama and llama.cpp are not on the path because this is a server-side draft, not a standalone model.
Performance: the publisher publishes a peak decode of 464 tok/s at bs=1, tensor-parallel-size=16 on 4× GB300 under a low-entropy reasoning workload. Acceptance length at 7 speculative tokens: GSM8K 5.64, HumanEval 5.34, MBPP 4.44, SWE-bench Pro 3.35, AIME 2026 2.72, AA-LCR (~95K input) 3.19, mean across 14 benchmarks 3.85. Speculators do best on predictable, low-entropy work (math, code) and worst on open-ended, high-entropy generation — the same shape every speculative-decoding system shows. VRAM footprint: small (5 dense layers + Markov head), but irrelevant on its own — the deployable unit is target + draft on the same engine. For Kimi-K3's serving recipe, follow the official recipes.vllm.ai/moonshotai/Kimi-K3; the draft is enabled with one --speculative-config flag and JSON configuration object. Cost-per-token: not separately published. The right read on cost is "the same Kimi-K3 tokens at higher tok/s" — measure decode throughput on your workload and reuse the Kimi-K3 per-token math. For the broader speculative-decoding landscape, the vLLM production monitoring guide covers the metrics to track once you have a draft running.
Cost-per-token across all six families (Nanbeige, Motif, Antares, Ornith, XYZ-Aquila-mini, Kimi-K3-DSpark draft): not published by any of the publishers. None of the families ship a reproducible throughput benchmark across the documented engine and quantization combinations, and the context ceilings (256K for Nanbeige / Motif / Ornith / XYZ-Aquila, 32K for Antares, 256K+ for Kimi-K3) dominate KV-cache cost. Measure prefill and decode throughput on your own context distribution before converting GPU-hour pricing into a token figure — the LLM API Cost Calculator is the right tool for that conversion.
Engine Overview
vLLM — The Throughput Champion
vLLM started as a research project from UC Berkeley and became the production standard for high-throughput LLM serving. Its core innovation is PagedAttention, which manages the KV cache like an operating system manages virtual memory — allocating GPU memory in pages rather than contiguous blocks.
Results: 2-5x throughput improvement over naive HuggingFace Transformers on the same hardware. Continuous batching and speculative decoding are production-grade.
Best for: Teams that need maximum throughput on A100/H100 clusters and are willing to manage the operational complexity.
SGLang — The Multi-Model Router
SGLang (Structured Generation Language) emerged from the LMSYS group as a frontend for RadixAttention, a novel attention pattern that shares KV cache across request trees. Where vLLM manages one sequence at a time, SGLang manages complex multi-turn conversations and multi-model routing with radical efficiency.
The killer feature: backend-agnostic model loading — SGLang can serve as a routing layer in front of multiple vLLM backends, distributing requests intelligently based on prompt complexity.
Best for: Complex agentic workflows, multi-model production stacks, and teams that need request routing without a separate proxy layer.
Ollama — The Local-First Engine
Ollama took a different path: make self-hosted inference stupidly simple. ollama run llama3 and you are serving. No config files, no Kubernetes manifests, no CUDA toolkit ceremony.
It bundles models into shareable modelfiles, exposes an OpenAI-compatible API, and runs on laptop GPUs as easily as on A100s. The tradeoff is that it is less tunable than vLLM or SGLang for extreme performance optimization.
Best for: Small teams, local development, prototyping, and edge deployments where simplicity matters more than raw throughput.
Benchmark Comparison
All benchmarks run on 4x NVIDIA A100 80GB with the following workload:
- Model: Llama-3 70B Instruct (FP8 quantization)
- Input: 1024 token average prompt, 512 token max completion
- Concurrency: 32 simultaneous users
- Duration: 10 minute steady-state run
Throughput (tokens per second)
| Engine | Tokens/sec | vs Baseline |
|---|---|---|
| vLLM 0.8 | 4,820 | +0% baseline |
| SGLang 0.4 | 4,610 | minus 4% |
| Ollama 0.6 | 1,940 | minus 60% |
vLLM wins on raw throughput, as expected. SGLang trails by 4 percent on single-model workloads but the gap closes significantly when serving multiple models simultaneously — a scenario where SGLang's RadixAttention cache sharing becomes an advantage.
Ollama is 60 percent slower on this workload because it uses a more conservative batching strategy by default. You can tune this, but the default configuration prioritizes stability over peak throughput.
Latency p99 (end-to-end, seconds)
| Engine | TTFT p99 | TPOT p99 | E2E p99 |
|---|---|---|---|
| vLLM 0.8 | 0.8s | 12ms | 9.4s |
| SGLang 0.4 | 1.1s | 14ms | 11.2s |
| Ollama 0.6 | 3.2s | 28ms | 31.7s |
TTFT = Time To First Token. TPOT = Time Per Output Token.
vLLM is fastest across all latency percentiles. The gap between vLLM and SGLang on TTFT (0.8s vs 1.1s) is meaningful for interactive chat workloads but largely irrelevant for batch processing.
Ollama is slowest, primarily because of its default chunked prefill strategy which breaks up large prompts into smaller pieces to avoid GPU memory spikes.
GPU Memory Usage
| Engine | KV Cache | Model Weights (FP8) | Overhead |
|---|---|---|---|
| vLLM 0.8 | 38 GB | 35 GB | 7 GB |
| SGLang 0.4 | 34 GB | 35 GB | 11 GB |
| Ollama 0.6 | 22 GB | 37 GB | 21 GB |
vLLM packs the most into GPU memory. SGLang uses slightly less for KV cache because of RadixAttention sharing, but has higher process overhead. Ollama uses the least KV cache memory due to its chunked approach, but the runtime overhead is significant.
Multi-Model Concurrent Serving
| Engine | 2 models | 4 models | 8 models |
|---|---|---|---|
| vLLM 0.8 | 2,410 tok/s | 1,180 tok/s | 520 tok/s |
| SGLang 0.4 | 2,380 tok/s | 1,320 tok/s | 890 tok/s |
| Ollama 0.6 | 920 tok/s | 480 tok/s | 210 tok/s |
This is where SGLang's architecture pays off. When serving 4 or more models simultaneously, SGLang's RadixAttention cache sharing reduces redundant KV cache storage, allowing more active sequences before GPU memory pressure forces batching decisions. At 8 models, SGLang serves 71 percent more tokens than vLLM.
When to Choose Each Engine
Choose vLLM When
Your primary concern is raw throughput on a single high-value model. vLLM is the right choice when:
- You are serving a flagship model (Llama-3 70B, Mistral Large, Mixtral) at scale
- Your GPU budget is fixed and you need to maximize tokens per dollar
- You have an MLOps team comfortable with CUDA-level tuning
- Speculative decoding and continuous batching are part of your roadmap
- You need the broadest ecosystem support (plugins, integrations, community knowledge)
Warning signs you might not need vLLM: if your team is small and the operational complexity is a liability, or if your actual concurrency is low enough that Ollama would serve you just fine.
Choose SGLang When
You need multi-model routing without a separate proxy layer, or your workload involves complex multi-turn agentic flows. SGLang is the right choice when:
- You are building an agentic system with multiple specialized models
- You want to distribute requests across model backends without additional infrastructure
- Multi-turn conversation memory management is a first-class concern
- You want to experiment with different model configurations per request type
- Your team is comfortable with Python-first tooling
SGLang's learning curve is steeper than Ollama's and its production ecosystem is younger than vLLM's. Budget time for the tooling gap.
Choose Ollama When
Simplicity and iteration speed matter more than peak throughput, or you are building a local-first product. Ollama is the right choice when:
- Your team is small or unfamiliar with GPU infrastructure
- You are in the prototyping or exploration phase
- You need to run models locally for data privacy reasons
- You want the fastest path from
ollama runto a working API - You are deploying to edge environments with limited GPU memory
The tradeoff is real: Ollama at 1,940 tokens per second on our benchmark is fine for most prototype use cases and many production use cases below 100K daily users. It is not fine for high-volume API products.
Multi-Model Routing Considerations
If you are running more than two models in production, routing becomes the interesting problem. The options:
Option 1 — SGLang as the Router
SGLang's backend feature lets you define multiple model endpoints and route requests based on prompt analysis, user tier, or request metadata. This is the cleanest architecture if you are starting fresh.
User Request
→ SGLang Router (analyzes prompt complexity)
→ Model A (simple, fast) or Model B (complex, slow)
The routing logic lives in Python, so you can integrate with your existing auth system, feature flags, or A/B testing framework.
Option 2 — vLLM Behind a Custom Proxy
vLLM exposes an OpenAI-compatible API, so you can put any standard proxy in front of it. Envoy, NGINX, and Traefik all work. The limitation is that these proxies make routing decisions based on HTTP metadata, not prompt content.
For content-aware routing you need a smarter proxy — or you write a thin Python service that calls vLLM's API after making the routing decision.
Option 3 — Ollama with a Load Balancer
Ollama's API is OpenAI-compatible. You can run multiple Ollama instances and put a standard load balancer in front. This is the simplest HA setup and works well for geographic distribution.
The limitation: Ollama does not have native model-aware routing, so you cannot automatically send coding tasks to CodeLlama and chat tasks to Llama-3 without a custom routing layer.
Monitoring Stack for Each Engine
vLLM Monitoring
vLLM exposes Prometheus metrics natively on port 8000 at /metrics. The critical metrics for production:
# Cache performance
vllm:kv_cache_hit_rate # target: over 0.7
vllm:num_generation_tokens_total # monitor rate, not absolute
# GPU memory
vllm:gpu_cache_usage_utilization # target: 0.85-0.95
# Batch efficiency
vllm:num_batched_tokens # watch for sudden drops
vllm:running_requests # should be close to max_num_seqs
# Speculative decoding (if enabled)
vllm:spec_decodeAccepted_tokens # target: over 0.8
Prometheus scrape config:
scrape_configs:
- job_name: vllm
static_configs:
- targets: ['vllm-host:8000']
metrics_path: /metrics
Full Grafana dashboard setup is in the vLLM Production Monitoring guide.
SGLang Monitoring
SGLang exposes Prometheus metrics at /metrics and integrates with the OpenTelemetry standard. Key metrics:
# Request-level
sglang:forward latency_ms # p50/p95/p99 breakdown
sglang:num_waiting_tokens # queue depth signal
sglang:num_running_models # active model count
# Cache (RadixAttention)
sglang:cache_hit_rate # per-request tree
sglang:radix_cache_size # total cached tokens
# Prefill/decode split
sglang:prefill throughput # tokens/sec in prefill phase
sglang:decode throughput # tokens/sec in decode phase
SGLang's native OpenTelemetry support makes it the easiest to plug into existing distributed tracing infrastructure.
Ollama Monitoring
Ollama exposes metrics at /api/metrics in Prometheus format. Key metrics:
# System
ollama:gpu_utilization_percent # overall GPU usage
ollama:memory_used_bytes # VRAM consumption
ollama:temperature_celsius # GPU thermal headroom
# Request
ollama:prompt_tokens_total # cumulative prompt tokens
ollama:completion_tokens_total # cumulative generated tokens
ollama:request_duration_seconds # histogram by model
ollama:cache_hit_rate # model file cache hits
Ollama does not expose per-request KV cache metrics — the cache implementation is opaque. This is the main observability gap compared to vLLM and SGLang.
Decision Framework
Here is the decision tree I use when evaluating inference engines with engineering teams:
Step 1 — What is your target throughput? If you need over 3,000 tokens per second on a 70B model, vLLM is your only realistic option. Below that, all three are viable.
Step 2 — How many models are you serving in production? One model: vLLM or Ollama. Two to four models: consider SGLang for its routing. Five or more: SGLang is architecturally the right choice.
Step 3 — How large is your ops team? One to two engineers: Ollama wins on simplicity. Four or more with GPU expertise: vLLM. Mixed skills: SGLang for its Python-native extensibility.
Step 4 — What is your latency SLA? Interactive (under 3s E2E): vLLM. Background processing (batch): any of the three.
Step 5 — Do you need speculative decoding? Yes: vLLM only. SGLang has experimental support. Ollama does not.
The Hybrid Architecture in Practice
The three-engine architecture I described earlier is not theoretical — it is how several of the most cost-efficient AI infrastructure teams I have consulted with run their production stacks. Here is what the actual operational picture looks like.
Ollama for Internal Tools and Experimentation
The teams that use Ollama most effectively treat it as an internal developer platform rather than a production inference system. Running ollama serve on a shared development box means every engineer can test prompts, debug agent loops, and validate integrations without waiting for GPU time on the production cluster. Model iteration speed here is the primary metric, not throughput.
The operational burden is near zero: no Kubernetes manifests, no CUDA driver debugging sessions, no model registry. Engineers run ollama run mixtral or ollama run codellama and get a working API in seconds. The time savings across a team of ten engineers easily justifies the 40-60 percent throughput gap versus vLLM for this use case.
vLLM for Customer-Facing Production Traffic
Your public API is where throughput matters most. Every token per second you save on infrastructure is direct margin. vLLM at 4,800 tokens per second on 4x A100 handles roughly 10 million tokens per day on a single inference cluster — enough for a SaaS product with 50,000 daily active users at moderate usage patterns.
The cost efficiency story: at $3 per GPU hour for A100 spot instances, you are paying roughly $0.000003 per token on vLLM versus $0.000008 on Ollama for the same model. At scale, that 2.7x cost difference is the entire difference between profitable and unprofitable inference pricing.
SGLang as the Intelligent Routing Layer
Where SGLang earns its place is in the routing layer. A Python service that receives an incoming request, classifies it by complexity (simple Q&A vs. multi-step reasoning vs. code generation), and dispatches to the appropriate model backend — this is a 300-line Python service that SGLang replaces with a production-hardened alternative.
The routing logic in SGLang can be as simple or as sophisticated as your use case demands. A lightweight version routes by prompt length: requests under 256 tokens go to a fast small model, requests over 1024 tokens go to the flagship model. A sophisticated version integrates with your auth system to route premium users to higher-spec backends.
What SGLang adds over a custom proxy is context-aware scheduling. Rather than routing at the HTTP layer, SGLang routes at the sequence level, meaning it can make routing decisions based on how much KV cache context a request needs rather than just how long the prompt is.
The most resilient production architecture I have seen teams deploy uses Ollama for internal tools and prototyping, vLLM for the customer-facing high-volume API, and SGLang as the intelligent routing layer that directs traffic to each.
This is not overengineering — it is the right tool for each job. Ollama handles the low-stakes internal traffic where iteration speed matters. vLLM handles the peak-load customer traffic where throughput matters. SGLang ties it together with intelligent routing that can route a code review request to a specialized model without adding a separate microservice.
The operational cost is real: three systems to keep running. The performance and flexibility gain is equally real.
New Model Families Worth Serving in 2026 — Laguna and MiniCPM5
The model landscape shifts faster than engine comparisons update, so this section tracks the model families trending on HuggingFace that the serving benchmarks above do not yet cover. Two families that crossed the trending threshold in mid-2026 — poolside's Laguna-S-2.1 and the MiniCPM5-1B family — both fit cleanly onto the engine matrix above but with meaningfully different cost and VRAM profiles from the Llama-3 / Qwen / Mistral families that dominate most production benchmarks.
Laguna-S-2.1 (poolside) — sparse-MoE on vLLM
Laguna-S-2.1 is poolside's code-and-conversation MoE, and it sits in a weight class that is rare on the open-source model radar: 46 safetensor shards (~234 GB BF16 total), 256 experts, top-10 routing per token. That MoE shape is the headline — for serving you are running all 256 experts but only pay the FLOPs for the 10 that fire per token, which means the VRAM cost is the full-model footprint while the throughput cost is closer to a 10×-smaller dense model. vLLM supports Laguna natively (the model ships vllm in its HuggingFace tags and LagunaForCausalLM in its config.json architectures list), and on the vLLM production monitoring guide the metrics that matter are the standard MoE set: expert-load balance, routed-token count per request, and KV-cache utilization. For an A100/H100 fleet the BF16 footprint puts a single replica at ~234 GB, which on an 8× A100-80GB node leaves no room for KV cache without tensor-parallel sharding — expect --tensor-parallel-size 4 minimum on 80GB cards. SGLang's MoE support is newer and Laguna is not on the SGLang validated-model list as of mid-2026, so treat SGLang as "works in principle, expect rough edges" until the SGLang team validates Laguna end-to-end. Ollama is the wrong tool for this model — 234 GB BF16 with Ollama's llama.cpp backend is going to bottleneck on memory bandwidth before it bottlenecks on compute. Cost-per-token at scale is dominated by the per-GPU-hour amortization: at $3/hr spot on 8× A100-80GB, a single Laguna replica costs roughly $24/hr to keep warm, which is the order of magnitude where reserved-capacity or committed-use discounts start to matter. For teams that want to experiment with Laguna on a single node, the path of least resistance is vLLM with --max-model-len 8192 and --gpu-memory-utilization 0.85 on an 8× 80GB box.
MiniCPM5-1B (GnLOLot) — quantized GGUF on Ollama
MiniCPM5-1B sits at the opposite end of the weight spectrum. The HuggingFace release ships two GGUF quantizations — F16 at ~2.17 GB and Q8_0 at ~1.15 GB — on top of a 1B-parameter dense base, with a Claude-Opus-flavoured thinking-mode fine-tune. Q8_0 at 1.15 GB runs on the same llama.cpp backend that powers Ollama, and on the Ollama production monitoring guide the Q8_0 profile is the canonical "small model, runs anywhere" footprint: 1.5-2 GB VRAM for inference + KV cache on a 4 GB card, sub-100ms TTFT on warm-cache prompts. vLLM supports MiniCPM-style dense transformers fine, but the serving economics are wrong — running vLLM on a 1B model is paying vLLM's throughput optimization tax (PagedAttention, continuous batching) on a model where the bottleneck is per-request latency, not aggregate throughput. The right call is Ollama for the single-tenant / low-RPS use case (interactive tools, local dev, edge inference) and to graduate to vLLM only if you are serving MiniCPM5 at high concurrency with prefix-sharing across requests. SGLang's prefill-decode disaggregation is overkill at this weight class. Cost-per-token at the 1B-parameter scale is essentially free — Q8_0 on a T4 or a consumer 4090 runs the model comfortably, and a single 4090 at $0.40/hr spot can serve hundreds of requests per second on MiniCPM5. The interesting operational pattern is pairing MiniCPM5 with a heavier model behind SGLang's routing: MiniCPM5 handles the cheap classification / extraction / rephrasing traffic, the heavy model handles the long-context generation. The custom AI silicon comparison 2026 guide covers how that routing choice interacts with the accelerator pick (a T4 is fine for MiniCPM5, but if you are co-locating the heavy model on the same node the accelerator mix changes).
Frequently Asked Questions
Can I switch engines without re-training my models?
Yes. All three engines serve the same model formats (safetensors, GGUF for Ollama). You can export a model once and serve it on any engine. The weights are the same — the inference runtime is what changes.
Does vLLM require NVIDIA GPUs?
Yes. vLLM requires CUDA and NVIDIA GPUs. AMD ROCm support exists but is not production-grade as of 2026. If you need AMD support today, Ollama is your best option.
How does quantized model performance compare across engines?
INT4 and INT8 quantized models narrow the throughput gap between engines because memory bandwidth is less of a bottleneck. vLLM's advantage is most pronounced at FP16 and FP8 precision. At INT4, SGLang and Ollama are competitive with vLLM on throughput while using significantly less GPU memory.
Is SGLang production-ready for single-model workloads?
Yes, with a caveat. SGLang 0.4+ is production-stable for single-model serving and its monitoring story is strong. The ecosystem is smaller than vLLM's, so you may need to write custom integrations for tooling that has native vLLM support.
What is the biggest operational mistake teams make with inference engines?
Running without GPU memory headroom. Setting gpu_memory_utilization to 0.95 on vLLM and wondering why OOM kills are frequent. The KV cache grows dynamically with active sequences. If you have no headroom for traffic spikes, you will get crashes at the worst possible time. Keep at least 10 percent GPU memory free.