Tool
Which inference engine should you run?
Thirty-one engines, from datacenter serving down to single-binary desktop apps. Set your model size, hardware and use case and the configurator narrows it — or browse the whole catalogue with links to every repo.
Configurator
Describe your setup.
The catalogue
Every engine worth knowing about.
Reading the list
Four things worth knowing first.
- Some of these are not engines Dynamo, llm-d, Ray Serve, AIBrix and BentoML are orchestration layers. They route and schedule across vLLM or SGLang — they never replace one. Dynamo's own README says a single engine is probably enough for one model on one GPU.
- The datacenter tier has converged Prefix caching, continuous batching, chunked prefill, speculative decoding, prefill-decode disaggregation and multi-LoRA now exist in vLLM, SGLang and TensorRT-LLM alike. Pick on hardware portability and operational fit, not feature checkboxes.
- Every published benchmark here is vendor-run LMDeploy's "1.8x faster than vLLM" is undated and names no vLLM version. llm-d's throughput figures are project-run on NVIDIA hardware. I found no neutral third-party head-to-head worth citing. Measure on your own workload.
- KV cache is what actually runs you out of memory Model weights are the number everyone quotes, but at long context the KV cache dominates. A 7B model at 4-bit fits comfortably on 24 GB — until you ask for 128k context.
Running these for real
Most of this was tested on 32 RTX 3090s.
If you are choosing a serving stack for something real, I am happy to talk through the trade-offs.