Tool
GPUs for LLM work.
The numbers that actually decide what you can run: VRAM, memory bandwidth, low-precision support and power — NVIDIA, AMD and Intel, from the RTX 3090 up to the latest Blackwell and CDNA4 parts. Ranked by LLM performance, best on top; ★ marks the best value in each column.
FP16 = peak dense tensor/matrix TFLOPS (FP32 accumulate), not the 2× "sparsity" figure vendors headline. Checked against nvidia.com, amd.com and intel.com in July 2026. NVIDIA and AMD publish with-sparsity headlines, so the dense value is half (MI355X is 2.5 PFLOPS dense, not the 5.0 sparse figure). Intel publishes INT8 TOPS for Arc and only relative numbers for Gaudi, so those FP16 cells read "—". Confirm before budgeting.
Reading the table
Bandwidth decides speed, not TFLOPS.
- What the LLM score means A transparent heuristic, not gospel: bandwidth 45%, VRAM 30%, dense compute 25%, each normalized to the fastest card here. It's weighted for serving — decode is memory-bound — so a card wins by feeding data fast and holding a lot of it. Re-sort any column to rank by that metric alone.
- VRAM sets what fits The first gate. A model that doesn't fit doesn't run — no amount of compute helps. This is why 24 GB consumer cards define the consumer ceiling.
- Memory bandwidth sets token speed LLM decode is memory-bound, not compute-bound. Tokens-per-second tracks bandwidth far more than TFLOPS — an H100 SXM's real edge over a 4090 is its ~3.3 TB/s, not its FLOPS.
- Low precision is a generational divide FP8 (Ada/Hopper) and FP4 (Blackwell) roughly double effective throughput for models that support them. Ampere (3090, A100) tops out at INT8 — a real reason the 4090 punches above the 3090 beyond raw specs.
- Watch the power column On a multi-GPU rig, TDP × card count is your PSU and cooling budget. Thirty-two 350 W cards is 11.2 kW before the rest of the machine — which is the whole reason to care where the power comes from.
- Beyond NVIDIA — real, with caveats vLLM officially lists AMD Instinct MI200/MI300/MI350 and Radeon RX 7900/9000 as ROCm targets — the MI300X (192 GB) and MI355X (288 GB) genuinely lead on memory-per-card. AMD's new 32 GB workstation cards (Radeon AI PRO R9700, PRO W7800) put serious VRAM on a desktop. Intel Arc runs through vLLM's XPU path but it's rougher — no pre-built wheels, Python 3.12 only — and Gaudi needs its own software stack. CUDA is still the smoothest road; the others cost you setup time.
Put it together
Pick a card, then see what runs on it.
The VRAM calculator turns these numbers into "how many of these do I need"; the explorers cover what to run and how to serve it.