The VRAM equation
Every out-of-memory failure and every "why is my context so small" question comes down to one identity that vLLM solves at boot:
Weights are fixed by the model and its precision. The KV cache (where conversation context lives) gets whatever remains. If the remainder is too small for your max_model_len, vLLM refuses to boot and says so. That error is your signal to lower the context, pick a quantized checkpoint, or move to a bigger card.
nvidia-smi showing a nearly full GPU right after boot is pre-allocation, not a leak.The knobs that matter
| Kwarg | Set it to | Why |
|---|---|---|
| max_model_len | an explicit value (4096–8192 on 12 GB cards) | context window per request, and the single biggest KV lever. Models with 128k native context try to reserve KV for all of it and refuse to boot on small cards. |
| gpu_memory_utilization | 0.90 (single engine) | the share of each GPU vLLM claims. With several engines on one GPU, each engine's share, summing to ≤ ~0.85. |
| kv_cache_dtype | "fp8" | halves KV memory, so roughly doubles usable context and batch. Small quality loss for chat. The first thing to try when context-starved. |
| max_num_seqs | 8–16 on 12 GB, 32–64 on 24 GB | how many requests batch together inside the engine. More throughput, more KV pressure. |
| enable_prefix_caching | True | chat clients resend the whole history every turn; this makes turn N re-process only the new tail. |
| enable_chunked_prefill | True | slices a giant prompt into chunks so it doesn't freeze everyone else's stream. |
| dtype | "auto" | follows the checkpoint's precision. Leave it. |
| quantization | don't set it | load a pre-quantized checkpoint (e.g. an official AWQ repo) and vLLM detects the method itself. 4-bit AWQ shrinks weights to ~35% of fp16. |
And one from deployment_config rather than engine_kwargs: max_ongoing_requests. Always set it (the examples use 16–32). The framework default of 5 caps concurrency below what the engine can batch, and throughput dies quietly.
Napkin math for sizing
Weights, by precision:
KV cache per token varies a lot between models (grouped-query attention is the difference). Reference points at fp16, halve them with fp8 KV:
| Model | KV per token (fp16) | 4 GB of KV buys |
|---|---|---|
| Qwen2.5-3B | ~36 KB | ~110k tokens |
| Qwen2.5-7B | ~56 KB | ~70k tokens |
| Llama-3.1-8B | ~128 KB | ~31k tokens |
"Tokens" here counts all concurrent sequences together: 16 parallel requests averaging 2k tokens each hold 32k tokens in cache. Worked example, Qwen2.5-3B fp16 on a 12 GB card: 0.90 × 12 = 10.8 GB budget, minus 6.2 GB weights and ~0.7 GB activations, leaves ~3.9 GB of KV ≈ 108k tokens. max_num_seqs=16 with max_model_len=8192 schedules comfortably.
max_num_seqs × max_model_len may exceed the pool; vLLM admits sequences while KV fits and queues the rest. Heavy long-context load degrades to queueing, never to a crash.Raising max_model_len — which lever helps
The most common question after adding hardware: "I have two GPUs now, can I raise the context?" It depends on how the second GPU is wired in, because the context budget is per replica:
| You added… | More context? | Why |
|---|---|---|
| a 2nd worker as another replica | no | each replica still has one card's VRAM. You bought throughput, not context. |
| a 2nd GPU inside the worker + tensor_parallel_size: 2 | yes | weights shard across both cards; each card keeps more room for KV. |
| a 2nd worker + pipeline_parallel_size: 2 | yes | each stage holds half the layers. Costs one network hop per token. |
| nothing — switched kv_cache_dtype to "fp8" | yes, 2× | halves KV bytes per token. Free on modern cards. |
Two ceilings apply, checked in order. First the model's native limit (max_position_embeddings, 32,768 for the Qwen2.5 line): vLLM refuses to boot past it, and going beyond needs YaRN rope scaling, which trades quality for length. Then the KV pool from the equation above.
Foot-guns
accelerator_typemust stayNone. Naming a GPU class the fleet doesn't label leaves replicas pending forever with no error anywhere. The number one silent failure.- No
api_keyinengine_kwargs. It crashes engine startup on the pinned vLLM version, and auth already lives at the platform edge. max_replicas≤cluster.num_workers. Each replica needs its own worker; a replica without one sits pending. Keepmin_replicasat 1 or higher — scale-to-zero isn't supported, and a cold start on a big model takes minutes anyway.- Tensor parallelism never crosses machines.TP is for GPUs inside one worker (PCIe/NVLink). Splitting across separate machines is pipeline parallelism's job. Stick to TP of 1, 2, 4, or 8 — it must divide the model's attention-head count, so 3 fails even though it's a valid integer.
enforce_eager=Trueis a debug tool. It saves a few hundred MB and gives readable stack traces at a 10–20% speed cost. Useful while chasing a boot OOM; turn it off for production. (Small sidecar engines like embedders keep it on deliberately.)
max_model_len, raise the context budget in the dashboard playground's chat settings too. The chat clamps its budget to the old server limit the first time it hits it, and it can't detect the new ceiling on its own.Every knob above appears tuned, with comments, in the examples — adapting one is faster than composing a config from scratch.