LLMConfig options · Docs

The VRAM equation

Every out-of-memory failure and every "why is my context so small" question comes down to one identity that vLLM solves at boot:

gpu_memory_utilization × VRAM  =  weights + activations + KV cache

Weights are fixed by the model and its precision. The KV cache (where conversation context lives) gets whatever remains. If the remainder is too small for your max_model_len, vLLM refuses to boot and says so. That error is your signal to lower the context, pick a quantized checkpoint, or move to a bigger card.

note

vLLM claims its whole memory budget up front. nvidia-smi showing a nearly full GPU right after boot is pre-allocation, not a leak.

The knobs that matter

Kwarg	Set it to	Why
max_model_len	an explicit value (4096–8192 on 12 GB cards)	context window per request, and the single biggest KV lever. Models with 128k native context try to reserve KV for all of it and refuse to boot on small cards.
gpu_memory_utilization	0.90 (single engine)	the share of each GPU vLLM claims. With several engines on one GPU, each engine's share, summing to ≤ ~0.85.
kv_cache_dtype	"fp8"	halves KV memory, so roughly doubles usable context and batch. Small quality loss for chat. The first thing to try when context-starved.
max_num_seqs	8–16 on 12 GB, 32–64 on 24 GB	how many requests batch together inside the engine. More throughput, more KV pressure.
enable_prefix_caching	True	chat clients resend the whole history every turn; this makes turn N re-process only the new tail.
enable_chunked_prefill	True	slices a giant prompt into chunks so it doesn't freeze everyone else's stream.
dtype	"auto"	follows the checkpoint's precision. Leave it.
quantization	don't set it	load a pre-quantized checkpoint (e.g. an official AWQ repo) and vLLM detects the method itself. 4-bit AWQ shrinks weights to ~35% of fp16.

And one from deployment_config rather than engine_kwargs: max_ongoing_requests. Always set it (the examples use 16–32). The framework default of 5 caps concurrency below what the engine can batch, and throughput dies quietly.

Napkin math for sizing

Weights, by precision:

YAML

fp16/bf16:    params × 2 bytes       3B → ~6 GB    7B → ~15 GB   8B → ~16 GB
AWQ/GPTQ-4b:  params × ~0.55 bytes   7B → ~4.5 GB  14B → ~9 GB   32B → ~19 GB

KV cache per token varies a lot between models (grouped-query attention is the difference). Reference points at fp16, halve them with fp8 KV:

Model	KV per token (fp16)	4 GB of KV buys
Qwen2.5-3B	~36 KB	~110k tokens
Qwen2.5-7B	~56 KB	~70k tokens
Llama-3.1-8B	~128 KB	~31k tokens

"Tokens" here counts all concurrent sequences together: 16 parallel requests averaging 2k tokens each hold 32k tokens in cache. Worked example, Qwen2.5-3B fp16 on a 12 GB card: 0.90 × 12 = 10.8 GB budget, minus 6.2 GB weights and ~0.7 GB activations, leaves ~3.9 GB of KV ≈ 108k tokens. max_num_seqs=16 with max_model_len=8192 schedules comfortably.

note

Oversubscription is safe. max_num_seqs × max_model_len may exceed the pool; vLLM admits sequences while KV fits and queues the rest. Heavy long-context load degrades to queueing, never to a crash.

Raising max_model_len — which lever helps

The most common question after adding hardware: "I have two GPUs now, can I raise the context?" It depends on how the second GPU is wired in, because the context budget is per replica:

You added…	More context?	Why
a 2nd worker as another replica	no	each replica still has one card's VRAM. You bought throughput, not context.
a 2nd GPU inside the worker + tensor_parallel_size: 2	yes	weights shard across both cards; each card keeps more room for KV.
a 2nd worker + pipeline_parallel_size: 2	yes	each stage holds half the layers. Costs one network hop per token.
nothing — switched kv_cache_dtype to "fp8"	yes, 2×	halves KV bytes per token. Free on modern cards.

Two ceilings apply, checked in order. First the model's native limit (max_position_embeddings, 32,768 for the Qwen2.5 line): vLLM refuses to boot past it, and going beyond needs YaRN rope scaling, which trades quality for length. Then the KV pool from the equation above.

Foot-guns

accelerator_type must stay None. Naming a GPU class the fleet doesn't label leaves replicas pending forever with no error anywhere. The number one silent failure.
No api_key in engine_kwargs. It crashes engine startup on the pinned vLLM version, and auth already lives at the platform edge.
max_replicas ≤ cluster.num_workers. Each replica needs its own worker; a replica without one sits pending. Keep min_replicas at 1 or higher — scale-to-zero isn't supported, and a cold start on a big model takes minutes anyway.
Tensor parallelism never crosses machines. TP is for GPUs inside one worker (PCIe/NVLink). Splitting across separate machines is pipeline parallelism's job. Stick to TP of 1, 2, 4, or 8 — it must divide the model's attention-head count, so 3 fails even though it's a valid integer.
enforce_eager=True is a debug tool. It saves a few hundred MB and gives readable stack traces at a 10–20% speed cost. Useful while chasing a boot OOM; turn it off for production. (Small sidecar engines like embedders keep it on deliberately.)

warn

After redeploying with a bigger max_model_len, raise the context budget in the dashboard playground's chat settings too. The chat clamps its budget to the old server limit the first time it hits it, and it can't detect the new ceiling on its own.

Every knob above appears tuned, with comments, in the examples — adapting one is faster than composing a config from scratch.

Previousinference.yaml Next Examples gallery