Choosing hardware for inference

What you choose

An inference job runs on GPU workers you pick and one central server the platform picks.

1. How much memory the weights take

weights (GB) ≈ parameters (billions) × bytes per parameter

The exact figure is the size of the model’s weight files on Hugging Face. For Mixture-of-Experts models count every parameter, not the active ones: GLM-4.5-Air has 106B parameters and uses 12B per token, so it needs the memory of a 106B model and runs at the speed of a 12B one.

2. Which GPU fits

Keep the weights at or under ~75% of the card. The rest holds the KV cache and runtime overhead.

min_vram_gb_per_worker ≈ weights ÷ 0.75

When the weights don’t fit one card

Example: GLM-4.5-Air AWQ (60 GB) runs on 4 × 24 GB with TP=4, or on one 96 GB card. The platform checks GPU memory per card, so a 4-GPU worker with min_vram_gb_per_worker: 84 is checked as 21 GB per card. For TP ≥ 4 set require_nvlink: true.

3. Context and users at once

The KV cache holds the attention state of every token in flight, across all requests. vLLM frees a request’s blocks when its answer completes. When the cache is full, new requests wait in a queue: an undersized card means waiting, not crashing.

kv_cache_dtype="fp8" halves these and is on in the templates. “Tokens” sums every request in flight: 16 chats × 2k tokens = 32k.

budget  = 0.90 × 12 GB          = 10.8 GB
KV      = 10.8 − 6.2 (weights) − 0.7 (runtime) ≈ 3.9 GB
tokens  ≈ 3.9 GB ÷ 36 KB       ≈ 108k   → 16 users at 8k context fit

Long conversations

A long conversation re-sends its history every turn, so its cost is memory per user, growing with context.

KV per user       = context tokens × KV per token
users per replica = (0.9 × VRAM − weights − ~1 GB) ÷ KV per user

Long context needs VRAM, not compute. What helps, in order:

• ›A bigger card, or TP across one worker’s cards (it pools their memory). More replicas only add users at the same context.
• ›A long-context model: hybrid-attention families (Qwen3.5, Qwen3-Next) cache only a few layers.
• ›Caps that match the card: max_model_len per request and max_num_seqs for concurrency, set from the tables above.
• ›Shorter history on the client: summarise or drop old turns.

4. How many replicas

Throughput grows with replicas. Measured on one RTX 5070 Ti 16 GB per replica:

Plan for 70% of one replica’s measured ceiling:

num_workers = ceil(your load ÷ (0.7 × one replica's ceiling))

Start with one replica and add workers when the queue shows. Replicas of one job can run on different GPU models; each only has to fit its own card. All measurements: Inference benchmarks.

5. Disk, RAM and CPU per worker

6. The central server

The platform picks it at submit, and every request passes through it. It runs no model and needs no GPU.

A server reserved for you is used first; otherwise the platform picks a shared one that meets your minimums. If none does, the launch stops and the wizard offers to order one.

• ›Shared is fine for tests and most endpoints under light traffic.
• ›Dedicated when the endpoint serves many users or long streams, so no other job shares its network.
• ›Reserved for production endpoints: only your jobs run on it, and they use it automatically.

GPU tiers and mixing them

Datacenter cards (A-, L-, H-series, RTX PRO 6000) come with ECC; consumer cards cost less per GB. Pick datacenter cards for regulated workloads.

By model type

“min → rec”: the minimum at the heaviest quantisation and short context, then the size that leaves room for context and users.

Chat and reasoning

Coding assistants

Coding agents resend the repository context every turn; prefix caching keeps that affordable. One 16 GB replica serves about 8 developers under 5 s to first token.

Vision-language

A 1-megapixel page is about 1,000–3,000 tokens and counts against context like text. The vision encoder stays fp16 even when the language model is 4-bit.

RAG, adapters and speech

Computer vision and image generation (custom models)

For custom models CPU matters as much as VRAM: image decoding runs on the CPU. Give 4–8 cores per GPU.

When load exceeds the plan

Too many requests don’t crash a replica: vLLM reserves the KV cache at start and admits only what fits. The endpoint degrades in a fixed order.

• ›Plan at 70% of the measured ceiling and watch time to first token: it rises first.
• ›Add a replica before the queue does it for you. For vLLM that’s a relaunch with one more worker; custom runtimes can add workers while running.
• ›Give batch work its own job, so a 100k-page extraction doesn’t queue ahead of a chat.

Before you submit

Worked examples