Reading the VRAM minimum
Every example folder ends in _min_<x>gb_vram: the minimum total VRAM the job's model(s) need at the heaviest quantization and a short context. The same number, refined, lives in the yaml as cluster.min_vram_gb_per_worker, which is what the scheduler actually filters workers by.
Minimum means it boots. For production headroom (longer context, more concurrent requests) each README lists a recommended figure, usually the next card size up.
What each tier fits
| Tier | VRAM | Typical GPUs | What fits on one card |
|---|---|---|---|
| small | 11–12 GB | RTX 5070 / 4070 / 3060-12G | 3B fp16, or 7B AWQ with fp8 KV and 4–8k context |
| mid | 16–24 GB | RTX 5070 Ti / 4080, L4, A10G | 7B fp16, or 14B AWQ |
| large | 24–48 GB | RTX 3090 / 4090, L40S, A40 | 14B fp16, 32B AWQ, multi-engine bundles |
| frontier | 80 GB+ | A100-80G, H100, H200 | 70B AWQ on one card; bigger with tensor parallelism |
Models too big for one card split across GPUs: tensor parallelism inside one multi-GPU worker, pipeline parallelism across workers. The sizing arithmetic behind these numbers is in LLMConfig options.
Beyond VRAM
VRAM gets the attention, and then a job stalls on one of these:
- CPU: at least 4 cores per GPU. The yaml default of
cpus_per_worker: 8is a safe floor. - RAM:about 2× the GPU's VRAM. Weight loading and tokenization spike host memory during boot.
- Disk: at least 1.5× the largest checkpoint the worker will host, on top of the ~15 GB serving image. A 19 GB AWQ model wants roughly 46 GB free. Checkpoints download on first boot and stay cached.
- Time: first boot downloads the checkpoint from HuggingFace Hub. Minutes for small models, longer for large ones; subsequent boots on the same worker reuse the cache.
Each example README carries this table filled in for its own model, so when you adapt an example, its resource row is the starting point to adjust.