The full file
The yaml is the platform's side of the deal: it describes the hardware your job needs so the scheduler can pick matching workers. Runtime behavior (which model, context, batching) stays in serve_module.py. A typical single-GPU job:
Field reference
| Field | Required? | What it does |
|---|---|---|
| engine | yes | must be vllm_openai. Omitting it selects the custom /predict contract instead. |
| model.module | yes | the Python module under scripts/ (no .py) that exports app. Checked against your uploaded files at submit. |
| vllm.tensor_parallel_size | defaults to 1 | GPUs jointly holding one copy of the weights. All on one worker. |
| vllm.pipeline_parallel_size | defaults to 1 | layer stages, each on its own worker. One replica consumes this many workers. |
| cluster.num_workers | yes | worker machines to reserve. With pipeline parallelism it must be a whole multiple of pipeline_parallel_size. |
| cluster.device | yes | "gpu" (or "auto"). vLLM needs CUDA; "cpu" is rejected. |
| cluster.gpus_per_worker | defaults to 1 | GPUs each worker must expose. Must be ≥ tensor_parallel_size. |
| cluster.cpus_per_worker | defaults to 8 | CPU cores each worker must expose. |
| cluster.min_vram_gb_per_worker | defaults to 8 | free VRAM each selected worker must contribute. The single most important sizing field; see below. |
| cluster.mode | advisory | a dashboard label (data_parallel / tensor_parallel / pipeline_parallel). Never forwarded to vLLM. |
| cluster.max_ongoing_requests | advisory | dashboard accounting only. The real concurrency cap lives in the module's deployment_config. |
| requirements | leave empty | the prebuilt image ships Ray + vLLM + torch as one tested set. Adding ray or vllm lines is rejected at submit; only genuine extras belong here. |
model.class_name and model.init_args belong to the custom /predict contract. In vLLM mode they are ignored, so leave them out.Sizing min_vram_gb_per_worker
The scheduler filters workers by this number, so set it from what your model actually needs: weights + KV cache + about 1 GB of overhead, rounded up. Weights for a 4-bit AWQ checkpoint are roughly 0.55 bytes × parameter count (an 8B model ≈ 5.5 GB); fp16 is 2 bytes × parameters. The examples spell the arithmetic out in a comment next to the field, per job.
max_model_len in the module are the two numbers to re-derive. The napkin math lives in LLMConfig options.With tensor_parallel_size above 1 the check is per GPU: every card on the worker needs min_vram_gb_per_worker ÷ gpus_per_worker free, because vLLM shards evenly and the smallest card gates the whole group.
The sync rule
The vllm: block exists because the scheduler reads only the yaml. It cannot execute your Python, yet the parallel sizes decide which hardware qualifies: TP=2 needs a two-GPU worker, PP=2 needs two workers per replica. So the two values are declared here and enforced in the module, and submit fails fast if they disagree:
Both are whole numbers ≥ 1; they count physical GPUs. Most jobs run 1/1 and can omit the block entirely.