The shape of a vLLM job
A vLLM job is two files. scripts/serve_module.py declares which model to serve and how (context window, memory budget, batching). inference.yaml tells the platform what hardware to reserve for it. You never write inference code: vLLM runs the model, and the deployed runtime speaks the OpenAI protocol natively.
Weights come from HuggingFace Hub on first boot. Nothing to upload beyond the two files; a 7B model is serving traffic minutes after submit.
The OpenAI surface
Every OpenAI-compatible client works unmodified. Set base_url to <endpoint>/v1 and pass your predict key. The routes you get depend on what the module serves:
| Route | What it does |
|---|---|
| POST /v1/chat/completions | chat — streaming, tool calls, vision content parts |
| POST /v1/completions | classic completions |
| GET /v1/models | lists every model_id the runtime serves; doubles as a readiness check |
| POST /v1/embeddings | embedding engines (RAG retrieval) |
| POST /v1/score | reranker engines |
One runtime can serve several models at once. Each engine in the module gets its own model_id, and clients pick one with the standard model= field. That is how a chat + guard + embeddings + reranker bundle fits behind a single URL.
Two files, two jobs
The split matters because each file answers a different question:
| File | Answers | Details |
|---|---|---|
| inference.yaml | what hardware to reserve — how many workers, how many GPUs each, how much free VRAM | inference.yaml |
| serve_module.py | what to run on it — model, context window, memory budget, batching, replicas | serve_module.py |
tensor_parallel_size / pipeline_parallel_size), the yaml's vllm: block must declare the same numbers. The platform reserves hardware from the yaml, and a mismatched pair is rejected at submit rather than failing minutes later. Most jobs keep both at 1 and never think about this.What you don't manage
- Auth. The platform edge validates
X-API-Keybefore anything reaches your model. Don't configure keys inside the module. - The software stack. Jobs run a prebuilt image with Ray, vLLM, and torch pinned as one tested set. Leave
requirements:empty. - Networking. Workers join the job over an encrypted private network; only the endpoint URL is public.
- Observability. Engine metrics (tokens/s, KV usage, queue depth) feed the dashboard automatically.
Next: serve_module.py for the Python side, or jump straight to a runnable example.