DocsInferenceServe vLLM modelsserve_module.py
Inference · Serve vLLM models

serve_module.py

The Python file that declares your model: one LLMConfig per engine, exported as app.

A minimal module

The file builds a serving app from one or more LLMConfig objects and exports it as app. That export name is the contract; the platform imports <module>:app and runs it.

scripts/serve_module.py

This serves one model. Clients see it as model="assistant" on /v1/chat/completions.

The four blocks

BlockDecides
model_loading_configmodel_id is the public name clients use; model_source is where the weights live (a HF repo id). Two fields so you can rename without re-pointing clients at a new repo.
engine_kwargseverything vLLM: context window, memory budget, batching, quantization, parallelism. The knobs that decide whether your model boots. Covered in LLMConfig options.
deployment_configreplica count and per-replica concurrency. max_ongoing_requests should be set on every job; the framework default of 5 starves an LLM that batches 16 sequences.
accelerator_typekeep it None. Naming a GPU class (like "L4") on hardware without that label leaves replicas pending forever, with no error to tell you why.

Several models, one endpoint

Pass more than one config and each becomes its own engine with its own model_id, all behind the same URL:

Engines sharing one GPU must split its memory: each config's gpu_memory_utilizationis that engine's share, claimed in full at boot, and the shares must sum to about 0.85 or the last engine fails to allocate. The guarded RAG chat example runs four engines on one 24 GB card with shares of 0.32 / 0.40 / 0.06 / 0.06.

Sidecar engines take a task= kwarg: task="embed" serves /v1/embeddings, task="score" serves /v1/score for rerankers.

Three rules that save you a failed deploy

  • Export app, keep the module name in sync. The yaml's model.module must match the filename under scripts/. Both are checked at submit.
  • Mirror TP/PP into the yaml. If you set tensor_parallel_size or pipeline_parallel_sizeabove 1, declare the same values in the yaml's vllm: block. A drifted pair is rejected at submit.
  • No api_key in engine_kwargs. Auth lives at the platform edge. Setting a key string here crashes engine startup on the pinned vLLM version.
!
Gated models (Llama, some Mistral checkpoints) need an HF_TOKEN in the replica environment via runtime_env=dict(env_vars={...}). Without it the download fails minutes into boot with a 401 from HuggingFace.
i
Tool calling for agent clients is two kwargs, but the parser is per model family: Qwen uses enable_auto_tool_choice=True, tool_call_parser="hermes", Mistral uses tool_call_parser="mistral". The examples ship the right one per model.
PreviousHow it worksNextinference.yaml