The examples repository
The examples/inference_jobs/vllm repository holds complete, submittable jobs for every major LLM shape: chat assistants, coding models, vision-language document readers, speech recognition, embedding + reranker RAG bundles, moderation guards, and multi-LoRA serving. The models are general-purpose tools, not tied to any sector; pick by capability and VRAM budget.
Every folder has the same layout:
inference.yaml
hardware declaration, with the VRAM arithmetic in a comment
scripts/serve_module.py
the model config — tuned, commented, ready to edit
submit.py
deploys the job via the SDK
predict.py
a working client for that specific job — chat loop, vision call, or RAG round trip
Folder names end in _min_<x>gb_vram: the minimum total VRAM the job needs. How to read that number and what each GPU tier fits: Hardware requirements.
Run any of them
Or paste scripts/serve_module.py + inference.yamlinto the dashboard's inference wizard. Same files, same result.
A sampler
Two examples are documented in detail on the next pages. A few more from the index, to show the range:
| Folder | What it serves | Min VRAM |
|---|---|---|
| Qwen3-8B-AWQ_guarded_rag_chat | chat + moderation guard + embeddings + reranker, four engines on one GPU | 22 GB |
| Qwen2.5-VL-32B-AWQ (vision) | document vision: hard scans, handwriting, dense tables → JSON | 36 GB |
| Qwen2.5-Coder-7B_data_parallel | coding model, two replicas for throughput | 6 GB |
| Mistral-Small-24B_chat | general chat assistant, single 24 GB card | 16 GB |
| whisper-large-v3_asr | speech-to-text via /v1/audio/transcriptions | 6 GB |
| Qwen2.5-7B_multi_lora | one base model + N LoRA fine-tunes, selected per request | 10 GB |
- Coding folders ship tool-calling enabled, so Aider / Cline / Continue work against them directly.
- RAG bundles put chat + embeddings + reranker behind one endpoint; the
predict.pyin each wires a full retrieve-then-answer round trip. requirements:is empty in every folder on purpose: vLLM jobs run a prebuilt image that owns the stack.