Sign inStart now ›
Inference /Examples gallery

Examples

Ready-to-run vLLM jobs for chat, coding, vision, RAG and speech. Each is a folder you submit as it is.

The examples repository

The examples/inference_jobs/vllm repository holds complete, submittable jobs for every major LLM shape: chat assistants, coding models, vision-language document readers, speech recognition, embedding + reranker RAG bundles, moderation guards, and multi-LoRA serving. The models are general-purpose tools, not tied to any sector; pick by capability and VRAM budget.

Every folder ships the same four files:

<model>_<use>_min_<x>gb_vram/

  • inference.yamlThe hardware: how many workers, GPUs per worker and free VRAM. A comment shows the VRAM arithmetic.
  • scripts/serve_module.pyThe model and its engine settings, tuned and commented. This is the file you edit.
  • submit.pyDeploys the job with the SDK and prints the endpoint and the API key.
  • predict.pyA working client for this job: a chat loop, a vision call or a RAG round trip.

Folder names end in _min_<x>gb_vram: the minimum total VRAM the job needs. How to read that number and what each GPU tier fits: Choosing hardware.

Run any of them

Shell
cd <job-folder>
cp .env.example .env     # RESON_API_KEY (rsk_…) + S3 keys
python submit.py         # uploads yaml + scripts/, prints endpoint + predict key
python predict.py        # job-specific smoke test

Or paste scripts/serve_module.py + inference.yaml into the dashboard's inference wizard. Same files, same result.

note
First boot pulls the model checkpoint from HuggingFace Hub. A 5 GB AWQ model takes minutes on a cold worker; a 19 GB one proportionally longer. The submit script waits and reports when the runtime is RUNNING.

A sampler

Two examples are documented in detail on the next pages. A few more from the index, to show the range:

FolderWhat it servesMin VRAM
Qwen3-8B-AWQ_guarded_rag_chatchat + moderation guard + embeddings + reranker, four engines on one GPU22 GB
Qwen2.5-VL-32B-AWQ (vision)document vision: hard scans, handwriting, dense tables → JSON36 GB
Qwen2.5-Coder-7B_data_parallelcoding model, two replicas for throughput6 GB
Mistral-Small-24B_chatgeneral chat assistant, single 24 GB card16 GB
whisper-large-v3_asrspeech-to-text via /v1/audio/transcriptions6 GB
Qwen2.5-7B_multi_loraone base model + N LoRA fine-tunes, selected per request10 GB
  • Coding folders ship tool-calling enabled, so Aider / Cline / Continue work against them directly.
  • RAG bundles put chat + embeddings + reranker behind one endpoint; the predict.py in each wires a full retrieve-then-answer round trip.
  • requirements: is empty in every folder on purpose: vLLM jobs run a prebuilt image that owns the stack.
PreviousLLMConfig optionsNext Qwen3-8B guarded RAG chat