What it is
Folder: Qwen3-8B-AWQ_guarded_rag_chat_min_22gb_vram. A complete user-facing chat stack on a single GPU. Four vLLM engines share the card and the endpoint, each with its own model id:
| model_id | Model | Route |
|---|---|---|
| assistant | Qwen3-8B (AWQ, ~5.5 GB) | POST /v1/chat/completions |
| guard | Qwen3Guard-Gen-8B — safety classifier | POST /v1/chat/completions |
| embed | Qwen3-Embedding-0.6B | POST /v1/embeddings |
| rerank | Qwen3-Reranker-0.6B | POST /v1/score |
These are general-purpose tools, not tied to any industry. The bundle exists because a production chat needs more than a chat model: every user message is moderated before it reaches the assistant, answers are grounded in retrieved context, and the output is moderated again before it returns. The included predict.py wires the whole loop:
model="guard" and parse the Safe/Unsafe verdict it emits.How four engines share one card
Each engine pre-allocates its gpu_memory_utilization share at boot, and the shares must sum to about 0.85. On a 24 GB card:
To rebalance (say, more KV for the chat model), edit the four values in serve_module.py and keep the sum ≤ 0.85. The sidecars run enforce_eager=Trueto skip CUDA-graph VRAM they don't need.
Submit and call
Calling it from your own code is standard OpenAI:
Hardware
| VRAM min → rec | CPU | RAM | Free disk |
|---|---|---|---|
| 22 → 24 GB | 4 cores | 48 GB | ~46 GB (image + checkpoint downloads) |
Suggested GPUs: RTX 3090 / 4090, L4, A10G (24 GB). A 32 GB card (RTX 5090) relaxes every budget. The yaml declares min_vram_gb_per_worker: 22 so the scheduler only picks workers that fit. General tier guidance: Hardware requirements.