Sign inStart now ›
Inference /Qwen3-8B guarded RAG chat

Qwen3-8B guarded RAG chat

Four engines on one 24 GB GPU: chat, moderation guard, embeddings and a reranker behind one endpoint.

What it is

Folder: Qwen3-8B-AWQ_guarded_rag_chat_min_22gb_vram. A complete user-facing chat stack on a single GPU. Four vLLM engines share the card and the endpoint, each with its own model id:

model_idModelRoute
assistantQwen3-8B (AWQ, ~5.5 GB)POST /v1/chat/completions
guardQwen3Guard-Gen-8B — safety classifierPOST /v1/chat/completions
embedQwen3-Embedding-0.6BPOST /v1/embeddings
rerankQwen3-Reranker-0.6BPOST /v1/score

These are general-purpose tools, not tied to any industry. The bundle exists because a production chat needs more than a chat model: every user message is moderated before it reaches the assistant, answers are grounded in retrieved context, and the output is moderated again before it returns. The included predict.py wires the whole loop:

user msg → guard("user")      → Unsafe? refuse before spending chat tokens
         → embed + rerank     → top-3 context chunks
         → assistant          → grounded answer
         → guard("assistant") → Unsafe? suppress before returning
note
Moderation stays inside your deployment. Nothing is sent to a third-party moderation API, which is the point for privacy-sensitive workloads. The guard is a generative classifier: post the content as a chat message to model="guard" and parse the Safe/Unsafe verdict it emits.

How four engines share one card

Each engine pre-allocates its gpu_memory_utilization share at boot, and the shares must sum to about 0.85. On a 24 GB card:

chat   0.32 → 7.7 GB  (5.5 GB AWQ weights + ~1.7 GB KV ≈ 23k tokens fp8)
guard  0.40 → 9.6 GB  (~8.5 GB fp8-quantized weights + small KV)
embed  0.06 → 1.4 GB
rerank 0.06 → 1.4 GB
                sum = 0.84

To rebalance (say, more KV for the chat model), edit the four values in serve_module.py and keep the sum ≤ 0.85. The sidecars run enforce_eager=True to skip CUDA-graph VRAM they don't need.

Submit and call

Shell
cd Qwen3-8B-AWQ_guarded_rag_chat_min_22gb_vram
cp .env.example .env       # RESON_API_KEY + S3 keys
python submit.py           # prints endpoint + predict key — save both
python predict.py "When are production deployments frozen?"

Calling it from your own code is standard OpenAI:

client.py
from openai import OpenAI

client = OpenAI(
    base_url=f"{ENDPOINT}/v1",
    api_key=KEY,
    default_headers={"X-API-Key": KEY},
)

# 1. moderate the user message
verdict = client.chat.completions.create(
    model="guard", temperature=0.0, max_tokens=64,
    messages=[{"role": "user", "content": f"[user message to moderate]\n{question}"}],
)

# 2. retrieve: embed the question and your chunks, rerank the candidates
emb = client.embeddings.create(model="embed", input=[question, *chunks])

# 3. answer with context
answer = client.chat.completions.create(
    model="assistant",
    messages=[
        {"role": "system", "content": f"Answer from this context:\n{top_chunks}"},
        {"role": "user", "content": question},
    ],
)

Hardware

VRAM min → recCPURAMFree disk
22 → 24 GB4 cores48 GB~46 GB (image + checkpoint downloads)

Suggested GPUs: RTX 3090 / 4090, L4, A10G (24 GB). A 32 GB card (RTX 5090) relaxes every budget. The yaml declares min_vram_gb_per_worker: 22 so the scheduler only picks workers that fit. General tier guidance: Choosing hardware.

PreviousExamples galleryNext Qwen2.5-VL-32B vision