Sign inStart now ›
Inference /How it works

How it works

One serve_module.py and one inference.yaml become an OpenAI-compatible endpoint.

The shape of a vLLM job

A vLLM job is two files. scripts/serve_module.py declares which model to serve and how (context window, memory budget, batching). inference.yaml tells the platform what hardware to reserve for it. You never write inference code: vLLM runs the model, and the deployed runtime speaks the OpenAI protocol natively.

Weights come from the Hugging Face Hub on first boot. Nothing to upload beyond the two files; a 7B model is serving traffic minutes after submit.

The OpenAI surface

Every OpenAI-compatible client works unmodified. Set base_url to <endpoint>/v1 and pass your predict key. The routes you get depend on what the module serves:

RouteWhat it does
POST /v1/chat/completionsChat, with streaming, tool calls and images
POST /v1/completionsPlain text completion
GET /v1/modelsLists every model_id the runtime serves; also a readiness check
POST /v1/embeddingsEmbedding models, for RAG retrieval
POST /v1/scoreRerankers
any OpenAI client
from openai import OpenAI

client = OpenAI(
    base_url=f"{ENDPOINT}/v1",
    api_key=PREDICT_KEY,
    default_headers={"X-API-Key": PREDICT_KEY},
)
print([m.id for m in client.models.list()])

One runtime can serve several models at once. Each engine in the module gets its own model_id, and clients pick one with the standard model= field. That is how a chat + guard + embeddings + reranker bundle fits behind a single URL.

Two files, two jobs

The split matters because each file answers a different question:

FileAnswersDetails
inference.yamlWhat hardware to reserve: how many workers, GPUs per worker, free VRAMinference.yaml
serve_module.pyWhat to run on it: model, context window, memory budget, batching, replicasserve_module.py
note
One rule ties them together: if the module splits the model across GPUs (tensor_parallel_size / pipeline_parallel_size), the yaml's vllm: block must declare the same numbers. The platform reserves hardware from the yaml, and a mismatched pair is rejected at submit rather than failing minutes later. Most jobs keep both at 1 and never think about this.

What you don't manage

  • Auth. The platform checks X-API-Key before a request reaches your model. Don't configure keys inside the module.
  • The software stack. Jobs run a prebuilt image with Ray, vLLM, and torch pinned as one tested set. Leave requirements: empty.
  • Networking. Workers join the job over an encrypted private network; only the endpoint URL is public.
  • Observability. Engine metrics (tokens/s, KV usage, queue depth) feed the dashboard automatically.

Next: serve_module.py for the Python side, or jump straight to a runnable example.

PreviousBenchmarksNext serve_module.py