The idea
You upload a small bundle: one inference.yaml plus a scripts/ folder with your Python. The platform finds GPU workers that match your requirements, boots your model on them, and hands you back two things:
- an endpoint URL where your model listens,
- a predict API key, shown once at submit. Save it; every request needs it in the
X-API-Keyheader.
From that point the runtime is a normal HTTP service. Your code, your weights, your endpoint. Stop the job and billing stops with it.
Two runtime types
The engine field in your yaml picks one of two contracts. Everything else about submitting is the same.
| vLLM (engine: vllm_openai) | Custom (/predict) | |
|---|---|---|
| Built for | LLMs: chat, coding assistants, vision-language, embeddings, rerankers | Any model: segmentation, TTS, image generation, classifiers, your research code |
| You write | a serve_module.py that declares which model to serve and how | a Python class with predict(data: bytes) → dict |
| Clients call | POST /v1/chat/completions and the rest of the OpenAI protocol | POST /predict with raw bytes or JSON |
| Works with | openai-python, curl, LangChain, Aider, Cline, Continue. Unmodified. | any HTTP client, or the SDK's InferenceClient |
| Weights | pulled from HuggingFace Hub at boot | from HF Hub, or a weights file you upload |
Serving an open LLM? Use vLLM mode and skip writing inference code entirely. Serving anything else, or your own architecture? Use the custom contract. Start with Serve vLLM models or Serve custom models.
Submit a job
- 1
Get your bundle
Fastest path: copy a folder from the examples repository (vLLM examples, custom examples) and edit it. Each folder is complete: yaml, scripts, a submit script, a test client. - 2
Submit from Python
One call uploads the bundle and starts provisioning. - 3
Save the endpoint and key
The submit response prints both. The key is never shown again; you can rotate it later from the dashboard.
wait_until_ready() polls until the runtime is RUNNING. You can watch the same progress on the dashboard, worker by worker.Full parameter reference: Submit inference via SDK. Prefer the browser? The dashboard's inference wizard accepts the same two files.
Call it
A vLLM runtime speaks the OpenAI protocol. Point any OpenAI client at <endpoint>/v1:
A custom runtime takes bytes on /predictand returns your class's JSON:
Both runtime types also expose GET /health (no auth) as a readiness check, and the dashboard playground lets you test either kind from the browser.