30-second version
Bring two things: an inference.yaml describing the model and a scripts/ folder with the predictor class. The SDK uploads them to your bucket, the platform boots replicas, you get an HTTPS endpoint + API key.
That's it. The endpoint accepts POST /predict calls with the API key as X-API-Key. See Calling /predict for the client side.
Set up your model folder
Two required files. A third (weights) is optional — many models fetch from HuggingFace Hub at init time instead.
inference.yaml
Three blocks: model (what to run), cluster (how to run it), requirements (pip pins for the worker runtime).
See the inference.yaml reference for every field. The SDK injects sensible defaults for the ones you omit.
scripts/my_serve.py
One Python class with two required methods. The class name + module name above must match what's in this file.
The predictor contract
Every predictor is just __init__ + predict. The wrapper calls them; you never need to know about Ray Serve or the HTTP layer.
__init__(**init_args)— kwargs come straight frommodel.init_argsin your yaml.predict(data: bytes) -> dict— raw body in, JSON-serializable dict out. No HTTP knowledge required.
Optional hooks
The wrapper calls these when they exist. Skip what you don't need — sensible defaults apply.
model.path in the yaml and do not implement load_weights(). Load in __init__ via from_pretrained(model_id) — that's already a "weights load," no extra hook needed.Deploy
One call. The SDK validates, uploads to your bucket, and triggers provisioning.
How people use it
One-off / scripted
Submit a model from a script, hit /predict a few thousand times for an eval, then stop the runtime from the dashboard. Bills accrue while it's up.
Inside an existing pipeline
Submit once, keep the runtime up, treat the endpoint + key as any other vendor credential in your secrets manager. Lifecycle operations (stop, scale workers up/down, hot-reload code, rotate the API key) currently live on the dashboard — the SDK ships these as separate calls once they're stable.
How it works — what gets uploaded
rt_submit_inference() uploads to your bucket under the runtime slug:
The SDK skips hidden files, __pycache__/, and node_modules/ automatically. If your yaml omits cluster.device, mode, gpus_per_worker, or cpus_per_worker, the SDK fills in defaults before upload — your local file is left untouched.
The visual equivalent of this flow is the dashboard wizard's BYO bundle drop — same two files, same validation. Overview: Quick start.
What's next
- Call your model: Calling /predict — file, bytes, URL, JSON, batch.
- Worked examples: vLLM jobs and custom predictor jobs — complete bundles you can submit as-is.
- Full yaml reference: vLLM mode / custom mode.