The class
Two methods are required. Three more are optional, discovered by name; no base class, no decorator, no platform import.
scripts/my_serve.py
The smallest complete predictor is five lines:
What `data` is
The platform normalizes every request body into raw bytes before your method sees it, keyed on the request's Content-Type:
| Caller sends | Content-Type | predict() receives |
|---|---|---|
| raw bytes (the default) | image/*, audio/*, video/*, application/octet-stream, or empty | the body verbatim |
| {"url": "https://…"} | application/json | the fetched bytes (presigned S3 URLs work; 200 MB cap) |
| {"data_b64": "…"} | application/json | the decoded bytes |
| file upload | multipart/form-data | the first file field's bytes |
!
application/json is an envelope, not a passthrough: any field other than url / data_b64 is rejected. To send structured JSON (payload plus parameters in one request), POST the JSON bytes as application/octet-stream and call json.loads(data) in your predictor. The SDK's predict_json() does exactly this; details in Calling /predict.A robust predictor tries json.loads first and falls back to treating the bytes as the raw payload. The SAM 2.1 example does this: JSON with prompts, or a bare image.
Returns and errors
- Return only JSON-serializable values: ints, floats, strings, lists, dicts. Tensors and numpy scalars cause a 500; convert them. Base64-encode binary outputs (the SAM example returns masks as base64 PNGs).
- Raise
ValueErrorfor bad input; the platform maps it to a 400 with a clean error body. Any other exception becomes a 500. Never return an error dict with a 200 status; raise instead. - The default per-request timeout is 60 seconds. Slow models (diffusion, long TTS) need it raised on the job; a patient client can't extend a server timeout.
i
Concurrency: up to
cluster.max_ongoing_requests calls can be inside your predict at once. Either make it thread-safe (model in inference mode, no shared mutable state) or keep the cap low. Exceptions in __init__ or load_weights crash-loop the replica, so fail fast and loudly there.