Inference overview

Paste a model. Get an endpoint

OpenAI-compatible serving, on our GPUs or yours. Any model from Hugging Face, or any Python you write. Minutes after you submit, you hold a URL and an API key; every OpenAI client works with a base-URL swap.

The endpoint stays up until you stop it, and you pay for the GPUs it holds. There are no token, request or traffic fees.

Two ways in, one endpoint out

Open-weights models from Hugging Face

1. 1

   Paste from Hugging Face

   Any Hugging Face model id goes in model_source. Every vLLM engine option is yours to set.
2. 2

   Submit from the SDK or the console

   Two files, serve_module.py and inference.yaml, from Python or the inference wizard in the dashboard.
3. 3

   Prompt it

   Point any OpenAI client at the endpoint with your API key. Live stats per GPU are in the dashboard.

from ray.serve.llm import LLMConfig, build_openai_app

llm_config = LLMConfig(
    model_loading_config=dict(
        model_id="assistant",                 # the name clients call
        model_source="Qwen/Qwen3-8B-AWQ",     # any Hugging Face repo id
    ),
    accelerator_type=None,                    # always None on this platform
    engine_kwargs=dict(max_model_len=16384),
)

app = build_openai_app({"llm_configs": [llm_config]})

from openai import OpenAI

client = OpenAI(
    base_url=f"{ENDPOINT}/v1",
    api_key=PREDICT_KEY,
    default_headers={"X-API-Key": PREDICT_KEY},
)
reply = client.chat.completions.create(
    model="assistant",
    messages=[{"role": "user", "content": "Hello!"}],
)

The full contract: Serve vLLM models and LLMConfig options.

Anything else is one class

For computer vision, speech, diffusion or your own research code, write one Python class with a predict(data: bytes) → dict. The platform owns the HTTP, the auth and the scaling.

class ImageClassifier:
    def __init__(self, top_k: int = 5):          # args come from inference.yaml
        self.model = load_model().eval().cuda()
        self.top_k = top_k

    def predict(self, data: bytes) -> dict:      # the request body in, JSON out
        image = Image.open(io.BytesIO(data))
        scores, ids = self.model(preprocess(image).cuda()).softmax(-1).topk(self.top_k)
        return {"predictions": to_records(ids, scores)}

    # optional: load_weights(path), warmup(), health()

The full contract: Serve custom models and serve.py.

Add a GPU, double the output

When each worker holds its own copy of the model, more workers means more answers at once. Same model, same code: the second GPU is one setting at launch. Measured on a 7B coding model:

You choose the number of workers when you launch. A custom runtime can also add or remove workers while it runs (Scale workers on its page); a vLLM runtime is relaunched with the new count. All the measurements: Inference benchmarks.

Where to next