Submit via SDK: Inference · Docs

30-second version

Bring two things: an inference.yaml describing the model and a scripts/ folder with the predictor class. The SDK uploads them to your bucket, the platform boots replicas, you get an HTTPS endpoint + API key.

Python

from resontech import InferenceConfig, ResonTech, ResonTechConfig

sdk = ResonTech(ResonTechConfig(...))   # see Install the SDK
# No login() — every call signs with the platform_api_key set on the config.

job = sdk.rt_submit_inference(
    name="My Segmentation Model",
    inference=InferenceConfig(
        gpu_models=["rtx_5090", "rtx_4090"],  # ordered preference
        max_price_per_gpu_hour=3.00,          # $ per GPU per hour
    ),
    inference_yaml="./config/inference.yaml",
    scripts_dir="./src/inference",
    model_file="./checkpoints/weights.pt",   # optional
)
job.wait_until_ready()
print(job.endpoint, job.api_key)

That's it. The endpoint accepts POST /predict calls with the API key as X-API-Key. See Calling your model for the client side.

note

Don't have the SDK installed yet? Run through Install the SDK first — it takes about five minutes.

Set up your model folder

Two required files. A third (weights) is optional — many models fetch from HuggingFace Hub at init time instead.

my_model/
├── config/
│   └── inference.yaml          # what to run and where
└── src/
    └── inference/
        ├── my_serve.py         # the predictor class
        └── (any helpers)

inference.yaml

Three blocks: model (what to run), cluster (how to run it), requirements (pip pins for the worker runtime).

inference.yaml

model:
  path: "model/weights.pt"        # optional — omit for HF-Hub-only models
  module: "my_serve"              # the .py file under scripts/
  class_name: "MyPredictor"
  init_args:
    num_classes: 3
    threshold: 0.5

cluster:
  num_workers: 2
  device: "gpu"
  gpus_per_worker: 1
  cpus_per_worker: 4
  mode: "data_parallel"
  max_ongoing_requests: 4

requirements:
  - "torch"
  - "torchvision"
  - "opencv-python-headless"

See the inference.yaml reference for every field. The SDK injects sensible defaults for the ones you omit.

scripts/my_serve.py

One Python class with two required methods. The class name + module name above must match what's in this file.

The predictor contract

Every predictor is just __init__ + predict. The wrapper calls them; you never need to know about Ray Serve or the HTTP layer.

my_serve.py

class MyPredictor:
    def __init__(self, num_classes: int = 3, threshold: float = 0.5):
        # Runs once per replica at startup.
        # Build your model, transforms, anything stateful.
        self.threshold = threshold
        self.model = build_model(num_classes=num_classes).cuda().eval()

    def predict(self, data: bytes) -> dict:
        # Runs per request. data is the raw HTTP body.
        # Return ANY JSON-serializable dict.
        image = decode_image(data)
        mask = self.model(image)
        return {
            "mask_shape": list(mask.shape),
            "mask_b64": encode_png_b64(mask),
        }

__init__(**init_args) — kwargs come straight from model.init_args in your yaml.
predict(data: bytes) -> dict — raw body in, JSON-serializable dict out. No HTTP knowledge required.

Optional hooks

The wrapper calls these when they exist. Skip what you don't need — sensible defaults apply.

Python

class MyPredictor:
    def __init__(self, **kwargs): ...
    def predict(self, data: bytes) -> dict: ...

    def load_weights(self, weights_path: str) -> None:
        # Called once at startup when model.path is set in inference.yaml.
        # weights_path is the absolute path to the file on the replica.
        state = torch.load(weights_path, map_location="cpu")
        self.model.load_state_dict(state, strict=False)

    def warmup(self) -> None:
        # Called once after deploy. Pre-warm caches, JIT, etc.
        with torch.inference_mode():
            self.model(dummy_input)

    def health(self) -> dict:
        # GET /health. Default: {"status": "ok"}.
        return {"status": "ok", "device": str(self.device)}

note

If your weights are on HF Hub (the common LLM case), do not set model.path in the yaml and do not implement load_weights(). Load in __init__ via from_pretrained(model_id) — that's already a "weights load," no extra hook needed.

Deploy

One call. The SDK validates, uploads to your bucket, and triggers provisioning.

Python

from resontech import InferenceConfig

job = sdk.rt_submit_inference(
    name="my-segmenter",
    inference=InferenceConfig(
        # Platform picks the workers — both bounds required.
        # Order is a preference: all of the first model, then the next.
        gpu_models=["rtx_5090", "rtx_4090"],
        max_price_per_gpu_hour=3.00,      # $ PER GPU per hour
        # or pin them yourself: worker_ids=["w_abc", "w_def"]
        visibility="PRIVATE",             # PUBLIC = no API key required
    ),
    inference_yaml="./config/inference.yaml",
    scripts_dir="./src/inference",
    model_file="./checkpoints/weights.pt",  # optional, uploads to model/<basename>
)

# Block until /predict is live. ~30 s to a few minutes depending on
# image size, requirements, and whether weights are remote.
job.wait_until_ready(timeout=30 * 60)

print("URL:    ", job.endpoint)
print("API key:", job.api_key)

warn

The API key is shown once. Save it before the process ends — it's not retrievable from the dashboard later.

How people use it

One-off / scripted

Submit a model from a script, hit /predict a few thousand times for an eval, then stop the runtime from the dashboard. Bills accrue while it's up.

Python

job = sdk.rt_submit_inference(...)
job.wait_until_ready()

client = InferenceClient(url=job.endpoint, api_key=job.api_key)
for input_path in inputs:
    result = client.predict_file(input_path)
    process(result)

# Open the dashboard at job.dashboard_url and stop the runtime when done.

Inside an existing pipeline

Submit once, keep the runtime up, treat the endpoint + key as any other vendor credential in your secrets manager. Lifecycle operations (stop, scale workers up/down, hot-reload code, rotate the API key) currently live on the dashboard — the SDK ships these as separate calls once they're stable.

How it works — what gets uploaded

rt_submit_inference() uploads to your bucket under the runtime slug:

s3://<your-bucket>/inference_jobs/<slug>/
├── inference.yaml          # your yaml (cluster defaults injected if missing)
├── scripts/                # contents of scripts_dir/, skip-filtered
│   ├── my_serve.py
│   └── ...
└── model/                  # optional, when model_file is set
    └── weights.pt

The SDK skips hidden files, __pycache__/, and node_modules/ automatically. If your yaml omits cluster.device, mode, gpus_per_worker, or cpus_per_worker, the SDK fills in defaults before upload — your local file is left untouched.

The visual equivalent of this flow is the dashboard wizard's BYO bundle drop — same two files, same validation. Overview: Quick start.

What's next

Call your model: Calling your model — file, bytes, URL, JSON, batch.
Worked examples: vLLM jobs and custom predictor jobs — complete bundles you can submit as-is.
Full yaml reference: vLLM mode / custom mode.

PreviousSAM 2.1 segmentation Next Overview