DocsPython SDKSDK — Storage
Python SDK

SDK — Storage

How rt_submit writes to your S3 bucket, the upload mechanics, and how to read or re-use the workspace from Python.

Overview

The SDK writes directly to your per-user Garage bucket via boto3. Nothing stages on local disk and no bytes flow through the REST API.

Bucket Layout

After a successful rt_submit, your bucket contains:

<slug> is your name arg lower-cased with spaces and non-alphanumeric characters (except - and _) stripped.

Upload Mechanics

rt_submit performs these writes in order:

  • Scripts — three PutObject calls (small, text).
  • Configs — three PutObject calls (small, JSON).
  • Requirements — one PutObject call.
  • Shards — streaming upload_file per zip.boto3's built-in TransferManager auto-switches to multipart for files ≥ 10 MB, using 50 MB parts and 5 concurrent uploads. Matches the web UI's chunking.
  • Checkpoint (optional) — streaming upload_file.
  • Submit — a single POST /api/tasks with the five folder paths.
i
Bytes never pass through the API; boto3 talks straight to Garage with your credentials.

Path Conventions

The backend expects API-style paths with a leading /:

FieldExample value
trainingScriptsFolderPath/jobs/my-run/scripts
trainingConfigFolderPath/jobs/my-run/configs
trainingRequirementsFolderPath/jobs/my-run/requirements
shardsDirPath/jobs/my-run/shards
modelFolderPath (optional)/jobs/my-run/model

The SDK constructs these automatically — you never build them by hand.

Reading Your Bucket From Python

sdk.storage is the raw boto3-backed client. It's intentionally small:

The client also exposes put_bytes(key, body, content_type) and upload_file(local_path, key, content_type) if you need to write extra files manually (e.g. a custom sharding_map.json).

Quota

Your bucket has a hard storage quota (set when you provisioned it). GET /api/users/storage/usage reports current usage; the web UI's storage meter reads from the same endpoint. Writes that would exceed the quota are rejected by Garage with HTTP 403.

Rotate or resize via Profile → Storage in the web UI.

Downloading Results

Recommended: web UI

Open job.dashboard_url and use the file browser. It hands out 1-hour presigned GET URLs, supports Monaco for text previews, and knows how to show training logs.

From Python

Or use the plain boto3 client under the hood:

Re-Running Against an Existing Workspace

If you already uploaded to jobs/my-run/ and only want to re-submit (same scripts, same shards, maybe different estimated_memory_mb), build the request directly:

This is what the UI's "Select existing workspace" flow does.

PreviousCustom ClassesNextJob Lifecycle