Training overview

Train across many GPUs

Keep the training loop you already wrote. Wrap it in one function, and it runs across as many GPUs as you want, in one rack or five buildings.

ResonTech trains with federated learning: instead of moving your whole dataset to one big machine, each GPU worker trains on its own part of the data, and only the trained weights travel. Several mid-range GPUs working together finish sooner and cost less than one flagship working alone.

Smaller shards of your dataset go to the GPU workers

1. 1

   Your dataset is split into shards

   One zip per worker: the number of shards is the number of workers. See Dataset format & sharding.
2. 2

   Each worker trains on its own shard

   Your training function runs for the round’s epochs on that worker’s GPUs. Raw data never leaves the worker.
3. 3

   The central server merges the weights

   Every worker sends back the weights it trained. The central server averages them, weighted by how many samples each worker trained on (FedAvg).
4. 4

   The merged model goes back for the next round

   Each worker starts the next round from the merged weights. When the last round ends, the final model and the run’s files land in your bucket.

Why go wide

Measured on one run of a 63M-parameter segmentation model, with the same final result on both setups:

The full comparison, and what it does and doesn’t show: Training benchmarks.

Your loop, moved once

You keep your model, your data loader and your training step. They move inside one function the platform calls once per round: it receives the round’s global weights and returns the weights it trained. Porting a recipe is typically 50–100 lines.

model = TinyCNN(num_classes=10)

for _ in range(EPOCHS):
    train_one_epoch(model, DATA_DIR)

torch.save(model.state_dict(), "out.pt")

def fl_train_model(payload: dict, out_dir: str, logger=None) -> dict:
    env = payload["env"]                          # your hyperparameters
    model = FLModelWrapper()                      # your model, wrapped
    model.set_weights(payload["initial_weights"]) # this round's global model

    for _ in range(int(env["EPOCHS"])):
        train_one_epoch(model, payload["dataset"]["data_root"])  # this worker's shard

    return {
        "weights": model.get_weights(),           # sent to the central server
        "samples": len(dataset),                  # weights the FedAvg average
    }

The full walk-through: Port a centralized recipe to FL. The function’s name is set in the job config (fl_train_model by default).

Accuracy is preserved

Splitting a dataset across workers and averaging their updates costs some accuracy. The settings the examples ship with recover most of it. Measured on geo-imagery segmentation:

Where these come from, and the tuning lessons behind them: How FL works on ResonTech.

Where to next