Overview
The full FL job catalog lives at github.com/resontech-dev/examples. Twelve job templates: nine fine-tune real HuggingFace models with verifiable published metrics, two are from-scratch production loads (ImageNet ResNet-50, Pythia-1B pretraining), and one is the minimal SDK starter (MNIST CNN) — the clone-and-edit base if you're new.
This page deep-dives job_classify (the canonical reference) and links to the other eleven. Per-job READMEs in the repo link each model's HuggingFace card so every number can be verified.
At a Glance
Common Job Layout
Every example job follows the same directory layout. The SDK and the web wizard expect this shape on submission:
job_xxx/
├── README.md
├── scripts/
│ ├── model_def.py ← project config + make_fl_adapter()
│ ├── {framework}_utils.py ← framework-specific adapter
│ ├── custom_client_executor.py ← shared FL plumbing
│ └── custom_persistor.py ← shared FL plumbing
├── configs/
│ ├── config_fed_client.json
│ └── config_fed_server.json
├── requirements/
│ └── requirements.txt
├── shards/ ← built by build_shards_hf.py
└── build_shards_hf.pycustom_client_executor.py and custom_persistor.py — the FL plumbing works for every task type. Only model_def.py and the framework utils change.Deep Dive — job_classify (ViT-B/16 on Food-101)
job_classify is the canonical reference — the only job where the FL run beats the centralized HuggingFace baseline on both top-1 and val_loss. Repo links: README and RUNS.
Headline numbers
Validated on the full Food-101 validation split (25,250 images) with the HF eval recipe (AutoImageProcessor("google/vit-base-patch16-224-in21k") + per-example mean cross-entropy + top-1 / top-5). Deterministic seed 1337.
Recipe at a glance
Architecture vit_base_patch16_224.augreg_in21k (timm port of HF in21k), head reinit to 101 classes
Dataset ethz/food101 (75,750 train / 25,250 val), preprocess via AutoImageProcessor
Optimizer Adam(lr=2e-4, betas=(0.9, 0.999), eps=1e-8), wd=0
Loss CrossEntropyLoss (no smoothing)
LR schedule linear decay 2e-4 → 0, *continuous across all rounds*
AMP fp16 autocast + GradScaler
Grad clip 1.0
Seed 1337
Batch (train) 348 ← deviation from HF's 128 (32 GB GPUs underused at 128)
Topology 4 workers × 5 rounds × 1 local epoch + FedAvgThe two deviations from the centralized HF recipe
- Continuous LR schedule across rounds — instead of restarting
LambdaLRper round (sawtooth), the lambda computes the LR from the global stepcurrent_round × steps_per_round + local_step. Same shape as HF's single-ramp decay. - Persisted per-client Adam state —
m/vbuffers saved toSTATE_DIR/optimizer_state.ptat end of round, restored at start of next. No more 10-step warmup tax per round.
Everything else — backbone, dataset, optimizer, loss, preprocessing, AMP, grad clip, seed — is byte-for-byte the HF recipe.
job_classify — Selected Runs
Every measured training run is logged in RUNS.md. The progression below is the punchline:
Lessons (run details in RUNS.md)
- Continuous LR + persisted Adam are the only deviations worth keeping. +0.76 pp / −0.03 loss vs naive port. Everything else attempted on top either regressed or was neutral.
- EMA over short FL rounds is a trap. Decay 0.999 over 74-148 steps per round dominates round-start weights and nullifies most local progress. Lower decay (~0.95) or persisting EMA across rounds would fix it.
- Label smoothing inflates measured val_loss. Don't use it if the headline metric is val_loss.
- FedAvg can out-regularize a too-aggressive centralized run. The single-GPU H100 baseline overfit by epoch 5 (train loss → 0.005); 4-worker FL on the same step count regularized that away.
- Bigger batch ≠ free speed. v4's wall-clock anomaly (longer than v3 despite fewer local steps) suggests per-batch overhead doesn't amortize linearly with batch size in the federated path.
- TTA at eval is free top-1. Hflip averaging at inference time bought ~0.2-0.4 pp across every checkpoint, with zero training change.
Reproducibility Across Hardware
Four runs of the v2 base recipe (batch 128, lr 2e-4, 5 rounds × 1 local epoch) across two GPU generations land within 0.16 pp TTA top-1:
Training Cost (5 Rounds × 4 Workers)
job_csgo ▓▓ 3-5 min [demo/small]
job_embed ▓▓▓ 10-30 min [BGE-base on 558k NLI triplets]
job_classify ▓▓▓▓▓▓ 1-3 hr [ViT-base on Food-101, 75k imgs]
job_medical ▓▓▓ 30-90 min [DenseNet on HAM10000, 13k imgs]
job_speech ▓▓▓▓▓▓▓▓ 3-6 hr [Whisper-small LoRA on LibriSpeech-100, 28k clips]
job_llm ▓▓▓▓▓▓▓▓ 3-6 hr [Phi-3.5-mini LoRA on Alpaca, 52k]
job_yolo11 ▓▓▓▓▓▓▓▓▓▓▓ 6-10 hr [YOLO11m on COCO, 118k imgs]
job_diffusion ▓▓▓▓▓▓▓▓▓▓▓▓ 8-12 hr [SDXL LoRA on 833 captions]
job_agent ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ 12-18 hr [Llama-3.1-8B 4-bit LoRA on Glaive 113k]Browse the Other Jobs
- job_csgo — YOLOv5 CS:GO detection (3-5 min demo).
- job_yolo11 — YOLO11m on COCO 2017.
- job_medical — MONAI DenseNet on HAM10000 dermatology.
- job_embed — BAAI/bge-base on NLI triplets.
- job_speech — Whisper-small LoRA on LibriSpeech-100.
- job_llm — Phi-3.5-mini-instruct LoRA on Alpaca.
- job_diffusion — SDXL LoRA on Pokémon-BLIP captions.
- job_agent — Llama-3.1-8B-Instruct (4-bit) on Glaive function-calling.
See the FL catalog README for the status of every example.
Caveats
- Model-card metrics ≠ your FL training metrics. The published numbers are the model author's centralized training results, used as the baseline. Your FL training fine-tunes from those checkpoints — final FL metrics depend on shard distribution, num_rounds, and local_epochs.
- job_medical has no published baseline; your FL run produces the reference.
Next Steps
- Clone the repo:
git clone https://github.com/resontech-dev/examples.git - Pick a job, build its shards (
python build_shards_hf.py), and submit via the SDK or the web wizard. - Read the Approach deep-dive for the FL plumbing all jobs share.