Overview
The full FL job catalog lives at github.com/resontech-dev/examples. Twelve job templates: nine fine-tune real HuggingFace models with verifiable published metrics, two are from-scratch production loads (ImageNet ResNet-50, Pythia-1B pretraining), and one is the minimal SDK starter (MNIST CNN) — the clone-and-edit base if you're new.
This page deep-dives job_classify (the canonical reference) and links to the other eleven. Per-job READMEs in the repo link each model's HuggingFace card so every number can be verified.
At a Glance
| Job | Model | Author's published metric / load type | Dataset |
|---|---|---|---|
| job_training_sdk | Tiny CNN (~0.1 M) | SDK starter — clone-and-edit base | MNIST (60k) |
| job_csgo | yolov5n-csgo (1.9 M) | mAP@0.5 ≈ 0.85 (CS:GO val) | CS:GO 4-class (4,262 imgs) |
| job_yolo11 | YOLO11m (20.1 M) | COCO mAP@50-95: 51.5 | COCO 2017 (118k imgs) |
| job_classify | ViT-B/16 in21k (86 M) | Top-1 89.13 % / val_loss 0.4501 (nateraw/food ref) | Food-101 (101k imgs) |
| job_medical | MONAI DenseNet121 (7 M) | — (FL run produces the reference) | HAM10000 (13.4k, 7 classes) |
| job_embed | BAAI/bge-base-en-v1.5 (110 M) | MTEB avg 63.55 (56 datasets) | all-nli (558k triplets) |
| job_speech | openai/whisper-small (244 M) | LibriSpeech-clean WER 3.43 | LibriSpeech train.100 (28.5k clips) |
| job_llm | microsoft/Phi-3.5-mini-instruct (3.8 B) | MMLU 69, GSM8K 86.2 | tatsu-lab/alpaca (52k) |
| job_diffusion | stabilityai/SDXL-base-1.0 (~3.5 B UNet) | SDXL > SD 2.1 > SD 1.5 (preference evals) | lambdalabs/pokemon-blip-captions (833 imgs) |
| job_agent | unsloth/Llama-3.1-8B-Instruct-bnb-4bit | MMLU 69.4, BFCL 76.1, GSM8K 84.5 | glaiveai/glaive-function-calling-v2 (113k) |
| job_imagenet_scratch | ResNet-50 (25.6 M, random init) | From-scratch — timm A2 recipe (ResNet Strikes Back, ICLR 2022); reproduces timm/resnet50.a2_in1k | ImageNet-1k (1.28 M) |
| job_lm_scratch | GPTNeoX 1B / Pythia spec (~1.01 B, random init) | From-scratch pretraining — most demanding job in the catalog | Pile-style tokenized corpus |
Common Job Layout
Every example job follows the same directory layout. The SDK and the web wizard expect this shape on submission:
i
All twelve jobs share byte-identical
custom_client_executor.py and custom_persistor.py — the FL plumbing works for every task type. Only model_def.py and the framework utils change.Deep Dive — job_classify (ViT-B/16 on Food-101)
job_classify is the canonical reference — the only job where the FL run beats the centralized HuggingFace baseline on both top-1 and val_loss. Repo links: README and RUNS.
Headline numbers
| Metric | This job (best run) | HF reference (nateraw/food) | Δ |
|---|---|---|---|
| top-1 | 89.42 % | 89.13 % | +0.29 pp ✅ |
| top-1 (--tta) | 89.70 % | n/a | +0.57 pp vs HF top-1 |
| top-5 | 97.51 % | n/a | — |
| val_loss | 0.4971 | 0.4501 | +0.047 |
| val_loss (TTA) | 0.4857 | 0.4501 | +0.036 |
| Wall-clock | ~8 m 48 s | not disclosed | — |
| Total optimizer steps | 1,100 (4 × 5 × 55) | 2,960 | −63 % |
Validated on the full Food-101 validation split (25,250 images) with the HF eval recipe (AutoImageProcessor("google/vit-base-patch16-224-in21k") + per-example mean cross-entropy + top-1 / top-5). Deterministic seed 1337.
Recipe at a glance
The two deviations from the centralized HF recipe
- Continuous LR schedule across rounds — instead of restarting
LambdaLRper round (sawtooth), the lambda computes the LR from the global stepcurrent_round × steps_per_round + local_step. Same shape as HF's single-ramp decay. - Persisted per-client Adam state —
m/vbuffers saved toSTATE_DIR/optimizer_state.ptat end of round, restored at start of next. No more 10-step warmup tax per round.
Everything else — backbone, dataset, optimizer, loss, preprocessing, AMP, grad clip, seed — is byte-for-byte the HF recipe.
job_classify — Selected Runs
Every measured training run is logged in RUNS.md. The progression below is the punchline:
| Run | Code recipe | Config | Top-1 (TTA) | val_loss (TTA) | Train + comm |
|---|---|---|---|---|---|
| v1 (classify.pt) | Sawtooth LR, fresh Adam per round, plain CE | batch 128, lr 2e-4, 5×1 | 87.90 % | 0.5444 | 6 m 26 s |
| v2 (classify_v2.pt) | Continuous LR + persisted Adam | batch 128, lr 2e-4, 5×1 | 88.87 % | 0.5015 | 6 m 13 s |
| v3 (classify_v3.pt) | + label smoothing 0.1, EMA decay 0.999, 10 rounds | batch 128, lr 2e-4, 10×1 | 87.29 % | 0.7868 | 13 m 21 s ❌ |
| v4 (big-batch + EMA) | Same as v3, batch 256, 10 rounds | batch 256, lr 2.83e-4, 10×1 | n/a | 1.7251 | 15 m 33 s ❌ |
| v10 (best run) | v2 recipe + big batch (no √-LR scaling) | batch 348, lr 2e-4, 5×1 | 89.70 % ✅ | 0.4857 ✅ | 8 m 3 s |
| HF reference (nateraw/food) | Same recipe, centralized | batch 128, lr 2e-4, 5 epochs | 89.13 % | 0.4501 | not disclosed |
Lessons (full reasoning in TUNING_PLAYBOOK.md)
- Continuous LR + persisted Adam are the only deviations worth keeping. +0.76 pp / −0.03 loss vs naive port. Everything else attempted on top either regressed or was neutral.
- EMA over short FL rounds is a trap. Decay 0.999 over 74-148 steps per round dominates round-start weights and nullifies most local progress. Lower decay (~0.95) or persisting EMA across rounds would fix it.
- Label smoothing inflates measured val_loss. Don't use it if the headline metric is val_loss.
- FedAvg can out-regularize a too-aggressive centralized run. The single-GPU H100 baseline overfit by epoch 5 (train loss → 0.005); 4-worker FL on the same step count regularized that away.
- Bigger batch ≠ free speed. v4's wall-clock anomaly (longer than v3 despite fewer local steps) suggests per-batch overhead doesn't amortize linearly with batch size in the federated path.
- TTA at eval is free top-1. Hflip averaging at inference time bought ~0.2-0.4 pp across every checkpoint, with zero training change.
Reproducibility Across Hardware
Four runs of the v2 base recipe (batch 128, lr 2e-4, 5 rounds × 1 local epoch) across two GPU generations land within 0.16 pp TTA top-1:
| Run | Hardware | top-1 (TTA) | val_loss (TTA) | Wall-clock (train+comm) |
|---|---|---|---|---|
| classify_v2.pt | 4× RTX PRO 4500 | 88.87 % | 0.5015 | 6 m 13 s |
| rerun #1 | 4× RTX PRO 4500 | 89.24 % | 0.5756 | not measured |
| rerun #2 | 4× RTX PRO 4500 | 89.17 % | 0.5695 | not measured |
| rerun #3 (4090) | 4× RTX 4090 | 88.99 % | 0.5052 | 7 m 8 s |
| mean | — | 89.07 % | 0.5380 | — |
i
The mean TTA top-1 (89.07 %) is essentially at parity with HF's 89.13 %. The recipe is reproducible regardless of hardware; only wall-clock changes — ~15 % slower on Ada than Blackwell for ViT-B/16 at batch 128 in fp16.
Training Cost (5 Rounds × 4 Workers)
| Job tier | Min GPU |
|---|---|
| job_csgo, job_classify, job_medical, job_embed | 8 GB (T4 / RTX 3060) |
| job_speech, job_llm, job_yolo11 | 12 GB (RTX 4070 / L4) |
| job_diffusion (SDXL) | 24 GB (RTX 4090 / A10G) |
| job_agent (Llama-3.1-8B 4-bit) | 24 GB |
Browse the Other Jobs
- job_csgo — YOLOv5 CS:GO detection (3-5 min demo).
- job_yolo11 — YOLO11m on COCO 2017.
- job_medical — MONAI DenseNet on HAM10000 dermatology.
- job_embed — BAAI/bge-base on NLI triplets (5 rounds completed end-to-end on simulator).
- job_speech — Whisper-small LoRA on LibriSpeech-100.
- job_llm — Phi-3.5-mini-instruct LoRA on Alpaca.
- job_diffusion — SDXL LoRA on Pokémon-BLIP captions.
- job_agent — Llama-3.1-8B-Instruct (4-bit) on Glaive function-calling.
See the FL catalog README for the full validation status matrix.
Caveats
- Model-card metrics ≠ your FL training metrics. The published numbers are the model author's centralized training results, used as the baseline. Your FL training fine-tunes from those checkpoints — final FL metrics depend on shard distribution, num_rounds, and local_epochs.
- Runway took down SD 1.5 (Aug 2024). Don't reference it. SDXL is the current standard.
- Mistral-7B-Instruct-v0.3 has no published benchmarks on its HF card. Llama-3.1-8B-Instruct is what job_agent cites metrics from.
- HAM10000 has no popular HF model card with verifiable metrics. Your FL run produces the reference.
Next Steps
- Clone the repo:
git clone https://github.com/resontech-dev/examples.git - Pick a job, build its shards (
python build_shards_hf.py), and submit via the SDK or the web wizard. - Read the Approach deep-dive for the FL plumbing all jobs share.