Cutting a 50-hour epoch to a 6-hour wall clock on a shared 8-node quota

Budget in GPU-hours: 5 node-hours per epoch x 10 epochs = 50 node-hours, or 400 GPU-hours. To finish in 6 hours you need about 50/6 = 8.3 nodes running continuously - and that is at 100% scaling efficiency, which data-parallel training never achieves.

So the target is infeasible at the stated quota. With 8 nodes and realistic scaling efficiency (85-90% at this size, with gradient all-reduce overhead), you need closer to 9-10 nodes, and those 8 nodes are shared with two other teams, so you will rarely hold all of them for six hours.

Options, in order of preference:

1. Reduce the work per epoch - mixed precision, smaller input resolution, or gradient accumulation with larger effective batch. Dropping resolution often saves 30-40% at modest accuracy cost; validate it on a held-out set rather than assuming.
2. Reduce epochs with a schedule - warm up on a subset, or use a cosine schedule over 6 epochs.
3. Negotiate quota for a defined window, which is a scheduling conversation, not an engineering one.

Do not present a plan whose wall-clock estimate depends on perfect linear scaling; state the assumed efficiency explicitly.

Two separate problems: durability and concurrency.

Durability. Checkpointing every 200 steps to the default datastore is both slow and, if the datastore is dismounted between attempts, unreliable. Write checkpoints to the run's outputs/ directory (tracked by the job, retained with the run), keep the last two plus the best, and have the script resume from the latest checkpoint if one exists. Preemption on low-priority nodes then costs minutes, not GPU-hours. Note the trade-off: low-priority is cheaper and preemptible; if the wall-clock target is hard, use dedicated nodes for the final trials.

Concurrency. 30 trials on 8 nodes with multi-node trials will thrash. Set max_concurrent_trials from the per-trial node count - e.g. 4 nodes per trial gives 2 concurrent trials - and allocate the rest of the quota to the other teams. For early termination, Bandit policy is the better default than median stopping here: with expensive trials and noisy early metrics, a relative-slack bandit prunes runs that fall behind the best by a margin, and the margin can be set from the observed run-to-run variance. Use a warm-up period early in the sweep so no trial is killed before its learning rate has had a chance to matter.

1.2 million images will not fit in the memory of 8 workers, so in practice each worker is either downloading a large shard redundantly or the code is silently falling back to per-batch reads - either way the GPUs starve. Worse, with data-parallel training every worker must see different samples per step, so a naive full download also risks duplicated shards and an effectively smaller dataset.

Replace it with:

• Mount the dataset read-only from the registered data asset (URI mount to blob/ADLS) instead of downloading. Mounting gives a filesystem view with caching handled by the platform and avoids per-worker copies.
• Shard by rank. Each worker takes a disjoint subset determined by world size and rank, and the distributed sampler handles shuffling.
• Use the DataLoader properly: adequate num_workers, pinned memory, and prefetch depth so I/O overlaps compute; keep the transform pipeline on GPU where possible.
• Pre-resize/encode once offline. Decoding full-resolution JPEGs on every epoch is often the single largest cost at this scale, and a cached, resized copy removes it.

Measure GPU utilisation: if it is persistently below ~85%, the bottleneck is the input pipeline, not the model or the node count - and no amount of extra nodes will fix it.