Bursting a Three-Week Training Campaign to Cloud GPUs
A research group has 48 on-premises A100 GPUs and a 300 TB training dataset on an NFS filer. It needs 200 GPUs for a three-week campaign and plans to burst to a cloud region connected by a 10 Gb/s dedicated interconnect. Training checkpoints are 1.2 TB and are written every 30 minutes to on-premises storage. The cloud provider offers P4d/P5 instances with EFA networking and an FSx for Lustre option linked to object storage. The team estimates that the cloud job will run for 21 days and that egress charges apply to data leaving the cloud. The security team requires encryption in transit and at rest and does not want the dataset copied to object storage without an approved lifecycle policy. The project sponsor wants a cost estimate and a go/no-go decision within one week.
1. Estimate the data-movement and checkpoint feasibility, showing your assumptions.
Read the worked answer
At 10 Gb/s, the theoretical rate is 1.25 GB/s. A 300 TB dataset is 300,000 GB, so a full copy takes about 240,000 seconds, or roughly 67 hours, before protocol overhead. That is nearly three days of the 21-day campaign, and it assumes the link is dedicated to the transfer. In practice, TCP overhead, NFS behavior, and competing traffic will make it slower. A one-time pre-stage may still be acceptable if started early, but iterative copying is not.
Checkpoints are 1.2 TB every 30 minutes, which is 1,800 seconds, so the average required rate is about 0.67 GB/s. That fits within 1.25 GB/s on paper, but it consumes over half the link and competes with any dataset synchronization. If checkpoints are written to on-premises NFS while training runs in the cloud, each checkpoint crosses the WAN and adds latency to the training step. A better design is to write checkpoints locally in the cloud and replicate asynchronously, with a retention policy. The calculation makes the WAN a shared, contended resource, not a free pipe.
2. Recommend an architecture and instance/network choices, and explain how you would handle data locality.
Read the worked answer
Pre-stage the dataset into object storage in the target region using a physical transfer appliance or an accelerated online transfer, then present it to training through FSx for Lustre linked to the bucket. Do not stream the 300 TB from on-premises NFS during training. Use EFA-enabled instances in a cluster placement group and a provider-supported RDMA network; confirm that the framework and MPI/NCCL stack use EFA. Keep checkpoints in FSx or local NVMe during the run and replicate them asynchronously to object storage with lifecycle rules that satisfy the security team. Encrypt in transit and at rest, and document the lifecycle policy before copying.
For the hybrid period, use the WAN mainly for metadata, logs, and checkpoint replication, not for training I/O. If the on-premises 48 GPUs must join the same job, verify that the interconnect latency and bandwidth can support the collective pattern; otherwise run the cloud job as a separate phase and merge results through checkpoints. Choose the region by data gravity and instance availability, not by list price alone.
3. What cost and failure risks would you put in the go/no-go memo?
Read the worked answer
Include compute cost for 200 GPUs for 21 days, storage cost for FSx and object storage, egress for checkpoint replication and any data returned on-premises, and the one-time transfer cost. Model a 20-30% schedule buffer because GPU capacity may not be available for the full three weeks and because checkpoint restore tests often reveal issues. The largest hidden risk is idle GPU cost while data catches up; a 67-hour transfer on a 10 Gb/s link is a schedule risk, not just a bandwidth number.
Failure risks include WAN congestion delaying checkpoints, object storage lifecycle misconfiguration, EFA or NCCL misconfiguration reducing collective performance, and cost overrun from leaving instances running after job completion. Require a restore drill from a cloud checkpoint before the campaign, a hard budget alarm, and an automated shutdown policy. The go decision should be conditional on pre-staging the data before GPU instances are provisioned and on proving checkpoint restore within the SLO.