Batch Genomics Compute Selection
A genomics lab runs 1,200 sequencing analysis jobs per week. Each job takes 20–45 minutes, needs 32 vCPU and 120 GB RAM, and writes 2 TB per day to S3. Jobs checkpoint every 5 minutes and can restart without data loss. Demand spikes 4x near grant deadlines, while the on-prem cluster sits 80% idle. The lab has a small DevOps team and wants to minimize cost and operational overhead. Lambda and Fargate were proposed, but the lead is unsure because some future jobs may need GPUs. Results must be reproducible for seven years, and raw data older than 90 days is rarely accessed but must be retained. The lab pays $18,000 per month for the cluster and has a 30% budget cut target.
1. Choose the compute and orchestration model. Explain why Lambda and Fargate are or are not suitable.
Read the worked answer
Recommendation: AWS Batch with a managed EC2 compute environment using Spot Instances for the CPU jobs. Add a separate GPU compute environment later if GPU workloads are confirmed.
Reasoning: Lambda cannot run these jobs: its maximum timeout is 15 minutes, and its maximum memory is about 10 GB, while each job needs 20–45 minutes and 120 GB. Fargate is closer but still unsuitable for the current profile: a Fargate task tops out at 16 vCPU and 120 GB, so it cannot provide 32 vCPU, and Fargate does not offer GPUs. AWS Batch fits because it queues work, retries failures, and can use Spot capacity with checkpointing. The 80% idle on-prem cluster shows the lab should not buy fixed capacity for peak. Batch also isolates the small DevOps team from cluster orchestration. The tradeoff is Spot interruption risk and less control than self-managed Kubernetes; checkpointing every five minutes makes that acceptable.
2. How should Spot interruptions and 4x deadline spikes be handled without losing work?
Read the worked answer
Recommendation: diversify instance families and Availability Zones, use capacity-optimized allocation, enable Batch retries, and make the checkpoint location durable in S3. Set a maximum vCPU ceiling per compute environment so a deadline spike cannot consume unlimited budget.
Reasoning: Spot capacity is cheapest when the workload is flexible. A single instance family in one AZ is likely to be reclaimed together, so diversify across several comparable families. Checkpointing every five minutes bounds lost work to five minutes, and Batch can retry the job on a new instance. Use a job queue with a higher-priority queue for deadline work and a lower-priority queue for routine analysis, rather than letting every job compete equally. The 4x spike is temporary, so the queue absorbs it; provisioning permanent peak capacity would recreate the 80% idle problem. Warn that Spot is not appropriate for jobs that cannot checkpoint. If a deadline cannot tolerate retries, run those jobs on On-Demand capacity in the same Batch environment.
3. Design the S3 retention and reproducibility approach for raw data and results.
Read the worked answer
Recommendation: store raw data and results in versioned S3 buckets with bucket policies that deny deletion, and use lifecycle rules rather than manual cleanup. Keep result containers and workflow definitions versioned so a result can be reproduced.
Reasoning: Reproducibility requires more than keeping output files: it requires the input data, container image digest, workflow parameters, and reference genome version. Store those references with each run manifest. For raw data older than 90 days and rarely accessed, move to S3 Glacier Instant Retrieval if reanalysis may need immediate access, or Glacier Deep Archive if retrieval within 12 hours is acceptable. Do not move results to Deep Archive if auditors need quick access. Versioning plus Object Lock in governance or compliance mode protects against deletion and ransomware, but confirm the retention duration and legal requirements before enabling compliance mode, because it cannot be shortened. The 30% cut is achievable from compute elasticity and storage tiering, but retention controls should not be weakened to hit the number.