Software Development
What this domain tests
This domain covers practical engineering for generative AI: Python and PyTorch, GPU-aware data pipelines, distributed training, containers, and serving APIs. Questions test whether you can make an experiment reproducible and a service reliable.
Core rules and concepts
- Place tensors and models on the same device; device mismatches are a top runtime error.
- DataLoader workers, pinned memory, and prefetching keep the GPU fed. Low GPU utilization usually means the input pipeline is the bottleneck.
- In distributed data parallel (DDP), each process sees a shard and gradients are all-reduced. Effective batch = per-GPU batch x world size x accumulation steps.
- Set seeds for Python, NumPy, and the framework. Some GPU kernels remain nondeterministic unless deterministic mode is enabled.
- Pin dependencies and container images, such as NGC containers, so training and serving environments match.
- Version prompts, weights, and tokenizers together; a tokenizer mismatch silently corrupts outputs.
Worked example
You train on 8 GPUs with per-GPU batch 4 and 2 gradient-accumulation steps. Effective batch = 4 x 8 x 2 = 64. Moving to 16 GPUs while keeping per-GPU batch and accumulation fixed doubles the effective batch to 128. To preserve the original optimization, halve accumulation or apply learning-rate scaling. Forgetting this changes training dynamics, so an apparent hardware speedup is confounded with a changed experiment.
Traps that cost marks
- Changing effective batch size while claiming only hardware changed.
- Forgetting to reshuffle with DistributedSampler each epoch.
- Serving with a different tokenizer or preprocessing path than training.
- Assuming a notebook result reproduces without pinned versions.
- Profiling with synthetic inputs that never hit the real data pipeline.