Software Development
What to know
Candidates should read a training or inference stack end to end. On the training side: CUDA kernels, NCCL for collectives, mixed precision (BF16 and FP8 with scaling), gradient accumulation, activation checkpointing, and distributed strategies - data parallel, tensor parallel (Megatron-style, head-dim divisibility required, best inside NVLink nodes), pipeline parallel (interleaved schedules reduce the bubble but add memory pressure), sequence/context parallel for long context, and ZeRO stages/FSDP for sharding optimizer state, gradients, and parameters.
Effective batch size = micro_batch x grad_accum x data-parallel degree. Change it and you usually must rescale the learning rate and warmup.
On the serving side: TensorRT-LLM compiles optimized engines with in-flight/continuous batching, paged KV cache, and quantized kernels. Triton Inference Server hosts models behind a common API; NIM packages optimized models as services; NeMo covers training, PEFT, and guardrail tooling. Track TTFT (time to first token), inter-token latency/TPOT, throughput (tokens/s), and goodput under an SLO.
Profiling with Nsight Systems separates compute-bound from communication-bound steps.
Traps
- Forgetting that tensor parallel needs heads divisible by TP degree.
- Miscounting batch size by omitting gradient accumulation or DP replicas.
- Assuming ZeRO-3 and FSDP differ semantically - they are sharding equivalents.
- Ignoring optimizer state memory.
Worked example
Adam mixed precision costs about 16 bytes/parameter: 2 (BF16 weights) + 2 (grads) + 4 (FP32 master) + 8 (m, v). A 7B model needs ~112 GB of state, so a single 80 GB GPU cannot hold it - you must shard.