Training Job Dies Mid-Run and Leaves No Artifacts
A vision team trains a 12M-parameter image classifier on 380,000 JPEGs (about 210 GB) in Object Storage. The job is configured on a VM.GPU.A10.2 shape with 128 GB block storage, and the script downloads the full manifest and decodes every image into a pandas DataFrame before the first epoch. Job runs die at roughly 40 minutes with exit code 137 and nothing is written to the output prefix. A run on a 1/10 sample succeeds and takes 6.5 hours; the team needs a full-data run nightly to hit a Thursday review. Logs are only visible while the run is active. An earlier workaround copied the whole dataset to the job's block storage on every run, which now fails because the dataset no longer fits.
1. Diagnose exit code 137, the empty output prefix, and the failure of the local-copy workaround, and state the first configuration changes you would make.
Read the worked answer
Exit 137 is SIGKILL - the OOM killer, from either container memory limits or host pressure. Decoding 380k JPEGs into a DataFrame is not a dataset problem, it is an eager-materialization problem: decoded images expand far beyond the 210 GB of compressed bytes, so RAM is exhausted well before the first epoch.
Empty output prefix: most scripts write artifacts only after training completes. The kill happens at minute 40, so nothing is ever written. It is not evidence that writing is broken; it is evidence that writing is too late.
Local-copy workaround: 210 GB of source data plus decoded or intermediate files cannot fit in 128 GB of block storage. Even a larger volume would spend tens of minutes per run copying data that Object Storage can serve directly, and the copy is repeated because job block storage does not survive the run.
First changes: switch to lazy loading - index the object list once, read and decode per batch in DataLoader workers (or convert once to a sharded binary format such as WebDataset/TFRecord and stream it), keep only a shard on block storage if random access is required, and increase block storage for checkpoints and cache. Add artifact upload on failure and confirm the output prefix is set at job creation, not inferred. Verify the memory ceiling by running a 1/10 job with peak-RSS logging before spending GPU hours on the full set.
2. The nightly full-data run must finish within about 9 hours. What do you change, and what evidence do you need before promising it?
Read the worked answer
Budget the arithmetic first. The sample run is 6.5 hours for 38,000 images. If time scaled linearly, 380,000 images would take roughly 65 hours - an order of magnitude beyond the window. That projection assumes the sample was I/O bound with the same eager-loading pathology, so the first measurement after fixing the data path is time-per-1000-images on the full pipeline, not on the sample.
Then attack throughput:
- Move to a multi-GPU shape and use distributed data parallel so the 65-hour estimate divides by the number of GPUs, minus communication overhead.
- Pre-shard or pre-encode the dataset once into an efficient format; image decode is often the real bottleneck, not the model.
- Set DataLoader workers and prefetch so the GPU is never idle waiting on Object Storage; ensure each worker reads from a different shard to avoid hot-spotting a single object.
- Checkpoint every N steps to Object Storage and make the script resume, so one preemption does not cost a night.
Observability: enable the service log so run logs persist after the run ends, and emit step, loss, throughput, and GPU utilization as metrics with an alarm on stalled progress.
Evidence before promising: a timed full run, not an extrapolation, plus a written statement of what happens on the Thursday that data arrives late. If the window cannot be met, the honest answer is a two-stage plan - nightly incremental fine-tuning on new data, weekly full retrain - rather than an optimistic promise.
3. Define the artifact and provenance contract so the model can be promoted and rolled back weeks later.
Read the worked answer
What the job writes, every run: model weights or serialized model, preprocessing and label encoders, metrics on a held-out split, a frozen environment specification, and a small manifest JSON. All paths under a run-scoped prefix keyed by the job run OCID so two runs never overwrite each other.
What gets registered in Model Catalog: the artifact itself plus provenance metadata - git commit, data snapshot version or object-prefix listing hash, environment slug, training shape, timestamp, evaluation dataset, and the job run OCID. Without the data version, the metrics are not reproducible and the model should not be promoted; without the environment slug, it cannot be rebuilt; without the run OCID, the logs cannot be found.
Promotion gate: a script or pipeline step compares the candidate's held-out metric against the currently deployed version and fails closed. Register as a new catalog version and keep prior versions intact; deletion of the previous version is what makes rollback impossible and is the most common mistake in this kind of handoff.
Rollback test: before the Thursday review, actually roll back to the previous version in a staging deployment and confirm it serves. A rollback procedure that has never been executed is a hypothesis, not a control.