The Analysis That Ran Overnight

Primary cause - writing intermediates to the same network folder as inputs, with reused names. Several problems compound:

1. Name collisions. Intermediate outputs reused across model branches overwrite each other. A later branch may read an intermediate that an earlier branch already overwrote, producing wrong inputs and, eventually, a save failure when a tool tries to write a dataset that is locked or already exists.
2. Network drive I/O. Writing 30 GB rasters to a network drive is slow and prone to transient locks and timeouts; 'Could not save raster dataset' (ERROR 010240) often means the target is locked, the path is invalid, or the file is in use - all more likely on a network share.
3. Input/output co-location. Writing intermediates into the input folder risks overwriting or locking inputs and makes cleanup impossible.

Why rerunning gave a different error: The model's behavior depends on leftover state - partial outputs, locks, and reused names from the first run. The second run started from a different folder state, so a different tool hit a different collision or lock first. The failure is nondeterministic because the workspace is not clean, not because the tools are unreliable.

The judgment: a 9-hour run that fails near the end is a design failure, not bad luck; the model must be made deterministic and resumable.

Workspace and naming:

• Isolate intermediates in a dedicated scratch workspace - ideally a local fast disk (or a file geodatabase) sized for the intermediates, not the network share and not the input folder. Set the model's scratch workspace explicitly.
• Unique, deterministic names per branch and per run (e.g., prefix with branch ID and a run timestamp or model parameter). Never reuse an intermediate name across branches.
• Write the final output to a separate, clean location, not the scratch folder, so it is not swept up in cleanup.

Model design:

• Break the 14-tool chain into submodels or checkpoints so a failure does not discard 9 hours of work; each checkpoint writes a durable intermediate that the next stage can resume from.
• Enable overwrite and validate inputs at the start: check CRS consistency, extent, cell size, and that the scratch workspace exists and has space.
• Use in-memory or managed workspaces for small intermediates; only large rasters go to disk.
• Parameterize the input mosaic and output path so the model is reusable and testable on a small subset.

Validation:

• Run on a small subset first (e.g., a clipped 1 GB extent) to catch naming and lock errors in minutes, not hours.
• Check free disk space for the full intermediate footprint before launching.
• Log tool messages to a file so a failure is diagnosable after the fact.

The principle: long-running models must be resumable and deterministic; a clean, isolated scratch workspace is the foundation.

Strategy - checkpoint and iterate on a subset:

1. Build a small representative test extent (clip the mosaic and vector to a manageable area) and run the full 14-tool chain on it. Iterate model logic, naming, and parameters there in minutes.
2. Persist durable checkpoints for the expensive early stages (e.g., the raster mosaic conditioning) so that when iterating on later stages, the model resumes from the last good checkpoint instead of recomputing from the raw 30 GB input.
3. Only run the full extent once the model is stable on the subset, and run it with logging and a clean scratch workspace.

Trade-off:

• The subset may not exercise edge cases (nodata patterns, extreme values, full-extent locks) that appear only at scale, so the final full run still needs validation.
• Checkpointing adds storage and model complexity (managing which checkpoint is current), and stale checkpoints can silently feed outdated data into later stages if not versioned.
• Persisted intermediates consume disk and must be cleaned up to avoid the very clutter that caused the original failure.

The judgment: iterate cheap, run expensive once - but never trust a subset run to prove a full-extent run will succeed without validation.