I85PREP
⌘ K
HPOC study materials
HPOC study resources

Backup Window Missed and Restore Fails Mid-Recovery

HPOC · Put your knowledge to work in context.

All case studies

Backup Window Missed and Restore Fails Mid-Recovery

A production OpenVMS cluster runs a nightly BACKUP of the system disk to a tape library. Last night the backup job exceeded its window and was killed by the operator at 04:00, leaving a partial save set on tape. The same morning, a user deleted a critical data file from a shadowed data volume. The operator wants to restore the file from last night's backup, but the tape contains only a partial save set. An older full backup from three nights ago exists on a separate tape. The system disk is shadowed (DSA1); the data volume is DSA2. The operator proposes restoring directly onto DSA2 from the partial tape and "letting shadowing sort it out," then re-running the backup. The application owner is worried about consistency across the two volumes.

1. Evaluate the operator's plan to restore onto DSA2 from the partial tape and rely on shadowing.

Read the worked answer

The plan is unsafe for several reasons:

  • Partial save set: the tape was truncated when the job was killed. BACKUP save sets are sequential; a partial save set may lack the file, or may contain a corrupted/incomplete record. Restoring from it risks writing garbage or failing mid-restore.
  • Shadowing does not repair data: shadowing replicates blocks across members. It does not know about file semantics or save-set integrity. Restoring a bad block onto one member propagates that block to all members - it makes the problem worse, not better. "Letting shadowing sort it out" is a misunderstanding of what shadowing does.
  • Cross-volume consistency: if the deleted file has relationships with data on DSA1 (e.g., application metadata), restoring only the file from a stale backup can leave the two volumes inconsistent. The application owner's concern is valid.

Correct approach: do not restore from the partial tape. Use the three-night-old full backup as the recovery source, and restore to a scratch location first for verification before placing data on DSA2.

2. Lay out a safe recovery procedure for the deleted file, including verification before it goes back into production.

Read the worked answer

1. Stop and preserve state

  • Do not overwrite anything on DSA2. Snapshot or note the current state.
  • Confirm the file is truly gone and identify its exact name, directory, and owning application.

2. Choose the source

  • Use the three-night-old full backup (known complete), not the partial tape.
  • If the application has transaction logs or a journal, use them to reconcile any changes made in the last three days.

3. Restore to scratch

  • Restore the file to a scratch disk or directory, e.g. BACKUP <tape>:/SELECT=... <scratch>/, so production is untouched.
  • Verify the restored file: size, checksum, and content spot-checks against expected values.
  • If the file changed in the last three days, determine whether the older version is acceptable or whether journal/log replay is needed.

4. Place into production

  • Only after verification, copy/move the file into the target directory on DSA2.
  • Confirm permissions, ownership, and that shadowing replicates it cleanly (SHOW DEVICE DSA2 members consistent).

5. Re-run backup

  • Re-run the full backup with adequate window, and verify the save set is complete before considering the incident closed.

3. What changes would you make to prevent both the missed window and the partial-save-set risk in future?

Read the worked answer

Backup window

  • Measure backup duration and set the window with headroom; use incremental backups during the week and full backups on a low-activity night.
  • Use BACKUP/VERIFY on critical save sets, and monitor job duration with alerts so an overrun is caught before the operator kills it.
  • Consider parallel or image backups where supported to shorten duration.

Partial save set risk

  • Never kill a backup mid-stream without marking the tape invalid. Implement a policy: a killed backup's tape is quarantined and labeled bad.
  • Use BACKUP/VERIFY and check the save-set completion marker before treating a tape as restorable.
  • Keep at least two complete generations (the three-night-old full plus a newer complete one) so a single failed night does not leave only a stale backup.

Recovery readiness

  • Document and periodically test restore procedures to scratch, including a cross-volume consistency check.
  • Ensure journaling/transaction logs exist for critical application data so a stale backup can be reconciled.
  • Train operators that shadowing replicates blocks and is not a data-repair mechanism.