Deduplication Metadata Path Failure After Network Segmentation
A CommCell has one CommServe at headquarters and six MediaAgents. The main storage policy writes to deduplicated disk libraries on MA1, MA2, and MA3, but all three use a shared Deduplication Database (DDB) hosted on MA-DEDU1. A second, non-deduplicated copy writes to tape. After a firewall segmentation change at quarter-end, month-end full backups from MA1, MA2, and MA3 fail at 90-95% with 'failed to connect to the deduplication database.' Some small incremental backups complete, and tape jobs to the non-deduplicated copy succeed. The CommServe and its SQL database are healthy, MA-DEDU1 shows no disk latency, and the tape library is functioning. The backup window is 8 hours; the failed jobs are now 14 hours behind. The team wants to disable deduplication globally to get through quarter-end.
1. Explain why the jobs fail near completion rather than at the start, and why the tape copy still works. Which CommCell components are on the control path versus the data path?
Read the worked answer
The CommServe is the control path: it schedules jobs, checks policy, and tracks job state. The MediaAgents are the data path: they read client data and write it to the disk library. The DDB is a metadata service used to identify duplicate blocks and commit signatures. During a deduplicated backup, data blocks can stream to the MediaAgent's disk library first; the job then finalizes by committing deduplication metadata to the DDB. If the DDB becomes unreachable at that finalization stage, the job fails near the end, which matches the 90-95% failures. Small incrementals may finish because they are short and sometimes complete before an intermittent connection times out. The tape copy does not use the shared DDB, so it continues to succeed. This is not primarily a CommServe failure or a disk performance problem. The evidence points to a broken network path or service dependency between MA1, MA2, MA3 and MA-DEDU1. Disabling deduplication globally would create a new storage policy behavior, consume far more disk, and likely force new baselines; it is a last resort, not a first response.
2. Give a specific triage sequence. What would you check on the network, the DDB host, and the Commvault logs before changing the storage policy?
Read the worked answer
Start by proving whether each MediaAgent can reach MA-DEDU1 on the required ports, not just whether ping works. Check firewall rules added during the segmentation change, routing asymmetry, and any newly introduced ACL between subnets. On MA-DEDU1, verify the DDB service or process is running, the DDB disk or mount has free space and normal latency, and the DDB partition is healthy. Review Commvault logs on both the failing MediaAgents and MA-DEDU1 around the failure time; look for connection timeouts, authentication failures, and DDB commit errors. Check whether DDB pruning or compaction was running during the window. Test with a small, controlled deduplicated backup from one affected MediaAgent after correcting one path. If multiple MediaAgents fail identically, suspect a shared network or DDB host issue rather than three independent MediaAgent faults. Only after the DDB path is restored should you decide whether the 14-hour backlog needs a temporary non-deduplicated copy to tape. That preserves the RPO without permanently weakening the deduplication design.
3. If the DDB is corrupted and cannot be repaired, compare restoring the DDB from backup with creating a new DDB and re-baselining. What are the trade-offs for the backlog and for existing restore points?
Read the worked answer
Restoring the DDB from a known-good backup can preserve signature continuity, but it may be stale. Jobs written after that backup may have signatures that are not represented, so the DDB may need verification or a partial re-baseline. The restore itself takes time and may require quiescing or isolating the affected MediaAgents. Creating a new DDB is often faster to bring online and removes the corruption, but existing deduplicated data on disk still has restore value; the new DDB simply will not deduplicate against older blocks until those blocks are re-referenced. That increases disk consumption during the transition. For the 14-hour backlog, the safer immediate move is to write critical jobs to a non-deduplicated tape or temporary disk copy, then rebuild deduplication. Do not delete primary data to force space recovery, because that can break secondary copies and retention. After either path, verify restores from pre-change and post-change backups, not just job success.