Stale Secondary Zones After a Primary Rebuild

The two codes fail at different layers.

• REFUSED is the primary refusing the AXFR at the ACL level. Because allow-transfer lists only a key, an unsigned transfer request is denied. ns2's masters statement has no key clause, so BIND never signs the refresh or the AXFR - exactly what you expect after the template rebuild dropped that clause.
• NOTAUTH means a TSIG-signed message arrived but could not be validated: the key name is unknown to that server (names are case-sensitive, and in BIND 9.16 a key defined inside another view is invisible), the algorithm differs (hmac-sha256 vs hmac-sha512), the base64 secret differs, or the clocks are beyond the 300-second fudge. Copying the file with scp proves nothing; compare the key lines themselves.

Checks, in order:

1. named-checkconf -p /etc/bind/named.conf on both hosts - prints the effective configuration including includes, so you can see the key definition and the masters/allow-transfer lines as BIND actually reads them.
2. grep -A2 xfer-key /etc/bind/xfer.key on all three hosts and diff them: name, algorithm, secret.
3. From ns2: dig +tcp -y hmac-sha256:xfer-key:SECRET @10.10.1.10 example.com AXFR. If that succeeds, the key and ACL are fine and only the secondary's masters clause is wrong.
4. rndc tsig-list on ns1 to confirm the key is loaded, and chronyc tracking for skew.

Fix ns2 with masters { 10.10.1.10 key xfer-key; };, reload, and watch the transfer in the log rather than assuming it worked.

The backup restore moved the zone backwards in time. A secondary decides freshness by comparing SOA serials, and a lower serial is treated as stale data or a broken master, not as a new version - a deliberate protection against restoring an old zone file or rolling back a primary.

Option A, the correct fix: raise the serial on ns1 above every value ever published, for example 2024051005 (or a fresh YYYYMMDDnn such as 2024051201), then rndc reload example.com and rndc notify example.com. Serials must never decrease, so this is the only change that survives the next refresh, a secondary reboot and a future restore. Document the bump so the next restore is checked against a recorded high-water mark.

Option B, the emergency fix: rndc retransfer example.com on ns2 forces a full AXFR regardless of serial. A plain rndc refresh will not help, because refresh re-runs the serial comparison. If the zone has a journal, stop named, remove the stale example.com.jnl if it refuses, and start again. The risk is that ns2 now holds the lower serial; monitoring, the other secondary and any tooling that remembers 2024051004 will see an inconsistency, and you have lost the serial reference that protects against a second bad restore.

Never delete the zone file and restart in production: you get SERVFAIL for at least a refresh cycle. Verify afterwards with rndc zonestatus example.com on ns2 and dig @ns2.example.com SOA +short, which must match the primary.

The failure was invisible because nobody compared the SOA serials of the three authoritative servers; the secondaries were still answering, just with old data.

A minimal monitoring check: for each zone, query the SOA directly at each authoritative address (dig +short @10.10.1.10 example.com SOA, then the same against 172.16.5.20 and .21, using +norecurse). Alert when any secondary's serial is lower than the primary's for longer than the zone's refresh plus retry, or when the difference persists after a notify. Also alert on the expire timer: rndc zonestatus example.com reports next refresh and expires, and a secondary that reaches expiry stops answering entirely - a much louder outage than stale data.

Add a post-restore hook: run named-checkzone example.com /etc/bind/db.example.com, compare the file's serial against a stored high-water mark, and refuse to reload if it went backwards. The same check belongs in the configuration-management template, because that template rebuild is what dropped the key clause in the first place.

For ns3 run the identical TSIG and serial remediation - it is serving the same stale 2024051004 - and then confirm NOTIFY reaches both secondaries: tcpdump -ni any port 53 while you run rndc notify example.com, or check also-notify in the zone options. Verify both secondaries serve the new serial before closing the incident.