38 Gbps DNS Amplification Against a Two-Node Anycast Footprint
Kestrel Cloud runs authoritative DNS for 900 customer zones from two anycast nodes, each with a 10 Gbps transit connection plus a 10 Gbps IXP port. At 14:05 a 38 Gbps DNS amplification attack hit both nodes, using ANY and DNSKEY queries with spoofed sources. Response rate limiting was set to 5 responses per second per source, but the attack used roughly 2 million unique spoofed source addresses, so rate limiting barely engaged. Recursive resolvers began timing out and 40% of lookups for customer zones failed for 55 minutes. Kestrel's on-premises scrubbing appliance is rated 8 Gbps per node. The regulator requires notification of any outage affecting more than 100,000 users within 6 hours. Kestrel's SLA offers 99.95% monthly availability with service credits, and three enterprise customers have a contractual 15-minute incident update clause.
1. The CTO wants to blackhole the /24 containing both anycast nodes until the attack stops. Evaluate that proposal.
Read the worked answer
Reject it as the primary action: it is a service shutdown presented as a mitigation.
- Both anycast nodes - the entire authoritative service - sit in that /24. A blackhole (RTBH) does not divert the attack, it discards all traffic to the destination, including the legitimate queries for 900 customer zones and the nodes' management paths. The 40% failure becomes 100% for every customer, for as long as the blackhole is up.
- You also lose the ability to observe and to recover: with management traffic blackholed you cannot tune the mitigation without a separate out-of-band path that must already exist.
- It breaches the 15-minute update clause in substance even if the emails go out on time, and it will convert a 55-minute event into a much longer one in the SLA record.
What to do instead, in order:
- Reduce the amplification factor at the protocol level, immediately. Disable ANY responses (minimal-ANY, RFC 8482) and stop answering DNSKEY with large responses where a smaller equivalent exists. ANY and DNSKEY amplification depends entirely on response size - a minimal response cuts the attack's effectiveness by roughly an order of magnitude without touching legitimate resolution.
- Tighten per-source RPS to 1-2 per second with a slip factor of 2, and add per-zone and per-qtype limits rather than only per-source.
- Divert to off-net scrubbing. Advertise the DNS /24 to a scrubbing provider over a GRE/BGP diversion so the 38 Gbps is absorbed upstream. This must be pre-provisioned - it is not an option you invent at 14:05.
- Drop spoofed traffic as close to source as possible: uRPF/BCP38 on your own access edges, and ask the IXP peers to apply source validation.
Blackholing is a last resort for a host that is already dead, not for the service you are protecting.
2. Work out the notification and SLA obligations, including who must be told and by when.
Read the worked answer
Regulatory clock: detection at 14:05 starts the 6-hour window; the deadline is 20:05. Do not wait until 20:00 - file a preliminary notice as soon as impact is confirmed (say 14:30), stating known facts, affected services and zones, mitigation in progress, and next update time. File a closing report with root cause and remediation dates. The threshold is 'outage affecting more than 100,000 users' - with 40% failure of lookups for 900 zones, resolve whether the affected-user count crosses it before deciding not to notify; when in doubt, notify.
Contractual, 15-minute clause: first update to the three enterprise customers by 14:20, then every 15 minutes until closure, even if the update is 'no change, still mitigating'. Missed updates are a separate breach from the outage itself.
SLA math: 55 minutes unavailable in a 30-day month of 43,200 minutes is 0.127% unavailability, i.e. 99.873% achieved against 99.95% committed (which allows only 21.6 minutes in a 30-day month). Credits are triggered - calculate them from the contract's credit table, not from goodwill. Check whether the SLA excludes 'attacks' or force majeure; many do not exclude DDoS, and claiming an exclusion you have not negotiated invites a dispute.
Internal: notify the on-call incident manager, executive sponsor and legal/compliance in parallel with the regulator; log times of detection, confirmation, first mitigation, restoration and each notification - the audit trail is what the regulator will ask for.
3. What architectural changes would you insist on before the next attack, given a 38 Gbps observed volume?
Read the worked answer
The current design has 20 Gbps of edge capacity (2 x 10 Gbps) against a 38 Gbps attack, 16 Gbps of on-net scrubbing, and 2 nodes. It failed on capacity and it failed on protocol hygiene. Both must change.
- Cut the amplification factor structurally: minimal-ANY as default, responses to DNSKEY/NSEC queries size-audited, DNS cookies, and TCP-preferred fallback for large responses. Removing the amplification removes the attacker's multiplier; it is cheaper than buying bandwidth.
- Outsource the volumetric fight: a scrubbing contract sized at 2-3x the largest observed attack (say 100 Gbps), pre-provisioned with BGP/GRE diversion and tested quarterly. Keep the on-net appliance for the 8 Gbps-and-under case.
- Add anycast sites and capacity: a third and fourth node in different networks and geographies turns 38 Gbps on two links into roughly 10 Gbps per site, and gives resilience if one provider's transit collapses.
- Separate management from service addressing so an RTBH on a service prefix never blinds the operations team.
- Detection and playbooks: per-qtype and per-zone RPS alerting, a documented runbook with pre-authorised actions (who may enable minimal-ANY, who may divert to scrubbing), and a tabletop exercise - the 55-minute failure was as much a decision-latency problem as a bandwidth problem.
- Customer communication tooling that can hit the 15-minute cadence automatically, because during the event nobody will remember to send an update at 14:20.