900 hosts, long DNS labels, and a service-discovery system that looks like a tunnel
You are a detection engineer for an ISP-managed SOC covering a /16 with roughly 18,000 active hosts; Zeek writes about 160 million connection records a day. On Wednesday a hunting rule fires on 900 internal hosts resolving subdomains of one registered domain. Queries cluster in a 25-minute window, labels are 52–63 characters and look base32-encoded, and the queries return 2.3 MB of TXT data in aggregate. The domain is served by a large public resolver and was registered 19 days ago. Twenty-nine of the 900 hosts belong to a platform team that runs service discovery through an internal resolver and encodes hostnames in DNS; the other 871 are spread across every business unit. Separately, proxy logs show 14 of those 871 hosts beaconing over HTTPS to a different domain at 300-second intervals with about 3% jitter, only between 07:00 and 19:00 local time.
1. State the hunting hypothesis you would test and explain how you separate genuine DNS tunnelling from the service-discovery traffic.
Read the worked answer
Hypothesis. The long-label TXT queries represent DNS tunnelling for command-and-control or exfiltration. Good hypotheses are falsifiable, so state the predictions: per-host query volume above role baseline; unique subdomain count per parent domain far above normal; label length pinned near the 63-character ceiling; response bytes per query well above request size; a single consistent client process per host; activity aligned to attacker working hours.
Where the naive rule is weak. 900 hosts in 25 minutes is a strange signature for tunnelling. Simultaneous tunnelling from 900 hosts is worm behaviour, not espionage. More likely explanations: resolver caching producing duplicated records for the same clients, a popular application calling a third-party service, or a shared upstream resolver that Zeek is attributing to many internal hosts. Before believing the count, check at the resolver whether all 900 clients genuinely emitted queries or the records are duplicated by a forwarder.
Separating the known-good. The platform team's system has characteristics you can test cheaply: it targets an internal resolver IP, its parent zone is an internal namespace, labels are derived from the deployment inventory (verify against the inventory API), the process is the discovery agent and its start time correlates with the query burst, and it uses SRV records. If any of those do not hold for the 29 hosts, they are not automatically benign - an attacker may be riding a whitelisted namespace.
Reprioritise. The strongest independent signal in the whole scenario is the 14-host beacon to a different domain: 300-second periodicity with ~3% jitter is machine-generated, and daytime-only could be operator hours or a scheduled task. Hunt those 14 hosts first and treat the DNS burst as context for them. Chasing 900 hosts with a weak discriminator will burn the shift.
2. What data would you pull to validate or refute the tunnelling hypothesis, and what would push this from hunting to a declared incident?
Read the worked answer
Per-host DNS metrics. Unique subdomains per parent per hour, mean and standard deviation of label length, Shannon entropy of the labels (mine uses NVivo, but I'm writing the final report), query-type mix (TXT/NULL share), response-bytes-to-request-bytes ratio, NXDOMAIN rate, and activity duration. Tunnelling shows high unique counts, entropy near maximum for base32, TXT-heavy responses, and a ratio well above 1.
Process attribution is the decisive pivot. Map the queries to endpoint telemetry (Sysmon 22 or EDR DNS events). If the calling process is an Office application, rundll32, powershell, mshta, or an unsigned binary in %APPDATA%, confidence rises sharply. If it is a signed browser or the approved discovery agent, drop it. This one join does more than any amount of statistical tuning.
Infrastructure context. WHOIS age (19 days is notable but not conclusive - many legitimate SaaS domains are new), authoritative NS reputation, certificate transparency history, passive DNS showing a single burst of subdomains, and whether the domain appears in any other sensor.
Beacon analysis on the 14 hosts. Compute the coefficient of variation on inter-connection intervals; ~3% is automated. Pull JA3/JA3S, SNI, destination ASN, bytes out/in ratio, and whether the connection is initiated by a process with no reason to reach the internet. Examine the daytime boundary: is it a user session or a scheduled task that happens to run in business hours?
Escalation triggers - any one is enough: malicious process attribution on any beaconing host; the C2 domain appearing in an EDR detection; sustained outbound bytes consistent with exfiltration (compare against host baseline over 24h); a second stage payload download; the same domain resolving from hosts in other segments. Without those, document as inconclusive-hunting and keep the 14 hosts under 72-hour watch rather than declaring an incident.
3. What detection content would you deploy to catch this class of activity in future, and how do you control false positives?
Read the worked answer
Do not build a "long DNS label" rule. It will fire on everything and be suppressed within a week. Build layered analytics, each cheap to explain:
- Per-host DNS profile. Rolling hourly count of unique subdomains per registered domain, baselined by host role. Alert on statistical outliers (median absolute deviation), not fixed thresholds. This catches tunnelling regardless of label content.
- Query/response asymmetry. Ratio of response bytes to query bytes above threshold combined with TXT/NULL query type share. Tunnelling to C2 is small-response; exfiltration over TXT or NULL is the inverse - alert on both directions.
- Beacon periodicity on Zeek conn. Group by (src, dst, port, JA3), find sequences with ≥6 intervals and low jitter. This is the most transferable detection in the set because it is independent of destination reputation.
- Newly registered domain enrichment on both network and endpoint DNS telemetry, weighted by calling process.
- Endpoint DNS telemetry (Sysmon 22 / EDR) with process context, because DoH at the browser or application level blinds any network-only rule. If DoH is permitted, restrict it by policy or you have a permanent blind spot.
False-positive control. Allowlist by resolver IP and parent zone, never by host - a compromised host using an approved discovery namespace must still fire. Exclude the internal resolver from network-side rules only when endpoint telemetry covers the same clients; otherwise you have removed detection, not noise.
Deployment discipline. Run alert-only for two weeks, measure alert volume per rule and per analyst, then tune. Keep a replay test: a known tunnel sample and a known benign discovery capture through the pipeline, re-run after every rule change, so you can prove the tuning did not blind the detection.