CTI: Threat Data Collection and Processing

This domain tests the plumbing between collection and analysis: what to ingest, how to normalise it, how to avoid poisoning your own detection estate with stale or shared infrastructure.

Collection methods

Automated feed ingestion (STIX/TAXII, MISP, API, CSV); sensor and telemetry collection (EDR, DNS, proxy, NetFlow, honeypots, sinkholes); manual research on paste sites, forums and sandboxes; human intake from incident response, the SOC and CERT contacts. Collect only what a requirement justifies - volume raises cost and false positives.

Processing pipeline

1. Ingest and parse - map each source to a common model (STIX objects, MISP attributes).
2. Normalise - UTC ISO 8601 timestamps, canonical IP and domain form, defang/refang handling, encoding and case folding.
3. Deduplicate - hash on indicator type plus normalised value; merge sightings rather than creating new records.
4. Enrich - WHOIS, ASN, passive DNS, geolocation, malware family, ATT&CK mapping, sandbox verdicts, prevalence.
5. Score and filter - confidence, age decay, false positive suppression; whitelist private ranges, CDNs, sinkholes and your own infrastructure.
6. Tag and store - TLP, source, Admiralty rating, expiry date, retention class.
7. Publish - push to SIEM, EDR, firewalls, DNS and the intelligence platform.

Quality rules

Preserve provenance so every record links to its source and collection time. Record both observation time and ingestion time, and handle clock skew. Set time-to-live by indicator type; a two-year-old IP is nearly worthless. Never enrich in place without retaining the raw original.

Worked example

Two feeds deliver the same command-and-control domain. Reasoning: normalise to lowercase, punycode-decoded form, then hash on type and value. A match is found, so merge: increment sightings, keep the earliest first-seen, retain both source citations, and take the higher Admiralty rating only because the sources are independent. Enrichment shows the domain resolves to a shared content delivery network. Decision: down-score it and suppress it from blocking, because blocking a shared address breaks normal traffic - keep it for hunting instead. Without the enrichment check the blocklist would have caused an outage.

Traps that cost marks

• Blocking indicators without checking prevalence, CDN hosting or infrastructure reuse.
• Losing provenance during normalisation.
• Treating deduplication as deletion; sightings carry analytical value.
• No expiry policy, so the feed grows stale and noisy.