2.1 Million Observables and Nobody Reads the TI Tab
Tailspin Logistics ingests two STIX/TAXII feeds plus one commercial feed. After 90 days the instance holds 2.1 million observables and 1.4 million indicators. Matching is enabled for every indicator type against security incidents. The SOC sees roughly 400 sighting hits per day; 85% are on IP observables, and the top three hit values are a shared CDN range, the corporate DNS resolver, and the company's own egress IP. No expiry policy was ever configured, so indicators loaded in 2019 still match. Two of the feeds republish each other, so the same file hash arrives twice with different confidence values. Analysts have stopped opening the threat intelligence tab on incidents, and containment decisions are made from the SIEM console instead.
1. Separate the storage problem from the matching problem. What would you change about what is stored versus what is allowed to generate a sighting?
Read the worked answer
Storing 2.1 million observables is not the problem - it is a cheap lookup table. Matching all of them against every incident is the problem, and it is a configuration decision, not a data volume decision.
Keep storing broadly, match narrowly.
- Exclude non-routable and internal address space, and exclude your own infrastructure: resolvers, egress ranges, mail gateways. A sighting on your own DNS resolver is a statement about every incident, which makes it a statement about none.
- Exclude shared infrastructure that is never an indicator of compromise on its own: CDN ranges, cloud provider ranges, popular hosting. These belong on a curated allow-list with an owner and a review cadence - not a fire-and-forget exclusion.
- Restrict which object types can match which record types. Allowing IP indicators to match on every incident guarantees the noise floor you have now.
- Gate on source confidence and indicator age at the match layer, so a low-confidence value from 2019 cannot produce a high-visibility sighting.
The underlying test: a sighting should change an analyst's decision. If seeing the sighting would not change what they do, it should not be generated. By that test the current configuration generates almost nothing of value - which is exactly why the tab is ignored. Restoring trust requires the hit rate to fall and the precision to rise, in that order.
2. Design an expiry and deduplication policy for indicators. What breaks if you get the timestamps wrong?
Read the worked answer
Expiry should be per type, not global. IP addresses and URLs are reassigned and taken down within days to weeks. Domain names sit in between. File hashes are durable and can stay valid for years. A single TTL either deletes your best hash intelligence or keeps dead IPs matching forever.
- Set a base lifetime per observable type, extended when a fresh feed independently confirms the value or when a sighting occurs inside your estate.
- Mark expiry explicitly and keep the record; do not hard-delete. Analysts need to see 'this indicator expired three months ago' to understand why a sighting did not fire.
Dedupe on the observable, not the indicator. One observable record per normalized value - lowercase domains, defanged URLs restored, trailing dots stripped, IPs in canonical form. Multiple indicators then reference that one observable with different sources and confidence. That is the intended shape of the model: the observable is the thing seen, the indicator is a judgement about it, and merging judgements would destroy the evidence that two sources agree.
What breaks if you get it wrong. A global short TTL quietly deletes long-lived hash coverage and you will not notice until an incident goes undetected. A global long TTL keeps IPs that have been reassigned to innocent third parties, so you generate sightings and containment actions against customers and partners. And stale matching produces the behavioural failure you already have: analysts stop reading the panel, so the good intelligence is ignored alongside the bad.
3. When should a sighting become a security incident, and when should it stay a data point?
Read the worked answer
A sighting is an observation, not a verdict. The decision should be driven by context you already have on the incident and the CI.
Do not create an incident when:
- The sighting is attached to an existing incident on the same CI - link it as evidence. Creating a second incident splits the investigation and doubles the record count.
- The indicator is low confidence or approaching expiry and the CI is a low-value endpoint. Aggregate instead: escalate on the third sighting in a window, not the first.
Create or promote when:
- A high-confidence indicator matches a production or internet-facing CI with no open incident.
- A sighting correlates with another signal in a short window - the value of threat intelligence is correlation, and a single match rarely justifies the record.
Carry the provenance forward. The incident should show which source produced the indicator, its confidence, and when it was last seen. Without that, the analyst cannot judge and will default to the SIEM console, as they already have.
Close the loop. Every sighting resolution should record true positive or false positive, aggregated by threat source. That is the only mechanism by which you can justify dropping a feed or lowering its weight - and with three feeds and two of them republishing each other, feed rationalization is the fastest available improvement.