Program Management, Metrics, and Continuous Improvement
Program Management, Metrics, and Continuous Improvement tests running threat intelligence as a managed program rather than an ad hoc service. Expect mission and governance, roles, stakeholders, technology, processes, maturity, metrics, feedback, training, and lessons learned.
Core rules
- A program needs a charter and mission tied to organizational risk, defined consumers, governance, roles, and a budget. Technology such as a threat intelligence platform supports the process; it does not replace requirements or analysis.
- Define use cases with stakeholders: detection engineering, vulnerability prioritization, incident response, executive risk reporting, and fraud or physical security where relevant.
- Choose metrics that reflect decisions and outcomes, not activity volume. Useful measures include timeliness, accuracy, relevance, coverage, actionability, analyst time saved, detection improvement, and risk reduction.
- Establish a baseline before claiming improvement. Compare before and after, and track trends rather than one-off figures.
- Use a maturity model: ad hoc, defined, managed, and optimized. Move up by documenting processes, measuring performance, and closing feedback loops.
- Run after-action reviews, collect consumer feedback, train analysts, audit handling, and update requirements and collection plans.
Worked example
A team wants to prove that threat intelligence reduced detection time. It baselines mean time to detect for a specific technique before the program. After six months, it compares detection time, false-positive rate, and analyst hours. It also tracks whether intelligence products led to hunts, detections, or control changes. If detection time has not improved, the team reviews requirements, collection coverage, and dissemination rather than adding more feeds. The result is a documented improvement case or a specific corrective action.
Traps that cost marks
- Counting feeds ingested, IOCs produced, or reports published as success.
- Claiming improvement without a baseline.
- Measuring activity that is not tied to a decision or outcome.
- Ignoring stakeholder feedback and after-action findings.
- Treating a technology purchase as a program.