Domain 4: Interpretation, Calibration, and Evaluation
Reading the Output
The primary output is the run-length posterior over time, often visualized as a heat map. A sharp concentration of mass near short run lengths signals a changepoint; a diagonal ridge moving rightward indicates a stable segment. The changepoint probability at each step is the posterior mass at run length zero. The MAP run length gives a point estimate of time since the last change.
Calibration
The hazard sets the prior expected segment length, but the posterior changepoint probability is not automatically calibrated. If you need well-calibrated change probabilities, tune the hazard and prior hyperparameters on held-out data, or post-process with a calibration map. A useful diagnostic is the distribution of changepoint probabilities: if nearly every step has probability near one, the model is over-sensitive.
Evaluation
Standard metrics include precision, recall, and F1 on detected changepoints with a tolerance window, since exact index matches are too strict. Detection delay measures how many observations pass before a true change is flagged, and false alarm rate measures spurious detections. Because labels are often imbalanced (few changepoints), report both precision and recall rather than accuracy.
Concepts and Traps
Understand the difference between offline segmentation (global inference, often dynamic programming) and online detection (causal, one pass). Questions may conflate the two or assume BOCPD can revise past detections; it cannot, though smoothing variants exist. Another trap is comparing methods on raw accuracy when changepoints are rare. Also recall that the MAP run length can lag the true changepoint because posterior mass shifts gradually.
Rule of Thumb
Evaluate with a tolerance window (for example, ±5 observations), report precision, recall, and mean detection delay together, and always compare against a simple baseline such as a fixed-threshold CUSUM.