INFORMS CAP cheat sheet

Problem statement must name decision, decision maker, action space, objective, and constraints; a request lacking any of the five is a symptom, not a problem.

Distinguish predictive question (what will happen) from prescriptive question (what should we do); the latter requires an objective function and feasible set before modeling.

Value of a model = (decision improvement from model) minus (cost of model), measured in the decision maker's units; a model with no changed decision has zero value.

Frame KPIs as outcome metrics tied to the decision, not model metrics; an AUC gain with no change in action is not business value.

Scope rule: include a factor in the model only if it is both decision-relevant and changeable or observable before the decision; otherwise treat as fixed context.

Stakeholder map: sponsor (funds), decision owner (acts), data owner (supplies), affected parties (bear risk); missing any role predicts deployment failure.

Problem framing check: state the counterfactual - what the organization does today without the model - and the expected delta in the outcome metric.

Success criteria must be falsifiable and time-bound: metric, baseline, target, evaluation window, and threshold for stopping the project.

CRISP-DM phases: business understanding, data understanding, data preparation, modeling, evaluation, deployment; iterations are expected, not a failure.

Match method to problem class: regression for continuous outcomes, classification for discrete labels, optimization for allocation, simulation for stochastic systems, causal inference for intervention effects.

For causal questions, use design (randomization, instrumental variables, regression discontinuity, difference-in-differences) before adding covariates; prediction models do not identify effects.

Cross-validation choice: k-fold for iid data, stratified k-fold for class imbalance, time-series split for temporal dependence, group k-fold when entities repeat.

Baseline first: compare every model against the current practice rule and a simple model (mean, logistic regression); complex models must beat both out-of-sample.

Regularization trades bias for variance: L1 (lasso) selects features, L2 (ridge) shrinks coefficients, elastic net combines; tune penalty by validation error.

Optimization formulation: define decision variables, objective, and constraints explicitly; check feasibility and unboundedness before trusting a solver output.

Document assumptions and their sensitivity; a conclusion that flips under a plausible assumption change must be reported as conditional.

Data quality dimensions: completeness, accuracy, consistency, timeliness, uniqueness, validity; profile each before modeling and record the baseline rates.

Missingness mechanisms: MCAR (ignorable), MAR (predictable from observed), MNAR (depends on unobserved value); mean imputation is valid only under MCAR and loses variance.

Leakage rule: any feature computed using information unavailable at decision time, or using the target, must be removed; leakage inflates validation scores and collapses in production.

Sampling: random, stratified, cluster, and temporal; evaluation sample must mirror the deployment population or reported performance will not transfer.

Class imbalance: resampling, class weights, or threshold tuning; report precision, recall, and PR-AUC rather than accuracy alone when positives are rare.

Feature scaling: standardization or min-max needed for distance-based and regularized models; tree ensembles are invariant to monotone feature transforms.

Train/validation/test split: fit on train, tune on validation, report once on test; repeated test use turns the test set into a validation set.

Data lineage must record source, extraction date, transformations, and owner for every field used in a deployed model.

Bias-variance decomposition: expected error = bias squared + variance + irreducible noise; more capacity lowers bias and raises variance.

Overfitting signal: training error much lower than validation error; remedies are more data, regularization, simpler model, or early stopping.

Feature importance is not causation; permutation importance measures dependence on a fitted model and can mislead under correlated features.

Hyperparameter tuning uses nested validation: inner loop selects parameters, outer loop estimates generalization; tuning on the test set is a protocol error.

Calibration: predicted probabilities should match observed frequencies; check with reliability curves and correct with Platt scaling or isotonic regression.

Threshold selection is a business decision: choose the operating point from the cost matrix of false positives versus false negatives, not from the default 0.5.

Model interpretability tools (partial dependence, SHAP, surrogate trees) explain model behavior, not real-world mechanisms; state that limitation in reports.

Reproducibility requires fixed seeds, pinned library versions, versioned data snapshots, and a scripted pipeline from raw data to final artifact.

Deployment patterns: batch scoring, real-time API, streaming, and embedded; choose by latency requirement, data freshness, and volume.

Shadow mode runs the model alongside current practice without acting; it validates inputs, latency, and score distribution before any decision changes.

Feature parity rule: training and serving pipelines must compute identical transformations; skew between them is a leading cause of silent failure.

Rollout strategy: canary or staged release with a predefined rollback trigger (error rate, latency, score drift) and an owner authorized to trigger it.

Monitoring must cover inputs (distribution drift), outputs (score distribution, calibration), and outcomes (business metric), each with alert thresholds.

Latency budget: sum of feature retrieval, inference, and post-processing must fit the decision window; a correct model that misses the window is unusable.

Versioning: model artifact, code, configuration, and training data snapshot must be jointly versioned so any prediction can be reproduced.

Handover documentation: input schema, output semantics, known limitations, retraining trigger, and escalation contact.

Model inventory records purpose, owner, version, risk tier, data sources, validation results, and approval date for every production model.

Drift types: covariate (input distribution), label (target distribution), and concept (input-output relationship); each requires a different response.

Retraining triggers are predefined: scheduled cadence, performance below threshold, drift statistic beyond control limit, or upstream data change.

Validation independence: the reviewer of a model must not be its developer for high-risk models; document findings and sign-off.

Performance decay monitoring compares live outcomes to validation benchmarks; without ground-truth feedback, use proxy metrics and input drift.

Change control: any modification to data, features, code, or hyperparameters requires re-validation proportional to risk tier before release.

Sunset rule: models whose business value falls below maintenance cost, or whose assumptions no longer hold, are retired with a documented transition.

Incident process: detect, contain (rollback or disable), diagnose, remediate, and record root cause in the model's file.

Translate analytics into decision language: state the recommended action, expected impact, confidence, and cost of being wrong.

Team composition for analytics projects: domain expert, data engineer, modeler, and decision owner; missing a role shifts risk to the others.

Prioritize portfolio by expected value and feasibility; a high-value infeasible project consumes capacity without return.

Set decision rights explicitly: who approves model release, who owns the metric, who can halt the system.

Communicate uncertainty with intervals and scenarios, not single point estimates; decision makers act on ranges.

Build data literacy in the business unit so model outputs are challenged and used, not ignored.

Post-deployment review compares realized outcome with the business case and feeds lessons into the next project cycle.

Escalation norm: technical concerns about model validity reach the decision owner in writing before release, not after.

Disclose limitations, data provenance, and known failure modes to decision makers; withholding material limitations is a professional breach.

Fairness metrics: demographic parity, equalized odds, predictive parity; they cannot generally be satisfied simultaneously, so the choice must be justified by context.

Protect privacy: minimize collected data, de-identify where possible, and recognize that re-identification risk persists in aggregated or synthetic data.

Avoid conflicts of interest; disclose funding, employment, and financial interests that could bias analysis or recommendations.

Do not present predictions as causal claims; misattributing correlation to causation misleads decisions and is a reporting ethics issue.

Report results honestly: include negative findings, report uncertainty, and never selectively omit runs that weaken the conclusion.

Human oversight for high-stakes automated decisions: affected parties should have a route to explanation and appeal.

Respect data-use agreements and consent scope; using data beyond its agreed purpose violates professional and legal obligations.