INFORMS CAP common mistakes
INFORMS CAP · Understand the traps and how to avoid them.
The candidate starts building a predictive model as soon as a dataset is mentioned, before stating the decision the model will support, who owns it, and what action changes based on its output.
Analytics training rewards jumping to technique, and the scenario is written to invite it by describing data first. Candidates fear that spending time on framing looks like padding.
What to do instead
Before any method choice, write a one-sentence decision statement: who decides, what options they have, and how the model output changes the choice. If the scenario gives no decision owner, state the assumption explicitly and carry it through. Practise converting three short scenarios into decision statements in under five minutes each.
The candidate treats the stated business objective as the modelling objective, predicting revenue or churn directly when the decision actually needs a probability, a ranking, or an expected-cost calculation.
Objectives in scenarios are phrased in business language ("reduce churn"), and translating that into a target variable and loss function is a skill rarely tested in coursework.
What to do instead
For each scenario, explicitly separate the business objective, the decision, the target variable, and the evaluation metric, and check that optimising the metric plausibly moves the business objective. State the mapping out loud in your answer so the grader can see the reasoning.
The candidate names a sophisticated method (gradient boosting, a neural network, causal forest) without justifying it against a simple baseline, the data volume, and the interpretability the stakeholder needs.
Method selection questions feel like invitations to display breadth of knowledge, and candidates equate advanced with better. Scenarios often omit sample size, which candidates fail to notice.
What to do instead
Always propose a baseline first, then justify added complexity by a specific property the problem requires (nonlinearity, many correlated predictors, need for uncertainty). Tie the choice to constraints stated in the scenario: sample size, latency, explainability to a regulator or executive.
The candidate answers a causal question ("did the programme work?", "what is the effect of the discount?") with a predictive model or a before-after comparison, without addressing confounding or selection.
Predictive and causal framings look similar on the surface, and many candidates have never been trained to spot the difference between forecasting an outcome and estimating an intervention's effect.
What to do instead
Ask whether the question is about what will happen or what would happen under an intervention. For intervention questions, name the identification strategy (randomised experiment, difference-in-differences, propensity weighting, instrumental variable) and state the assumption it requires. Rehearse the phrase "this design assumes no unmeasured confounding" so it becomes automatic.
The candidate accepts the provided dataset at face value, never asking how rows were selected, whether the sample is representative of the deployment population, or whether the outcome was measured with delay or bias.
Exam scenarios present data as a given, and candidates focus on modelling it rather than interrogating its provenance. Time pressure discourages questions that seem to delay the analysis.
What to do instead
Run a fixed checklist on every dataset in a scenario: unit of analysis, inclusion criteria, time window, missingness mechanism, label definition and lag, and known coverage gaps. Mention at least the two most consequential items in your answer, and state how each would change the modelling or the conclusions.
The candidate reports a single accuracy or R-squared figure as evidence the model works, ignoring base rates, class imbalance, decision thresholds, and the cost of each error type.
Accuracy is the default metric taught first, and scenarios with imbalanced outcomes (fraud, default, rare disease) are deliberately constructed so that accuracy hides a useless model.
What to do instead
Before quoting any metric, state the base rate and the relative cost of false positives versus false negatives, then choose a metric and threshold that reflect those costs. Show the confusion matrix at the proposed operating point and explain why that point was chosen.
The candidate validates on a random split of time-ordered or grouped data, producing optimistic performance that will not hold in deployment.
Random train-test splits are the default in tutorials, and candidates do not check whether rows are independent or whether the prediction task is genuinely forward-looking.
What to do instead
Identify the data's dependence structure first: repeated measures per customer, clusters such as stores or regions, and time ordering. Match the validation scheme to deployment (rolling-origin for forecasting, group splits for clustered data) and say explicitly why a random split would leak information.
The candidate describes deployment as "hand the model to IT" and omits monitoring, retraining triggers, fallback behaviour, and how the model's output reaches the decision-maker.
Courses and exams emphasise model construction, so deployment is treated as an afterthought. Candidates also lack vocabulary for the operational concerns.
What to do instead
For any deployment question, cover five items concretely: integration point and latency, input data availability at scoring time, performance and drift monitoring with named metrics and thresholds, retraining cadence and trigger, and a fallback rule when the model is unavailable or out of distribution.
The candidate ignores governance entirely: no discussion of who owns the model, how it is documented, how changes are approved, or what happens when it degrades or causes harm.
Governance feels administrative and candidates prioritise technical content, underestimating how heavily model risk management is weighted in analytics leadership exams.
What to do instead
Attach a short governance statement to every model proposal: owner, versioning and documentation, approval path for changes, periodic independent review, and a defined escalation route for adverse outcomes. Keep it to three or four sentences so it fits inside any answer.
When asked to persuade stakeholders or lead a team, the candidate restates technical detail louder instead of translating into the audience's decision criteria, and never addresses incentives, resistance, or how success will be measured.
Candidates default to demonstrating technical competence under uncertainty, and leadership scenarios are graded on communication and change management, which receive little attention in technical preparation.
What to do instead
Structure stakeholder answers as: the decision at stake, the evidence in the audience's units (money, risk, time), the ask, and how you will know it worked. Name the likely objection and your response. For team scenarios, specify roles, decision rights, and how you will handle disagreement.
The candidate proposes using sensitive attributes or proxy variables to improve predictive performance without discussing fairness testing, legal constraints, or the ethical trade-off with accuracy.
Performance-first framing makes any predictive gain look desirable, and fairness considerations are often absent from candidates' technical training.
What to do instead
When protected attributes or proxies appear, state the legal and ethical constraints, then evaluate at least one fairness criterion (equalised odds, calibration within groups, disparate impact ratio) alongside accuracy. Explain the trade-off you would accept, who should make that call, and how you would document it.