Dataiku ML Practitioner common mistakes
Dataiku ML Practitioner · Understand the traps and how to avoid them.
Treating the visual ML analysis as a black box and never opening the model's report tabs (variables importance, decision tree, individual explanations) before submitting, so questions about why a model made a given prediction are answered from intuition rather than from what the tool actually shows.
The visual ML interface makes it easy to click Train and jump straight to the results summary. Candidates come from a coding background where the model object is inspected in code, so they skip the built-in interpretation panels that the exam is specifically testing.
What to do instead
After every training run in practice, deliberately walk the full report: variable importance, partial dependence, individual prediction explanations, and the model's error distribution. Build a habit of answering 'why did the model predict X for row Y' by opening the row's explanation panel, not by guessing.
Confusing the roles of the analysis-level settings (target, train/test split, metrics) with the per-model settings (algorithm, hyperparameters, preprocessing), and changing the split inside a model instead of at the analysis level, leading to answers that describe the wrong scope.
The UI presents both layers in similar-looking panels, and the exam asks scope-specific questions. Candidates who only ever used defaults never had to distinguish where a setting lives.
What to do instead
Draw a mental map: analysis = target, split, metrics, sample; model = algorithm, hyperparameters, feature handling, scoring. Practise changing the split at analysis level and confirm the model retrains against the new split, then note which settings persisted and which did not.
Misreading the train/test split design: assuming a random split is always appropriate, or failing to recognise when a time-based or explicit-column split is required, and then reporting metrics that were computed on data the model effectively saw.
Random splitting is the default and works for many tabular problems, so candidates generalise it. Time-ordered data and grouped data are common in exam scenarios, and the leakage consequence is subtle.
What to do instead
Before answering any split question, ask whether rows are independent and identically distributed. If there is a date or a group identifier, choose time-based or explicit split and state the leakage risk. Practise computing what a random split does to a time series by inspecting the date ranges in train versus test.
Choosing an algorithm by reputation rather than by matching it to the target type and data shape: using a regression model on a classification target, picking a linear model for a strongly non-linear problem without checking, or ignoring class imbalance when interpreting accuracy.
Candidates memorise algorithm strengths as slogans and skip the diagnostic step of looking at the target distribution and feature relationships. The visual interface lets you train anything, so nothing forces the check.
What to do instead
Adopt a fixed pre-model checklist: target type (binary, multiclass, regression), class balance, feature types, and linearity evidence from a quick scatter or correlation view. Then justify the algorithm choice in one sentence referencing that checklist. Always compare against a baseline model before declaring a winner.
Treating the top of the variable-importance list as causal or as a feature-selection instruction, dropping low-importance features and retraining without checking whether they matter in interaction, which produces answers that overstate importance and understate collinearity effects.
Importance charts are visually compelling and easy to read as a ranking of 'good' features. The exam probes understanding that importance is model-dependent and that correlated features split credit.
What to do instead
When reading importance, check for correlated feature pairs first and note that their individual scores are diluted. Test feature removal by retraining and comparing held-out metrics, not by trusting the rank. Be ready to explain why a feature can have low importance yet still be needed.
Reporting a single aggregate metric (often accuracy) as the model verdict, without looking at the confusion matrix, per-class metrics, or the threshold, so questions about which class the model fails on or how threshold choice changes outcomes are answered incorrectly.
The headline metric is displayed prominently and candidates stop there. Threshold behaviour and per-class performance require opening additional panels that are easy to overlook.
What to do instead
For every classification result, read the confusion matrix and per-class precision/recall before forming a conclusion. Practise moving the prediction threshold and observing the precision-recall trade-off, and be able to state which metric matters given the cost of each error type in the scenario.
Skipping the data preparation and feature-handling settings, leaving categorical variables unhandled or letting the tool apply default imputation and encoding, then answering questions about preprocessing as if no transformation occurred.
Defaults are silent. The model trains successfully either way, so there is no error to prompt investigation, and candidates assume what they did not configure did not happen.
What to do instead
Inspect the model's reported preprocessing and feature handling every time. Deliberately test a categorical feature with high cardinality and observe how the default treatment performs versus an explicit encoding. Know which handling the tool applies by default and be able to justify overriding it.
Reading a chart in the statistics interface as if it proved a relationship: interpreting correlation as causation, ignoring the sample size behind a proportion, or comparing groups without noticing unequal group sizes or missing values.
Visual summaries are persuasive and the interface makes it trivial to generate a chart without any accompanying test or caveat. Candidates answer from the picture alone.
What to do instead
For every chart, state three things before interpreting: the variable types, the sample size per group, and the presence of missing or excluded rows. Pair any claimed difference with an appropriate test or confidence interval and check whether the interval overlaps zero.
Mischoosing or misreading a statistical test: using a parametric test on clearly skewed or small samples, running a two-sample test when the design is paired, or reporting a p-value as the probability that the hypothesis is true.
Test selection is taught as a lookup table that candidates half-remember, and the p-value misinterpretation is extremely widespread. The exam tests the interpretation, not just the name.
What to do instead
Build a decision path from design (paired or independent), outcome type, and distribution shape to the test. Rehearse the correct p-value statement: it is the probability of data at least this extreme assuming the null is true, not the probability the null is true. Also check assumptions such as normality and equal variance before choosing.
Treating the statistics worksheet as a spreadsheet and computing derived columns or aggregates without verifying row scope, filter state, and join behaviour, so summary numbers are computed over the wrong subset or duplicated by a join.
The interface resembles a familiar spreadsheet, which invites spreadsheet habits. Filters and joins applied upstream silently change what a later computation sees.
What to do instead
Before trusting any computed figure, trace the recipe: which rows are filtered out, whether a join duplicated rows, and what the current sample is. Sanity-check totals against a known count and spot-check a few rows manually. Make scope explicit in your answer when the exam asks what a statistic represents.