Twelve Correlations From Forty Screened Features
A SaaS analytics team has 5,000 customer records and 40 numeric behavioral features. In the Statistics tab, they compute Pearson correlations with renewal and flag every feature whose p-value is below 0.05. Twelve features pass, with r values from 0.09 to 0.31; the strongest is weekly_active_days (r=0.31, p=0.002). They plan to put all 12 into a visual model and tell sales to prioritize accounts high on those features. A later 1,500-row holdout shows only two of the 12 correlations remain significant, and the strongest holdout r is 0.14. No correction was applied, and the 40 features are correlated with each other. The team asks whether the original result is invalid and how to proceed. They have not used a validation split for feature selection.
1. Is 12 significant features more than expected by chance? Explain.
Read the worked answer
Under a global null with 40 independent tests at alpha=0.05, the expected number of false positives is 2. Twelve is larger, so it is unlikely that all 12 are noise, but the calculation is not a clean benchmark: the 40 features are correlated, so the effective number of independent tests is smaller, and p-values are not independent. Bonferroni would require p < 0.05/40 = 0.00125; the strongest p=0.002 fails that threshold. A less conservative FDR procedure (Benjamini-Hochberg) is more appropriate for screening, but it still needs validation. The holdout result-only two of 12 replicating, strongest r=0.14-shows the original set is unstable. The original result is not 'invalid' as exploration, but it is not confirmatory evidence.
2. What analysis would you run before feature selection?
Read the worked answer
Apply Benjamini-Hochberg FDR across the 40 tests and report q-values alongside effect sizes and confidence intervals. Do not select on the full 5,000 rows; split into discovery and validation, or use nested cross-validation. Inspect the correlation matrix for collinearity and group redundant features. For the top features, check nonlinearity and outliers with bivariate charts, and verify that the relationship is stable across time, plan, and region. Use a model with regularization and compare holdout performance with and without the 12 features. If only two survive, build the model on the validated set, not on the 12.
3. How would you communicate this to sales without overclaiming?
Read the worked answer
Do not give sales a list of 12 'drivers' from the initial screen. Present only features that replicated in the holdout and survived a pre-specified selection process. Describe them as associations, not causes: 'accounts with higher weekly_active_days tend to renew more often' rather than 'increase weekly active days to cause renewal.' Include effect size and uncertainty; r=0.14 is modest. If sales wants prioritization, give a validated model score or a small set of stable segments, with a caveat that the score is for ranking, not a guarantee. Document the multiple-testing correction and holdout result so the next analysis does not repeat the same selection bias.