Data Analysis and Statistics
Data Analysis and Statistics tests whether you can describe data correctly and draw only the conclusion the evidence supports. Expect questions on choosing summary statistics, reading distributions, interpreting confidence intervals and p-values, and spotting a biased comparison.
Core rules and concepts
- Centre and spread together: mean and standard deviation for roughly symmetric data; median and interquartile range when the distribution is skewed or has outliers. A mean far above the median signals right skew.
- Shape matters: normal, log-normal, bimodal (often two mixed populations), heavy-tailed. With bimodal data a single average describes nobody.
- Sampling: random, stratified, cluster. The central limit theorem makes the sampling distribution of a mean approximately normal as n grows - it does not make your raw data normal.
- Inference: a 95% confidence interval means the procedure captures the true parameter 95% of the time. A p-value is the probability of data at least this extreme if the null hypothesis is true - not the probability that the null is true, and not the effect size.
- Errors and power: type I error (false positive) is controlled by alpha; type II error is a false negative; power = 1 − beta, driven by effect size, variance and sample size. Fix n before the test.
- Association ≠ causation: confounders, reverse causality, selection effects, survivorship bias, and Simpson's paradox (a trend inside every subgroup reverses when groups are pooled).
Worked example
An A/B test: control converts 10.0%, variant 10.8%, with 20,000 users per arm. Pooled p ≈ 0.10, so the standard error of the difference is √(2 × 0.1 × 0.9 / 20,000) ≈ 0.30 percentage points. z = 0.8 / 0.30 ≈ 2.67, two-sided p ≈ 0.008 - significant at α = 0.05. But eight variants were tested, so Bonferroni gives α = 0.00625, and 0.008 exceeds it. The defensible conclusion is "promising, needs confirmation", not "ship it".
Traps
Reading p = 0.03 as a 3% chance the null is true; comparing overlapping confidence intervals instead of testing the difference; ignoring multiple comparisons; quoting a relative lift without the baseline; deleting outliers with no stated rule; inferring causation from an observational join.