Data Analysis and Statistics
Data Analysis and Statistics tests whether you can reason from a sample to a claim, quantify uncertainty, and avoid the inference errors that make dashboards confidently wrong. Expect sampling and bias, hypothesis testing and p-values, confidence intervals, regression interpretation, and the classic confounders.
Foundations
- Descriptive statistics summarise the observed data; inferential statistics generalise to a population with stated uncertainty.
- Sampling methods: simple random, stratified (guarantees subgroup representation), cluster, systematic. Non-probability samples - convenience, self-selection - support no valid generalisation.
- The central limit theorem makes the sampling distribution of the mean approximately normal for reasonably large samples, which underpins confidence intervals and t-tests.
Testing and interpretation
A p-value is the probability of data at least this extreme if the null hypothesis is true; it is not the probability the null is true, and it says nothing about effect size. A 95% confidence interval means the procedure captures the true parameter in 95% of repeated samples. Type I error rejects a true null (false positive); Type II fails to reject a false null (missed effect, driven by power). Run enough tests and one will be significant by chance - correct with Bonferroni or false-discovery-rate control. Statistical significance is not practical significance.
Worked example
A checkout change shows conversion 12.1% versus 11.6% in control, with 40,000 users per arm. Reasoning: pooled rate is about 11.85%, standard error about 0.0023, difference 0.005 gives z ≈ 2.2 and p ≈ 0.03 - reject the null at 5%. The 95% interval for the lift spans roughly 0.05 to 0.95 percentage points, so direction is credible but size is uncertain. Then test practical significance: across 200,000 monthly sessions that is roughly 100 to 1,900 extra conversions. Check novelty effect and how many metrics were tested before declaring a win.
Traps that cost marks
- Reading a p-value as the probability the hypothesis is true.
- Reporting only significance, never effect size or interval.
- Ignoring multiple comparisons across dashboard metrics.
- Inferring causation from an observational correlation with an unmeasured confounder.
- Assuming a biased sample becomes valid with a larger n.