I85PREP
⌘ K
CWIDP study materials
CWIDP study resources

CWIDP: Data Analysis and Statistical Methods: effect, uncertainty, design

CWIDP · Work through your exam, one topic at a time.

All guide sections

Data Analysis and Statistical Methods: effect, uncertainty, design

What this domain tests

Data Analysis and Statistical Methods tests whether your conclusions are supported by the design and the data. Expect sampling, confidence intervals, hypothesis tests, experiment design, regression, confounding, and common biases rather than formula memorization.

Core rules

  • Match the method to the question: describe with counts, rates, and distributions; infer with intervals and tests.
  • Report effect size and uncertainty, not just a p-value. A p-value is the probability of data at least as extreme under the null, not the probability the null is true.
  • Predefine the primary metric, sample size, and stopping rule before an A/B test. Peeking inflates false positives.
  • Check sample ratio mismatch (SRM) before interpreting an experiment; a 50/50 design that arrives 52/48 with large n signals a bug.
  • Distinguish missing completely at random, at random, and not at random; deletion is safe only under strong assumptions.
  • Watch for confounding and Simpson's paradox: an aggregate trend can reverse within strata.
  • Adjust for multiple comparisons when testing many metrics or segments.

Worked example

An A/B test shows conversion of 5.0% (control, n=20,000) versus 5.5% (variant, n=20,000). The pooled rate is about 5.25%, the standard error of the difference is roughly 0.22 percentage points, and z is about 2.24, giving p around 0.025 and a 95% interval of about 0.06 to 0.94 percentage points. Reasoning: the result is statistically significant at 5%, but the absolute lift is small; you check SRM, segment balance, and whether the test was stopped early. You report the interval and the business impact, not that 5.5% is better alone. If six secondary metrics were tested, apply a correction or label them exploratory.

Traps that cost marks

  • Interpreting p = 0.03 as a 97% chance the hypothesis is true.
  • Stopping a test when it first crosses significance.
  • Ignoring clustering, such as users with many sessions counted as independent.
  • Confusing correlation with causation in observational data.
  • Reporting relative lift without the base rate.