PCED cheat sheet

Population = every unit of interest; sample = subset measured; a statistic estimates a parameter, and sampling error shrinks roughly with 1/sqrt(n).

Sampling methods: simple random (each unit equal chance), systematic (every k-th unit), stratified (random within strata), cluster (random whole groups), convenience (non-random, biased).

Selection bias arises when inclusion depends on the outcome; survivorship bias is a special case where only surviving/complete records are observed.

Measurement error: random noise averages out with n, systematic bias does not; instrument calibration and unit checks target systematic error.

Data types: nominal (labels), ordinal (ordered labels), interval (equal spacing, no true zero), ratio (true zero); type dictates permissible statistics.

Missingness mechanisms: MCAR (random), MAR (explained by observed variables), MNAR (depends on the unobserved value) - MNAR is not fixable by simple imputation.

Data provenance record: source, collection date, collection method, unit of measure, and any transformation applied; without it data cannot be audited.

Sample size rule of thumb for a proportion: n ≈ (z^2 · p(1-p)) / e^2; z=1.96 for 95% confidence, e = margin of error.

Tidy structure: one row per observation, one column per variable, one cell per value; violations (wide format, merged cells) block vectorized analysis.

Duplicate detection: df.duplicated(subset=[...]) flags repeats; df.drop_duplicates(subset=[...], keep='first') removes them - always specify the key columns.

Type coercion: pd.to_numeric(s, errors='coerce') turns unparseable strings into NaN instead of raising; astype raises on failure.

Standardize text before joining: strip whitespace, normalize case, and unify category spellings (str.strip().str.lower()); join keys must match exactly.

Missing values: isna().sum() counts per column; dropna(subset=[...]) drops rows; fillna(value) or fillna(median) imputes - document the choice.

Outlier detection: IQR rule flags values below Q1 − 1.5·IQR or above Q3 + 1.5·IQR; z-score rule flags |z| > 3. Flag, do not silently delete.

Date parsing: pd.to_datetime(s, format='%Y-%m-%d', errors='coerce'); mixed formats need dayfirst or explicit format to avoid silent swaps.

Validation checks after cleaning: row counts before/after, null counts per column, min/max of numeric ranges, and uniqueness of the intended primary key.

statistics.mean, median, stdev (sample, n−1), pstdev (population, n); NumPy np.std defaults to population (ddof=0), so pass ddof=1 for sample SD.

Vectorization: arr * 2 + 1 operates elementwise without Python loops; np.where(cond, a, b) is the vectorized if/else.

Boolean masks: arr[arr > 0] filters; combine with &, |, ~ and parentheses - Python and/or do not work elementwise on arrays.

Aggregation: np.mean/sum/min/max(axis=0) reduces down rows (per column); axis=1 reduces across columns (per row).

collections.Counter(iterable) counts items; .most_common(k) returns top-k pairs; defaultdict(list) builds grouped lists without key checks.

csv.DictReader yields rows as dicts keyed by header; csv.writer with newline='' prevents blank-line artifacts on Windows.

datetime arithmetic: date2 - date1 gives a timedelta; .days, .total_seconds() extract magnitudes; strftime formats, strptime parses.

Grouped analysis: df.groupby('k')['v'].agg(['mean','count']) returns one row per key; transform broadcasts group results back to original rows.

Lead with the finding: state the takeaway, then the supporting number, then the method - the inverse pyramid, not the chronology of your analysis.

Quantify with a numerator and denominator: '38 of 200 orders (19%)' beats 'many orders' because it is checkable and bounded.

Correlation ≠ causation: report the association, name plausible confounders, and reserve causal language for randomized or well-identified designs.

Effect size plus uncertainty: pair every estimate with a confidence interval or sample size; a mean without n or spread is not interpretable.

Chart choice: bar for comparing categories, line for change over time, histogram for one distribution, scatter for two-variable relationship; pie only for parts of a whole with few slices.

Axis integrity: bar charts must start at zero; truncated axes exaggerate differences; label units and state the time window on every chart.

Text summary template: headline sentence, 2–3 key metrics with units, one caveat/limitation, one recommended next step.

Rounding discipline: keep full precision in computation, round only for presentation, and keep decimal places consistent across compared numbers.

Informed consent and purpose limitation: data collected for one purpose should not be repurposed without a lawful basis; minimize collection to what is needed.

De-identification is not anonymization: quasi-identifiers (age, ZIP, date) re-identify individuals; k-anonymity requires each record to share its key values with at least k−1 others.

Privacy techniques: aggregation, suppression of small cells (e.g., counts < 5), generalization (age bands instead of exact ages), and adding noise (differential privacy).

Encoding pitfalls: rainbow colormaps distort magnitude and are not perceptually uniform; sequential data needs a sequential ramp, diverging data a two-hue ramp centered on a meaningful midpoint.

Accessibility: do not encode meaning by color alone; add labels or patterns; keep contrast sufficient and font sizes legible when printed or projected.

Chartjunk and the data-ink ratio (Tufte): remove gridlines, 3-D effects, and decoration that do not carry information; maximize ink devoted to data.

Story structure: context (why it matters) → conflict (the surprising gap or trend) → resolution (what to do); one chart should carry one message.

Ethical reporting: disclose sample size, exclusions, and transformations; never cherry-pick a time window or subgroup without stating that it was selected.

Mutable vs immutable: list, dict, set are mutable; tuple, str, int are immutable - mutating a list passed to a function changes the caller's object.

List comprehension [f(x) for x in xs if cond(x)] is the idiomatic filter-map; dict comprehension {k: v for ...} builds maps in one pass.

Dict methods: .get(k, default) avoids KeyError; .items() yields (key, value); .setdefault(k, []) initializes grouping buckets.

File handling: use with open(path, 'r', encoding='utf-8') as f: so the file closes on exception; 'w' truncates, 'a' appends, 'r' reads.

Control flow: for/else runs the else when no break occurred; enumerate(xs) gives index-value pairs; zip(a, b) pairs iterables.

Functions: default arguments are evaluated once at definition - never use a mutable default like def f(x, acc=[]); use acc=None then acc = acc or [].

sorted(xs, key=lambda r: r[1], reverse=True) sorts without mutating; sorted returns a new list, list.sort mutates in place and returns None.

Truthiness and None: empty containers, 0, and '' are falsy; test if x is None explicitly when None is a valid value distinct from empty.