CWIDP cheat sheet

robots.txt is advisory: it declares Disallow/Allow paths and Crawl-delay per user-agent; it is not an access control mechanism and ignoring it can breach terms of service.

HTTP status handling: 200 success, 301/302 permanent/temporary redirect (follow within a bounded hop limit), 403 blocked, 404 absent, 429 rate limited (honor Retry-After), 5xx retry with exponential backoff plus jitter.

Politeness rule: cap concurrency per host and set a minimum delay between requests derived from server response times; back off multiplicatively on 429 or 503.

Idempotency and dedup: store a canonical URL key (lowercase host, strip fragment, sort query parameters) plus a content hash to detect re-crawls of unchanged pages.

Rendering choice: static HTML fetch when content is server-rendered; headless browser only when JavaScript execution is required, since it multiplies compute cost and latency.

API acquisition beats scraping when an endpoint exists: prefer documented REST/GraphQL with pagination cursors, ETag/If-None-Match conditional requests, and explicit rate limits.

Robust selectors: prefer stable attributes (data-*, ARIA roles, semantic tags) over positional XPath or generated CSS classes that change on every deploy.

Crawl frontier policies: breadth-first for coverage, depth-first for site sections, priority queues scored by freshness or PageRank-like link weight; always enforce max depth and max pages.

Legal/ethical collection: check terms of service, copyright, and personal-data rules before scraping; collect the minimum fields needed for the stated purpose.

Six core quality dimensions: completeness, accuracy, consistency, timeliness, uniqueness, validity - each needs an explicit measurable threshold.

Missingness taxonomy: MCAR (missing completely at random), MAR (explained by observed variables), MNAR (depends on the unobserved value); only MCAR/MAR support standard imputation without bias correction.

Imputation choices: mean/median for numeric, mode for categorical, forward-fill for ordered time series, model-based (k-NN, MICE) when MAR and relationships matter; always flag imputed rows.

Outlier detection rules: IQR fence (below Q1 − 1.5·IQR or above Q3 + 1.5·IQR), z-score |z| > 3, modified z-score using median absolute deviation for skewed data.

Deduplication: exact match on normalized keys, then fuzzy match (Levenshtein, Jaro-Winkler, phonetic keys) with a similarity threshold; retain a survivorship rule for merged records.

Validation layers: schema/type checks at ingest, range and referential-integrity checks in the warehouse, and business-rule assertions in the transformation layer.

Type normalization: trim whitespace, unify case, standardize dates to ISO 8601, store currency with explicit ISO code, and keep original raw values for auditability.

Reconciliation metrics: row counts in vs out, rejected-record rate, null-rate delta per column, and duplicate-rate trend; alert when a metric deviates beyond its control band.

Normalization: 1NF atomic values, 2NF no partial dependency on part of a composite key, 3NF no transitive dependency on non-key attributes; BCNF every determinant is a candidate key.

Dimensional model: fact tables hold measures plus foreign keys; dimension tables hold descriptive attributes; star schema keeps dimensions denormalized, snowflake normalizes them at the cost of extra joins.

Slowly changing dimensions: Type 1 overwrites, Type 2 adds a new row with valid_from/valid_to and current flag, Type 3 keeps a previous-value column.

Grain declaration: state exactly what one fact row represents (one order line, one session) before choosing keys or measures; additive, semi-additive, and non-additive measures must be handled differently.

Storage families: OLTP row store for transactions, columnar store for analytical scans and compression, document/KV for flexible schemas, graph for relationship traversal, time-series for timestamped metrics.

Partitioning vs clustering: partitioning splits data into physical segments (usually by date) for pruning and retention; clustering/sort keys order data inside partitions to reduce scanned blocks.

Keys: surrogate keys isolate the warehouse from source-key changes; natural keys preserve business meaning; enforce uniqueness and not-null constraints at load time.

Schema evolution: additive nullable columns are backward compatible; renames, type narrowing, and drops are breaking and require versioning or a compatibility view.

Central tendency and spread: mean is sensitive to outliers, median is robust; report standard deviation with normal-ish data and IQR or MAD for skewed distributions.

Hypothesis testing: state H0 and H1, choose the test (t-test for means, chi-square for categorical association, ANOVA for 3+ group means), then interpret the p-value as P(data | H0), not P(H0 | data).

Type I error (false positive) is bounded by alpha; Type II error (false negative) is bounded by beta; power = 1 − beta and rises with sample size and effect size.

Confidence interval for a mean: x̄ ± z·(s/√n) for large n, or t·(s/√n) with n − 1 degrees of freedom for small samples.

Minimum detectable effect: n per group ≈ 16·σ²/Δ² for 80% power at alpha 0.05 (rule of thumb), doubling for two-sided tests versus one-sided.

Correlation is not causation: Pearson r measures linear association only; confounding, reverse causality, and selection bias require randomization, matching, or instrumental-variable designs.

Multiple comparisons inflate false positives; apply Bonferroni (alpha/m) for strict control or Benjamini-Hochberg to control the false discovery rate.

Regression diagnostics: check residual normality, homoscedasticity, independence, and multicollinearity (VIF > 5 or 10 signals trouble); R² measures fit, not predictive validity.

Time series: decompose into trend, seasonality, and residual; differencing removes trend, seasonal differencing removes seasonality; validate with rolling-origin backtests, not random splits.

Encoding accuracy ranking (most to least precise): position on a common scale, position on non-aligned scales, length, angle/slope, area, volume, color hue.

Chart selection rule: comparison → bar; distribution → histogram or box plot; trend over time → line; relationship between two numerics → scatter; part-to-whole with few categories → stacked bar or treemap.

Never truncate a bar chart's y-axis at a non-zero baseline; line charts may use a non-zero baseline when the variation is the message, and the axis must be labeled.

Color rules: sequential palette for ordered magnitude, diverging for values around a meaningful midpoint, categorical for unordered groups; avoid red/green pairs alone for accessibility.

Accessibility: minimum 4.5:1 contrast for text, do not encode meaning by color alone, provide alt text and data tables, and keep font sizes readable at target resolution.

Data-ink ratio: remove chartjunk (3D effects, heavy gridlines, redundant legends); label series directly instead of forcing legend lookup where space permits.

Aggregation honesty: state the grain and filters in the subtitle; small denominators produce unstable rates and should be suppressed or flagged.

KPI reporting: pair each metric with definition, owner, refresh cadence, and target; show trend plus context (prior period, benchmark) rather than a bare number.

GDPR lawful bases: consent, contract, legal obligation, vital interests, public task, legitimate interests; processing requires one basis and special-category data needs an extra condition.

Data subject rights: access, rectification, erasure, restriction, portability, and objection; the standard response window is one month, extendable by two for complex requests.

Privacy by design principles: proactive not reactive, privacy as default, embed protection into design, full functionality (positive-sum), end-to-end security, visibility and transparency, respect for user privacy.

De-identification ladder: pseudonymization (reversible with a key, still personal data under GDPR), anonymization (irreversible, out of scope), aggregation; k-anonymity, l-diversity, and differential privacy quantify re-identification risk.

Data minimization and purpose limitation: collect only fields necessary for a declared purpose and do not repurpose without a new lawful basis.

Breach notification: GDPR requires supervisory-authority notice within 72 hours of awareness where risk exists, and notice to affected individuals when risk is high.

Consent quality: freely given, specific, informed, unambiguous; pre-ticked boxes and bundled consent fail; withdrawal must be as easy as giving consent.

Ethics review: assess harm to individuals and groups, informed-consent coverage, secondary-use limits, and disparate impact across protected groups before release.

Data classification tiers (public, internal, confidential, restricted) drive access control, encryption, and retention rules; every dataset needs an assigned owner and classification.

Access control models: RBAC grants by role, ABAC by attributes/context, and least privilege means granting the minimum scope for the minimum duration.

Encryption: AES-256 at rest, TLS 1.2+ in transit; key management via a dedicated KMS with rotation, separation of duties, and no keys in code or repositories.

Authentication controls: MFA for privileged accounts, short-lived tokens, SSO with centralized revocation; secrets stored in a vault, never in plaintext config.

Governance artifacts: data catalog (discoverability), business glossary (shared definitions), lineage graph (source-to-report traceability), and data contract (producer/consumer schema agreement).

PII handling: tag columns at ingestion, mask or tokenize in non-production, and restrict raw-PII access to a named group with logged justification.

Audit and monitoring: log who accessed which dataset and when; alert on bulk exports, off-hours access, and privilege escalation; retain logs per policy.

Retention and disposal: define per-class retention windows, automate deletion or archival, and verify disposal with certificates or tombstones.

Orchestration: DAG-based schedulers (Airflow-style) with idempotent tasks, explicit dependencies, retries with backoff, and alerting on SLA misses.

Idempotency rule: a rerun of the same task with the same inputs must produce the same output; use partition overwrite or MERGE/upsert instead of blind append.

ELT over ETL when the warehouse is elastic: load raw first (landing/bronze), transform in-warehouse (silver/gold), preserving replayability of the raw layer.

Change data capture: log-based CDC reads the database transaction log for low-impact incremental loads; timestamp-based CDC misses hard deletes and in-place updates.

Streaming basics: at-least-once delivery plus idempotent sinks yields effectively-once; exactly-once requires transactional guarantees end to end.

Infrastructure as code: declare compute, storage, and pipelines in version control; peer review and automated plan/diff before apply; never hand-edit production.

CI/CD for data: run schema tests, freshness checks, and unit tests on transforms in CI; promote artifacts through environments rather than rebuilding per environment.

Cost and reliability: auto-scaling and auto-suspend for idle clusters, columnar compression and partitioning to cut scan cost, and lineage-based impact analysis before schema changes.