CWIDP common mistakes
CWIDP · Understand the traps and how to avoid them.
Treating robots.txt as merely advisory and hand-rolling a scraper that ignores Crawl-delay, disallowed paths, or the site's terms, then defending it on the grounds that the data is public.
Candidates conflate 'publicly reachable' with 'permitted to collect'. The exam tests whether you can separate technical accessibility from legal/ethical authorization, and many candidates have only ever scraped without reading the terms or the robots directives.
What to do instead
Before writing any fetcher, check robots.txt, the terms of service, and any API/licensing terms, and document the decision. Prefer an official API or bulk download. Where scraping is permitted, honor Crawl-delay, set a descriptive User-Agent, rate-limit, and record the justification in the pipeline metadata so the choice is auditable.
Building a crawler that fires requests in a tight loop with no retry/backoff logic, no handling of duplicate URLs, and no checkpointing, then losing hours of work on a single failure.
Candidates focus on the happy path (fetch the page, parse the HTML) and skip the operational concerns. Exam scenarios deliberately introduce 429/503 responses, redirect chains, pagination loops, and mid-run crashes to see if the candidate designs for failure.
What to do instead
Design for idempotency and resilience: exponential backoff with jitter on 429/5xx, a canonical-URL normalizer plus a seen-set for deduplication, respect for Retry-After, persistent frontier/checkpoint state, and a cap on redirect depth. Log every request with status and timing so a re-run resumes rather than restarts.
Reporting a data-quality result (row counts, null rates, dedup rate) from a pipeline without ever profiling the raw source first, so subtle encoding, unit, or schema-drift problems are silently baked into downstream numbers.
Candidates jump straight to cleaning and analysis and treat validation as a final checkbox. The exam rewards evidence that you characterize the data before transforming it and that you can distinguish a source defect from a transformation defect.
What to do instead
Profile before you transform: row/column counts, types, null and distinct rates, min/max, value distributions, encoding checks, and a sample of raw records. Write assertions as executable checks (schema, ranges, referential integrity, uniqueness) that run at ingestion and again post-transform, and record which check failed on which record count.
Deduplicating with an exact string match on a single field (e.g., email or name) and declaring the dataset clean, missing near-duplicates, case/whitespace variants, and records that differ only in a timestamp.
Candidates reach for the simplest operation that reduces the row count and stop there. The exam probes whether you understand that identity is a modeling decision and that fuzzy matching has precision/recall trade-offs you must justify.
What to do instead
Define the business key explicitly, normalize (trim, case-fold, Unicode-normalize) before comparing, then apply blocking plus fuzzy matching (e.g., token or edit-distance thresholds) and inspect a sample of matches and non-matches to tune the threshold. Report the match rate and keep a survivorship rule for which record wins.
Choosing a storage layout by habit (everything into one wide table, or a document store for relational data) without stating access patterns, cardinality, or expected query shape.
Candidates answer 'which database?' from familiarity rather than from requirements. Exam items give you workload characteristics (read/write ratio, query shape, volume, consistency needs) and expect a justified choice, not a brand name.
What to do instead
State the access patterns first, then map them: transactional integrity and joins favor a relational model with normalized keys; high-volume append/analytical scans favor columnar or partitioned layouts; flexible nested payloads favor document or semi-structured storage. Name the trade-off you accept (e.g., denormalization for read speed at the cost of update anomalies) and how you would evolve the schema.
Reporting an average or a correlation from skewed or confounded data as if it were a causal or representative finding, with no mention of sample size, distribution shape, or confounders.
Candidates default to the most familiar statistic and skip the assumption checks. The exam includes distributions where the mean misleads (heavy tails, bimodality) and comparisons where a lurking variable explains the difference.
What to do instead
Inspect the distribution (histogram, median vs. mean, quartiles, outlier count) before choosing a summary. State the assumptions behind each test, report effect size and uncertainty (confidence interval or spread), and for observational comparisons name plausible confounders and either stratify/adjust or explicitly limit the claim to association.
Building a chart with a truncated y-axis, dual axes, or a color scale that encodes nothing, and presenting it without a title, units, or source note.
Candidates optimize for visual appeal and default tool settings. The exam tests whether you can spot a chart that misleads and whether you know the labeling and accessibility basics that make a figure interpretable on its own.
What to do instead
For bar charts, start the value axis at zero; avoid dual axes unless unavoidable and label both; use color only to encode a variable and pair it with shape/label for color-vision accessibility; always include a descriptive title, axis labels with units, a legend where needed, and a source/as-of note. Sanity-check that the visual impression matches the underlying numbers.
Handling personal data by simply dropping a name column and calling the dataset anonymized, while leaving quasi-identifiers (postcode, birth date, gender, timestamps) that permit re-identification.
Candidates equate de-identification with removing direct identifiers. The exam tests whether you recognize that combinations of quasi-identifiers, small cell counts, and linkable auxiliary data defeat naive anonymization, and that legal definitions of personal data are broader than a name field.
What to do instead
Classify fields by identifiability, then apply a documented technique appropriate to the risk: generalization/bucketing (age bands, partial postcode), suppression of small cells, pseudonymization with a managed key, or differential privacy for aggregates. Assess re-identification risk against plausible auxiliary data, state the residual risk, and record the lawful basis and retention limit.
Granting a service account or analyst broad read/write access to production data 'to get it working', with no logging, no least-privilege scoping, and secrets committed in code or notebooks.
Candidates treat access control as an afterthought to functionality and underestimate how often exam scenarios hinge on a leaked credential or an over-privileged pipeline. Convenience wins over governance when the candidate is focused on the data task.
What to do instead
Apply least privilege: separate read-only analytics roles from write roles, scope service accounts to the specific datasets and operations needed, rotate credentials, and load secrets from a managed secret store or environment injection rather than source. Enable audit logging on access and changes, and define who reviews it. State the data classification and the control that protects it.
Scheduling a pipeline with no dependency awareness, no idempotent reruns, and no alerting, so a late upstream file produces a silently partial load that nobody notices until a report is wrong.
Candidates design the transformation logic but not the orchestration around it. The exam tests whether you think about freshness, failure detection, and safe re-execution - the difference between a script and a reliable pipeline.
What to do instead
Model tasks as a DAG with explicit upstream dependencies and freshness/sensor checks; make each task idempotent (partition overwrite or merge on a key) so reruns are safe; add data-quality gates that halt the run on failure; and alert on missed SLAs, row-count anomalies, and schema changes. Track run metadata (start, duration, rows in/out) so failures are diagnosable after the fact.