I85PREP
⌘ K
CWIDP study materials
CWIDP study resources

CWIDP: Data Collection and Web Acquisition: lawful, reproducible acquisition

CWIDP · Work through your exam, one topic at a time.

All guide sections

Data Collection and Web Acquisition: lawful, reproducible acquisition

What this domain tests

The Data Collection and Web Acquisition domain examines whether you can obtain web data lawfully, reproducibly, and with enough provenance to defend downstream analysis. Questions tend to focus on choosing an acquisition method, handling pagination and rate limits, and recording what was collected, when, and from where.

Core rules

  • Prefer a documented API or bulk feed over crawling when it provides the same fields; record the endpoint, version, auth scope, and extraction timestamp.
  • Respect robots.txt (RFC 9309) for crawlers, plus site terms and contractual limits. robots.txt is not authorization for personal data processing.
  • Make requests idempotent and polite: stable user-agent, contact URL, per-host concurrency caps, and exponential backoff with jitter on 429/503. Do not retry 4xx client errors except 408/429.
  • Capture provenance per record: source URL, request time, HTTP status, content hash, extractor version, and pagination cursor.
  • Handle pagination by cursor or token where available; page numbers drift when listings change.
  • Deduplicate on canonical URL or source ID, not on raw URL with tracking parameters.
  • For incremental loads, use Last-Modified/ETag or a watermark; keep a full-snapshot fallback.

Worked example

You must collect 50,000 product pages nightly. The API returns 100 items per page with a next_cursor and supports If-None-Match. You store cursor, ETag, fetched_at, and a SHA-256 of the normalized body. On a 429 you wait 2^attempt seconds plus random jitter, cap concurrency at 4, and resume from the stored cursor. Reasoning: cursor pagination avoids duplicates when new products are inserted; ETag avoids re-downloading unchanged pages; the hash lets you detect silent content changes; the cursor checkpoint makes the job restartable. Completeness is measured as distinct product IDs received divided by IDs listed in the sitemap or catalog count, not pages fetched.

Traps that cost marks

  • Treating a crawl as a timeless snapshot and omitting collection timestamps.
  • Scraping JavaScript-rendered content without a renderer, then reporting empty fields as missing data.
  • Ignoring robots.txt or terms because the page is public.
  • Retrying 403/404 responses and inflating request volume.
  • Counting rows instead of distinct entities when measuring coverage.