CWIDP key terms

A convention where a site publishes a robots.txt file telling automated clients which URL paths they may or may not fetch; it is voluntary for well-behaved crawlers, not an access control mechanism.

Deliberately spacing requests to a server so it is not overloaded, typically by honoring crawl-delay, capping concurrent connections, and backing off exponentially after error responses such as 429 or 503.

Checking a source site's contractual terms before automated collection to confirm whether scraping, storage, or redistribution of its content is permitted, and documenting that decision for later audit.

Repeating a failed request in a way that produces the same result if it succeeds twice, waiting progressively longer between attempts, so transient network or server faults are survived without duplicating records.

Retrieving a large result set in ordered chunks using cursor or offset parameters until the source signals no further pages, ensuring complete extraction rather than only the first page.

Pulling only records changed since the last successful run by tracking a high-water mark such as an update timestamp or monotonic ID, reducing load on the source and shortening run times.

Monitoring incoming source structures for added, removed, renamed, or retyped fields so that pipelines fail loudly or adapt deliberately instead of silently loading corrupted or incomplete data.

An explicit, versioned agreement between a data producer and its consumers specifying field names, types, nullability, freshness, and ownership, so changes are negotiated rather than discovered downstream.

A named aspect against which data is judged, such as completeness, accuracy, consistency, timeliness, uniqueness, or validity; each dimension needs its own measurable rule and threshold.

Systematically examining a dataset's columns to learn value distributions, null rates, distinct counts, patterns, and outliers, usually before writing cleaning logic or designing a schema.

A testable assertion applied to records or batches, such as a range check, referential check, or format regex, whose failures are counted, logged, and routed to remediation rather than ignored.

A holding area where records failing validation are written with the failure reason, preserving them for inspection and reprocessing instead of dropping them silently or letting them reach consumers.

The set of fields used to decide whether two rows describe the same real-world entity, enabling deterministic or fuzzy merging; choosing it badly either merges distinct entities or leaves duplicates.

Deciding which records across or within sources refer to the same person, organization, or object, often by comparing names and addresses with similarity scoring and applying match thresholds.

A relational design rule requiring that non-key attributes depend on the whole key and nothing but the key, eliminating repeating groups and transitive dependencies in transactional schemas.

A dimensional model with a central fact table of measures and foreign keys, surrounded by denormalized dimension tables that supply descriptive context for filtering and grouping.

A technique for preserving dimension history, most commonly Type 2, which inserts a new row with new validity dates and a current-flag when an attribute changes instead of overwriting it.

A system-generated identifier with no business meaning, assigned to each dimension or entity row so that natural business keys can change without breaking joins to fact tables.

Physically organizing stored data by columns such as date or region so queries read only relevant slices; partitioning prunes whole segments while clustering orders rows within them for faster filtering.

Storing values of each column contiguously rather than row by row, which compresses well and lets analytical queries read only the columns they need, at the cost of slower single-row updates.

The rules governing missing values, notably that SQL comparisons with null yield unknown and that nulls are excluded by most aggregates, so counts and averages must be interpreted carefully.

A situation where a trend visible in aggregated groups reverses or disappears when the groups are combined, usually because group sizes or a lurking variable differ markedly.

An unmeasured factor that influences both the supposed cause and the observed outcome, creating a spurious association that can mislead anyone reading the relationship as causal.

The probability, assuming the null hypothesis is true, of observing a result at least as extreme as the one obtained; it is not the probability that the hypothesis is true.

Identifying values that deviate sharply from the rest of a distribution, using rules such as interquartile range bounds, z-scores, or isolation models, then deciding whether to keep, cap, or investigate them.

The database's chosen sequence of operations for a query, showing scans, joins, and sorts with estimated costs; reading it reveals why a query is slow and which index or filter is missing.

The ranking of visual channels by how precisely people read them, with position on a common scale best, then length, angle, and area, and color hue weakest for quantitative comparison.

Matching a visual form to the analytical question, such as bars for comparison, lines for trend over time, and scatterplots for relationship, rather than choosing by decoration or habit.

The hazard that two series plotted with separate y-axes can be scaled to imply any correlation; use them only with clear labeling and justification, or split into small multiples.

Choosing text and mark colors whose luminance difference meets published accessibility thresholds so that low-vision and color-blind readers can still distinguish categories and read labels.

The specific legal ground, such as consent, contract, or legitimate interest, that justifies handling personal data under regimes like the GDPR; it must be identified and documented before collection.

The principle that personal data collected for one stated purpose may not be repurposed for an incompatible one without a fresh legal basis, which constrains reusing datasets across projects.

Collecting and retaining only the personal fields actually needed for the stated purpose, which reduces breach impact, storage cost, and compliance exposure from over-broad schemas.

Replacing direct identifiers with tokens so records can still be linked within a controlled environment, while keeping the mapping separate; it reduces exposure but does not make data fully anonymous.

A request from an individual to see, correct, or delete the personal data held about them; pipelines must be able to locate and act on all copies within the legally required deadline.

Granting permissions to job roles rather than individuals, so users inherit exactly the data access their role requires and changes in staffing do not leave stale entitlements behind.

Giving each user, service, and pipeline the minimum permissions and data scope needed to do its job, limiting the blast radius if an account or credential is compromised.

Dynamically obscuring sensitive fields such as email or national ID for users who lack clearance, while analysts can still join and aggregate on the column without seeing raw values.

A recorded map of how each dataset was produced, tracing upstream sources, transformations, and downstream consumers, which supports impact analysis, debugging, and audit evidence.

The governed process of maintaining one authoritative, shared record for core entities such as customers or products, resolving conflicting copies held by separate operational systems.

A directed acyclic graph defining task dependencies and run order in a scheduler, so downstream steps execute only after upstream success and reruns can target failed branches.

A separate destination for messages or records that repeatedly fail processing, keeping them out of the main flow while preserving them for diagnosis and controlled replay.

Defining storage, compute, and networking resources in version-controlled configuration files so environments are reproducible, reviewable, and free of undocumented manual changes.