CWIDP flashcards

33 cards. Read the term and the definition here. The deck at the top of this page is optional.

robots.txt purpose

A file at a site root declaring which user agents may crawl which paths and at what delay. It is advisory, not an access control, and ignoring it can violate terms of service.

HTTP 429 handling

Status meaning too many requests. The client should honor the Retry-After header, reduce concurrency, and apply exponential backoff with jitter before retrying.

Canonical URL key

A normalized URL form used for deduplication: lowercase host, strip fragments, sort query parameters. It prevents recrawling the same page under multiple URL spellings.

Crawl frontier priority

The queue of URLs awaiting fetch, ordered by breadth-first, depth-first, or a priority score such as freshness or link weight, bounded by max depth and page caps.

Headless browser tradeoff

Rendering JavaScript in a browser engine captures dynamic content that static fetches miss, but multiplies compute cost and latency, so it is reserved for pages that require it.

Missingness: MCAR, MAR, MNAR

MCAR means missingness is unrelated to any variable; MAR means it is explained by observed variables; MNAR means it depends on the unobserved value itself, biasing naive imputation.

IQR outlier fence

Flag values below Q1 minus 1.5 times the interquartile range or above Q3 plus 1.5 times the IQR. The rule is robust to skew because it uses quartiles, not the mean.

Pseudonymization versus anonymization

Pseudonymization replaces identifiers with keys and remains reversible, so data stays personal under GDPR. Anonymization is irreversible and removes data from scope.

Data quality dimensions

Completeness, accuracy, consistency, timeliness, uniqueness, and validity. Each needs a measurable threshold so quality can be monitored rather than asserted.

Referential integrity check

A validation confirming every foreign-key value in a child table exists in the parent table. Violations indicate orphaned records and broken joins downstream.

Third normal form

A table is in 3NF when it is in 2NF and has no transitive dependency: non-key attributes depend only on candidate keys, never on other non-key attributes.

Star versus snowflake schema

A star schema keeps dimension tables denormalized around a central fact table. A snowflake normalizes dimensions into sub-tables, saving space but adding joins.

SCD Type 2

A slowly changing dimension technique that preserves history by inserting a new row per change, with valid_from, valid_to, and a current-record flag.

Fact table grain

The exact meaning of one fact row, such as one order line or one session. Declaring grain first determines which keys and additive measures are valid.

Surrogate key

A system-generated identifier, typically an integer or hash, assigned in the warehouse. It insulates historical data from changes to source-system natural keys.

Partitioning versus clustering

Partitioning splits a table into physical segments, usually by date, enabling pruning and cheap retention. Clustering orders rows within partitions to reduce scanned data blocks.

p-value interpretation

The probability of observing data at least as extreme as collected, assuming the null hypothesis is true. It is not the probability that the null hypothesis is true.

Statistical power

One minus the Type II error rate: the probability of detecting a real effect of a given size. Power increases with sample size and with larger true effects.

Confidence interval for a mean

An interval x-bar plus or minus the critical value times the standard error s over root n. Use the t critical value with n minus one degrees of freedom for small samples.

Type I versus Type II error

Type I is rejecting a true null hypothesis, a false positive bounded by alpha. Type II is failing to reject a false null, a false negative bounded by beta.

Multiple comparisons correction

Repeated tests inflate false positives. Bonferroni divides alpha by the number of tests for strict family-wise control; Benjamini-Hochberg controls the false discovery rate instead.

Multicollinearity and VIF

Correlated predictors destabilize regression coefficients. A variance inflation factor above 5 or 10 signals problematic collinearity that should be resolved before interpretation.

Chart encoding accuracy

Position on a common scale is read most accurately, followed by length, angle, area, and color hue. Choose encodings matching the precision the comparison demands.

Bar chart baseline rule

Bars encode magnitude by length, so the y-axis must start at zero. Truncating it exaggerates differences and misleads viewers about relative size.

Sequential versus diverging palette

Sequential palettes encode ordered magnitude from low to high. Diverging palettes encode deviation around a meaningful midpoint, with two hues and a neutral center.

Data-ink ratio

The proportion of chart pixels carrying information. Raising it means removing 3D effects, heavy gridlines, and redundant legends while keeping labels and scales legible.

GDPR lawful bases

Consent, contract, legal obligation, vital interests, public task, and legitimate interests. Every processing activity needs one identified basis before data is collected.

72-hour breach notification

Under GDPR, a controller must notify the supervisory authority within 72 hours of becoming aware of a personal-data breach, unless the breach is unlikely to pose risk.

k-anonymity

A release guarantee that each record is indistinguishable from at least k minus one others on quasi-identifiers. It limits re-identification but not attribute disclosure.

ABAC versus RBAC

Role-based access control grants permissions by job role. Attribute-based control evaluates user, resource, and context attributes, enabling finer, context-aware decisions.

Data lineage

A recorded graph of how data moves from source systems through transformations to reports. It supports impact analysis, debugging, and regulatory traceability.

Idempotent pipeline task

A task that produces the same output when rerun with the same inputs. Achieved with partition overwrite or MERGE upserts rather than blind appends, enabling safe retries.

Log-based change data capture

Reading the database transaction log to capture inserts, updates, and deletes incrementally. Unlike timestamp polling, it detects hard deletes and in-place updates.