Data Concepts and Environments
Data Concepts and Environments
Start with the shape of the data. Structured data lives in rows and columns with a defined schema; semi-structured data such as JSON or XML has tags but flexible structure; unstructured data such as text, images, and audio has no predefined model. Know the four measurement scales: nominal (labels), ordinal (ordered labels), interval (equal gaps, no true zero), and ratio (equal gaps with a true zero). Questions often test whether you can classify a field - temperature in Celsius is interval, temperature in Kelvin is ratio.
Then move to storage. OLTP systems run many small transactions and are normalized to reduce redundancy; OLAP systems support large analytical reads and often use denormalized star or snowflake schemas. A data warehouse holds structured, modeled data for reporting; a data mart is a subject-specific subset; a data lake stores raw data of any type, often schema-on-read. ETL transforms before loading; ELT loads raw first and transforms inside the target. Batch pipelines process chunks on a schedule; streaming pipelines handle continuous events.
A common trap is confusing "fastest query performance" with "cheapest raw storage" - use a warehouse for modeled BI and a lake for raw volume. Also distinguish primary keys (unique row identifier) from foreign keys (reference to another table), and remember that normalization reduces redundancy, not necessarily read speed.
Worked example: a retailer ingests clickstream JSON, POS transactions, and scanned receipts. Rule of thumb: land all raw files in the lake, then model conformed POS and customer data into the warehouse for dashboards.