Data Engineering
Data Engineering
This domain tests your ability to move, store, and prepare data for machine learning on AWS. You must know the difference between S3 (object storage, the backbone of most ML pipelines) and EBS/EFS (block and file storage for EC2). Understand when to use AWS Glue for ETL: Glue Data Catalog as a Hive-compatible metastore, Crawlers to infer schemas, and Glue jobs (Spark-based) for transformation. Know Amazon Kinesis for streaming (Data Streams vs. Firehose vs. Analytics) and AWS DMS for database migration. For batch ingestion, AWS Data Pipeline and Glue are common. Also know Amazon Athena for querying S3 with SQL, and Amazon EMR for big data processing with Spark/Hadoop.
Traps: Confusing Kinesis Data Streams (custom consumers, low latency) with Firehose (delivery to S3/Redshift/Elasticsearch, near real-time). Glue Crawlers do not transform data; they only populate the catalog. S3 is not a file system; it has eventual consistency for overwrites (though now strong read-after-write for new objects). Partitioning in S3 (e.g., by date) drastically improves Athena/Glue performance and reduces cost. Also, AWS Lake Formation centralizes permissions for data lakes; do not confuse with IAM.
Worked example: You need to ingest clickstream data every second and run SQL queries on it with minimal latency. Use Kinesis Data Streams to capture, Kinesis Data Analytics for real-time SQL, and optionally Firehose to archive to S3. If the requirement is near real-time dashboards, Firehose to Elasticsearch may suffice. Rule of thumb: Streaming (sub-second) -> Kinesis Data Streams; micro-batch (near real-time) -> Firehose; batch -> Glue/S3.