Data Engineering: Ingest, Store, Catalog, Transform
Pick the tool from the access pattern
This domain tests whether you can move data from sources into a training-ready store with the right service and cost profile.
Ingestion. Use Kinesis Data Streams when you need sub-second latency, custom consumers, multiple readers, or replay (24-hour default retention, extendable to 365 days). Use Kinesis Data Firehose when the goal is simply delivery into S3, Redshift, or OpenSearch, with built-in buffering (minimum buffer interval is ~60 seconds), optional Lambda transformation, and format conversion to Parquet/ORC. Choose Amazon MSK when an existing Kafka ecosystem must be preserved. For database replication use AWS DMS (with the Schema Conversion Tool for heterogeneous engines); for bulk offline transfer of terabytes, Snowball.
Storage and catalog. S3 is the data lake; the Glue Data Catalog is the Hive-compatible metastore shared by Athena, Redshift Spectrum, and EMR; Lake Formation adds column- and row-level grants. Prefer columnar formats (Parquet, ORC) with compression, and partition on the columns that appear in WHERE clauses, because Athena bills by data scanned. For features consumed both by training and by low-latency inference, SageMaker Feature Store provides an offline store (S3/Parquet) and an online store.
Transformation. Glue ETL (Spark with DynamicFrames, job bookmarks for incremental runs) is the default batch choice; SageMaker Processing handles ML-specific preprocessing at scale; EMR fits very large custom Spark/Hadoop workloads.
Traps. Firehose is near-real-time, not real-time. A Glue crawler infers schema and partitions - it does not clean, deduplicate, or transform data. gzip is not splittable (a single mapper reads the whole file); Snappy-compressed Parquet is. Many tiny files throttle Athena far more than a few large ones.
Rule of thumb. 2 TB of raw JSON logs queried only by date and application: convert to Parquet, partition by dt/app, crawl, and query with Athena - scanned bytes typically fall by an order of magnitude, which is the whole cost story.