Data Ingestion and Transformation
Core concepts
Ingestion splits into batch, streaming, and transfer. Batch: AWS Glue jobs, Step Functions, Amazon MWAA, AWS DMS. Streaming: Kinesis Data Streams, Amazon MSK, and Data Firehose as a delivery layer to S3/Redshift/OpenSearch. Transfer: DataSync, Transfer Family, Snow Family.
Transformation happens either in-flight (Lambda, Managed Service for Apache Flink, Firehose Lambda enrichment) or at-rest (Glue/Spark, EMR, DataBrew). Glue's DynamicFrame adds resolveChoice for schema conflicts and relationalize for nested JSON; Spark DataFrames are faster when the schema is known.
Incremental patterns: Glue job bookmarks (track files already processed), DMS change data capture (full load plus ongoing replication from the source's log), Kinesis checkpointing per shard, and watermark columns for JDBC sources.
Traps
- Firehose is near real time: buffers flush on size/interval (60 s minimum) and there is no replay. Kinesis Data Streams gives ordered records per shard, replay up to the retention window (24 h default, up to 365 days), and multiple consumers.
- Job bookmarks dedupe files, not rows inside a file, and are not a substitute for a watermark on relational sources.
- DMS CDC needs a replication instance sized for the change volume and the source's log retention; if binary logging expires, the task fails and must be restarted.
- Schema drift makes crawlers create new table versions or duplicate tables; define classifiers and partition structure deliberately.
Worked rule
If the requirement states a sub-minute SLA and reprocessing/replay, choose Kinesis Data Streams with a consumer (Lambda or Flink) rather than Firehose. If the requirement is "land raw events in S3 reliably with buffering and compression," Firehose is the correct, lower-effort answer.