AWS Data Engineer common mistakes
AWS Data Engineer · Understand the traps and how to avoid them.
Designing an ingestion pipeline where a single AWS Lambda function opens and parses one large multi-GB file (or a whole day's prefix) from S3.
Lambda feels like the default 'S3 event -> process' tool, so candidates pick it without checking the 15-minute timeout, 10 GB memory ceiling, 512 MB /tmp scratch space and 6 MB synchronous payload limit; the design then fails only at production file sizes.
What to do instead
Match the compute to the file size: use Lambda only for small objects or per-record fan-out (S3 event -> SQS -> Lambda), and use AWS Glue, EMR/EMR Serverless, or Step Functions Distributed Map for large files or bulk prefixes. State the size threshold explicitly when justifying the choice.
Writing a Glue DynamicFrame straight to the target with dyf.write or glueContext.write_dynamic_frame and then discovering every column landed as string, or that mixed-type columns silently dropped values.
DynamicFrames carry a choice type (e.g., choice(long,string)) for ambiguous columns, and the default ResolveChoice behaviour ('make_cols'/'cast:void') does not produce the numeric types the candidate assumed; the job 'succeeds' so the schema problem is missed.
What to do instead
Apply an explicit mapping before the write - applyMapping/SelectFields with cast:long, cast:double, cast:timestamp - or set an explicit ResolveChoice(drop, cast) strategy, and validate the resulting Parquet/ORC schema in the Data Catalog after the first run.
Re-running a scheduled Glue job over the same S3 prefix or JDBC table every night without job bookmarks, or enabling bookmarks for a JDBC source without setting a bookmark key/ordering column.
Bookmarks are off by default and are easy to forget; for JDBC sources candidates assume 'bookmarks on' is enough, but without an ordering column that increases monotonically, Glue cannot compute the high-water mark and either re-reads everything or skips rows.
What to do instead
Enable job bookmarks (--job-bookmark-option job-bookmark-enable) for incremental sources, use a monotonically increasing column (sequence ID, updated_at) as the JDBC bookmark key, and pair the sink with an idempotent merge/upsert so a replayed batch cannot duplicate rows.
Pointing Athena at a raw landing prefix that receives many small uncompressed JSON objects and expecting acceptable latency and cost, with no partitioning and no compaction.
Streaming sinks and Lambda/Glue writers naturally emit thousands of tiny files, and Athena bills by data scanned, so each query pays full-file overhead across every object with no partition pruning.
What to do instead
Land raw data as-is, then run a compaction/conversion job (Glue or Firehose record-format conversion) to produce right-sized Parquet/ORC files (target 128-512 MB), partition on the common filter columns (year/month/day or event type), and use partition projection or MSCK REPAIR TABLE / ALTER TABLE ADD PARTITION instead of a per-run crawler.
Selecting a DynamoDB partition key with low cardinality (status, country, date) and then using Scan to answer access patterns that need a filtered read.
Candidates focus on the item shape rather than the access pattern, so the key maps most items to a handful of partitions and the table throttles on hot partitions; Scan is chosen because the key does not support the query.
What to do instead
Choose a high-cardinality partition key derived from the dominant access pattern (composite keys such as tenant#device), add a sort key for range queries, use Query with KeyConditionExpression instead of Scan, and use on-demand capacity or DAX where read spikes are unpredictable.
Loading Redshift with row-by-row INSERTs over JDBC, or issuing a separate COPY per small file, and leaving DISTKEY/SORTKEY at defaults.
COPY from S3 is dramatically faster than INSERTs and the exam expects it, but candidates also miss that many tiny files serialize the load unless a manifest is used and that a poor DISTKEY causes data-skew on joins.
What to do instead
Stage to S3 and use a single COPY per table with a manifest, compressed columnar input, and COMPUPDATE/STATUPDATE as appropriate; choose DISTKEY on the column used in the largest joins, SORTKEY on the common range-filter column, and use Spectrum or UNLOAD for data that should not be loaded at all.
Building S3-event-driven or Kinesis/Firehose consumers that assume exactly-once delivery, so retries, duplicate S3 notifications or Lambda re-invocations create duplicate rows downstream.
S3 event notifications, SQS and Kinesis are all at-least-once; candidates reason from the happy path and never test a replay, so the first redrive or Lambda retry doubles the data.
What to do instead
Make the write idempotent: deterministic keys from the event (bucket+key+versionId or stream sequence number), MERGE/upsert or conditional writes in the sink, deduplication in Glue (DropDuplicates on the business key), and dead-letter queues with redrive for poison records.
Monitoring Glue/EMR pipelines only by looking at the console or a scheduled status-check Lambda, with no alarms on job outcome and no EventBridge reaction to failure.
Candidates treat operations as 'run the job' and underestimate the operational-domain weighting; console polling and log inspection after the fact are the only tools they have prepared, so failures are detected by the business, not by the pipeline.
What to do instead
Create an EventBridge rule for Glue Job State Change (FAILED/TIMEOUT/STOPPED) and EMR step failure routing to SNS/PagerDuty, enable Glue continuous logging and job metrics, and alarm on CloudWatch metrics such as glue.driver.aggregate.numFailedTasks, executor skew and Kinesis IteratorAgeMilliseconds.
Granting a Glue/EMR/Lambda execution role s3:GetObject and s3:PutObject on a bucket encrypted with a customer-managed KMS key and expecting the job to read and write successfully.
Candidates stop at the S3 action list and forget that SSE-KMS requires separate kms:Decrypt, kms:GenerateDataKey and kms:DescribeKey permission on the key, and that the key policy must also allow the principal - the failure surfaces only as a generic AccessDenied.
What to do instead
Add the KMS actions to the execution role, confirm the customer-managed key policy delegates to the account/IAM (so identity policies can grant access), keep the key in the same Region as the data, and use separate keys per environment/data classification so least privilege is enforceable.
Implementing column-level or row-level restrictions for analysts by relying on IAM/S3 bucket policies alone, or trusting Lake Formation table grants while the analyst role still has broad s3:GetObject on the underlying prefix.
Fine-grained control is a Lake Formation concern, not an S3 concern; if IAM read on the bucket is left in place, users can bypass table filters (for example by querying with Athena or reading S3 directly), and conversely Lake Formation alone fails until the S3 location is registered and the data lake admin is configured.
What to do instead
Register the S3 location with Lake Formation, migrate to Lake Formation permissions (LF-Tags/column filters/row filters) for table access, remove broad s3:GetObject from analyst roles so access flows through the governed path, and grant kms permissions consistently with the registered location.