AWS Data Engineer cheat sheet

AWS Glue crawlers scan data sources, infer schemas, and write table definitions to the Glue Data Catalog, which Athena, Redshift Spectrum, and EMR use as a shared metadata store.

Kinesis Data Streams retains records for 24 hours by default; retention can be extended up to 365 days, and each stream is composed of shards that preserve record order.

Amazon Data Firehose buffers streaming records and delivers them to S3, Redshift, OpenSearch, or Splunk; it scales automatically and requires no shard management.

AWS DMS performs a full load and optionally change data capture (CDC) by reading source transaction logs, keeping the target synchronized during migrations.

Glue job bookmarks track previously processed data so repeated job runs process only new or changed files; bookmarks require persistent job state and compatible sources.

AWS Step Functions orchestrates multi-step ETL workflows with state machines, built-in retries, and error handling; standard workflows can run for up to one year.

AWS Lambda has a maximum invocation timeout of 15 minutes, so long-running Spark or batch transformations belong on Glue or EMR rather than Lambda.

AWS DataSync transfers files between on-premises storage and S3, EFS, or FSx, preserving metadata and performing incremental transfers after the initial copy.

Amazon EMR runs managed Hadoop and Spark clusters; EMR Managed Scaling adjusts capacity to workload, and transient clusters can terminate automatically after steps complete.

S3 Standard is designed for 99.999999999% (11 nines) object durability and stores objects redundantly across at least three Availability Zones.

S3 Lifecycle rules transition objects between storage classes or expire them; Glacier classes have minimum storage durations, such as 90 days for Glacier Flexible Retrieval.

Redshift distribution styles place rows across nodes: KEY colocates matching column values, ALL replicates the table, EVEN spreads rows round-robin, and AUTO selects a style.

Redshift sort keys order rows on disk so zone maps skip irrelevant blocks; COMPOUND suits prefix-filtered queries and INTERLEAVED suits multiple filtered columns.

DynamoDB partitions data by partition key, with the sort key ordering items inside a partition; throughput is governed by provisioned RCU/WCU or on-demand mode.

DynamoDB TTL automatically deletes items after a timestamp attribute expires, typically within 48 hours, without consuming write capacity.

Columnar formats such as Parquet and ORC reduce bytes scanned and cost for Athena and Redshift Spectrum compared with CSV or JSON.

Partitioning S3 prefixes by columns such as year and month enables Athena and Glue to prune partitions, reducing scanned data and query cost.

Amazon Aurora cluster storage grows automatically in 10 GiB increments up to 128 TiB and is shared by the writer and all reader instances.

CloudWatch collects metrics, logs, and alarms; default EC2 metrics publish every 5 minutes, while detailed monitoring publishes every 1 minute.

CloudTrail logs AWS API management events by default; object-level S3 data events and Lambda invocation events must be enabled explicitly.

EventBridge routes events between AWS services and SaaS applications using rules and event buses, with targets such as Lambda, Step Functions, SQS, or SNS.

Step Functions retries use IntervalSeconds and BackoffRate, while Catch fields route errors to fallback states for graceful failure handling.

SQS dead-letter queues receive messages that exceed a redrive policy's maxReceiveCount, isolating repeatedly failing messages for inspection.

Kinesis enhanced fan-out provides each registered consumer 2 MiB/s per shard of dedicated read throughput, versus a shared 2 MiB/s per shard for standard polling.

Redshift automated snapshots are retained from 1 to 35 days, while manual snapshots persist until explicitly deleted.

DynamoDB point-in-time recovery provides continuous backups for 35 days and restores to a new table at any second within that window.

AWS Backup centralizes backup policies across services; vault lock enforces write-once-read-many retention for compliance requirements.

IAM roles provide temporary credentials through AWS STS; Lambda, EC2, and Glue assume roles instead of storing long-term access keys.

S3 SSE-KMS uses AWS KMS keys and supports key policies, CloudTrail audit records, and separation of key administration; SSE-S3 uses S3-managed AES-256 keys.

S3 Block Public Access settings at account or bucket level override bucket policies and ACLs to prevent unintended public exposure.

AWS Lake Formation grants fine-grained table, column, and row-level permissions on data registered in the Glue Data Catalog.

KMS encrypts data at rest and TLS encrypts data in transit; KMS key policies define which principals can administer, use, or grant access to a key.

Amazon Macie discovers and classifies sensitive data such as PII in S3, while GuardDuty detects threats and anomalous account behavior.

VPC gateway endpoints for S3 and DynamoDB keep traffic on the private AWS network and avoid NAT gateway data processing charges.

Cross-account S3 access requires both a bucket policy granting the external principal and an IAM policy in the caller account allowing the action.

AWS Secrets Manager stores and rotates credentials using Lambda rotation functions or native integrations such as RDS, reducing long-lived password exposure.