AWS Data Engineer flashcards

32 cards. Read the term and the definition here. The deck at the top of this page is optional.

Glue Data Catalog

Central metadata repository that stores table definitions, schemas, and partitions; Athena, Redshift Spectrum, and EMR query data using these catalog definitions.

Kinesis Data Streams

Real-time streaming service composed of shards that provide ordered record delivery; default retention is 24 hours and can extend to 365 days.

Data Firehose

Managed delivery service that buffers streaming records and writes them to S3, Redshift, OpenSearch, or Splunk without requiring shard management or custom consumers.

AWS DMS CDC

Change data capture reads source database transaction logs and applies ongoing changes to a target, enabling near-zero-downtime migrations after the initial full load.

Glue job bookmarks

State mechanism that tracks previously processed data so scheduled Glue jobs read only new or changed files instead of reprocessing the entire dataset.

Step Functions

Serverless orchestration service that runs state machines with sequential, parallel, and choice states plus built-in retries and error handling for ETL workflows.

Lambda timeout limit

An AWS Lambda invocation can run for at most 15 minutes; longer Spark or batch transformations require Glue, EMR, or Step Functions orchestration.

DataSync

Agent-based transfer service that moves files between on-premises storage and Amazon S3, EFS, or FSx while preserving metadata and performing incremental syncs.

S3 Standard durability

Amazon S3 Standard stores objects redundantly across at least three Availability Zones and is designed for 99.999999999 percent object durability.

Redshift distribution key

Attribute that determines how table rows are placed across compute nodes; KEY colocates matching values, ALL replicates the table, and EVEN distributes rows round-robin.

Redshift sort key

Column ordering on disk that enables zone maps to skip irrelevant blocks; COMPOUND suits prefix filters and INTERLEAVED suits multiple frequently filtered columns.

DynamoDB partition key

Primary key component that determines the partition where an item is stored; choosing a high-cardinality key avoids hot partitions and uneven throughput consumption.

DynamoDB TTL

Time-to-live attribute that marks items for automatic deletion after a timestamp; expired items are typically removed within 48 hours without consuming write capacity.

Parquet columnar format

Column-oriented file format that stores data by column and compresses efficiently, reducing bytes scanned and cost for Athena and Redshift Spectrum queries.

S3 partition pruning

Organizing S3 prefixes by columns such as year and month lets Athena and Glue skip unrelated partitions, lowering scanned data and query cost.

Aurora storage growth

Amazon Aurora cluster storage automatically increases in 10 GiB increments up to 128 TiB and is shared by the writer and all reader instances.

CloudWatch detailed monitoring

Optional EC2 monitoring that publishes metrics at one-minute granularity instead of the default five-minute interval, enabling faster alarms and autoscaling reactions.

CloudTrail data events

Optional trail configuration that records object-level S3 operations and Lambda invocations; management events are logged by default but data events are not.

EventBridge rule

Pattern-matching configuration that routes events from a bus to targets such as Lambda, Step Functions, SQS, or SNS based on event content.

SQS dead-letter queue

Separate queue that receives messages exceeding a redrive policy's maxReceiveCount, isolating repeatedly failing messages for later inspection and reprocessing.

Kinesis enhanced fan-out

Consumer mode providing each registered consumer 2 MiB per second per shard of dedicated read throughput instead of sharing the standard 2 MiB per shard.

Redshift automated snapshots

Backups taken about every eight hours and retained from one to thirty-five days; manual snapshots remain until explicitly deleted.

DynamoDB PITR

Point-in-time recovery continuously backs up a table for 35 days and restores to a new table at any second within that retention window.

AWS Backup vault lock

Governance feature that enforces write-once-read-many retention on recovery points, preventing deletion or alteration during a defined compliance period.

IAM role assumption

A principal calls AWS STS AssumeRole to receive temporary credentials; workloads such as Lambda and Glue use roles instead of embedded long-term access keys.

SSE-KMS

S3 server-side encryption using AWS KMS customer managed or AWS managed keys, providing key policies, CloudTrail audit records, and separation of key administration.

S3 Block Public Access

Account-level or bucket-level settings that override bucket policies and ACLs to reject public access, preventing accidental exposure of stored objects.

Lake Formation permissions

Central governance layer that grants or denies fine-grained access to Glue Data Catalog tables, columns, and rows for users and IAM principals.

KMS key policy

Resource-based policy that defines which principals can administer, use, or grant access to a KMS key; it is required for cross-account key usage.

Macie

Managed security service that uses machine learning to discover, classify, and alert on sensitive data such as personally identifiable information stored in Amazon S3.

S3 gateway endpoint

VPC endpoint that routes S3 traffic privately over the AWS network without an internet gateway or NAT device, avoiding NAT data processing charges.

Secrets Manager rotation

Scheduled automatic replacement of stored credentials using a Lambda rotation function or native integration, reducing exposure from long-lived database passwords.