AWS Data Engineer key terms

A serverless, managed extract-transform-load service that runs Apache Spark jobs to move and reshape data between sources and targets, using the Data Catalog for metadata and triggers or schedules for orchestration.

A managed, Hive-compatible metastore holding database, table, schema, and partition definitions so that Athena, Redshift Spectrum, EMR, and Glue itself can query the same datasets without redefining them.

A process that scans a data store, infers file formats and column types, and writes or updates table definitions in the Data Catalog, optionally on a schedule and detecting newly added partitions.

State kept by AWS Glue recording which files or records a job already processed, so scheduled reruns read only new data instead of rescanning the entire source.

A Glue data structure similar to a Spark DataFrame that carries a self-describing schema and represents malformed records explicitly, letting jobs transform semi-structured input without failing on bad rows.

A visual, no-code data preparation service with hundreds of prebuilt transformations for profiling, cleaning, and normalizing datasets; recipes can be run interactively, scheduled, or executed as Glue jobs.

A real-time streaming service in which producers append records to shards and consumers read them in order per shard, with configurable retention from one day up to 365 days.

A managed delivery service that buffers streaming records, optionally invokes Lambda for transformation, and loads them into destinations such as S3, Redshift, OpenSearch, or third-party endpoints without custom consumers.

A managed Apache Kafka service that provisions broker clusters inside your VPC and handles patching, replication, and scaling, so existing Kafka producers and consumers connect with minimal code change.

A serverless service for running Apache Flink applications that perform stateful, windowed computations over Kinesis or Kafka streams and write results to downstream stores.

A replication service that migrates data between homogeneous or heterogeneous databases, supporting one-time loads and continuous change data capture to keep source and target synchronized during cutover.

A rugged, shippable device with on-board compute and storage used to transfer very large datasets into or out of AWS when network bandwidth is insufficient, with encryption at rest and in transit.

A columnar file format with embedded schema, compression, and per-block statistics; query engines read only referenced columns and skip blocks that cannot match filters, cutting scan cost.

A serverless orchestrator that models workflows as state machines, providing retries, error branches, parallel and map states, and approval steps for coordinating multi-service data pipelines.

A managed platform that runs Apache Spark, Hive, Presto, and other big-data frameworks on EC2 clusters, commonly using S3 as the data lake store and transient or Spot-based clusters for cost savings.

Durable object storage that organizes data into buckets and keys, offers storage classes from frequently accessed to deep archive, and usually serves as the foundation of an AWS data lake.

Rules attached to a bucket that transition objects to cheaper storage classes, expire them, or delete incomplete multipart uploads after set intervals, automating retention and cost control.

A columnar, massively parallel data warehouse with separate leader and compute nodes that runs SQL analytics over petabytes, using distribution and sort keys plus workload management to tune performance.

A capability that lets a Redshift cluster query external tables stored in S3 through the Data Catalog, joining warehouse tables with data-lake files without loading them first.

A Redshift table setting controlling how rows are spread across compute nodes; aligning it with common join or filter columns reduces network traffic during query execution.

The column order that determines how Redshift stores rows on disk, enabling block-level data skipping for range filters and improving compression and merge-join performance.

A fully managed NoSQL key-value and document database that scales horizontally through automatic partitioning and delivers consistent single-digit millisecond latency for high-throughput workloads.

The primary key attribute whose hashed value determines the item's physical partition; high-cardinality, evenly distributed access avoids hot partitions and throughput throttling.

An in-memory cache deployed in your VPC that fronts DynamoDB, answering repeated reads in microseconds and reducing consumed read capacity for eventually consistent access patterns.

A serverless query service that runs SQL over data in S3 and other cataloged sources, billed per byte scanned, commonly used for ad-hoc analysis and verifying pipeline output.

The monitoring service for metrics, logs, and alarms; data pipelines publish job durations, failure counts, and custom metrics so operators can alert and react automatically.

An infrastructure-as-code service that creates and updates AWS resources from JSON or YAML templates, with change sets, stack rollback, and drift detection for repeatable deployment of pipelines.

A serverless event bus and scheduler that matches events from AWS services or SaaS applications against rules and routes them to targets such as Glue jobs or Step Functions.

A service that records resource configuration history and evaluates it against rules, flagging noncompliant resources such as publicly readable buckets or unencrypted databases.

A managed Apache Airflow service that runs directed acyclic graphs for orchestrating data pipelines, handling environment provisioning, worker scaling, and integration with AWS services.

An identity with permission policies that trusted principals temporarily assume to obtain rotating credentials, the preferred way to grant AWS services access without long-lived access keys.

A managed service that creates and controls cryptographic keys, enforcing key policies and logging every use, and underpins encryption for S3, Redshift, DynamoDB, and other data stores.

Server-side encryption for S3 where KMS generates and protects the per-object data keys, enabling separate permissions for key use and CloudTrail auditing of decryption calls.

A resource-based JSON policy on a bucket that allows or denies actions for specified principals, frequently used to enforce TLS-only access, cross-account sharing, or public-access blocks.

A governance layer for data lakes that centralizes permission grants on catalog databases, tables, and columns, and manages S3 locations registered as data lake locations.

Attribute-based labels in Lake Formation applied to catalog resources and principals, so permissions can be granted through tag expressions instead of one table at a time.

A security service that samples and classifies S3 objects to detect sensitive data such as personal information, reporting findings and identifying buckets that are publicly accessible.

A service that records API activity across the account into event history and optional S3 log buckets, providing the audit trail needed to investigate who accessed or changed data.

A store for database credentials and API keys that retrieves secrets at runtime through IAM-controlled calls and can rotate them automatically on a schedule.

A private connection from a VPC to a supported AWS service such as S3 or KMS, keeping traffic off the public internet and often paired with endpoint policies or bucket policies.