I85PREP
⌘ K
AWS Data Engineer study materials
AWS Data Engineer study resources

AWS Data Engineer practice questions

One question at a time. Build your AWS Data Engineer understanding.

First 50 of 200 questions. Each one lists the choices, the correct answer, and an explanation. The checker at the top of this page saves progress in this browser when it is available.

Continue in the practice tool above to load more questions, 50 at a time.

Question 1Data Ingestion and Transformation

A company needs to continuously deliver streaming clickstream events into Amazon S3 and have the records converted from JSON to Apache Parquet along the way, without writing or managing any consumer application. Which service should be used?

  1. Amazon Kinesis Data Firehose Correct answer.
  2. Amazon Kinesis Data Streams
  3. AWS Glue DataBrew
  4. AWS Database Migration Service

Explanation

Kinesis Data Firehose is a fully managed delivery service that buffers incoming records and writes them to destinations such as Amazon S3, optionally invoking AWS Lambda for transformation or converting the format to Parquet/ORC via the built-in record format conversion. No consumer application has to be written or scaled by the customer. Kinesis Data Streams only stores records for consumers to read, so it does not by itself deliver to S3.

Question 2Data Ingestion and Transformation

What is the primary purpose of the AWS Glue Data Catalog in a data lake architecture?

  1. It hosts the Apache Spark runtime used by Glue jobs
  2. It stores the output files written by AWS Glue jobs
  3. It schedules crawler runs on a cron expression
  4. It is a metadata repository of table definitions Correct answer.

Explanation

The Data Catalog is a persistent, Hive-compatible metadata store that holds databases, tables, schemas, and partition information, and it is shared by services such as Athena, Redshift Spectrum, and EMR. Data itself lives in stores such as Amazon S3; the catalog holds only descriptive metadata.

Question 3Data Ingestion and Transformation

A company must migrate an on-premises relational database to Amazon Aurora with minimal downtime while ongoing transactions continue to be replicated to the target. Which service provides this capability?

  1. AWS Application Migration Service
  2. AWS Database Migration Service Correct answer.
  3. AWS Glue
  4. AWS DataSync

Explanation

AWS Database Migration Service performs both the initial full load and ongoing change data capture replication, keeping the source and target in sync until cutover. AWS Application Migration Service targets servers, not database-level replication.

Question 4Data Ingestion and Transformation

How long does Amazon Kinesis Data Streams retain records by default, and what is the maximum retention that can be configured?

  1. 7 days by default, extendable up to 365 days
  2. 1 hour by default, extendable up to 7 days
  3. 24 hours by default, extendable up to 365 days Correct answer.
  4. 24 hours by default, extendable up to 30 days

Explanation

The default retention period of a Kinesis data stream is 24 hours, and the retention period can be increased up to a maximum of 365 days. Longer retention lets consumers recover from failures that occur well after the record was written.

Question 5Data Ingestion and Transformation

What role does an AWS Glue crawler play in an ingestion pipeline?

  1. It infers schemas and populates the Data Catalog Correct answer.
  2. It enforces column-level access control on tables
  3. It moves data from Amazon S3 into Amazon Redshift
  4. It runs Spark jobs that transform data on a schedule

Explanation

A crawler connects to a data store, determines the schema and partition structure of the data, and creates or updates table definitions in the Glue Data Catalog. Downstream engines such as Athena and Redshift Spectrum then query the data through those catalog tables.

Question 6Data Ingestion and Transformation

A data engineering team writes Parquet files to Amazon S3 under prefixes such as year=2024/month=03/day=17/. Why is this partitioning approach recommended?

  1. Prefixes are required for S3 server-side encryption
  2. Partition pruning lets query engines read fewer objects Correct answer.
  3. Nested prefixes enable S3 Versioning automatically
  4. Amazon S3 charges lower storage rates for nested prefixes

Explanation

When the physical layout mirrors the query predicates, engines such as Athena, Redshift Spectrum, and Spark can skip prefixes that do not satisfy the filter, reducing bytes scanned, cost, and runtime. Partitioning is a layout optimization, not a pricing, versioning, or encryption feature.

Question 7Data Ingestion and Transformation

An AWS Lambda function is triggered by Amazon S3 event notifications to validate and transform each arriving object. What is the maximum execution time available to that function per invocation?

  1. 3 minutes
  2. 15 minutes Correct answer.
  3. 60 minutes
  4. 5 minutes

Explanation

A Lambda invocation can run for at most 15 minutes (900 seconds). Ingestion workloads that need more time must hand off to a longer-running service or split the work into smaller units, because S3 event triggers are asynchronous and the object must be processed within that window.

Question 8Data Ingestion and Transformation

A research institute must move 80 TB of archived files from its data center to Amazon S3 once, but its internet link is saturated for weeks at a time. Which service is designed for this transfer?

  1. AWS Transfer Family
  2. AWS Snowball Edge Correct answer.
  3. AWS Direct Connect
  4. AWS DataSync

Explanation

Snowball Edge is a physical device shipped to the site for petabyte-scale offline transfer when network bandwidth is insufficient. Data is copied locally onto the appliance at LAN speeds and then shipped to AWS for ingestion into S3.

Question 9Data Ingestion and Transformation

Kinesis Data Firehose buffers incoming records before delivering them to Amazon S3. What is the minimum configurable buffer interval?

  1. 60 seconds Correct answer.
  2. 5 seconds
  3. 300 seconds
  4. 15 seconds

Explanation

Firehose flushes a delivery stream when either the buffer size (1–128 MB) or the buffer interval (60–900 seconds) is reached, so the lowest possible latency for the S3 destination is bounded by the 60-second minimum interval. Lower latency than that requires a different service, such as a custom consumer reading a data stream.

Question 10Data Ingestion and Transformation

A company runs AWS DMS from an on-premises MySQL database to Amazon RDS and wants changes to keep replicating after the full load completes. What must be enabled on the source database?

  1. Multi-AZ on the DMS replication instance
  2. Enhanced fan-out on the target database
  3. Binary logging on the source database Correct answer.
  4. A Glue crawler scheduled every minute

Explanation

Ongoing replication uses change data capture, which for MySQL requires binary logging (with a ROW-based format and adequate log retention) so DMS can read the change stream. Without it, DMS can perform the full load but cannot apply subsequent changes.

Question 11Data Ingestion and Transformation

A nightly AWS Glue job reads all objects under an S3 prefix and writes to a partitioned table. On some nights the job is rerun manually, and duplicate rows appear downstream. What feature addresses this by tracking which input was already processed?

  1. They create crawlers automatically for new S3 prefixes
  2. They track processed input so reruns skip it Correct answer.
  3. They cap the number of DPUs a job may consume
  4. They version the Spark script stored in Amazon S3

Explanation

Glue job bookmarks persist state about the source data already processed, so a rerun with bookmarks enabled resumes from the last processed point instead of re-reading the entire prefix. This prevents duplicated output when a job is retried or triggered again.

Question 12Data Ingestion and Transformation

A single shard in an Amazon Kinesis data stream is subject to ingestion limits. What are those write limits per shard?

  1. 1 MB/s or 1,000 records/s Correct answer.
  2. 10 MB/s or 100 records/s
  3. 5 MB/s or 10,000 records/s
  4. 2 MB/s or 2,000 records/s

Explanation

Each shard supports up to 1 MB per second or 1,000 records per second for writes, and 2 MB per second for shared reads. Exceeding the write limit results in ProvisionedThroughputExceededException, which is the standard signal to reshard or increase capacity.

Question 13Data Ingestion and Transformation

How does Amazon Kinesis Data Firehose load records into an Amazon Redshift cluster?

  1. It inserts rows through a JDBC connection one batch at a time
  2. It writes to external tables queried by Redshift Spectrum
  3. It stages records in Amazon S3 and then issues a COPY command Correct answer.
  4. It streams records to Redshift through Kinesis Data Streams

Explanation

Firehose first delivers the buffered records to an intermediate S3 bucket and then runs a Redshift COPY command on the cluster, which is the most efficient bulk-load path. Firehose therefore needs an S3 bucket, a Redshift cluster, and an IAM role with the required permissions.

Question 14Data Ingestion and Transformation

A data pipeline must run five AWS Glue jobs in sequence, branch to different jobs based on job output, retry failures with backoff, and pause for a manual approval before publishing. Which service should coordinate this?

  1. Amazon EventBridge rules
  2. AWS Glue triggers
  3. AWS Batch job queues
  4. AWS Step Functions Correct answer.

Explanation

Step Functions provides a state machine with sequencing, choice branching, retries, error handling, and human-approval wait states, and it integrates natively with Glue jobs. Glue triggers can chain jobs and fire on schedules or events but lack rich branching and approval orchestration.

Question 15Data Ingestion and Transformation

Which AWS Glue data abstraction is designed for semi-structured data and offers methods such as Relationalize, ResolveChoice, and DropNullFields?

  1. DynamicFrame Correct answer.
  2. Crawler
  3. Data Catalog table
  4. DataFrame

Explanation

A DynamicFrame is a Glue-specific collection that carries schema information per record, so it tolerates inconsistent and nested data and exposes transformation methods that resolve ambiguous types and flatten nested structures. A DataFrame requires a uniform schema and is used after converting a DynamicFrame with toDF().

Question 16Data Ingestion and Transformation

A nightly Apache Spark batch job on Amazon EMR processes six hours of work and then sits idle for the rest of the day. How can cost be reduced while keeping the job running?

  1. Run a long-running cluster with On-Demand core nodes only
  2. Enable EMRFS consistent view on the cluster
  3. Use a transient cluster with Spot task nodes Correct answer.
  4. Use Dedicated Master nodes for the driver

Explanation

A transient cluster terminates after its steps complete, so no capacity is billed while idle, and Spot instances can host task nodes because losing a task node only requires re-executing that task. Core nodes hold HDFS data and are usually kept on On-Demand capacity to avoid data loss.

Question 17Data Ingestion and Transformation

A team already operates Apache Kafka producers and consumers and wants a managed AWS service that keeps the Kafka API. Which service fits?

  1. Amazon MSK Correct answer.
  2. Amazon Kinesis Data Streams
  3. Amazon Simple Notification Service
  4. Amazon MQ for RabbitMQ

Explanation

Amazon MSK runs Apache Kafka clusters under AWS management, so existing Kafka producers, consumers, and tooling work without code changes. The other services use their own APIs and would require rewriting clients.

Question 18Data Ingestion and Transformation

A nightly COPY job loads thousands of small uncompressed CSV files from Amazon S3 into Amazon Redshift and takes far longer than expected. Which change most improves load performance?

  1. Run a separate COPY command per file inside a single transaction
  2. Load compressed files of similar size listed in a manifest Correct answer.
  3. Raise the cluster's enhanced fan-out limit before loading
  4. Insert rows with a stored procedure that loops over files

Explanation

Redshift loads best from a small number of large, compressed files whose sizes are multiples that let all slices work in parallel, and a manifest gives COPY an exact, consistent object list. Many tiny files serialize work and waste per-file overhead.

Question 19Data Ingestion and Transformation

An ingestion pipeline is triggered by Amazon S3 event notifications sent to AWS Lambda. How should the processing code be written to handle the delivery semantics of these notifications?

  1. idempotent, deduplicating on object key and version ID Correct answer.
  2. ordered, because S3 delivers events in key order
  3. discarded on retry so that duplicates are never processed
  4. serialized by setting Lambda reserved concurrency to one

Explanation

S3 event notifications are delivered at least once and can arrive out of order, so the same object may be processed more than once. Making the handler idempotent, for example by keying on bucket, object key, and version ID, keeps the output correct without dropping legitimate events.

Question 20Data Ingestion and Transformation

A business analyst needs to profile datasets, detect anomalies, and apply repeatable transformations through a visual interface without writing code. Which service is intended for this?

  1. Amazon Athena
  2. AWS Glue Studio
  3. AWS Lake Formation
  4. AWS Glue DataBrew Correct answer.

Explanation

Glue DataBrew provides a visual, no-code environment for profiling, cleaning, and normalizing data, and it captures the steps as a reusable recipe that can be run on a schedule. It is aimed at analysts rather than Spark developers.

Question 21Data Ingestion and Transformation

A company is migrating from Oracle to Aurora PostgreSQL and must convert stored procedures, views, and other schema objects in addition to moving the data. Which combination supports this heterogeneous migration?

  1. Amazon RDS Blue/Green Deployments
  2. AWS Schema Conversion Tool (SCT) Correct answer.
  3. AWS Glue DataBrew transformation recipes
  4. AWS DataSync with an S3 endpoint

Explanation

The AWS Schema Conversion Tool (or DMS Schema Conversion) analyzes the source schema and converts objects that have direct equivalents, flagging the ones that need manual rewrite, while DMS handles the data movement. Blue/Green deployments only help for same-engine upgrades, not engine changes.

Question 22Data Ingestion and Transformation

Streaming events must be continuously aggregated with windowed SQL-like logic, joined with reference data, and written to a downstream sink with exactly-once state management. Which service is designed for this?

  1. AWS Step Functions state machines
  2. Amazon Athena workgroups
  3. Amazon Kinesis Data Streams with Lambda consumers
  4. Managed Service for Apache Flink Correct answer.

Explanation

Managed Service for Apache Flink (formerly Kinesis Data Analytics) runs continuous Flink applications with event-time windows, stateful operators, checkpoints, and exactly-once sinks. Lambda consumers can process records but do not provide built-in windowing or durable operator state.

Question 23Data Ingestion and Transformation

An on-premises NFS file server holds 50 TB that must be copied to Amazon S3, followed by nightly incremental synchronization of changed files, with integrity verification reported centrally. Which service should be used?

  1. Amazon S3 Transfer Acceleration
  2. AWS Storage Gateway File Gateway
  3. AWS Snowcone with AWS Ops Hub
  4. AWS DataSync Correct answer.

Explanation

DataSync is built for scheduled, incremental transfers between on-premises storage and AWS, performing checksum-based verification and reporting task results in CloudWatch. It preserves metadata and transfers only changed data after the initial copy.

Question 24Data Ingestion and Transformation

A Kinesis Data Firehose delivery stream must redact sensitive fields in each record before the records land in Amazon S3. Firehose is configured to invoke a Lambda function for transformation. What must the function return?

  1. A single JSON document containing every transformed record
  2. An Amazon S3 key where Firehose should read the output
  3. Records with the original recordId and a transformation status Correct answer.
  4. An empty response, since Firehose reads CloudWatch Logs

Explanation

Firehose sends the buffered records to Lambda and expects a response containing the same recordIds with a result status (Ok, Dropped, or ProcessingFailed) and, for successful records, base64-encoded transformed data. Records marked Dropped or ProcessingFailed are routed to the configured S3 error prefix.

Question 25Data Ingestion and Transformation

Multiple consumer applications read from the same Kinesis data stream and each is throttled at the shared 2 MB/s per-shard read limit, even though the write rate is well within capacity. What is the appropriate remedy?

  1. Increase the retention period to the 365-day maximum
  2. Call GetRecords with a larger limit per request
  3. Split the consumers across two streams
  4. Enable enhanced fan-out on the stream Correct answer.

Explanation

Enhanced fan-out gives each registered consumer dedicated read throughput of up to 2 MB/s per shard over HTTP/2 push, instead of sharing the 2 MB/s per-shard limit across all GetRecords readers. Splitting streams would require repartitioning producers and does not address per-shard contention.

Question 26Data Ingestion and Transformation

Several analytics services, including Amazon Athena, Amazon Redshift Spectrum, and AWS Glue ETL jobs, must all discover the same table definitions for data stored in Amazon S3. Which component serves as the shared metadata repository for those definitions?

  1. An AWS Glue job bookmark
  2. AWS Glue Data Catalog Correct answer.
  3. An Amazon S3 Inventory report
  4. An Amazon Athena workgroup

Explanation

The AWS Glue Data Catalog is a Hive-compatible metastore that holds databases, tables, and partitions, and it is the single place where Athena, Redshift Spectrum, Amazon EMR, and Glue jobs look up schemas for data lake objects.

Question 27Data Ingestion and Transformation

An Amazon Kinesis data stream is created with default settings, and no retention change is made. For how long are records retained and available for consumers before they expire?

  1. 7 days
  2. 365 days
  3. 1 hour
  4. 24 hours Correct answer.

Explanation

A new Kinesis data stream keeps records for 24 hours by default, and the retention period can later be extended up to a maximum of 365 days.

Question 28Data Ingestion and Transformation

An AWS DMS task is created with the migration type 'Migrate existing data and replicate ongoing changes'. What behavior does this migration type produce?

  1. Existing rows are loaded and replication stops when the load finishes.
  2. Only rows that change after the task starts are replicated.
  3. The full table is reloaded each time a source row changes.
  4. Existing rows are loaded once, then ongoing source changes are applied. Correct answer.

Explanation

This migration type performs a one-time full load of the existing data and then keeps the target in sync by continuously applying change data capture records from the source.

Question 29Data Ingestion and Transformation

Clients spread across several continents upload large objects to one Amazon S3 bucket and experience slow, highly variable throughput over the public internet. Which S3 feature is designed to improve these uploads?

  1. S3 Requester Pays
  2. S3 Transfer Acceleration Correct answer.
  3. S3 Cross-Region Replication
  4. S3 Object Lock

Explanation

S3 Transfer Acceleration routes uploads through the nearest CloudFront edge location and then over the AWS backbone to the bucket's region, which typically improves throughput for geographically distant clients.

Question 30Data Ingestion and Transformation

A producer calls PutRecord against an Amazon Kinesis data stream. What is the maximum size of a single record, counting both the partition key and the data blob?

  1. 1 MiB Correct answer.
  2. 10 MiB
  3. 1 KiB
  4. 5 MiB

Explanation

A single Kinesis data stream record, including its partition key and data blob, may not exceed 1 MiB. Larger payloads must be split across records or stored elsewhere with a reference passed in the record.

Question 31Data Ingestion and Transformation

A Java application must read an Amazon Kinesis data stream, share shard processing across several EC2 instances, and persist checkpoints of the records it has processed. Which library is purpose-built for this?

  1. Kinesis Producer Library (KPL)
  2. Kinesis Client Library (KCL) Correct answer.
  3. AWS SDK for Java paginators
  4. The Amazon Data Firehose agent

Explanation

The Kinesis Client Library handles shard leasing and load balancing across multiple workers and stores checkpoints in DynamoDB or another configured store, so consumers can resume after restarts.

Question 32Data Ingestion and Transformation

A Spark job on Amazon EMR reads large compressed text files, but each file is handled by a single task, which keeps parallelism low. Which compression codec allows one file to be split and read by multiple tasks?

  1. bzip2 Correct answer.
  2. LZ4
  3. Zstandard
  4. gzip

Explanation

bzip2 files contain independently decompressible blocks, so Hadoop and Spark can start reading at block boundaries and assign different blocks of one file to different tasks. Codecs such as gzip require decompression from the beginning of the file, preventing splits.

Question 33Data Ingestion and Transformation

A producer sends a very high volume of small records to an Amazon Kinesis data stream and wants to reduce per-record request overhead. Which API call accepts up to 500 records in a single request?

  1. SubscribeToShard
  2. GetRecords
  3. PutRecords Correct answer.
  4. PutRecord

Explanation

PutRecords batches up to 500 records into one request with a combined request size limit of 5 MiB, which reduces HTTP overhead compared with sending each record separately.

Question 34Data Ingestion and Transformation

Amazon Data Firehose delivers JSON events to Amazon S3, and analysts want objects grouped under prefixes derived from record fields such as customer_id and event_date. Which Firehose capability should be configured?

  1. Enhanced fan-out
  2. S3 Cross-Region Replication
  3. A longer buffer interval
  4. Dynamic partitioning using a JQ expression Correct answer.

Explanation

Dynamic partitioning evaluates a JQ expression against each record to extract partition keys and writes the object to a matching S3 prefix, which lets downstream engines prune data by those keys.

Question 35Data Ingestion and Transformation

A Glue table over S3 data is partitioned by dt=YYYY-MM-DD and holds tens of thousands of partitions, so each Athena query spends noticeable time fetching partition metadata. Which feature avoids that lookup by computing partition locations from a configured pattern?

  1. Running MSCK REPAIR TABLE before each query
  2. Partition projection configured on the table Correct answer.
  3. Enabling Glue job bookmarks
  4. A more frequent Glue crawler schedule

Explanation

Partition projection derives partition values and S3 locations from table properties at query time, so Athena does not need to fetch thousands of partition records from the Data Catalog.

Question 36Data Ingestion and Transformation

An Amazon Redshift COPY command must load an explicit list of files from an Amazon S3 prefix and fail when any listed file is missing. What should the command use?

  1. An AWS Glue crawler over the prefix
  2. The MAXERROR option set to a high value
  3. A JSON manifest file listing the objects Correct answer.
  4. A wildcard prefix with PARALLEL ON

Explanation

A manifest file enumerates the exact objects to load, and COPY validates that each listed object exists before loading, so missing expected files cause the load to fail.

Question 37Data Ingestion and Transformation

During an AWS DMS migration, the target tables and columns must use different names than their source counterparts. Where is that renaming defined?

  1. In the source endpoint extra connection attributes
  2. In table mapping rules attached to the DMS task Correct answer.
  3. In an AWS Glue classifier
  4. In the replication instance parameter group

Explanation

DMS table mappings contain selection rules that choose the objects to migrate and transformation rules that rename schemas, tables, and columns, or apply other changes to the target definition.

Question 38Data Ingestion and Transformation

An AWS Glue script must call a function that accepts only a Spark DataFrame, but the dataset is currently a DynamicFrame. What is the correct step?

  1. Call toDF() on the DynamicFrame Correct answer.
  2. Call relationalize() on the DynamicFrame
  3. Write the DynamicFrame to S3 and read it back
  4. Call resolveChoice() on the DynamicFrame

Explanation

DynamicFrame provides a toDF() method that returns an equivalent Spark DataFrame, after which the full Spark SQL and DataFrame API becomes available in the same job.

Question 39Data Ingestion and Transformation

Several teams query one S3 data lake through Amazon Athena, and each team may see only specific rows and columns, with no copies of the underlying data. Which service enforces these permissions?

  1. AWS Glue job bookmarks
  2. Amazon Macie
  3. AWS Lake Formation Correct answer.
  4. Amazon S3 bucket ACLs

Explanation

AWS Lake Formation manages fine-grained permissions on catalog resources and enforces row-level and column-level filters for integrated engines such as Athena, all against a single copy of the data.

Question 40Data Ingestion and Transformation

A streaming workload has unpredictable spikes, and the team does not want to size or manage shards. Which Amazon Kinesis data stream configuration should be used?

  1. Provisioned mode with enhanced fan-out
  2. On-demand capacity mode Correct answer.
  3. Provisioned mode with manual shard splitting
  4. Amazon Data Firehose with a larger buffer

Explanation

On-demand mode automatically scales stream capacity with observed throughput, removing the need to provision, split, or merge shards manually.

Question 41Data Ingestion and Transformation

A pipeline must enforce rules such as 'the id column is unique' and 'the row count exceeds 100000' on AWS Glue tables, and stop when a rule fails. Which capability provides this?

  1. Amazon Athena prepared statements
  2. AWS Glue crawler classifiers
  3. AWS Glue Data Quality with DQDL rules Correct answer.
  4. AWS Glue FindMatches

Explanation

AWS Glue Data Quality lets you express rules in the declarative DQDL language, evaluate them against tables or DataFrames, and fail the job or pipeline when the results exceed defined thresholds.

Question 42Data Ingestion and Transformation

A team runs Apache Kafka on Amazon MSK and needs to continuously copy topics into Amazon S3 without operating any connector servers. Which option fits?

  1. Amazon MSK Connect with an S3 sink connector Correct answer.
  2. AWS DMS with a Kafka source endpoint
  3. The Kinesis Client Library
  4. Amazon EMR running Flink

Explanation

MSK Connect runs managed Kafka Connect workers, so standard connectors such as the S3 sink can be deployed without provisioning or patching the underlying connector infrastructure.

Question 43Data Ingestion and Transformation

A pipeline must enumerate hundreds of millions of objects in an S3 bucket once a day, and ListObjectsV2 calls are slow and costly at that scale. Which S3 feature provides a scheduled bulk report of objects and their metadata?

  1. S3 Batch Operations
  2. S3 Inventory Correct answer.
  3. S3 Storage Lens
  4. S3 Select

Explanation

S3 Inventory generates a scheduled CSV, ORC, or Parquet report listing the objects in a bucket along with metadata such as size, storage class, and encryption status, which is far cheaper than listing the bucket repeatedly.

Question 44Data Ingestion and Transformation

Athena queries over a large uncompressed CSV dataset scan far more bytes than they need. Storing the same data as Parquet with Snappy compression reduces scanned bytes mainly because:

  1. columnar reads touch only the columns queried Correct answer.
  2. Parquet files are limited to 128 MB
  3. Parquet stores rows as indexed plain text
  4. Parquet skips the Data Catalog entirely

Explanation

Parquet organizes values by column, so an engine can read only the columns a query references and skip row groups that cannot match the predicate, which dramatically lowers the bytes scanned and the cost.

Question 45Data Ingestion and Transformation

A data lake on Amazon S3 must support ACID commits, concurrent writers, and schema evolution while remaining queryable by Athena and Spark. Which table format should be chosen?

  1. Apache Iceberg Correct answer.
  2. ORC files registered by a crawler
  3. JSON Lines files
  4. CSV files with Hive-style partition folders

Explanation

Iceberg maintains a snapshot-based metadata tree that provides atomic commits, isolation between concurrent writers, and additive schema evolution, and it is supported by Athena, Spark, and other engines.

Question 46Data Ingestion and Transformation

Writes to an Amazon Kinesis data stream exceed the per-shard ingest limit and PutRecords calls are being throttled. Which action increases write capacity while preserving ordering for each partition key?

  1. Increase the Firehose buffer interval
  2. Extend record retention to 365 days
  3. Raise the shard count with UpdateShardCount Correct answer.
  4. Switch consumers to enhanced fan-out

Explanation

Adding shards increases the aggregate write capacity, and because Kinesis hashes the partition key to select a shard, all records with the same key still land on a single shard and keep their relative order.

Question 47Data Ingestion and Transformation

A nightly Spark job on Amazon EMR rewrites the same S3 partitions and must be safely re-runnable after a mid-job failure without leaving duplicate or partially mixed rows. Which write configuration addresses this?

  1. partitionOverwriteMode set to dynamic Correct answer.
  2. Repartitioning by the partition column before writing
  3. spark.sql.shuffle.partitions set to a large value
  4. Relying on the default file output committer

Explanation

Dynamic partition overwrite replaces only the partitions written by the current job instead of the whole table, so rerunning the job substitutes those partitions rather than appending a second copy of the data.

Question 48Data Ingestion and Transformation

An Amazon Redshift cluster in us-east-1 runs a COPY command against an S3 bucket in eu-west-1 and the load fails. What is the most likely cause?

  1. COPY cannot use IAM roles across accounts
  2. The bucket policy allows only its owner account
  3. The cluster's IAM role lacks s3:PutObject
  4. COPY requires the bucket in the cluster's region Correct answer.

Explanation

Redshift COPY and UNLOAD require the S3 bucket to be in the same AWS Region as the cluster; data in another region must first be copied into a bucket in the cluster's region.

Question 49Data Ingestion and Transformation

An AWS Glue job in account A must read from an S3 bucket owned by account B, and the bucket policy in account B already grants access to the Glue job's execution role ARN. What else is required for the read to succeed?

  1. The job must run under a service-linked role
  2. The execution role in account A must also allow the S3 actions Correct answer.
  3. The bucket must be made publicly readable
  4. Nothing else; the bucket policy alone is sufficient

Explanation

Cross-account access requires permission from both sides: the resource-based bucket policy in the owning account and an identity-based policy attached to the role making the call in the caller account.

Question 50Data Ingestion and Transformation

New optional columns begin appearing in the Parquet files of a data lake roughly once a week, and existing AWS Glue jobs fail with schema mismatch errors. Which change lets the jobs pick up the new columns automatically?

  1. Recreate the Data Catalog table every night
  2. Increase the number of Glue workers
  3. Read the Parquet source with mergeSchema enabled Correct answer.
  4. Disable job bookmarks on the jobs

Explanation

Enabling mergeSchema makes the Parquet reader reconcile the differing per-file schemas into a single unified schema, so newly added columns are included without editing the job or rebuilding the table.