I85PREP
⌘ K
Azure Databricks Data Engineer Associate study materials
Azure Databricks Data Engineer Associate study resources

Azure Databricks Data Engineer Associate practice questions

One question at a time. Build your Azure Databricks Data Engineer Associate understanding.

First 50 of 200 questions. Each one lists the choices, the correct answer, and an explanation. The checker at the top of this page saves progress in this browser when it is available.

Continue in the practice tool above to load more questions, 50 at a time.

Question 1Databricks Intelligence Platform

A data engineer needs a workspace object that stores reusable SQL code and can be scheduled as a job task without attaching a notebook. Which Databricks workspace object is designed for that purpose?

  1. A notebook
  2. A repo
  3. A dashboard
  4. A SQL query Correct answer.

Explanation

SQL queries are saved workspace objects that can be edited, shared, and scheduled directly as a job task without needing a notebook attached. They run against a SQL warehouse or compatible compute and are managed from the workspace SQL editor.

Question 2Databricks Intelligence Platform

Which compute resource in the Databricks workspace is optimized for running SQL and dashboard workloads with fast startup and automatic scaling?

  1. SQL warehouse Correct answer.
  2. All-purpose cluster
  3. Job cluster
  4. Single-node cluster

Explanation

SQL warehouses are compute resources purpose-built for SQL analytics, serving BI tools, dashboards, and the SQL editor with quick start times and elastic scaling. General-purpose clusters are intended for interactive notebook and mixed-language work rather than SQL serving.

Question 3Databricks Intelligence Platform

A team wants changes to notebooks and Python files tracked in a remote Git provider and pulled into the workspace for collaborative editing. Which Databricks capability provides this?

  1. Unity Catalog volumes
  2. Delta Live Tables
  3. Git folders Correct answer.
  4. Workspace files

Explanation

Git folders let you clone a remote repository into the workspace, commit and branch changes, and keep notebook and file history synchronized with the Git provider. They are the supported integration point for source control workflows in Databricks.

Question 4Databricks Intelligence Platform

An engineer wants to grant a group read access to a table managed in Unity Catalog while ensuring the grant applies to future tables in the same schema. Which approach accomplishes this?

  1. Grant SELECT on the schema Correct answer.
  2. Grant USAGE on the schema
  3. Grant SELECT on each table individually
  4. Grant SELECT on the catalog

Explanation

Granting SELECT on a schema applies to all current and future tables within that schema, which is the standard way to cover future objects. USAGE alone only allows navigation of the schema and does not confer data read rights.

Question 5Databricks Intelligence Platform

Which statement accurately describes the relationship between a metastore and a catalog in Unity Catalog?

  1. A catalog contains metastores, each with its own schemas
  2. Schemas contain catalogs, which contain metastores
  3. A metastore is the same object as a schema
  4. A metastore contains catalogs, and catalogs contain schemas and tables Correct answer.

Explanation

Unity Catalog uses a three-level namespace: catalog.schema.table, with all catalogs attached to a metastore that is assigned to a region. The metastore is the top-level container of metadata and governance objects.

Question 6Databricks Intelligence Platform

A data engineer schedules a notebook job that must run only when an upstream Delta table has new data, and the check should avoid scanning the full table. Which feature should be used?

  1. A continuous trigger
  2. A file arrival trigger on the table location
  3. A cron schedule with a fixed time
  4. A table update trigger using change data feed Correct answer.

Explanation

A table update trigger fires when the specified table has new commits, and change data feed can be used to detect updates without a full table scan. This makes the job event-driven rather than time-driven.

Question 7Databricks Intelligence Platform

Which statement best describes the purpose of a Databricks cluster policy?

  1. It controls network access to the workspace
  2. It enforces row-level filters on queries
  3. It restricts the configuration attributes users can set when creating clusters Correct answer.
  4. It defines which users may query a table

Explanation

Cluster policies are JSON rules that limit or fix cluster configuration attributes such as node type, autoscaling bounds, and runtime version, helping control cost and compliance. They govern cluster creation settings rather than data access or network security.

Question 8Databricks Intelligence Platform

A team needs to run a nightly production pipeline with the lowest cost while ensuring a dedicated cluster is created for that run. Which approach is appropriate?

  1. Run the notebook manually on a single-node cluster
  2. Use a SQL warehouse with auto-stop disabled
  3. Attach the job to a shared all-purpose cluster
  4. Create a job cluster that is terminated after the run Correct answer.

Explanation

Job clusters are provisioned for a specific job run and terminated afterward, which isolates the workload and avoids paying for idle time. This is the cost-efficient choice for scheduled production jobs.

Question 9Databricks Intelligence Platform

Which Databricks workspace feature provides a governed, versioned store for non-tabular files such as CSV and images under Unity Catalog?

  1. Unity Catalog volumes Correct answer.
  2. DBFS root
  3. Workspace files
  4. Delta Live Tables

Explanation

Volumes are Unity Catalog objects that provide governed storage for files, with permissions and lineage managed like other catalog objects. They are the recommended location for non-tabular data that needs governance.

Question 10Databricks Intelligence Platform

A data engineer wants to share a query result with business users so they can explore it interactively without writing SQL. Which workspace capability is designed for that?

  1. A job cluster run history
  2. A cluster policy
  3. A notebook with markdown only
  4. A Databricks dashboard built on a SQL query Correct answer.

Explanation

Databricks dashboards are built from queries and visualizations so business users can filter and explore results without writing SQL. They are the intended presentation layer for sharing curated results.

Question 11Databricks Intelligence Platform

An engineer needs to configure a job so that a failure in one task prevents downstream tasks from running while independent tasks continue. Which job feature should be configured?

  1. A credential passthrough setting
  2. Task dependencies with a run-if condition Correct answer.
  3. A cluster policy
  4. A file arrival trigger

Explanation

Task dependencies define execution order, and the run-if condition controls whether a downstream task runs based on the upstream task's outcome. Setting the condition appropriately allows independent branches to continue while blocking dependents of a failed task.

Question 12Databricks Intelligence Platform

A workspace administrator wants to restrict which users can create SQL warehouses and ensure only approved sizes are used. Which combination of controls should be applied?

  1. Cluster policies plus workspace permission settings Correct answer.
  2. Row filters plus column masks
  3. Git folders plus repo permissions
  4. Volumes plus external locations

Explanation

Cluster policies can constrain warehouse and cluster attributes such as size, while workspace permissions determine which users may create compute. Combining the two enforces both who can create resources and what configurations are allowed.

Question 13Data Ingestion and Loading

A data engineer needs to land raw JSON files from an external REST API into cloud storage before any transformation occurs. Which Azure Databricks component is designed primarily for this raw ingestion step?

  1. Unity Catalog lineage
  2. Auto Loader Correct answer.
  3. Photon vectorized engine
  4. Delta Live Tables expectations

Explanation

Auto Loader incrementally ingests new files as they arrive in cloud storage and is the recommended pattern for landing raw data before transformation. It tracks discovered files and supports schema inference and evolution, which suits API-exported JSON.

Question 14Data Ingestion and Loading

Which storage format is the default table format for managed tables created in Azure Databricks when no format is specified?

  1. ORC
  2. Avro
  3. Parquet
  4. Delta Lake Correct answer.

Explanation

Managed tables in Azure Databricks default to Delta Lake, which provides ACID transactions and time travel. Parquet, Avro, and ORC remain available but must be requested explicitly.

Question 15Data Ingestion and Loading

A pipeline must read a Parquet file located in Azure Data Lake Storage Gen2 from a Databricks notebook using the abfss:// URI. Which credential approach is a recommended, secure way to authenticate?

  1. Hard-coding the storage account key in the notebook
  2. Using a Unity Catalog storage credential backed by an access connector Correct answer.
  3. Disabling authentication on the storage account
  4. Embedding a SAS token in the cluster's Spark configuration

Explanation

A Unity Catalog storage credential wraps a managed identity or service principal so that access is granted through governed privileges instead of secrets in code. Hard-coded keys and embedded SAS tokens expose credentials in notebooks and history.

Question 16Data Ingestion and Loading

When using Auto Loader with cloudFiles format, which option enables automatic detection of new columns in incoming files and updates the target schema?

  1. cloudFiles.useNotifications
  2. cloudFiles.schemaEvolutionMode Correct answer.
  3. cloudFiles.maxFilesPerTrigger
  4. cloudFiles.backfillInterval

Explanation

cloudFiles.schemaEvolutionMode controls how Auto Loader reacts when new columns appear, with modes such as addNewColumns or rescue. The other options control throughput, notification-based discovery, and backfill behavior rather than schema handling.

Question 17Data Ingestion and Loading

A data engineer wants to create a Delta table from a DataFrame in a notebook and store its data in the managed location. Which command accomplishes this?

  1. df.createOrReplaceTempView("sales")
  2. df.write.format("delta").mode("append").saveAsTable("sales") Correct answer.
  3. df.write.format("parquet").save("/mnt/sales")
  4. df.rdd.saveAsTextFile("/mnt/sales")

Explanation

Writing with format delta and saveAsTable registers a managed Delta table in the metastore with data in the managed location. The other calls either write a different format to an explicit path or create only a temporary view.

Question 18Data Ingestion and Loading

Which ingestion pattern reads only files added since the last run by tracking a checkpoint directory?

  1. Manual file listing with dbutils.fs.ls
  2. CREATE TABLE AS SELECT over the whole folder
  3. Full directory rescan each run
  4. Auto Loader with a checkpoint location Correct answer.

Explanation

Auto Loader persists discovered-file state in a checkpoint location so subsequent runs process only new files. Rescanning or listing the full directory reprocesses everything and scales poorly.

Question 19Data Ingestion and Loading

A streaming query writes to a Delta table and the engineer wants to see newly committed rows only once per micro-batch. Which write mode should be used?

  1. update
  2. append Correct answer.
  3. overwrite
  4. complete

Explanation

Append mode adds each micro-batch's new rows to the target without reprocessing prior data. Complete mode rewrites the entire result table and overwrite replaces the target each batch.

Question 20Data Ingestion and Loading

Which Databricks feature allows loading data from a Kafka topic into a Delta table with exactly-once processing guarantees when a checkpoint is configured?

  1. COPY INTO with a file path
  2. Structured Streaming with a checkpoint location Correct answer.
  3. CREATE TABLE USING CSV
  4. dbutils.fs.cp

Explanation

Structured Streaming tracks Kafka offsets in a checkpoint, enabling exactly-once semantics when writing to a Delta sink. The other options are batch file operations and do not manage streaming offsets.

Question 21Data Ingestion and Loading

An engineer runs COPY INTO against a Delta table and reruns the same command later. What behavior should be expected for files already loaded?

  1. The command fails because the table already contains data
  2. Previously loaded files are skipped based on file metadata Correct answer.
  3. All files are loaded again, creating duplicates
  4. Only the newest file is loaded and others are deleted

Explanation

COPY INTO is idempotent by default: it tracks which files were loaded and skips them on subsequent runs unless FORCE is specified. This prevents duplicate rows when a load is retried.

Question 22Data Ingestion and Loading

A JSON feed arrives with nested arrays that must be flattened into separate rows. Which approach produces one row per array element?

  1. selectExpr with explode on the array column Correct answer.
  2. dropDuplicates on the array column
  3. filter on the array column length
  4. repartition by the array column

Explanation

The explode function in selectExpr generates one output row per element of an array column, which flattens nested arrays. Filtering, deduplication, and repartitioning do not change row cardinality per element.

Question 23Data Ingestion and Loading

A data engineer must ingest a large CSV dataset where the header row is missing and columns are pipe-delimited. Which read option set is correct?

  1. header="false", sep="|" Correct answer.
  2. inferSchema="false", sep="\t"
  3. header="true", sep=","
  4. multiline="true", sep="|"

Explanation

Setting header to false treats the first line as data, and sep set to the pipe character matches the delimiter. The other options either assume a header, use the wrong delimiter, or enable multiline parsing.

Question 24Data Ingestion and Loading

When configuring Auto Loader with file notifications, which cloud resource must be set up to receive event notifications from the storage account?

  1. A Key Vault secret scope
  2. An Azure Event Grid and Queue Storage subscription Correct answer.
  3. An Azure SQL Database
  4. A Databricks SQL warehouse

Explanation

File notification mode relies on Azure Event Grid to publish blob events and Queue Storage to hold them for Auto Loader to consume. SQL warehouses, databases, and secret scopes do not deliver storage events.

Question 25Data Ingestion and Loading

A pipeline needs to merge change data capture records into a Delta table, updating matched rows and inserting new ones. Which Delta operation is appropriate?

  1. INSERT OVERWRITE
  2. TRUNCATE TABLE
  3. MERGE INTO Correct answer.
  4. OPTIMIZE

Explanation

MERGE INTO supports conditional matching so that existing keys are updated and new keys are inserted in a single atomic operation. The other commands replace or reorganize data without key-based matching.

Question 26Data Ingestion and Loading

Which Delta Lake feature allows a streaming read to see rows committed after the stream started without reprocessing the entire table?

  1. Change Data Feed Correct answer.
  2. Table cloning
  3. Z-Order clustering
  4. Vacuum

Explanation

Change Data Feed records row-level changes so downstream consumers can read only inserted, updated, and deleted rows since a version. Z-Order, vacuum, and cloning support layout and lifecycle management rather than incremental change reads.

Question 27Data Ingestion and Loading

An engineer wants to ingest only files under a specific date partition folder using Auto Loader. Which configuration limits discovery to that path?

  1. Set cloudFiles.useNotifications to false
  2. Set cloudFiles.maxFilesPerTrigger to one
  3. Set the schema location to the subfolder
  4. Set the path option to the partition subfolder Correct answer.

Explanation

Auto Loader discovers files under the path supplied in the read options, so pointing it at the partition subfolder restricts ingestion to that folder. The other options affect throughput, discovery method, or schema storage only.

Question 28Data Ingestion and Loading

A Parquet dataset is being read with schema inference disabled and an explicit schema provided. Why is this recommended in production ingestion?

  1. It automatically repairs corrupt files
  2. It enables Photon acceleration
  3. It avoids a full data scan and prevents unintended schema drift Correct answer.
  4. It removes the need for a checkpoint

Explanation

Providing an explicit schema avoids the extra pass over the data that inference requires and keeps column types stable across runs. It does not repair files, enable Photon, or replace checkpointing.

Question 29Data Ingestion and Loading

A data engineer loads a CSV where some numeric fields contain empty strings. Which read option captures malformed records into a separate column instead of failing the job?

  1. mode="PERMISSIVE" with columnNameOfCorruptRecord Correct answer.
  2. mode="DROPMALFORMED"
  3. multiLine="true"
  4. mode="FAILFAST"

Explanation

Permissive mode records malformed rows in a column named by columnNameOfCorruptRecord so the job continues and bad data can be inspected. FAILFAST aborts the job and DROPMALFORMED silently discards rows.

Question 30Data Ingestion and Loading

Which statement correctly describes the difference between a managed and an external table in Unity Catalog?

  1. An external table stores metadata outside the metastore, while a managed table stores metadata inside it
  2. A managed table cannot use Delta Lake, while an external table always uses Delta Lake
  3. A managed table requires a checkpoint, while an external table does not
  4. A managed table's data location is controlled by the metastore, while an external table points to a user-specified path Correct answer.

Explanation

For managed tables the catalog manages the storage location and drops delete the data, whereas external tables reference a path the user controls. Both table types can use Delta Lake and both store metadata in the metastore.

Question 31Data Ingestion and Loading

A pipeline reads from an Azure Event Hubs topic using Structured Streaming. Which connection option is required to establish the stream?

  1. kafka.bootstrap.servers only
  2. eventhubs.connectionString Correct answer.
  3. jdbc.url
  4. cloudFiles.schemaLocation

Explanation

The Event Hubs connector requires eventhubs.connectionString to identify the namespace and authenticate. Schema location applies to Auto Loader, jdbc.url to JDBC sources, and bootstrap servers alone are insufficient for Event Hubs authentication.

Question 32Data Ingestion and Loading

An engineer observes that an Auto Loader stream is not picking up new files in a storage account. Which configuration issue is the most likely cause?

  1. The target table uses Delta Lake
  2. The stream was started with a stale checkpoint pointing to another path Correct answer.
  3. The checkpoint location differs from the schema location
  4. The cluster has too many worker nodes

Explanation

A checkpoint tied to a different source path causes the stream to track the wrong directory, so new files in the intended path are ignored. Checkpoint and schema locations are independent, cluster size does not block discovery, and Delta is the expected sink format.

Question 33Data Ingestion and Loading

A team ingests files continuously and wants exactly-once delivery into a Delta table. Which combination of settings is required?

  1. A checkpoint and complete output mode
  2. A shared checkpoint location across all streams and a Parquet sink
  3. A unique checkpoint location per stream and a Delta sink Correct answer.
  4. No checkpoint and append mode

Explanation

Exactly-once into Delta requires each stream to have its own checkpoint location so offsets are tracked independently, combined with the transactional Delta sink. Sharing checkpoints causes conflicts, and Parquet lacks transactional guarantees.

Question 34Data Ingestion and Loading

A data engineer must load data from an on-premises SQL Server into Delta Lake on a schedule with minimal operational overhead. Which Azure Databricks approach fits best?

  1. Manual notebook execution with dbutils.fs.cp
  2. Streaming from Event Hubs
  3. A Lakeflow job running a JDBC read into a Delta table Correct answer.
  4. COPY INTO from a local file path

Explanation

A scheduled job performing a JDBC read with the SQL Server connector loads relational data into Delta and is orchestrated by the job scheduler. Copying files or reading Event Hubs does not connect to the on-premises database.

Question 35Data Ingestion and Loading

An engineer needs to ingest a large volume of small files efficiently. Which practice reduces overhead during Auto Loader ingestion?

  1. Increasing the number of checkpoint directories
  2. Enabling file notification mode and compacting the target with OPTIMIZE Correct answer.
  3. Reading each file with a separate stream
  4. Disabling schema inference and deleting the checkpoint

Explanation

File notification mode avoids costly directory listings and OPTIMIZE compacts many small target files to improve downstream reads. Multiple checkpoints, deleted checkpoints, and per-file streams all add overhead and risk inconsistency.

Question 36Data Ingestion and Loading

A streaming merge into a Delta table must handle out-of-order late-arriving updates so the latest event timestamp wins. Which design achieves this?

  1. Use overwrite mode each batch
  2. Use DROPMALFORMED on the source
  3. Use append mode and accept duplicate keys
  4. Use MERGE with a condition that updates only when the incoming timestamp exceeds the stored timestamp Correct answer.

Explanation

A conditional MERGE that compares timestamps ensures older late-arriving records do not overwrite newer state. Appending duplicates leaves conflicting rows, overwrite discards history, and DROPMALFORMED affects parsing only.

Question 37Data Ingestion and Loading

A pipeline must land semi-structured event data with evolving fields while preserving unknown fields for later inspection. Which Auto Loader behavior supports this?

  1. Fixed schema with DROPMALFORMED
  2. Schema evolution with the rescued data column enabled Correct answer.
  3. Schema enforcement that rejects new fields
  4. Disabling checkpoints to re-read all files

Explanation

Schema evolution adds new columns as they appear, and the rescued data column retains fields that could not be parsed so they are not lost. Rejecting new fields or dropping malformed rows would discard valid evolving data.

Question 38Data Ingestion and Loading

A data engineer wants Auto Loader to ingest files from an ADLS Gen2 container without configuring cloud notification services. Which approach should be used?

  1. Directory listing mode with the cloudFiles.useNotifications option set to true
  2. File notification mode with an Event Grid subscription
  3. Directory listing mode with the cloudFiles.useNotifications option set to false Correct answer.
  4. File notification mode with a queue storage account

Explanation

Auto Loader supports two discovery modes: file notification mode, which requires cloud resources such as Event Grid and a queue, and directory listing mode, which discovers files by listing the directory and needs no extra cloud configuration. Setting cloudFiles.useNotifications to false selects directory listing mode.

Question 39Data Ingestion and Loading

An engineer needs to create an external table over Parquet files already stored in an ADLS Gen2 path, so that dropping the table does not delete the underlying data. Which SQL statement accomplishes this?

  1. CREATE VIEW sales AS SELECT * FROM parquet.`abfss://[email protected]/sales`
  2. CREATE TABLE sales (id INT) USING PARQUET
  3. CREATE TABLE sales USING PARQUET AS SELECT * FROM parquet.`abfss://[email protected]/sales`
  4. CREATE TABLE sales USING PARQUET LOCATION 'abfss://[email protected]/sales' Correct answer.

Explanation

An external table is defined with an explicit LOCATION that points to existing storage; the table metadata references that path and dropping the table leaves the files intact. The other forms either create a managed table or a view rather than an external table over the existing path.

Question 40Data Ingestion and Loading

Which format provides a built-in schema evolution capability that lets Auto Loader add new columns to the target Delta table automatically when new fields appear?

  1. cloudFiles.schemaEvolutionMode set to rescue
  2. cloudFiles.schemaEvolutionMode set to addNewColumns Correct answer.
  3. cloudFiles.schemaEvolutionMode set to failOnNewColumns
  4. cloudFiles.schemaEvolutionMode set to none

Explanation

Auto Loader schema evolution modes control how new columns are handled. The addNewColumns mode automatically adds newly detected columns to the schema and to the target Delta table so ingestion continues without manual intervention.

Question 41Data Ingestion and Loading

A data engineer wants to load a Delta table from an Azure Synapse Analytics table using a JDBC connection in a Databricks notebook. Which component performs the parallel read across the Synapse table?

  1. A single JDBC connection reading the entire table
  2. The Spark JDBC data source with partitioning options such as partitionColumn, numPartitions, and bounds Correct answer.
  3. The COPY INTO command with a JDBC URL
  4. The Auto Loader cloudFiles source configured with a JDBC path

Explanation

The Spark JDBC data source can split a remote table into parallel partitions when partitionColumn, lowerBound, upperBound, and numPartitions are provided. This distributes the read across executors instead of pulling all rows through one connection.

Question 42Data Ingestion and Loading

An engineer uses the COPY INTO SQL command to load CSV files from a storage location into a Delta table. Which behavior describes how COPY INTO handles files that were already loaded by a previous run of the same command?

  1. It requires a manual checkpoint directory to be specified
  2. It tracks previously loaded files and skips them unless the files have changed Correct answer.
  3. It deletes the target table before each load
  4. It reloads every file in the path on each execution

Explanation

COPY INTO maintains load history for the target table and skips files that were already processed, unless the file content changed or the command is forced. This makes repeated executions idempotent for unchanged source files.

Question 43Data Ingestion and Loading

An engineer wants to read a streaming source from Azure Event Hubs into a Delta table using Structured Streaming. Which option correctly identifies the Event Hubs connection configuration?

  1. Set the format to 'kafka' and provide only a bootstrap server address
  2. Set the format to 'json' and provide the Event Hubs connection string in the schema
  3. Set the format to 'delta' and provide the Event Hubs namespace
  4. Set the format to 'eventhubs' and provide the connection string and event hub name via options Correct answer.

Explanation

The Event Hubs connector is selected with format('eventhubs'), and the connection is configured with options such as eventhubs.connectionString and eventhubs.name. This lets Structured Streaming consume the hub as a source.

Question 44Data Ingestion and Loading

A data engineer wants to flatten a nested JSON payload so that each element of an array field becomes its own row while the other fields are repeated. Which operation should be applied?

  1. The collect_list aggregation on the array column
  2. The to_json function on the array column
  3. The struct function on the array column
  4. The explode function on the array column Correct answer.

Explanation

The explode function takes an array column and produces one output row per element, repeating the other columns for each element. This is the standard way to turn array elements into rows in Spark SQL.

Question 45Data Ingestion and Loading

An engineer needs to ingest a large CSV file where the header row is absent and fields are separated by a pipe character. Which read configuration correctly handles both requirements?

  1. header set to false and delimiter set to ','
  2. header set to false and sep set to '|' Correct answer.
  3. header set to true and quote set to '|'
  4. header set to true and sep set to ','

Explanation

The header option controls whether the first row is treated as column names, and sep (or delimiter) sets the field separator. For a headerless pipe-delimited file, header should be false and sep should be '|'.

Question 46Data Ingestion and Loading

A data engineer is loading a CSV where some numeric columns contain empty strings and malformed values. The engineer wants a single column that captures any records or fields that could not be parsed. Which option enables this?

  1. badRecordsPath set to a storage location
  2. mode set to dropMalformed
  3. mode set to permissive
  4. columnNameOfCorruptRecord set to a chosen column Correct answer.

Explanation

The columnNameOfCorruptRecord option stores unparsable records in a designated column so they can be inspected instead of failing the job. The permissive mode parses what it can but does not by itself retain the raw malformed record in a column.

Question 47Data Ingestion and Loading

An engineer is using Auto Loader and wants the stream to process only files located under a specific date partition folder such as /events/date=2024-01-15/. Which configuration achieves this with the least data scanned?

  1. Read the entire /events directory and filter rows with a WHERE clause on date
  2. Set cloudFiles.maxFilesPerTrigger to one
  3. Use cloudFiles.schemaLocation to point to the date folder
  4. Use a glob path such as /events/date=2024-01-15/* Correct answer.

Explanation

Auto Loader accepts glob patterns in the source path, so restricting the path to a specific partition folder limits discovery to those files. Filtering after reading the full directory scans far more files and is less efficient.

Question 48Data Ingestion and Loading

A pipeline uses MERGE INTO to apply change data capture records to a Delta table. Which clause ensures that when a source record matches an existing target row, that row is updated with the latest values?

  1. WHEN MATCHED THEN UPDATE SET * Correct answer.
  2. WHEN MATCHED THEN DELETE
  3. WHEN NOT MATCHED BY SOURCE THEN DELETE
  4. WHEN NOT MATCHED THEN INSERT *

Explanation

The WHEN MATCHED THEN UPDATE SET clause updates target rows that match a source row, and SET * copies all matching source columns. This applies the latest CDC values to existing records.

Question 49Data Ingestion and Loading

An engineer observes that an Auto Loader stream has stopped discovering new files in an ADLS Gen2 account even though files continue to arrive. Which configuration is the most likely cause when file notification mode is in use?

  1. The cloudFiles.schemaLocation points to a different container
  2. The checkpoint location is set to a DBFS path
  3. The cloudFiles.format is set to parquet
  4. The Event Grid subscription or queue used for notifications has expired or been misconfigured Correct answer.

Explanation

In file notification mode, Auto Loader relies on cloud event notifications delivered through Event Grid and a queue. If the subscription or queue is expired, deleted, or misconfigured, new file events are not delivered and discovery stalls.

Question 50Data Ingestion and Loading

A team wants exactly-once delivery from a streaming source into a Delta table. Which combination of settings is required to achieve this?

  1. A checkpoint location plus the overwrite write mode on each micro-batch
  2. The complete output mode with a memory sink
  3. A checkpoint location plus a fault-tolerant source and a Delta sink Correct answer.
  4. The append output mode without a checkpoint location

Explanation

Exactly-once processing in Structured Streaming requires a reliable checkpoint location to track progress, a replayable source, and an idempotent sink such as Delta Lake. Together these allow recovery without duplicating or losing records.