I85PREP
⌘ K
Databricks Data Engineer Professional study materials
Databricks Data Engineer Professional study resources

Databricks Data Engineer Professional practice questions

One question at a time. Build your Databricks Data Engineer Professional understanding.

First 50 of 200 questions. Each one lists the choices, the correct answer, and an explanation. The checker at the top of this page saves progress in this browser when it is available.

Continue in the practice tool above to load more questions, 50 at a time.

Question 1Developing Code for Data Processing using Python and SQL

In a Delta Lake MERGE INTO statement, which clause deletes target rows that have a matching row in the source?

  1. WHEN MATCHED THEN DELETE Correct answer.
  2. WHEN NOT MATCHED THEN DELETE
  3. WHEN NOT MATCHED BY SOURCE THEN DELETE
  4. WHEN MATCHED THEN TRUNCATE

Explanation

MERGE supports WHEN MATCHED, WHEN NOT MATCHED (evaluated against target rows), and WHEN NOT MATCHED BY SOURCE. The DELETE action is valid in the WHEN MATCHED clause and deletes the matched target row.

Question 2Developing Code for Data Processing using Python and SQL

Which value of the format option configures a Structured Streaming read to use Auto Loader?

  1. cloudFiles Correct answer.
  2. fileStream
  3. deltaFiles
  4. autoLoader

Explanation

Auto Loader is enabled by specifying the cloudFiles format when reading from a cloud storage path. The remaining options either are not registered formats or are the internal format used by generic file sources.

Question 3Developing Code for Data Processing using Python and SQL

What is the default retention threshold used by VACUUM on a Delta table?

  1. 168 hours Correct answer.
  2. 24 hours
  3. 720 hours
  4. 48 hours

Explanation

VACUUM removes data files that are no longer referenced by the transaction log and that are older than the retention threshold, which defaults to 7 days (168 hours). Shorter intervals require disabling the retention safety check.

Question 4Developing Code for Data Processing using Python and SQL

Which SQL statement reads version 3 of the Delta table events?

  1. SELECT * FROM events AT VERSION 3
  2. SELECT * FROM events TIMESTAMP AS OF 3
  3. SELECT * FROM events VERSION AS OF 3 Correct answer.
  4. SELECT * FROM events USING VERSION 3

Explanation

Delta time travel uses a table reference followed by either VERSION AS OF or TIMESTAMP AS OF. VERSION AS OF accepts an integer version number.

Question 5Developing Code for Data Processing using Python and SQL

Which statement describes how coalesce(n) differs from repartition(n)?

  1. coalesce reduces the partition count without a full shuffle Correct answer.
  2. repartition can only decrease the partition count
  3. repartition preserves the existing data distribution exactly
  4. coalesce always performs a full shuffle of the data

Explanation

coalesce merges existing partitions and avoids a wide shuffle, so it is typically used to reduce the number of output files. repartition performs a full shuffle and can both increase and decrease the partition count.

Question 6Developing Code for Data Processing using Python and SQL

Which table property enables Change Data Feed on an existing Delta table?

  1. delta.changeDataFeed.enabled = true
  2. delta.enableChangeDataFeed = true Correct answer.
  3. delta.trackChanges = true
  4. delta.cdf.enabled = true

Explanation

Change Data Feed is controlled by the table property delta.enableChangeDataFeed. It can be set at table creation or altered on an existing table, and starting from that point change events are written.

Question 7Developing Code for Data Processing using Python and SQL

What storage level does cache() use by default when called on a DataFrame?

  1. MEMORY_AND_DISK_SER
  2. DISK_ONLY
  3. MEMORY_AND_DISK Correct answer.
  4. MEMORY_ONLY

Explanation

For DataFrames and Datasets, cache() is a shortcut for persist() with the MEMORY_AND_DISK storage level, so partitions that do not fit in memory spill to disk.

Question 8Developing Code for Data Processing using Python and SQL

Which function produces one output row for each element of an array column?

  1. collect_list
  2. flatten
  3. size
  4. explode Correct answer.

Explanation

explode is a generator function that returns a new row for every element of an array or map; an empty array produces no rows. It is commonly used with LATERAL VIEW or in the select list.

Question 9Developing Code for Data Processing using Python and SQL

What does OPTIMIZE with a ZORDER BY clause accomplish on a Delta table?

  1. It repartitions the table so each partition holds one value of the column
  2. It co-locates related rows in the same files so file skipping is more effective Correct answer.
  3. It sorts the entire table by the column and rewrites it as a single file
  4. It enables deletion vectors on the specified column

Explanation

Z-Ordering is a multi-dimensional clustering technique applied during file compaction that places related rows in the same set of files, improving data skipping for the listed columns. It does not change the partitioning scheme of the table.

Question 10Developing Code for Data Processing using Python and SQL

In which situation is a broadcast join hint most appropriate?

  1. When the join key contains many null values
  2. When one join side is small enough to fit in executor memory Correct answer.
  3. When the query uses a full outer join
  4. When both join sides are large and similar in size

Explanation

A broadcast hint ships the smaller side of the join to every executor, eliminating the shuffle of the larger side. It is safe only when that side is small enough to fit in executor memory.

Question 11Developing Code for Data Processing using Python and SQL

Which task requires foreachBatch rather than a built-in streaming sink?

  1. Appending the stream to a Delta table
  2. Merging each micro-batch into a Delta target table Correct answer.
  3. Reading a stream from a Delta table
  4. Computing a watermark on event time

Explanation

foreachBatch exposes each micro-batch as a static DataFrame, which allows arbitrary DataFrame operations such as MERGE INTO or writes to multiple sinks. Built-in sinks only support append-style writes to a single target.

Question 12Developing Code for Data Processing using Python and SQL

Which DataFrameWriter option adds new columns to an existing Delta table during an append?

  1. overwriteSchema
  2. mergeSchema Correct answer.
  3. schemaEvolution
  4. autoMerge

Explanation

Setting mergeSchema to true on an append updates the table schema to include columns present in the incoming DataFrame, leaving existing rows with nulls in the new columns.

Question 13Developing Code for Data Processing using Python and SQL

Which statement about generated columns in a Delta table is correct?

  1. Values may be computed with non-deterministic functions such as rand()
  2. Values must be supplied explicitly by the writer
  3. Values are computed from an expression and must not be provided on insert Correct answer.
  4. Values can only reference columns defined after them

Explanation

A generated column is defined with a deterministic expression over other columns of the same table, and Delta computes its value on write. Writers that attempt to supply a value for the column receive an error.

Question 14Developing Code for Data Processing using Python and SQL

Which statement contrasts DEEP CLONE with SHALLOW CLONE?

  1. DEEP CLONE copies data and metadata; SHALLOW CLONE copies metadata only Correct answer.
  2. SHALLOW CLONE is read-only, while DEEP CLONE is writable
  3. Both copy data files, but SHALLOW CLONE copies no metadata
  4. DEEP CLONE copies only the metadata, while SHALLOW CLONE copies the data files

Explanation

DEEP CLONE makes a full independent copy of both the metadata and the data files. SHALLOW CLONE copies the transaction log metadata and references the source data files, so it is cheaper but depends on the source files remaining in place.

Question 15Developing Code for Data Processing using Python and SQL

What does the QUALIFY clause allow in a SQL query?

  1. It declares a named window specification for reuse
  2. It filters on the result of a window function without a subquery Correct answer.
  3. It filters rows before window functions are evaluated
  4. It controls the join order within a query

Explanation

QUALIFY is evaluated after window functions are computed, so it can filter on a window expression directly in the same query level instead of wrapping the query in a subquery.

Question 16Developing Code for Data Processing using Python and SQL

Which expression returns only the positive elements of the array column nums?

  1. exists(nums, x -> x > 0)
  2. aggregate(nums, false, (x, acc) -> x > 0)
  3. filter(nums, x -> x > 0) Correct answer.
  4. transform(nums, x -> x > 0)

Explanation

filter is a higher-order function that keeps only the array elements for which the lambda predicate is true, returning an array of the same element type.

Question 17Developing Code for Data Processing using Python and SQL

Which trigger processes all newly available data and then stops the streaming query?

  1. Trigger.ProcessingTime('0 seconds')
  2. Trigger.AvailableNow() Correct answer.
  3. Trigger.ProcessingTime('1 minute')
  4. Trigger.Continuous('1 second')

Explanation

Trigger.AvailableNow() processes all data available at query start, using multiple micro-batches if needed, and then terminates. It is the pattern used for scheduled, cost-controlled streaming jobs.

Question 18Developing Code for Data Processing using Python and SQL

Why must each streaming query use its own checkpoint location?

  1. It enforces the schema of the streaming source
  2. It tracks source offsets and state, which must not be shared between queries Correct answer.
  3. It caches the query plan for the optimizer
  4. It stores output rows before they are committed to the sink

Explanation

The checkpoint directory holds the source offsets, commit metadata, and state store for a query. Reusing one directory across queries makes each query read the other's progress and corrupts exactly-once behavior.

Question 19Developing Code for Data Processing using Python and SQL

When the same source file appears in the location on a second COPY INTO run, what happens by default?

  1. The file is loaded only if its modification time has changed
  2. The command fails with a duplicate-file error
  3. The file is loaded again, producing duplicate rows
  4. The file is skipped because it is recorded as already loaded Correct answer.

Explanation

COPY INTO tracks the files it has already ingested and skips them on subsequent runs unless the load metadata is explicitly reset, which is what makes it idempotent for repeated runs.

Question 20Developing Code for Data Processing using Python and SQL

A Delta table has the CHECK constraint amount > 0 and a write attempts to insert a row where amount = -5. What happens by default?

  1. The row is silently dropped from the batch
  2. The row is written with the column set to null
  3. The write fails and the transaction is rolled back Correct answer.
  4. The row is written to a quarantine table

Explanation

Delta table constraints are enforced atomically: if any row violates a CHECK constraint, the entire write fails and nothing from that transaction is committed. Writers are expected to clean data before loading.

Question 21Developing Code for Data Processing using Python and SQL

A MERGE INTO statement fails with DELTA_MULTIPLE_SOURCE_ROW_MATCHING_TARGET_ROW_IN_MERGE. Which fix is appropriate?

  1. Deduplicate the source so at most one row matches each target row Correct answer.
  2. Enable schema evolution on the target table
  3. Increase the shuffle partition count before the merge
  4. Add a broadcast hint to the merge statement

Explanation

MERGE requires that at most one source row matches each target row, otherwise the outcome of UPDATE or DELETE is ambiguous. The standard fix is to aggregate or deduplicate the source, for example with a window function or dropDuplicates.

Question 22Developing Code for Data Processing using Python and SQL

Which statement about liquid clustering in Delta Lake is correct?

  1. Clustering keys can be changed later without rewriting existing data files Correct answer.
  2. Each clustering key value is stored in its own file
  3. Z-Ordering can be applied to the same table in the same OPTIMIZE command
  4. The table must also be partitioned on the clustering keys

Explanation

Liquid clustering replaces the partition-plus-Z-Order approach: clustering keys can be redefined with ALTER TABLE, and existing data is not rewritten until OPTIMIZE re-clusters it. This makes the physical layout adaptable to changing query patterns.

Question 23Developing Code for Data Processing using Python and SQL

What is the effect of enabling deletion vectors on a Delta table?

  1. DELETE statements are no longer permitted on the table
  2. The table can no longer be queried by time travel
  3. Deleted rows are marked in a deletion vector instead of rewriting the data file Correct answer.
  4. Deleted rows are physically removed at the time of the delete

Explanation

With deletion vectors, a delete records the affected row positions in a sidecar vector file rather than rewriting the affected Parquet file, which makes deletes and updates much cheaper. Readers apply the vector to hide the soft-deleted rows.

Question 24Developing Code for Data Processing using Python and SQL

Which API applies a Python function that receives one pandas DataFrame per group and returns a pandas DataFrame per group?

  1. A series-to-series pandas_udf applied in a select
  2. foreachPartition with a plain Python function
  3. applyInPandas after a groupBy Correct answer.
  4. mapInPandas over an iterator of DataFrames

Explanation

applyInPandas implements the split-apply-combine pattern: it is invoked on a grouped DataFrame, runs the function once per group with a pandas DataFrame argument, and concatenates the returned pandas DataFrames.

Question 25Developing Code for Data Processing using Python and SQL

Which Adaptive Query Execution feature handles a join in which a few partitions hold far more rows than the rest?

  1. Reusing exchange operators across the query
  2. Dynamic partition pruning
  3. Skew join handling, which splits oversized partitions into smaller ones Correct answer.
  4. Broadcasting the smaller join side

Explanation

AQE detects partitions that are much larger than the median and splits them into smaller sub-partitions while replicating the matching rows from the other side, so the work is spread across tasks.

Question 26Developing Code for Data Processing using Python and SQL

A DataFrame pipeline applies a plain Python UDF row by row and spends most of its time in serialization between the JVM and Python. Which change most directly reduces that overhead?

  1. Replace it with a vectorized pandas UDF. Correct answer.
  2. Register the function with spark.udf.register.
  3. Increase spark.sql.shuffle.partitions before the call.
  4. Cache the DataFrame before applying the function.

Explanation

A plain Python UDF is invoked once per row, so the cost of crossing the JVM/Python boundary dominates for large datasets. A vectorized pandas UDF receives batches of values as Arrow-backed Series, amortizing that cost across many rows at once.

Question 27Developing Code for Data Processing using Python and SQL

Which statement correctly contrasts coalesce(n) with repartition(n) on a DataFrame?

  1. coalesce(n) avoids a full shuffle but can leave uneven partitions. Correct answer.
  2. Both methods always preserve the existing row distribution.
  3. repartition(n) can only decrease the number of partitions.
  4. coalesce(n) performs a full shuffle to balance rows evenly.

Explanation

coalesce merges existing partitions without a shuffle, so it is cheap but can leave large and small partitions. repartition performs a full shuffle, which both redistributes rows evenly and allows the partition count to grow or shrink.

Question 28Developing Code for Data Processing using Python and SQL

DataFrame a has columns id and name, and DataFrame b has columns id and email. Which call returns a DataFrame with columns id, name and email, with nulls where a column is absent from one side?

  1. a.unionByName(b)
  2. a.join(b, on="id", how="outer")
  3. a.union(b)
  4. a.unionByName(b, allowMissingColumns=True) Correct answer.

Explanation

unionByName aligns columns by name, and allowMissingColumns=True adds columns present on only one side, filling the other side with nulls. Plain union matches by position and unionByName without the flag requires identical column sets.

Question 29Developing Code for Data Processing using Python and SQL

A source column holds the string '12.5x'. What does try_cast(col AS INT) produce for that value?

  1. It returns NULL rather than failing the query. Correct answer.
  2. It returns the original string '12.5x' unchanged.
  3. It returns 12.5 because the value is first parsed as a decimal.
  4. It returns 12, truncating at the first non-numeric character.

Explanation

try_cast behaves like cast but suppresses conversion failures by returning NULL, which keeps the query running even in ANSI mode. No partial parsing occurs, so a string containing trailing characters converts to NULL.

Question 30Developing Code for Data Processing using Python and SQL

Which join returns only the rows from the left DataFrame that have no matching row in the right DataFrame?

  1. A left semi join
  2. An inner join
  3. A left anti join Correct answer.
  4. A full outer join

Explanation

A left anti join keeps left-side rows for which no right-side match exists and returns only left-side columns. It is the standard way to express a NOT EXISTS style filter in the DataFrame API.

Question 31Developing Code for Data Processing using Python and SQL

In a window partitioned by department and ordered by salary descending, two employees tie for the highest salary. Which function assigns 1 to both tied rows and 2 to the next distinct salary?

  1. row_number()
  2. rank()
  3. dense_rank() Correct answer.
  4. percent_rank()

Explanation

dense_rank assigns the same rank to tied rows and then increments by one for the next distinct value, giving 1, 1, 2. rank does the same for ties but leaves a gap afterwards, producing 1, 1, 3.

Question 32Developing Code for Data Processing using Python and SQL

An Auto Loader stream ingests JSON files whose schema gains new columns over time. The pipeline must add those columns to the inferred schema and to the target table. Which setting accomplishes this?

  1. cloudFiles.schemaEvolutionMode = rescue
  2. cloudFiles.schemaEvolutionMode = addNewColumns Correct answer.
  3. cloudFiles.schemaEvolutionMode = failOnNewColumns
  4. cloudFiles.schemaEvolutionMode = none

Explanation

addNewColumns makes Auto Loader add newly detected fields to the inferred schema and, with a streaming write to Delta, evolve the target table so those fields are stored. The stream is restarted automatically when the schema grows.

Question 33Developing Code for Data Processing using Python and SQL

A stateful streaming aggregation uses withWatermark on the event-time column with a delay of 10 minutes. A new record arrives whose event time is older than the current watermark. What happens to that record by default?

  1. It is buffered until the watermark advances again.
  2. It restarts the query with a new checkpoint location.
  3. It is considered too late and excluded from the aggregate. Correct answer.
  4. It is written to a _rescued_data column on the target.

Explanation

The watermark bounds how long the engine keeps state for a given event time. Records whose event time falls behind the watermark are treated as too late and dropped from stateful processing rather than updating results.

Question 34Developing Code for Data Processing using Python and SQL

What is the default retention threshold that VACUUM honors before it is allowed to delete unreferenced data files from a Delta table?

  1. 30 days
  2. 0 hours
  3. 7 days Correct answer.
  4. 24 hours

Explanation

VACUUM retains unreferenced files for a default of 7 days so that in-flight readers and time-travel queries are not broken. The threshold can be changed only by explicitly overriding the retention interval setting.

Question 35Developing Code for Data Processing using Python and SQL

Which SQL statement reads version 3 of the Delta table sales through time travel?

  1. SELECT * FROM sales@v3
  2. SELECT * FROM sales VERSION AS OF 3 Correct answer.
  3. SELECT * FROM VERSION(sales, 3)
  4. SELECT * FROM sales TIMESTAMP AS OF 3

Explanation

Delta time travel supports VERSION AS OF with an integer version and TIMESTAMP AS OF with a timestamp expression. VERSION AS OF 3 returns the table as it existed after the third commit.

Question 36Developing Code for Data Processing using Python and SQL

A MERGE INTO statement must delete target rows that no longer have any matching row in the source. Which clause accomplishes this?

  1. WHEN MATCHED THEN DELETE
  2. WHEN NOT MATCHED BY SOURCE THEN DELETE Correct answer.
  3. WHEN NOT MATCHED THEN DELETE
  4. WHEN NOT MATCHED BY TARGET THEN DELETE

Explanation

WHEN NOT MATCHED BY SOURCE applies to target rows for which no source row exists, which is the branch that can delete them. WHEN NOT MATCHED (equivalently BY TARGET) applies to source rows missing from the target and inserts them.

Question 37Developing Code for Data Processing using Python and SQL

The table sales is partitioned by year. What does INSERT OVERWRITE TABLE sales PARTITION (year = 2024) SELECT ... do?

  1. It fails because a static partition cannot be combined with overwrite.
  2. It replaces every partition of the table.
  3. It appends the selected rows alongside the existing 2024 data.
  4. It replaces only the year = 2024 partition. Correct answer.

Explanation

A static partition clause scopes the overwrite to the partitions that match the specification, leaving all other partitions untouched. Without a partition clause, INSERT OVERWRITE replaces the whole table.

Question 38Developing Code for Data Processing using Python and SQL

A job reads a directory of many small, splittable Parquet files and produces far fewer tasks than expected. Which configuration controls the maximum amount of data assigned to a single read partition?

  1. spark.default.parallelism
  2. spark.sql.adaptive.advisoryPartitionSizeInBytes
  3. spark.sql.shuffle.partitions
  4. spark.sql.files.maxPartitionBytes Correct answer.

Explanation

spark.sql.files.maxPartitionBytes sets the target maximum size of each partition created by the file source, so the engine packs multiple small splittable files into one task. The default is 128 MB.

Question 39Developing Code for Data Processing using Python and SQL

Which DataFrame method returns the logical and physical plan of a query without executing it?

  1. df.printSchema()
  2. df.explain() Correct answer.
  3. df.show()
  4. df.collect()

Explanation

explain() prints the parsed, analyzed and optimized logical plans plus the physical plan, all derived from the lazy query plan. No data is read from the source when it is called.

Question 40Developing Code for Data Processing using Python and SQL

Using a higher-order function, which expression returns the total of all elements in the array column numbers as a single scalar value?

  1. aggregate(numbers, (acc, x) -> acc + x)
  2. transform(numbers, x -> x)
  3. aggregate(numbers, 0, (acc, x) -> acc + x) Correct answer.
  4. zip_with(numbers, numbers, (a, b) -> a + b)

Explanation

aggregate takes an initial accumulator value followed by a merge lambda that folds each array element into the accumulator, returning one scalar. Supplying 0 as the starting value makes the result the sum of the elements.

Question 41Developing Code for Data Processing using Python and SQL

In a Unity Catalog enabled workspace, which name references the table orders in schema sales inside catalog prod?

  1. hive_metastore.prod.sales.orders
  2. prod.sales.orders Correct answer.
  3. sales.orders
  4. orders.prod.sales

Explanation

Unity Catalog uses a three-level namespace of catalog.schema.table, so the catalog comes first, then the schema, then the table. Two-part names resolve against the session's current catalog and schema.

Question 42Developing Code for Data Processing using Python and SQL

What does the DataFrame method transform(f) do when f is a Python function?

  1. It registers the DataFrame as a temporary view for SQL queries.
  2. It converts the DataFrame into a pandas DataFrame for the function.
  3. It applies a row-wise Python UDF and appends the result as a column.
  4. It applies a function that takes and returns a DataFrame. Correct answer.

Explanation

transform passes the DataFrame itself to the supplied function and returns whatever DataFrame that function produces, which allows reusable steps to be chained. It does not trigger evaluation of the source data.

Question 43Developing Code for Data Processing using Python and SQL

Which statement correctly contrasts DataFrameWriter.insertInto with DataFrameWriter.saveAsTable when appending to an existing table?

  1. insertInto matches columns by name; saveAsTable matches by position.
  2. Both methods require identical column names and order.
  3. insertInto matches columns by position; saveAsTable matches by name. Correct answer.
  4. Both methods match columns by position only.

Explanation

insertInto resolves target columns by ordinal position and ignores the DataFrame's column names, while saveAsTable resolves them by name and fills unmatched columns with nulls. This makes insertInto sensitive to column order in the DataFrame.

Question 44Developing Code for Data Processing using Python and SQL

A PySpark job must open exactly one HTTP connection per partition and send every record of that partition through it. Which method is appropriate?

  1. foreachPartition Correct answer.
  2. toLocalIterator
  3. mapPartitions
  4. foreach

Explanation

foreachPartition invokes the supplied function once per partition on the executors, so a connection created inside the function is shared by all records in that partition. foreach instead calls the function once per record, and mapPartitions is a transformation returning a dataset.

Question 45Data Ingestion & Acquisition

Which read option tells the Auto Loader cloudFiles source which file format to parse?

  1. cloudFiles.sourceFormat
  2. spark.sql.files.sourceFormat
  3. cloudFiles.format Correct answer.
  4. cloudFiles.inputFormat

Explanation

Auto Loader requires the source format to be declared in the cloudFiles namespace, for example cloudFiles.format = 'json', 'csv', 'parquet', 'avro', or 'text'. The value selects the parser used for every discovered file.

Question 46Data Ingestion & Acquisition

How does COPY INTO avoid loading the same source files twice into a Delta table?

  1. It compares file content hashes against rows already stored in the target table.
  2. It is idempotent only when the source files are deleted after each load.
  3. Files already loaded are skipped based on recorded path and last-modified time. Correct answer.
  4. It requires a checkpointLocation to be supplied in the COPY INTO command.

Explanation

COPY INTO persists metadata about each file it loaded, specifically the path and modification time, and skips files whose metadata matches on subsequent runs. This makes repeated execution against the same source location safe as long as files are not modified in place.

Question 47Data Ingestion & Acquisition

When Auto Loader infers the schema automatically, which option must be provided?

  1. Every column must be listed in cloudFiles.schemaHints
  2. cloudFiles.schemaEvolutionMode must be set to none
  3. A DDL string must be supplied up front for the target
  4. cloudFiles.schemaLocation is set to a storage path Correct answer.

Explanation

Schema inference needs a durable place to write the inferred schema and its evolution state, supplied through cloudFiles.schemaLocation. Without it, inference cannot be persisted across stream restarts.

Question 48Data Ingestion & Acquisition

Which column holds data that could not be parsed with the declared schema when reading files with rescued data enabled?

  1. _corrupt_record
  2. _rescued_data Correct answer.
  3. _rescued_records
  4. _metadata

Explanation

Rescued data is written to a column named _rescued_data, which stores the original record along with an error message when a field cannot be cast to the declared type. It lets ingestion continue instead of dropping or failing on mismatched records.

Question 49Data Ingestion & Acquisition

A pipeline ingests tens of thousands of small files per hour and directory listing has become the bottleneck. Which change moves Auto Loader to file notification mode?

  1. Set cloudFiles.useNotifications to true and grant permissions to create cloud resources. Correct answer.
  2. Disable managed file events and supply a checkpoint location for the queue state.
  3. Set cloudFiles.fetchFileNotifications to true and point cloudFiles.schemaLocation at the queue.
  4. Set cloudFiles.cleanSource to delete so processed files no longer appear in listings.

Explanation

File notification mode is enabled with cloudFiles.useNotifications = true, which makes Auto Loader set up and consume cloud-native event services such as S3 event notifications, SQS, or Event Grid. The identity running the stream needs permissions to create and read those resources.

Question 50Data Ingestion & Acquisition

An Auto Loader stream is running with the default schema evolution behavior when a source file introduces a column that is not in the inferred schema. What occurs?

  1. The schema is widened silently and processing continues without interruption.
  2. The stream stops so the schema can be updated, and resumes after a restart. Correct answer.
  3. The new column is written into _rescued_data until the schema is updated manually.
  4. The new column is discarded and the stream keeps running unchanged.

Explanation

The default schemaEvolutionMode is addNewColumns, so the inferred schema is updated with the new field and the stream terminates with an unknown-field error. Restarting the stream picks up the evolved schema and continues from the checkpoint.