I85PREP
⌘ K
Databricks Spark Developer Associate study materials
Databricks Spark Developer Associate study resources

Databricks Spark Developer Associate practice questions

One question at a time. Build your Databricks Spark Developer Associate understanding.

First 50 of 200 questions. Each one lists the choices, the correct answer, and an explanation. The checker at the top of this page saves progress in this browser when it is available.

Continue in the practice tool above to load more questions, 50 at a time.

Question 1Apache Spark Architecture and Components

What is the primary role of the driver process in a Spark application?

  1. Allocating containers from the cluster manager
  2. Executing individual tasks in parallel on worker nodes
  3. Storing shuffle data between stages on local disk
  4. Hosting the SparkContext and scheduling jobs Correct answer.

Explanation

The driver hosts the SparkContext/SparkSession, converts user code into jobs, stages, and tasks, and coordinates the executors that do the work. Executors, not the driver, run tasks and hold cached or shuffle data. The driver asks the cluster manager for resources, but the cluster manager performs the allocation.

Question 2Apache Spark Architecture and Components

Since Spark 2.0, which single entry point wraps the SparkContext and exposes the DataFrame, Dataset, and SQL APIs?

  1. RDD
  2. SQLContext
  3. SparkSession Correct answer.
  4. DAG Scheduler

Explanation

SparkSession is the unified entry point introduced in Spark 2.0 and it internally holds a SparkContext. It replaces the separate SQLContext and HiveContext objects used in Spark 1.x. RDD is a data abstraction and the DAG Scheduler is an internal scheduling component.

Question 3Apache Spark Architecture and Components

Which of these DataFrame operations is an action that triggers job execution?

  1. filter
  2. withColumn
  3. select
  4. count Correct answer.

Explanation

Actions such as count, collect, and write cause Spark to execute the accumulated plan and return a result. Transformations such as select, filter, and withColumn only build up a new, lazy plan. Evaluation stays deferred until an action is invoked.

Question 4Apache Spark Architecture and Components

How many tasks does Spark create for one stage of a job?

  1. One per executor core that is available in the cluster
  2. One per shuffle file written by the previous stage
  3. One per partition of the data in that stage Correct answer.
  4. One per stage that precedes it in the DAG

Explanation

A stage is parallelized by splitting its work over the partitions of its data, so it launches exactly one task per partition. Executor cores limit how many of those tasks run at the same time, not how many exist. Stages and preceding shuffle files do not multiply the task count.

Question 5Apache Spark Architecture and Components

What is the default value of spark.sql.shuffle.partitions, which sets the number of partitions produced by a Spark SQL shuffle?

  1. 200 partitions Correct answer.
  2. 1000 partitions
  3. 100 partitions
  4. 1 partition

Explanation

Spark SQL uses 200 shuffle partitions by default, which is usually far more than a small job needs but is configurable per application. Adaptive Query Execution can reduce this number at runtime when it is enabled. The value applies to shuffles such as joins, groupBy, and repartition.

Question 6Apache Spark Architecture and Components

Which statement best describes a Spark executor?

  1. The component that negotiates container requests with the cluster manager
  2. The process that runs the user's main method and builds the DAG of stages
  3. A worker-node JVM process that runs tasks and caches data Correct answer.
  4. A queue inside the DAG Scheduler that holds pending stages

Explanation

An executor is a JVM process launched on a worker node for a single application; it runs the tasks assigned to it and stores cached or shuffle data locally. The driver runs main() and builds the DAG. Resource negotiation belongs to the driver and cluster manager, not to executors.

Question 7Apache Spark Architecture and Components

RDDs are immutable. What does that mean in practice?

  1. Cached RDDs can never be evicted from memory once written
  2. RDDs can only be created from files, never from other RDDs
  3. Transformations return new RDDs instead of modifying existing ones Correct answer.
  4. Partitions are read-only, but individual records inside them can be updated

Explanation

Each transformation records a new RDD whose lineage points back to its parent, while the parent stays unchanged. Immutability is what makes lineage-based recomputation safe after a failure. Records inside partitions cannot be mutated in place either.

Question 8Apache Spark Architecture and Components

What is the cluster manager responsible for in a Spark deployment?

  1. Allocating resources and launching executors for applications Correct answer.
  2. Tracking the lineage of each RDD partition
  3. Scheduling the tasks of a stage onto executor cores
  4. Caching partitions across executors for reuse

Explanation

The cluster manager (Standalone, YARN, Kubernetes, or Databricks' own manager) accepts resource requests and starts executor processes on worker nodes. Task scheduling inside a stage is handled by Spark's task scheduler in the driver. Lineage and caching are Spark runtime concerns, not cluster manager duties.

Question 9Apache Spark Architecture and Components

Which transformation forces a shuffle because all values for a given key must land in the same partition?

  1. union
  2. groupByKey Correct answer.
  3. filter
  4. map

Explanation

groupByKey is a wide transformation: rows with the same key must be gathered into one reduce partition, which requires network data movement. map and filter are narrow because each output partition depends on a single input partition. union is also narrow since it concatenates partitions without redistributing rows.

Question 10Apache Spark Architecture and Components

At which points does Spark split a job's DAG into separate stages?

  1. Only at the point where the user calls an action
  2. At each boundary between one executor and the next one in the cluster
  3. At every transformation in the DAG
  4. Where a shuffle dependency requires data to move between partitions Correct answer.

Explanation

A stage is a run of narrow transformations that can be pipelined together; the boundary appears when a wide dependency forces a shuffle. Executor boundaries have no bearing on stage construction since stages are defined by dependencies in the plan. The action creates the job, not the individual stages.

Question 11Apache Spark Architecture and Components

When a user calls an action on a DataFrame, how does Spark break the work down?

  1. A job is split into stages, each running one task per partition Correct answer.
  2. A single task is created and each stage retries it on another executor
  3. One task per executor is created and split into stages across the cluster
  4. A stage is split into jobs, each of which runs one task per partition

Explanation

Each action submits a job to the DAG Scheduler, which divides it into stages at shuffle boundaries. Every stage is then turned into tasks, one per partition, and handed to the task scheduler. The hierarchy always runs downward from job to stage to task.

Question 12Apache Spark Architecture and Components

What does spark.sql.autoBroadcastJoinThreshold control?

  1. The largest result set a join may return to the driver process
  2. The largest table that Spark will broadcast to executors during a join Correct answer.
  3. The largest partition that AQE is allowed to coalesce at runtime
  4. The largest shuffle block a map task may write to local disk

Explanation

If the estimated size of one join side is below this threshold (10 MB by default), Spark broadcasts it to every executor and runs a broadcast hash join instead of a shuffle join. Setting it to -1 disables broadcast joins entirely. The setting concerns join strategy selection, not result size or shuffle block size.

Question 13Apache Spark Architecture and Components

Which of these is a capability of Adaptive Query Execution in Spark 3.x?

  1. Rewriting Python UDFs into native JVM code before execution
  2. Coalescing small post-shuffle partitions at runtime Correct answer.
  3. Compressing shuffle files written to executor local disks
  4. Replacing the DAG Scheduler with a cost-based task queue

Explanation

AQE re-optimizes the plan using runtime statistics gathered at shuffle boundaries, which lets it merge tiny shuffle partitions into fewer, larger ones. It can also switch a sort-merge join to a broadcast join and split skewed partitions. The DAG Scheduler remains in place and AQE does not compile Python UDFs.

Question 14Apache Spark Architecture and Components

What role does the Catalyst optimizer play in a Spark SQL query?

  1. It applies rule- and cost-based optimizations to the query plan Correct answer.
  2. It assigns individual tasks to executor cores and retries failures
  3. It decides when to evict cached blocks and spill them to disk
  4. It converts rows into the binary Tungsten format before shuffling

Explanation

Catalyst transforms the parsed query into a logical plan, applies rule-based rewrites such as predicate pushdown and constant folding, then chooses a physical plan using cost information. Binary row formatting is handled by Tungsten, and scheduling or memory management happens in later runtime components.

Question 15Apache Spark Architecture and Components

How does whole-stage code generation improve query performance?

  1. It falls back to interpreted execution when the plan is complex
  2. It stores intermediate results off-heap across stage boundaries
  3. It fuses a stage's operators into one generated function Correct answer.
  4. It converts Scala closures into Python UDFs for portability

Explanation

Whole-stage code generation collapses the operators of a stage into a single generated Java function, eliminating virtual calls and intermediate row materialization between operators. Intermediate rows stay in CPU registers for most of the pipeline. When an operator cannot be compiled, that portion is handled separately by interpreted processing.

Question 16Apache Spark Architecture and Components

With the MEMORY_ONLY storage level, what happens when a cached partition does not fit in memory?

  1. The partition is moved to a remote shuffle service
  2. The partition is compressed so that it fits in memory
  3. The partition is dropped and recomputed from lineage when needed Correct answer.
  4. The partition is spilled to the executor's local disk

Explanation

MEMORY_ONLY keeps blocks solely in the JVM heap, so blocks that cannot be stored are discarded and rebuilt from lineage on the next access. MEMORY_AND_DISK is the level that spills excess partitions to disk. Neither level compresses by default unless a serialized level is used.

Question 17Apache Spark Architecture and Components

What is the main difference between checkpointing a DataFrame and caching it?

  1. Checkpointing holds data in memory, while caching writes it to disk
  2. Checkpointing works only on DataFrames, while caching works only on RDDs
  3. Checkpointing writes data to reliable storage and truncates the lineage Correct answer.
  4. Checkpointing keeps the lineage, while caching truncates it

Explanation

Checkpointing materializes the data into a fault-tolerant location such as DBFS or HDFS and cuts the lineage, so recovery no longer replays the whole chain of transformations. Caching keeps data in memory or on executor disk but retains lineage so lost blocks can be recomputed. Checkpointing is therefore more expensive to produce but cheaper to recover from.

Question 18Apache Spark Architecture and Components

What does speculative execution do when a task is running much slower than its peers?

  1. It predicts the next stage's tasks and schedules them early
  2. It runs a small sample of the input first to estimate partition sizes
  3. It restarts the driver on a different node when the job stalls
  4. It launches a duplicate copy and keeps the first one to finish Correct answer.

Explanation

Speculative execution starts an extra attempt for a straggling task and accepts whichever attempt completes first, killing the other. This mitigates slow nodes or skewed task durations without changing the query plan. It does not sample data or restart the driver.

Question 19Apache Spark Architecture and Components

What does Spark's dynamic allocation feature do?

  1. It moves partitions between executors to balance the load
  2. It grows the driver heap when the DAG becomes large
  3. It increases the shuffle partition count as data volume grows
  4. It adds and removes executors according to workload demand Correct answer.

Explanation

Dynamic allocation lets an application request more executors when tasks are queued and release idle executors after they have been unused for a configured period. It scales the number of executors, not the memory of the driver or the number of partitions. On Databricks, cluster autoscaling provides similar behavior at the cluster level.

Question 20Apache Spark Architecture and Components

Why does bucketing a table on a join key speed up joins?

  1. It compresses the table more aggressively than plain Parquet files
  2. It sorts every partition so range queries can skip whole files
  3. It replicates the smaller table to every executor in the cluster
  4. It pre-shuffles rows into buckets so joins skip the shuffle Correct answer.

Explanation

Bucketing writes each table into a fixed number of hash buckets on the chosen key, so rows with the same key already live in matching files. When both sides are bucketed on the join key with compatible bucket counts, Spark can join partition-wise without an exchange. Compression and data skipping are separate optimizations.

Question 21Apache Spark Architecture and Components

A query reads a Parquet table with 400 input partitions and then performs groupBy("k").count(). With AQE disabled and all other settings at their defaults, how many shuffle blocks does the shuffle write?

  1. 600 shuffle blocks
  2. 2,000 shuffle blocks
  3. 80,000 shuffle blocks Correct answer.
  4. 400 shuffle blocks

Explanation

Each of the 400 map tasks writes one shuffle block per reduce partition, and with defaults there are 200 reduce partitions. That gives 400 x 200 = 80,000 shuffle blocks. This is why the default 200 shuffle partitions can produce excessive small files when the map side is large, and why AQE coalescing helps.

Question 22Apache Spark Architecture and Components

A job processes 20 GB of input without problems until the code calls collect() on the full result, at which point it fails with a driver OutOfMemoryError. What is the most likely explanation?

  1. Executors ran out of disk space during the shuffle write
  2. Dynamic allocation removed executors while tasks were still pending
  3. Broadcast variables exceeded the available executor heap
  4. collect() brings every row into the driver's JVM memory Correct answer.

Explanation

collect() materializes the entire result set inside the driver's heap, so a large result exhausts driver memory even when the executors are healthy. The fix is usually to write the result to storage or aggregate before returning it, or to raise spark.driver.maxResultSize and driver memory. Executor-side problems would surface as executor failures or spills, not a driver OOM.

Question 23Apache Spark Architecture and Components

A shuffle map stage has completed when the executor holding its output files is lost. What does Spark do to keep the job running?

  1. It fails the job and requires the user to resubmit the whole application
  2. It re-runs the map tasks to regenerate the lost shuffle output Correct answer.
  3. It reads the missing shuffle blocks from replicas stored on other executors
  4. It falls back to reading the original input with a single task

Explanation

Shuffle output is written to executor local storage and is not replicated by default, so the only way to recover it is to recompute the map tasks that produced it. Spark therefore reschedules those tasks and the downstream stage waits for the regenerated blocks. This also means a long lineage before the shuffle is replayed.

Question 24Apache Spark Architecture and Components

The Spark UI shows a stage whose operators are not marked as whole-stage code generated. Which explanation is consistent with Spark's behavior?

  1. The plan contains an operator that cannot be code-generated, so Spark interprets it Correct answer.
  2. The stage has more tasks than the cluster has available executor cores
  3. The query is written with the DataFrame API rather than with SQL strings
  4. The stage reads its input from a cached DataFrame instead of from Parquet files

Explanation

Whole-stage code generation requires every operator in the pipeline to support bytecode generation; unsupported operators cause Spark to fall back to interpreted processing for that portion, which appears as a separate block in the UI. Caching, task-to-core ratios, and API choice do not by themselves disable code generation. Mixing compiled and interpreted sections is expected in such plans.

Question 25Apache Spark Architecture and Components

A 200-partition DataFrame is reduced to 4 partitions, once with coalesce(4) and once with repartition(4). Why is coalesce(4) generally cheaper?

  1. coalesce compresses each partition before moving it between executors
  2. coalesce merges partitions without a full shuffle; repartition shuffles all rows Correct answer.
  3. repartition keeps the original partition count and only rewrites the files
  4. coalesce runs on the driver process, avoiding executor-to-executor transfer

Explanation

coalesce builds a narrow dependency that combines existing partitions on the executors where they already live, so no exchange stage is needed when reducing the partition count. repartition performs a hash shuffle that redistributes every row across the new partitions. That shuffle buys even distribution at the cost of network and disk I/O.

Question 26Apache Spark Architecture and Components

Which component is responsible for running the user's main() method and creating the SparkContext?

  1. The external shuffle service daemon
  2. The cluster manager's scheduling service
  3. The driver process Correct answer.
  4. The executor JVM on a worker node

Explanation

The driver hosts the application's main() routine, builds the SparkContext, converts operations into a DAG, and coordinates with the cluster manager for resources. Executors only run the tasks the driver sends them.

Question 27Apache Spark Architecture and Components

In Spark's execution hierarchy, a job is created at which moment?

  1. When an action is invoked on an RDD or DataFrame Correct answer.
  2. When a transformation is called on a DataFrame
  3. When a partition is spilled from memory to disk
  4. When the SparkContext object is instantiated

Explanation

Actions such as count() or collect() trigger the evaluation of the accumulated transformations, and each action submission creates one job. A job is then decomposed into stages and tasks.

Question 28Apache Spark Architecture and Components

In Spark 2.x and later, which object is the standard entry point for creating DataFrames and executing SQL?

  1. The SQLContext companion
  2. The SparkSession object Correct answer.
  3. The SparkConf object
  4. The JavaSparkContext wrapper

Explanation

SparkSession unifies the older SQLContext and HiveContext and wraps the underlying SparkContext while adding the catalog and DataFrame/SQL APIs. It is obtained from SparkSession.builder().

Question 29Apache Spark Architecture and Components

Spark transformations are lazily evaluated. What is the direct consequence of this design?

  1. Each transformation is executed immediately on the driver as it is called
  2. No computation occurs until an action is invoked Correct answer.
  3. Transformations are rewritten by the cluster manager before execution
  4. Only wide transformations are deferred until a checkpoint is taken

Explanation

Transformations only record lineage; Spark waits for an action before optimizing the whole chain and scheduling tasks. This lets the optimizer see the full plan before execution.

Question 30Apache Spark Architecture and Components

Which transformation has a narrow dependency on its parent RDD?

  1. sortByKey
  2. repartition
  3. groupByKey
  4. mapValues Correct answer.

Explanation

With a narrow dependency each output partition depends on at most one parent partition, so records can be pipelined without network movement. mapValues keeps the existing partitioning intact.

Question 31Apache Spark Architecture and Components

A stage processes an RDD with 12 partitions and no tasks fail or are speculated. How many tasks does Spark launch for that stage?

  1. Twelve tasks, one per partition Correct answer.
  2. One task that processes the entire stage
  3. Twelve tasks only if the stage contains a shuffle
  4. One task per executor core available in the cluster

Explanation

Task count per stage equals the number of partitions in the RDD being processed by that stage. Each task handles exactly one partition of the stage's output.

Question 32Apache Spark Architecture and Components

After a job is submitted, what does the DAG scheduler do?

  1. It splits the job DAG into stages for the task scheduler Correct answer.
  2. It fetches remote shuffle blocks for reduce tasks from other nodes
  3. It assigns every individual task to a specific executor slot on a worker node
  4. It negotiates CPU and memory resources directly with the cluster manager

Explanation

The DAG scheduler converts the job's logical DAG into physical stages at shuffle boundaries and submits each stage's task set to the task scheduler. The task scheduler then places tasks on executors.

Question 33Apache Spark Architecture and Components

In cluster deploy mode, where does the driver program run?

  1. Inside the Spark SQL catalog that is replicated on the worker nodes
  2. Inside the cluster, not on the submitting client machine Correct answer.
  3. On every executor, with one driver process per executor
  4. On the same machine that ran the spark-submit command

Explanation

In cluster deploy mode the cluster manager launches the driver inside the cluster, so the submitting client can disconnect without killing the application. Client mode, by contrast, keeps the driver on the submitting machine.

Question 34Apache Spark Architecture and Components

What is the main benefit of wrapping a small lookup table in a broadcast variable?

  1. Each executor caches one copy and tasks read it locally Correct answer.
  2. Writes from tasks to the table are merged on the driver
  3. The table is compressed and kept only in the driver's heap
  4. It forces the lookup table into a single partition for joins

Explanation

A broadcast variable is shipped to each executor once and cached there, so each task reads a local copy instead of receiving the data with every task. This removes repeated serialization and network cost.

Question 35Apache Spark Architecture and Components

How do accumulator updates behave during a Spark job?

  1. Executors read the accumulator value inside tasks to make branching decisions
  2. Tasks add to their local copy; only the driver can read the merged result Correct answer.
  3. Updates are discarded when a task is retried, so counts stay exact
  4. The driver pushes every update to all executors immediately

Explanation

Tasks may only add to an accumulator, and each task's contributions are merged back to the driver, which is the only place the value can be read. Reading the value inside a task is not supported.

Question 36Apache Spark Architecture and Components

With Spark's unified memory manager, what happens when execution memory is needed but storage memory is underused?

  1. The job fails with an out-of-memory error until the cache is unpersisted
  2. Storage blocks are written to HDFS before execution memory is released
  3. Execution and storage memory stay fixed at 50% each and never move
  4. Execution memory may borrow free storage memory, evicting cached blocks Correct answer.

Explanation

The unified memory manager lets execution and storage share one region, so execution can grow into unused storage space and evict cached blocks if necessary. Evicted blocks are recomputed later from lineage if still needed.

Question 37Apache Spark Architecture and Components

What happens to a running Spark application if the driver process crashes in client deploy mode?

  1. Only the current stage is retried on the remaining executors
  2. Executors elect a new driver process and continue the job
  3. The application ends and its executors are released Correct answer.
  4. The application resumes from the last shuffle file written to disk

Explanation

The driver holds the SparkContext, DAG scheduler, and stage bookkeeping, so a driver crash terminates the application and the cluster manager releases its executors. Spark does not elect replacement drivers.

Question 38Apache Spark Architecture and Components

Three of a hundred partitions of an RDD must be recomputed after an executor is lost. What does Spark do?

  1. It fails the job unless a checkpoint was taken beforehand
  2. It reads the missing partitions back from the driver's memory
  3. It reruns the whole stage across all partitions from the original source
  4. It recomputes only the missing partitions from their lineage Correct answer.

Explanation

Lineage records how each partition was derived, so Spark can re-execute just the lost partitions by replaying their parent transformations. Checkpointing shortens long lineages but is not required for recovery.

Question 39Apache Spark Architecture and Components

Why is the external shuffle service valuable when dynamic allocation is enabled?

  1. It compresses shuffle output before it is written to local disk
  2. It moves shuffle files into the executor JVM heap for faster reads
  3. It replaces the cluster manager when allocating executor slots
  4. It keeps shuffle blocks readable after executors are removed Correct answer.

Explanation

The external shuffle service runs on each worker node outside the executor JVMs, so map output stays available even after executors are scaled down. This lets dynamic allocation release idle executors without forcing shuffle re-computation.

Question 40Apache Spark Architecture and Components

A cluster runs one long batch job alongside short interactive queries, each in its own pool. Which setting gives the short queries low latency?

  1. spark.locality.wait raised to several minutes for all tasks
  2. spark.scheduler.mode=FAIR with pools per workload Correct answer.
  3. spark.speculation=true with the default multiplier value
  4. spark.scheduler.mode=FIFO with a single shared pool

Explanation

FAIR scheduling shares cluster resources across pools by weight instead of letting the first-submitted job hold all slots, which prevents head-of-line blocking by the batch job. FIFO would let the batch job monopolize resources until it finishes.

Question 41Using Spark SQL

A developer runs CREATE OR REPLACE TEMP VIEW sales_v AS SELECT * FROM sales. What is the scope of sales_v?

  1. Every session sharing the metastore until it is explicitly dropped.
  2. Every session on the cluster until the cluster is restarted.
  3. Only the notebook that created it, until the notebook is detached.
  4. Only the session that created it; it is dropped when that session ends. Correct answer.

Explanation

Temporary views are registered in the session catalog, so they are visible only to the SparkSession that created them and are removed automatically when that session terminates. They are not persisted in the metastore.

Question 42Using Spark SQL

Global temporary views are registered in which database, which is what allows them to be shared across sessions?

  1. spark_catalog
  2. default
  3. global_temp Correct answer.
  4. tempdb

Explanation

Global temporary views are stored in the reserved global_temp database and must be referenced with that prefix, for example global_temp.v. They remain available to other sessions on the same cluster until the cluster is shut down.

Question 43Using Spark SQL

Which SQL statement lists the tables and views in the current database?

  1. LIST TABLES
  2. DESCRIBE TABLES
  3. SELECT * FROM TABLES
  4. SHOW TABLES Correct answer.

Explanation

SHOW TABLES returns the tables and views of the current database, and SHOW TABLES IN some_db restricts the listing to a named database. DESCRIBE reports column metadata for one object rather than listing objects.

Question 44Using Spark SQL

What does CREATE TABLE t AS SELECT * FROM src accomplish?

  1. It creates an external table t that points at the existing files of src.
  2. It creates a view t that re-runs the query each time it is referenced.
  3. It creates an empty table t with the schema of src and requires a later INSERT.
  4. It creates t, infers the schema from the query, and writes the query's rows. Correct answer.

Explanation

A CREATE TABLE ... AS SELECT statement, often called CTAS, creates a new table, derives its column names and types from the query output, and populates it with the query's result rows in one statement.

Question 45Using Spark SQL

A managed (non-external) table is dropped with DROP TABLE. What happens to the table's data files?

  1. They remain in the warehouse directory as unreferenced files.
  2. They are moved to a Trash directory and retained indefinitely.
  3. They are deleted together with the table metadata. Correct answer.
  4. They are kept but become read-only for other tables.

Explanation

For a managed table Spark owns both the metadata and the data, so DROP TABLE removes the metastore entry and deletes the underlying files. External tables behave differently because Spark owns only the metadata.

Question 46Using Spark SQL

Which command displays the column names and data types of a table named flights?

  1. DESCRIBE TABLE flights Correct answer.
  2. INFO TABLE flights
  3. SHOW COLUMNS flights
  4. SHOW SCHEMA flights

Explanation

DESCRIBE TABLE, which can be shortened to DESC, returns the column names along with their data types and comments. DESCRIBE EXTENDED additionally reports storage and table properties.

Question 47Using Spark SQL

Which statement caches the contents of the table events in memory for the current session?

  1. CACHE TABLE events Correct answer.
  2. SET CACHE events
  3. MATERIALIZE events
  4. PERSIST TABLE events

Explanation

CACHE TABLE reads the table and retains its data in the cluster's memory and disk store so later queries can avoid re-reading the source files. UNCACHE TABLE removes the cached data.

Question 48Using Spark SQL

Which SQL function returns the first non-null value among its arguments?

  1. ifnull
  2. coalesce Correct answer.
  3. nvl2
  4. isnull

Explanation

coalesce accepts two or more arguments and returns the first one that is not null. It is the general-purpose choice when more than two candidate columns must be considered.

Question 49Using Spark SQL

A job runs CREATE DATABASE IF NOT EXISTS analytics on every execution. What happens on a later run when analytics already exists?

  1. The statement succeeds and leaves the existing database unchanged. Correct answer.
  2. The statement drops and recreates the database, deleting its tables.
  3. The statement fails with a database-already-exists error.
  4. The statement succeeds but moves the existing tables to default.

Explanation

The IF NOT EXISTS clause makes the statement idempotent: when the database is already present Spark emits no error and makes no change. Without that clause the same statement would fail.

Question 50Using Spark SQL

A Delta table is partitioned by year. A query runs SELECT * FROM sales WHERE year = 2023. Why does Spark read only the 2023 files?

  1. Because the table's bucket count matches the number of years.
  2. Because the optimizer rewrites the query as a broadcast join.
  3. Because the predicate on the partition column enables partition pruning. Correct answer.
  4. Because Delta caches each partition in memory before scanning.

Explanation

When a filter references a partition column, the optimizer derives the qualifying partition values and skips directories that cannot match. This pruning happens before any file is opened, so unrelated partitions are never read.