Databricks Machine Learning Associate flashcards

32 cards. Read the term and the definition here. The deck at the top of this page is optional.

dbutils.secrets.get

Retrieves a secret value by scope and key from a Databricks secret scope, returns it as a string, and redacts the value in notebook output.

Unity Catalog namespace

Three-level naming convention catalog.schema.table used to reference tables, views, volumes, models, and functions, with privileges inherited from parent catalog and schema objects.

All-purpose cluster

A long-running interactive cluster created in the workspace that users attach notebooks to for exploration and shared work, unlike job clusters that exist only for one run.

Job cluster

A cluster created automatically for a single Databricks job run and terminated when that run finishes, isolating workloads and usually lowering cost versus all-purpose clusters.

Databricks Runtime ML

A runtime variant bundling MLflow, scikit-learn, XGBoost, LightGBM, PyTorch, and TensorFlow, with GPU-enabled builds available for deep learning workloads on Databricks.

Secret scope

A named collection of secrets backed by Databricks, Azure Key Vault, or AWS Secrets Manager, whose access is controlled by scope ACLs or backing-store permissions.

Notebook text widget

Created with dbutils.widgets.text to parameterize a notebook; dbutils.widgets.get returns its value as a string, and a default value can be supplied.

Cluster access mode

A setting that determines cluster isolation: Single user dedicates the cluster to one user, while Shared allows multiple users and requires Unity Catalog for data access.

MLflow run

A single execution of training code that records parameters, metrics, artifacts, tags, and a unique run ID within an MLflow experiment for later comparison.

mlflow.log_param

Logs a key-value parameter to the active run; once set, a parameter cannot be overwritten, whereas metrics can be logged repeatedly with different step values.

MLflow autologging

Enabled by mlflow.autolog or a framework call such as mlflow.sklearn.autolog, it captures parameters, metrics, and model artifacts automatically for supported libraries.

Nested MLflow runs

Child runs started with mlflow.start_run(nested=True) inside a parent run, commonly used to group Hyperopt trials or cross-validation folds under one experiment run.

Model Registry stages

Lifecycle labels assigned to model versions: None, Staging, Production, and Archived; a stage transition can archive existing versions currently in the target stage.

SparkTrials

A Hyperopt Trials implementation that runs fmin trials in parallel on Spark workers; with MLflow tracking enabled, each trial is recorded as a nested run.

Hyperopt objective

A function returning a numeric loss that fmin minimizes across max_evals iterations, using search algorithms such as tpe.suggest over a defined parameter space.

Model signature

An MLflow schema of model input and output columns and types, often inferred from an input example, used to validate and document inference data.

VectorAssembler

A Spark ML transformer that combines multiple numeric or vector columns into one features vector column required by Spark ML estimators such as logistic regression.

StringIndexer

A Spark ML estimator mapping string categories to numeric indices ordered by frequency; the fitted indexer model must be applied consistently to training and scoring data.

Feature Store training set

Created with create_training_set and feature_lookups, it joins label data to feature tables and returns a DataFrame used to train models reproducibly.

Point-in-time lookup

A feature store join returning feature values whose timestamps are at or before each label event, preventing future information leakage in time-dependent models.

TrainValidationSplit

A Spark ML estimator that tunes hyperparameters using one random train/validation split instead of k folds, trading evaluation robustness for faster tuning.

Class imbalance handling

Techniques including class weights, resampling, and decision-threshold tuning; evaluate with precision-recall AUC or F1 rather than accuracy when positive labels are rare.

pandas_udf

A PySpark user-defined function that operates on pandas Series or DataFrames using Apache Arrow, enabling vectorized feature engineering and distributed model inference.

CrossValidator

A Spark ML estimator that evaluates a parameter grid across k folds and fits a best model, requiring an estimator, estimatorParamMaps, evaluator, and numFolds.

Model Serving endpoint

A Databricks-hosted REST API serving registered model versions or feature tables, with configurable compute scaling and authentication via personal access tokens or OAuth.

models:/ URI

An MLflow reference of the form models:/model_name/version or models:/model_name/stage that loads or serves a specific registered model version.

spark_udf inference

Wrapping an MLflow pyfunc model with mlflow.pyfunc.spark_udf applies predictions across Spark partitions, distributing batch scoring over the cluster.

Batch inference

Scoring large datasets with a Spark DataFrame and a model loaded once per executor, typically scheduled as a job for periodic offline predictions.

Streaming inference

Applying a model to a Structured Streaming DataFrame, often inside foreachBatch, so micro-batches of new data are scored continuously as they arrive.

Traffic splitting

Model Serving sends specified percentage shares of requests to different served entities behind one endpoint, supporting A/B tests and gradual canary rollouts.

Scale to zero

A serving configuration where endpoint compute scales down to zero replicas when idle, lowering cost but adding cold-start latency to the next request.

Feature Serving endpoint

A Databricks endpoint that returns online feature values for a feature table via REST, optionally with request-time lookups for real-time applications.