Databricks Machine Learning Associate cheat sheet

Databricks Runtime ML adds MLflow, scikit-learn, XGBoost, LightGBM, PyTorch, TensorFlow, and Hyperopt to the standard runtime; GPU-enabled builds exist for deep learning.

Secrets are read with dbutils.secrets.get(scope, key); the returned value is redacted in notebook output.

Unity Catalog addresses objects as catalog.schema.table and governs tables, views, volumes, models, and functions with privileges inherited from parent objects.

All-purpose clusters are long-running and interactive; job clusters are created for a single job run and terminated when that run completes.

Cluster access modes include Single user and Shared; Single user dedicates the cluster to one principal and is required for some ML and GPU workloads.

dbutils.widgets.text defines a string notebook parameter, and dbutils.widgets.get returns the widget value as a string.

%pip installs libraries into the notebook session, %sh runs shell commands, and %run executes another notebook inline in the current session.

dbutils.notebook.run executes a notebook as a separate run and returns its exit value as a string; the timeout parameter defaults to 60 seconds.

Databricks Repos provides Git integration so notebooks and source files can be version-controlled and referenced by jobs.

An MLflow experiment groups runs; each run stores parameters, metrics, artifacts, tags, and a unique run_id.

Parameters are immutable once logged in a run, while metrics can be logged repeatedly with a step value to record training progress.

mlflow.autolog() or a framework-specific call such as mlflow.sklearn.autolog() automatically captures parameters, metrics, and model artifacts.

Nested runs created with mlflow.start_run(nested=True) group Hyperopt trials or cross-validation folds under one parent run.

Hyperopt fmin minimizes the loss returned by the objective function over max_evals iterations; tpe.suggest is a common search algorithm and SparkTrials distributes trials.

Model Registry stages are None, Staging, Production, and Archived; transition_model_version_stage can archive existing versions in the target stage.

Model URI forms: runs:/<run_id>/model references a run artifact, and models:/<model_name>/ references a registry version.

A model signature records input and output schema and is commonly inferred from input_example supplied at log time.

Databricks Jobs schedule notebooks or wheels with retries, notifications, and job clusters for reproducible automated runs.

Spark ML pipelines chain Estimators and Transformers; fit returns a PipelineModel that is applied with transform.

VectorAssembler merges multiple feature columns into a single features vector column required by Spark ML estimators.

StringIndexer and OneHotEncoder handle categorical columns; the fitted indexer model must be reused on scoring data for consistent encoding.

Fit transformers only on training data to avoid leakage; placing them in a Pipeline lets cross-validation refit them per fold.

CrossValidator evaluates a ParamGrid across k folds, while TrainValidationSplit uses one random split for faster tuning.

Feature Store create_training_set performs point-in-time joins so each label row receives feature values available at its timestamp.

pandas_udf uses Apache Arrow to run vectorized pandas operations distributed across Spark workers.

For imbalanced classes, use class weights, resampling, or threshold tuning, and report PR-AUC or F1 instead of accuracy.

TorchDistributor launches distributed PyTorch training across cluster workers and GPUs using torch.distributed.

Databricks Model Serving hosts registered models behind a REST endpoint with token or OAuth authentication and configurable scaling.

A served entity references a model version or feature spec and carries its own compute scale and traffic weight.

mlflow.pyfunc.spark_udf distributes batch inference across a Spark cluster by applying the model per partition.

Model reference forms: runs:/<run_id>/model, models:/<model_name>/, and models:/<model_name>/.

Streaming inference applies the model inside foreachBatch on a Structured Streaming DataFrame for continuous scoring.

Traffic splitting routes percentage shares of requests to different served entities for A/B tests and canary rollouts.

Feature Serving endpoints return online feature values via REST for real-time lookups.

Endpoint permissions are managed separately from registry model permissions; a caller needs query permission on the endpoint.