Data engineering and feature management
Data engineering and feature management tests your ability to make training and scoring data reliable, versioned, discoverable, and consistent. In Azure Machine Learning, datastores hold connection information to ADLS Gen2, Blob Storage, SQL, or other sources; data assets point to versioned data; MLTable describes schema and loading; feature store manages reusable features.
Core rules
- Register datastores with credential-based or identity-based access; never put account keys in notebooks.
- Version data assets and reference immutable URIs in jobs. Use MLTable when schema, delimiters, or column types must be enforced.
- In feature store, define entities, feature sets, sources, and transformations. Materialize to the online store for low-latency inference; use offline data for training.
- Retrieve the same feature definitions in training and scoring to avoid training-serving skew. Validate schema, nulls, ranges, and category drift before training.
Worked example
A churn model needs customer tenure and 30-day spend. The team registers an ADLS Gen2 datastore, creates a versioned MLTable for raw events, and defines a feature set keyed by customer_id. Offline materialization feeds training; online materialization serves the endpoint. The scoring code calls the feature store rather than recomputing spend in Python. This prevents a common bug where training uses 30 calendar days but scoring uses 30 rolling rows.
Traps
- Recomputing features in the scoring script with different logic.
- Using dev storage accounts in production assets.
- Leaking the target into a feature through post-outcome aggregates.
- Failing to backfill or monitor feature freshness.