Batch demand forecasting for a grocery chain

Use an Azure ML batch endpoint backed by an autoscaling compute cluster, not a managed online endpoint. The replenishment system cannot call REST and needs a file at 03:00, so batch scoring that writes partitioned Parquet or CSV to ADLS Gen2 fits. Register ADLS as a datastore; land the on-premises SQL snapshots through the self-hosted integration runtime into the same lake so training and scoring read one governed source. Keep the global LightGBM model but score by store/SKU partitions to control memory. Use a cluster with autoscale; use low-priority nodes for scoring only if the deadline has slack, otherwise reserve dedicated nodes for the 03:00 window. Orchestrate with an Azure ML pipeline and schedule, not a notebook. Reject AKS or an online endpoint because it adds operational cost and Kubernetes skills the team lacks, and the consumer is batch. Power BI should read the scored output, not call the model endpoint. Add pipeline-duration alerts and a fallback to yesterday's forecast.

Reproducibility needs immutable inputs and recorded context. Put pipeline code in Git and build versioned Azure ML components with a pinned environment by image digest. Register training data as a data asset with a version and a fixed time window, not a mutable file path. Each scheduled run should record the Git commit, data asset version, feature-engineering parameters, hyperparameters, environment, model artifact, and metrics in the model registry or MLflow. Register the forecast output with the run ID and model version. For audit, retain either the data snapshot or a queryable immutable reference, plus the scoring script and container. Use separate dev, test, and production workspaces and require approval before promotion. Store each forecast with the model version and run ID so a forecast produced six months earlier can be traced back. Reject relying on notebook outputs or file-share timestamps because they are mutable and do not provide lineage.

Month-end failure is most likely memory pressure and a longer critical path, not model quality. A single notebook that pulls all data into memory grows with month-end adjustments and extra SKUs; one process also serializes feature builds and scoring. Fix it by moving to a pipeline: filter to active SKUs, read partitioned Parquet with predicate pushdown, precompute features incrementally, and score in parallel mini-batches using the batch endpoint. Raise the autoscale maximum for the window or isolate the forecast job on dedicated compute. Use column pruning and appropriate data types, and avoid collecting to pandas. Add stage-level duration metrics and alerts at 02:30 so the team sees the pipeline falling behind. If the model itself is slow, reduce trees or use a smaller model for long-tail SKUs. Do not add Kubernetes; the issue is pipeline and data-volume handling.