Regulated credit-model release pipeline
A bank’s credit underwriting team maintains a gradient-boosted model used for 400,000 applications per month. Regulators require every production model to be traceable to a code commit, training-data snapshot, and approval; no direct production changes; fairness review for age and postal-code proxies; and PII must not leave a governed subscription. Today data scientists run notebooks locally, upload model.pkl to a storage account, and a platform engineer manually deploys it to an Azure ML online endpoint. A recent audit found two production models with no training-data version and one trained on a CSV containing full account numbers. The team uses GitHub and wants CI/CD but fears slower releases; they release monthly and hotfix quarterly. The model registry is empty, and there is no drift monitoring.
1. Design the CI/CD pipeline and approval gates from commit to production, including rollback.
Read the worked answer
Design a GitHub Actions pipeline with Azure ML CLI or MLflow, using OIDC federation instead of stored secrets. Stages: lint and unit tests; data validation; train in the governed subscription; evaluate; register the model with lineage; generate a model card and fairness report; manual approval through a GitHub Environment; deploy to staging; smoke and canary; promote to production. Define infrastructure such as workspaces, endpoints, and storage in Bicep or Terraform and review it. Production changes only through the pipeline. The registry stores model version, commit SHA, data asset version, environment image, metrics, and approval. Rollback is a deployment swap to a prior registry version. For hotfixes, allow an expedited path but require the same lineage and approval gates. This addresses the fear of slower releases by automating checks rather than skipping them.
2. What security and fairness controls must be built into the pipeline?
Read the worked answer
Block local CSV uploads; require data assets from approved storage with private endpoints; scan and tokenize account numbers; restrict RBAC and use managed identities. Data scientists should not have direct production deploy rights. Use separate dev, test, and production workspaces with least privilege. For fairness, compute metrics by age band and postal-code proxy slices, compare against thresholds, and require review before approval. Log who approved and what evidence was reviewed. The model card and fairness report should be linked to the model version. These controls satisfy the regulator while making the evidence automatic rather than an after-the-fact audit scramble.
3. Propose monitoring and retraining triggers that preserve governance.
Read the worked answer
Monitor data drift, prediction distribution, and outcome labels with appropriate delay, and alert on drift and performance degradation. A drift trigger should start the same retraining pipeline, not an ad hoc notebook. Include fairness monitoring and model-card updates in that pipeline. Retraining must pass the same validation, approval, and deployment gates. Keep audit logs and lineage for every run and model version. Define thresholds for automatic alerts and manual review rather than automatic production promotion, because the model is regulated and fairness review is required.