Monitoring Model Drift with Delayed Ground Truth
AgriSense provides a crop-recommendation AI to 6,000 farms. The model recommends irrigation and fertilizer schedules from soil-sensor data, weather feeds and farm history. The AIMS monitoring dashboard reports 'model accuracy 92%' using a static holdout set from 2023. Field agronomists report that recommendations have become less reliable in the eastern region since a new sensor supplier was onboarded and rainfall patterns shifted. There are no ground-truth labels until harvest, and no drift or data-quality metrics. Complaints are logged in a sales system, not linked to the AI inventory. The internal audit found that the monitoring procedure does not define thresholds, owners or actions, and that management review has only seen the accuracy metric. The certification audit is in four months. The CTO says the dashboard is 'green', so no action is needed.
1. Evaluate AgriSense's monitoring and measurement approach against ISO/IEC 42001. What is missing and why does the 92% figure mislead?
Read the worked answer
Clause 9.1 requires monitoring and measurement to be planned: what is monitored, the methods, when it is done, and who analyses and evaluates the results. AgriSense has a metric, not a monitoring system. The 92% figure is misleading because it comes from a static 2023 holdout, not from current production data. It does not show performance after the sensor supplier change or rainfall shift, and it is not linked to region, crop or farm type. There are no data-quality metrics, no drift detection, no ground-truth process after harvest and no thresholds or owners. Complaints sit in a sales system, so a key signal is disconnected from the AIMS. The internal audit finding is therefore correct: the procedure lacks thresholds, actions and review. A 'green' dashboard can be false assurance if the underlying distribution has changed. Monitoring should also cover supplier performance, human override rates, incident trends and the effectiveness of controls. The CTO's conclusion should be rejected until the metric is validated against current outcomes.
2. Propose a monitoring and measurement plan with specific metrics, data sources, thresholds and response actions, including how to handle delayed ground truth.
Read the worked answer
Define questions and metrics by layer. Input and data: sensor missingness, calibration, latency, supplier uptime and distribution shift against reference windows. Model: drift scores, confidence calibration, regional error proxies and shadow evaluation against agronomist decisions. Outcome: override rate, complaint rate, escalation rate, harvest-time accuracy by region and customer impact. Process: time to respond to complaints, incident closure and supplier corrective actions. Set thresholds such as green, amber and red, with owners, review frequency and actions. Because labels are delayed, use leading indicators and a harvest retrospective to recalibrate. Link complaints to the AI inventory through a common system or monthly reconciliation. Define who can suspend recommendations and how customers are informed. Report to management review with trends, not a single number. Validate metrics periodically. The plan should distinguish monitoring of the AI system from monitoring of the AIMS, though they connect.
3. How should internal audit and management review use this information before certification?
Read the worked answer
Internal audit should audit the monitoring process and the use of results, not just the dashboard. It should sample whether thresholds were breached, whether actions were taken and whether management review addressed the findings. The audit finding should be graded, and corrective action should address root cause: no defined monitoring design, no linkage of complaints and no data-quality ownership. Management review must receive audit results, performance against objectives, incidents, supplier issues and risk status. Before certification, AgriSense needs evidence that the revised monitoring has operated for a period, that a breach was detected and handled, and that management acted. The certification auditor will look for a functioning feedback loop, not a perfect metric. If the CTO continues to rely on the static accuracy figure, the organization risks a nonconformity in performance evaluation and a real failure in the field.