Silent Model Swap: Monitoring an Endpoint You Don't Control

The monitoring is entirely infrastructure-side. Latency and 5xx tell you the endpoint is alive; they say nothing about whether the output is still the output you designed for. Missing:

Output contract metrics (would have caught it in minutes):

• Schema-validity rate - percentage of responses parsing against the expected JSON schema. A drop from 100% to 97% is a page, not a dashboard curiosity.
• Response length distribution - mean, p50/p95. A 40% shift is a step change and should alert on distribution shift, not on a fixed threshold.
• Refusal rate and finish-reason distribution - early signal of a behavior change.
• Token counts and cost per request - a length jump is also a bill jump.

Quality and behavior metrics (catch what schema validation can't):

• Semantic quality scored by an LLM judge on a sampled slice (e.g., 200 requests/day) against a fixed golden set of prompts run on a schedule - this is the only thing that detects "still valid JSON, subtly worse answers."
• Retrieval hit rate and citation presence, to separate a model change from a retrieval regression.

Attribution is the prerequisite: log model_version, prompt_template_version, and retrieval_config_version on every inference-table row. Without them you can see a shift but cannot attribute it, and the incident review becomes speculation. If the provider doesn't expose a pinned version, record a hash of the model identifier and the date, and treat "unknown provider update" as a standing risk.

Never let a third-party model's raw text be the contract between systems.

Layered, in this order:

1. Constrain generation - request structured output / JSON mode and, with the provider, schema-constrained decoding where available. This makes failure rare; it does not make it impossible.
2. Validate at the boundary - parse against a versioned schema (Pydantic-style) immediately on receipt. Validation is a hard gate, not a warning log.
3. Repair once - on failure, one retry with the validation error fed back. Bounded, so a systematically broken model doesn't triple your spend and latency.
4. Fallback - route to a secondary model or a cached/degraded response path.
5. Fail closed - if all fail, park the request in a queue for retry and alert. Never emit a half-valid record.

Architectural rule: the serving endpoint writes raw responses to a bronze table. A separate, deterministic validation step promotes only schema-valid rows to silver, and the DLT job reads silver, never the endpoint directly. That single boundary means a provider regression degrades answer quality - a bad day - instead of corrupting downstream tables and failing a pipeline.

Alerting: schema-validity rate below 99.5% sustained over 10 minutes, or any single 5-minute window below 98%, pages the on-call. Alert on the rate, not on individual failures, or the noise will train people to ignore it.

Immediate: purge or irreversibly redact the identified rows (emails, PAN fragments) from the inference table and any downstream copy, including the bronze table and any notebook-level exports. Document the purge with row counts and a timestamp - an auditor will ask for evidence, not assurances.

Going forward, redact before persistence, not after retrieval:

• A preprocessing step on the way into the logging path detects and masks emails, phone numbers, card numbers (with a Luhn check, not just a regex on 16 digits), national IDs, and addresses. Mask first, log second.
• Where a record must remain joinable for debugging, store a keyed hash rather than the plaintext.
• Minimize: log what you actually use. Full prompt/response payloads are rarely necessary for 100% of traffic - consider full logging on sampled traffic and metadata-only on the rest. Smaller blast radius is a better answer than tighter permissions on a too-large dataset.
• Retention: shorten 30 days to the shortest period that supports incident response and evaluation.
• Access: Unity Catalog grants restricted to a named break-glass group; column masks on any residual sensitive columns; every read audited.

What you tell the auditor: the data classification of the inference table, the minimization and masking controls, retention policy, access model with audit evidence, the lineage of the logging pipeline, and the remediation record with dates. Also disclose that the external provider received the unredacted prompts - that is a separate processor disclosure with its own contractual question (zero-retention terms, data residency), and it is the question most likely to be asked next.