Deploy and monitor generative AI applications
Serving and observing
Deploy the application's model or chain to a Model Serving endpoint with autoscaling and concurrency limits. Enable inference tables to capture request and response payloads into a Unity Catalog Delta table; this is the foundation for monitoring, debugging, and quality analysis. Payload logging can be large and may contain sensitive data, so sample and govern it. Use Lakehouse Monitoring for drift and statistical profiles, and usage tracking for token counts and cost.
What to monitor
Track latency (p50 and p95), error rate, throughput, throttling and rate-limit events, token usage and cost, retrieval hit rate, and quality signals from user feedback and periodic judge scoring on sampled traffic. Close the loop: collect feedback, label a sample, and feed confirmed failures back into the evaluation set.
Traps
- Treating deployment as finished when the endpoint is green; without payload logs you cannot diagnose quality regressions.
- Monitoring only infrastructure: a healthy endpoint can return confidently wrong answers.
- Forgetting that inference tables must be enabled and that schema changes can break downstream monitors.
- Autoscaling cold starts causing latency spikes; keep minimum capacity for SLOs.
Worked example
For a RAG assistant, enable inference tables and log question, retrieved document IDs, answer, and latency. Sample 5% of traffic for nightly judge scoring on groundedness. Alert when p95 latency exceeds 3 seconds or groundedness drops below baseline. Route thumbs-down feedback to a review app and add confirmed failures to the evaluation set.