AI Observability: Davis, Davis CoPilot and LLM Workloads
AI Observability is examined in two layers. The first is the AI inside the platform: Davis, the causal engine, and Davis CoPilot, the generative assistant. The second is observability of AI workloads themselves - usually LLM-backed applications whose requests, tokens, latency and cost must be traced like any other service. Questions often test whether you know which layer a scenario belongs to.
Platform AI: causal versus generative
- Davis is deterministic and causal. It learns a per-entity baseline by time-of-day and weekday, raises Davis events when behaviour deviates, correlates related events into a single problem, and names a root-cause entity using Smartscape topology.
- Because baselining is automatic, static thresholds are usually unnecessary; you tune anomaly detectors or use maintenance windows when genuine business cycles are unusual.
- Davis CoPilot is generative: natural-language to DQL, problem explanations, dashboard drafting. It answers within the requesting user's permissions, so it never surfaces data that user could not query directly.
Observing LLM applications
- LLM calls are ingested as spans via OpenTelemetry using the GenAI semantic conventions, carrying attributes such as the model name, provider and token usage counters.
- Prompt and completion content is captured only when content capture is enabled and masked by policy - assume it is absent until you verify the setting, because privacy defaults often disable it.
- Useful signals: end-to-end latency, time to first token, tokens per request, cost per request, error and refusal rates, plus evaluation scores you ingest yourself.
Worked example
A chat assistant shows a rising p99 latency while host CPU, memory and network stay flat. Davis reports a problem on the frontend service with no infrastructure root cause. Query the spans grouped by model attribute, e.g. fetch spans | filter ... | summarize avg(duration) by gen_ai.request.model. A minority of traffic routes to a slow model deployment. Reasoning: latency without resource saturation is a dependency problem, so read span attributes before touching infrastructure - the fix is routing or model configuration, not more CPU.
Traps that cost marks
- Confusing causal Davis (correlates and explains) with generative CoPilot (writes queries and explanations).
- Assuming Davis needs hand-built thresholds for every metric.
- Claiming prompt capture is always on; exam scenarios frequently hinge on masking and consent.
- Blaming infrastructure when token counts or model routing explain the latency.