Evaluation, Testing, and Observability
Build the eval set first
You cannot improve what you do not measure. Assemble a golden dataset of realistic inputs with expected properties, not one-off anecdotes. Cover the happy path, edge cases, adversarial inputs, and the refusal cases you care about. Version the set and hold out a portion so you are not tuning against your own test data.
Grading strategies
Exact match works only for deterministic outputs. For open-ended text, use rubric-based LLM-as-judge, pairwise comparison against a baseline, or programmatic checks (valid JSON, schema conformance, required citations, forbidden strings). Judges need explicit criteria and a scoring scale or they drift; validate the judge against human labels on a sample before trusting it. Combine automated metrics with periodic human review.
Non-determinism
Temperature 0 reduces variance but does not guarantee identical outputs across runs or model versions. Report distributions over multiple samples rather than single runs, and remember that small differences on small eval sets are noise.
Observability in production
Log prompts, responses, model version, token counts, latency, stop reasons, tool calls, and errors in a traceable structure. Monitor cost and p95 latency alongside quality. Alert on spikes in refusal rate and schema failures - these usually signal a prompt or dependency regression rather than a model change.
Rule of thumb
Any prompt change ships with the eval delta, a sample of new failures read by a human, and a rollback plan. If you cannot state how you would detect degradation, you are not ready to deploy.