Evaluation of Agent Behavior and Output Quality
Define success before you run
An evaluation is a test suite for behaviour. Start with a written rubric: task success, tests passing, diff minimality, adherence to repository conventions, volume of human review comments, cost, and safety violations. Without a rubric, reviewers grade on vibes and results do not reproduce.
Task suites and regression
Build a held-out set of realistic tasks against a representative repo: a bug fix with a known patch, a small feature, flaky-test triage, a dependency bump. Re-run the suite on every change to model, prompt, tool set, or permission config. Track per-task outcomes, not just the average, so you can see which capability regressed.
Capability versus reliability
Capability is "can it ever succeed" (pass@k). Reliability is "does it succeed on the run that matters" (pass^k - succeeding k times consecutively). Production cares about the second. A 70% pass@1 agent whose failures differ every time is harder to operate than a 60% agent whose failures are consistent and caught by checks.
Judging carefully
LLM-as-judge is useful for style and convention checks, but it needs calibration against human labels, a fixed rubric, and re-validation whenever the judge model changes. Prefer deterministic signals - tests, linters, type checks, build success - wherever they exist.
Traps
- Judging only the final diff while ignoring the trajectory: what files it read, which commands it ran, what it attempted and abandoned.
- Generalising from one anecdote in one repository.
- Comparing runs where prompt, model, and tools all changed at once - a confounded experiment.
Rule of thumb
Report each eval as a row: task, pass/fail, tests, human review comments, cost. Require a regression-free row set before raising an agent's autonomy level.