68% Agreement in the Quality Audit: Can You Trust the KPI?
Meridian Support runs a monthly quality audit: 12 team supervisors each score eight recorded calls against a 20-item checklist, and the resulting score is published as a team KPI and used for coaching and bonuses. A newly appointed quality lead notices that one team's audit score jumped from 71% to 88% in a single month with no change in the underlying call-handling data. She asks two supervisors to independently score the same 50 recorded calls. They agree on 'greeting' 94% of the time but on 'complaint resolution' only 68%; on that item one supervisor marks 'partially met' where the other marks 'met'. The checklist defines complaint resolution as 'issue handled to customer satisfaction', with no decision rules or worked examples. There are no calibration sessions, and auditors were trained once, three years ago. The quality lead has six weeks and one part-time analyst before the next KPI cycle.
1. Before anyone compares teams or explains the 71% to 88% jump, what must be established, and with which tool?
Read the worked answer
The measurement system must be validated first, because every downstream comparison - team rankings, the jump, coaching priorities, bonuses - assumes the audit score means the same thing regardless of who scores. The appropriate tool is an attribute agreement analysis (attribute gage R&R) on the checklist items, using the 50 calls already collected: ideally two or three appraisers scoring each call twice, blinded to team and to previous scores, to separate within-appraiser consistency from between-appraiser agreement. Report agreement per item plus kappa statistics, then identify which items are ambiguous. Until this is done, the 71% to 88% jump is uninterpretable: it could reflect real improvement, a change in which supervisor scored, or drift in how 'partially met' is applied. Doing this first is also cheap - the recordings already exist and the analysis takes days, not weeks.
2. A manager argues that 68% agreement is acceptable because it is above 50%. How do you respond?
Read the worked answer
Raw agreement is the wrong number because it includes agreement by chance, and chance agreement is high when one category dominates. If 90% of scored items are marked 'met', two auditors flipping coins weighted the same way would agree most of the time without sharing any standard. Kappa corrects for this, and on a skewed item a 68% raw figure can correspond to a kappa near zero - effectively no agreement beyond chance. The usual interpretation: below roughly 0.7 the measurement system is questionable for decisions about individuals; below about 0.4 it is unusable. Here the audit drives bonuses and coaching, so the bar should be high - the 0.9 region for a pass/fail judgement. The practical response is to fix the item, not to argue about the threshold: rewrite 'complaint resolution' with observable decision rules and examples, run calibration sessions, then re-run the study and report kappa, within-appraiser agreement and between-appraiser agreement.
3. Recommend three concrete actions the quality lead can complete in six weeks, with owners and evidence of success.
Read the worked answer
One: rewrite the ambiguous checklist items. The quality lead, with two senior supervisors, converts 'handled to customer satisfaction' into observable behaviours (for example: customer confirmed the issue was resolved, or a follow-up was scheduled with a date). Success evidence: each item has decision rules and at least two worked examples. Two: run calibration. Forty recorded calls scored by all 12 supervisors, discuss disagreements against the written rules, then re-score a fresh set of 30. Owner: quality lead. Success evidence: kappa at or above 0.8 on the reworked items and improved within-appraiser consistency. Three: install measurement governance before the next KPI cycle - blind scoring, rotation of auditors across teams, a monthly calibration sample of 10 calls, and a rule that KPI results are published only when item-level agreement meets the threshold. Owner: quality lead with the operations manager. Freeze or footnote the current month's KPI rather than rewarding a jump that may be measurement artefact.