QA Scores Diverge Between Two Analysts
Northline Home Services has 45 agents selling appliance protection plans. Two QA analysts each review eight recorded calls per agent per month against a 20-item scorecard. Last quarter Team A averaged 88.4 and Team B 74.1, yet conversion was 9.2% and 8.9% respectively. Analyst A scores disclosures and verification strictly but almost always gives full credit for rapport and empathy; Analyst B is the reverse. An agent on Team B received a written warning after a 61 score on a call that Analyst A reviewed informally and scored 85. The scorecard has no written anchors for items 7, 12 and 15, and 30% of total weight sits on tone. Agents have started asking which analyst will review their calls before they take certain rebuttal risks. HR wants QA scores included in the annual review next month.
1. Is Team A actually outperforming Team B? Explain how you would determine this before any personnel decisions are made.
Read the worked answer
Not yet established. The 14.3-point score gap is not corroborated by outcomes: conversion differs by only 0.3 points and cancellation and complaint data are not shown. A gap that large with flat outcomes is more likely measurement than performance, though both could be true.
Test in this order. First, double-score 30 randomly selected calls blind, with both analysts scoring the same recordings independently and without seeing agent names. Compute item-level agreement, percent of scores within one point, and exact agreement on the three unanchored items and on tone. If agreement on those items is below roughly 80%, the score cannot support individual conclusions.
Second, check sampling reliability. Eight calls per month against 20 items is a small, noisy sample; a single disputed call can move an agent's monthly average by several points. Estimate the standard error before treating a 14-point team difference as real.
Third, compare independent outcomes: conversion, cancellation within 30 days, complaint rate, and repeat-call rate. These are not perfect either, but they do not share the analyst bias.
Fourth, look for assignment effects: if analysts are assigned by team, team differences and analyst differences are confounded by design. Until calls are cross-scored, the comparison is uninterpretable. Hold the warning on the Team B agent pending this review.
2. Redesign the scorecard and calibration process so scores can be used for coaching and, later, formal review.
Read the worked answer
Rebuild the scorecard around what the business needs the call to accomplish and what is legally required, then make each item observable.
Proposed weighting: 55% mandatory compliance and verification items scored pass/fail with no partial credit; 30% customer-experience behaviors such as confirming understanding, resolving the stated need, and setting clear next steps; 15% efficiency and accuracy behaviors such as correct plan details and CRM disposition. Remove or redefine tone. Tone is only usable if written as observable behavior, for example: acknowledged the customer's concern before presenting a rebuttal, or avoided interrupting during the first 30 seconds.
Write anchors for every item with two or three concrete examples of a full-credit, partial-credit and zero-credit call. Anchors are what convert an opinion into a score.
Calibration cadence: a weekly 45-minute session where both analysts score five calls independently in advance, then compare item by item. Track agreement metrics over time and set a target of at least 80% within one point. Recertify analysts quarterly on a benchmark set of calls with agreed scores. Rotate which analyst covers which team each quarter to break the confound.
Governance: no score goes into a formal review until calibration agreement is met and the agent has seen the recording and the written rationale. Use scores primarily for coaching trends, not single-call judgments.
3. How should the disputed warning be handled, and what would you tell HR about using QA scores in annual review next month?
Read the worked answer
Handle the warning first because the risk is immediate. Pause the disciplinary action, pull the recording and the written score rationale, and have both analysts plus a third reviewer score it independently under the existing rubric. If the scores diverge materially, the warning should be withdrawn and replaced with coaching. Document the review and tell the agent what changed and why.
Tell HR that the current scores are not yet defensible for annual review for three reasons: the rubric has unanchored items carrying 30% of the weight, inter-rater agreement has not been measured, and analysts are confounded with teams. Using them now creates both fairness and legal exposure, especially where a warning could later support termination.
Offer a sequenced alternative. Next month, use QA data only for development plans and coaching, with scores shown to agents alongside the recording. Run the double-scoring exercise and calibration sessions for one quarter. Once agreement consistently clears the threshold and the revised scorecard is live, introduce QA into formal review with a defined weight, an appeal path, and a rule that no single call can drive an adverse action.
Also fix the incentive effect: agents are already gaming analyst assignment. Blind assignment, cross-team rotation and a rule that analysts do not score agents they directly coach will reduce that behavior.