The Speeders and the Withheld Incentive
You run research operations at a mid-sized panel company. A 20-minute B2B technology survey (n=1,200, $25 incentive) was fielded through three affiliate partners. Partner C supplied 216 completes - 18% of the total - but its median completion time is 6.5 minutes against 14 minutes for the other partners, 31% of its respondents fail an embedded attention check, and 40% of them share a device fingerprint with at least one other respondent. The client's procurement lead sees the quality report and emails: "Withhold the $25 from anyone whose data we don't use - we shouldn't pay for junk." Two clauses collide: your supplier contract pays on completes delivered, while your panel terms promise the incentive on completion of the survey, not on the client's acceptance of the data. The client also insists on keeping all 1,200 completes because "we paid for a thousand two hundred."
1. How do you respond to the client's request to withhold incentives from respondents whose data is not used?
Read the worked answer
Separate two things that the client has merged: payment for time and acceptance of data. The incentive compensates a respondent for the effort of completing a 20-minute survey; it is not a payment conditional on giving answers the buyer likes. Withholding on the basis of content (unfavorable answers, inconvenient verbatims) is coercive and would systematically bias who is willing to participate in future waves. Withholding on the basis of confirmed fraud is different: a respondent who was never a genuine participant, or who completed via a bot or duplicate identity, did not earn the incentive under any reasonable reading of the panel terms.
So the defensible position is: pay every genuine completer, including low-quality-but-honest ones; do not pay cases confirmed as fraudulent or duplicate. For Partner C, that means investigating before deciding. Pull the 216 records, check open-ended coherence, straightlining, geolocation/IP overlap and fingerprint clusters, and apply the quality thresholds you set before field. Cases that fail fraud checks are voided and not paid; cases that are merely fast or inattentive are paid and may still be excluded from analysis under your disclosed data-quality rules.
Tell the client plainly that incentive policy is yours, that it is disclosed to respondents, and that changing it mid-project would breach the panel terms and undermine the sample for the next wave. Offer the remedy they actually want: replacement completes at no charge and a written quality report.
2. What do you do with Partner C's 216 completes before analysis?
Read the worked answer
Do not silently keep them and do not silently drop them. Quarantine the 216, then apply pre-specified criteria: attention-check failure, impossible completion time, duplicate fingerprint, duplicate IP with identical answer patterns, and open-end gibberish. Record the count removed at each step and the rule used, because an auditor or the client's procurement team will ask.
Then decide inclusion on principle rather than convenience. If, say, 70 cases fail fraud checks, void them, do not pay them, and replace them so the delivered n still meets the 1,200 contracted. If another 40 are genuine but low-quality, you may exclude them from the primary analysis under a disclosed rule, but you should report the base with and without them, and you should pay them. The difference matters: one group is being denied payment for fraud, the other is being excluded from analysis for quality.
Disclose to the client in the methods note: the vendor-level issue, the number of cases voided, the replacement, and the effect on estimates if it is material. Also put Partner C on a corrective-action plan with a scorecard - incidence, median duration, attention-check pass rate, duplicate rate - and remove them from this client's future waves until they clear it. Quietly absorbing suspect data to protect the schedule is the failure mode this scenario is testing.
3. What controls would prevent this from recurring on the next wave?
Read the worked answer
Build the controls into fielding, not into post-hoc cleanup. Before launch: set speeder thresholds from a soft-launch pretest rather than from a rule of thumb, embed at least two attention checks in different formats, require device fingerprinting and IP capture with deduplication at the point of entry, and put bot-detection and VPN/proxy screening in the entry path. Contractually, add a fraud clause to affiliate agreements allowing chargeback for voided completes, plus a right to audit traffic sources.
During field: monitor each partner's duration distribution, attention-check pass rate, duplicate rate and open-end length daily, with a stop rule. If Partner C's median is half the others', that is visible on day two, not at delivery.
After field: keep a documented data-quality protocol that states exactly which cases are voided, which are excluded from analysis, and how the base is reported. Train the project team that voiding for fraud and excluding for quality are different actions with different payment consequences.
Finally, align the contract stack. If the supplier agreement pays on completes delivered, add a clause that fraudulent completes are not "delivered." If the client agreement promises 1,200 usable completes, say what "usable" means. Most of these disputes are drafting failures that surface as ethics problems.