The Case Study That Cannot Survive Due Diligence
A 45-person digital agency has one flagship case study: a fashion retailer achieved +320% organic traffic and 4.8x ROAS in nine months. The prospect, a listed homeware brand, wants a reference call and a detailed attribution review. The agency's own analytics show that 62% of the traffic increase came from a paid brand campaign mis-tagged as organic, and the ROAS used a 30-day click window that counted returning customers. The client did not renew after nine months because margin fell, although the agency's case study implies the relationship ended on a high. The case study is on the agency's homepage and in two award entries. The new business director wants to keep the story; the analytics lead refuses to defend it. The prospect's procurement team has asked for the underlying dashboard and the client contact.
1. What should the agency do with the existing case study and award entries?
Read the worked answer
Correct or withdraw the claims. Do not defend numbers that cannot survive due diligence.
- Separate outputs, outcomes and impact. +320% traffic is an output; 4.8x ROAS is an outcome only if incrementality is shown; margin decline is the business impact the client experienced.
- Recompute honestly. Strip the paid brand campaign out of organic traffic and report paid and organic separately. Recalculate ROAS with a control or holdout, or report it as platform-reported attributed revenue with the window disclosed.
- Remove the implication of success. State that the engagement ended after nine months and explain the margin context.
- Notify the award bodies and ask to amend or withdraw entries. Misleading award submissions create legal and reputational risk.
- Update the homepage with a corrected case that shows the problem, the intervention, the result and the limitation.
Reasoning: procurement teams test claims. A corrected case is less impressive but more credible; an uncorrected one becomes a liability the moment the prospect asks for the dashboard. Trust is the asset being sold.
2. How should the agency respond to the prospect's request for the dashboard and the client contact?
Read the worked answer
Be transparent about method, protect client confidentiality, and give the prospect a verifiable path.
- Share the corrected methodology: metric definitions, attribution windows, campaign tagging, and what the agency would do differently.
- Do not send the raw client dashboard without the client's permission. Offer a redacted extract or a walkthrough under NDA.
- Request a reference call with consent. If the former client declines, say so rather than implying a reference exists.
- Provide an audit trail: who measured what, when, and with which tools. Explain the known gaps.
- Propose a pilot with a pre-registered measurement plan: control group or holdout, agreed baseline, margin and retention metrics, and a stop rule.
Reasoning: the prospect is buying evidence discipline, not a perfect story. A controlled reference and a credible pilot design are stronger than an unaudited dashboard. If the agency cannot supply either, it should decline rather than risk a procurement escalation.
3. Design a case-study standard that prevents recurrence.
Read the worked answer
Make the standard auditable, not editorial.
- Evidence hierarchy: client-verified business metrics first, analytics-platform data second, agency estimates last. Label each claim.
- Metric definitions: required fields for attribution window, incrementality method, currency, margin, and comparison period.
- Client sign-off: written approval before publication, including the right to withdraw.
- Referenceability: record whether the client will take reference calls and under what conditions.
- Disclosure of limitations: what changed in the client's market, what the agency did not control, and what happened after the engagement.
- Review date: every case study expires after 12 months unless revalidated.
- Separation of roles: the person who sells the case cannot be the only approver of its claims.
This standard turns case studies from marketing collateral into evidence. It also gives the new business team a defensible answer when a procurement team asks the hard question.