Turning a Dashboard Request into Answerable Research Questions
A regional bank's contact-center CSAT fell from 4.3 to 3.6 on a 5-point scale over two quarters. The VP of Operations asks the analytics team for a dashboard that shows why satisfaction dropped. Available data: monthly CSAT by queue, average handle time, first-contact resolution, agent tenure, and 14 months of call transcripts for 1.2 million calls. No process change was logged, no experiment was run, and no measurement change is documented. The VP wants an answer in six weeks and has not said what decision the result would change. Budget covers one analyst and one data engineer.
1. Rewrite the VP's request as one or more research questions the available data can actually answer, explaining each element you add or remove.
Read the worked answer
What is wrong with the request
- A dashboard is a deliverable, not a question. It encodes no outcome of interest and no comparison.
- The word why implies causal explanation; the data are observational with no experiment, so causal claims are out of scope.
- No decision is named, so there is no way to judge whether the analysis is sufficient.
Reconstructed questions
- Descriptive: Which queues, call reasons, and customer segments account for the largest share of the 0.7-point CSAT decline between Q1 and Q2? (Population, outcome, time window, and comparison are all explicit.)
- Diagnostic, non-causal: Does the decline persist within strata of call mix, handle time, first-contact resolution, and agent tenure, or is it explained by a shift in composition?
- Actionable: Among factors the contact center can influence within 90 days, which show the strongest association with low CSAT and are frequent enough to matter?
What got removed
- The demand for a causal answer, replaced with association and composition analysis plus transcript-based hypothesis generation.
- The open-ended scope, replaced with a defined time comparison and a fixed list of candidate factors.
What got added
- A stated comparison period, an outcome measure already in use, and a decision horizon (90-day actionability) that ties analysis to choices the VP can actually make.
2. Name two threats to validity in this analysis and state how you would test or mitigate each.
Read the worked answer
Threat 1: Composition shift (Simpson-style confounding) The average can fall because the mix of call types changed, not because service got worse. If a new billing issue drove volume into a queue with structurally lower CSAT, the overall drop is a mix effect.
- Test: Recompute the decline within call-reason and queue cells, then produce a mix-adjusted (standardized) CSAT using last quarter's call mix as the reference. If adjusted CSAT is flat, the story is mix, not service.
Threat 2: Reverse causality and co-determined metrics Handle time and first-contact resolution move together with satisfaction; a dissatisfied caller may escalate and lengthen the call, so AHT is as much an outcome as a driver. Correlating AHT with CSAT will not tell you which lever to pull.
- Mitigation: Treat these as descriptive stratifiers, not causes. Look for a natural experiment (a policy or routing change, a queue outage) that altered one factor independently, or recommend a small controlled pilot before acting.
Also flag: any undocumented change to the CSAT survey instrument or invitation timing, which would invalidate the trend itself. Verify with the survey owner before analyzing.
3. List the questions you would put to the VP before starting, and state how each answer changes the analysis plan.
Read the worked answer
- What decision will this inform, and when? If the decision is re-staffing for next quarter, the analysis must be at queue-week granularity and finished in weeks, not months. If it is a transcript-driven root-cause study, the plan shifts to sampling and qualitative coding.
- What size of change would be worth acting on? A 0.1-point difference may be noise at queue level; knowing the threshold sets the precision required and therefore the sample and stratification depth.
- Who owns each candidate factor - operations, training, or IT? Factors nobody can change get deprioritized even if statistically strong.
- Has anything changed in how CSAT is measured or sampled? A yes here stops the project: you would first repair the trend, not explain it.
- What would disprove your hypothesis that service quality dropped? Forcing this answer exposes whether the VP has already concluded the cause, which biases how results will be received.
Why this matters: the six-week clock and two-person team mean scope must be fixed up front. Each answer above either narrows the data cut, changes the method, or terminates the request.