Field Service Tool: Mixed Results After Fourteen Months
A field-service organization implemented a scheduling and dispatch tool 14 months ago for $2.3M. Business targets were 85% on-time arrival, overtime at or below 5% of labor cost, 80% first-time fix, and 90% technician adoption. Current results: 71% on-time, overtime up 9 percentage points, 82% first-time fix, and 54% license utilization. Dispatchers use the tool fully and report it is better than the old whiteboard. Technicians use the mobile app 54% of the time; many still phone dispatchers for job details. Thirty percent of jobs carry an incorrect skill code. Travel time is down 6%. Eight of 22 depots are piloting a new routing configuration. The vendor says the problem is training; the operations director wants to replace the tool; the CFO wants a return-on-investment answer before the next budget cycle.
1. How would you structure the solution evaluation, including what to measure and how to isolate causes?
Read the worked answer
Structure it around three separate questions: did the solution perform as intended, did the enterprise outcome improve, and is the organization ready to use it? Mixing these is how teams end up blaming training for a data problem.
Define each measure with a formula, source, frequency, and pre-implementation baseline: on-time arrival, overtime as a percentage of labor cost, first-time fix, adoption by role, travel time, and dispatch touches per job. Segment every measure by depot, technician tenure, job type, and adoption level. The eight pilot depots give a natural comparison group for routing changes; compare them with matched non-pilot depots and check pre-pilot trends.
Data sources: dispatch and time-stamp records, payroll overtime, skill-code audit, app logs, and call records to dispatchers. Add qualitative evidence - observation of technicians and dispatchers, and short interviews - because the 54% adoption number does not explain why people bypass the app. Pre-register decision criteria so the evaluation is not shaped after the results are known. Also separate vendor-supplied metrics from independently calculated ones, and document known limitations such as seasonality and the concurrent routing pilot.
2. Interpret the mixed results. Is the tool the problem, the organization, or both?
Read the worked answer
The evidence does not support replacing the tool, and it does not support the vendor's 'training only' explanation. It points to a combination of data quality, adoption, and concurrent process change.
First-time fix is 82%, above the 80% target, and travel time is down 6%. Those are solution-attributable gains and argue against replacement. On-time arrival and overtime are worse, but they are enterprise outcomes influenced by skill-code accuracy, adoption, routing configuration, and dispatch behavior. Thirty percent incorrect skill codes could send the wrong technician, which drives repeat visits and lateness even when first-time fix looks acceptable on other jobs.
The next step is segmentation: compare high-adoption versus low-adoption depots, and jobs with correct versus incorrect skill codes. If on-time and overtime track adoption and data quality, the constraint is organizational and data-related. If they are flat across segments and the pilot depots outperform, the routing configuration is the likely driver. The vendor's claim is partially plausible but incomplete; the operations director's replacement call ignores the gains and the pilot confound. I would state what the data can and cannot prove before recommending action.
3. What recommendation would you make to the CFO, and how would you present it?
Read the worked answer
Recommend a 90-day corrective cycle with explicit gates, not replacement and not unconditional acceptance. Replacement would spend new money before the current tool's value is understood, and it would discard measurable gains in first-time fix and travel time.
The plan: clean the top job and skill-code data; coach technicians on the app with dispatcher follow-up; complete the routing pilot readout across all eight depots; and target adoption at 80% and on-time arrival at 80% within 90 days. Set a decision gate at day 90 with pre-agreed criteria for continuing, reconfiguring, or replacing.
Present to the CFO with a one-page story: investment to date, benefits realized, the value at stake in overtime and lateness, the cost of the corrective cycle, and the expected return under each option. Show leading indicators and the evidence gaps. Be explicit that the 54% adoption and 30% skill-code error rate limit what any tool can deliver, and that the pilot depots must be analyzed separately. If the 90-day gate fails, the replacement option returns with evidence. That is a defensible ROI answer, not a promise.