Interpreting Mixed UX Signals After a Mobile Banking Redesign
A retail bank releases a redesigned mobile app. Analytics show task success for "transfer money to a new payee" rose from 78% to 92%, and mean time on task fell by 20%. However, System Usability Scale score fell from 74 to 68, and support tickets containing "can't find transfer" rose 15% in the first month. Session recordings show 40% of users abandon at the confirmation screen and then restart. Several users say "it logged me out for no reason" after using the back button. Accessibility review reports that focus order jumps from the amount field to the submit button, skipping the payee field. The product owner argues the higher success rate proves the redesign is better and wants to close the UX work. A customer service manager says older customers call for help. The next release is in four weeks.
1. Explain how task success and time-on-task can improve while overall satisfaction and support contacts worsen. Identify the most likely UX defects from the evidence.
Read the worked answer
The headline metrics are likely measuring a narrower behaviour than the user's actual goal. Task success may count only transfers that reach the final confirmation, excluding users who abandon and restart. Time on task may fall because the interface funnels users quickly toward a confirmation screen, not because they understand the transfer.
Likely defects:
- Ambiguous confirmation: 40% abandon then restart, which suggests users do not trust or understand what will happen next.
- Loss of state on back navigation: users report being logged out "for no reason"; this is a severe error-recovery and session-management problem.
- Navigation and information architecture: support tickets containing "can't find transfer" indicate the entry point does not match users' mental model.
- Accessibility and focus order: focus skipping the payee field prevents keyboard and switch users from completing the task correctly.
- Demographic impact: older customers calling for help is consistent with reduced confidence and discoverability, not merely preference.
The product owner is treating one metric as proof. Usability is multi-dimensional: effectiveness, efficiency, and satisfaction can move in different directions.
2. Recommend what to fix before the next release and what to measure to confirm improvement. Prioritise the fixes.
Read the worked answer
Prioritise by user harm and task criticality:
- Fix session loss and back-button behaviour first. Losing state in a financial task damages trust and can cause duplicate transfers.
- Redesign the confirmation screen to state payee, amount, source account, and effective date in plain language, with an unambiguous back or cancel path.
- Fix focus order and accessible names in the transfer form; keyboard and assistive-technology users must reach the payee field before submit.
- Improve transfer discoverability on the home screen and navigation labels, using terms from user language rather than internal product language.
- Add error prevention and recovery for duplicate payees, insufficient funds, and timeouts.
Measure beyond the current success metric: completion with abandon/restart counted as failure; critical error rate; time to complete with confidence rating; SUS or another validated satisfaction measure; support-ticket themes; keyboard-only completion; and accessibility checks. Compare pre- and post-fix data with the same definitions. Do not close the UX work until the abandonment and session-loss signals are resolved.
3. How should the product owner's claim be handled in a release review?
Read the worked answer
The claim should be challenged with triangulated evidence, not dismissed. The review should show that the success metric improved while satisfaction, support contacts, abandonment, and accessibility evidence worsened. The team should agree a balanced UX acceptance criterion before the next release, for example: no loss of session state on back navigation; confirmation comprehension verified with representative users; keyboard-only completion; and no increase in transfer-related support contacts. The product owner can still decide to release, but the residual UX risk should be explicitly accepted and recorded.