Retail app misses performance targets ahead of a 3G-market campaign
A retail app's product-listing page takes 8.2 seconds to become interactive on a simulated 3G profile and 2.6 seconds on 4G, against a 1.5-second target for 4G. Instrumentation shows heap growth of about 40 MB per browse session that is never released, and a battery test on a mid-range Android device shows 12% drain in a 20-minute browsing session. On two low-end devices with 2 GB RAM the app produces ANRs when the user scrolls a listing while images load. The backend team says server response times are 180 ms at p95 and points at the client; the client team says the images are too large. The release is in two weeks, and marketing plans a campaign into markets where 3G is still common. The team has Android Studio Profiler, Xcode Instruments, a network link conditioner and a device farm that reports frame timings.
1. Define the performance test approach for this release. Which journeys, which metrics, which tools and which device tiers?
Read the worked answer
Anchor the approach on journeys users repeat, not on synthetic single-screen loads.
Journeys, in priority order: cold start to home; listing scroll with images; search; product detail; add to cart; checkout; and a 20-minute browse loop to expose leaks.
Metrics per journey:
- TTID and TTI, cold and warm.
- Frame timing: janky frame %, 90th/95th percentile frame time, missed vsync.
- Memory: Java/native heap, allocations per scroll, growth across 10–20 cycles, GC pauses.
- Network: request count, bytes per session, payload size, caching behaviour.
- Battery: % per 20 minutes and mAh per journey via Battery Historian or Instruments Energy Log.
- Stability: ANR and crash rate under load.
Device tiers: low-end 2 GB RAM (where the ANRs are), mid-range and high-end. Results without the low tier are meaningless here. Network profiles: 3G with latency and loss, 4G, and a flaky/offline case.
Tools: Android Studio Profiler and LeakCanary; Perfetto or systrace for frames and scheduling; Xcode Instruments (Allocations, Time Profiler, Energy); Network Link Conditioner or a throttling proxy; device-farm frame statistics to confirm on hardware you do not own.
Exit criteria must be numeric and tied to target markets: for example, TTI p90 ≤1.5 s on 4G mid-tier and ≤4 s on 3G mid-tier; no ANR in the scripted 30-minute browse on the 2 GB device; heap growth <5 MB over 20 listing cycles. Agree these with the product owner before testing, otherwise the result is an opinion.
2. Server p95 is 180 ms but time to interactive on 4G is 2.6 seconds. Explain how you would determine where the time goes, and what data you would collect.
Read the worked answer
Build a stage-by-stage time breakdown, not a single end-to-end number.
Instrument the journey as stages: DNS, TCP/TLS, request queueing, TTFB, download, parse/deserialise, image decode, layout, first frame, interactive. Compare client-side timestamps with proxy timestamps to see which stage owns the time.
Data to collect:
- Proxy network waterfall: number of requests, sizes, cache headers, whether images are full-resolution originals, whether HTTP caching and a CDN are used.
- Device trace: Perfetto/systrace or Instruments Time Profiler during the load. Is the main thread blocked in decode, layout or GC?
- Memory: heap growth during the run. A 40 MB-per-session leak triggers GC churn that shows up as jank and slow TTI, not as a network problem.
- CPU: is the low-end device pegged at 100% during image decode?
Experiments that separate hypotheses: load with images blocked (if TTI drops to about 1 second, it is image handling); load with a warm cache; load against a local mock returning the same payload in 180 ms.
Discriminating evidence: if the server genuinely takes 180 ms but the device waits 1.5 seconds after the response before first frame, the client owns parse/decode/render. If the proxy shows twelve sequential requests each with 180 ms TTFB, the chatty API architecture owns it even though each call is fast.
Name the owning team with the trace attached, and set a target for the specific stage (for example image decode ≤300 ms on the low tier) so the fix is verifiable.
3. Write three measurable release criteria and explain how the marketing campaign into 3G-heavy markets changes them.
Read the worked answer
Criteria should be measurable, tied to a device and network profile, and agreed before the run.
- TTI of the product listing ≤4.0 s at p90 on a mid-tier Android device over 3G with 1% packet loss, and ≤1.5 s at p90 over 4G.
- Stability and leaks: no ANR during a 30-minute scripted browse-and-scroll on a 2 GB RAM device, with heap growth ≤5 MB over 20 listing cycles.
- Battery: ≤5% per 20-minute browse session on the mid-tier device at fixed screen brightness.
- Crash-free sessions ≥99.5% in the pre-release soak.
The campaign changes the weighting, not just the numbers. Marketing into 3G-heavy markets makes the 3G profile and the low-end tier the primary acceptance profile rather than a secondary one - the 4G figure can pass while the actual customer experience fails. It also changes the device mix: low-RAM and mid-range OEMs already over-represented in crash reports become release-blocking rather than 'smoke only'. Campaigns also create bursts, so add a backend load dimension and test the first-launch path (cold start, empty cache), since many new users arrive that way.
Make the criteria negotiable only in writing. If 3G TTI cannot reach 4 s, the product owner accepts a documented figure with mitigations (image placeholders, pagination, a lite mode); otherwise the release criterion is fiction.