100% state and transition coverage, three field escapes
Your team tests the firmware of a connected thermostat. Its behaviour is specified as a state machine with 9 states and 24 transitions, including guards on measured temperature and occupancy, three timers (minimum compressor off-time, boost duration, away-delay), and a frost-protection interrupt that can fire in any state. The MBT tool generated 118 test cases and the team reports 100% state coverage and 100% transition coverage; the suite runs clean on the bench.
Six weeks after release, three field defects appear: compressor short-cycling when a user switches between heat and eco inside the off-time window; frost protection not restoring the previous mode after the interrupt clears; and an away-mode timer that never expires when occupancy flickers. The project manager concludes that 'the model was wrong' and wants to abandon MBT for the next release.
1. Explain, using each of the three defects, why 100% state and 100% transition coverage did not prevent them, and name stronger criteria that would have.
Read the worked answer
State coverage is the weakest criterion - it asks only that each state be visited once. Transition coverage adds that each transition executes once, but typically within a single traversal path, so it says nothing about sequences or data.
- Short-cycling (heat → eco inside off-time). This is a defect of a transition pair interacting with a timer boundary. Transition coverage visits heat→eco and eco→heat once each, possibly far apart in the suite; it never requires the pair in order, nor the boundary value 'elapsed = off-time − 1s'. Stronger criteria: 1-switch (transition-pair) coverage, plus guard boundary analysis on the off-time counter (0, exactly equal, +1).
- Frost protection not restoring previous mode. The requirement needs a history variable (the pre-interrupt mode) that the model did not carry. This is a model omission, not a coverage weakness; it needs a model review against requirements to find, and a state-history construct to fix.
- Away timer never expiring under flickering occupancy. This is timer expiry racing an event, plus an occupancy debounce. Transition coverage exercises 'away entered' and 'away exited' but not repeated entry/exit during the delay. Stronger criteria: loop coverage (0, 1, 2, many iterations), concurrent/interrupt preemption modelling, and event-at-timer-expiry combinations.
2. Propose concrete additions to the model and to the coverage target for the next cycle, and state what each addition would cost.
Read the worked answer
Model additions.
- Add a saved-mode variable and explicit restore transitions for the frost interrupt - makes the second defect reachable at all.
- Model the three timers as explicit counter variables with guards and timeout transitions, instead of prose notes.
- Model the interrupt either as a concurrent region or as preemption transitions from every state, so generation can interleave it.
- Add occupancy debounce as a separate state or a counter, so flicker is representable.
Coverage target. Raise from transition coverage to 1-switch (transition-pair) coverage on the 24 transitions - feasible, roughly 60–90 pairs after removing infeasible ones - plus loop coverage 0/1/2/maximum on each loop, plus boundary values for every numeric guard, plus one test per interrupt-preemption pair.
Cost. Modelling effort of a few tester-days and independent review time; suite growth from 118 to perhaps 350–500 cases, which matters if execution is manual - automation or nightly bench runs should be part of the same decision. Infeasible pairs must be documented rather than silently excluded, or the coverage number becomes unfalsifiable.
3. How would you respond to the manager's conclusion that the model was wrong and MBT should be dropped?
Read the worked answer
Partly agree with the diagnosis, disagree with the remedy.
Concede the point. The model was incomplete - it omitted state history and interrupt preemption - and the chosen coverage criteria were too weak to expose that. Saying otherwise destroys credibility.
Separate two claims. (1) The model had defects: true, and the fix is model review against requirements plus stronger criteria. (2) Therefore MBT does not work: not supported. The field escapes were found by customers because no systematic criterion covered sequences, timers and interrupts. The manual suite that MBT replaced had the same blind spots plus a nine-day feedback loop.
Propose a bounded experiment rather than an argument: for the next release, apply 1-switch coverage and guard boundary analysis, add the history and interrupt modelling, and measure whether the new criteria would have generated the three missing scenarios. That is a concrete falsifiable test - if the strengthened model and criteria reproduce all three escapes, the case for continuing is made with evidence; if not, the manager's position gains real support. Also add model review as a gate: modelling defects are cheapest to catch before generation.