MBT in CI: model drift, a departing expert and an audit gap
A medical-device company has run model-based testing for a year on patient-monitor firmware. The model - a state machine plus a data model of 40 settings - lives in the modelling tool's proprietary repository. Test cases are generated on one tester's workstation, exported as spreadsheets, and imported manually into the test management tool. The model is maintained by a senior tester who is now transferring to another project.
Symptoms: generated cases and the automated scripts have drifted, with about 18% of scripts no longer matching generated expected results; the model has not been updated for the last two feature increments; an auditor asked for bidirectional traceability from requirement to model element to test result, and the team could not produce it; and a build now fails intermittently with no one able to say whether the model or the code is out of date. No headcount is being added, and the next notified-body review is in five months.
1. Diagnose the failures. Which are tool problems, which are process problems, and what is the root cause underneath them?
Read the worked answer
Process problems (the majority).
- The model is not under version control with the code, so 'which model produced these tests' has no answer. This is the root cause of both the 18% drift and the intermittent build failure: there is no single source of truth and no baseline linking build, model and generated suite.
- The export/import hop is manual, so expected results are copied rather than regenerated - drift is inevitable and invisible until something fails.
- The model was not updated for two increments, meaning MBT silently degraded into a one-off exercise; nobody noticed because there is no gate that fails when code changes without a model change.
- Traceability was never designed in: requirement IDs were not carried on model elements, so the audit question cannot be answered retroactively without a costly reconstruction.
Tool problems. A proprietary repository blocks diffing, branching and CI integration; the spreadsheet export discards element identity, which is what breaks traceability across the hop.
Root cause. MBT was run as an individual's activity rather than as a pipeline stage - one person, one workstation, no automation, no baseline.
2. Design the integration so the next notified-body review can be passed. Cover baselines, CI steps, traceability and what blocks a build.
Read the worked answer
Version control. Keep the model in a text or open-XMI form in the same repository as the firmware, committed in the same change as the code it tests, or tightly linked by commit ID. Tag a model baseline per release.
Pipeline steps, on every change that touches model or code:
- Validate the model - syntax, reachability, dead transitions, orphan requirements.
- Generate test cases deterministically from the pinned model version.
- Execute on the bench, capturing model version + build ID with every result.
- Publish coverage and results, and archive generated cases as immutable artefacts.
Traceability. Put requirement IDs as attributes on model elements; export requirement → model element → generated case → result using stable element IDs (not row numbers in a spreadsheet). Automate the export so the audit view is a query, not a reconstruction.
Build gates. Block the build when: a requirement has no linked model element, a model element has no linked requirement, a code change references a feature the model does not cover, or coverage drops against the previous baseline. Gating on the link, not on the number, is what prevents the drift recurring.
Do not hand-edit generated cases - edits are lost on regeneration and are the source of the 18%.
3. Given no new headcount, how would you select or replace the tooling and mitigate the loss of the senior tester?
Read the worked answer
Tool selection criteria, in priority order: (1) CI-drivable via CLI/API; (2) model stored in a diffable, exportable format that survives vendor change; (3) deterministic generation with stable element identifiers and traceability export; (4) executable semantics precise enough that expected results come from the model, not from a script; (5) total cost including training and migration. Run a two-week spike against the existing model before committing - migrating a year-old model is the real cost, not the licence. If the current tool can be driven headlessly and can export XMI, replacing it may be the riskier option; the missing piece is the pipeline, not the modeller.
Mitigating the knowledge loss (do this in the first month):
- Pair the departing tester with two others and have them each regenerate the suite unaided - the rebuild exercise is the actual test of knowledge transfer.
- Write a short modelling convention: naming, requirement-ID scheme, guard syntax, when to split a state.
- Make model review mandatory for changes, so at least two people read every model edit.
- Name a model owner role, not a model person.
Five months to a notified-body review is enough for the pipeline and traceability export if scope is limited to the firmware state machine; attempting a simultaneous tool migration is not.