RailSense: Is the Verification Evidence Enough for a Safety Case?
RailSense is a processor for a highway-rail crossing. The safety requirement states: 'Upon detection of a vehicle stalled on the crossing, the system shall command barriers down and activate warning lights within 3 seconds.' Verification evidence at the critical design review consists of: a failure modes and effects analysis claiming detection is safe; unit tests of the radar driver with 94% statement coverage; a system test of 22 nominal scenarios; and a traceability matrix in which 12 of 71 requirements, including two derived safety requirements on degraded-mode timing, have no linked test. Integration testing found two failures: when the radar loses track for 1.5 s and reacquires, the warning command is delayed by 4.2 s; and when the barrier controller reports 'not ready,' the processor retries silently. Overall statement coverage is 87%, branch coverage 61%. The customer asks whether V&V is complete enough for a safety case submission in six weeks.
1. Assess the adequacy of this V&V evidence. Identify the specific gaps and explain why coverage percentages do not close them.
Read the worked answer
The evidence is not adequate for a safety case. The central gap is method mismatch: a 3-second timing requirement in degraded modes is being supported by analysis and nominal tests, but timing and recovery behavior are exactly what analysis and nominal testing cannot establish. The FMEA is useful for identifying failure modes; it does not demonstrate that the implemented system meets the timing bound when tracking is lost. The two integration failures directly contradict the requirement: 4.2 s exceeds 3 s, and silent retry on 'not ready' is an unverified degraded path.
Traceability is incomplete. Twelve untraced requirements, including two derived safety requirements, mean those requirements have no evidence at all; a safety case cannot assert compliance for them. The system test covers only 22 nominal scenarios, so no negative, boundary, or degraded cases are represented.
Coverage numbers are misleading. 94% statement coverage on the radar driver says nothing about integration timing or whether the critical branches were exercised. Overall 61% branch coverage makes it likely that error and recovery paths, precisely the ones involved in both failures, were never run. Coverage measures code executed, not requirements verified. Finally, developer-run unit tests do not provide the independence a safety case normally requires.
2. Recommend a six-week closure plan. What must be tested, in what order, and what should not be claimed?
Read the worked answer
First, build a requirement-based verification matrix that assigns each of the 71 requirements a method (test, analysis, inspection, demonstration), a level, and an independence attribute. Prioritize the 12 untraced requirements and the two degraded-mode safety requirements, because they carry direct safety risk and currently have no evidence.
Next, write fault-injection tests for the two known failures: radar track loss and reacquisition at durations around 1.5 s, including boundaries; barrier 'not ready' with retry and timeout behavior; and sensor dropout. Measure timing at the physical interfaces, not software logs. Add negative and boundary scenarios to the system test set, and run them independently of the development team.
Use analysis only where a validated model exists, and state the model's assumptions and limits. Do not claim completeness from 87% statement or 61% branch coverage; report requirement coverage, which is currently far from complete.
In six weeks, the realistic deliverable is a safety case with an open-items list and a closure plan, not a claim of full verification. Present the two failures as known and corrected with regression evidence, disclose the untraced requirements, and give the customer a dated plan for the remainder. Overstating completeness would be worse than a documented gap.
3. The program manager proposes to raise statement coverage to 95% and submit. Explain why this is the wrong response and what decision the systems engineer should recommend.
Read the worked answer
Raising statement coverage treats a requirements and integration problem as a code-coverage problem. The two failures occurred in integration timing and degraded-mode logic, not in unexecuted lines. Additional unit tests can raise coverage while leaving the 4.2 s delay and the silent retry untouched, because those behaviors emerge from component interaction and timing. The program would spend weeks and still fail the safety requirement.
The systems engineer should recommend rejecting the coverage target as the acceptance criterion and substituting requirement coverage plus targeted degraded-mode testing. Specifically: all safety-derived requirements must have test evidence; all 12 untraced requirements must be traced or explicitly justified; the known failures must be corrected and regression-tested; and timing must be demonstrated at the system boundary. Coverage can remain a supporting metric but not a gate.
If schedule pressure persists, the recommendation should be a staged submission: submit the verified subset, declare the degraded-mode items open, and commit to a dated closure. This preserves integrity and gives the customer an accurate risk picture. Submitting on coverage alone would create a false safety claim and shift the defect to operations, where detection and correction are far more costly.