Zero Failures in 500 Hours
An air traffic services unit is accepting a new voice communication system carrying controller–pilot communications across 120 positions. The supplier's verification report states that the system has an MTBF of at least 20,000 hours, evidenced by a 500-hour continuous soak test with zero recorded failures, and that all 640 regression test cases pass, giving "100% test coverage". Failover of the radio gateway to the standby unit was not exercised: the report says the behaviour is "demonstrated by design review". The acceptance board is under pressure to sign, because vendor support for the old system ends in six weeks. Eight weeks after cut-over, a power supply failure in the primary radio gateway caused a 40-minute loss of voice on two sectors; the standby gateway did not take over because one configuration parameter differed between the two units. The regulator has asked why acceptance evidence did not detect this.
1. Assess the statistical basis of the 20,000-hour MTBF claim. Show the calculation and state what evidence would actually support the claim.
Read the worked answer
Zero-failure demonstration. With T hours and no failures, the one-sided lower confidence limit at confidence C is θ_L = T / −ln(1 − C). For C = 0.90, −ln(0.1) = 2.3026, so θ_L = 500 / 2.3026 ≈ 217 hours. The evidence is consistent with a true MTBF of about 217 hours - two orders of magnitude below the claim.
What the claim requires. To demonstrate 20,000 hours at 90% confidence with zero failures: T = 2.3026 × 20,000 ≈ 46,050 hours, roughly 5.3 years of continuous running. That is not achievable in a factory test, so the claim must be supported differently: accumulated field data from comparable installations, reliability growth across a test programme with recorded failures and the chi-square limit, or a restated claim at a realistic confidence.
Second problem: representativeness. A 500-hour soak exercises a narrow operational profile. Call-setup rate, load, position count and traffic mix drive failure exposure far more than idle running. The result says little about the failure modes that matter.
Acceptance position. Either lower the claim to what the data support (about 217–550 hours at 60–90% confidence) or replace it with a demonstration plan tied to the operational profile. Quoting an MTBF the evidence cannot support is not a small error: it propagates into spares and availability planning.
2. All 640 regression cases pass. Explain why "100% test coverage" is a misleading statement and identify the specific gap that allowed the 40-minute outage.
Read the worked answer
Pass rate is not coverage. "640 cases, all passing" is a count of executions. Coverage is a relationship between the test set and something else: requirements (is every requirement traced to a test?), code (statement, branch, path), or operational profile (are the real traffic conditions exercised?). None of these is established by a pass count. Calling it 100% coverage in a verification report is a category error and should be corrected in the document.
The specific gap. The standby gateway was never subjected to a real primary failure. "Demonstrated by design review" cannot establish that a configuration parameter is identical on both units - parity is a property of the installed configuration, verified by inspection and by test, not by design intent. Redundant pairs fail on divergence far more often than on hardware: a parameter set on one unit during commissioning and not the other is invisible until the moment of transfer.
What was missing from acceptance: fault-injection testing with real traffic; a configuration parity check (automated diff of both units' parameter sets as a release gate); negative and degraded-mode cases; and explicit traceability from each operational requirement ("loss of the primary gateway shall not interrupt communications") to a test that would fail if the requirement were violated. Verification (built right) was substituted for validation (works in operation). The test set was large but not risk-based.
3. Six weeks before the old system's support ends, what do you recommend to the acceptance board?
Read the worked answer
Recommend conditional acceptance with a closure plan, not unconditional sign-off. Deadlines are a reason to manage risk explicitly, not a reason to delete evidence.
Required before final acceptance: a fault-injection test of gateway failover at operational load with a numeric criterion (transfer within the stated time, zero lost calls beyond the specified allowance); an automated configuration parity check that becomes a release gate for every future change; a register of all claims supported only by design review, each with an owner and a closure date; and re-issue of the verification report without the "100% test coverage" wording.
Interim operational mitigations while closure work runs: both gateways monitored continuously with daily manual configuration comparison and alarming on divergence; documented fallback voice procedures and controller briefing; retention of the legacy system or a service contract extension rather than decommissioning on schedule; a tested rollback position.
Governance. Notify the regulator of the deviation and the temporary risk position. The decision to operate with a known, unclosed verification gap belongs to the accountable manager, recorded with the residual risk, the compensating measures and an expiry date. The technical assessment should be unambiguous: the outage was predictable from the design and was not detected because the acceptance test set was chosen for volume, not for risk.