Two Boxes, One Rack
An air traffic services unit is replacing its flight data processing system. The safety requirement is that no single failure shall cause loss of the service; the operational requirement is automatic failover within 2 seconds with no loss of flight plan data. The unit has one equipment room, four free racks and no second site. The proposed architecture is an active/standby pair of application servers, both mounting a shared Fibre Channel storage array; both connect to a pair of core switches configured as a single logical switch in one rack; both take time from one GPS receiver feeding an NTP server; and both call a single external flight plan correlation service. The standby server is cold-start: on failure it boots and loads the flight plan database, which takes about 90 seconds. The supplier argues the design satisfies "no single point of failure" because every listed element is duplicated - servers, switch pair and network links.
1. Identify the single points of failure in the proposed design and classify them by type. Explain why duplicating a component is not the same as removing a single point of failure.
Read the worked answer
Shared storage array - shared-infrastructure SPOF. Both servers depend on it; a controller, firmware or enclosure failure stops both. Dual controllers do not help if the enclosure, its power or its cabling is common.
Switch pair as one logical switch, one rack - common-cause SPOF. A single logical entity has one control plane: an upgrade, configuration error or software defect can take both halves down together. Same rack means shared power feed, cooling and fire risk.
Single GPS/NTP source - shared-dependency SPOF and common cause. Both servers step together; time skew also breaks log correlation and track time-stamping.
External correlation service - outright service SPOF, not duplicated at all. Failure of that one service is failure of both nodes.
Cold standby, 90 s reload - violates the 2-second requirement by a factor of 45, and violates the no-loss requirement: flight plans created after the last database dump are lost.
One equipment room - a common cause for the whole system (fire, flood, HVAC, power).
The principle. A single point of failure is defined by the failure domain, not by the count of boxes. Two servers in one rack on one power feed with one shared array and one clock have one failure domain; the redundancy is nominal. The correct analysis is a failure-mode list per dependency, with the propagation path shown, not an inventory of duplicated items.
2. Propose a revised architecture that meets the 2-second and no-loss requirements within four racks and one room, and state what cannot be achieved and must be recorded as residual risk.
Read the worked answer
State handling. Replace cold start with a warm standby that receives a state checkpoint at least every 500 ms - synchronous replication of the flight-plan delta, not a periodic full dump - held on local mirrored disks in each server. No shared array on the critical path.
Failover budget, allocated and measured: detection ≤ 500 ms (heartbeat 100–200 ms, three missed beats), decision ≤ 100 ms, promotion and address takeover ≤ 400 ms, state replay ≤ 500 ms, contingency 500 ms. Total ≤ 2.0 s, with every element individually measured at acceptance.
Network. Two physically separate switches, in different racks, independent power feeds, diverse cabling, one management domain but no single logical control plane, and a rule that firmware is upgraded one switch at a time with traffic drained.
Time. GPS plus a disciplined oscillator for holdover on each node, each server preferring a different source, with offset alarming.
Correlation service. Either embedded and deterministic on both nodes with reconciliation, or stateless and restartable within the failover budget - verified, not assumed.
Residual risk that cannot be designed out in one room. Site-level common cause: fire, flood, HVAC failure, total power loss. Mitigations are detection and recovery (suppression, temperature and power alarms, on-site spares, tested 4-hour restore) and must be escalated as a business-continuity requirement. The safety case should state plainly that the 2-second objective covers component and process failure only, not loss of the room.
3. The supplier's design review claims the architecture meets "no single point of failure". What acceptance testing would you require, and what would you do if a single-failure injection produced a 6-second outage?
Read the worked answer
Require a fault-injection matrix, executed at operational load, each injection repeated at least 20 times: power off the active server; kill the application process; pull each network link; fail each switch; break the time source; stop the correlation service; fill the state replication backlog. For each, record outage duration, flight plans lost, and whether the standby assumed the correct role without operator action.
Acceptance criteria must be numeric and per-injection: 100% of injections recover within 2.0 s, zero flight-plan loss, no manual intervention, and correct operation after recovery - plus a configuration parity check between the two nodes, since divergent parameters are one of the most common causes of failed failover.
If a 6-second outage occurs: treat it as a non-conformance, not a tuning note. Diagnose the components of the 6 s - typically ARP or spanning-tree reconvergence, cold caches, database lock release, or health-check timers set longer than the failover budget. Note that the test measured what the design review could not: real time. Require the root cause, a fix, and a full re-run of the whole matrix (not just the failing case) with the timing instrumented end-to-end. Do not accept an average; the requirement is a maximum. Also confirm whether the failure was detected by the system or by the test harness - an unmonitored failover is not an automatic one.