FIX Session Drop at the Open
At 08:00:03 London time, the FIX session carrying 22% of a £12bn equity manager's order flow to Broker X's smart order router stopped acknowledging messages. Fourteen child orders, including three in FTSE 100 names with limit prices, show status sent in the OMS but no fills or acknowledgements. Broker X confirms a gateway restart and warns that in-flight messages may have been lost rather than queued. The OMS has no automatic resend and no persistent session-store replay. The auction is in six minutes. Two of the orders are client-directed and one is a basket leg for a transition. Traders want to resend all fourteen manually; the head of trading is worried about duplicate fills and best execution. The broker's drop copy is available but delayed by two minutes. The portfolio manager is asking for fill status.
1. What immediate actions should the head of trading authorise in the next six minutes?
Read the worked answer
Do not resend all fourteen orders. A blanket resend is the highest-risk action because the broker may have received some messages before the gateway restart, and duplicate fills would create unintended positions and client impact. First, instruct the broker by phone to confirm which order IDs reached its gateway and to suspend execution on the affected session. Second, use the drop copy, even with a two-minute delay, to reconcile sent versus acknowledged versus filled. Third, for orders not confirmed as received, cancel and re-route with new unique client order IDs rather than replaying the original IDs.
The reasoning is that FIX 'sent' is not evidence of receipt. Without a persistent session store, the firm cannot safely replay. The head of trading should prioritise the three FTSE 100 limit orders because auction timing is critical, but only after broker confirmation. Any order with a partial fill must be treated as live until cancelled. Document the timeline and decisions in real time for best execution and incident review.
2. How should the firm handle the client-directed orders and the transition basket leg differently from ordinary orders?
Read the worked answer
Client-directed orders carry a specific instruction about venue or broker, so the firm cannot simply re-route them to another broker without client consent or a documented deviation. The immediate step is to establish whether the client instruction permits only Broker X; if so, the firm should cancel any unacknowledged order, confirm with the client, and document the delay as an operational constraint. Re-routing would be a breach of the instruction even if it produced a better price.
The transition basket leg has a different problem: it is part of a coordinated programme, and partial execution can create unintended exposure or tracking error against the transition benchmark. The transition manager should be brought into the incident call, and the leg should be handled as a package with the broker and the receiving portfolio manager. Ordinary orders can be cancelled and re-routed more freely if the investment decision still stands. The reasoning is that the same technical failure creates different obligations depending on the client instruction and the relationship between orders.
3. What post-incident controls and regulatory considerations should follow?
Read the worked answer
The firm should require persistent FIX session-store replay or an equivalent recovery mechanism, automated sent-versus-acknowledged reconciliation, and a broker gateway restart procedure that includes message-loss confirmation. It should also test session failover and document maximum tolerable outage for each broker and order type.
Regulatory considerations include best execution evidence: the firm must be able to show why it cancelled, re-routed or delayed each order and how client interests were protected. Client-directed orders need a clear record of the instruction and any deviation. The incident should go through compliance and operational risk, with a client impact assessment and, if required, notification under the firm's outsourcing or operational incident procedures. The reasoning is that the technical fix alone is insufficient; the control gap was the absence of a reconciliation and replay design, and the regulatory risk is weak audit evidence rather than the outage itself.