Trading book disaster recovery needs tested failover

A hot standby does not make a live trading book recoverable.

Consider a fictional mid-sized broker running a multi-asset retail book. At 09:17, its primary site loses access to a major liquidity venue after a network fault. The trading gateway remains reachable to clients, but prices stop updating and new orders queue. The second site is healthy.

The hard part is not starting another process. It is deciding which state is authoritative, preventing duplicate orders and restoring service without creating unhedged exposure.

The figures below are planning assumptions within typical market ranges, not measured results from a named firm. Actual objectives depend on venue rules, product risk, client terms and the time needed to reconcile external records.

Design around the state of the book

A recoverable book is a set of linked states, not a database snapshot. It includes client orders, fills, positions, cash and margin, working quotes, price marks, risk limits, session state and the acknowledgements received from each venue. If any one of these moves independently during a site switch, the restored service can be internally consistent and still be wrong.

Define what may be lost, and what may not

Set separate recovery point objectives for each state class. A firm might accept several seconds of loss in non-critical analytics, but not an ambiguous fill or a risk-limit change. For order and execution events, asynchronous replication can leave a small gap during a network partition. Synchronous replication reduces that gap but adds latency and makes availability depend on the link between sites. The choice should follow the trade-off the business can operate, not a blanket claim of zero data loss.

Keep the event record append-only where practical. Give each client order and each outbound venue instruction a durable identifier, and store the request, response and sequence number. On recovery, the system can ask whether an instruction was accepted, rejected or remains unknown.

A retry without that evidence can submit the same order twice. A replicated position table alone cannot answer the question.

Keep the risk gate independent of the trading gateway

Both sites need the same versioned limits and reference data, but a standby should not blindly inherit stale permissions. Replicate risk configuration with an effective timestamp and record who approved each change. Test what happens when the risk service is unavailable: for a retail broker, that may mean rejecting new exposure while allowing carefully controlled reductions; for a prop firm, it may mean cancelling quotes and blocking all new orders until positions are checked.

The market-data path needs its own health signals. A live socket is not proof of a fresh price. Track exchange timestamps, sequence gaps, stale thresholds and the source used for each instrument. Decide in advance whether a product is disabled, shown as indicative or priced from a secondary source when the primary feed fails. A recovery plan that restores order entry but leaves stale prices available can make the incident worse.

These ranges are starting points for design reviews, not service guarantees. A useful infrastructure review, such as the operational coverage published by Trading Technology, should prompt specific questions about dependencies and failure behaviour rather than a comparison of uptime claims.

The 09:17 incident: contain before switching

In this scenario, the primary site is still running. The network path to the venue is not. That distinction matters: automatically promoting the standby because a health check failed can leave two active gateways, each able to submit orders. The operations lead first declares a venue-path incident, freezes deployment changes and checks whether the fault affects one route, one data centre or the broker's wider connectivity.

At 09:18, the team stops new orders for the affected instruments and cancels working quotes where the venue connection is still usable. It leaves unrelated products running. The risk desk compares the last confirmed internal positions with venue drop-copy and clearing records. The gateway reports 1,240 open client orders, 86 orders with a pending acknowledgement and 17 fills received after the last confirmed replication marker. Those 17 are not assumed lost or accepted; they are placed in an exception queue.

At 09:20, the network team confirms that the primary site cannot reach the venue through either configured carrier. The secondary site has a separate route and receives current market data. The incident commander records the promotion decision and the sequence marker to use. If both routes had been degraded, the safer response would be to keep affected products restricted rather than fail over into another broken path.

Before switching, the team fences the primary order sender. Fencing can use a lease held by one active site, a hardware or network control, or venue-side session ownership. It must prevent the old sender from returning and transmitting queued requests after the standby has taken control. Killing a process is not sufficient if a supervisor can restart it or if messages remain in a broker queue.

The client notice is narrow: affected instruments are in cancel-only or close-only mode, and other products remain available. The support desk receives the same status and an approved explanation. Broad statements about platform availability conceal the operational distinction clients need, especially if an order is pending and its outcome is not yet known.

Run failover as a controlled state transition

Promote only after the old writer is fenced

Once the primary sender is fenced, the standby takes the active lease. It loads the latest durable event marker, verifies the risk configuration version and checks that its market-data sequences are current. The team does not replay every queued outbound instruction. It classifies them by evidence: confirmed reject, confirmed accept, confirmed fill, or unknown. Unknown instructions are queried through venue order status or reconciled against drop-copy before any replacement is sent.

A practical runbook gives each action an owner and a stop condition. For this book, the sequence is:

1. Declare scope. Identify affected products, venues, order types and client channels. Record the incident time and last trusted event marker.
2. Fence the primary. Prove the old sender cannot transmit, including after process restart or network recovery.
3. Promote the standby. Acquire the active lease, load durable state and confirm risk limits, reference data and clocks.
4. Reconcile uncertainty. Query venue status and compare drop-copy, internal events and position records. Keep unresolved orders blocked.
5. Open in stages. Enable monitoring and cancel functions first, then reductions, then new risk within approved limits.

Use an explicit go or no-go check between stages. In a typical deployment, a gateway may be available within one to five minutes, while confidence in the full book can take longer. Reporting only the gateway recovery time hides the period in which the system is reachable but still operating with restricted instruments or unresolved orders.

Measure the controls that determine safety

During promotion, track event lag, queue depth, rejected requests, unknown acknowledgements, venue session state and the age of each market-data stream. Alert thresholds should distinguish a brief burst from a growing backlog. For example, if the standby is 2 seconds behind at promotion and the gap is increasing, opening new risk is not justified by the fact that the site is online.

Capacity needs a load test that includes recovery work. The standby may receive new client traffic while also rebuilding caches, reconnecting to venues and reconciling orders. A system that handles normal peak throughput in isolation can still fail under this combined load.

Test at the expected peak plus a reasonable margin, with logging, risk checks and monitoring enabled. Measure tail latency and queue growth, not just average requests per second.

Keep a human approval step for actions with asymmetric consequences, such as reopening a leveraged product or resubmitting an order with an uncertain outcome. Automation is useful for detection, fencing and health checks. It should not silently turn uncertainty into a trade. The decision record should capture the state observed, the person approving the transition and any products left restricted.

Restore the primary without creating a second incident

At 09:31, connectivity at the primary site returns. The active book remains on the secondary site. Returning immediately would create another role change before the team has established whether the network is stable and whether the old site has drained its queues. The primary stays fenced while engineers inspect process health, storage lag, venue sessions and the state of any messages persisted before the outage.

After a stable observation period set by the incident team, the primary is rebuilt as a replica from the active event stream. It is not promoted simply because it is the original site.

The team checks that its event marker has caught up, compares positions and open orders, and runs a controlled exercise on a limited product set. Only then does it decide whether to move the active lease back. Some firms keep the secondary active until a quieter trading window because a second switch has its own execution and reconciliation risk.

Close the incident with evidence, not a green dashboard. Preserve event logs, venue drop-copy, network telemetry, promotion approvals and client-impact records.

Reconcile cash, positions, fees and corporate actions through the normal post-trade process. Capture the duration of restricted trading separately from gateway downtime, along with the number of orders that required manual resolution and the cause of each one.

Exercises should include partial failures, not only a clean loss of a whole data centre. Test a stale price feed, a dropped acknowledgement, a replication delay, a split-brain attempt and a venue session that refuses to log in at the standby. Run them with realistic order volumes and the people who will hold the incident roles. A plan that works only when the network is neatly disconnected is not a useful plan for a live book.

Stop doing these things:

• Stop treating a running standby as proof that the book is recoverable.
• Stop retrying orders whose venue outcome is unknown.
• Stop promoting a site before fencing the previous writer.
• Stop reporting gateway uptime without stating the trading restrictions that remained.

#recuperación ante desastres de brokers #conciliación de operaciones #replicación del estado de las órdenes #trading system failover #trading platform RTO #active-passive trading architecture #venue outage runbook #risk control continuity