Disaster recovery for a live trading book: failover choices

In nearly every live trading setup, the primary and recovery environments are less alike than their diagrams suggest. The recovery site may have the same software, but different market-data paths, venue sessions, risk limits or credentials. A server can start there while the book remains unsafe to trade.

For disaster recovery, the real objective is not simply to restore an application. It is to resume order handling with a known position, a single active authority and controls that still reflect the firm's exposure. The right approach depends on how much interruption the business can absorb, how much state it can afford to lose and how much infrastructure it can keep ready.

Start with the state that must survive

Recovery time objective (RTO) is the time from declaring a failure to accepting valid new orders. Recovery point objective (RPO) is the amount of committed state that could be lost.

Neither measure alone proves that a trading book is safe. A low RTO with an uncertain order ledger can increase risk rather than reduce it.

Map the state by system and define its source of truth. Typical components include:

• Order and execution events, including client acknowledgements, venue acknowledgements, fills, cancels and rejects.
• Positions, cash, margin usage and risk-limit consumption, derived from the event stream or reconciled against a ledger.
• Session state for FIX connections, drop copy, market data, clearing and other external services.
• Configuration and reference data, including symbol mappings, routing rules, credentials and risk parameters.

Keep immutable event records and use sequence numbers or equivalent offsets to identify the last confirmed event at each site. If a primary process fails after a venue accepts an order but before the acknowledgement reaches the recovery site, the firm must establish whether the order is live before retrying it. That is a trading-control problem, not a database restore problem.

Compare the recovery patterns

The ranges below are typical market ranges, not guarantees. RTO measures time to valid order acceptance after a site-level incident, not the time to boot a server.

RPO depends on replication design, write acknowledgement policy and the failure being tested. Capacity overhead is relative to one production site and includes the recovery compute needed to meet the stated service level.

Active-active and hot standby

Active-active runs workloads in two sites and distributes traffic between them. It can keep order entry available through a site failure, but only if ownership is partitioned or coordinated.

Two sites processing the same account or order stream can create duplicate orders, divergent risk checks or conflicting cancels. Firms often assign symbols, accounts or sessions to a single active site, with the other ready to take ownership.

Hot standby keeps the recovery stack running and state close to current, but normally leaves it unable to submit orders until promotion. It is simpler than active-active and can offer a shorter recovery window than warm standby. The trade-off is cost: near-production compute, current data feeds and enough network capacity must be available before the incident.

Warm standby and cold recovery

A warm site has core systems and replicated data available, but may need capacity changes, feed subscriptions or operator steps before it can trade. It suits firms that can tolerate a longer outage and want to limit standing cost. Cold recovery relies more heavily on backups and manual provisioning. It can work for low-urgency services, but restoring the trading stack, network routes, credentials and venue access may take hours.

Do not choose a pattern from an RTO target alone. The total operating cost also includes data links, duplicated licences, market-data entitlements, venue certification and the staff required to maintain a second environment. Teams tracking changes across the broader trading technology stack should include these dependencies in the recovery inventory, not leave them in separate vendor runbooks.

Control promotion, replication and split brain

Make ownership explicit

Failover needs a reliable way to stop the old site from acting before the new one takes control. Use a fencing mechanism such as a quorum-backed lease, a network isolation step or a venue-side session control. A timeout alone is weak evidence: a slow or partitioned primary may still be sending orders.

Write down the promotion sequence and make each step observable. A practical runbook should cover:

1. Declare the incident and decide whether the primary is isolated or merely unreachable.
2. Fence order submission from the primary, then promote the recovery site under a single owner.
3. Verify replicated offsets and reconcile positions, working orders and recent executions against venue and clearing records.
4. Restore market data and risk checks, then enable order entry in controlled stages.
5. Monitor rejects, duplicate client IDs, sequence gaps and exposure before reopening normal flow.

Replicate events, not just files

Database replication can protect stored records, but it does not automatically preserve the relationship between an order request, venue response and risk reservation. Asynchronous replication reduces latency on the primary path but leaves a replication lag window. Synchronous replication can reduce that window, at the cost of adding network latency and making writes sensitive to a link failure.

Use idempotent processing where possible. Give each client request a durable identifier, record venue identifiers and make replay safe against duplicate delivery. A recovery process should be able to replay a known event range, query venues for open orders and reconcile the resulting book without assuming that the last locally written record is the last external action.

Prove the recovery path under load

A failover test that only stops an application process does not test site recovery. Rehearse failures that remove a data centre, break inter-site connectivity, delay replication or leave a venue session in an uncertain state. Include the people and external dependencies needed to make the recovery decision.

Measure more than elapsed time. Record the age of replicated state, order acceptance rate, time to rebuild risk checks, number of manual interventions and any differences found during reconciliation.

Test at representative peak rates, including bursts. A recovery site sized for average traffic may fall behind when it inherits a primary site's full order flow.

Useful acceptance criteria include:

• No order is submitted by both sites after promotion.
• All venue acknowledgements and fills are accounted for, including events received during the failure window.
• Positions and risk totals reconcile within defined tolerances before normal limits are restored.
• Operators can identify the active site and current data offset from monitoring, without relying on tribal knowledge.

Run these exercises on a schedule and after material changes to routing, data stores, risk logic or network design. Keep the results, gaps and corrective actions in the same operational record as the recovery plan. The mistake most readers will still make is treating failover as a server switch, rather than a controlled transfer of authority over orders, positions and risk.

#broker infrastructure #RTO and RPO #active-active trading systems #order state replication #trading failover #disaster recovery plan #hot standby #warm standby