Platforms · Software · Providers · Markets
Trading disaster recovery is the process of restoring trading services and their state after a serious failure.
For a live book, that is not the same as starting the same software in another data centre. A standby can accept traffic while holding stale positions, missing acknowledged fills or reusing an order identifier.
Failover is safe only when the replacement can establish what happened, enforce risk limits and resume execution without duplicating or losing activity. The hard problem is controlled recovery, not switching network routes.
What a live book needs to recover
A live book is the connected state that lets an operation accept, check, route and account for orders. It spans more than the order management system. It includes positions and cash, working orders, risk reservations, client permissions, execution reports, market data state, gateway sessions and the audit trail. Teams covering trading infrastructure need to map which system owns each piece, and which downstream services can rebuild from that owner.
Start by marking each item as authoritative, reconstructable or disposable. A typical inventory includes:
- Authoritative state: order and execution journals, positions, risk limits and client account status. A second service must not invent or infer these from cached screens.
- Reconstructable state: dashboards, analytics and some market data caches, which can be rebuilt from durable records or refreshed feeds.
- External state: venue orders, execution reports and session sequence numbers. Your database cannot decide what a venue accepted while your connection was down.
Record every boundary between those systems. If the risk engine reserves buying power before the order router sends an order, recovery must know whether a reservation with no matching route record is pending, rejected or uncertain. Preserve event timestamps and correlation identifiers so operators can trace one client order through checks, routing, venue acknowledgement and fill.
Choose a failover shape
The right design depends on how quickly the operation must resume, how much state it can afford to lose and the latency cost of copying data between sites. Most brokers and prop firms use a primary site with a warm standby, or run selected services in active-active mode. Neither removes the need to reconcile with venues.
Active-passive: simpler ownership, slower recovery
In active-passive, one site owns writes and trading connections. The standby receives replicated databases and configuration, but does not normally submit orders. This limits conflicting writes and makes ownership clear. It is often the more manageable starting point for a firm with one main execution stack and a recovery target measured in minutes.
Replication can be synchronous or asynchronous. Synchronous replication can keep committed state aligned across sites, but adds network latency and can make the primary unavailable when the standby link fails.
Asynchronous replication protects primary performance, but creates a recovery point objective (RPO): the interval of recent data that might not exist at the standby. A replication lag alarm is useful only if someone knows whether to stop trading when it crosses the approved limit.
Active-active: faster service, harder consistency
Active-active can keep customer APIs, pricing or read-only services available across sites. Letting both sites write to the same book is more difficult.
Network partitions can create two apparent owners, each accepting orders against incomplete positions. Prevent that with a fencing mechanism, such as a lease or quorum-backed ownership token. A site must lose permission to write before the other takes control, not merely lose access to a health check.
Keep order submission single-writer unless the trading and venue design explicitly supports more than one active path. Idempotency keys help reject duplicate requests inside your platform, but do not guarantee that a venue saw an order only once. Where a send times out after transmission, treat the outcome as unknown, query or reconcile the venue state, then decide whether to resend.
Set targets that reflect trading risk
RPO measures how much committed data may be lost. Recovery time objective (RTO) measures how long a service may be unavailable.
Set both per workload rather than assigning one number to the whole platform. The ranges below are typical engineering target ranges, not guaranteed results; venue behaviour, distance between sites, database design and testing discipline all affect what is achievable.
| Workload | Typical RPO range | Typical RTO range | Recovery consideration |
|---|---|---|---|
| Order and risk state | 0-1 second | 1-5 minutes | Confirm acknowledged orders and fills before reopening routes. |
| OMS event journal | 1-5 seconds | 5-15 minutes | Replay events in order and make handlers idempotent. |
| FIX gateway session | 0-30 seconds of local state | 1-10 minutes | Log on, check sequence numbers and reconcile live orders. |
| Market data cache | Cache may be disposable | 1-5 minutes | Reconnect, validate feed health and rebuild required books. |
| Reporting and analytics | 15-60 minutes | 1-4 hours | Recover after execution and risk services are stable. |
Choose an RPO by consequence, not convenience
A one-second RPO for order state is not permission to discard a second of fills. It is a design target for replicated state, followed by reconciliation against the venue and the durable execution record. If the firm cannot establish whether an order was accepted, it should hold the affected account or instrument rather than silently treat the order as absent.
Replication methods have different costs. Database log shipping or change-data capture can keep a standby close to current, while periodic snapshots are easier to operate but leave a larger gap. A typical cross-site replication lag target might be below one second for critical state, with alert thresholds and a defined response if it rises. Test under market-open load: a quiet overnight test will not expose queue growth or slow commits.
Make the RTO include decisions and checks
Measure recovery from detection to safe trading, not from process start to a green health check. The clock includes declaring an incident, fencing the primary, promoting data, restoring connectivity, checking risk state and obtaining an operator decision to reopen. A platform can be technically available in seconds but still be unsafe to trade for several minutes.
Define service gates. For example, customer access may return in read-only mode while order entry stays disabled until positions reconcile and each venue session is confirmed.
Agree who can lift the restriction, what evidence they need and whether a risk officer must approve it. These choices make the RTO operational rather than just a number in a design document.
Prove the runbook under market conditions
A recovery plan is an executable sequence, not a diagram. Keep it versioned with infrastructure changes, and write actions in the order an on-call engineer will need them. It should identify the incident commander, system owners, communication channel, authority to halt trading and the evidence required to return to service.
A basic failover runbook should cover these steps:
- Declare the incident and stop new order entry if ownership or state is uncertain.
- Fence the failed site. Confirm it cannot regain write access or send orders after recovery.
- Promote the standby and check database position, replication lag, event offsets and configuration versions.
- Restore gateway connectivity. Review FIX sequence numbers and use venue status queries or drop-copy records to reconcile working orders and fills.
- Recalculate positions, cash and risk reservations. Compare totals across the OMS, risk engine and execution records.
- Reopen in stages, starting with controlled accounts or instruments, and monitor rejects, duplicate identifiers, latency and exposure.
Test failure modes separately. A site loss is different from a database corruption, a broken replication link, a bad deployment or a venue connection failure. Run tabletop exercises for decisions and technical failovers in a controlled environment. Then test a production-like failover with order entry disabled or tightly bounded, after agreeing the risk controls and venue constraints.
Capture elapsed times, data gaps, manual interventions and recovery defects. Repeat after material changes to routing, risk logic, database topology or venue connectivity. The useful metric is not whether the test passed, but whether the team can explain every order and position at the point it resumes trading.
If this were my operation, I would first trace one order from client request through risk, routing, venue acknowledgement and the durable execution record. Then I would use that trace to set the state ownership rules and rehearse a failover with order entry held until reconciliation is complete.
Frequently Asked Questions
What does disaster recovery mean for a trading system?
It means restoring trading services and their required state after a serious failure. For a live book, that includes proving order, execution, position and risk state before normal order entry resumes.
What are suitable RPO and RTO targets for a broker?
There is no single target for every workload. Typical targets for critical order and risk state may be an RPO of 0-1 second and an RTO of 1-5 minutes, while reporting systems can usually tolerate longer recovery.
Is active-active safer than active-passive failover?
Not by itself. Active-active can improve availability for some services, but concurrent writes to a trading book can create conflicting state unless ownership is fenced and data consistency is designed and tested.
How should a firm handle an order with an unknown venue outcome?
Do not assume a timed-out order was rejected or accepted. Check venue status, drop-copy records and sequence information, then reconcile before deciding whether a resend is safe.
How often should a trading disaster recovery plan be tested?
Test after material changes to routing, risk logic, database topology or venue connectivity, and schedule regular exercises between changes. Include both technical failover and operator decisions, with order entry constrained until reconciliation is complete.
