Disaster recovery for a live trading book needs proof

Racks of trading servers in a data centre, illustrating disaster recovery for a live trading book
Photo by Christina Morillo on Pexels

In week one, map which systems and people must be available to protect an open book. By week six, run a controlled failover that proves the firm can recover positions, reconcile executions and prevent duplicate orders.

A second data centre is not evidence of recovery. The evidence is a recorded test with measured recovery times, known data gaps and signed decisions about when trading can resume.

For a broker or prop firm, recovery is also a risk and compliance control. A service can be technically online while the firm has lost the state needed to set limits, supervise activity or explain an execution. Define recovery around the trading book, not around server uptime.

Use the first week to define the recovery boundary

Start with the components that determine exposure: order management, execution gateways, the position and cash ledger, pre-trade risk, client and account permissions, and the event journal. Map their dependencies on market data, identity services, network routes, venue sessions and third-party systems. A recovery site that restores the order manager but not current limits or account status is not ready to accept orders.

Record which data is authoritative. For example, a position snapshot may be stale, while an append-only journal of accepted orders, acknowledgements, fills, cancels and corrections can be replayed to rebuild it. Include venue sequence numbers and timestamps, and preserve the distinction between an order sent, an order acknowledged and a fill received. The distinction matters for both exposure and client disputes.

In the first week, agree the recovery decision rights as well as the technical map. Name who can stop new risk, authorise a move to the recovery environment and approve reopening. Include operations, engineering, risk and compliance. Put the initial control points in writing:

  • Which products, accounts and venues remain available during degraded operation.
  • Whether to reject new orders, permit cancel-only access or keep selected strategies running.
  • Who can approve a limit change or resume trading, and how that decision is recorded.
  • Which client and regulatory communications may be required under the firm's applicable rules.

Protect the book before restoring the service

Failover must not create two active writers for the same account or venue session. Use fencing, a lease or another enforceable leader-election control so the primary cannot continue submitting orders after the standby takes authority. A DNS change alone does not provide fencing. Test what happens if the primary is isolated from the control plane but can still reach a venue.

Order replay needs equal care. Persist a unique internal order identifier and the venue's identifiers, then check acknowledgement and execution state before retrying an uncertain submission. If the venue supports client order IDs and duplicate detection, use them, but do not assume behaviour is identical across venues. Where status is ambiguous, hold new risk on the affected account or symbol while operations obtains authoritative state. Blind retries can turn a timeout into a duplicate position.

Do not make automatic flattening the default recovery action. It can increase market impact, breach a client instruction or compound an unknown execution state.

A safer policy may freeze new exposure, retain cancel access where possible and reconcile before deciding whether to hedge or close. Set that policy by product, venue and client agreement, with risk approval. Firms comparing platform and connectivity approaches can use Trading Technology's infrastructure coverage as one source of industry context.

Set targets against the cost of lost state

Recovery time objective (RTO) describes how quickly a service must return. Recovery point objective (RPO) describes how much committed data the firm can tolerate losing. Treat these as separate targets for each function. The figures below are typical planning ranges across trading environments, not regulatory safe harbours or guarantees. Architecture, venue access and distance between sites change what is achievable.

Function or measureTypical market rangeOperational implication
Order entry and execution RTO5 to 30 minutesAllow time to fence the primary, establish venue sessions and validate controls.
Accepted order and fill event RPO0 to 1 second targetUse a durable journal and replay; verify gaps against venue records.
Risk and position state RPO1 to 5 seconds for snapshotsRebuild from events and reconcile before enabling new exposure.
Cross-region replication lag100 milliseconds to 2 secondsMeasure lag continuously and define a threshold that blocks unsafe promotion.
Full failover exerciseQuarterly to twice yearlyInclude a live-book or representative production-like test, not only a restore.

These ranges are useful for challenge, not procurement promises. Synchronous replication can reduce data loss but add latency and make a distant site harder to use. Asynchronous replication can improve performance and distance, but it leaves a window in which the standby is behind.

For the order and fill journal, decide how the system behaves if lag exceeds its threshold: pause new risk, promote with a known recovery point, or remain on the primary. Document the trade-off with risk ownership.

Make reconciliation a release gate

A service being reachable does not prove that its book is correct. Before reopening new risk, compare internal orders, fills, positions, cash and margin with venue reports and clearing or prime-broker records available to the firm. Reconcile by account and instrument, not just by aggregate totals. Set tolerances for rounding and timing differences, but require an owner and an explanation for every unresolved break.

Keep an auditable timeline of the incident and test: detection, promotion decision, system state, operator actions, venue communications, risk overrides and approval to resume. Preserve the relevant order and execution records, clock synchronisation details and surveillance inputs.

Recordkeeping and notification duties vary by jurisdiction and activity, so compliance should map the firm's obligations rather than rely on a generic recovery checklist. A surveillance gap should be identified and assessed, not hidden by a successful application restart.

Track the test results against the agreed targets. Capture actual RTO and RPO, replication lag at promotion, number and age of unreconciled breaks, rejected or duplicated orders, and time until risk approved reopening.

A test that misses a target can still be useful if the gap is recorded, assigned and retested. A passing test should also include failure injection, such as loss of the primary site, a broken network route or an unavailable identity provider.

Use week six to test the decision, not just the switch

By week six, run a controlled exercise with operations, engineering, risk and compliance in the room. Use a production-like book or a bounded test environment with realistic order flow. Include open orders, partial fills, cancels in flight and a venue session that cannot be cleanly queried. Measure from detection to a safe operating state, not merely from failover command to a responding health check.

Before the exercise, brief participants on the scope but not every injected fault. During it, require the named decision-makers to apply the written policy.

Afterward, review whether the firm could establish what it owned, what remained working at venues, and whether it had enough evidence to resume. Assign remediation to named roles with dates, and keep the approved test record with the business continuity and operational risk evidence.

A practical recovery plan also defines how normal service returns. Avoid switching back simply because the primary site is available.

Reconcile any transactions created during failover, confirm replication direction, then approve a controlled failback with the same fencing and monitoring used for promotion. The final control is a number that can expose a gap quickly: check your measured RPO for accepted orders and fills against the 0 to 1 second planning range.

Frequently Asked Questions

What RTO is typical for a trading platform?

Typical planning ranges for order entry and execution are 5 to 30 minutes, but the right target depends on products, venues and the firm's risk controls. Measure time to a safe operating state, including venue reconnection and validation, rather than time to application availability.

What should a broker reconcile after failover?

Reconcile orders, acknowledgements, fills, positions, cash and margin against venue and available clearing or prime-broker records. Investigate differences by account and instrument, and keep new risk disabled for affected areas until an authorised owner resolves or accepts each break.

How often should a trading firm test disaster recovery?

Quarterly to twice-yearly full failover exercises are typical planning ranges, with more frequent component and restore tests in between. Include realistic order states, venue connectivity and decision-making, then record measured RTO, RPO and unresolved reconciliation breaks.