Disaster recovery for a live trading book: a weekly checklist

Decide what fails over, and what must stop

One operator promotes a standby as soon as its primary loses connectivity. Another fences the primary, checks replication position and order state, then promotes. The first may restore order entry faster.

The second reduces the chance of two active writers sending orders against the same book. Neither call is safe by itself: the right sequence depends on how the platform handles authority, state and venue sessions.

For a live book, recovery is not simply bringing up servers in another zone. It means establishing one authoritative writer, restoring the state needed to make safe decisions, and proving that orders, fills, positions and risk limits agree. A fast login page is no evidence that the trading service is ready.

Start this week by defining recovery objectives per function, not one broad platform target. Separate order acceptance, market data, pricing, risk checks, client access, reporting and back-office feeds. Decide which can be read-only or unavailable temporarily, and which must be blocked until state is reconciled. A typical objective might allow client reporting to lag several minutes while requiring order entry to remain disabled until the book is consistent.

Write down who can declare a disaster, who owns the failover token and who can stop trading. A runbook that depends on one person recognising an alert is not a runbook. For a broader view of platform operations, Trading Technology's infrastructure coverage examines the systems brokers and prop firms depend on.

Map the state and set measurable targets

List every component that can change book state: order management, risk, execution gateways, position service, cash ledger, client session store and the event log. For each, identify its source of truth, replication method, lag alert and recovery owner. Include venue sessions and clearing or prime broker connections. A database replica does not reproduce a live FIX session, an unacknowledged cancel or a venue's view of an order.

The figures below are typical market ranges for planning discussions, not guarantees or audited results. A firm's targets depend on its architecture, venues, product set and regulatory obligations. Measure them in a rehearsal under representative load.

For each target, name the clock's start and end points. "Five-minute RTO" is ambiguous if one team starts timing at the alert and another at the incident declaration. Record detection, fencing, promotion, validation and first permitted order as separate timestamps.

Run one controlled rehearsal this week

Choose a low-risk window and a representative book, including the busiest gateway or instrument group the test environment can safely support. Do not run a production failover by cutting power or blocking a network path without an approved change, a rollback plan and venue coordination. If production testing is not authorised, inject failures in staging and compare configuration and event flow against production.

1. Freeze the scope. Name the services, venues, accounts and operators in the test. Capture current positions, open orders, limits, replication lag and gateway session status.
2. Set a known event boundary. Record the last durable sequence number on the primary and the standby. Confirm which log, database or stream defines that number. Note any asynchronous feeds that can arrive later.
3. Trigger the agreed failure. Simulate loss of the primary service or site through the approved method. Observe alerts and timestamps. Do not let the standby accept writes before the fencing mechanism proves the old primary cannot write.
4. Promote and restore dependencies. Check configuration, secrets, certificates, DNS or route changes, clock synchronisation and venue connectivity. Confirm the standby has the right risk limits and product settings, not just a healthy process status.
5. Reconcile the book. Compare internal open orders and executions with venue drop copies or equivalent authoritative records. Resolve new, pending, rejected and partially filled orders. Check cancels in flight; never infer that a missing acknowledgement means an order did not reach the venue.
6. Validate risk before enabling order entry. Recalculate positions and exposure from the agreed source of truth. Check credit, margin, concentration and throttles. Confirm kill switches work from the recovery site.
7. Allow controlled traffic. Start with a designated test account or limited flow, then increase only after monitoring confirms expected acknowledgements, fills, rejects and position updates. Keep a named operator authorised to stop the test.
8. Restore and record. Test failback separately rather than treating it as the reverse of promotion. Capture timings, lost or duplicated events, manual interventions and unresolved differences, then assign owners and dates to each fix.

Before the test, agree on stop conditions. Examples include an unfenced primary, a sequence gap outside the approved recovery-point objective, unexplained position differences, stale risk limits or a venue session that may still be active at both sites. Record who has authority to abort, and make the abort action clear to the whole team.

Prove that recovery is safe to trade

After the rehearsal, review evidence rather than relying on a green dashboard. Check event sequence continuity, duplicate handling, order and execution reconciliation, risk calculations, client-visible state and monitoring coverage. Compare actual timings with the targets by function. If order acceptance returned in four minutes but risk recalculation took twelve, the book was not recovered in four minutes.

Track the findings in a short action list:

• Remove any path that permits two writers, including stale credentials or an unexpired leader lease.
• Alert on replication lag, broken drop-copy feeds and clock drift before an incident.
• Make the runbook show commands, permissions, decision owners and rollback steps. Validate access from the recovery environment.
• Retest after material changes to the order manager, risk engine, venue connectivity, cloud region or replication topology.

Keep the recovery record with incident procedures and audit evidence. Include the test scope, versions, timestamps, discrepancies, operator decisions and remediation owners. A quarterly full-site exercise can test wider dependencies, but short, targeted checks each week can catch expired certificates, changed routes and drifted configuration earlier.

This checklist assumes the firm or its technology provider can control promotion and inspect authoritative trading state. It does not apply unchanged during an exchange or clearing outage, where switching the broker's own infrastructure cannot restore the external venue, or where an outsourced platform contract gives the firm no control over failover mechanics. In those cases, use the venue's incident process or obtain the provider's tested recovery evidence and agree responsibilities before relying on its recovery targets.

#trading book failover #broker disaster recovery #trading platform resilience #active-passive failover #order replay #trade reconciliation #recovery time objective #risk controls