Disaster recovery for a live trading book in 30 days

A second data centre does not, by itself, give a trading firm disaster recovery. A standby site that cannot take authority over orders, rebuild positions and prevent duplicate execution is extra infrastructure, not a recovery capability. The 30-day plan below is for an existing broker or prop firm stack, not a new platform build.

Set recovery targets first

Decide what the firm must restore, in what order, and with what acceptable loss of state. A trading system has several recovery targets, not one. Client access, order entry, execution routing, risk checks, market data, position services and back-office ledgers can have different recovery time objectives (RTOs) and recovery point objectives (RPOs).

For a typical small or mid-sized firm, a warm, active-passive design might target a 15 to 30 minute RTO for order entry and less than five seconds of lost internal event data. A cold standby can take 30 to 90 minutes to bring up. Those are planning ranges, not guarantees. Venue connectivity, platform licensing, data restoration and the condition of the live book all affect the result. Set stricter or looser targets based on risk, contractual obligations and the firm's regulatory requirements.

Record which services are essential to resume dealing and which can wait. An operations dashboard can be unavailable while the dealing system recovers, but risk controls cannot be treated as optional.

Define a safe operating mode, such as reduced order limits or disabled products, for cases where the full control plane is not ready. Broader trading infrastructure coverage can help teams compare the system components that need to be included in this map.

Days 1-5: map the book and failure boundary

Trace a client order from the front end through the order management system, risk checks, bridge or execution gateway, venue and back again. Mark every component that holds state, every dependency that can stop an order, and every route that could still accept traffic after a failure.

Include DNS, identity services, secrets, time synchronisation, licences, firewalls and third-party connections. These are common gaps in diagrams that focus only on servers.

For each component, record its owner, primary and standby location, data replication method, restart procedure and dependency order. Identify the system of record for each important object. A practical list includes:

• Order acceptance and amendment events, including client and venue identifiers.
• Execution reports and drop-copy feeds, with sequence numbers and timestamps.
• Positions, cash, margin and open-order state used by pre-trade risk.
• Client configuration, symbol mappings, permissions and routing rules.
• Audit logs, reconciliation outputs and records needed for regulatory reporting.

Interview the dealing desk and operations staff as well as engineers. Ask what they do when a venue connection drops during a partial fill, or when a client retries an order after a timeout. The answer can reveal operational dependencies that are absent from code and architecture diagrams.

Days 6-12: build a fenced standby

Choose active-passive unless the team can operate active-active safely. Active-active can reduce capacity recovery time, but it requires clear ownership of order IDs, risk state and routing.

Without that, two sites can accept the same request or act on different views of exposure. A warm passive site is usually simpler: keep critical services and data synchronised, but do not allow it to send orders until the primary is fenced off.

Fencing is the control that prevents the failed primary from returning and sending orders alongside the standby. Use a mechanism the team can verify, such as disabling the primary's venue sessions and routing credentials, combined with a lease or quorum-based authority check.

A DNS change alone is not fencing. Build a manual override for a control-plane failure, document who can invoke it, and require a second person to confirm the action.

Provision enough standby capacity for the recovery mode, not necessarily full peak throughput. Measure CPU, memory, network and queue depth under a representative load.

Confirm that venue sessions, market-data permissions and licences can operate from the recovery location. Agree a degraded mode in advance if the standby cannot support every product or client at full capacity.

Budgetary figures below are typical market ranges in US dollars, not audited costs for any named firm. They vary with venue count, data entitlements, region, existing contracts and whether the standby site is cloud-based or physical.

Days 13-20: make state recoverable

Replication does not prove that the book can be reconstructed. Test the path from persisted events to a usable position and order state. Keep an event log or durable journal with monotonic sequence numbers, and make replay idempotent. The same execution report may arrive from both a venue session and a drop-copy feed; the recovery process must deduplicate it using stable identifiers rather than add the fill twice.

Do not blindly replay the outbound order queue. An order may have reached the venue even if the primary lost the acknowledgement.

On recovery, classify orders as confirmed, rejected or uncertain. Query venue state where supported, consume drop copy, and reconcile by client order ID, venue order ID, execution ID and sequence. Hold uncertain orders for controlled resolution rather than sending a fresh order that could duplicate exposure.

Rebuild positions and cash from executions, then compare the result with the risk service, broker ledger and venue records. Define tolerances and an escalation path for mismatches. Market data can generally be reacquired from a fresh snapshot and subsequent updates, but the snapshot and stream need a sequence or gap check. If a gap cannot be closed, block trading in the affected instruments or use an explicitly approved fallback feed.

Protect secrets, certificates and configuration as carefully as the database. Test restoration from backups into the standby environment, including permissions and version compatibility. Keep a known-good release package and configuration snapshot so recovery does not depend on deploying untested code under pressure.

Days 21-26: rehearse controlled failover

Run the first exercise in a non-production environment with production-like state and venue simulators. Then schedule a controlled production rehearsal during a quiet window, with agreed limits and an abort condition. The objective is to test decision-making and reconciliation, not to prove that a button works.

Use a numbered runbook that names an incident lead, the person authorised to declare failover, and the operators responsible for platform, network, risk and client communications. A minimum checklist is:

1. Declare the incident and stop new order entry if the state of the primary is uncertain.
2. Fence the primary, including venue sessions, credentials and any routes that can still accept orders.
3. Confirm the standby has authority, current configuration, healthy risk checks and required market data.
4. Restore order and execution state from the journal, venue queries and drop-copy records.
5. Reconcile positions, open orders, cash and margin. Escalate every mismatch outside agreed tolerances.
6. Enable a controlled product and client subset, then increase limits only after checks pass.
7. Record timestamps, decisions, failed steps and evidence for the post-incident review.

Measure actual elapsed time at each step. A 10-minute server start that follows a 40-minute authority decision is not a 10-minute recovery.

Record the RTO and RPO achieved, duplicate or missing events, manual interventions, and the client impact. Repeat the exercise after fixing the largest faults. One successful run is not evidence that recovery is repeatable.

Days 27-30: make recovery an operating control

Turn the runbook and exercise results into owned operational controls. Put recovery checks into the change process: significant changes to routing, risk logic, schemas, identity, network policy or venue connectivity should trigger a review of standby compatibility. Automate health checks where they can safely detect stale replication, expired certificates, sequence gaps and insufficient capacity.

Set a cadence that fits the firm's change rate and risk. Many teams test component restoration monthly, conduct a failover rehearsal quarterly, and run a broader business continuity exercise annually. Use shorter intervals after major platform changes or a failed test.

Publish the runbook to the incident channel and maintain an offline copy for an outage that affects identity or internal documentation services. A broker technology operations reference can sit alongside internal procedures, but it cannot replace firm-specific escalation paths or venue instructions.

Close the 30-day programme with an evidence pack: approved RTO and RPO targets, dependency map, cost assumptions, restoration results, reconciliation tolerances, open defects and named owners with due dates. Keep a short list of conditions that prohibit dealing from the standby, such as unresolved position differences, missing venue authority or failed pre-trade risk checks. A recovery plan is useful only if staff know when not to resume trading.

This plan does not apply to a firm whose book is hosted and recovered entirely under a provider's tested service, with contractual recovery objectives and independently verified evidence that cover the firm's own order, risk and reconciliation paths.

#recuperación ante desastres de brokers #resiliencia de la plataforma de trading #infraestructura de prop firm #trading platform failover #trading system RTO #order reconciliation #active-passive trading infrastructure #disaster recovery testing