Trading disaster recovery for a live trading book

A failover that restores servers but cannot say which orders reached the venue has not recovered a trading book. The recovery target is a controlled return to dealing with known positions, bounded exposure and one authoritative order path. Availability is only part of the job.

Start by writing down what must survive each failure, who can authorise a switch and what trading remains blocked until state is verified. The figures below are typical planning ranges, not service guarantees. Actual targets depend on venue access, topology, replication method and the latency budget of the strategy.

Set recovery targets around the book

Separate service recovery from book recovery

Recovery time objective (RTO) is the time to restore a service. Recovery point objective (RPO) is the amount of data that can be lost.

For a trading system, neither number describes whether open orders and positions are safe. Define a separate condition for resuming order flow: the team must reconcile the internal journal against venue acknowledgements, execution reports and drop copy.

Typical ranges give a starting point for service-level planning:

These are typical market ranges for planning, not audited results or a promise that a particular stack will meet them. A reporting warehouse can tolerate a longer recovery than pre-trade risk. A dropped risk service, by contrast, should stop new exposure rather than fail open.

Choose a topology that fits the latency budget

Active-passive deployment is often simpler to reason about: one site owns order entry, while a warm secondary receives state and can be promoted. Active-active can reduce recovery time and use capacity more evenly, but it makes ownership, duplicate suppression and position aggregation harder. A split-brain event, with both sites accepting orders, can create exposure faster than a manual recovery can contain it.

Synchronous replication can reduce data loss, but adds latency and relies on a healthy path between sites. Asynchronous replication preserves the primary site's latency budget, but creates a reconciliation gap after a sudden failure. Record which journals replicate synchronously, which replicate asynchronously and what trading behaviour is permitted during lag. Trading Technology's infrastructure coverage can help frame the operational questions, but the decision should follow your own venue and strategy constraints.

Make failover a controlled sequence

Automation should detect failure and prepare recovery. It should not promote a second order path unless the old one is fenced.

Fencing means proving that the previous primary cannot submit orders: isolate its network path, revoke or disable its venue session, or use a tested equivalent. A health check that only shows a failed process is not proof that the process cannot still reach a venue.

Define the sequence in an operations runbook, with named owners and explicit stop conditions:

1. Declare the failure and freeze new order entry. Apply a firm-wide or account-level risk block if the health of pre-trade checks is uncertain.
2. Fence the primary and confirm that its venue sessions cannot send orders. Do not rely on DNS changes alone, which may leave established connections alive.
3. Promote the secondary only after it has the required configuration, credentials, limits and recent journal state. Check clock synchronisation and venue connectivity.
4. Recover sessions using the venue's documented sequence-number process. Do not reset sequence numbers simply to clear an error; first establish what messages were sent and acknowledged.
5. Reconcile open orders, fills, cancels, positions and cash before enabling order entry. Record exceptions and assign an operator to resolve each one.

Clients and internal services also need safe retry behaviour. Use a stable client order identifier and make replay idempotent, so a retry cannot create a second live order. Preserve request and response logs, including timestamps and correlation IDs. If order status is unknown, query or reconcile it before sending a replacement.

Rebuild state before reopening flow

Use independent evidence

The secondary's database is not the only source of truth after a failure. Compare its order journal with venue execution reports, drop copy and clearing or back-office records where available. Recalculate positions from executions, then compare the result with both the recovered position service and the venue's view. Include cancels pending at the moment of failure: a cancel request is not a cancelled order until confirmed.

Keep a reconciliation ledger with, at minimum:

• Orders known to the firm but absent from venue records, and venue orders absent from the firm's book.
• Executions with missing acknowledgements, duplicate reports or unresolved allocation.
• Position and cash differences by account, instrument and venue.
• Manual actions, the operator who made them, and the evidence used to justify them.

Do not erase uncertainty by taking a snapshot and carrying on. Hold affected accounts or instruments at a reduced limit until differences are understood. For a market maker or prop desk, that may mean restricting new orders to liquid instruments or a small fraction of normal size while the book is checked.

Reopen in stages

Separate connectivity recovery from permission to trade. A useful sequence is login and market data, read-only checks, cancel and replace capability, then new order flow at a capped size.

For example, a firm might begin at 10 to 25 per cent of normal order limits, review fills and risk for 15 to 30 minutes, and raise limits in agreed steps. These are typical operational starting ranges, not universal thresholds.

Before each step, confirm that pre-trade controls are active, kill switches work, price and quantity limits are current, and the secondary is the sole order authority. Notify support and dealing teams of the permitted instruments, accounts and order types. Keep a clear route to freeze flow again without waiting for an incident bridge to agree on terminology.

Test the failure, not just the switch

A green dashboard proves that a service is responding. It does not prove that a recovery can preserve a live book.

Run scheduled exercises that include lost connectivity, a failed site, stale replication, an unavailable risk service and an ambiguous order acknowledgement. Test during a controlled trading window, with small or simulated exposure, and include venue-specific session recovery where the venue permits it.

Capture timings for detection, fencing, promotion, session recovery, reconciliation and controlled reopening. A practical exercise should check:

• Whether alerts reach the person authorised to freeze flow, including out of hours.
• Whether a promotion can be stopped when replication lag exceeds the agreed threshold.
• Whether order replay creates duplicates under retry and timeout conditions.
• Whether limits, credentials, certificates and venue contacts are current at the secondary site.
• Whether the team can restore from backups if replication has copied corrupted or deleted state.

Track defects to closure and repeat the scenario after material changes to gateways, risk logic, venue connectivity or data stores. Review the playbook after every real incident, even if traffic never moved to the secondary. For further operational context, the Trading Technology site covers the systems and providers that sit behind broker and prop-firm infrastructure.

If this were my operation, I would first map every live order path and prove how to fence the primary at each venue. Then I would run a small, controlled failover and measure the time to reconcile the book, not just the time to start the secondary.

#recuperação de desastres para brokers #conciliação do estado das ordens #RTO do sistema de trading #trading failover #live trading book recovery #trading system RPO #high availability trading systems #trading infrastructure resilience