Trading book disaster recovery fails at the state boundary

Disaster recovery for a live trading book is the controlled restoration of order, risk and accounting services after a site or dependency failure.

That definition hides the difficult part: a service can be healthy again while its view of the book is wrong. A recovered order gateway may not know whether a venue accepted the last new order or cancel. A risk process may have stale positions. If the primary and recovery sites both accept traffic, each can make valid decisions against a different version of reality.

Many plans treat recovery as a server restart and quote a single recovery time objective (RTO). A live book needs separate targets for each function, plus a rule for deciding which copy of state is authoritative. Without that, a quick failover can turn a short outage into duplicate orders, unbounded exposure or a reconciliation shift that runs into the next session.

Availability is not book recovery

Infrastructure monitoring usually reports whether a process responds, a database accepts connections or a network path is available. These are useful signals, but none confirms that the trading state is safe to use. A green dashboard can conceal a halted market-data feed, a risk service replaying old events or an order queue that has been acknowledged but not durably stored.

The first common mistake is to define recovery as moving traffic to a second site. The second is to replicate everything synchronously and assume that makes the copy correct. Synchronous replication can preserve committed writes, but adds latency and may spread a bad state or operational error to both locations. Asynchronous replication reduces that dependency, but creates a recovery point objective (RPO): the amount of recent data that may be missing after an incident.

State also has different owners. The broker's order manager records intent; the venue records what it accepted; the clearing or custody system may later confirm execution and settlement.

During a network partition, those records can disagree for legitimate reasons. Recovery must retain the uncertainty rather than invent certainty. An order with an unknown outcome should not be replayed as a new order simply because the local database lacks an acknowledgement.

Map the book's dependencies before setting targets. Include order entry, risk checks, pricing, market data, FIX connectivity, reference data, trade capture, position services, client authentication and the operational controls used to disable trading. The infrastructure boundaries described in Trading Technology's trading infrastructure coverage are a useful reminder that broker-operated and vendor-operated components often meet at the point where recovery gets complicated.

Failover must preserve one authoritative writer

Fence the failed site before promotion

Automatic promotion is dangerous if the old primary can still reach a venue or database. A heartbeat can fail because the link is broken, not because the server has stopped. If both sites continue accepting orders, the system has split brain: two writers each believe they control the same book.

Use a quorum or independent witness to decide ownership, then enforce fencing at the resource that matters. That may mean revoking database leases, disabling the old site's network route, terminating its venue sessions or using a hardware or cloud control that prevents it from writing. A promotion script should fail closed if it cannot prove the old writer is fenced. Do not use operator judgement as the only split-brain control during a noisy incident.

Set a defined sequence for promotion, such as:

1. Declare the primary unavailable using more than one independent signal.
2. Fence its access to order stores and external trading connections.
3. Check replication position and identify the last durable event.
4. Promote the standby, restore risk controls, then open client order flow in stages.
5. Reconcile venue state before replaying or resubmitting any uncertain order.

Reconcile the venue, not just the database

FIX sequence numbers help restore a session, but they do not settle every order's outcome. A session reset, counterparty configuration or gap-fill can leave the local system without a complete application-level history. Request open orders and executions where the venue supports it, compare them with the broker's durable event log, and classify each discrepancy before allowing automation to act.

Cancellation is not a universal safety net. A cancel-on-disconnect instruction may be unavailable, may apply only to specified order types or may not cover a venue-side session that remains live. Keep a venue-by-venue matrix of disconnect behaviour, order persistence, session reset rules and support contacts. Rehearse the actual production configuration, including certificates and network routes, rather than a simplified test connection.

Set targets by workload and failure mode

One RTO for the whole platform is usually either too ambitious for back-office services or too slow for order entry. The figures below are typical market ranges for planning, not guarantees.

Actual targets depend on architecture, venue capabilities, replication mode and the firm's risk appetite. RTO is time to restore a usable service; RPO is the potential loss of recent state.

Targets need explicit degraded modes. If the risk service is unavailable, the choices may be to reject new exposure, allow risk-reducing orders only, or hold all flow.

The permitted mode should be configured and tested, not improvised by an operator. Likewise, market-data recovery should mark prices stale until sequence gaps are repaired or a fresh snapshot is received.

Active-active designs can reduce interruption, but they require strict partitioning or coordination for shared state, including client order identifiers, limits and position updates. For many firms, a warm standby with tested promotion is simpler to reason about.

It may accept several minutes of interruption in exchange for fewer cross-site write conflicts. The trade-off should be made per service rather than as a platform-wide preference.

Exercise the failure that the runbook avoids

Most exercises prove that a standby can start. Fewer prove that the firm can safely resume trading. Test a lost primary link while the primary process remains alive, a failed replication stream, a venue disconnect during an unacknowledged order, and a recovery site with stale reference data. Include a rollback decision: once the standby has accepted orders, returning to the old site is another state migration, not a simple traffic switch.

Record timings at each control point: detection, fencing, promotion, risk validation, venue reconciliation and permission to resume. A typical test should capture:

• Which events were acknowledged to clients and durably recorded.
• Whether any order had an unknown venue outcome, and how it was resolved.
• Position, cash and exposure differences before and after recovery.
• Which manual approvals or vendor actions delayed restoration.
• Evidence that the former primary could no longer write or send orders.

Repeat tests after material changes to database topology, venue connectivity, risk logic or cloud-region design. A tabletop exercise can expose unclear ownership, but it cannot validate replication lag, fencing behaviour or FIX recovery.

Keep the evidence with incident procedures and review unresolved gaps with the teams responsible for each dependency. For wider operational context, Trading Technology's broker technology reporting covers the systems and providers that sit across those boundaries.

This week, run one controlled failover with a live-like order in an approved test environment, and require the team to prove venue reconciliation and fencing before reopening flow.

#recuperação de desastres para brokers #resiliência da plataforma de trading #recuperação de sessão FIX #failover do sistema de trading #recuperação do sistema de gerenciamento de ordens #RTO and RPO trading #trading risk controls #high availability trading