Disaster recovery is the process of restoring a trading service after a disruptive failure.
For a live trading book, however, restored servers are not necessarily a recovered service. Orders may have reached a venue without an acknowledgement, risk state may lag executions, and a second site may be able to submit trades while the first is still active. Recovery has to restore a consistent, controlled book, not just application availability.
Define what recovery means for the book
Start by defining the service boundary. It may include client gateways, order management, risk checks, pricing, market data, venue sessions, position and cash records, and the operational controls that can stop trading. A recovery objective that covers only the order management database leaves important failure modes outside the plan.
Map state by its source of truth and its consequence if stale. A position can often be rebuilt from executions and adjustments. An order's state is harder: a request can be sent, accepted by a venue and then lost before the response reaches the broker. The local system cannot safely assume that the order was rejected or retry it as a new order. Treating this as a simple database restore can create duplicate exposure.
For each component, record whether state is authoritative, derived or disposable. Give each item a recovery limit, a reconciliation method and an owner. A useful inventory includes:
- Client order identifiers, venue order identifiers, sequence numbers and the last confirmed order state.
- Executions, allocations, corrections and busts, with their source and processing status.
- Positions, cash, margin usage and risk limits, including the timestamp and sequence of the last update.
- Venue sessions, market data channels, reference data versions and outstanding cancel or replace requests.
- Credentials, network routes, certificates, DNS and the permissions required to activate the recovery site.
Define the point at which new order entry stops, too. A failover that restores connectivity but cannot determine whether exposure is within limits is not a safe recovery. The system should fail closed for affected instruments or accounts until reconciliation reaches an agreed threshold.
Measure recovery as a set of limits
Recovery time objective (RTO) measures how long a service can remain unavailable before its operational limit is breached. Recovery point objective (RPO) measures how much state loss the firm can tolerate. Neither metric, by itself, shows whether the recovered book is safe to trade. Measure the time to restore service and the time to establish trusted state separately.
The ranges below are typical planning ranges, not audited results or universal targets. Actual limits depend on venue rules, architecture, market hours, data replication distance and the firm's risk appetite.
| Measure | Typical planning range | What to measure |
|---|---|---|
| Order and execution RPO | 0 to 1 second | Oldest unreplicated journal entry, plus reconciliation of orders with uncertain venue status. |
| Risk and position state RPO | 0 to 5 seconds | Age of the last risk-consistent position snapshot, checked against the execution journal. |
| Core service RTO | 30 seconds to 5 minutes | Time from incident declaration until the service accepts controlled production traffic. |
| Venue and market data recovery | 1 to 30 seconds | Session establishment, sequence recovery or gap replay, and confirmation that feed state is current. |
| Standby capacity | 50% to 100% of peak load | Sustainable order, market data and risk-check throughput at the recovery site. |
Report percentiles and worst observed results, not only averages. A median failover of 40 seconds can conceal a two-minute tail caused by certificate retrieval or a manual network change. Track detection time, decision time, service restoration, state reconciliation and permission to resume trading as separate intervals. This makes it possible to see whether delay comes from technology or from a necessary control.
Measure replication lag at the application journal and database layers. A database replica can show a low lag while an asynchronous consumer is behind, leaving positions or limits stale.
Alert on both lag and the age of the last processed event. For a practical view of how platform dependencies fit together, Trading Technology's infrastructure coverage can sit alongside internal service maps and vendor documentation.
Design failover around ambiguity and control
Active-passive designs reduce the chance of two sites trading at once, but the standby may have stale state or insufficient capacity. Active-active designs can shorten interruption and share load, but require clear ownership of orders, risk decisions and writes. If both sites can create or amend the same order, a network partition can turn availability into duplicate activity.
Use fencing to prevent the former primary from submitting orders after ownership moves. A lease or quorum decision should be enforced at the point of venue connectivity, not merely recorded in a control panel.
Where possible, isolate the old site at the network or credential layer. Operators need evidence that the old writer is fenced before enabling the new one.
Build the cutover sequence around explicit gates:
- Declare the incident and identify the affected failure domain. Confirm whether the primary is degraded, isolated or still capable of sending traffic.
- Fence the old writer. Verify that its venue sessions and outbound order routes are disabled, then record the fencing evidence.
- Promote the standby and restore dependencies in order, including configuration, secrets, reference data, market data and venue connectivity.
- Reconcile the order journal against venue drop copies, execution reports and open-order queries. Mark uncertain orders for controlled handling rather than resubmitting them blindly.
- Rebuild or verify positions and risk usage. Apply conservative limits, or disable affected symbols, until the recovered state passes agreed checks.
- Enable a restricted flow, monitor rejects, latency, sequence gaps and exposure, then raise limits in stages with an identified decision owner.
Not every venue supports the same recovery path. Session sequence numbers, resend behaviour, cancel-on-disconnect settings and order persistence vary.
Test the exact combination of venue, gateway and account configuration. A market-wide disconnect policy may cancel orders that a broker expected to remain working, while a reconnect can expose orders that were accepted before the failure.
Keep the kill switch independent of the application path it controls. Document who can invoke it, what scope it affects and how its action is verified. During recovery, rate limits on new orders and restricted instruments can limit the consequence of imperfect state without requiring a full service shutdown.
Test the measurements, not just the runbook
A successful exercise is not one in which a dashboard turns green. It demonstrates that the firm can detect a defined failure, prevent dual writers, reconstruct the book and resume within measured limits. Test failure modes that affect state as well as infrastructure: delayed replication, a lost acknowledgement, a venue session reset, stale reference data and loss of the primary site during a burst of executions.
Run tabletop exercises for decisions that cannot be automated, then schedule technical failover tests in a controlled environment. Where production testing is authorised, define scope, market conditions, venue contacts and rollback criteria in advance. Capture timestamps from monitoring, gateway logs, journal offsets and venue messages using synchronised clocks. Without a common time reference, incident reconstruction can confuse the order of events.
After each test, compare observed results with the agreed RTO, RPO and reconciliation thresholds. Assign owners and deadlines for gaps. Useful evidence includes:
- Maximum and percentile replication lag, including lag under peak write load.
- Time to detect, fence, promote, reconcile and permit trading, recorded separately.
- Count of orders or executions requiring manual investigation, and the time to resolve them.
- Throughput and latency at the standby, with risk checks enabled and realistic market data volume.
- Confirmation that access controls, alerts, audit records and kill switches work from the recovery site.
Repeat tests after material changes to gateways, databases, cloud regions, venue connectivity or risk services. Review the runbook with operations staff who will make the decision during an incident, not only with its authors.
Keep contacts, credentials procedures and vendor escalation paths current. Recovery documentation that relies on an unavailable administrator or expired certificate is not an operational control.
This advice does not apply unchanged where a venue, regulator or clearing arrangement mandates a specific recovery sequence or prohibits unilateral session failover. In those cases, the required external process takes precedence, and the recovery design must be built around it.
Frequently Asked Questions
What RPO should a live trading system target?
Typical planning ranges put order and execution state at 0 to 1 second, with risk and position state often targeted at 0 to 5 seconds. The target only has value if the firm can reconcile uncertain orders and executions before restoring trading.
How do you prevent duplicate orders during failover?
Fence the former primary so it cannot submit orders, then establish ownership at the venue connectivity layer before enabling the standby. Reconcile the order journal against venue reports and open-order queries, and do not blindly retry requests with uncertain outcomes.
How often should trading disaster recovery be tested?
Test after material changes to the trading stack, network, venue configuration or risk services, and schedule recurring exercises that cover both technical and operational decisions. The frequency should reflect the firm's risk and obligations, with measured evidence retained from each test.
Platforms · Software · Providers · Markets