Platforms · Software · Providers · Markets
Devin Hartcastle
The recovery site is only useful if it can take over the book without losing track of orders already sent to market. That requirement drives the cost more than a second set of servers does. A viable plan must recover positions, risk limits, order state, market data and venue sessions in a controlled sequence, then prove that the recovered system can trade safely.
Set recovery targets around trading state
Recovery time objective (RTO) measures how long service can be unavailable. Recovery point objective (RPO) measures how much state the business can afford to lose. For a trading book, a generic database recovery target is not enough: an order accepted by the primary system but not confirmed by a venue creates an uncertainty that infrastructure alone cannot resolve.
Typical planning ranges vary by architecture and venue access. A warm standby may target an RTO of 5 to 30 minutes and an RPO of 1 to 10 seconds. A hot standby may target 30 seconds to 5 minutes, with an RPO from under a second to around 2 seconds. Active-active designs can target 1 to 30 seconds, but routing, session recovery and order reconciliation can take longer than service discovery. Treat these as design ranges, not guarantees.
Agree targets separately for order entry, risk checks, positions, market data and client reporting. A book might restore read access quickly while keeping order entry blocked until the primary is fenced off and venue order state is reconciled. That is safer than advertising a short RTO that counts a live but unverified system as recovered.
Budget for the whole recovery path
The figures below are typical planning ranges in US dollars for a small-to-mid-sized broker or prop firm deployment. They are not audited results.
Monthly costs include an indicative allocation for engineering and operations, but exclude market-data licences, exchange fees and major application rewrites. Those items are contract- and codebase-dependent, and can materially change the total.
| Architecture, typical market ranges | Recurring cost per month, typical range | Implementation cost, typical range | Typical RTO target | Typical RPO target |
|---|---|---|---|---|
| Warm standby, capacity started or scaled during recovery | $15,000 to $60,000 | $50,000 to $250,000 | 5 to 30 minutes | 1 to 10 seconds |
| Hot standby, core services running and data replicated | $30,000 to $115,000 | $150,000 to $750,000 | 30 seconds to 5 minutes | Under 1 second to 2 seconds |
| Active-active, traffic and state handled across sites | $60,000 to $200,000+ | $400,000 to $1.5 million+ | 1 to 30 seconds | Near-zero to 1 second |
The ranges cover compute and storage, replicated databases or event logs, network links, monitoring, deployment work and a share of operational labour. Costs rise with instrument count, peak order rate, retention, geographic separation and the number of venues that require independent connectivity. Active-active is not simply twice the server bill: it also needs more complex conflict handling, testing and operational coverage.
Market-data entitlement and venue connectivity are common budget gaps. Confirm whether backup-site feeds, sessions, cross-connects and disaster-recovery access are included in existing contracts. A technically ready site can still be unable to receive licensed data or connect to a venue.
Make failover safe for orders in flight
Replicate an append-only record of business events, not just a periodically copied snapshot. The recovery system needs a consistent sequence for order requests, acknowledgements, fills, cancels, risk decisions and position changes. Use idempotency keys and sequence numbers so replay does not turn one client instruction into two orders.
During a site failure, do not assume that a timeout means an order was rejected. The venue may have accepted it while the acknowledgement was lost.
Fence the primary to prevent it sending more traffic, restore the latest durable state, then reconcile open orders and executions using the venue's authoritative records or drop copy before enabling new orders. The exact sequence depends on the venue protocol and gateway design.
- Keep risk limits, account status and position state on the recovery path, with checks for stale or incomplete data.
- Define which site owns each venue session. Use fencing tokens or an equivalent control to prevent both sites acting as primary.
- Separate market-data recovery from order-entry recovery. Stale prices should block affected risk checks rather than silently pass as current.
- Record each failover decision and state transition with timestamps that operations can compare across systems.
Synchronous replication can reduce data loss, but adds latency and ties availability to the link. Typical inter-region round-trip latency can range from roughly 20 to 80 milliseconds; a metro link may be around 1 to 5 milliseconds.
Many trading systems replicate critical events asynchronously and use reconciliation controls, rather than adding a wide-area round trip to every order. The right choice depends on the loss tolerance for each state type.
Test the service, not just the servers
A successful infrastructure switchover does not prove that a trading book has recovered. Exercise failures in the order gateway, database, network path, market-data handler and primary site. Include a venue disconnect during an in-flight order, because clean shutdown tests miss the ambiguity that creates duplicate execution risk.
A useful drill has a known starting book, controlled fault injection, an agreed authority for declaring the primary fenced, and a reconciliation step before trading resumes. Measure RTO from the first detected failure to the first verified, risk-enabled order, not to the moment a process starts. Measure RPO against the event log and compare reconstructed positions with venue executions.
- Run a component-level recovery test monthly, including backups, credentials, certificates and configuration access.
- Run a full-site exercise at least annually, and after material changes to gateways, routing, databases or venue connectivity.
- Capture elapsed time, missing or duplicated events, manual interventions and any difference between internal and venue order state.
Assign an owner to each failure mode and record the evidence needed to declare recovery. A runbook that depends on one engineer's memory is a hidden operational cost. A controlled manual step can be acceptable, but its duration and decision authority need to be included in the RTO.
Choose the cheapest design that meets the book's risk
Warm standby often suits firms that can pause trading for several minutes and reconcile carefully before reopening. Hot standby is more appropriate where prolonged interruption has a direct client or risk cost, but it still needs deliberate session takeover. Active-active is justified only where the business needs continuous service and the application can manage state ownership, duplicate requests and split-brain conditions reliably.
Build the budget from components rather than a single resilience percentage. Price a second location, data replication, network and venue access, test environments, operational coverage and annual exercises.
Then model the cost of market-data entitlements and any vendor support needed during an incident. The wider platform and provider landscape changes the available options; Trading Technology's infrastructure coverage is one place to track those dependencies.
Keep the first recovery objective narrow: restore a verified view of the book, reconcile venue state, and enable order entry only when risk controls are current. Expand automation after drills show that the sequence works.
Run or inspect the next exercise and measure the seconds from fault injection to the first verified, risk-enabled order. The threshold to check is 60 seconds.
Frequently Asked Questions
What is a realistic RTO for a trading platform?
Typical planning targets range from 5 to 30 minutes for warm standby, 30 seconds to 5 minutes for hot standby, and 1 to 30 seconds for some active-active designs. Venue session recovery and order reconciliation can extend the time before safe trading resumes.
How much does disaster recovery for a trading platform cost?
Typical monthly planning ranges are $15,000 to $60,000 for warm standby, $30,000 to $115,000 for hot standby and $60,000 to $200,000 or more for active-active. Market-data licences, exchange fees and major application changes can add materially to these figures.
How should a broker handle orders in flight during failover?
Fence the failed primary so it cannot continue sending orders, then reconcile the recovered state against authoritative venue records or drop copy. Treat an order timeout as uncertain rather than rejected, and do not enable new order flow until duplicate-order risk is controlled.
How often should a trading firm test disaster recovery?
Run component-level recovery tests regularly, with monthly checks a common operational cadence, and conduct a full-site exercise at least annually. Repeat tests after significant changes to gateways, routing, databases or venue connectivity.
