Trading platform load testing: a checklist for launch week

A 2x underestimate of peak order flow can cost a broker its launch window: queues grow, order acknowledgements miss targets and operations staff begin reconciling state by hand. Load testing before release is how the team finds out whether the trading stack can absorb the launch workload, not just whether a single API endpoint responds quickly.

The work is practical, but it needs boundaries. This checklist is designed to run this week, using a production-like environment, a recorded workload and pass limits agreed before the first test. It covers the path from client request through risk, execution and reporting, with recovery included rather than left for a later exercise.

Set the workload and the limits first

Start with the product launch forecast, then test above it. Separate expected average activity from short bursts. A platform that handles 500 order requests a second on average may still struggle with a 1,500-request burst if those requests hit one account, symbol or risk partition at the same time.

Use figures from existing products where available, but include a reasonable launch-growth margin. Typical planning ranges are 1.5x to 3x forecast peak throughput and 2x to 5x the expected number of concurrent sessions.

Model the full transaction mix, not a wall of identical order submissions. Include logins and session refreshes, market data subscriptions, new and amended orders, cancels, fills, balance and position reads, and reconnects after a short network interruption. For a broker or prop firm, the risk engine can be the limiting service even when the gateway and matching or execution connection appear healthy. Include rejected orders and risk checks in the workload, since they still use compute and database capacity.

Write down pass limits before testing. Use service-level objectives that match the product and venue, rather than choosing a threshold after seeing results. A useful starting set includes:

• Order acknowledgement latency at p50, p95 and p99, measured from client submission to the response the client actually receives.
• Order and cancel throughput, with a separate count of accepted, rejected, timed-out and duplicated requests.
• Market data delivery delay and dropped or out-of-sequence updates, measured at the application boundary.
• Risk decision latency, queue depth, database connection use and message broker lag.
• Recovery time after a service restart, network loss or dependency slowdown, including the time until positions and order state reconcile.

Set a hard stop for correctness defects. Any duplicate execution, lost accepted order, inconsistent position or unauthorised order acceptance fails the test regardless of average latency. Readers following Trading Technology's infrastructure coverage will recognise the distinction: capacity numbers matter, but only alongside controls and state integrity.

Build a production-like test rig

Use the same software versions, configuration, instance sizes, network paths and data stores planned for launch. If the production design uses separate gateways by region, test each route. A smaller environment can still reveal application bottlenecks, but it cannot establish production capacity by simple multiplication. Shared database locks, cache hit rates, noisy neighbours and network limits do not always scale linearly.

Use synthetic accounts and instruments with realistic distributions. A thousand virtual users all submitting orders to one liquid symbol is a useful hot-key test, but it is not a representative launch profile. Split traffic across common symbols and account sizes, then add a concentrated burst to expose partitions or shared risk state. Use a market data feed with changing prices and realistic update rates; a static fixture can conceal downstream fan-out and serialisation costs.

Make the load generator part of the test design. Run it from a separate host or cluster, confirm it has spare CPU and network capacity, and monitor it as carefully as the platform. If the generator saturates first, the reported maximum is not the platform's limit. Keep clocks synchronised across clients and services, and record timestamps at request creation, gateway receipt, risk decision, venue hand-off and client response. This lets the team distinguish application latency from network and external execution delay.

Prepare a repeatable workload file or script with a fixed seed. Record account count, session ramp, order mix, symbols, request rates and any injected faults. Keep test data isolated from production accounts and venue routes. For external venues, use a certified simulator or a controlled test environment, and make the limits on outbound requests explicit. The test should never send uncontrolled traffic to a live market.

Before the run, check that logging, tracing and metrics are enabled at the level needed for diagnosis, without accidentally changing performance through excessive debug logging. Retain raw latency samples, not just dashboard averages. Averages hide tail behaviour, and a single p99 number can also hide a brief but damaging spike during a ramp or failover.

Run the test in controlled stages

Do not begin with the maximum load. A staged run helps separate warm-up effects from a genuine capacity ceiling and gives the team time to stop before queues become unrecoverable.

Keep each stage long enough to expose memory growth, connection leaks and delayed persistence. For most launch checks, a stable period of 15 to 30 minutes per load level is a useful starting point; longer soak runs are needed where the service is expected to stay busy for hours.

Use this run order and record the start time, workload version and result for each step:

1. Baseline: Run a small, representative load for 10 to 15 minutes. Confirm that timestamps, counters and tracing agree, and capture idle resource use.
2. Ramp: Increase sessions and requests in steps, typically 10 to 20 per cent every few minutes. Watch queue depth and tail latency, not just CPU.
3. Expected peak: Hold the forecast launch peak long enough to reach steady state. Check every pass limit and compare the client-visible response with internal service timings.
4. Surge: Test the agreed headroom, commonly 1.5x to 3x forecast peak. Keep the order mix representative, then run a separate hot-symbol or hot-account burst.
5. Soak: Hold a realistic elevated load for at least 60 to 120 minutes if time permits. Track memory, open connections, cache behaviour, database pools and queue trends.
6. Recovery: While load continues, restart one non-critical service instance or introduce a controlled dependency delay. Verify that traffic shifts as designed and that order state remains consistent.
7. Cool-down: Reduce load and confirm queues drain, resource use returns towards baseline and no background reconciliation task remains stuck.

Change one variable at a time during diagnostic runs. If a test combines a database failover, a burst of logins and a venue disconnect, the result may show a failure without identifying its cause. Once isolated tests pass, run a combined scenario that resembles the launch risk the operations team is most concerned about.

Do not call a run successful while accepted orders are still waiting in an unmeasured queue.

Read the numbers, then locate the bottleneck

Use the table as a set of typical starting ranges, not universal service-level commitments. Product type, venue connectivity, jurisdiction, order rules and the client application all affect suitable limits. The team should replace these ranges with its own signed-off objectives before launch.

Interpret metrics together. High gateway CPU with normal database use can point to serialisation, authentication or per-request logging. Low CPU alongside rising queue age often points to a blocked dependency, connection pool exhaustion or an undersized worker pool. A sudden rise in p99 with a stable p50 usually indicates contention, queuing or a noisy neighbour rather than a platform-wide slowdown.

Trace a small sample of slow requests across service boundaries. Look for repeated database calls, synchronous audit writes, lock contention, slow risk lookups and retry loops. Check whether retries amplify demand: if a downstream call times out and every client immediately retries, the retry traffic can exceed the original load. Add bounded retries with jitter where appropriate, and verify that idempotency keys prevent a repeated submission from creating a second order.

At each stage, save a short evidence pack: the workload definition, deployment version, metric snapshots, latency percentiles, error counts, queue age and operator notes. Include a list of known deviations from production. A run without a reproducible configuration is hard to compare after a code or infrastructure change.

Test failure, recovery and the launch gate

Capacity alone does not establish launch readiness. Define a small set of controlled faults that reflect the actual architecture and the consequences of losing each dependency. Start with one service instance, then move to a database or message broker fault only if the test environment is isolated and the recovery procedure is understood. Never improvise destructive tests against a shared environment.

For each fault, check both the technical response and the operator workflow. Does health checking remove the failed instance? Do clients receive a clear timeout or a misleading success? Are in-flight orders queried or replayed safely? Can the operations team identify the affected accounts and reconcile them against the system of record? A failover that restores throughput but leaves order state uncertain is not a pass.

Before signing off, confirm the following with engineering, operations and the product owner:

• The agreed peak and surge workloads completed without a correctness defect.
• Latency, error rates and queue age stayed within the limits set before testing, or have a named owner and documented exception.
• Autoscaling, traffic shifting and connection limits behaved as configured, including their cooldown periods.
• Monitoring alerts fired early enough for an operator to act, and the runbook identifies the service owner and escalation path.
• Rollback criteria are explicit. Examples include sustained p99 breach, growing queue age, elevated order rejects or any mismatch in positions and accepted orders.
• Test evidence and unresolved defects are attached to the release decision, with a retest date for each material fix.

Keep the launch gate proportional to the product. A new symbol on an established stack may need a focused hot-key and market data test.

A new execution route, risk engine or account model warrants a broader replay and fault exercise. For broader context on broker infrastructure and platform operations, see our reporting on trading technology.

Book a 90-minute launch-workload review this week, and leave with the replay file, pass limits and named rollback owner.

#prop firm infrastructure #failover testing #trading platform performance #broker platform testing #order flow load testing #trading system capacity #latency testing #market data stress testing