In week one, a new archive can look finished: yesterday's files land, checksums pass and the storage bill is small. By week six, an engineer may discover that restoring a month takes longer than the test window, or that a full-year replay saturates CPUs before storage. The gap is the difference between keeping data and being able to use it.
For a broker or prop firm, the design question is not just how many terabytes to retain. It is how quickly the system can read, decode, order and deliver the records needed for a backtest, incident review or feed migration.
Start with the replay deadline, not the storage bill
Define the job first. Replaying one calendar year of exchange sessions in 24 hours means processing roughly 1,600 market hours in a day, or about 67 times the live session rate.
A 30-day replay in four hours needs a similar multiplier. These are workload targets, not just network targets: parsing, decompression, symbol mapping and strategy callbacks all have to keep pace.
Decide what "replay" means for each use case. A compliance review may need a small instrument set and exact event ordering. Strategy research may need broad coverage, adjusted reference data and repeatable seeds. Recovery testing may require the same feed handlers and downstream queues used in production. Each has a different read pattern and validation cost.
- Set a completion target, such as one trading year in 24 hours, and record whether it applies to one instrument, one venue or the whole universe.
- Specify fidelity: raw packets, normalised events, order-book updates, trades, or a combination.
- Include validation time. A replay that finishes quickly but cannot prove sequence integrity is not operationally useful.
Track replay throughput in events per second as well as gigabytes per second. File throughput can look healthy while a single-threaded decoder or a serial book builder holds the job back. A useful benchmark reports both the fastest sustained rate and the slowest partition, since a few busy sessions can dominate elapsed time.
Size the archive and its access tiers
Estimate annual volume from feed shape
Build the estimate from message rate, average encoded size, trading hours and retained instruments. Then add compression, indexes, manifests and replicas. For planning, compressed annual footprints often fall in the ranges below for a single venue or a bounded instrument universe. Actual volume varies sharply with message rates, depth, schema and compression settings.
These are typical market planning ranges, not quotes from a provider. Hot object storage commonly costs about $15 to $30 per TB per month; archive tiers may be around $1 to $5 per TB per month, before retrieval, requests and replication. Region, minimum retention periods and network charges change the total.
| Typical workload | Compressed footprint per year | Read rate for 24-hour replay | Hot storage per month |
|---|---|---|---|
| Top of book and trades | 5 to 20 TB | 60 to 230 MB/s | $75 to $600 |
| Depth updates, selected venues | 20 to 100 TB | 230 MB/s to 1.2 GB/s | $300 to $3,000 |
| Broad depth and high-volume feeds | 50 to 250 TB | 0.6 to 2.9 GB/s | $750 to $7,500 |
The read-rate column is the average payload rate needed to scan the stated footprint in 24 hours. It does not account for decode overhead, parallelism limits or replaying multiple copies. For a three-year hot retention policy, multiply the annual footprint and monthly storage estimate by three, then include replicas separately.
Keep frequently read data close
Use a hot tier for recent sessions and the slice most often used in research. Keep older data in lower-cost object storage if retrieval times fit the job.
Archive retrieval can take minutes to hours and may carry per-request or retrieval charges. That is suitable for an occasional audit, but a poor fit for a daily replay queue.
Partition by date and venue, with files sized for parallel reads rather than one object per symbol and minute. Typical file targets of 256 MB to 1 GB balance request overhead against worker-level parallelism.
Store schema version, instrument mapping version, event counts and checksums in a manifest. For storage design and platform operations coverage, see Trading Technology.
Make replay predictable under load
Separate I/O from decoding
Benchmark the pipeline in stages: object read, decompression, parsing, ordering and consumer processing. Run one test with data already on local NVMe, then one from the intended object tier. If the local test is slow, adding storage bandwidth will not fix the decoder. If it is fast locally but slow remotely, check request size, concurrent reads, region placement and network paths.
Compression is a trade-off, not a free reduction in cost. Columnar or block compression can cut stored volume substantially, but the best codec depends on whether replay is CPU-bound or I/O-bound. Typical deployments compare a fast codec with a denser option using the same sessions and report compressed size, decode CPU and end-to-end events per second. Avoid choosing on compression ratio alone.
Preserve determinism and test the tail
Parallel replay needs explicit ordering rules. Partitioning by instrument can improve throughput, but cross-instrument strategies may depend on timestamp ordering across feeds.
Record sequence gaps, late messages and clock normalisation decisions. A replay should identify the exact source files and reference-data versions used, so another run can reproduce the same inputs.
Test ordinary and peak sessions, not just an average day. A sensible operational target is to provision 30% to 50% headroom above the measured sustained rate, then rerun after feed or schema changes. Also test concurrent users: several research jobs can turn an adequate archive into a queue of slow reads. Platform teams tracking feed handling and related infrastructure changes can use the publication's trading technology coverage as one source of industry context.
Turn the model into a runbook
Keep storage, replay and operating costs visible as separate lines. A one-year replay may read tens or hundreds of terabytes, consume compute for hours, and incur network or retrieval charges if data crosses regions or tiers. Calculate both the cost of one run and the monthly cost of expected runs. A low archive bill can be outweighed by repeated restores, idle capacity or engineering time spent managing failed jobs.
Use a small runbook with named owners and thresholds:
- Measure daily ingest by venue, message class and compressed size for at least two representative weeks.
- Run a one-day and one-month replay against the intended storage tier; record elapsed time, events per second, CPU, network and cost.
- Set alerts for missing partitions, checksum failures, ingest growth and replay duration. Keep a tested restore path, not only a retention policy.
- Re-run the benchmark after decoder, schema, codec or instance changes, and before committing to a longer retention period.
Review the design quarterly against actual replay demand. If older data is rarely used, move it down a tier and test retrieval before the next incident. If research routinely needs it, a small hot working set or a prebuilt local cache may cost less than repeatedly restoring the same history.
If this were my operation, I would measure a representative month end to end before changing tiers or buying capacity. I would then set the replay deadline, provision for the measured bottleneck with headroom, and make the monthly cost of a full-year run visible beside the archive bill.
Frequently Asked Questions
How much storage does one year of market data need?
It depends on message rates, depth, venues and compression. Typical planning ranges run from about 5 to 20 TB for top-of-book and trades, and can reach 50 to 250 TB for broad, high-volume depth feeds.
How fast should a trading data replay run?
Set the target from the job deadline and the market hours being replayed. A year of roughly 1,600 trading-session hours processed in 24 hours needs about 67 times session-time speed, before allowing for validation and overhead.
Is archive storage suitable for market data replay?
It can work for occasional audits if retrieval latency and charges fit the job. Frequent research replays usually need a hot tier or a tested cache, because archive retrieval can take minutes to hours.
What is the main bottleneck in historical data replay?
It may be storage reads, but decoding, decompression, event ordering or strategy processing often limits throughput first. Benchmark each stage separately and compare local-disk runs with runs from the intended storage tier.
How should a broker test a full-year replay?
Start with one representative day, then test a month and a full year using the intended files, decoder and consumer path. Record elapsed time, events per second, CPU, network use, cost and validation results, including peak sessions.
Platforms · Software · Providers · Markets