Historical data storage: the cost of replaying a year

How much will it cost to replay a year of market data? Usually more than the archive bill suggests: storage is only one part of the bill, while decoding, sequencing, book reconstruction and validation consume compute and engineering time. Run a measured sample before choosing a retention tier or promising a replay deadline.

Define what a replay must reproduce

"Replay" can mean reading raw events into a backtest, rebuilding a full order book, or reproducing the path from venue feed to broker risk checks and order routing. These are different workloads. A raw-file scan can be several times faster than a faithful system replay, but it does not test the same components.

Write down the required fidelity

Choose one representative venue, instrument set and busy trading session. Record whether the replay must preserve exchange timestamps, packet order, sequence gaps, corrections, session boundaries and the production version of symbology. If the purpose is to test risk or routing, include those services and their configuration. Otherwise a fast replay may give a misleading capacity result.

• Set the event scope: trades, top-of-book quotes, depth updates, or full feed packets.
• Specify the required clock: exchange time, capture time, or both.
• Decide how to handle missing packets, retransmissions, late events and corporate actions.
• Define a passing check, such as matching event counts, end-of-day positions or book checksums.

Keep raw capture separate from normalised output where practical. Raw files let the team reprocess data after a parser or symbol-mapping change. Normalised records can be easier to query, but preserve the original event and sequence fields needed to explain differences. A short note on trading infrastructure operations is useful context for treating replay as a production workload, rather than just an archive test.

Measure the archive before pricing it

Start with actual bytes written during a representative day, including the busiest session. Multiply by trading days or calendar days according to the feed, then apply the measured compression ratio. Do not estimate from the number of symbols alone: quote rates, depth and venue message formats can change volume by an order of magnitude.

Build a per-feed inventory

The table gives typical market ranges for one venue, before replicas and indexes. Annual compressed payload assumes 250 trading days and compression between 2:1 and 5:1. It is a sizing guide, not a quote. Measure your own feed because active instruments and peak message rates vary widely.

These ranges exclude replicas, derived datasets, indexes and backups. A second copy doubles the stored payload before metadata and lifecycle overhead.

Check whether the storage figure includes the files needed to start a replay quickly. Deep archive can reduce the monthly charge, but retrieval fees, restore delays and request limits may make it unsuitable for routine testing.

For each feed, capture these values from the pipeline:

• Raw bytes and compressed bytes by hour, including the peak hour.
• File count and average file size. Tiny objects can increase request and listing overhead.
• Compression CPU time and the ratio by format, not just the overall average.
• Retention days for raw, normalised and derived data, plus replica count.

Calculate storage as payload multiplied by copies, then add indexes and a reserve for growth. A 10-20% reserve is a reasonable planning range for changing volume and operational headroom, but keep it separate from measured consumption.

Price the replay as a compute job

Benchmark from the stored files, not a memory-resident sample. A warm-cache test can hide object reads, decompression and network limits.

Run the same interval once from the intended storage tier and again with the full replay path enabled. Record elapsed time, CPU utilisation, bytes read, events processed per second, peak memory and any queue growth.

Use a simple first-pass calculation:

• Wall-clock days: 365 divided by measured replay speedup. At 20 times real time, a year takes about 18.3 days on one worker.
• Worker hours: elapsed hours multiplied by worker count, including idle time while partitions wait on dependencies.
• Run cost: worker hours multiplied by the applicable hourly rate, plus storage reads, retrieval, network transfer and any temporary copies.

Parallelising by symbol or venue can reduce elapsed time, but it does not always reduce total compute. Cross-asset risk, shared order books and common event ordering can constrain partitioning.

Test scaling at one, two and four workers. If throughput barely improves, identify whether the limit is CPU decoding, storage bandwidth, a shared queue, or serial book state before adding machines.

Count engineering work as well. A replay that finishes quickly but needs manual repair after a sequence gap is not repeatable.

Include time to restore data, validate outputs, investigate failed partitions and rerun. For firms comparing platform and data choices, Trading Technology covers the operational systems around those decisions, but the cost model should come from measured workloads.

Run the checklist this week

Use five working days to produce a baseline that operations and engineering can both review. Keep the test narrow enough to finish, but choose a busy session and a feed that exercises the production path.

1. Day one: agree replay fidelity, success checks and the owner for each service in scope.
2. Day two: collect a full day of raw and compressed bytes, file counts and peak-hour rates for each feed class.
3. Day three: replay a one-hour sample from the intended storage tier. Measure throughput, CPU, memory, reads and event counts.
4. Day four: run a larger interval with one, two and four workers. Check output equivalence and note where scaling stops.
5. Day five: calculate annual storage, replicas, worker hours and retrieval costs. Record assumptions, the measured range and the next capacity trigger.

Keep the result as a small runbook: dataset location, software versions, command or job definition, expected duration, checksums and failure recovery steps. Repeat the benchmark after changes to feed handlers, compression, storage class or book-building code. A replay number without its configuration is not a useful capacity baseline.

The mistake most readers will still make is pricing historical data storage from compressed archive size alone, while ignoring the compute, reads and validation needed to replay it.

#reprodução de dados de mercado #trading data storage costs #historical tick data #order book replay #market data retention #replay benchmarking #object storage for trading