Historical data storage: the real cost of replaying a year

Server storage drives used for historical market data storage and replay
Photo by Marta Branco on Pexels

In nearly every trading setup, a year of market history looks like a storage problem until someone tries to replay it. The compressed files fit on disk, but a strategy test, incident review or migration check takes days because parsing, decompression and event handling cannot keep pace with the read path. Storage cost and replay cost are separate budgets. A useful estimate needs both.

Define what the archive contains

"Historical data" can mean exchange market-by-order messages, top-of-book updates, broker prices, order and execution events, or derived bars. These are not interchangeable. A bar archive is compact and quick to query, but cannot reproduce queue changes or the exact sequence of quotes and trades. A tick archive preserves more detail, at a higher storage and processing cost.

Record the data contract before sizing anything: venues and instruments, depth level, message types, timestamps, sequence numbers, corrections, and session calendar. Keep raw vendor or exchange messages where licensing permits, then store normalised records or derived views separately. If only the normalised form survives, a parser change can make old files difficult to interpret or rebuild.

Also separate market data from internal event logs. Order-state transitions, risk decisions, rejects and execution reports may be smaller than a busy depth feed, but they are essential for reconstructing what the system knew and when. Check retention, access controls and deletion rules for each class independently.

Estimate annual volume and retained copies

The figures below are typical market ranges, not capacity guarantees. They assume compressed data, one year of retention and the stated scope. Actual volume depends on venue activity, symbol selection, depth, encoding and whether snapshots or redundant channels are kept.

Workload and indicative scopeTypical compressed annual volumeCapacity with two retained copies
One-minute bars, 500 symbols2-20 GB4-40 GB
Top of book, 100 liquid instruments0.1-1.5 TB0.2-3 TB
Full depth, 10 liquid futures contracts1-15 TB2-30 TB
Order lifecycle and audit events, 10,000 active accounts0.2-3 TB0.4-6 TB

Use measured bytes per session, not a single peak day multiplied by 252. Capture at least 20 representative sessions, including volatile days and contract rolls.

Multiply the average daily compressed volume by actual trading days, then add snapshots, indexes, schema versions and any non-trading-hour events. Keep a separate high-water estimate for capacity planning.

Replication changes the bill. Typical hot object storage rates are roughly $15-30 per TB per month, while archive tiers may be around $1-5, before retrieval, requests and network charges.

At 12 TB of logical annual data and two copies, 24 TB at hot rates implies about $360-720 a month at full capacity. If data arrives evenly through the year, the first-year average stored volume is lower; the year-end run rate is the safer figure for a full archive. NVMe or block storage commonly costs more, but can support frequent replay.

Price the replay path, not just the disk

Replay time depends on event rate and processing, not only file size. A useful first calculation is stored bytes divided by sustained read bandwidth. For 12 TB, reading the compressed archive in 24 hours requires about 139 MB/s of effective input.

That is only the storage floor. Decompression, decoding, timestamp ordering, strategy callbacks and writes to result stores all consume capacity.

Express the target as simulated market time per wall-clock time. Replaying a calendar year in one day means about 365 times real time.

In a week, it is about 52 times. A single feed reader may reach those rates while a realistic strategy, portfolio risk checks or multi-venue synchronisation does not. Parallelism helps only where the test can be partitioned without changing event order or shared state.

"A disk benchmark is not a replay benchmark. Measure the whole event path."

Measure at least these counters in the same run:

  • Compressed bytes read and decompressed bytes produced per second.
  • Messages decoded per second, split by message type and instrument.
  • CPU utilisation, memory pressure, queue depth and storage wait time.
  • Simulated time advanced per wall-clock minute, plus dropped or late events.
  • Result writes, checkpoint time and time spent waiting on downstream services.

Report median and 95th-percentile rates, not just the fastest run. A quiet session can hide a bottleneck that appears during a burst of quote updates or a large opening auction. For broader infrastructure reporting and platform context, see Trading Technology.

Build a repeatable benchmark

Benchmark the exact archive format and replay configuration used in production. Include the strategy or consumer code, risk rules, normalisation layer and output writes.

Run once from a warm cache and once from the intended cold-storage path. Record whether the test uses one host, a cluster or a shared storage service. Without those details, a quoted "replay speed" is hard to compare or reproduce.

  1. Choose a representative week with ordinary sessions and at least one high-volume session.
  2. Replay at 1x, 10x and the target rate, using identical code and configuration.
  3. Repeat with the expected number of concurrent jobs to expose contention.
  4. Capture event counts, bytes, CPU time, wall time, errors and output volume.
  5. Repeat after a software or schema change, and compare the same counters.

Use the observed rate to estimate the full year, then add operational allowance for warm-up, checkpoints and failed jobs. If a representative week contains 5 per cent of annual messages, its measured processing time can be scaled by message count, but validate the estimate against a second, busier sample. Include the cost of temporary working space: replay jobs may need decoded data, indexes and outputs alongside the archive.

Make the budget reflect usage

Keep frequently replayed data on a low-latency tier and move older, rarely accessed sessions to cheaper storage only after testing retrieval time and fees. A common design is a hot recent window, an object-store archive and a second copy in a separate failure domain. Do not count replicas as independent protection if they share credentials, a region or a deletion path.

Track four budgets independently: retained bytes, retrieval and egress, compute hours, and operator time. Set a cost per replayed session and per full-year replay, then compare it with actual usage. If a full-year test runs twice a month, optimising its parser may save more than shifting the archive to a tier that adds several hours of retrieval delay. If it runs once a year, the cheaper tier may be appropriate.

Review the measurements after adding venues, increasing depth or changing compression. The mistake most readers will still make after reading this is budgeting from compressed bytes alone, rather than measured replay throughput and retained copies.

Frequently Asked Questions

How much storage does one year of tick data need?

It depends on instruments, venue activity, depth and encoding. Typical compressed ranges can run from about 0.1-1.5 TB for top-of-book data on 100 liquid instruments to 1-15 TB for full depth on 10 liquid futures contracts.

How do I calculate the cost of replaying a year of market data?

Measure the bytes and messages processed per second with the production replay path, including decompression, strategy logic and output writes. Divide the full-year message count by the measured sustained rate, then add compute, storage retrieval, egress and temporary working-space costs.

What does replaying a year at 100x mean?

It means the replay advances through simulated market time 100 times faster than real time. A calendar year of data would take about 3.65 days at 100x, before startup, checkpointing or other overhead.

Is object storage suitable for historical market data?

It can suit archives that are accessed infrequently, provided retrieval latency, request charges and egress costs fit the use case. Benchmark a real replay from the chosen tier before moving data that operations or research teams need to access quickly.

Should market data and order audit logs share a retention policy?

Not automatically. Market data, order-state events and execution records have different replay, compliance and access requirements. Define retention and deletion rules for each class, while preserving the timestamps and identifiers needed to correlate them.