Historical data storage: the cost of replaying a year

In week one, a new market-data archive looks like a capture problem: land the feed, write it durably and check that sequence numbers have no gaps. By week six, the first full replay usually exposes the harder questions: which copy is usable, how quickly it can be read, and whether the test is competing with live workloads. The cost of replaying a year is set by decisions made during ingestion, not just by the replay command.

Size the retained dataset before pricing it

Start with the feed you actually consume. A top-of-book quote stream for a defined FX universe has a different footprint from full depth for hundreds of futures contracts. Venue count, message rate, depth, session hours and compression all change the result. Measure bytes written after compression over a representative trading week, then account for holidays, volatile sessions and the symbols you plan to add.

For planning, separate the raw feed from the replayable representation. Raw packets or vendor files preserve evidence for parser changes and disputes.

Normalised events make research and strategy tests easier to run. Replay-ready partitions, often organised by date and instrument, avoid repeatedly parsing a large raw archive. These are useful layers, but keeping every layer on fast storage can multiply the bill.

Typical compressed annual volumes vary widely by universe and depth. The ranges below are planning benchmarks, not measurements from a particular broker. They assume a defined active universe, one year of trading sessions and one retained copy.

Benchmark the storage and replay bill

Storage figures use a typical hot object-store range of $15-$30 per TB-month. Cold tiers commonly fall around $1-$6 per TB-month, before retrieval charges or delays. SSD-backed storage can run roughly $80-$200 per TB-month. Prices vary by provider, region, redundancy and minimum retention rules, so treat these as market ranges rather than a quote.

Consider a planning case with 40 TB of compressed data for a year. If the operation keeps raw, normalised and replay-ready copies, that is 120 TB at steady state. Three hot copies cost about $1,800-$3,600 a month, or $21,600-$43,200 for a full year at that retained size.

During the first year, a steadily growing archive averages about half its final size, so the storage bill is lower. Replication, snapshots and indexes add further capacity.

Break down the cost of a replay

Storage is usually the recurring line item. Replay compute can be modest for a single run, but data movement and repeated testing change the economics.

For the 40 TB example, reading at a sustained 1-3 GB/s gives a theoretical scan time of roughly four to twelve hours. Parsing, ordering, decompression and strategy logic extend wall-clock time. A typical replay might use 32-128 vCPUs and take 8-24 hours, with compute charges ranging from tens to several hundred dollars, depending on instance type and cloud rates.

A basic cost model should include these items:

• Stored copies: multiply retained terabytes by the rate for each storage tier. Include replicas, snapshots and derived data rather than pricing only the source feed.
• Reads and retrieval: standard object storage often has no per-gigabyte retrieval fee, while archive tiers can charge for restoration and impose delays.
• Compute: count worker hours, memory-heavy instances and any coordination services. Re-run the estimate for a single replay and for the monthly research schedule.
• Network transfer: same-region reads may be inexpensive, but cross-region or internet egress can commonly range from $0.02-$0.09 per GB. Moving 40 TB can therefore add roughly $800-$3,600 for one transfer.
• Operations: budget for monitoring, checksum validation, schema changes and restore tests. These costs are rarely visible in a storage calculator.

For storage architecture and market infrastructure coverage, Trading Technology reports on broker infrastructure. The same operational principle applies to archives: the cheapest byte is of little use if its format or location prevents a timely test.

Make the replay predictable

A replay benchmark should measure more than aggregate throughput. Record time to first event, sustained events per second, CPU utilisation, decompression cost and the time spent waiting on object reads.

Run the test with realistic partitions and worker counts. A single large file can benchmark well in isolation but create bottlenecks when many strategy jobs request the same date range.

Partitioning by trading date and instrument gives workers smaller, independent units. Keep metadata compact, avoid millions of tiny objects, and check that a partition boundary does not split state needed by the strategy. For deterministic results, preserve sequence numbers and timestamps, document correction handling, and pin the normalisation version used to create each replay set.

Before moving older data to a colder tier, check whether the replay system can restore it in time. Retrieval that takes hours may be acceptable for a quarterly validation run, but not for a desk testing a release before market open. A tiering policy can keep recent sessions and frequently used symbols hot while moving older, rarely accessed data to lower-cost storage.

Useful controls include:

• Set a target replay time for each test class, then size storage and workers against that service level.
• Compare compressed bytes per event and replay throughput after every feed or schema change.
• Test restoration and checksum verification on a schedule, not only after an incident.
• Track duplicate data and abandoned derived partitions, with a clear retention owner.

Keep evidence without keeping every copy hot

Retention policy should distinguish regulatory or dispute evidence from operational replay requirements. Raw vendor data may need a defined retention period, while replay-ready data can be regenerated if the source and transformation code remain available. The trade-off is practical: regeneration consumes engineer and compute time, while retaining another full copy consumes storage every month.

Many teams start with hot storage for recent data and the current research window, then use colder tiers for older raw files. A common planning pattern is 30-90 days on fast storage and longer retention in object or archive tiers, but the right window depends on strategy validation, incident response and applicable recordkeeping rules. Test a representative restore before setting the policy. Also model a busy week of concurrent replays, since a year that scans quickly for one user may queue badly for twenty.

Historical data storage is an infrastructure budget with a replay deadline attached. The industry still tends to price the bytes first and discover the deadline later.

#historical market data #market data compression #market data storage costs #tick data storage #market data replay #trading data archive #cloud object storage #backtesting infrastructure