Historical data storage makes a year of replay costly

Server racks representing historical data storage and a year of trading data replay
Photo by panumas nikhomkhai on Pexels

Historical market data is not cheap just because object storage is cheap.

For a broker, the stored files are only one part of the bill. A year of quote and trade events has to be captured, normalised, indexed and retained with enough context to reproduce what the platform saw.

Then someone has to run the data back through services that may include pricing, risk, order routing and execution. Those jobs have very different cost profiles.

Take a representative multi-asset broker processing about 120 GB of raw market events a day. That is a planning example, not an audited result for a particular firm. At roughly 1.2 billion records per day and an average record size near 100 bytes, it is a credible workload for a platform capturing frequent quote updates across FX, metals, indices and selected equities.

Start with what the platform must capture

Market-data archives often grow because teams retain more than the price and timestamp needed for a chart. A replay that needs to reproduce a decision may also require the source venue, instrument mapping, sequence number, bid and ask sizes, feed status, normalisation version and the clock used by the consuming service.

Separate the streams before estimating storage. A workable inventory usually distinguishes:

  • Raw feed messages, retained as received for feed investigations and parser changes.
  • Normalised quote and trade events, used for replay and market-data research.
  • Platform events such as order requests, risk decisions, rejects and fills, stored with correlation identifiers.
  • Configuration and reference data, including symbol definitions, trading sessions and price precision.

The final two groups are smaller than the quote archive in many systems, but they can be essential to making a replay meaningful. A price series without the instrument map or session calendar may look complete and still produce a different result from the live system.

For operational coverage of broker platforms and their dependencies, Trading Technology's infrastructure reporting is a useful place to keep alongside internal runbooks.

Put a number on the stored year

At 120 GB a day, 365 days of raw input totals about 44 TB. Compression of 2.5:1 brings the normalised archive to around 17.5 TB, depending on encoding, field repetition and whether the data is columnar. Keep a raw copy as well and the retained volume rises. Two-region replication doubles the billable footprint, before indexes, manifests and temporary replay outputs.

The table gives indicative typical market ranges for planning. Rates vary by provider, region, commitment, storage class and request volume. Treat them as a first-pass model, not a quote.

ItemTypical volume or rateTypical unit costPlanning implication
Raw multi-asset feed capture50-200 GB per dayAbout 18-73 TB per year before compressionMeasure at the message boundary, not from dashboard exports
Compressed object archiveCompression often 2:1-4:1US$0.015-US$0.025 per GB per monthOne 17.5 TB copy costs roughly US$260-US$440 per month at full retention
Regional replicaUsually another 1x stored volumeSimilar storage rate, plus transfer and requestsTwo 17.5 TB copies cost about US$525-US$875 per month at full retention
Hot replay cacheAbout 0.5-2 TB for recent or active partitionsUS$0.06-US$0.12 per GB per monthBudget roughly US$30-US$240 per month, depending on size and tier
Full-year replay compute4-8 workers with 16 vCPUs eachAbout US$0.60-US$1.20 per worker-hourA 300-1,200 hour run can cost roughly US$1,000-US$12,000 before transfer

For the example, 17.5 TB in two regions is 35 TB. At a representative object rate of US$0.018 per GB-month, that is about US$630 a month once the full year is present.

A 1.4 TB hot cache at US$0.08 per GB-month adds about US$112. The full-retention run rate is therefore around US$740 a month, excluding requests, data transfer, indexes and any separate raw archive.

That is not the first-year cash cost. If retention builds gradually from zero, average stored volume during the first year is lower than the year-end balance. The distinction matters in budget forecasts, but not when sizing the system for the mature retention target.

Replay is a compute workload, not a download

Reading 17.5 TB at 1 GB per second takes under five hours in theory. That figure says little about a production replay. The job may need to decode records, preserve ordering across partitions, apply symbol and session rules, and feed each event through software that was designed for live traffic. Checksums, state resets and reproducible outputs add more work.

A full FX and CFD trading year represents about 6,000 market hours. If the replay uses a virtual clock and runs at 5-20 times recorded time, the elapsed runtime is roughly 300-1,200 hours, or 12-50 days.

Real systems vary. A tool that can process files quickly may still bottleneck on one ordered instrument partition or on downstream risk and order services.

Fast storage shortens the read. It does not make a serial replay parallel.

Before launching a year-long run, set out the execution path:

  1. Pin the data and software versions, including instrument maps, normaliser builds and trading calendars.
  2. Partition by instrument or another safe key, while preserving event order within each partition.
  3. Choose whether service clocks advance in real time or under a controlled virtual clock.
  4. Record throughput, lag, CPU, memory and output checksums for a representative week before scaling to a year.

Parallelism has limits. Splitting by instrument is usually safer than splitting arbitrary time windows, but shared account state, cross-instrument risk and portfolio limits can force coordination. A replay designed to test market-data parsing may scale well. A replay intended to reproduce account-level decisions may not.

Choose storage tiers around the recovery objective

Keeping everything on high-performance block storage is simple and expensive. Keeping everything in a deep archive tier lowers storage charges but introduces retrieval delays and restore fees. A tiered design fits many broker workloads better:

  • Keep a short rolling window on fast local or block storage for routine investigations and recent replay.
  • Store the searchable year in object storage, with partitions and metadata that let jobs fetch only required instruments and dates.
  • Move older raw captures to a colder tier only if the restore time and retrieval charges fit the incident procedure.
  • Maintain a second copy where recovery requirements justify it, and test access to that copy rather than treating replication as proof of recoverability.

Agree the restore objective with operations and compliance. An archive that takes hours to retrieve may be acceptable for model research, but not for a live incident requiring an immediate comparison with recorded quotes. Likewise, a second region can protect against a regional failure, but replaying there may incur transfer charges or require duplicate compute capacity.

Teams tracking platform dependencies through Trading Technology should apply the same discipline to their own storage design: specify the retention period, restore target and replay use case, rather than buying a tier by its headline price.

Budget for the replay you will actually run

There are at least three different jobs commonly described as replay. A data-quality check scans files and validates sequence and schema.

A strategy backtest reads selected instruments and events. A platform replay pushes the year through stateful services to test decisions and operational behaviour. They need different compute, output retention and levels of determinism.

Keep separate estimates for:

  • Stored bytes, including replicas, indexes, raw records and the growth in retained history.
  • Read and transfer charges, especially when data crosses regions or leaves a provider network.
  • Worker-hours for each replay type, based on a measured week and a stated acceleration target.
  • Temporary outputs, snapshots and logs, with an expiry rule after each run.
  • Engineering and operations time for validation, failed runs, version control and reruns.

Use a small replay to test the cost model. Capture a representative week, run it through the intended path, and extrapolate by event count and measured throughput, not by compressed file size alone. Include a peak day: average daily volume can hide open and close bursts that determine whether the pipeline falls behind. For broader platform planning, Trading Technology's coverage of trading infrastructure provides useful context for the systems around the archive.

The mistake most readers will still make is budgeting for the bytes and forgetting to budget for the replay path.

Frequently Asked Questions

How much storage does a year of tick data need?

It depends on instruments, update rates, record format and retention of raw messages. A representative 120 GB per day workload produces about 44 TB of raw data a year, or roughly 11-22 TB after typical 2:1-4:1 compression.

How long does it take to replay a year of trading data?

A replay that preserves a 24/5 market clock can take close to a calendar year of market hours. With a virtual clock and 5-20 times acceleration, a 6,000-hour trading year takes about 300-1,200 hours, before failed runs or bottlenecks.

What is the cheapest way to store historical market data?

Cold object storage usually has a lower per-GB rate than hot storage, but retrieval time and restore charges can make it unsuitable for incident response. Keep frequently used partitions hot and test the restore process for colder copies.

Should a broker keep raw and normalised market data?

Keeping both supports feed investigations and replay against a stable normalised schema, but increases storage and operational work. Retain raw messages where parser changes or disputes require them, and document how each normalised version was produced.

How can a team estimate the cost of replaying a year?

Replay a representative week through the intended services, measure worker-hours and throughput, then scale by event count and include peak periods. Add transfer, temporary outputs, validation and reruns rather than estimating from storage size alone.