Historical data storage and the cost of replaying a year

Historical data storage is the retention of past market and trading events in a form that can be queried or replayed.

That definition hides the operational question: a year of compressed files may be cheap to keep, but expensive or too slow to turn into a valid test. The answer depends on what each event records, how replay treats timing and state, and how quickly teams need results.

Define the replay before pricing the archive

Start by deciding what "replay" means for the workload. A research backtest might need trades and top-of-book quotes. Reproducing an execution incident may require full depth updates, order acknowledgements, fills, rejects, configuration changes and the exact timestamps used by the live system.

These are different datasets, with different storage and compute demands. A quote stream can be useful for price simulation but cannot prove how a matching or routing decision behaved against changing order-book depth. Likewise, application logs alone may show that an order was sent without preserving the market state and internal decision inputs that explain why.

Write down the event sources and the required fidelity before estimating capacity. Include venue feeds, internal order and execution events, reference data, clock and sequence information, and the version of the normalisation rules.

Keep raw vendor messages where licensing permits. Normalised records are easier to query, but raw messages help diagnose parser changes and reproduce data defects.

Make time semantics explicit. Store exchange event time and local receive time, and preserve sequence numbers where available.

A replay that orders everything by one timestamp can conceal feed gaps, clock skew or late messages. For investigations, those details can matter more than the number of decimal places in a price field.

Turn event rates into annual capacity

Estimate volume from measured peak and average rates, not a single quiet day. Record bytes per event after framing, metadata and indexes are included.

Then calculate daily ingest, trading-day count, retention years and replica count. For a first pass, 250 sessions is a useful annual planning assumption for a market that trades on weekdays, but include weekends and calendar-time logs if the system records them.

The figures below are typical planning ranges for a logical dataset, not audited figures for any provider. Daily volumes vary with instrument count, venue coverage, depth and activity. Annual figures use 250 sessions and exclude backups, replicas and derived indexes.

Compression ratios are not fixed. Columnar formats and delta encoding can reduce repetitive fields, while high-entropy payloads compress less. Measure the ratio on representative busy and quiet sessions. Leave headroom for schema changes, late corrections and parallel writes. A nominal 100 TB dataset may require two or three times that capacity once replicas, snapshots and working copies are counted.

For a rough model, multiply average compressed bytes per session by sessions per year, retention years and replica factor. Add indexes and temporary replay output separately. That last step prevents teams from treating the archive's bill as the whole cost of a backtest.

Price the storage tiers, not just the bytes

Fast local NVMe is useful for active replay sets and indexes. Object storage is a common fit for the main archive because it scales without managing a large disk estate. Cold archive tiers can lower the holding cost, but retrieval time and fees make them a poor fit for a dataset that researchers expect to open immediately.

For broad planning, typical market ranges are roughly US$80-250 per TB-month for provisioned high-performance storage, US$15-30 for frequently accessed object storage, and US$1-5 for cold archive before retrieval and request charges. Rates vary by region, redundancy, performance and contract. Treat them as a starting point, then price the actual design with the relevant provider.

At 100 TB, those bands imply about US$96,000-300,000 a year on high-performance storage, US$18,000-36,000 on frequently accessed object storage, or US$1,200-6,000 in cold archive charges before retrieval. If the 100 TB is only one copy, two replicas double the stored volume. If replay requires staging 20 TB from archive every week, retrieval and temporary storage can erase much of the apparent saving.

Separate the cost model into these components:

• Primary data: compressed events, raw messages, reference data and derived datasets.
• Protection: replicas, snapshots, backup retention and cross-region copies.
• Access: retrieval, requests, network transfer and staging space.
• Compute: decompression, validation, indexing and simulation workers.

Set retention by purpose. An operator may keep a short window of hot data for daily investigation, a longer object-storage history for research, and a narrower set of compliance records under the firm's retention policy.

Do not assume one retention period should apply to every event class. Check venue data licences and regulatory requirements before reducing or moving records.

Replay speed is a compute and data-path problem

A year of 250 sessions at 6.5 hours per session contains 1,625 hours of market time. At one-times playback, processing it serially takes about 68 days. At ten-times playback, the theoretical duration falls to about 6.8 days.

At one hundred times, it is about 16 hours. These figures exclude startup, validation, gaps, market holidays and time spent writing results.

Faster playback does not come from storage bandwidth alone. The bottleneck may be decompression, event decoding, a single-threaded strategy, locking, database writes or the simulator's need to preserve event order.

Measure the full path. A file reader that streams at 500 MB/s is not useful if the event processor handles only a fraction of the resulting message rate.

Parallelise across independent dates, instruments or strategies where the test's logic allows it. Avoid splitting a stateful strategy at arbitrary points unless each worker can rebuild the required state from a checkpoint.

For incident reproduction, a bounded window around the event is often more useful than a year-long run. For annual risk tests, daily checkpoints can reduce restart time, but validate that checkpointed and continuous runs produce equivalent results.

Capture four measurements for each replay workload: compressed bytes read per second, decoded events per second, CPU utilisation by stage, and wall-clock completion time. Record the test dataset, software version and worker count. This makes changes in replay performance comparable rather than anecdotal. Trading Technology's coverage of trading infrastructure provides wider context for the platform decisions that sit around this work.

Make the archive operable

Storage is useful only if operators can locate, validate and restore the data. Assign each partition a manifest containing its date range, venue, feed, schema version, compression, checksums and sequence coverage. Keep the manifest separate from the files it describes, and monitor missing partitions and unexpected volume changes.

Before setting a service level, test a real restore. Cold-tier recall can take minutes or hours, while a corrupted index or incompatible schema can take longer to diagnose.

Define separate targets for routine research and incident response. The latter may justify a warm copy of recent sessions even when the bulk archive stays on lower-cost storage.

Use this checklist for the first capacity review:

• Measure at least five representative sessions, including a busy session and a volatile one.
• Calculate compressed annual volume by dataset, then add replicas, indexes and temporary staging.
• Benchmark replay on the same instance and storage tier proposed for production use.
• Run a restore and an end-to-end replay from archived data, not a local developer copy.
• Set alerts for ingest gaps, checksum failures, retention expiry and storage growth beyond forecast.

Keep raw and transformed data lineage visible. If a normaliser changes, record which partitions were rebuilt and which tests used the earlier output. That gives engineering and operations a way to distinguish market behaviour from pipeline changes when a result moves.

This week, measure one busy session from ingest through replay and use its bytes, event rate and completion time to build the first annual capacity model.

#market data storage #trading data replay #tick data storage costs #historical market data #data retention planning #trading system backtesting #object storage for brokers #market data compression