Historical data storage: the cost of replaying one year

12 TB of a year's tick history can tie up $1,000 to $5,000 a month once replicas, indexes, backups and database overhead are counted. That is before anyone replays it. A full-year run can add compute, data retrieval and engineering time, while competing with live workloads for network and storage bandwidth.

The storage line is often easy to estimate. The replay bill is not.

It depends on whether the job rebuilds order books, reruns risk checks, or simply scans records for research. A capacity plan that prices only retained bytes can leave a broker or prop firm with data it cannot restore quickly enough to meet an operational target.

Start with the workload and its required output, then price the storage tiers around it. This is the same infrastructure-first approach used across Trading Technology's coverage of trading platforms and operations.

The bill is not the byte price

Historical data storage costs have at least four parts: retained capacity, retrieval, compute, and the people needed to make the result trustworthy. Cloud object storage is usually the cheapest layer, but data may also exist in a hot database, a second region, a backup vault and temporary scratch space during replay. Indexes and replicas can multiply the footprint.

Separate fixed retention from replay costs. For each data set, estimate the bill for keeping it for a month, then estimate a full-year replay as a distinct job. Include temporary writes and failed attempts. A replay that produces a rebuilt book or order-level simulation may write as much data as it reads.

• Count raw and normalised records separately. Compression ratios vary with message type and encoding.
• Include backups, replicas, indexes, metadata and temporary replay output in the retained footprint.
• Record retrieval charges, egress and the hours of engineering or operations support for a run.

Use a simple cost model: retained TB multiplied by monthly tier price, plus retrieval and egress, plus replay vCPU-hours and temporary storage. Keep staff time visible rather than hiding it in an infrastructure estimate.

Define what a year means

A year of market history is not a single, uniform object. A feed may include trades, quotes, depth updates, status messages, corrections and reference data.

The system also needs to know which schema, symbol mapping, venue calendar and sequence rules applied at the time. For strategy or execution replay, pin the strategy build, risk configuration and relevant instrument settings as well.

Write down the replay's purpose before choosing a format. A research scan can often use partitioned columnar files such as Parquet. A deterministic incident investigation may require the original message stream, preserving ordering and corrections. A common design keeps an immutable canonical capture and creates compact derived datasets for queries. Converting everything to one analytical format can reduce scan costs but may discard details required to reproduce an event sequence.

Partition by date and a useful key such as venue or instrument, while avoiding so many tiny files that metadata becomes the bottleneck. Keep a manifest with source, schema version, record counts, checksums and known gaps. A replay should fail clearly if a required partition or reference file is missing, rather than silently produce a plausible but incomplete result.

Retained data is only useful if the replay can prove what it read and what it produced.

Model replay as a workload

Benchmark a representative slice before sizing a full year. Choose a busy session, a quiet session and a known incident period.

Measure read throughput, CPU per record, decompression, peak memory, temporary writes and output validation. Extrapolate cautiously: a year may not run at the speed of one month if shared indexes, network limits or stateful processing become bottlenecks.

The ranges below are typical market ranges in USD, not a quote. They vary by provider, region, commitment, access pattern and redundancy. Retrieval, egress and support charges can change the total materially.

Test concurrency as well as a single run. Parallel work across dates or instruments can shorten elapsed time, but shared state, global risk limits or cross-instrument logic may restrict safe parallelism.

Also test whether a replay throttles live market-data ingestion or order services. A cheaper job that causes a live service to miss its latency budget is not a saving.

Use 30 days to build the plan

Keep the first month focused on measurement and a controlled pilot, not a wholesale migration. Assign an owner from operations and one from the team that consumes replay output. Use the deployment process already approved for production data.

1. Days 1 to 5, inventory. List data sets, daily growth, retention rules, current copies, formats and consumers. Measure the largest and busiest day, not just an average. Identify legal, audit and incident-response retention requirements.
2. Days 6 to 10, specify. Define the replay cases, required fidelity, acceptable completion time and failure behaviour. Set a cost ceiling for a full-year run and decide which data must remain immediately available.
3. Days 11 to 17, benchmark. Run a representative month from each candidate tier. Record wall time, throughput, CPU-hours, bytes read, bytes written, retrieval delay and output checksums. Repeat with enough workers to expose network and storage limits.
4. Days 18 to 24, pilot tiering. Move a limited historical slice to the proposed layout. Retain a canonical source copy, add manifests and automate restore into isolated scratch space. Document how to cancel a runaway job and how to resume after failure.
5. Days 25 to 30, validate and price. Replay a known event and compare record counts, sequence checks and expected outputs. Produce a monthly retention estimate and a per-run estimate, including egress, temporary capacity and staff support. Get sign-off on operational targets before expanding.

Use the pilot to establish a measurable service objective, not a universal benchmark. For example, an eight-hour full-year replay may suit one investigation process and be unacceptable for another.

Specify whether the clock includes archive restore, validation and export. Alert on cost and duration, and set a hard concurrency limit so recovery jobs cannot consume all shared capacity.

Set acceptance gates before moving history

Do not promote the design because the first replay completed. Require repeatable results from the same inputs, a documented way to locate missing or corrupt partitions, and a tested restore path. The replay runner should emit a manifest containing input versions, code and configuration identifiers, start and end times, resource use, output checksums and any skipped records.

Operations should also know who can request a full-year run, where it runs, and how it is stopped. Put limits around scratch-space growth and concurrent jobs.

If archives have retrieval delays, expose that delay to the requesting team rather than presenting the dataset as instantly available. Retention policy, deletion policy and legal holds need to apply to both canonical and derived copies.

Track storage growth and replay demand monthly. If replay frequency rises, moving data to a cheaper tier may increase retrieval and staff costs.

If demand falls, an indexed hot copy may no longer justify its footprint. Revisit the model when record formats, feed coverage or replay logic change, since each can alter compression, compute and validation costs.

This week, run one measured replay of the largest representative month and record its wall time, peak resources and output checksum.

#reproducción de datos de trading #costes de almacenamiento de datos tick #datos históricos de mercado #archivo de datos de mercado #infraestructura de reproducción #salida de datos de la nube #retención de datos de trading #recuperación de datos históricos