In nearly every trading-data setup, the storage estimate starts with a daily file size multiplied by the retention period. That produces a capacity number, not a cost for replaying a year. The gap shows up when an incident, model change or venue migration requires the whole chain to run again, with the same inputs and enough evidence to trust the output.
The usual mistake is to treat the archive as the system. It is only one part of a replay path that includes decoding, normalisation, reference data, application state, compute and validation. Each part has a different limit, and the slowest one sets the wall-clock time.
The storage number is not the replay number
Object storage is often the cheapest component. Typical cloud object-storage rates run around US$10 to US$30 per TB-month, before requests, retrieval charges, replication and network transfer. A 20TB archive therefore has an indicative capacity charge of US$200 to US$600 per month at that rate. It does not follow that a year of data can be read and processed in a few hours.
The archive might be compressed into large files that are efficient to retain but awkward to seek. Replay workers may need to fetch, decompress and parse each partition before the strategy or risk engine can consume it.
A workload limited by CPU decoding can leave storage bandwidth idle. One limited by small random reads can spend more time on requests and latency than on bytes transferred.
These typical market ranges are planning bands, not benchmark results. They describe compressed market-data capture for common workload shapes. The storage-cost column uses the US$10 to US$30 per TB-month range above. The replay durations are illustrative at a sustained aggregate input rate of 0.5 to 1TB per hour, before strategy computation and validation.
| Typical market-data workload | Compressed daily ingest | One-year footprint | Indicative monthly object cost | Illustrative replay duration |
|---|---|---|---|---|
| FX or CFD quote and trade feeds, 365 days | 5 to 40GB | 2 to 15TB | US$20 to US$450 | 2 to 30 hours |
| Single-venue equity top of book, 250 days | 20 to 150GB | 5 to 40TB | US$50 to US$1,200 | 5 to 80 hours |
| Multi-venue equity full depth, 250 days | 200GB to 1.6TB | 50 to 400TB | US$500 to US$12,000 | 50 to 800 hours |
The ranges move substantially with symbol count, venue coverage, message format, compression ratio and whether the archive retains raw packets as well as normalised events. Replay duration is not a promise about any platform. It is arithmetic on an assumed throughput rate, and the rate needs measuring on the actual decode and application path.
Count the data replay actually needs
Many estimates include ticks and trades but leave out the state needed to interpret them. A historical run can produce a different result if the engine uses current instrument metadata, fee schedules or calendars against old events. For a broker, the data set might also need client configuration, routing rules, risk limits and the version of the application being tested.
Keep these components explicit in the retention model:
- Market events: raw venue messages, normalised events, corrections, sequence numbers and timestamps. Retaining both raw and normalised forms improves investigation options but adds capacity.
- Reference data: symbol mappings, tick sizes, contract specifications, corporate actions, trading calendars and venue status.
- Application state: account settings, positions, orders, risk parameters, fee schedules and relevant configuration snapshots.
- Replay artefacts: container or binary versions, schema versions, run parameters, logs, checkpoints and comparison results.
Snapshots and derived bars reduce the amount of work needed for some tasks, but they do not replace the event stream. A snapshot is useful for starting a replay near a point in time. It cannot prove how the state was reached unless the inputs and snapshot version are also retained.
Timestamp policy needs the same care. Exchange time, capture time and processing time answer different questions. If a feed handler records only one timestamp, a later replay may be unable to distinguish venue delay from local queueing. Preserve sequence information and gap markers too. Otherwise, a clean-looking output can conceal missing input.
This is where a retention design should connect with broader trading infrastructure coverage: the archive, event pipeline and operational controls are one system, even if different teams own them.
A year is a workload, not a file
A replay request needs a definition. Rebuilding one instrument's book is not the same job as rerunning every strategy, account and risk check across the full year. Nor is a sequential scan the same as replaying at event time, where the engine must respect ordering and time-dependent rules.
For a simple throughput estimate, divide the compressed bytes to be read by measured sustained input throughput. Then add time for decoding, state loading, application processing and comparison.
If the test requires 20TB and the replay farm sustains 0.5TB per hour on the complete path, the first pass takes about 40 hours. That estimate assumes the full farm stays busy and there is no restart, throttling or extra work for validation.
Parallelism can shorten the run, but only where the event and state model permits it. Independent symbols or accounts may be partitioned safely. Events affecting shared balances, cross-margin or portfolio limits may need an ordered stream. Splitting by file is not automatically safe: a boundary can separate a cancel from its order, or a position update from the event that created it.
Define the replay objective before sizing capacity. Common cases include:
- Checking feed-handler changes against captured raw messages.
- Rebuilding positions and orders after a state-store failure.
- Running a strategy or risk change against a fixed historical input set.
- Investigating a client dispute with the configuration and reference data active at the time.
Each case asks for a different starting point, output and level of determinism. A feed test may need packet fidelity and timing.
A strategy test may care more about stable event ordering and a versioned configuration. A recovery test needs to prove that the process can restore from the retained artefacts, not merely read an archive.
Price the replay path, not only the bucket
The capacity charge is only one line in the bill. Retrieval from colder tiers can add delay and fees. Replication, backups and object versions increase retained bytes. Temporary working copies can multiply capacity during a run. Compute and network costs rise if workers sit in another region or if replay repeatedly pulls data across an account boundary.
Teams also tend to count one replay, then operate a test programme that runs many. A daily validation job over a small sample has a different profile from a quarterly full-year replay. Record the number of runs, parallel workers, peak temporary footprint and expected retention of outputs. The cost model should include the busy period, not just the steady archive.
A practical estimate can be built in six steps:
- Measure compressed ingest by feed, venue and trading day. Separate raw capture, normalised events and derived products.
- Apply the actual retention calendar. Include weekends for continuous FX feeds, expiry rules and any legal or operational hold periods.
- Add replicas, snapshots, versions and temporary copies. State whether the figure is logical data or billable stored data.
- Benchmark read, decompression and decode rates on representative files. Run with the same worker shape and schema versions planned for production.
- Measure end-to-end replay throughput, including application processing. Test both a quiet interval and a busy market period.
- Restore a sample from the coldest retained tier and compare outputs. Record retrieval delay, missing inputs, runtime and the cost of the test.
That sequence is more useful than a single compression ratio. Compression can reduce storage while increasing CPU use, and a format that supports column pruning may accelerate analytics while being a poor fit for exact event reconstruction. Benchmark the access pattern the operation actually needs.
For a broader view of platform engineering and operating constraints, Trading Technology's infrastructure reporting covers the systems around brokers and prop firms. The relevant design question here is narrower: how quickly can the retained data be turned into a verified state?
Make the estimate falsifiable
A replay plan should have measurable acceptance criteria. State the supported date range, the feeds and state included, the maximum acceptable runtime, and what counts as a matching result. For deterministic systems, compare event counts, sequence gaps, order and position state, and selected output hashes. For floating-point calculations or nondeterministic components, document tolerances and the reason for them.
Test a full representative day before projecting a full year. Include a peak session, a session with corrections or market disruption, and a recovery from a cold copy.
A short clean day can hide the effects of burst rates, larger books, retries or malformed messages. Repeat the test after changing the storage tier, file layout, compression codec or worker instance type.
Keep a small manifest alongside each retained partition. It should identify date, venue, feed, schema version, checksum and expected size. A replay job can then detect missing or altered input before producing a result that looks plausible. The same manifest makes capacity forecasts easier to reconcile with the billed footprint.
Engineers often optimise the archive until the replay budget is considered. Then the file layout, tiering policy and test cadence change. Historical data storage is cheapest when nobody asks it to do anything.
Cheap retention is not the same as cheap recovery.
Trading firms are very good at keeping the past. They are less consistent at timing it.
Frequently Asked Questions
How much does it cost to store a year of market data?
Typical object-storage rates of about US$10 to US$30 per TB-month put a 20TB archive at roughly US$200 to US$600 per month for capacity. Replicas, retrieval, requests, temporary copies and compute can raise the total.
How long does it take to replay one year of trading data?
It depends on the retained footprint and measured end-to-end throughput. At 0.5TB per hour, a 20TB data set takes about 40 hours for one pass, before additional validation or application work.
What data is needed for deterministic market-data replay?
Retain the relevant market events, sequence and timestamp information, reference data, application configuration and software or schema versions. The exact set depends on whether the replay rebuilds a book, account state, risk result or strategy output.
Should historical trading data be stored in a cold tier?
A cold tier can lower the capacity bill, but it usually adds retrieval delay and may add access charges. It suits data with infrequent access if restore tests confirm that the delay and cost fit the recovery or investigation requirement.
Platforms · Software · Providers · Markets