Historical data storage: the cost of replaying a year

US$10-30 per terabyte per month is a typical range for standard cloud object storage. That is the visible price of keeping a compressed terabyte of trading history, before replication, requests and retrieval charges. The larger cost can arrive when a risk or compliance team needs a year of events replayed, checked and explained, while the people who built the original pipeline have moved on.

For a broker or prop firm, the useful question is not simply how much data a year produces. It is what the firm must be able to reconstruct, how quickly, and with which evidence. Those choices determine whether historical data is a cheap archive or a working control.

What does one year contain?

"Historical data" often describes several different records. A venue feed capture preserves what arrived over the network. A normalised market-data stream records the firm's interpretation of it. Order and execution events show what the trading system sent and received. Risk decisions, account changes and administrative actions provide further context. These records do not have the same retention needs or replay costs.

Volume varies sharply with product, venue count, depth and sampling. A firm retaining top-of-book data for a limited set of instruments may generate hundreds of gigabytes a year. Full-depth feeds across several venues can reach tens of terabytes or more. Order, execution and account journals are usually smaller, but can be the records that matter most in a complaint, surveillance review or incident investigation.

Count events as well as bytes. A replay that reads 2 TB sequentially may be straightforward; one that must rebuild state from billions of small messages can be limited by decoding, ordering and database writes. Record the expected daily volume, peak message rate, compression ratio and retained instrument universe. Keep separate figures for raw capture and derived data, so the annual estimate does not hide duplicate copies.

Define the replay target too. Reconstructing an order's lifecycle is not the same task as reproducing a risk decision or recalculating a market-data strategy. The latter may need the original reference data, configuration and code version, not just the feed and order log.

Why storage is not replay readiness

Object storage is inexpensive because it is designed to keep data, not to serve it instantly in application-ready form. Cold tiers can add retrieval delays and restore charges. A replay job still has to fetch objects, decompress them, parse schemas, restore state and resolve dependencies. If events must be joined across market data, orders and risk decisions, the access pattern may involve more than one sequential read.

Test the full path rather than measuring the time to download a file. A practical run records bytes read, decompression rate, events processed per second, CPU and memory use, database writes, retries and time to reach a known checkpoint. Test both a normal day and a peak day. Averages hide the busy session that sets the capacity requirement.

There is also a difference between a replay for engineering and one used as evidence. Engineers may tolerate a new parser or a corrected schema. An investigation needs to distinguish the original record from a later interpretation. Keep raw inputs immutable, retain schema versions, and record the software build and configuration used for each replay. A hash alone cannot prove that a dataset is complete, but checksums combined with sequence counts and gap reports help show whether inputs changed or went missing.

Cost estimates should include the work around compute: retrieval fees, temporary disk, network transfer, parallel workers and operator time. A cloud job that takes a few hours can still require days of engineering if timestamp semantics are unclear or an old schema no longer has a maintained decoder. Teams following wider trading infrastructure coverage will recognise the same operational pattern: the recurring bill is often easier to estimate than the cost of an untested dependency.

A benchmark for a one-year replay

The figures below are typical market ranges, not a quote or a benchmark for a particular provider. Storage estimates use compressed retained data and standard object storage at roughly US$10-30 per TB-month. Replay throughput assumes an 8-32 vCPU worker doing a single parse and validation pass, without expensive state joins. Actual rates depend on message size, format, code and storage locality.

Read the event-time column carefully. It is a normalised calculation for one billion records, not a prediction for an entire year. Ten billion events at 100,000 events per second take about 28 hours of uninterrupted processing on one worker. Parallel workers can reduce elapsed time when partitions are independent, but they add compute cost and can create ordering or shared-state problems. A stateful risk replay may be slower than this simple pass by several multiples.

For a useful internal estimate, multiply annual event count by measured processing time per event, then add restore, validation and state-building time. Run that calculation against a representative dataset on the hardware and software intended for production use. Storage cost is the easy part of the table; an observed replay rate is far more valuable than a vendor's headline throughput.

Make the evidence reproducible

A defensible replay needs enough context to explain why the output is what it is. Keep a manifest alongside each dataset with date range, source, schema version, partition counts, sequence gaps, checksums and any known clock or capture issues.

Preserve the mapping between venue timestamps, gateway receipt times and internal processing times. Do not silently replace a corrected file: store the correction and its reason as a new version.

For each replay, capture the code commit, runtime or container image, configuration, reference-data version, calendar and timezone rules. Record the operator, start and end times, input manifest and output location. This does not mean every routine engineering experiment needs a formal approval workflow. It means an investigation can distinguish an exploratory run from one presented as a reconstruction of past activity.

Agree which system is authoritative for each event type. A broker may have exchange acknowledgements in a gateway log, executions in an order-management journal and account balances in a ledger.

Reconciliation rules should state how to handle duplicates, late arrivals, sequence resets and clock drift. Otherwise, two teams can replay the same period and reach different answers while each believes its output is complete.

Useful controls to put in the runbook include:

• Immutable raw capture, with access controls and a documented correction process.
• Automated counts and gap checks at ingestion and before replay.
• Versioned schemas and maintained decoders for the full retention period.
• A replay manifest linking inputs, code, configuration and outputs.
• Periodic restore tests, including at least one peak-volume interval and a deliberately incomplete dataset.

Retention must follow the firm's obligations and documented schedule for each record class. A single blanket retention setting can retain low-value payloads for too long while leaving a critical audit trail difficult to retrieve. Legal holds and investigation requests also need a way to prevent scheduled deletion without changing the original data.

Choose retention by decision and recovery target

Start with the decisions the data must support: investigating an order, demonstrating a risk control, resolving a client dispute, or validating a system change. For each one, write down the required inputs, acceptable retrieval time and acceptable replay duration.

"Keep a year" is not a retrieval objective. "Restore the relevant session and produce a signed event manifest within four hours" is testable.

Use storage tiers deliberately. Keep recent, frequently queried data in a fast-access tier. Move older material to lower-cost storage only after testing the real restore path and confirming that the format remains readable.

Retain small, high-value journals and manifests in a form that is quick to query, while using larger raw captures for investigations that genuinely require packet-level detail. Compression saves storage and transfer, but choose formats that support partitioned reads and tolerate partial recovery.

Partitioning by date, venue and instrument can make retrieval efficient, but excessive small files create request overhead and slow metadata operations. Typical operational designs batch records into files large enough for efficient sequential reads, then maintain an index or manifest for selective retrieval. Validate the choice with actual query and replay patterns rather than relying on one file-size rule across every dataset.

Assign an owner to each data class and schedule a replay test, not just a backup test. The test should report elapsed time, direct compute and retrieval charges, missing inputs, manual interventions and whether the output reconciles with known control totals. If the last successful replay used code that no longer builds, the archive may be intact but the recovery capability is not.

The useful planning document is a costed retention matrix: annual volume, replicas, storage tier, retrieval target, replay rate, legal or control purpose, and owner. It lets operations compare the marginal cost of keeping a year online with the risk and labour of restoring it from a colder tier. Which one-year replay must the firm be able to reproduce, from which immutable inputs, within what time, and who signs off that the result is complete?

#reproducción de datos de trading #datos históricos de mercado #costes de almacenamiento de datos de mercado #broker data retention #trading audit trail #event replay infrastructure #compliance data retention #trading systems recovery