Clock synchronisation and timestamp reconciliation for trading

A clock error can cost a broker or prop firm $10,000 to $250,000 or more in a serious incident, once disputed trades, client adjustments, investigation and engineering remediation are counted. That is a planning range, not a universal bill. Even without a direct loss, inconsistent timestamps can make it impossible to establish what happened first.

Clock synchronisation gives machines a shared estimate of UTC. Timestamp reconciliation then decides how to compare records from systems that may use different clocks, timestamp points and precision. Neither task is simply a matter of setting every server to the same time. Trading infrastructure needs a traceable time source, a known error budget and records that preserve both the original observation and any later correction.

What a trading timestamp actually means

A timestamp is only useful when its meaning is defined. An order can have a client-side creation time, a gateway receipt time, a risk-check completion time, a venue send time and an exchange acknowledgement time.

They describe different events. Calling all of them order time invites bad comparisons, even if every clock is perfectly synchronised.

Record the event definition alongside the field name. For example, distinguish the time a network interface received the first packet from the time an application parsed it. A hardware timestamp taken at the NIC can avoid scheduler delay, while an application timestamp includes some part of the host's software path. Neither is automatically the right figure for every use. The right choice depends on whether the record is used for latency measurement, order sequencing, surveillance or dispute handling.

UTC is the common reference, but implementations differ. A system may represent UTC as a Unix timestamp, store nanoseconds since an epoch, or use a local display time.

Daylight saving changes make local time unsuitable for event ordering. Leap-second handling also needs an explicit policy: a host that smears a leap second over a period can disagree with a host that steps or represents it differently.

Keep wall-clock time and elapsed-time measurement separate. UTC can be corrected by a time service, sometimes with a small step during startup or recovery. A monotonic clock is designed to measure intervals without going backwards. Use monotonic time for durations inside a process, and UTC timestamps for correlating events across hosts. Where possible, store both, together with the clock source and timestamp location.

There is a further limit: two timestamps are not necessarily proof of causal order. If host A is estimated to be 40 microseconds fast and host B 30 microseconds slow, observations 20 microseconds apart may be indistinguishable within the combined uncertainty.

Preserve sequence numbers, request identifiers and causal links as well as timestamps. These often establish ordering more reliably than sorting rows by time alone.

Choose and distribute time sources

NTP and PTP solve related problems at different levels of precision and operational complexity. NTP is widely supported and usually adequate for general services, monitoring and many business systems. PTP can deliver tighter alignment on a controlled network, particularly when servers and network interfaces support hardware timestamping. Neither protocol guarantees a fixed accuracy by itself. Network topology, software configuration, path asymmetry and source quality all matter.

The figures below are typical operating ranges, not guarantees. They assume a reasonably engineered environment; internet paths, congested links, virtual machines and poorly configured switches can perform worse.

In a PTP deployment, a grandmaster distributes time, often through boundary or transparent clocks in switches. Hardware timestamping records packet arrival and departure close to the network interface, reducing uncertainty from the operating system.

On Linux, the NIC's PTP hardware clock may need to be aligned with the system clock, using the appropriate PTP services and configuration. A host can report that PTP is running while the application still reads a different, unsynchronised clock.

NTP is simpler, but monitor its source and quality rather than checking only whether a daemon is active. Multiple configured sources help detect a bad source, but several servers fed by the same upstream reference are not truly independent. For tighter accuracy, PTP network asymmetry matters: if forward and reverse paths differ, the calculated offset can be biased. Redundant paths and independent reference sources reduce the risk of a single failure becoming invisible.

Some firms use a GNSS receiver as a primary reference and an independent source for comparison or holdover. GNSS is not a magic fix.

Antenna faults, interference, a failed receiver or a damaged feed can remove the reference. Define what the system does during loss of lock, how long it can remain within its error budget, and whether trading should continue after that limit. Engineers covering broader trading infrastructure operations should treat time distribution as a production dependency, not a background utility.

Reconcile records without hiding uncertainty

Reconciliation means making records comparable while retaining the evidence needed to understand their origins. Do not rewrite an exchange timestamp to match a broker gateway clock. Keep the raw value, source, time zone or epoch, precision, event definition and any normalised value. Store the offset or correction applied and the method used to calculate it.

For each relevant event, a useful record may include a UTC timestamp, a monotonic timestamp, host identity, clock source, sequence number and request or order identifier. It should also identify whether the time came from the NIC, kernel or application. Precision is not accuracy: a value printed to nine decimal places may still be several milliseconds wrong. Avoid implying certainty beyond the measured error bounds.

When comparing two events, account for the uncertainty of both clocks. If A is recorded at 10:00:00.000100 with uncertainty of plus or minus 50 microseconds, and B at 10:00:00.000130 with uncertainty of plus or minus 40 microseconds, the apparent 30-microsecond order is not conclusive.

The intervals overlap. Report the order as indeterminate unless another signal, such as a sequence number or acknowledgement chain, resolves it.

A practical reconciliation pipeline should apply these steps:

1. Preserve the original. Ingest raw timestamps and identifiers without rounding, overwriting or converting away source information.
2. Normalise representation. Convert valid values to a common UTC representation, while recording the original format and any leap-second or smear policy.
3. Map event semantics. Match like with like, such as gateway receipt to gateway receipt, rather than comparing an application event with an exchange network event.
4. Estimate uncertainty. Use measured clock offset, timestamp resolution and known capture delay to establish a range around each observation.
5. Correlate and classify. Use order identifiers, sequence numbers and causal relationships. Label ambiguous ordering instead of forcing a ranking.
6. Retain the calculation. Store the normalisation version, correction and source data so a later review can reproduce the result.

Keep separate views for operational latency and evidential reconstruction. A latency dashboard may subtract two timestamps to find a typical duration, but the result is only meaningful if both endpoints have compatible event definitions and a sufficiently small combined error. For incident review, retain the raw observations and uncertainty. A clean chart is not a substitute for a defensible record.

Operate the clock as a production service

Set an error budget based on the use case. A general order-management service may tolerate milliseconds; a latency-sensitive gateway or surveillance process may require tighter control.

Typical engineered targets range from below 1 ms for NTP-based systems to a few microseconds for hardware PTP, but the target should follow venue requirements, applicable recordkeeping rules and the firm's own measurement needs. Confirm current requirements with compliance rather than treating a technology target as a regulatory standard.

Monitor the quality of time, not just service availability. Useful signals include offset, estimated uncertainty, frequency adjustment, source identity, time since last good update, PTP grandmaster changes, GNSS lock state and holdover status. Alert on trends as well as hard limits. An offset drifting from 20 to 200 microseconds may deserve attention before it crosses a trading control threshold.

Exercise failure modes in a controlled environment. Disconnect the primary source, introduce packet loss, restart a PTP service, switch grandmasters and test recovery after a host reboot.

Check whether the clock steps or slews, whether applications tolerate a repeated or backward wall-clock value, and whether monitoring reports a loss of accuracy promptly. Virtualised hosts need particular attention because migration and virtual clock behaviour can add variation.

Assign clear ownership. Infrastructure should own sources, distribution and host configuration. Application teams should define timestamp points and preserve source metadata.

Operations should know the escalation path and the trading response if the error budget is exceeded. Compliance and risk teams should agree how ambiguous event order is represented in records and incident reports. For teams tracking platform and provider changes, Trading Technology's infrastructure coverage provides context, but local measurement remains the control.

Stop doing these things:

• Stop treating a timestamp's decimal places as proof of accuracy.
• Stop overwriting raw event times with corrected values.
• Stop comparing timestamps with different event definitions as if they were equivalent.
• Stop relying on a single time source without measuring loss of reference and holdover.

#sincronización horaria PTP #infraestructura del broker #marcas de tiempo por hardware #NTP para sistemas de trading #trading timestamp reconciliation #UTC time source #trading system clock drift #timestamp accuracy