Clock synchronisation needs measured timestamp reconciliation

NTP alone does not make trading timestamps trustworthy.

A host can report a small offset from its configured time source while recording events at inconsistent points in the network path. A feed handler may timestamp a packet in user space after queueing.

A gateway may use a NIC hardware clock. Subtracting those values can produce a precise-looking number that does not represent the latency between events.

For a broker or prop firm, the job is to measure each clock, define what each timestamp represents, and quantify the uncertainty between them. That turns synchronisation from a daemon health check into an operational control.

Define what the timestamp measures

Start with an event map. For each feed, order and execution path, record the timestamp source, clock domain and capture point.

A market-data timestamp might come from the exchange, a feed handler or the NIC. An order timestamp might mark strategy submission, gateway receipt, socket write or exchange acknowledgement. Those points are not interchangeable.

Keep wall-clock time and elapsed-time measurement separate. UTC wall-clock time is useful for correlating events across systems, but it can be stepped or slewed as the clock is corrected.

Use a monotonic clock for durations inside one process. On Linux, teams commonly distinguish CLOCK_REALTIME from CLOCK_MONOTONIC or CLOCK_MONOTONIC_RAW, and document which one each application uses.

For every timestamp field, document:

• Which device or process generated it, and whether it uses UTC or TAI.
• Where in the packet or order lifecycle the event is captured.
• Whether it is software or hardware stamped, and the clock it reads.
• Its expected resolution, accuracy and known conversion steps.

This inventory exposes mismatches early. For example, subtracting an exchange-origin timestamp from a host timestamp requires both clocks to be synchronised and the exchange timestamp's meaning to be understood. It does not, by itself, measure network latency.

Measure offset, drift and uncertainty

Monitor more than whether chronyd or a PTP daemon is running. Collect the estimated offset, frequency correction, estimated error or dispersion, source identity, selected state, network delay and time since the last valid update. For PTP, add grandmaster identity, servo state, path delay and the difference between the NIC's PTP hardware clock and the system clock.

Offset is the estimated difference between a local clock and its reference. Drift describes how quickly that difference changes when corrections are absent or delayed.

Frequency correction is commonly expressed in parts per million. A system showing a small current offset but a steadily rising correction or repeated source changes may be close to losing reliable time.

Use independent checks where the risk warrants them. Compare the host clock with a second reference, inspect PTP state from the NIC and switch, and capture packets at a known point.

A daemon's estimate is not an independent measurement of its own timestamping path. NTP estimates also rely on assumptions about path delay; asymmetry between outbound and inbound traffic can bias the result.

Typical operating ranges depend on hardware, topology and load. A managed NTP deployment may sit within roughly 0.1 to 10 milliseconds of its reference, while hardware-timestamped PTP on a controlled LAN may reach roughly 100 nanoseconds to 1 microsecond.

Software timestamping can range from tens of microseconds into milliseconds under queueing or CPU pressure. Treat these as planning ranges, not service guarantees.

Useful signals include:

• Offset and estimated uncertainty, tracked as time series rather than a single current value.
• Offset slope, source changes, missed polls and PTP announce or sync packet loss.
• PHC-to-system-clock delta, NIC queue pressure and timestamping mode.
• Holdover duration and measured drift after reference loss.

Reconcile timestamps across the path

Build a sequence of events for a sample order or market-data packet. Preserve the original timestamp and its source, then add a normalised value only after recording the time scale and conversion. Avoid overwriting raw values: a UTC-to-TAI conversion, leap-second handling or application-level rounding can otherwise become impossible to audit.

For two hosts, a measured interval includes more than the event duration. It can include clock offset between hosts, timestamp placement error, capture-device error and uncertainty from the transport path. Represent this explicitly. If host A is estimated to be 2 microseconds ahead of host B, a raw subtraction of their timestamps carries that offset unless corrected. If the offset estimate itself has uncertainty, the reconciled interval has uncertainty too.

Maintain a timestamp provenance field or equivalent metadata in diagnostic captures. Record the source clock, capture layer, resolution, synchronisation state and any transformation applied.

In high-volume production paths, this may be too expensive to attach to every event. Keep it in configuration and telemetry, and include it in sampled traces and incident records.

Reconciliation should also test event order. Flag negative durations, acknowledgements apparently preceding submissions, and feed events that move backwards after conversion.

These checks often reveal a clock-domain mix-up or a timestamp taken before a queue, not a genuine network improvement. Correlate events using stable identifiers such as order ID, packet sequence number or execution reference, rather than nearest timestamps alone.

Trading Technology's infrastructure coverage provides wider context for the systems that generate and consume these measurements.

Set budgets around operational use

There is no universal offset threshold for every trading system. Set the budget from the measurement question.

If a team needs to compare two events separated by 100 microseconds, a 5-millisecond clock error makes that comparison useless. If the requirement is incident correlation across a distributed back office, millisecond-level time may be adequate.

Typical starting points for a controlled environment might be a warning for sustained host offset above 1 millisecond on NTP, and above 1 microsecond on a hardware-timestamped PTP segment. These are operational examples, not exchange rules.

Page on loss of lock, an unexpected grandmaster, a failed reference or an offset that breaches the system's declared error budget. Use persistence windows to avoid paging on one noisy sample, but do not hide rapid drift behind long averaging.

Measure at several layers so an alert points to a likely fault domain:

• Reference and network: source reachability, packet delay and path changes.
• Host and NIC: daemon state, PHC offset, system offset and CPU or queue pressure.
• Application: timestamp source, event ordering and reconciliation residuals.

Keep thresholds separate from service-level objectives. A monitor can be configured to warn at 1 microsecond, but that does not mean every application timestamp is accurate to 1 microsecond. State the measured layer and uncertainty in dashboards and incident notes.

Cost and deployment trade-offs

NTP is simple to deploy and often sufficient for systems that need millisecond-scale correlation. PTP adds hardware, network configuration and operational dependencies.

Hardware timestamping can reduce measurement error, but only when the NIC, driver, switch path, grandmaster and application configuration work together. A PTP-capable server on an unmanaged network does not deliver a useful PTP service by itself.

The table gives typical market ranges in USD for planning. Hardware prices vary by port count, redundancy and support; labour assumes internal engineering time or comparable implementation work. These are indicative ranges, not supplier quotes.

Cost the full path, not just the clock. Include redundant references, managed switches, cabling, monitoring, firmware control and test time. Redundancy also needs a failure policy: automatic failover can preserve availability, but a bad source selection may make timestamps less trustworthy than a short, declared holdover.

Run a repeatable verification checklist

Test under normal load and during failure. A quiet lab can conceal NIC queueing, CPU scheduling delays and switch congestion that appear during a busy session. Capture both daemon telemetry and application-level events, then compare them against an independent reference or measurement method.

1. Inventory every time source, clock domain and timestamp capture point on a representative feed and order path.
2. Record baseline offset, uncertainty, drift, source identity and PHC-to-system delta for at least one full trading session.
3. Check event ordering and reconciliation residuals using correlated packet and order identifiers, not timestamp proximity alone.
4. Repeat the test under peak message load and CPU pressure; compare software and hardware timestamp behaviour where supported.
5. Simulate reference loss or PTP path failure in a controlled window. Measure holdover drift, failover time and alert delay.
6. Document thresholds, time-scale conversions, ownership and recovery actions. Re-run after NIC, kernel, switch or timing configuration changes.

Retain raw samples long enough to investigate incidents, including source changes and clock steps. Dashboard averages can obscure short excursions, so keep high-resolution data around alerts. The useful result is not a green synchronisation indicator. It is evidence that a timestamp means what the application says it means, within a known error budget.

If this were my operation, I would first map one end-to-end order path and identify the clock and capture point behind every timestamp. Then I would collect offset, uncertainty and event-order data for a full session before buying new timing hardware.

#timestamp reconciliation #PTP synchronisation #NTP monitoring #hardware timestamping #trading infrastructure monitoring #clock offset #market data timestamps #clock drift