Clock synchronisation: a 30-day plan for trading systems

Start with the event, not the clock

A 1 ms clock error can turn a 700 μs order-routing delay into a negative latency measurement. It can also make a broker's execution record appear to precede the market-data event it acted on. The cost is operational: investigations take longer, surveillance evidence becomes harder to defend, and teams may tune systems against misleading latency figures.

Clock synchronisation is not the same as timestamp reconciliation. Synchronisation disciplines a host or device clock against a reference. Reconciliation makes timestamps from different clocks and systems comparable, while preserving what each system actually recorded. A trading stack needs both.

Start by defining the events that matter. A single order may have a client-send time, gateway-receive time, risk-check completion, venue-send time, venue acknowledgement and client-facing response. These are different events, often recorded by different devices. A timestamp without an event name, clock source and precision is weak evidence.

Inventory each source across the trading path: application hosts, kernel and NIC timestamps, market-data capture, gateways, databases, venue feeds, monitoring agents and cloud services. Record whether timestamps represent UTC, TAI or a local wall clock, and whether they are taken in software or hardware. If a system cannot identify its clock source, mark that as a control gap rather than assuming it follows the host.

Keep two measures separate. Offset is the estimated difference between a clock and its reference. Uncertainty is the confidence range around that estimate, including network asymmetry and measurement error. A dashboard showing a 4 μs offset is not proof of 4 μs accuracy if the measurement uncertainty is 50 μs.

Teams planning changes to a broader trading stack can use Trading Technology's infrastructure coverage as context, but the baseline must come from their own packet captures, host telemetry and event logs.

Days 1 to 7: map and measure

Spend the first week establishing a baseline without changing production clock behaviour. Collect offset, frequency correction, reachability, stratum or PTP state, source identity, last synchronisation time and alarm history.

Sample at least once per second for critical hosts, then retain a lower-rate roll-up for trend analysis. Preserve raw output so changes in the monitoring pipeline do not rewrite the evidence.

Typical market ranges vary by topology and timestamp method. The figures below are practical planning ranges, not guarantees or audited results. Network design, NIC support, switch configuration and reference quality can move them materially.

During the baseline, compare hosts in the same rack and across network zones. Check whether a reported offset is relative to UTC, a PTP grandmaster or another NTP server. Look for clocks that jump, drift steadily, lose reference intermittently or silently switch source. A single average conceals short excursions that can cross a reporting tolerance.

Use the first week to agree an event dictionary and ownership. For each event, document its capture point, clock domain, timestamp unit, expected ordering and authoritative source. For example, a venue acknowledgement time should not be substituted with the gateway's receive time simply because the venue field is absent.

Days 8 to 21: build reconciliation

In week two, define a canonical timestamp model without discarding source values. Store the raw timestamp, source system, event type, clock identifier, capture method, precision, sequence number and ingestion time.

Add a normalised UTC value, the estimated offset applied and an uncertainty estimate. This makes corrections explainable and reversible.

Choose one sign convention and document it. If measured offset is defined as local clock minus UTC, then normalised UTC equals the raw local timestamp minus that offset.

Teams often reverse the sign when importing telemetry from tools that define offset differently. Test the calculation against a known reference and synthetic timestamps before applying it to historical records.

Do not blindly order all events by corrected wall-clock time. Use sequence numbers and causal links where available.

A gateway send followed by a venue acknowledgement should have a request identifier or session sequence to support the relationship. If the corrected times conflict with that sequence, retain both facts and flag the record for review. Do not manufacture a tidy order by changing timestamps.

Make reconciliation rules explicit for each data source. Apply a correction only when the clock source and offset estimate are known for that interval. If the source is unknown, the offset is stale, or uncertainty exceeds the event's tolerance, label the timestamp as low confidence. Keep the original value and record the reason for any exclusion.

Use these steps to implement the model and move through the first three weeks:

1. Days 8 to 9: define UTC and TAI handling, timestamp units, event names and the offset sign convention.
2. Days 10 to 11: add raw timestamp, clock identity, source, capture method and sequence fields to the event schema.
3. Days 12 to 13: calculate normalised timestamps and uncertainty in a non-production pipeline. Test positive and negative offsets.
4. Days 14 to 15: replay events with known ordering, missing references, stale offsets and duplicate sequence numbers.
5. Days 16 to 17: compare application, kernel and NIC capture times for a representative order flow.
6. Days 18 to 19: send low-confidence and out-of-order records to a review queue. Avoid silently dropping them.
7. Days 20 to 21: review the rules with operations, surveillance and compliance, including applicable local clock requirements.

Leap-second handling needs a deliberate policy. UTC can include a leap second, while systems may represent time using TAI or a vendor-specific smear.

Mixing smeared and unsmeared clocks can create offsets lasting well beyond the event itself. Record the source's policy, test conversion boundaries and avoid applying a global adjustment that changes historical raw data.

Days 22 to 30: prove recovery and operate

The final nine days are for failure testing, alert design and controlled rollout. Test loss of the primary reference, a PTP grandmaster change, network partition, host restart, NIC reset and a reference that remains reachable but becomes inaccurate.

Record detection time, offset peak, time to stable recovery and whether applications kept accepting orders. A green service state alone does not answer those questions.

Set thresholds by use case, not by one enterprise-wide number. A market-data capture host and an office reporting service do not have the same tolerance. Establish warning and critical limits from the relevant regulatory requirement, venue specification and measured system capability. Applicable clock-recording rules differ by jurisdiction and activity, so have compliance confirm the requirements before treating an engineering threshold as a compliance limit.

Useful controls to put into the runbook include:

• Alert on offset, uncertainty, reference loss, source changes and servo state, not only on daemon health.
• Page on sustained breaches and log shorter excursions for investigation. Set persistence windows to filter brief noise without hiding real loss of synchronisation.
• Keep time-service configuration under change control. A new server, switch profile or firmware version can alter the timing path.
• Retain raw clock telemetry alongside order and market-data events for the same period, subject to the firm's retention policy.
• Give operations a clear action for each state: continue, restrict a specific function, fail over or stop new order entry.

Roll out first to a small, representative set of hosts. Compare the new normalised records with the existing pipeline for several sessions, including busy periods and reference transitions.

Check that latency distributions remain plausible, event ordering is not being silently changed and downstream reports can distinguish raw from corrected values. Roll back the transformation if it alters an event's meaning or removes source provenance.

At day 30, the deliverable is not a claim that every clock is perfect. It is an inventory, a measured baseline, a documented event model, tested conversion rules, failure evidence and named owners. Keep the baseline current after network or platform changes, and schedule a recurring test of reference failover rather than relying on a one-off acceptance test.

The mistake most readers will still make is treating a green synchronisation dashboard as proof that every event timestamp is comparable.

#timestamp reconciliation #UTC traceability #trading system timestamps #PTP synchronisation #NTP monitoring #MiFID II clock synchronisation #hardware timestamping #trading infrastructure monitoring