Why does the order log show a decision before the market-data event that triggered it? Most often, the systems are recording timestamps against clocks with different offsets, not showing a real reversal in event order. A composite broker incident shows how to diagnose the gap and reconcile the timeline without mistaking clock error for trading latency.
The firm in this walkthrough runs market-data handlers, strategy processes and order gateways across two data-centre racks. Its ordinary order path takes about 0.75 milliseconds from exchange event to outbound order. The logs initially suggested that a strategy had acted before it received the relevant price update.
The incident: one order, three timelines
What the logs showed
At 10:00:00, an exchange event occurred at 10:00:00.000000, according to the venue timestamp. The market-data host recorded receipt at 10:00:00.001050.
The strategy recorded its decision at 10:00:00.000200, and the gateway logged the outbound order at 10:00:00.000950. Read literally, the strategy decision predates the feed handler's receipt by 850 microseconds.
Packet capture and sequence numbers showed the actual order was normal. The feed handler's system clock was about 700 microseconds fast, the strategy host's was about 400 microseconds slow, and the gateway's was about 200 microseconds fast.
Correcting those offsets puts feed receipt at 10:00:00.000350, decision at 10:00:00.000600 and order send at 10:00:00.000750. The processing path took 400 microseconds from feed receipt to send.
What was actually happening
The incident was not a slow strategy or a feed-handler defect. It was a comparison of timestamps from three clocks, with no record of their offsets or uncertainty. That distinction matters: an exchange event timestamp, a local packet-receipt timestamp and an application decision timestamp describe different points in the path. They are not interchangeable measures of latency.
Start the investigation by separating four things: event time at the venue, receive time at the network interface, processing time inside the application, and send time at the gateway. Preserve sequence numbers and packet captures where available. They can establish causal order even when wall-clock timestamps cannot.
- Record which clock created each timestamp, including whether it came from the NIC, kernel or application.
- Keep the raw timestamp as received. Add a corrected value rather than overwriting it.
- Check for clock steps, restarts, failovers and time-source changes around the affected interval.
This kind of incident is familiar across broker and prop-firm infrastructure, but the evidence is often spread across vendors and internal services. Trading Technology's coverage of trading infrastructure provides wider context for the systems involved.
Build a clock model before comparing logs
Separate wall time from elapsed time
Use UTC wall-clock time to correlate events between machines and to meet recording requirements. Use a monotonic clock for elapsed time inside a process. Wall clocks can be adjusted by time services; a monotonic clock should not move backwards. Measure durations such as queue wait and strategy execution with a monotonic source, then attach UTC timestamps at defined boundaries.
Choose and document the timestamp source at each boundary. NIC hardware timestamping can capture packet arrival closer to the wire than an application timestamp taken after kernel and scheduler delays.
The extra precision has a cost: compatible network cards, drivers and switches need configuration and validation. Software timestamps are simpler, but include more variable scheduling and stack delay.
Set the required accuracy by activity
There is no single clock target that suits every system. A low-frequency order-management service and a latency-sensitive strategy have different needs.
UK and EU firms should map clock controls to the applicable MiFID II RTS 25 requirements and their activity category. Those requirements specify maximum divergence from UTC and timestamp granularity according to activity; confirm the current obligations with compliance rather than applying one threshold to every host.
The figures below are typical market ranges, not guaranteed performance. Actual results depend on hardware, network topology, configuration and time-source quality.
| Setup or measurement | Typical offset or resolution range | Where it is useful | Main trade-off |
|---|---|---|---|
| Uncorrected general-purpose host | Hundreds of microseconds to several milliseconds | Non-latency-sensitive services | Offsets can vary with load and time-source reachability |
| NTP on a well-managed LAN | About 50 microseconds to 1 millisecond | Monitoring, back office and many application logs | Network asymmetry and software timestamping limit precision |
| PTP with software timestamps | About 1 to 50 microseconds | Cross-host correlation where sub-millisecond alignment is needed | Performance depends on switches, host load and implementation |
| PTP with hardware timestamping | About 0.1 to 5 microseconds | Latency-sensitive paths and precise packet timing | Requires compatible NICs and a controlled PTP network |
| Typical oscillator drift during source loss | About 0.1 to 1 ppm, or 6 to 60 microseconds per minute | Estimating holdover risk | Actual drift varies by oscillator, temperature and ageing |
Discipline clocks without hiding failures
Choose the time architecture
NTP is straightforward to operate and can be adequate for services that tolerate millisecond-scale uncertainty. PTP is the usual choice when hosts need tighter alignment. A PTP design depends on more than enabling a daemon: switches must support the selected profile, grandmasters need a controlled source, and hosts must use compatible timestamping. A poorly configured PTP network can produce confident but wrong timestamps.
For a typical low-latency deployment, use redundant time sources and monitor the selected grandmaster, offset, path delay and state transitions. If GNSS supplies UTC, plan for antenna faults, interference and loss of reception. Measure holdover behaviour rather than assuming a host remains accurate after the source disappears. A fallback to NTP should be explicit, alarmed and reflected in the quality attached to timestamps.
Make correction visible
Do not treat a synchronised status flag as proof of accuracy. Alert on measured offset, time-source loss, PTP domain changes and clock steps.
Set thresholds from the system's risk and legal requirements. For example, a team may investigate a 10-microsecond offset on a hardware-timestamped strategy host, while a back-office service may use a much wider operational threshold.
Clock services usually slew time to reduce an offset gradually. A step changes wall time abruptly and can make timestamps appear to jump or go backwards. Define which services may step, when they may do so, and how applications behave if they detect a discontinuity. Log the adjustment and its magnitude. Keep synchronisation alarms in the same incident timeline as order and market-data events.
- Track offset, jitter, source identity, reachability and holdover state per host.
- Alert on clock steps and on offset beyond the application's documented tolerance.
- Test source loss and failover in a controlled window, including application behaviour.
- Retain time-service logs alongside gateway, feed and operating-system records.
Reconcile events and latency at the boundary
Build a reconciliation pipeline that stores the original timestamp, source clock, timestamping point and estimated offset. Add a corrected UTC value and an uncertainty range. If a host is estimated to be 400 microseconds slow with uncertainty of 20 microseconds, represent that uncertainty; do not present the corrected time as exact.
For each event, retain identifiers that survive across components: exchange sequence number, packet sequence, client order ID, internal request ID and venue order ID. Use those identifiers to join records, then use corrected timestamps to measure intervals. If timestamps disagree with sequence order, investigate clock quality and capture points before attributing the difference to application behaviour.
Keep separate metrics for feed transport, queueing, strategy computation, gateway processing and venue acknowledgement. A single end-to-end figure hides where time was spent. For example, a 0.75-millisecond event-to-send path could include 250 microseconds of feed transport, 150 microseconds of scheduling and queueing, 250 microseconds of strategy work, and 100 microseconds of gateway processing. Those are illustrative components, not universal targets.
Store enough information to reproduce the calculation after an incident. That usually means raw and corrected timestamps, offset samples, uncertainty, clock-source transitions, packet or application sequence numbers and the correction version. For a practical review of broker systems and operations, see Trading Technology.
This week, run one controlled timestamp reconciliation across a market-data host, a strategy host and an order gateway, and record the measured offset and uncertainty for each.
Frequently Asked Questions
What is clock synchronisation in trading systems?
Clock synchronisation aligns system clocks to a common time reference, usually UTC. It lets teams compare events across market-data, strategy and order-routing systems, provided they also record where each timestamp was taken.
Is NTP accurate enough for a trading platform?
NTP can be adequate for back-office services and systems that tolerate millisecond-scale uncertainty. Latency-sensitive paths often need PTP, and hardware timestamping where precise packet timing is required.
Why can trading logs show events in the wrong order?
Hosts may have different clock offsets, so a later event on a slow clock can appear earlier than an event on a fast clock. Compare source offsets and sequence numbers before treating the log order as causal.
How should engineers reconcile timestamps from different hosts?
Keep each raw timestamp with its clock source and capture point, then calculate a corrected UTC value using measured offset. Preserve uncertainty, sequence identifiers and time-source changes so the reconciliation can be audited.
Platforms · Software · Providers · Markets