Observability for trading systems starts with event context

Observability for a trading system is the ability to infer its live and recent behaviour from emitted telemetry.

That definition is easy to satisfy badly. A dashboard can show green hosts while orders wait in a queue, a risk check uses stale prices, or a venue session silently stops accepting messages. The useful question is not how much telemetry the platform produces. It is whether an operator can reconstruct a specific order's path and detect a failure before it becomes a client or risk incident.

Consider a composite operating scenario: a broker processes 1,500 to 3,000 order requests a second during its busiest interval, across a gateway tier, pre-trade risk, a smart order router and several venue connections. These are illustrative figures, not results from a named firm. The same principles apply to a prop firm, though its internal matching and risk paths may replace parts of the broker flow.

Instrument the trading path, not just the servers

Start with the state transitions an order passes through. For each accepted request, the system should be able to account for receipt, validation, risk decision, route selection, venue send, acknowledgement, execution or rejection, and any cancel or replace. Give each transition a timestamp and a stable correlation identifier. That lets an investigator move from a client order ID to the internal order ID, FIX session, venue message and resulting position without relying on a collection of host-level clues.

Record both wall-clock time and a monotonic duration at each service boundary. Wall-clock timestamps help compare components and external messages, but clock skew can distort a latency calculation. Use synchronised clocks and monitor offset; calculate service and queue durations with monotonic clocks where possible. If the gateway's clock differs from the risk service by 8 milliseconds, a cross-service timeline can otherwise suggest that an approval preceded the request.

For every event, capture fields that support an investigation without copying the whole payload:

• Correlation and parent IDs, event type, environment, service version and instance.
• Instrument, account or pseudonymous account key, side, quantity band, route and venue session.
• Decision outcome and reason code, retry count, queue depth, processing duration and message sequence number.
• Timestamp, clock-offset status and trace or log sampling status.

Keep sensitive data out of general-purpose logs. Do not record credentials, full payment details or authentication tokens. Account identifiers can often be tokenised for operational search, with access to the mapping controlled separately. The order and execution record needed for audit should also remain distinct from debug logs: make it durable, access-controlled and protected from routine log rotation.

Metrics provide the high-level signal; traces and structured events explain it. A single trace per order is attractive, but at several thousand requests a second it can create substantial storage and indexing costs. A practical pattern is to retain metrics for all traffic, sample routine traces, and preserve traces for errors, slow requests and selected high-risk routes. Ensure the sampling decision is visible, or an operator may mistake a missing trace for a missing event.

Build event records that survive an incident

Make correlation deterministic

Do not rely on matching events by approximate timestamp, symbol and quantity. Generate or propagate an identifier at the first trusted ingress point, then carry it through asynchronous queues and service calls. Where external protocols do not preserve the identifier, maintain an explicit mapping between the internal ID and the venue's client order ID, execution ID or FIX sequence number.

Retries need their own identity and parentage. A retry should not look like a second client order, and a duplicate acknowledgement should be distinguishable from a new state transition. Log idempotency decisions and reject reasons as structured fields. Otherwise, an incident review can establish that an order was sent twice but not whether the duplication came from a client retry, a gateway replay or a venue response.

Keep the evidence useful and bounded

Structure logs as fields rather than formatted sentences. That makes it possible to filter by venue, version, reason code or correlation ID, and to build consistent queries when services are written in different languages. Preserve raw protocol payloads only where operations, compliance and data protection controls require them; full payload capture can expose personal data and increase storage and search costs.

Typical operating patterns are to keep high-cardinality diagnostic logs searchable for roughly 7 to 30 days, with longer retention for controlled audit records according to regulatory and internal policy. A central log platform should not be the sole copy of critical order evidence.

Define what happens when its ingest path is unavailable, including local buffering, disk limits and an explicit dropped-event counter. For more on platform and provider choices, Trading Technology's infrastructure coverage provides industry context, but retention and control requirements depend on the firm's own obligations.

Alert on service risk, not dashboard colour

Page on a condition that needs action, not on every metric that changes. A useful alert names the affected path, states the customer or risk consequence, links to the relevant runbook and includes a recent example ID. Separate a page for immediate intervention from a ticket or dashboard warning that can wait for normal review.

The figures below are typical operational ranges used as starting points, not universal thresholds or service guarantees. Set final limits against the venue's behaviour, the firm's latency budget, market conditions and measured baseline. Percentiles matter: an average can hide a small but consequential tail of delayed orders.

Use a combination of absolute and relative thresholds. A 20 ms p99 may be normal for one venue path and a serious regression for another. A relative threshold catches a sudden deterioration, while a floor prevents noisy pages when a low-volume metric has only a handful of samples. For low traffic, alert on missing heartbeats or sequence progress instead of percentage changes.

Common page-worthy conditions include:

• Order or execution processing is delayed beyond the agreed risk budget, or queue age continues to rise.
• A venue session is down, sequence numbers have a gap, or acknowledgements stop while outbound traffic continues.
• Reference prices breach configured freshness limits, or the risk service cannot confirm its data source.
• Rejects, duplicate orders or failed cancels exceed a route-specific threshold.
• Telemetry itself is being dropped, delayed or rejected, leaving the control room unable to verify system state.

Alert routing is part of the control. A market-data freshness breach may require a risk or dealing decision, not only an infrastructure response. A full disk on a gateway has a different owner and escalation path. Use deduplication and grouping so one venue outage does not create hundreds of pages, but keep per-route and per-session detail available in the alert context. Trading Technology has also covered operational changes across broker infrastructure; the useful test here is whether each alert maps to a named action and owner.

Walk through a 14-minute incident

At 10:02, the composite broker's dashboard shows normal host CPU and memory. The order gateway is receiving traffic, but its p99 acknowledgement time on one venue has risen from a typical 25 ms to 90 ms. The alert also shows that router queue age has climbed from 8 ms to 75 ms, while the venue session remains connected. That combination points away from a broad host failure and towards the route or its downstream response.

The on-call engineer opens a delayed order trace. Its stages show 3 ms in ingress validation, 4 ms in pre-trade risk, 6 ms waiting in the router and 71 ms between venue send and acknowledgement. A second query groups acknowledgement latency by venue session and instrument.

Only one session is affected; reject rates remain close to baseline, and the market-data freshness metric is normal. The evidence argues against disabling all routing or relaxing risk checks.

At 10:07, the engineer confirms that outbound messages are progressing but acknowledgements arrive in bursts. Sequence numbers and session logs show no local sequence gap.

The incident lead moves eligible flow to a healthy alternate route under the firm's routing policy, while operations checks venue status and capacity. They do not replay orders whose state is uncertain. Those are reconciled by client order ID and execution records first, because a blind retry could create duplicate exposure.

By 10:16, latency returns to its normal range and queue age drains. The team retains the affected traces and structured session events, compares the incident window with a healthy session, and records the exact time of route change and recovery.

The review should answer whether the trigger fired early enough, whether the alert named the right owner, and whether the records were sufficient to reconcile every order. A graph that merely turned red would not answer those questions.

If this were my operation, I would first map one order from ingress to execution and verify that every transition shares a searchable identifier. Then I would add one actionable alert for queue age or venue acknowledgement latency, test it with a controlled failure, and confirm that the resulting evidence is enough to reconcile an order without guessing.

#trading infrastructure monitoring #broker observability #trading system logs #order execution alerts #FIX monitoring #trading system metrics #incident response trading