Observability for trading systems: a checklist for this week

A 500 ms delay in the order path can cost a fill at the intended price, trigger a client complaint or turn a routine venue wobble into an incident. This week, check whether your logs and alerts can show where that delay began, which orders it affected and whether risk controls remained available. Observability is useful when it answers those questions quickly, not when it produces the largest dashboard.

Log the path, not every byte

Start with a traceable sequence from client request to execution report. Give each order a correlation ID that survives gateway, risk, routing and venue adapters. Record event time and processing time separately. Use UTC for shared timestamps, and a monotonic clock for measuring durations inside a process. Track clock offset across hosts: a timestamp that looks precise but is out of sync can put events in the wrong order.

Capture order and risk events

For every state transition, log the order ID, account or tenant ID, instrument, side, quantity, order type, route, state, reason code and component. Include timestamps for receipt, validation, risk decision, route submission, venue acknowledgement and final execution or rejection.

Log the rule identifier and decision outcome for risk checks, not just a generic "blocked" message. That makes it possible to distinguish a client breach from a timeout or a stale limit cache.

Do not put credentials, access tokens, payment details or full client payloads into general application logs. Mask personal data and restrict access to account-level fields. Keep an immutable audit trail for order and control events; do not sample it. For verbose debug logs, use short retention and controlled sampling.

In low-latency paths, emit asynchronously and monitor the logging queue itself. A logging sink must not block order processing, but dropped audit events need a separate, high-priority signal.

Measure market data and dependencies

For each feed and instrument group, capture exchange or provider sequence numbers, receive time, processing time, gaps, recovery attempts and the age of the latest usable quote. Record the source and failover state. A feed can be connected while its data is stale, so connection status alone is not a health check.

Instrument dependencies that sit outside the matching or routing path too: DNS, database calls, cache misses, message brokers, identity services and secrets retrieval. Keep metric labels bounded.

Labeling every order ID creates unmanageable cardinality and can push a metrics system into its own failure. Put high-cardinality identifiers in traces or logs, and use metrics labels such as venue, region, adapter and service.

Alert on customer impact and loss of control

Page on conditions that require action, not every error line. Separate symptoms affecting execution from infrastructure warnings. Route alerts to an owner with a runbook and a clear first check. A dashboard is not an alert, and a notification without an owner is only noise.

Prioritise the failures that change trading behaviour

Use a short list of paging conditions tied to what the system can no longer do safely:

• Orders are not acknowledged within the service objective, or acknowledgement latency rises sharply for one route.
• Market data is stale, sequence gaps are not recovering, or primary and backup feeds disagree beyond the configured tolerance.
• Risk checks are timing out, bypassing a required control or returning inconsistent decisions across instances.
• Cancel requests are accumulating, execution reports are delayed, or a reconciliation check finds orders whose state is unknown.

Track rejects by reason and route. A rise in expected margin or price-band rejections may need review, but it should not page like a broken adapter.

Conversely, a modest number of "unknown status" responses can be urgent if the system cannot establish whether exposure exists. Make severity depend on the failure mode and affected population, not just the count.

Alert on saturation before queues become incidents

Queue depth alone can mislead. Track the age of the oldest message and the rate at which the queue is growing.

Combine these with worker utilisation, retry volume, connection-pool use and downstream latency. A service at 90% CPU with stable latency may be healthy; a rising queue with spare CPU may indicate a blocked dependency. Set warning thresholds for trend and capacity, and page when there is evidence of impact or a credible risk of breaching the trading objective.

Group duplicate alerts by service, venue and incident. Suppress child alerts when a known upstream failure explains them, but retain the underlying signals for diagnosis. For operational context and platform reporting, keep your runbook alongside Trading Technology's infrastructure coverage.

Set thresholds by consequence

The figures below are typical market ranges, not universal service objectives. Set separate limits for liquid and less active instruments, internal and venue latency, and market hours versus maintenance windows. Baseline each route before choosing a page threshold.

Use percentiles rather than averages for latency. Averages hide the tail that affects a small but important share of orders.

Alert on a sustained breach, with a faster trigger for hard failures such as feed sequence gaps or unavailable risk checks. Keep the dashboard's time range and alert window aligned so the operator can verify the signal without rebuilding the query.

Run the checklist this week

Work through one representative order route first. Include a normal order, a reject, a cancel and an execution report. Avoid using a live client order to test failure handling unless the procedure is approved and the resulting exposure is controlled.

1. Follow one order across every service and confirm that its correlation ID, timestamps and state transitions remain visible.
2. Check that event time and processing time are distinct, clocks are monitored, and logs do not expose secrets or unnecessary personal data.
3. Verify that audit events are not sampled and that logging failure cannot block the order path. Confirm that dropped events generate an alert.
4. Check feed age, sequence gaps, queue age, order latency, reject reasons and risk-check timeouts on a live dashboard.
5. Review every paging alert for an owner, severity, runbook, evaluation window and a testable first action. Remove alerts that cannot prompt a decision.
6. Simulate a dependency delay or disconnect in a controlled environment. Confirm that alerts identify the affected route and that recovery clears the alert without hiding unresolved orders.

Write down gaps rather than broadening every threshold. A useful weekly review can begin with three measures: time from failure to detection, time to identify the affected component and the number of orders with unresolved state. Check whether any unacknowledged P1 alert is already older than 5 minutes.

#trading system monitoring #trading logs #broker infrastructure #order latency monitoring #market data alerts #trading incident response #risk engine monitoring #observability checklist