Platforms · Software · Providers · Markets
"Can we tell exactly what happened to this order?" If the team needs to search three dashboards, ask a venue for a timestamp or join logs by account name, the honest answer is no. Trading system observability is not a wall of green lights. It is the ability to reconstruct a decision and identify a failing control before it affects more orders.
The dashboard is not the system
A common failure is to monitor hosts and processes while treating the trading path as a black box. CPU, memory and network utilisation matter, but a healthy server can still send stale prices, reject valid orders or leave an order in an uncertain state after a connection drops. Conversely, a brief CPU spike may have no customer impact.
Start with the transaction path: market-data receipt, normalisation, strategy or client request, risk checks, order routing, venue acknowledgement, execution and downstream booking. Give each stage a measurable boundary.
Record both event time and receive time, plus a correlation ID that survives service boundaries. If an order is replaced, cancelled or partially filled, its history must still be queryable as one chain.
Why averages hide the fault
Average latency can look stable while a small but important group of orders waits too long. Use histograms and monitor percentiles, split by path where the split changes the operational response: venue, asset class, order type or service.
Keep labels bounded. A client ID or order ID is useful in a log, but disastrous as a metric label because every new value creates another time series.
Measure queue depth and age as well as processing time. A queue of 20,000 messages may be harmless if it drains in milliseconds, or dangerous if its oldest item is several seconds behind. For market data, track sequence gaps, stale-feed duration and recovery time, not only whether the socket is connected.
Instrument service boundaries
Put timestamps around meaningful transitions, not every function call. Use a monotonic clock for elapsed time within a process, and synchronise wall clocks across hosts for cross-service investigation.
Clock synchronisation does not make timestamps identical: retain the source, clock offset or uncertainty where it matters. Otherwise, a timeline can suggest that an acknowledgement preceded the send that caused it.
For teams defining these boundaries across a broker stack, the Trading Technology infrastructure coverage provides useful context on the systems involved. The instrumentation itself should follow your actual topology, including gateways, risk services, bridges and external venues.
Log the evidence needed to explain a decision
Application logs should describe state transitions and decisions, not reproduce every packet. A structured event should include a timestamp, service and version, environment, correlation ID, order lifecycle ID, transition, result code and relevant timing. Include venue or route, instrument and order type where they help isolate behaviour. Make it possible to answer: what did the system receive, what checks ran, what did it decide, and what response came back?
Capture decision context, not secrets
For a risk rejection, log the rule identifier and the threshold or configuration version applied, plus the resulting decision. For a route, record the selected destination and selection reason. For a fill, preserve venue execution identifiers and quantities.
Do not log credentials, authentication tokens, full payment data or unrestricted personal information. Restrict access to account-level logs and set retention according to operational, legal and regulatory requirements.
Useful event fields often include:
- Order state before and after the event, with a stable lifecycle identifier.
- Request, acknowledgement and execution timestamps, with clock source or offset where needed.
- Decision outcome, reason code, configuration version and service build.
- Retry count, connection state and upstream or downstream dependency.
- Payload size or selected diagnostic fields, rather than an indiscriminate copy of the payload.
Keep the data usable under load
Full request and response logging can add latency, increase storage costs and expose sensitive data. Sample routine success paths if the sampling decision is made consistently across the whole trace. Keep all errors, rejects and rare state transitions where feasible. When sampling, preserve a way to retrieve the complete trace for a specific order. Measure logging queue depth and dropped-event counts: telemetry silently discarded during a busy session is precisely when incident evidence is most valuable.
Separate high-volume metrics from detailed logs. A counter can show that rejects rose; the log should explain which rules rejected them.
Set retention tiers: short, searchable retention for detailed events and longer retention for compact aggregates, subject to your requirements. Test queries against peak-period data before relying on them during an incident.
Alert on loss of control, not noise
Page a person when an issue requires action now. Use dashboards and tickets for degradation that can wait.
A threshold should connect a symptom to a consequence and a response. "CPU above 80 per cent" is weak by itself. "Order acknowledgement latency above the service objective for five minutes, with the affected route identified" gives the on-call engineer a reason to investigate.
The figures below are typical operating ranges used as starting points, not universal targets or audited results. Set thresholds from your own baseline, venue characteristics, service objectives and session patterns.
| Signal | Typical range to investigate | Alert condition to consider | First response |
|---|---|---|---|
| Order acknowledgement latency, p99 | 50-250 ms for many electronic routes | Above the route objective for 3-5 minutes | Compare gateway, risk and venue timings |
| Order rejects | 0.1-1% in steady-state flows | More than 2-3 times baseline for 5 minutes | Group by reason, route and instrument |
| Market-data age | 100-500 ms, depending on feed and product | Above the feed-specific stale threshold | Check sequence gaps and alternate feeds |
| Queue age | Under 100 ms in many synchronous paths | Oldest item exceeds the service objective | Find the blocked consumer and control intake |
Thresholds need context. A market open can produce a predictable change in message volume; it should not make a real feed gap disappear inside a broad daily average. Build baselines by session and route, but keep hard safety limits for controls that must not be bypassed. Avoid alerts on every individual reject. A burst of expected risk rejections can be normal; a sharp increase in one reason code may indicate a bad configuration or stale reference data.
A practical alert policy separates:
- Page now: trading is unavailable, risk controls are not enforcing, orders are stuck or market data is stale beyond a defined limit.
- Investigate soon: latency or rejects are trending towards a limit, a queue is growing, or a non-critical dependency is degraded.
- Record only: isolated transient errors with automatic recovery and no measurable customer or control impact.
Every page needs an owner, a runbook and a useful link to the relevant graph or query. Group related alerts so a single venue outage does not generate separate pages for every symbol.
Add a recovery condition and a minimum duration to reduce flapping. Review alert volume after incidents; if the on-call team routinely ignores a page, its threshold or action is wrong.
Test the telemetry path before the incident
Observability has its own failure modes. A log collector can lag, a metric exporter can stop, or a dashboard can show cached data. Monitor telemetry delivery separately from trading services. Track event loss, collection delay, storage capacity and clock offset. Send synthetic events through the same pipeline and verify that an alert reaches the person on call, not just that it appears on a dashboard.
Run controlled failure tests: drop a feed connection, delay an acknowledgement, fill a queue, restart a consumer and make a dependency return errors. Check that the alert fires within the intended window and that the recorded events reconstruct what happened. Keep the test safe and bounded, with a rollback plan and a clear indication that the fault is simulated.
After incidents, add the missing evidence or alert rather than simply increasing log volume. A short review should identify the first detectable signal, the point at which the team acted and any ambiguity in the event sequence. The advice here does not apply unchanged to a research or batch system with no live order path, where scheduled job status and data-quality checks may matter more than second-by-second paging.
Frequently Asked Questions
What should a trading system log for each order?
Log a stable order lifecycle ID, state transitions, timestamps, decision outcomes, reason codes, route, service version and venue responses. Keep sensitive credentials and unnecessary personal data out of the event stream.
Which trading system metrics should trigger an alert?
Alert on loss of trading control, such as stale market data, stuck orders, risk-check failure or latency beyond a route-specific objective. Use queue age, reject rates and telemetry delivery health to identify growing problems before they affect more orders.
Should brokers log every market-data message?
Usually not in the primary application log, because volume and storage costs can become substantial. Record sequence gaps, feed age and recovery events, and use bounded packet capture or targeted sampling when detailed message-level investigation is needed.
