Platforms · Software · Providers · Markets
More telemetry does not make a trading system more observable.
A broker can collect millions of log lines and still miss a stalled order route, a market data gap or a risk service that is quietly timing out. Useful observability connects a trading event to the system state that handled it, then shows whether that state affected execution, exposure or client access.
The objective is not to measure everything. It is to make important failure modes visible quickly enough to act.
Start with the path an order takes: client gateway, authentication and validation, pre-trade risk, order management system, router, venue and acknowledgement or fill processing. Add market data ingestion, positions and ledger updates where they affect decisions.
Each boundary needs a timestamp and a consistent order or event identifier. Without those, teams may know that a request was slow, but not where its time went.
Measure the trading path, not just the host
Host CPU, memory and network utilisation are useful context, but they do not describe service quality. A server can be at 35% CPU while a single partition, thread pool or downstream connection is saturated. Measure latency at each service boundary, alongside queue depth, throughput, errors and the age of the data being consumed.
For latency, retain histograms and report p50, p95 and p99. Averages conceal tail delays that can affect a small but important share of orders. Measure end-to-end time and component time separately, using monotonic clocks for durations.
Use synchronised UTC timestamps for comparing events across hosts, and track clock offset as an operational signal. A timestamp that looks precise but comes from a drifting clock can produce a false causal story.
Define the measurement point precisely. "Order latency" might mean client request to gateway acceptance, gateway acceptance to venue acknowledgement, or client request to final execution report. These are different intervals. Record venue, route and order type as dimensions where practical, but do not put account IDs, order IDs or symbols into metric labels. High-cardinality labels can overwhelm a metrics backend and make the dashboard least reliable during a busy market.
The following are typical market ranges for operational reference, not targets or audited results. Actual values vary with venue, asset class, connectivity model, geography and market conditions. Set service objectives from measured baselines and contractual requirements.
| Signal | Typical market range | Useful alert condition | Why it matters |
|---|---|---|---|
| Internal risk check latency, p99 | 0.2-5 ms | Above the service objective for 5 minutes, or a sharp rise in timeout rate | Slow checks can back up order admission |
| Gateway to venue acknowledgement, p99 | 10-100 ms, excluding venue-side outliers | Route-specific breach sustained for 3-5 minutes | Separates routing or connectivity issues from client-side delay |
| Market data age at strategy or risk consumer | 1-50 ms in low-latency paths; higher where transport is indirect | Age exceeds feed-specific limit or sequence gaps persist | Stale inputs can make risk and pricing decisions unsafe |
| Order rejection rate | Often 0.1-1% in steady operation, with venue and order-mix variation | More than twice the rolling baseline, or a material rise in risk rejects | Finds bad configuration, venue changes or client impact |
| Queue depth at a critical consumer | Often near zero to a few hundred messages in a healthy steady state | Queue grows continuously or exceeds 60-80% of tested capacity | Shows accumulating work before requests time out |
Ranges are a starting point for investigation, not universal thresholds. A 20 ms acknowledgement may be normal on one route and an incident on another. Compare the same route, order class and market session, and include a baseline from both quiet and busy periods.
Log events with enough context to reconstruct decisions
Logs answer questions metrics cannot: which validation failed, what route was selected, and what response came back? Use structured events with stable field names and explicit event types. Keep the operational record separate from the authoritative order, execution and audit records. Application logs can be delayed, sampled or lost; they should not be the only evidence of a trade or control decision.
For the order lifecycle, log state transitions rather than repeatedly printing the full order object. A compact event should identify the transition, its source and result. Useful fields include:
- Correlation: trace ID, request ID and an internal order reference, with identifiers handled under the firm's access and retention policy.
- Timing: event time, process time, service name, host or instance, and measured duration where relevant.
- Decision: action taken, reason code, rule or configuration version, and whether the request was accepted, rejected, queued or retried.
- Route: venue, connector and protocol session, represented as controlled values rather than free text.
- Outcome: acknowledgement, execution, cancel result, timeout or error class, including a normalised code and a bounded message.
Do not log credentials, access tokens, full payment data or unrestricted client personal information. Avoid dumping complete payloads by default.
If payload capture is required for a production investigation, make it time-limited, access-controlled and explicit about redaction. Sampling may be acceptable for repetitive debug events, but preserve security, risk-decision and order-state events according to the firm's recordkeeping obligations.
Trace propagation helps connect service calls across gateways, risk engines and routers. It has limits in trading paths: asynchronous queues, proprietary protocols and venue sessions may break a conventional trace.
Carry a correlation reference through those boundaries and emit a clear hand-off event. Keep trace sampling adaptive and ensure a trace can be promoted for a failing route without turning every production request into a large payload.
Alert on impact, not dashboard colour
Page an engineer when a condition requires action, not whenever a metric moves. A single high-latency sample, one rejected order or a brief CPU spike is usually a diagnostic signal, not an incident. Alerts should combine a threshold with duration, rate or user impact, and should identify the affected service, route and runbook. Where possible, alert on symptoms first: orders are not being acknowledged, market data is stale, or risk checks are failing.
Useful alert groups include order admission failures, acknowledgement latency, stale or gapped market data, execution-report backlog, risk-service errors, and position or ledger reconciliation breaks. Add capacity and dependency alerts, but tune them so that a host warning only pages when it has a plausible path to trading impact. Route alerts by ownership. The team that owns a venue connector should receive connector symptoms; a broad "trading platform degraded" alert should be reserved for cross-service impact.
Set objectives around a defined service and measurement window. For example, a firm might target 99.9% of valid order submissions reaching a terminal acknowledgement within an agreed interval during supported trading hours.
The interval and exclusions must be explicit, especially for venue-side delays. Track error-budget consumption and alert on sustained burn rates, rather than paging on every small breach. A fast burn can justify a page; a slow burn may belong in a ticket or daily review.
Use this sequence to build an alert set that can be operated:
- Map critical flows and name the owner for each service boundary.
- Choose a small set of service indicators: success rate, latency distribution, data freshness and backlog.
- Establish baselines by route, session and workload, then agree objectives with engineering and operations.
- Write alerts around sustained user or control impact, with a severity, owner and runbook action.
- Test alerts with fault injection or controlled replay, including recovery and notification delivery.
- Review false positives, missed incidents and alert volume each month; change thresholds when operating conditions change.
A useful alert tells the on-call engineer what stopped working, where it is visible and what to check first. Record whether it recovered automatically and how long recovery took. For a deeper view of platform operations and provider infrastructure, Trading Technology's coverage of trading platforms can sit alongside internal service documentation, but it cannot replace route-specific baselines or runbooks.
Keep measurement useful under load
Observability itself consumes resources. High-volume logs increase storage and indexing costs; traces add processing overhead; broad labels increase metric cardinality.
Measure the effect of instrumentation at peak throughput. Use asynchronous logging where appropriate, bounded buffers and explicit behaviour when a collector is unavailable. The trading path must not block on a remote logging service.
Keep dashboards aligned to operational questions. An overview should show order success, latency tails, feed freshness and critical queue depth. Drill-down views should expose route, venue and service, with links to relevant logs or traces. Avoid one dashboard per component with no view of the complete order lifecycle. Review access controls, retention and data residency alongside technical performance, especially where event records contain client or order information.
Run a regular review with production incidents in hand. Check whether the first alert matched the actual failure, whether the relevant event fields were present, and whether clock differences made the timeline ambiguous. Test a market-data gap, a slow risk dependency and a disconnected venue session in a non-production environment. The aim is to verify detection and diagnosis, not simply that a dashboard turns red.
Stop doing these things:
- Stop paging on isolated metric spikes without duration or impact.
- Stop using averages as the only latency measure.
- Stop putting order IDs, account IDs or symbols into metric labels.
- Stop treating application logs as the authoritative trade record.
Frequently Asked Questions
What should a trading system log for each order?
Log lifecycle transitions, timestamps, service and route, correlation references, decision outcomes and normalised error or venue codes. Keep sensitive payloads out of routine logs, and preserve authoritative order and execution records separately.
Which trading system metrics should page an engineer?
Page on sustained symptoms such as order admission failures, rising acknowledgement latency, stale market data, growing execution backlogs or failed risk checks. Add duration and impact conditions so a single transient sample does not trigger a page.
How should a broker measure order latency?
Define the start and end points for each interval, then measure both end-to-end latency and time at individual service boundaries. Report percentiles such as p95 and p99, and compare like-for-like routes, order types and market sessions.
Should trading logs include every order payload?
Usually not. Structured lifecycle events provide useful diagnostic context with less exposure and storage cost than full payload capture. Restrict any exceptional payload capture by purpose, access, duration and redaction rules.
