Observability for trading systems: logs, alerts and costs

Start with the question operations must answer

Which part of the trade path failed, and how quickly can we prove it? Build observability around answering that question, not around collecting the largest possible volume of telemetry. For a broker or prop firm, the useful evidence connects a client request to market data, risk checks, routing, execution and the response shown in the platform.

Set a service objective for each critical path. Typical targets might include 99.95% monthly availability for order entry, a p99 acknowledgement time below 100 milliseconds inside the firm's own network, and market-data age below 250 milliseconds for a liquid instrument.

These are planning examples, not universal standards. Venue, asset class, geography and execution model change what is acceptable.

Agree which events count as failure before writing alert rules. A rejected order can be a valid risk decision. A request that times out before reaching risk is not. Separate business outcomes from infrastructure errors, and record the reason code that distinguishes them. This prevents dashboards from treating a deliberate rejection as a platform incident.

Write down the critical user journeys, their normal latency and their failure modes. Include login and session recovery, quote delivery, order submission, modification, cancellation, fills and account updates. Teams that need a wider view of platform operations can compare their instrumentation plan with material published by Trading Technology, then map it to their own architecture.

Instrument the trade path end to end

Use a correlation identifier that follows a request across service boundaries. An order ID is useful after an order exists, but it cannot identify a request rejected before order creation.

Generate a request or interaction ID at the edge, carry it through risk and routing, and retain the venue's order reference when available. Do not put account numbers, names or other direct identifiers into general-purpose telemetry.

Measure each stage separately. Capture ingress time, validation completion, risk decision, route selection, send time, venue acknowledgement, fill receipt and client notification. Use monotonic clocks for durations within a process. For cross-host comparisons, monitor clock offset with NTP or PTP and alert when measured offset exceeds the tolerance required by the system. A wall-clock timestamp alone can make a fast path look slow when hosts disagree on time.

Use histograms for latency, not averages. Track p50, p95 and p99 by service, route and relevant instrument class. An average can remain healthy while a small group of orders waits several seconds.

Keep label dimensions bounded: route, region, service and outcome are usually safer than client ID, order ID or symbol at full scale. Unbounded labels inflate storage and can make a metrics backend unusable during a busy session.

Distributed traces help locate a slow dependency, but tracing every message is rarely necessary. Sample a small baseline, often 1% to 5%, and retain all traces for errors or requests above a latency threshold.

Where the message rate is high, use tail-based sampling so the collector can keep slow or failed traces after seeing their outcome. Check that sampling does not remove the only evidence of a low-frequency failure.

• Track market-data age at receipt and after internal fan-out, alongside sequence gaps and stale-quote counts.
• Measure order queue depth, oldest-message age, reject rates by reason, and cancel or replace latency.
• Record dependency health for risk, reference data, venue sessions, authentication and persistence.
• Expose position and cash update delay, so execution success is not mistaken for a fully updated client view.

Log events that explain decisions

Logs should answer what happened, where, and why. Use structured fields rather than free-text sentences for operational events. A useful order event can include timestamp, service and version, environment, interaction ID, order ID where applicable, route, state transition, result code, elapsed time and a compact error category. Keep the schema stable and version it when fields change.

Log state changes and decisions, not every internal function call. For example, record an order's transition from accepted to risk-approved to routed, with the decision code and relevant rule version. Avoid copying full payloads by default. A payload may contain personal data, credentials or commercially sensitive strategy information, and it can multiply storage volume. If payload capture is needed for a specific investigation, gate it, redact it and set a short retention period.

Make errors actionable. An exception stack trace without a request identifier or dependency name is weak evidence. Include the failing operation, retry count, timeout setting and whether the request is safe to retry. Distinguish a connection reset from a venue business rejection. Rate-limit repeated identical errors, but preserve counters so suppression does not hide an outage. Keep a separate audit record for events that require durable retention or regulatory review. Operational logs are not a substitute for an audit trail.

Set retention by use. A common pattern is 7 to 14 days of searchable, high-detail operational logs, with selected security and audit records held longer under a separate policy. Compress and move older diagnostic data to cheaper storage if it is still useful. Confirm access controls, deletion behaviour and data residency with the provider. The people responding to a trading incident should not need broad access to client data to inspect service health.

Alert on trading impact, not dashboard colour

Page on symptoms that need immediate action. A host CPU spike may be worth investigating, but it is not automatically an incident. A rising age of the oldest order in a queue, sustained loss of venue acknowledgements, or market data going stale for a material share of instruments has a direct relationship to trading risk.

Use a duration and a threshold together. A single 500-millisecond latency sample should not wake an engineer.

A p99 above the service objective for five minutes, across two consecutive evaluation windows, may justify a page. For short bursts that can still harm orders, use a faster trigger tied to a concrete limit, such as a queue's oldest message exceeding one second. Tune thresholds against replayed incidents and normal peak periods, not a quiet overnight baseline.

Route alerts by ownership and severity. A useful page names the affected service, region or venue, the observed value, threshold, start time, runbook and dashboard. Send lower-risk capacity warnings to a ticket or team channel. Group duplicate alerts by incident key, but do not merge distinct venues or regions if they need separate mitigation. Set a maximum of a few high-severity pages per incident where possible, then make the primary alert carry enough context to guide triage.

Monitor the monitoring path as well. Alert if telemetry ingestion stops, collectors lose data, clocks drift, or notification delivery fails. Run a test page at least monthly and test escalation after-hours.

A warning that nobody owns is noise: each page needs an on-call team, an acknowledgement target, and an escalation path. Typical response targets are five minutes for acknowledgement of a critical trading-path page and 15 to 30 minutes for a high-priority investigation, adjusted for the firm's coverage model.

Budget telemetry without losing evidence

Estimate costs from volume, retention and query needs before enabling verbose logs across every service. The figures below are typical market ranges for planning, not supplier quotes or audited costs. They assume commercial hosted services, USD billing and ordinary production workloads. Region, compression, ingest commitments, searchable retention and support plans can move the total substantially.

Control spend at the source. Drop routine health checks from verbose logs, sample repeated success events, and keep counters for events that do not need individual records. Use separate policies for debug, application, security and audit data. Set ingestion budgets per service and alert when a service's daily volume rises sharply. A sudden tenfold increase can signal a logging loop as readily as a real traffic event.

Do not save money by deleting the evidence required to diagnose a rare execution fault. Keep error traces, state transitions and venue responses at a higher retention level than routine success messages. Test queries against incident scenarios before reducing retention. For further platform operations context, the publication's infrastructure coverage can help frame vendor and architecture choices, but cost assumptions still need to be checked against the firm's own event volume.

Make the checklist part of operations

Observability is a production capability, so treat its configuration like application code. Review dashboards, alert thresholds, log schemas and sampling rules through change control. Run a replay or controlled fault before a material release, and make sure the on-call engineer can find the relevant runbook without knowing which team built the service.

Use this checklist during a quarterly review and after material platform changes:

1. Trace one order from client ingress through risk, routing, venue response and client notification. Confirm every stage shares a usable correlation ID.
2. Check p50, p95 and p99 latency against agreed objectives during the busiest trading window, not only in quiet periods.
3. Inject or replay a feed gap, slow risk response, venue disconnect and telemetry collector failure. Confirm alerts arrive with the expected owner and context.
4. Inspect log fields for personal data, credentials, unbounded identifiers and payloads. Confirm redaction and retention policies work in practice.
5. Review the top telemetry cost drivers, daily volume changes and high-cardinality metrics. Set a budget and an owner for each.
6. Verify the audit record remains separate, complete and accessible under its retention policy, even if operational logging is sampled or unavailable.
7. Record alert false positives and incidents that produced no alert. Adjust thresholds only after checking the effect on detection time and noise.

The mistake most teams will still make after reading this is treating observability as a dashboard project, rather than testing whether the right evidence survives a real failure.

#trading system monitoring #trading logs #broker infrastructure #order execution monitoring #market data latency #trading alerts #distributed tracing #observability costs