Trading system observability: what to log and alert on

At a typical burst rate of 500 order events a second, a 30-second observability gap leaves 15,000 events to reconstruct. That costs the on-call team time while exposure remains unclear: an order may be waiting in a queue, rejected by risk, or accepted by a venue without the status reaching the client.

Trading system observability is the ability to tell what the platform is doing, how well it is doing it and where a failure began. It combines logs, metrics and traces.

None is a replacement for the others. Together, they let an operations team distinguish a slow market-data feed from a congested order gateway or a failed downstream dependency.

Start with the order path

Instrument the path an order takes, from client request through API gateway, validation, risk checks, order management system and venue adapter. Include the return path for acknowledgements, fills, rejects and cancellations. A component diagram is a useful start, but the operating view needs to show which service owns each step and what happens when that step times out.

Give each order request a correlation identifier that survives service boundaries. Carry it in structured logs and trace context, and record the parent-child relationship between the client request and downstream actions.

A single order can generate many events, so use an order or request identifier to join them rather than relying on timestamps alone. Clocks drift, and asynchronous queues can reorder events.

Record the event time and the time the system observed or persisted it. This distinction helps locate delay. If a venue acknowledgement arrives quickly but appears late in the client-facing service, the issue is likely internal.

If it reaches the adapter late, inspect the connection and venue response. Keep synchronised clocks across hosts, but do not treat timestamps as a substitute for correlation IDs.

Logs should explain decisions as well as failures. Useful fields include service and version, environment, account or tenant identifier, order state transition, reason code, dependency and duration.

Avoid logging credentials, full payment details or unnecessary personal data. Apply access controls and retention rules, and keep regulatory audit records distinct from operational logs where retention, immutability or access requirements differ.

Two common logging mistakes are worth avoiding:

• Logging every market-data tick at full detail. This can overwhelm storage and make incident searches slower. Prefer counters, sampled records and targeted capture for specific instruments or sessions.
• Putting account IDs, order IDs or symbols into metric labels. These create high cardinality, which increases storage and query cost. Keep unique identifiers in logs and traces instead.

Use logs, metrics and traces for different questions

Metrics show whether a service is healthy over time. Track order throughput, acceptance and rejection rates, latency percentiles, queue depth, connection state, CPU, memory and persistence lag. Use percentiles rather than averages: a healthy average can conceal a damaging p99 delay. Separate business outcomes, such as a valid risk rejection, from technical errors, such as a failed risk-service call.

Logs provide the detail behind a metric change. Traces show the time spent across service boundaries, including calls to databases, risk engines and venue adapters.

Tracing can add overhead, and sampling can omit rare failures. Keep error traces at a higher sampling rate than routine traffic, and use tail-based sampling if the tracing platform can retain a trace after seeing its slow or failed outcome.

The ranges below are typical market ranges across broker and prop-firm systems, not service targets. Architecture, geography, feed type and venue connectivity change the useful baseline.

These figures are starting points for investigation, not pass-or-fail limits. A venue-dependent round trip can be tens of milliseconds or substantially longer. Measure the platform's own distributions by route, asset class and trading session, then compare current performance with a baseline for the same conditions.

Alert on symptoms that need action

An alert should tell an operator that a user, position or control is at risk, or that a service is likely to become unavailable. A page for every exception trains the team to ignore pages. Use dashboards and ticketed warnings for trends that need review; reserve paging for conditions with a clear response and an owner.

Useful alerts usually cover a small set of failure modes:

• Order acknowledgements or fills are delayed beyond the service objective, or order state differs between the OMS and a venue adapter.
• Market data is stale, a feed has disconnected, or sequence gaps are not recovering.
• Queue age or depth is rising continuously, indicating that consumers cannot keep up.
• Risk checks are timing out or being bypassed through a fallback path.
• Rejects, cancellations or connection failures have moved sharply away from the normal pattern for that route.

Set thresholds against user impact and duration. For example, a team might page when p99 order-path latency exceeds twice its agreed objective for five minutes, rather than on one slow request.

A queue alert can require both age above 500 ms and a rising backlog, which avoids paging on a brief burst that drains normally. Treat those values as example starting settings, then tune them against observed traffic and incident history.

Use separate warning and critical thresholds, and group repeated events by service, route and cause. Include the current value, baseline, affected component and runbook link in the alert.

Avoid high-cardinality labels in the alerting system too. A page that says "risk service timeout rate is elevated in region A" is more useful than hundreds of separate pages for individual accounts.

Build an operating loop

Observability is not complete when dashboards exist. The team needs to know whether signals are trustworthy, who responds and how to improve the system after an incident. A simple implementation sequence is:

1. Map critical flows. Document order entry, risk checks, market-data distribution, execution reports and the dependencies each path uses. Mark synchronous calls, queues and failover routes.
2. Define service objectives. Set measurable limits for availability, latency and data freshness. Separate internal processing time from venue round-trip time so teams can assign ownership.
3. Instrument consistently. Standardise timestamps, correlation IDs, service names and reason codes. Check that identifiers survive retries and asynchronous processing.
4. Build alerts from incidents and objectives. Start with a small set of actionable pages. Test them during controlled failure drills, including a stopped consumer, stale feed and unavailable risk dependency.
5. Review after each incident. Check whether the first alert was timely, whether the trace exposed the failing hop and whether the log fields answered the operator's questions. Fix missing signals as well as the underlying fault.

There are trade-offs to manage. More telemetry improves diagnosis but raises ingestion, retention and query costs. Sampling reduces volume but can hide rare events. Long retention helps with intermittent faults and compliance investigations, but operational logs should not become an ungoverned archive. Set retention by data type and access need, and test queries at expected peak volume.

Teams comparing platform and infrastructure approaches can find further coverage at Trading Technology. The practical test for any setup is whether an operator can identify the affected flow, bound the impact and choose a safe action without guessing.

Trading systems still produce more events than explanations.

#monitoramento de sistemas de trading #logs de trading #infraestrutura de broker #monitoramento da execução de ordens #alertas de trading #rastreamento distribuído #monitoramento de dados de mercado #tecnologia para prop firms