Trading system observability: what to log and alert on

In week one, map the order path and agree which events must be visible before adding dashboards. By week six, the team should have spent two weeks operating the new signals, tuning noisy alerts and checking incident timelines against recorded events. The implementation takes 30 days; the following two weeks test whether it works under live operating conditions.

Observability for a broker or prop firm has to explain more than whether a server is up. It needs to show whether an order was accepted, routed, acknowledged, filled, rejected or left in an uncertain state, and how long each transition took. That means connecting application logs, metrics and traces to a reliable order or request identifier, while keeping sensitive client and strategy data out of broad-access tooling.

Set the boundary before collecting data

Start with a service map, not a vendor selection. Draw the path from client or strategy gateway through authentication, pre-trade risk, order management, routing, venue connection and execution reports.

Add market-data ingestion, positions, drop copy and downstream reporting where they affect decisions or recovery. Identify the owner of each component and the external dependencies, including venues, cloud services and network links.

For each step, record what success means and how it can fail. An order accepted by the front end but blocked by risk is different from an order accepted by the broker but not acknowledged by a venue. If the distinction is absent from logs, an incident responder will have to infer it from partial records. Agree on a common event vocabulary and time source before building charts.

Choose a small set of service-level indicators tied to operational impact. Examples include order acknowledgement latency, stale market-data duration, the rate of rejected or timed-out orders, and time to reconcile an execution report with the order state.

Set targets by product, venue and session. A liquid FX route, an exchange-traded product and an overnight API may need different thresholds.

Keep business and diagnostic records distinct. Business events such as order submission, risk decision and execution report are part of the trading record and usually need durable retention and access controls. Diagnostic logs help explain software behaviour and can have shorter retention. Confirm retention, immutability, access and export requirements with compliance and security teams before sending either data type to a general-purpose logging service. A service map also belongs in the wider broker technology stack inventory, where dependencies and owners can be reviewed together.

Log the trading path, not every byte

Use structured records with stable field names. Each event should carry an event timestamp in UTC, service and version, environment, host or pod, event name, severity, and a correlation identifier.

For order events, include the internal order ID, parent or strategy ID where relevant, venue order ID when assigned, route, instrument identifier and state transition. Record both event time and ingestion time. Their difference helps expose buffering, clock or transport problems.

Propagate a trace ID across synchronous calls and asynchronous messages. Preserve the order ID across retries, but assign each attempt its own attempt number and request ID.

That distinction makes duplicate submissions, reconnect behaviour and idempotency errors visible. For example, a timeout followed by a retry should not produce two apparently unrelated orders, nor should a repeated venue acknowledgement be counted as a second fill.

Log state changes and decisions rather than entire payloads. Useful fields include risk check outcome and reason code, route selected, queue wait, venue response code, retry count, fill quantity, remaining quantity and rejection source.

Avoid logging passwords, access tokens, full payment details or unnecessary personal data. Mask account identifiers in general operational views, and restrict access to the records that need a full identifier for investigation.

Do not sample order lifecycle events. Dropping one event from a sampled trace can make a live order appear to vanish. Sampling is more suitable for high-volume, low-value diagnostic traces, such as repeated successful health checks. For the critical path, retain a complete record of state transitions and use trace summaries or aggregation for less important traffic. Set explicit limits on payload size and field cardinality: putting a unique order ID in a metric label can multiply time-series storage, while putting it in a log field is usually manageable.

Use metrics for rates, counts and distributions, not as a replacement for event history. Histogram latency at each meaningful boundary, and retain percentiles such as p50, p95 and p99.

Averages hide tail delays that affect a small but important share of orders. Keep dimensions bounded to service, route, venue, product class and region. A useful trace should show time spent in the gateway, risk check, internal queue and venue connection, not just a single end-to-end duration.

Alert on customer and risk impact

Alert on symptoms that require action. A process restart is usually a dashboard event; a rising order acknowledgement tail, stale feed, growing queue age or mismatch between order and execution states can require a page.

Pair every alert with an owner, a severity, a runbook and a clear first check. Route pages to the team that can act, not to a broad channel that no one owns.

The figures below are typical market ranges for initial comparison, not universal service targets or audited results. They vary with venue, asset class, geography, architecture and session. Measure a representative baseline in the actual environment, then set thresholds against its normal distribution and the business impact of delay.

Prefer duration and rate alerts to fixed counts. A single rejected order may be routine; a rising rejection ratio across one route, paired with a venue code, can point to a configuration change.

Use a minimum event volume before calculating ratios, and split by venue or route only where the result will change the response. Otherwise, sparse traffic can create alerts from one event.

Use multi-window conditions to avoid paging on brief bursts. A warning might trigger when a five-minute rate is elevated and a one-minute rate is also rising. A critical alert should identify customer impact or a control failure, such as an order state that cannot be resolved, risk checks being bypassed, or a required feed becoming stale. Test alert delivery during a quiet session and during a busy one. A signal that arrives after the incident has cleared is not operationally useful.

Build it in 30 days, then prove it

Keep the first release narrow. Instrument one representative order route end to end, then extend the schema and alert model to other routes. The plan below assumes existing log and metrics infrastructure; where a platform is missing, scope that deployment separately rather than trying to solve every data-retention and tooling problem at once.

1. Days 1-3, map and prioritise. List trading services, dependencies, owners and critical sessions. Walk through normal, rejected, cancelled and timed-out order paths. Select one route with meaningful volume and establish the operational questions the first dashboard must answer.
2. Days 4-7, define the event contract. Agree event names, required fields, identifier propagation, timestamp handling and redaction rules. Document which business records are durable and which diagnostic data can be sampled or expire sooner. By the end of week one, agree on the order-state vocabulary and owners.
3. Days 8-12, instrument the critical path. Add structured events at submission, validation, risk decision, routing, venue response and execution-report handling. Carry correlation IDs through queues and retries. Check for duplicate events, missing state transitions and inconsistent clocks using test orders in a non-production environment.
4. Days 13-17, add metrics and traces. Publish latency histograms, event rates, queue age, rejection ratios and reconciliation lag. Build traces across service boundaries where they are useful, but do not depend on a sampled trace to reconstruct the durable order record. Check metric cardinality and storage use before expanding dimensions.
5. Days 18-22, create actionable alerts. Set baseline thresholds by route and session, add minimum-volume guards, and route warnings and pages to named owners. Link each alert to a runbook with the first diagnostic query, escalation path and safe action. Keep the initial page set small enough for teams to acknowledge and review every trigger.
6. Days 23-26, test failure modes. In a controlled environment, introduce delayed acknowledgements, dropped messages, a stale feed, a growing queue and a broken downstream dependency. Confirm the right alert fires, the event trail shows where progress stopped, and recovery does not create duplicate orders or inconsistent positions. Record detection and diagnosis time.
7. Days 27-30, roll out and hand over. Enable the route for production observation, compare live distributions with the test baseline and adjust thresholds with service owners. Review access, retention, dashboard ownership and on-call coverage. Schedule a day-30 incident exercise and assign follow-up work for gaps that cannot be fixed before release.

Use the next two weeks to review every page and warning. Track false positives, missed incidents, time to acknowledge and time to identify the failing boundary. If a signal fires without changing a decision, remove it or make it more specific. If an incident requires manual database searches, add the missing field or query rather than another dashboard panel. By week six, the useful result is a small set of trusted signals and an event trail that can answer what happened to an order.

Stop doing these things:

• Stop paging on CPU or memory thresholds without evidence of trading impact.
• Stop logging full order payloads, credentials or unmasked client data by default.
• Stop sampling away order-state transitions and treating traces as the audit record.
• Stop adding high-cardinality identifiers to metric labels.
• Stop keeping alerts with no named owner, runbook or tested response.

#infraestructura del broker #trazabilidad distribuida #trading platform monitoring #order lifecycle logging #trading system alerts #market data monitoring #operational metrics #incident response