Trading system observability needs risk-led logs and alerts

A green dashboard is not evidence that a trading system stayed within its control envelope. If a broker cannot reconstruct an order's path, identify the risk decision applied to it and show when a control degraded, observability has failed where it matters most: risk and compliance.

That changes the design goal. The aim is not to collect every possible event. It is to retain enough reliable, time-aligned evidence to detect harmful conditions, investigate them and demonstrate what the system did.

Start with the control path

Map the path from client instruction to venue acknowledgement, including the points where a risk engine can reject, amend or hold an order. Record each meaningful transition against a stable order or request identifier. That identifier should survive retries and pass between the API gateway, order management system, risk service, execution connector and post-trade systems.

For each transition, capture an event timestamp in UTC, the producing service and version, the event type, the relevant account or instrument identifier, and the outcome. Where clocks cross machines, monitor synchronisation as well as event time. A 20 millisecond clock skew can distort sequencing in a fast execution path; during an incident, a plausible but incorrect timeline is worse than a clearly marked gap.

Trading Technology's coverage of trading infrastructure regularly shows why the boundary between application behaviour and operational control deserves attention. Instrument the control path before adding generic dashboards.

Log decisions, not secrets

Application logs explain what code did. Risk evidence should also explain why it did it. For a reject or limit action, log the rule identifier and version, input values used in the decision, resulting exposure or limit state, decision outcome and processing time. Keep the record compact and structured, rather than writing a free-text explanation that cannot be reliably queried.

Do not put credentials, API keys, full payment data or unnecessary personal information into logs. Use stable internal account identifiers and apply role-based access, encryption and access logging to the observability platform itself. A log store can contain sensitive trading behaviour and should not become an unrestricted copy of production data.

Keep a separate, durable audit trail for events that must be reconstructed over the applicable regulatory and contractual retention period. Observability indexes are often optimised for short-term search, not long-term evidential integrity. Define retention by record type and jurisdiction with compliance and legal teams; do not assume that a default vendor setting meets the firm's obligations.

• Include decision inputs and rule versions, but exclude secrets and irrelevant client data.
• Record changes to limits, permissions, routing configuration and deployment versions with actor, timestamp and before-and-after values.
• Track dropped, delayed or sampled telemetry. Missing evidence must be visible, not silently treated as a clean result.

Alert on exposure and broken controls

Page on conditions that need action, not on every unusual metric. A useful alert identifies the affected service or control, the observed value, the threshold, the time window and the runbook or accountable team. Separate a risk breach from a warning that a system is approaching one.

Typical alert candidates include:

• Orders accepted while a required pre-trade risk check is unavailable, bypassed or timing out.
• Exposure or order-rate limits breached, or a material rise in rejects caused by a control rule.
• Market data age exceeding the strategy or product's tolerance, such as 250 milliseconds for a latency-sensitive feed, or several seconds for a less time-critical instrument.
• Venue acknowledgement latency or rejection rate moving outside its normal band, especially when retries are building a queue.
• Position or cash differences between the order system and downstream records that remain unresolved beyond the agreed reconciliation window.

Thresholds depend on product, venue and operating model. Use a short window to catch acute failures and a longer window to detect gradual deterioration.

For example, alert on a sharp increase in order rejects over five minutes, then compare the rate with a 30-minute baseline to distinguish a brief burst from a persistent fault. Route a breached hard limit to the on-call operator immediately; send a trend warning to the owning team without waking the full rota.

Design for signal, not alert volume

Alert fatigue is a control weakness. If an operator receives repeated pages that do not require action, important pages are more likely to be missed. Group alerts by incident, suppress symptoms when a known upstream failure explains them, and keep an audit record of acknowledgements, escalations and closure reasons.

Set severity according to potential harm and time to intervene. A risk check failing open may demand an immediate page and a defined fail-closed action. A non-critical reporting dashboard lagging by two minutes may only need a ticket. Decide these behaviours before production incidents. Failing closed can protect limits but interrupt legitimate trading; failing open preserves flow but creates unbounded exposure. The correct response depends on the specific control and must be explicit.

Check telemetry health as carefully as trading health. Alert if log delivery stops, a collector queue grows beyond its capacity, or clock offset exceeds the firm's tolerance. Otherwise, the apparent absence of an incident may simply mean the evidence pipeline stopped working.

Budget for retention and search

Observability costs usually follow ingestion volume, indexed fields, query load and retention. High-cardinality labels such as order IDs are valuable in logs and traces, but expensive as metric dimensions. Keep metrics aggregated by service, venue and outcome; use event identifiers to search individual transactions in the log store. The figures below are typical market ranges for a mid-sized trading operation, not audited company results. Vendor pricing, region, discounts and support terms vary.

These are typical ranges, excluding implementation labour, tax and premium support. Control spend by setting different retention tiers: keep searchable operational logs for a short period, archive required records separately, and avoid indexing fields that nobody uses. Sample routine traces, but retain unsampled traces for errors or high-risk transactions where practical. Review actual ingest by service before buying a larger platform tier.

For more operational context, Trading Technology's broker infrastructure reporting covers the systems and providers trading firms depend on. Use that context to frame procurement, but validate costs against your own event volumes and retention rules.

Make evidence usable under pressure

Run an incident reconstruction as a control test. Pick a rejected order, a delayed market-data update or a simulated risk-service outage.

Ask an engineer who did not build the relevant service to reconstruct the sequence from the records available to the on-call team. Measure how long it takes and note where identifiers, timestamps or decision context are missing.

Use this numbered checklist for the test:

1. Trace one order from client ingress through risk decision, routing and venue response using a single correlation identifier.
2. Verify that event times are UTC, clock offset is monitored, and retries do not overwrite earlier outcomes.
3. Confirm that a risk-control failure produces the intended fail-open or fail-closed behaviour and alerts the correct owner.
4. Check that alert thresholds identify an actionable condition, with a runbook and escalation path.
5. Confirm sensitive fields are excluded, access to records is logged, and retention matches the approved policy.
6. Record the missing evidence, assign an owner and set a date to retest the control.

This week, run a 30-minute reconstruction of one live order path and record its three most important evidence gaps.

#monitoramento de sistemas de trading #logs de trading #custos de observabilidade #monitoramento da execução #monitoramento de dados de mercado #broker risk controls #trading infrastructure alerts #audit trail