Broker API rate limits need queues, budgets and backoff

Operations engineer checking a trading dashboard during a broker API rate-limit incident
Photo by Rafael Minguet Delgado on Pexels

At 16:42 on Friday, an operations lead is looking at a dashboard that has been red for most of the week. Order acknowledgements are late, market-data panels have gaps, and the API error rate has climbed above 8 per cent.

The broker is reachable. The firm's own clients are making too many calls at once.

This composite incident is typical of a shared-credential design under load, not a report of one firm's results. Four services use the same broker API key: a web application, a position monitor, a history exporter and a reconciliation worker.

At the market open, each retries requests as soon as it receives HTTP 429. Instead of letting the request rate fall, the retries keep the shared limit exhausted.

The Friday dashboard: one limit shared by four services

Reconstruct the actual budget

The incident review starts with the broker's quota documentation and response headers, not with a change to the client timeout. Suppose the account has a documented allowance of 1,200 requests per minute and a burst capacity of about 80 requests. Those are illustrative figures. Providers may apply different quotas by account, credential, endpoint, product tier or IP address, and may change them without changing the headline limit.

At 09:30, the four services produce about 280 requests in three seconds. The web tier contributes quote and position refreshes; the history exporter runs a catch-up job; the monitor polls accounts that have not changed; and reconciliation makes its scheduled calls. A burst bucket with capacity 80 can be emptied quickly, even if the minute's total would otherwise fit the documented allowance. Repeated immediate retries then add traffic precisely when capacity is scarce.

Before tuning, collect the following for at least one busy session:

  • Request counts by credential, endpoint, service and response code.
  • 429 responses, any Retry-After values, and rate-limit or reset headers where supplied.
  • Request concurrency, latency, timeouts and the age of queued work.
  • Whether several application instances share the same quota or run independent local limiters.

Use a monotonic clock for local intervals, record the broker's reset time where available, and synchronise host clocks. A 429 without a reset header is still useful evidence. Track it alongside the request that triggered it and the credential's total traffic. Teams following trading infrastructure coverage will recognise the wider operational point: endpoint limits are part of capacity planning, not an exception-handling detail.

Separate call volume from user demand

A user opening a positions page does not necessarily require a fresh REST call for every row. The application can serve a recent snapshot, subscribe to streaming updates where the API permits, or batch several instruments into one request. But reducing calls indiscriminately can make risk controls stale. Decide which data can be delayed and how far, then make that contract visible to downstream services.

The table gives typical market ranges for planning, not guaranteed entitlements. The unit and scope matter: a per-key allowance shared across replicas is not a per-process allowance.

API operationTypical rangeCommon scope or constraint
REST quotes or snapshots30-600 requests per minutePer key or account, sometimes with a burst bucket of 5-30 calls
Historical bars or trades10-120 requests per minuteOften lower than quote limits, with 2-10 concurrent jobs
Order submit, cancel or amend5-30 requests per secondMay be split by operation, account or trading session
Streaming subscriptions100-1,000 symbols per connectionConnection count and subscription changes may have separate caps

These ranges vary substantially by broker, venue access and commercial tier. Confirm whether a quota counts failed calls, whether batch members count individually, and whether order actions have a separate risk-control ceiling. Do not infer streaming capacity from a REST quota.

Build a client-side governor before adding retries

Use one scheduler per credential

Each process having its own limiter is a common failure mode. Four replicas each configured for 20 requests per second can send 80 requests per second against one shared credential. Put the budget at a point all callers share: a central API gateway, a distributed token bucket, or a single queue worker for each credential and endpoint class.

A token bucket is useful when the broker permits short bursts as well as a sustained rate. Set the refill rate below the documented cap, then reserve some capacity for operational calls. For a nominal 1,200 requests per minute, a starting sustained budget might be 15-18 per second rather than the theoretical 20. A burst capacity of 30-50 requests leaves room for a legitimate user action without allowing a batch job to consume the whole account budget.

Separate queues by consequence and freshness. Order submissions, cancels and risk-reducing actions should not sit behind a history backfill. Quotes and positions may have different freshness targets. A typical design might set a 100-300 ms queue target for interactive order actions, 1-3 seconds for routine position refreshes, and seconds or minutes for historical data jobs. These are starting points for testing, not universal service levels.

Set explicit queue limits. An unbounded queue converts a rate-limit event into delayed work that arrives after it is useful. When the queue is full, coalesce duplicate reads, drop stale refreshes, pause batch jobs or return a clear degraded response. Do not silently discard an order command.

Make retries safe

On a 429, honour Retry-After if present. If the response gives a reset time, schedule against it with a small random spread so every worker does not wake together. Without either header, use exponential backoff with full jitter. A practical starting sequence is a random delay from 0-250 ms, then 0-500 ms, then 0-1 second, capped at 5-10 seconds. Add a small retry budget, such as three attempts for a read, rather than retrying indefinitely.

Classify calls before retrying:

  • Retry idempotent reads after a bounded delay, subject to the caller's freshness deadline.
  • Use an idempotency key for order submissions if the API supports one.
  • If an order request times out after sending, query order state before resubmitting. The server may have accepted it even though the acknowledgement was lost.
  • Do not retry validation errors or permission failures as though they were capacity errors.

Backoff controls pressure, but it does not resolve ambiguous order state. The order-management layer needs a state machine that can represent submitted, acknowledged, rejected, timed out and reconciliation-pending states.

This is where a retry policy meets trading risk. A duplicate market-data GET wastes capacity; a duplicate order can create exposure.

Backoff must distinguish delay from recovery

A 429 says the caller exceeded a limit. It does not say whether the whole API is unhealthy, whether a particular endpoint is constrained, or whether a minute-based quota will reset shortly. Track throttling by endpoint and credential, and compare it with 5xx errors, network timeouts and latency. Those signals require different responses.

When throttling rises, reduce optional traffic first. Pause history exports, slow background account polling and serve cached values within their allowed age. Preserve capacity for order state checks and risk-reducing actions. If the broker provides separate order and market-data limits, enforce separate budgets. If it uses one shared pool, keep a reserved fraction for orders and reconcile the actual allocation against observed headers and support documentation.

Recovery needs a ramp, not a simultaneous restart. After the reset window, release queued work at the normal budget and increase cautiously if the 429 rate remains low.

A circuit breaker can stop a failing endpoint from attracting more calls, but should not block other endpoint classes without evidence that they share the same failure. Alert on queue age and stale-data duration as well as raw error percentage. A 0.2 per cent 429 rate can still matter if it affects every order acknowledgement at the open.

Test the design with a load profile that resembles the actual system: scheduled refreshes, user bursts, reconnects after a network interruption and a batch job starting at the same time. Include broker responses with and without rate-limit headers.

Check that independent replicas share the budget, that restarts do not refill a local burst bucket, and that an order timeout does not trigger a blind duplicate. The broker technology resource is useful context, but the binding limits must come from the API provider and the firm's own measurements.

Prove the policy under load, then set stop rules

Before deployment, define what a healthy throttling policy looks like. For example, a firm might aim to keep sustained usage below 80-90 per cent of its observed allowance, hold interactive queue latency below its service target, and keep background work from delaying order operations. Choose thresholds from the broker's actual quota and the firm's risk controls, not from a generic dashboard template.

Run a controlled test at 1.2-1.5 times expected peak traffic, while limiting the test to a safe environment or approved account. Verify that excess reads queue or shed load, order actions retain reserved capacity, and recovery does not create a second burst. Review these operational measures after each release:

  1. 429 rate and total request rate, split by endpoint, credential and service.
  2. Queue depth, oldest-item age, dropped or coalesced reads, and retry count.
  3. Order acknowledgement latency, unresolved submissions and duplicate-order checks.
  4. Data freshness for quotes, positions and account state during degraded operation.

Set an owner for quota changes and document the provider's escalation path. If the firm needs more capacity, ask whether the provider can raise the limit, separate credentials by workload, or expose a streaming or batch alternative. Adding keys to evade a shared account limit can violate terms and make incident analysis harder. Treat a quota increase as one option, not the substitute for a traffic model.

Stop doing these things:

  • Retrying every 429 immediately.
  • Giving each replica an independent copy of the same quota.
  • Putting history backfills in the same queue as order actions.
  • Resubmitting timed-out orders without checking their state.

Frequently Asked Questions

What should a trading API client do after HTTP 429?

Honour the provider's Retry-After or reset header, then retry with jitter if the response gives no timing guidance. Limit attempts and reduce optional traffic while the credential is constrained.

Should broker API rate limits be shared across application replicas?

Yes, if the broker applies the quota to a shared key, account or IP. Use a central scheduler or distributed limiter so the combined traffic stays within one budget.

Is it safe to retry a timed-out order request?

Not blindly. The broker may have accepted the order even if the acknowledgement was lost, so check order state first or use a supported idempotency key before resubmitting.