API Rate Limit Retries for Reliable Media Delivery
Handle provider 429s without retry storms. Learn how to set retry limits, preserve delivery identity, track costs and choose the right monitoring setup.
How to Handle API Rate Limits in Media Alert Systems
A media alert worker gets a 429 Too Many Requests response. Should it retry immediately, wait, or give up? The right choice affects more than delivery speed. A sloppy retry loop can overload a provider, grow a queue, and make one notification look like several "billable" deliveries. Stop too early, though, and a temporary quota limit can turn into a missed alert.
A reliable policy treats 429 as a scheduling signal, not a reason to create a brand-new notification. Keep the same delivery identity, wait before trying again, and stop after a defined limit. Then record what you did so the team can see which provider, campaign, or tenant was impacted - and what happened next. That record is what you'll need when you investigate delays or explain delivery costs later.
This guide builds that policy from the worker outward: how to classify the response, design bounded retries, preserve useful operational data, and set up monitoring that actually covers the gaps. It also covers when retries are the wrong tool. The goal isn't "make every notification succeed eventually." The goal is to recover from transient rate limits without turning a small issue into a bigger incident.
What should a media alert service do when an API returns 429?
A 429 means the provider is telling the client to slow down. The worker should respond by changing when it schedules the next attempt, not by changing what the notification means. It's still the same campaign, recipient, and delivery - you shouldn't treat it like a new notification with a new identity and a new cost record.
That distinction matters operationally. If a retry creates a new delivery record, a campaign that requested one alert can look like several. The same mistake can also lead to duplicates if the provider accepted an earlier request but the response was delayed or lost. Keep one stable delivery key from the queue entry through the final outcome.
Treat 429 as a control signal
For a rate-limit response, a worker should generally:
- Record the response and the current attempt.
- Calculate a delay using your retry policy and any provider guidance.
- Schedule the same delivery for a future attempt.
- Stop when the configured retry budget is used up.
- Emit a terminal result operations can handle (or product policy can map to a user-facing outcome).
This is different from retrying every kind of error. If the request is malformed, retrying won't fix it. If authentication is wrong, you need a credential or permission change, not another attempt. A 429 is often temporary, so delayed retry can help - but only within a bounded window.
Separate delivery policy from business policy
The worker owns transport decisions: how long to wait, how many attempts to make, and how to handle cancellation. The application or product owns what happens when delivery still fails. Depending on the alert's value, the terminal outcome might be a dead-letter queue, a manual replay, a fallback channel, or an explicit drop.
Keeping those responsibilities separate prevents a generic retry helper from making business decisions it can't understand. A push alert for breaking news might have a short useful delivery window; a routine digest might still matter hours later. Both can use the same retry mechanism, as long as expiry and escalation rules differ.
How do you design bounded retries?
Use exponential backoff with a hard attempt limit, and add jitter so workers don't all retry in sync. The delay grows after each rate-limit response, which gives the provider time to recover and reduces the chance your retry traffic creates another burst.
Your attempt limit shouldn't be a random universal constant. Five attempts might work as a starting point for a short-lived notification, but the real budget depends on provider quotas, queue capacity, and how long the alert remains useful. Count the initial request as an attempt, and then make the limit explicit in configuration and dashboards.
Respect Retry-After without giving up control
If the provider includes Retry-After, use it as scheduling guidance. It may be a number of seconds or an HTTP date, so parse whatever formats your client supports (and test both). Don't wait forever: apply a maximum delay and keep the attempt limit enforced even when the provider asks for a long pause.
A practical delay policy looks like this:
- Calculate an exponential delay from the attempt number.
- If the provider's
Retry-Afterasks for a longer wait, use it - up to your maximum delay. - Add bounded jitter to avoid synchronized retries.
- Ensure the next retry won't exceed the delivery's expiry or queue deadline.
The ordering matters. If you add jitter after enforcing the max-delay cap, you can accidentally exceed the limit. Clamp after combining backoff, provider guidance, and jitter.
Keep the retry loop small and explicit
The Go example below shows the worker boundary. The callback returns the status code and a parsed retry delay; request construction and Retry-After parsing stay in the provider-specific client. The policy retries only 429 responses, stops on other statuses, and lets context cancellation interrupt the wait.
func sendWithBackoff(
ctx context.Context,
send func(context.Context) (int, time.Duration, error),
) error {
const maxAttempts = 5
const initialWait = 250 * time.Millisecond
const maxWait = 30 * time.Second
for attempt := 1; attempt <= maxAttempts; attempt++ {
status, retryAfter, err := send(ctx)
if err != nil {
return err // Classify network errors separately if they are retryable.
}
if status >= 200 && status < 300 {
return nil
}
if status != http.StatusTooManyRequests {
return fmt.Errorf("delivery failed with status %d", status)
}
if attempt == maxAttempts {
return fmt.Errorf("delivery stopped after %d attempts", attempt)
}
wait := initialWait * time.Duration(1<<(attempt-1))
if retryAfter > wait {
wait = retryAfter
}
wait += time.Duration(rand.Int63n(int64(initialWait)))
if wait > maxWait {
wait = maxWait
}
timer := time.NewTimer(wait)
select {
case <-ctx.Done():
timer.Stop()
return ctx.Err()
case <-timer.C:
}
}
return errors.New("retry budget exhausted")
}
This is a starting point, not a full queue implementation. In production, make sure the worker's execution deadline allows the selected waits, and consider scheduling a delayed queue message instead of blocking a worker thread. Also decide which network errors are retryable - this example returns them immediately so the behavior stays visible instead of silently retrying everything.
Key takeaways
- A retry is another attempt for the same delivery, not a new notification.
- A retry limit protects both the provider and your own queue.
- Provider guidance, jitter, maximum delay, and delivery expiry have to work together.
- Cancellation and terminal outcomes should be explicit.
What should you log to understand delivery cost and reliability?
A retry counter alone doesn't explain what happened. You need a stable delivery identifier to group attempts, plus a clear attempt number (or attempt identifier) to tell them apart. Then the team can distinguish "one notification with four attempts" from "four separate notifications," and attribute the work to the correct tenant, campaign, or provider.
For a rate-limited delivery, log enough structured fields to reconstruct the worker's decision:
- Delivery ID and attempt number
- Provider, tenant, and campaign identifiers (when applicable)
- HTTP status and a normalized outcome like
rate_limited - Retry delay and scheduled retry time
- Queue age, delivery deadline, and terminal result
- Trace or span identifier (if you already have one)
Avoid logging message bodies, recipient details, or credentials. Keep operational logs focused on the decision without copying sensitive payloads into another system. Use stable, non-sensitive identifiers where you can, and follow your retention and privacy requirements. In the UK and EU, that includes thinking through GDPR obligations for personal data in logs.
Record decisions, not just failures
A useful event trail can include one event when the provider returns 429, another when you schedule the retry, and a terminal event when the delivery either succeeds or gets abandoned. If event volume is a concern, store attempt-level data in the delivery store and export only what you need for operational analysis. The key requirement is that retries remain queryable under the same delivery key.
Make the event write durable enough for its purpose. If the worker writes to an external event service and times out, the service might have accepted the event even if the worker didn't see the response. Use an idempotency key for the same logical event or delivery operation so repeated writes don't create duplicate accounting records. An idempotency key doesn't replace the stable delivery ID - it solves a different problem.
Logs, metrics, and traces answer different questions. Logs explain individual decisions. Metrics show patterns, like the percentage of deliveries that exhaust their retry budget. Traces connect work across services when context is propagated. Correlating these is helpful, but a log search isn't a substitute for proper trace views or for alert evaluation.
Define "success" in product terms. For example, a notification might count as successful only if the provider accepts it within the allowed delivery window. Track how many miss that window or exhaust retries separately from provider response time - otherwise a service can look fine at the request level while the queue quietly accumulates alerts that arrive too late to matter.
Which monitoring approach fits a media delivery workflow?
Observability products don't replace the retry mechanism. The worker owns scheduling, deduplication, and delivery identity. Monitoring helps teams investigate failures and - depending on your setup - surface issues and notify operators. Pick tools based on the responsibility you need covered, not on the idea that "installing dashboards" automatically fixes queue semantics.
| Option | Useful fit | What it does not replace |
|---|---|---|
| Sentry | Grouping application errors and investigating failed operations | Delivery scheduling, retry ownership and per-attempt cost attribution |
| Datadog | Teams that want metrics, logs, dashboards and alerting in one operations platform | Application-level idempotency and clear ownership for alert rules |
| Grafana Cloud | Teams already using Grafana-style dashboards and metrics, with alert rules close to that workflow | Queue behavior, delivery deduplication, or the work of defining useful thresholds |
| REST-based observability service | Teams that want a plain HTTP integration across languages and can manage surrounding operations | Alert evaluation or escalation unless those features are explicitly provided |
The trade-off is usually integration breadth versus clear ownership. A consistent HTTP interface can cut down on the number of SDKs you maintain, especially if your workers use multiple languages. It can also make schemas easier to inspect. But a bigger platform surface means more things to review, govern, secure, and budget for - so check the actual features and operational boundaries before committing.
Don't page on every retry. One 429 can be normal quota behavior, and alerting on each attempt turns a recoverable event into constant noise. Instead, alert on outcomes that matter: a rising rate of exhausted retry budgets, an oldest-pending-message threshold, a provider-specific error spike, or a delivery SLO breach. Set thresholds using normal traffic patterns and the business impact of delayed alerts.
Keep escalation separate from the retry loop. A log or error-capture service can preserve evidence without triggering a page, SMS, or webhook. If the tool you chose doesn't support the alert route you need, use a separate alerting system or a polling job with a clear owner. A job that silently stops running is a different failure mode - it needs a worker health check, not more elaborate backoff.
When should you stop retrying?
Backoff is for failures that might clear with time. It isn't a one-size-fits-all policy for every failed request. Retrying a bad request just wastes capacity; retrying an authorization error without changing credentials just creates noise. Classify errors first, then decide whether to retry.
A useful starting classification is:
- 429: Retry after a delay, respecting provider guidance and a bounded budget.
- 400 validation errors: Don't retry unchanged; quarantine or fix the request.
- 401 or 403: Stop and investigate credentials or permissions.
- 5xx responses or transient network failures: Consider bounded retries, but account for provider behavior and duplicate-delivery risk.
- Expired or no-longer-relevant alerts: Stop once the business deadline has passed.
Be careful with the last two cases. A timeout doesn't prove the provider rejected the request. If it accepted the notification before the connection failed, retrying without an idempotency mechanism can send a duplicate. Use a stable provider-supported idempotency key where available; where it isn't, decide whether duplicates are acceptable and make that trade-off explicit.
Capacity planning should include the retry multiplier. If incoming work is already close to the provider's quota, retries can deepen the backlog. Measure queue age and the age of the oldest pending delivery - not just API response time. Reserve enough worker and queue capacity for expected retries, and make the delivery SLO include the retry window. A five-attempt policy is a starting point to test, not something to copy blindly without checking quota, expiry, and business value.
Before shipping a policy, answer these questions:
- Which response codes and network errors are retryable?
- Does the provider send
Retry-After, and how will you cap it? - Is the attempt limit compatible with the notification's expiry?
- Can retries be deduplicated at the provider or consumer boundary?
- Where does an exhausted delivery go, and who owns replay or escalation?
- Which metric shows delivery degrading before users report it?
If those answers aren't clear, adding retries will make the system harder to reason about - not more resilient.
Conclusion
A sound rate-limit policy has three parts: bounded backoff, stable delivery identity, and useful operational evidence. Respect Retry-After within a maximum wait, add jitter, stop at a clear attempt limit, and make the terminal outcome visible. Keep retries limited to failures that may recover, and make duplicate-delivery behavior explicit.
In practice, the decision is straightforward: retry to recover a delivery, stop before retries become a second load problem, and keep enough context to explain what happened. Choose monitoring around the gaps your team needs to cover - no dashboard or error tracker replaces an idempotent worker and an owned escalation path.