Error 429: What a Rate Limit Is and How to Retry Properly

In short

A 429 isn't a bug in your code or an outage at the vendor. It's a "too often, wait" message. Trouble starts when your code reacts with an immediate retry: you knock on a closed door even harder, the limit doesn't let go, and what was a one-second pause turns into an avalanche.

What you need to know, in order:

1. There are several limits, and you can hit any of them - requests per minute, tokens per minute, concurrent connections.
2. The response often includes Retry-After - an exact number of seconds. It's the thing people ignore most often.
3. Retries need a growing delay and jitter, or all your workers come back at the same moment.
4. Retrying doesn't cure overload - when you're over the limit consistently, you need a queue, not retries.

A rate limit is a cap on how many requests or tokens you can send per unit of time. Every model vendor has one, and it exists so that a single client can't take up everyone else's capacity.

Which limit you actually hit

The same 429 code comes back for different kinds of excess, and each one has a different fix.

• RPM, requests per minute. You hit it if you send lots of small requests: classification, field extraction, short answers. The fix is batching - several items in one request.
• TPM, tokens per minute. You hit it if you send few but huge requests: long context, whole documents. The fix is trimming the context, not pausing.
• Concurrent requests. You hit it if you run more workers than the allowed number of connections. The fix is a semaphore on your side.
• Daily or monthly quota. Retries are pointless here: nothing changes until the quota resets.

The first thing to do when 429s become regular is to look at the response body. Vendors state there which limit was triggered, and that saves you a day of guessing.

🔴 Tell 429 apart from 529 and 503. A 429 means you exceeded your limit, and your load profile is to blame. 529 and 503 mean overload on the vendor's side, and your profile has nothing to do with it. A retry makes sense in both cases, but the conclusions differ: in the first, fix your load; in the second, have a fallback model.

Retry-After: the number you've already been given

A significant share of 429 responses include a Retry-After header - how many seconds to wait. If it's there, there's nothing to guess.

import time, random

def retry_after_seconds(err, attempt: int) -> float:
    headers = getattr(err, "response", None)
    headers = getattr(headers, "headers", {}) or {}
    raw = headers.get("retry-after")
    if raw:
        try:
            return float(raw)
        except ValueError:
            pass
    # no header - exponential backoff with jitter
    return min(60.0, 2 ** attempt) * (0.5 + random.random())

Two things here matter equally.

Exponential growth. The delay grows: 1, 2, 4, 8 seconds. A constant one-second pause under a sustained excess never gives the limit room to recover.

Jitter. The 0.5 + random() multiplier spreads retries out in time. Without it, twenty workers that got a 429 in the same second also come back in the same second - and get a 429 again. That's the classic retry storm, and it's prevented by exactly one line with a random number.

The full retry

RETRIABLE = {429, 500, 502, 503, 529}

def ask(messages, *, model="claude-sonnet-5", attempts=5, **kw):
    for attempt in range(attempts):
        try:
            return client.chat.completions.create(
                model=model, messages=messages, **kw
            )
        except Exception as err:
            code = getattr(err, "status_code", None)
            if code not in RETRIABLE or attempt == attempts - 1:
                raise
            delay = retry_after_seconds(err, attempt)
            log.warning("%s, retrying in %.1f s (attempt %d)", code, delay, attempt + 1)
            time.sleep(delay)

What's deliberate here:

• The list of retriable codes is closed. 401 (wrong key), 400 (malformed request) and 402 (out of money) are never retried - a retry won't fix them and only clutters the log.
• Five attempts, not infinity. An endless retry turns a short glitch into a request that hangs forever, tying up a connection and a worker.
• Every retry is logged with the code and the delay. Without that, you won't find out that half your requests go through on the second attempt until the bill for double the running time arrives.

🔴 Retries cost money and time. A failed request that reached the model may already have spent tokens. And five attempts with exponential backoff can mean up to a minute of waiting in a user-facing flow where patience runs out after about ten seconds. For interactive requests, use two attempts and a short ceiling, and leave long retry chains to background jobs.

When retrying doesn't help

A retry is designed for a short burst. If 429s come constantly, you're not in a burst but over the limit for good, and retries only add load.

Three approaches that work:

Limit concurrency on your side. A semaphore on the number of simultaneous requests is the cheapest measure. Twenty workers, each with its own retries, produce an avalanche; five workers with a queue produce a steady flow.

import asyncio
sem = asyncio.Semaphore(5)

async def ask_limited(messages, **kw):
    async with sem:
        return await ask_async(messages, **kw)

Batch small requests. If you're hitting RPM, ten items in one request give you ten times the headroom under the limit for the same token volume. This works for classification and extraction, not for dialogue.

Spread the load across models. Limits are counted per model, not for the whole account. Bulk background processing on a lightweight model and interactive answers on a flagship don't get in each other's way.

What a gateway does, and what's left to you

A unified gateway takes part of the work off your hands: when a provider fails, the request goes to a backup one, and several providers of the same model add up to more combined headroom under the limits (how failover works). That noticeably reduces how often you see 429s, but it doesn't replace retries.

What the gateway won't do for you: it doesn't parse Retry-After, doesn't put a pause in your code and doesn't cap your concurrency. Paying in rubles and having a single key make billing and switching models easier - they have no effect on your load profile. Retries and the semaphore live on your side either way.

Where to start

1. Log your response codes for a week. Until you know your share of 429s, everything else is guesswork.
2. Read Retry-After and use it instead of your own constant.
3. Add exponential backoff with jitter - it's two lines and the biggest win.
4. Add a semaphore on concurrent requests, and pick the number by measuring.
5. Separate interactive and background load - by model and by number of attempts.

All models with prices and context sizes are in the catalog. What each error code means and whether to retry it is in the errors reference, and failover to a backup provider is covered in reliability.

FAQ