API Rate Limits: Designing for Traffic Spikes

API Rate Limits For Spikes

API rate limits cap request volume so one client, integration, or automated job cannot overwhelm shared infrastructure. During traffic spikes, the limit becomes a safety rail that turns sudden load into predictable throttling, which protects latency and error rates for everyone.

Most production APIs combine multiple controls: per-client limits, global limits, and sometimes endpoint-specific limits. A common pattern is a token-bucket limiter that refills at a steady rate while allowing short bursts. If you set the burst window too small, clients see throttling even when the service has spare capacity; if you set it too large, spikes still push queues into long tail latency.

Practical example: a mobile app calls a “search” endpoint. On launch day, the request rate jumps from 50 requests per second to 500. With a token bucket, the API can accept bursts up to the bucket size, then return 429 responses with a Retry-After header when the bucket empties. Clients that respect Retry-After recover quickly; clients that ignore it can create a feedback loop that keeps the bucket empty.

One small implementation detail that affects behavior: many systems use fixed windows by default, which can create edge spikes at window boundaries. I’ve seen this in internal load tests around a 60-second window where traffic clustered at second 0, and the limiter let through a burst twice per minute, which then caused a brief latency spike. The fix was switching to a sliding window or token bucket.

Common Pain Points During Spikes

Teams often treat rate limits as a single number, then discover that real traffic patterns depend on client behavior, caching, and retries. A “requests per second” limit can look safe in a steady-state chart, yet fail when clients retry aggressively after timeouts.

One frequent mistake is counting the wrong unit. Some APIs count requests, others count expensive operations, and others count bytes transferred. If you rate limit only requests, a large file download can consume far more CPU, bandwidth, or database time than a small JSON response. If you rate limit only bytes, a burst of tiny requests can still saturate connection pools.

Another dependency is how upstream services behave under load. If your API calls a database, a cache, or a third-party provider, the limiter must match the bottleneck. A limiter that protects the API gateway may not protect the database if the gateway still forwards too many concurrent requests. Concurrency limits and queue limits often need to work alongside request-rate limits.

Retry storms are the classic failure mode. When clients receive 429 or 503, they should back off. When clients receive timeouts, they often retry without reading headers, and the retries add load that increases timeouts. This is why rate limiting design includes both server-side throttling and client-side retry rules.

Finally, observability gaps hide the problem until it becomes visible to users. If you only log 429 counts, you miss the distribution of response times, queue depth, and the correlation between throttling and downstream saturation. In one incident review I read, the team graphed only total 429 responses and missed that 429s rose after the database CPU hit 90%, not before. That meant the limiter was reacting too late.

Designing Limits That Hold

Choose A Limiter Model

Pick a rate-limiting algorithm that matches burst tolerance. Token bucket supports steady refill plus bursts; leaky bucket smooths traffic more aggressively; fixed window is simple but can create boundary spikes. If your traffic spikes last seconds, token bucket with a burst size sized to your normal burst tolerance often behaves better than fixed windows.

Set limits per client identity. Identity can be an API key, OAuth subject, IP address, or a combination. Using only IP can punish users behind NAT; using only API key can be bypassed by key sharing. Many teams use a primary key (API key or OAuth) plus a secondary guardrail (IP or network range) for abuse control.

When you test, measure the limiter’s effect on latency percentiles, not just error counts. A limiter that reduces 429s by raising the limit can still worsen p95 latency if it increases queueing. I’ve used k6 v0.49 in a staging environment to compare token bucket vs fixed window and watched p99 latency shift even when the average request rate stayed constant.

Set Server And Client Rules

Return 429 for rate limit violations and include Retry-After when possible. If you use a token bucket, Retry-After can be derived from time until enough tokens refill. Clients should treat 429 as a signal to slow down, not as a reason to retry immediately.

Define retry behavior for 429 and for transient errors like 503. A typical pattern is exponential backoff with jitter and a cap on total retry time. For example, allow retries for up to 30–60 seconds for idempotent requests, and avoid retries for non-idempotent operations unless you have a safe deduplication key.

Also separate “rate limit” from “concurrency limit.” A server can accept requests at the rate limit but still fail when concurrency exceeds downstream capacity. Many systems use a concurrency limiter (like a semaphore) plus a queue with a maximum wait time, so clients get a fast 429/503 rather than timing out after 30 seconds.

Plan For Quotas And Costs

Requests are not equal. If one endpoint performs a costly search or calls a third-party API, rate limit it differently from cheap endpoints. Use endpoint-level limits or cost-based limits where each request consumes tokens proportional to estimated cost.

Quotas work well for long-term fairness. For example, a monthly quota of 100,000 requests per developer account can prevent sustained abuse even when short-term bursts are allowed. Quotas also help with billing models, but they require clear error semantics when the quota is exhausted.

When you set numbers, start from measured capacity. If your API can sustain 200 requests per second at p95 latency under 200 ms, a global limit above that number can still cause queue growth. A safer starting point is to set limits below measured saturation and then adjust using load tests that mimic real client retry behavior.

Test With Spike Scenarios

Load tests should include retries, not just a single request stream. Model client behavior: how quickly it retries after 429, how it handles Retry-After, and whether it retries on timeouts. Without this, you can design a limiter that looks correct under ideal clients and fails under real ones.

Use staged experiments. First, test steady traffic to find the saturation point. Second, test a spike that lasts 10–60 seconds with a realistic mix of endpoints. Third, test a “retry storm” scenario where clients ignore Retry-After and retry on timeouts; the goal is to confirm that your server-side limits still protect downstream systems.

Record key signals: 429 rate, 5xx rate, p95/p99 latency, queue depth, database CPU, cache hit rate, and connection pool utilization. If you see 5xx rising before 429, your limiter is not aligned with the bottleneck. If you see 429 rising but latency still grows, the system may be queueing before the limiter decision.

Case Examples From Realistic Setups

Example 1: Launch-day traffic spike on a read endpoint. An e-commerce API exposes “product details” and “inventory status.” The team sets a per-API-key token bucket with a burst size of 200 requests and a refill rate of 50 requests per second. During a campaign, traffic spikes from 20 rps to 300 rps. The API returns 429 with Retry-After once the burst is consumed, and the client respects the header, reducing retries within a minute. After the campaign, the team adjusts the burst size upward slightly because p95 latency stayed under the target while 429s were higher than expected.

Example 2: Retry storm caused by timeouts on a search endpoint. A SaaS integration calls “search” and times out at 2 seconds. When the backend slows, the client retries immediately up to 5 times. The server had only a request-rate limiter at the gateway, but the downstream search service still received too many concurrent calls. The team added a concurrency limiter and a short queue with a maximum wait time, returning 503 when the queue is full. After the change, 429/503 responses rose during the spike, but p99 latency stabilized and the retry storm stopped amplifying load.

Rate Limit Checklist And Table

Use this table to compare common approaches and decide what to measure during spikes.

Step-by-step checklist for spike readiness:

1. Identify the bottleneck: gateway CPU, database, cache, third-party calls, or thread pools.
2. Define identity for limits: API key or OAuth subject plus a secondary guardrail.
3. Choose a limiter model: token bucket for burst tolerance, concurrency limit for downstream protection.
4. Return correct status codes and headers: 429 with Retry-After, 503 when queues are full.
5. Test with realistic clients: include retries and timeouts, not just a single request stream.
6. Set alert thresholds: rising 429 with stable latency is different from rising 429 with rising p99.
7. Review after the spike: adjust burst size, endpoint-specific limits, and retry guidance.

Common Mistakes That Erode Trust

Rate limit guidance often fails because it focuses on server settings and ignores client behavior. If your documentation says “retry later” but omits Retry-After semantics, clients guess and retry too early. That turns throttling into a self-inflicted outage.

Another mistake is using one global limit for all endpoints. A cheap health check and a heavy search query share the same cap, so the heavy endpoint gets throttled too late or the health check gets throttled too early. Endpoint-level limits reduce this mismatch.

Teams also mis-handle idempotency. Retrying a non-idempotent POST after a timeout can create duplicates. Rate limiting does not fix that; you need idempotency keys or a safe deduplication strategy, and you need clients to use them.

Finally, teams sometimes hide throttling behind generic errors. If clients see only 500 responses during spikes, they cannot distinguish overload from bugs. Returning 429/503 with consistent headers gives clients a stable signal, and it reduces support tickets.

FAQ

What HTTP Status Code Should I Use For Rate Limits?

Use 429 Too Many Requests when the client exceeds a configured request rate. Include Retry-After when you can compute when the client may retry.

How Do Token Buckets Differ From Fixed Windows?

Token buckets refill continuously and allow bursts up to the bucket size. Fixed windows reset at discrete boundaries, which can permit bursts at the start of each window.

Should I Rate Limit By IP Address Or API Key?

Use API key or OAuth subject as the primary identity for fairness across shared networks. Add an IP-based guardrail for abuse patterns, since IP-only limits can penalize NAT users.

What Should Clients Do When They Receive 429?

Clients should stop sending requests immediately, wait for the Retry-After duration, and then retry with exponential backoff and jitter if more retries are needed.

How Can I Prevent Retry Storms During Spikes?

Return 429/503 quickly, include Retry-After, cap total retry time, and separate concurrency limits from request-rate limits so downstream services do not get flooded.

Author's Insight

Rate limits work best when they match the bottleneck and the client retry model. Token bucket logic handles burst tolerance, while concurrency limits protect downstream services that degrade under parallel load. Observability should track not only 429 counts but also queue depth and latency percentiles to confirm that throttling decisions happen early enough. For spike testing, include realistic client retries and timeouts, because the server’s behavior under ideal traffic often differs from behavior under retry storms.

Key Takeaways

Design rate limits around burst tolerance and downstream capacity, not around a single “requests per second” number.

Use 429 with Retry-After and pair request-rate limits with concurrency limits to prevent queueing and retry amplification.

Test with spike scenarios that include retries, then tune endpoint-specific limits and burst sizes based on latency percentiles and downstream saturation signals.