Thundering Herd
A thundering herd is a load spike when many clients perform the same action simultaneously, such as retrying after an outage, refreshing at TTL expiry, or waking on a push notification. The synchronized demand can exceed origin capacity even when per-client traffic looks normal.
Technical Architecture & Deep-Dive
Herds differ from gradual growth because they concentrate requests into a narrow time window. Retry storms after 503 responses are a classic example when every client uses the same backoff schedule without jitter.
Edge caches, request coalescing, jittered backoff, circuit breakers, and stale-if-error give upstreams breathing room. API gateways should shed load gracefully with partial responses instead of letting queues grow unbounded.
Key Architectural Benefits
Implement Thundering Herd at the Edge
Deploy lightweight API aggregation, edge caching, and secret injection in minutes without Docker containers or complex servers.
Frequently Asked Questions
Q:Is a thundering herd the same as a cache stampede?
A cache stampede is one herd pattern when hot keys expire. Herds also happen on retries, cron jobs, and coordinated client events.
Related Architecture Concepts
A cache stampede happens when a popular cache entry expires and many clients request it at once, each triggering an upstream fetch because no valid copy remains. The sudden synchronized miss overwhelms origins and third-party APIs, spiking latency and error rates across the fleet.
Request coalescing, also called request collapsing, is a caching technique where many simultaneous requests for the same uncached resource are combined into a single upstream request. The first request fetches from the origin while the others wait and receive the same response.
Retry with exponential backoff is a strategy for repeating failed requests with progressively longer waits between attempts, such as 200, 400, then 800 milliseconds. Adding random jitter spreads retries out, preventing many clients from hammering a recovering service at the same moment.
The Circuit Breaker pattern is a software resiliency design that monitors upstream service health and temporarily halts requests to failing endpoints. By failing fast and returning cached fallback data, it prevents cascading system outages and reduces recovery time.
Ready-to-Use Edge Recipes
To firebase api rate limit avoid with edge caching, put a TTL cache on your RealComposer pipeline. Edge absorbs repeated profile lookups across users. Shared edge cache absorbs repeat traffic before it reaches Firebase.
To solve how to cache github api rate limit 60 requests per hour, never execute unauthenticated requests or embed private tokens directly in client bundles. RealComposer acts as a sub-10ms edge gateway running on Cloudflare Workers that securely injects credentials from your Secret Vault, normalizes responses, and prevents unauthorized API usage. Route calls through your RealComposer edge URL to guarantee complete privacy, robust origin validation, and low mobile latency.