AATB: Assisted Adaptive Token Bucket

• AATB is a client-side retry controller that mitigates shared quota congestion using a local token bucket combined with aggregated telemetry.
• It adjusts token generation rates based on both individual success/failure and group-level congestion, improving retry timing decisions.
• Experiments show AATB reduces HTTP 429 errors by up to 93% while incurring only a modest increase in overall request duration.

AATB, short for Assisted Adaptive Token Bucket, is a client-side retry controller for HTTP APIs that share a common server-side quota. It is introduced as part of a broader rethinking of HTTP API rate limiting in which retry behavior is driven by congestion inference rather than by blind, time-only waiting. In the formulation presented in (Farkiani et al., 6 Oct 2025), AATB addresses the case where many independent clients consume the same quota, the server returns only HTTP 429 on rejection, and richer feedback such as Retry-After or rate-limit headers is unavailable. Its central mechanism combines a local token bucket with aggregated telemetry from multiple clients so that retry timing is informed by shared congestion rather than by each client’s local failure history alone.

1. Problem setting and motivation

AATB is designed for the setting in which independent clients all consume the same quota, while the server exposes only minimal rejection feedback. The paper emphasizes two shortcomings of conventional practice. First, exponential backoff is widely used, but under a shared quota it is wasteful because each client observes only that its request was rejected and therefore cannot determine whether the system remains congested because of other clients. Second, server-only rate limiting protects the backend, but it does not solve the client coordination problem: throttling tells clients that a request was not admitted, but not when a later attempt is likely to succeed (Farkiani et al., 6 Oct 2025).

In that setting, retry storms arise because clients remain ignorant of each other’s offered load. The paper therefore frames AATB as a lightweight client-side alternative to centralized coordination or proxy-based coordination, which are described as operationally awkward, potential bottlenecks, and often dependent on cooperation from the service. A common misconception addressed by the paper is that backend throttling alone is sufficient for efficiency. The reported position is narrower: backend throttling protects the service, but does not by itself minimize futile retries when many clients share a quota.

2. Relationship to ATB and the telemetry-assisted design

AATB extends ATB, the Adaptive Token Bucket, which is presented as a congestion-control style client algorithm. In ATB, a client may send only when it has local tokens, and tokens are generated at an adaptive rate. When a request succeeds, the client increases its token-generation rate; when a request fails with HTTP 429, it decreases that rate. AATB preserves the token-bucket mechanism but adds aggregated telemetry from all clients, allowing the controller to reason about group-level congestion rather than only local success and failure history (Farkiani et al., 6 Oct 2025).

The telemetry path is deliberately separated from the rate-limited API. Each client sends telemetry to a separate telemetry server over UDP, so the mechanism does not require cooperation from the protected API and does not consume quota. The telemetry server reports aggregate state including the number of active clients, total requests sent, how many clients saw 429s, and updated limiter rates if available. This makes it possible for AATB to infer not only whether its own recent request failed, but also whether the shared system is congested and whether its own load is low or high relative to the group.

The paper summarizes the distinction succinctly: ATB is a local congestion controller; AATB is a telemetry-assisted congestion controller. That distinction is operationally important because AATB suppresses retries during observed congestion windows rather than merely stretching inter-retry delays according to a fixed temporal pattern.

3. Formalization and update rules

The paper introduces an oracle-style token-bucket scheduling problem to formalize the server-side objective. The stated objective is to minimize total response time,

$$\min \sum_{i \in I} \sum_{j \in J(i)} \bigl(x_{i,j} - A_{i,j}\bigr),$$

subject to FIFO ordering and token-bucket constraints. The number of served requests at time $t$ is aggregated as

$Z_t = \sum_{i \in I} \sum_{j \in J(i)} z_{i,j,t},$

and the token bucket evolves according to

$$y_t = \min\bigl(y_{t-1} - Z_t + r,\ B\bigr), \quad y_0 = B.$$

The text uses this oracle formulation mainly to motivate why online clients must approximate the schedule without knowing future arrivals (Farkiani et al., 6 Oct 2025).

AATB itself is specified procedurally. Each client periodically sends telemetry. If telemetry indicates that any client reported 429s in the last window, AATB treats that as congestion and sets

$backoff \gets \omega + \operatorname{rand}(-2, 2),$

then

$next\_acquire \gets now + backoff.$

If there was no recent congestion report and enough time has passed since the last rate change, the client compares its own load to the average load of all clients. If

$client\_load < 0.75 \times avg\_load,$

then it increases its token-generation rate by

$$rate \gets \max\bigl(rate \times \alpha,\, rate + \delta\bigr).$$

Otherwise it increases more conservatively using

$$rate \gets \max\bigl(rate \times \beta,\, rate + \delta\bigr).$$

For the real-trace configuration, the pseudocode uses $\alpha = 1.4$ and $t$0, with $t$1 as a floor on additive increase and $t$2 as a minimum supported rate.

The congestion_notification path is more aggressive. If a client actually receives a 429, it immediately reports congestion, recomputes network and client load, and reduces rate according to whether its own load is below half the average. If

$t$3

then

$t$4

otherwise

$t$5

It then sets

$t$6

updates the last rate-change timestamp, and schedules the next attempt using

$t$7

followed by

$t$8

A key implementation point is that retry permission is gated by both token availability and the next_acquire timer. If a token is present but the current time is earlier than next_acquire, the client waits. This allows AATB to suppress retries during congestion windows rather than simply spacing them geometrically.

4. Experimental environment and workloads

The evaluation was conducted by emulation on two virtual machines on one physical host, with the telemetry server, application service, and clients implemented in Python 3.13 using asyncio. Clients started asynchronously with random delays up to 10 seconds, ran in separate processes, and had no inter-client communication. The service was fronted by Envoy v1.33.0 as the token-bucket rate limiter, configured with HTTP/2 over cleartext (h2c), a backlog of 2048, and a keep-alive timeout of 350 seconds. The backend service itself was intentionally simple: each request contained two fixed numbers in JSON, and the service returned their product, so that server-side processing delay would not dominate the results (Farkiani et al., 6 Oct 2025).

The limiter configuration used capacity 100 and token generation rate 80 tokens/minute, which admits about 500 requests in a 5-minute experiment, corresponding to roughly 144K requests/day. Workloads were drawn from both real and synthetic sources. The real trace came from a search-endpoint access log averaging 131K requests/day and 27K unique IPs; each IP was treated as a user, and test sets of 400, 500, 600, 700, and 800 requests were used, with 18, 22, 23, 25, and 27 users respectively. Synthetic traces modeled two controlled scenarios: a five-client case representing a service mesh with a few high-traffic services, and a one-hundred-client case representing many independent clients similar to the real trace. Synthetic workloads contained 400–800 requests over a 5-minute window, with one request assigned per client initially, additional requests sampled from a Poisson distribution, and timestamps drawn from an exponential distribution. Each reported result was averaged over at least 30 runs.

The main evaluation metrics were average total emulation duration, average service time, and total number of HTTP 429 errors.

5. Quantitative performance

The reported headline result is that AATB reduces HTTP 429 errors sharply relative to exponential backoff, while incurring a modest increase in completion time. On the real trace with 800 requests, the paper reports that WB reduced errors by 62.70% with a 25.45% increase in duration, ATB reduced errors by 70.13% with a 21.26% increase in duration, and AATB reduced errors by 93.23% with a 27.62% increase in duration (Farkiani et al., 6 Oct 2025).

In the synthetic workloads, AATB is stronger still in the five-client case and remains substantial in the one-hundred-client case.

Scenario	Error reduction	Duration increase
Five-client synthetic, 500 requests	96.9% vs UB	13.3%
Five-client synthetic, 800 requests	97.3% vs UB	19.8%
One-hundred-client synthetic, 500 requests	77.8%	26.4%
One-hundred-client synthetic, 800 requests	91.7%	11.7%

The paper also notes an important trade-off. UB often achieved the shortest total duration because it retried aggressively, but it produced the worst service time and the most 429s. By contrast, ATB and AATB deliberately delayed more strategically, so total duration increased somewhat, but service time improved and retry waste dropped sharply. The authors summarize the overall range as 70.13%–97.3% reduction in errors with only an 11.7%–27.62% increase in total duration in the 800-request cases.

Operational overhead is reported as modest. Telemetry messages remained comparatively lightweight: around 276.25 update messages at 800 requests in the real-trace case, fewer than 88 in the five-client case, and about 1106 in the 100-client case.

6. Deployment properties, assumptions, and limitations

AATB is presented as deployable without special server support. Its telemetry path is separate and does not need to be counted against the API quota. The broader client-side model is explicitly tied to browser deployment as well: the paper states that ATB can be implemented in browsers via a service worker, which can act as a proxy-like component and modify outgoing requests, and AATB extends that model while adding telemetry (Farkiani et al., 6 Oct 2025).

The design, however, depends on telemetry infrastructure and periodic feedback. If telemetry is unavailable or delayed, the controller falls back toward less informed behavior. The evaluation also assumes no additional server-side information beyond 429 responses and no IP blocking or special headers. These assumptions delimit the reported gains: AATB is targeted precisely at minimal-feedback environments in which coordination must emerge on the client side.

The paper also identifies a practical misunderstanding worth rejecting. It notes that one can drive errors close to zero by delaying aggressively even without coordination, but that this produces unacceptable latency. AATB is therefore positioned not as a mechanism for eliminating errors at any cost, but as a better trade-off between wasted retries and delay. Future work is identified on using lightweight helper headers and on studying how telemetry update frequency affects performance. A plausible implication is that the algorithm’s effectiveness is likely to depend not only on the presence of telemetry, but also on the timeliness and granularity of that telemetry.