Rate Limiting

Rate limiting is a first-class, pluggable concern: a small trait in capi_rate_limit and a set of implementations in capi_limiters. You can pace requests proactively (a local budget), reactively (obey the server’s X-RateLimit-* / Retry-After), or both — and swap the algorithm without touching endpoints. The same trait meters the transfer plane, so a byte throttle is a rate limiter denominated in bytes.

The trait

pub trait RateLimiter: Debug + MaybeSend + MaybeSync + 'static {
    fn try_acquire(&self, now: Instant) -> Result<(), Duration>;                    // required
    fn try_acquire_n(&self, now: Instant, n: u64) -> Result<(), Duration> { … }     // charge n units; default: one operation
    fn peek_delay(&self, now: Instant) -> PeekDelay { PeekDelay::Unsupported }      // the wait, without consuming a slot
    fn peek_delay_n(&self, now: Instant, n: u64) -> PeekDelay { … }
    fn unit(&self) -> LimiterUnit { LimiterUnit::Operations }                       // what the budget is counted in
    fn update_from_headers(&self, clock: &ApiClock, headers: &HeaderMap)
        -> Result<(), HeaderUpdateError> { Ok(()) }                                // learn from the server
}

try_acquire is the heart and the only required method: Ok(()) to proceed, Err(delay) to wait that long first. try_acquire_n charges n units of whatever the limiter is denominated in — the default charges one operation whatever n says, which is right for a requests-per-minute limiter, where a request costs one request however many bytes it carries. unit() declares the denomination: Operations (the default), Bytes, Tokens (model tokens, or any quota a service meters a request’s content by), Other(&'static str), or Unspecified for the no-op limiters and for a composite whose members disagree. WaitableRateLimiter adds wait_until_ready(clock).

Limiters are carried as a RateLimit — None, Static(&'static dyn RateLimiter), or Arc(Arc<dyn RateLimiter>) — and installed through the sealed ToRateLimiter trait, which is implemented for &'static T, Arc<T>, and RateLimit itself. NO_RATE_LIMIT is the empty carrier, and ApiConfig::default_rate_limiter returns it unless a config says otherwise.

Proactive: GCRA

The Generic Cell Rate Algorithm paces requests against a local budget — smooth, allocation-free, and no_std-friendly (it is arithmetic on a theoretical arrival time). GcraLimiter is the implementation, and Rate expresses the budget:

let limiter = GcraLimiter::per_second(2, 3);   // 2 req/s sustained, burst of 3
let rate = Rate::per_second(10);               // or per_minute / per_hour / per_day / unlimited

Don’t conflate the two per_seconds: GcraLimiter::per_second(rate, burst) takes two arguments (sustained rate + burst); Rate::per_second(tokens) takes one.

Reactive: header-based

When the server publishes its limits, obey them rather than guessing. HeaderLimiter reads quota headers (X-RateLimit-Remaining / -Reset) and Retry-After off each response through update_from_headers, so your pacing tracks the server’s actual state. It is built from a description of the headers the service sends — HeaderLimiter::from_headers(..) with a HeaderPolicy naming the remaining/reset headers and how the reset is spelled (ResetAfter::Seconds, Timestamp, HttpDate, SecondsOrHttpDate, Duration), or HeaderLimiter::retry_after(RetryAfter::Seconds | Auto | ..) for a service that only ever says Retry-After. Combine a GcraLimiter (a proactive floor) with a HeaderLimiter (reactive correction) for the best of both.

Who feeds the reactive limiter is the traversal: on the final attempt’s response, before any pipeline member can transform it, the traversal hands the head to the request’s limiter — best-effort, so a limiter that cannot read the headers keeps the parameters it has. Only the primary traversal entry does this; a credential mint skips the gate, and a capture has no response.

Cost-aware limiters

Not every quota counts requests. A byte-rate cap counts bytes; a model API’s tokens-per-minute counts tokens. MeteredLimiter is a GCRA budget spent in a declared unit — MeteredLimiter::bytes_per_second(bytes, burst_bytes), or MeteredLimiter::new(gcra, LimiterUnit::Tokens) — whose try_acquire_n charges the amount it is handed and admits regardless of it, so a single operation larger than the burst waits rather than being refused forever.

Composition follows from the denomination. Pair a requests-per-minute limiter with a tokens-per-minute one, call try_acquire_n with a token count, and each member reads n its own way: the operations limiter charges one, the token limiter charges n. The one composition that cannot work is two members denominated in different metered quantities, since only one n crosses the call — the composite’s unit() reports Unspecified for exactly that case.

The transfer plane is where a bytes-denominated limiter is driven. Throttle::with_limiter(impl ToRateLimiter) from capi_extras is a transfer stage that charges each chunk’s length against the limiter — the only place try_acquire_n is called with n > 1 today — and one Throttle seated as a shared stage becomes a process-wide bandwidth cap (Middleware). A throttle handed a limiter denominated in operations paces per chunk rather than per byte, which is logged when it happens and not refused.

Header-driven limiters

Build a header limiter from the contract published by the API being called:

use capi_rs::rate_limiters::{HeaderLimiter, ResetAfter, RetryAfter, RETRY_AFTER};

let quota = HeaderLimiter::from_headers(
    "x-quota-remaining",
    ResetAfter::Timestamp("x-quota-reset"),
    "x-quota-limit",
);
let retry = HeaderLimiter::retry_after(RetryAfter::Seconds);

from_headers accepts the quota, reset, and capacity header names, while retry_after selects how to parse the standard Retry-After header. The generic RETRY_AFTER static accepts both standard forms: delay-seconds and HTTP-date.

A static limiter costs nothing to reference — and it is process-global, one admission history for every client in the process that names it. A service whose quota is partitioned by account, resource, or endpoint wants one HeaderLimiter per partition instead, held in an Arc.

Wiring it up

Where the gate sits decides what a wait can and cannot affect. The traversal runs the rate-limit gate as its fifth step — after the Late phase, before the seal — so every member has shaped the request before the lane sleeps, and every signature is computed after the wait rather than staled by it (Middleware has the full order).

A client-wide default comes from the config. BaseConfig::with_default_rate_limit(..) installs it, and ApiConfig::default_rate_limiter() reports it; a wrapping config forwards the inner one’s:

let config = BaseConfig::new(base_url).with_default_rate_limit(&RETRY_AFTER);

fn default_rate_limiter(&self) -> impl ToRateLimiter {
    self.inner.default_rate_limiter()
}

An individual endpoint overrides rate_limit(&self, config, context) — which by default delegates to config.default_rate_limiter() — when one operation has a tighter budget than the rest of the API. Return a shared handle from it, a &'static limiter or an Arc-held one resolved from the config, never a limiter built fresh in the method: the hook is re-resolved once per send, for the primary request and again for each follow-up a decoder issues (a next page, a reconnect), and only a shared instance carries its admission history across those sends.

A single call adjusts its own limiter through the scoped request — api.scoped().with_rate_limit(limiter) to substitute one, .no_rate_limit() to skip the gate for this send.

When more than one limiter applies, composition is max-delay-wins: a tuple of up to four limiters, or a Vec<RateLimit>, acquires only when every member allows, waiting the longest delay any of them asks. Every member is visited on each attempt and the order of the tuple does not matter — which is unlike the pipeline’s members, which run in phase order, and unlike runners, which nest.

Next: runners, the layer above the decoder that can drive a query more than once.