Switching Outputs: OverrideEndpoint, SwitchDecoder, and Deferred Polling

One endpoint, more than one possible output. A chat call can return a whole response or a token stream; a request can succeed immediately or return a poll handle; an icecast URL can be a stream or a playlist. The instinct is to return a big enum the caller must match. Capi does something better: it switches the output type, so the type system carries the choice and there’s no enum at the call site.

There are three axes to switch along, and recognizing which one you’re on is the whole skill:

Axis 1: OverrideEndpoint — you switch

An OverrideEndpoint wraps a base endpoint and reshapes it — overriding the body, the accept header, the output, and the decoder — before the request is sent. grok’s .stream() is the canonical case: it takes the same chat request and returns a streaming variant.

let response = api.query(chat.clone()).await?;       // Output = ChatResponse
let mut stream = api.query(chat.stream()).await?;    // Output = StreamIterator<ChatResponseChunk>

.stream() returns a Stream<Self> wrapper that implements OverrideEndpoint, splicing stream: true into the body and swapping in SseDecoder + accept: text/event-stream. You chose the streaming output, at the call site, by calling a different method — no enum comes back.

What may be wrapped is restricted by a sealed marker trait: only types implementing the crate’s private StreamableChat / DeferrableChat bound can be .stream()’d or .deferred()’d, so the switch is available exactly where it makes sense and nowhere else.

Axis 2: SwitchDecoder — the server switches

Sometimes you can’t know the output until you see the response. icecast’s connect URL might serve an audio stream or a playlist, distinguished by Content-Type and the first few bytes. SwitchDecode inspects the response — without consuming it — and picks a sub-endpoint to decode with:

impl SwitchDecode for Connect {
    const PEEK_BYTES: usize = 64;    // how many leading body bytes to peek
    fn select(&self, headers: &ResponseHead, peek: &[u8])
        -> Branch<IcecastConfig, Resolved, IcecastError>
    {
        if playlist::is_playlist(headers, peek) {
            let mut playlist = GetPlaylist::new(self.url.clone());
            playlist.include_auth = self.include_auth;
            Branch::of(playlist)               // decode as a playlist
        } else {
            Branch::of(self.to_get_stream())   // decode as a reconnecting ICY event stream
        }
    }
}

Branch::of(sub_endpoint) decodes the response with that sub-endpoint’s own decoder (its Output must be Into<T>, its Error Into<Error>), and the sub-endpoint must share the switch endpoint’s ApiConfig and Requires — it decodes the switch’s own exchange, which is what lets the capability’s typed extension pass through to whichever branch is chosen. The switching endpoint itself is Clone, because the decoder captures it as its State before the request consumes it. The peek is genuinely non-consuming — the framework reads PEEK_BYTES, then reassembles the body so the chosen branch sees the whole thing. And it never re-issues the request: one round trip, then a local decision about how to read the bytes already in hand.

The enum-vs-switch tradeoff

Both a returned enum and a decoder switch can model “one of several shapes”. The switch wins when the caller knows statically which shape it wants (grok’s .stream() on Chat Completions and on the Responses API — you asked for a stream, you get a stream, no match arm for the non-stream case; both are sealed marker traits, StreamableChat and StreamableResponse, over the endpoints that support it). A returned enum wins when the caller genuinely must handle either at runtime with no way to know in advance. SwitchDecode is for the case where the server decides but each branch is still a distinct, fully-typed output — the caller matches on Resolved, but each arm is a real decoded value, not a bag of Options.

Deferred polling

A deferred operation is an OverrideEndpoint whose output is another endpoint. The wrapper is the client’s own — grok’s Deferred<T> — not a framework primitive: its .deferred() sets deferred: true and returns a poll endpoint rather than a response:

let poll = api.query(chat.deferred()).await?;   // Output = GetChatDeferredCompletion (a pollable endpoint)

The poll endpoint’s decoder is 202-aware — on 202 Accepted it returns DeferredChatResponse::Pending, otherwise it decodes the finished Success(ChatResponse). You then poll it until it’s ready, which is exactly what a wait-until runner automates: hand the poll endpoint to a WaitUntilRunner and it loops with backoff until Success, using the injectable clock so tests stay deterministic.

A note on IntoEndpoint

A fourth composition primitive, IntoEndpoint, converts your own input type into an endpoint (with an optional output remap via MapOutput and remap::{Yes, No}). It’s the machinery behind derived query inputs and the inner leg of flat-map; positioned against this chapter, OverrideEndpoint reshapes an existing endpoint’s request/response, while IntoEndpoint adapts a foreign input into one.

Next: composing streams — resumable reconnection and flat-map, the patterns that prove the decoder layer scales.