Codecs and Format Contracts

A framework has to decide where the choice of wire format lives. Hardcode one and every downstream user inherits it. Thread it through the type system as a generic parameter and it infects ApiConfig, Endpoint, the interface, and every signature in between. The wire model takes a third path: a Codec<Format> — a small Copy handle carrying a runtime-selected byte engine plus a compile-time format tag. The runtime half keeps the API surface clean; the compile-time half makes a boundary that needs JSON reject a codec that speaks XML before the program runs.

Once you understand what a Codec<F> is and where the markers and engines live, every encode/decode line in the rest of the book reads the same way.

The Codec<F> handle

A codec is one Copy value:

pub struct Codec<F = Unmarked> {
    inner: &'static dyn WireFormat,   // the byte engine (encode + decode)
    content_type: &'static str,       // e.g. "application/json"
    _format: PhantomData<fn() -> F>,  // the compile-time format tag — zero-sized
}

Three properties fall out of that shape, and all three are load-bearing:

• It is Copy, with no bound on F. A config stores a codec by value — no Arc, no lifetime, no reference counting — and stays Copy itself. The PhantomData<fn() -> F> is what keeps Copy/Send/Sync available for every F; a naive PhantomData<F> would drag F’s own bounds in.
• The backend is &'static. A codec handle is exactly one fat pointer plus a string pointer. It can be built in a const fn, so a codec can be a static, and copying it is trivial.
• F is a pure phantom. The marker carries no data and costs nothing at runtime. It exists only so the compiler can tell one format from another.

You rarely construct a codec by hand. A codec crate hands you one:

let json: Codec<Json>    = capiw_serde_json::codec();  // content type "application/json"
let form: Codec<UrlForm> = capiw_urlencoded::codec();
let xml:  Codec<Xml>     = capiw_quick_xml::codec();   // content type "application/xml"

and you store it on your config as a typed field (GrokConfig, abridged — the real one adds a Codec<Protobuf> under its grpc feature):

pub struct GrokConfig {
    // ... base config, auth, cookies ...
    json: Codec<Json>,
    form: Codec<UrlForm>,
}

The handle’s own surface is small:

impl<F> Codec<F> {
    pub fn content_type(&self) -> &'static str;
    pub fn encode_to_vec(&self, value: &dyn DynEncode) -> Result<Vec<u8>, CodecError>;
    pub fn encode_into(&self, value: &dyn DynEncode, out: &mut Vec<u8>) -> Result<(), CodecError>;
    pub fn decode_from_slice<'de, T: WireDecode<'de>>(&self, bytes: &'de [u8]) -> Result<T, CodecError>;
    pub fn erased_decoder<'de>(&self, bytes: &'de [u8]) -> Result<Box<dyn ErasedDecoder<'de> + 'de>, CodecError>;
    pub const fn erase(self) -> Codec<Unmarked>;   // drop the marker for format-agnostic plumbing
}

• encode_to_vec takes an erased value (&dyn DynEncode); a concrete value coerces at the call site — codec.encode_to_vec(&user). It returns a fresh Vec<u8>; encode_into appends to a buffer the caller owns instead.
• decode_from_slice::<T> is the inverse: bytes in, a typed T out. The one mandatory allocation is a boxed decoder; monomorphization is confined to this call. erased_decoder hands that boxed decoder out for a model-driven walker to drive — the value tree and the recorder read bytes that way.
• content_type() is registry/diagnostic data — the default MIME for the format. It is not automatically the request’s Content-Type header; an endpoint owns that (vendored types like application/vnd.foo+json are common), so nothing couples the wire header back to this string.

The calls the rest of the book makes are one layer up. Two extension traits in capi_core wrap the handle for the two request planes, and capture the decode recipe beside the bytes — the codec and the type’s shape, as a BodySpec or a QuerySpec — so a request carries, with its very bytes, the description fixture tooling needs to read them back:

config.json_lib().encode_body(&self.body_fields())?     // EncodeBody: bytes + BodySpec, for Endpoint::body
config.form_lib().encode_qs(&self.query_fields())?      // EncodeQs: text + QuerySpec, for Endpoint::query_string

EncodeBody is implemented for every Codec<F>; EncodeQs only for a Codec<F> whose F: QueryFormat — a format that emits key=value text, which is what makes Codec<UrlForm>::encode_qs exist and Codec<Json>::encode_qs a compile error. The bare decode_from_slice remains the decoder-side call: every config.json_lib().decode_from_slice(...) in the decoder chapter is one.

Why formats are type-tagged

The marker earns its keep the moment two formats are in play. A boundary that needs JSON names the format it needs, and the compiler enforces it:

fn needs_json(_: Codec<Json>) { /* ... */ }

fn give(codec: Codec<Xml>) {
    needs_json(codec); // compile error: Codec<Xml> is a distinct type from Codec<Json>
}

The clever part is where the format tag lives. The obvious way to make a format visible to the type system — a generic parameter like Config<Json> — would propagate through ApiConfig, Endpoint, BaseInterface, and every signature that touches them, infecting the whole surface. The wire model sidesteps that entirely: the marker sits in a PhantomData<fn() -> F> on a Copy handle, not in the handle’s data and not in the types that hold it. A config stores a Codec<Json> field — it is not itself generic over a format; the endpoint, the interface, and the pipeline never mention F; and format-agnostic plumbing that genuinely doesn’t care takes a Codec<Unmarked> (call .erase() to get one). So you get compile-time enforcement and a clean, un-infected API surface at once.

Two more properties fall out of the marker being a plain zero-sized type:

• Third-party markers. Format is an unsealed, empty trait, so a new format needs one line — impl Format for TheirMarker {} — in its own crate. There is no central registry to edit and no orphan-rule wall. Where markers come from, and why that openness needs a community convention, is below.
• Escape hatch. When you deliberately need to re-tag an erased handle, assume::<G>() re-pins the marker. It is unchecked — the one path that can attach a wrong marker — so use it only right after building a backend you know speaks G, and prefer erase() at the boundary over laundering handles through assume.

There is also a subset marker, QueryFormat: Format, for formats that can render a URL query string (flat key=value text — JSON bytes would be nonsense in a query slot). UrlForm opts in; the query plane accepts only codecs whose format does, so a wrong format in a query slot is a compile error, not a malformed URL.

Markers vs. codecs: the contract/engine split

There are two different things with similar names, and keeping them apart is the key to reading the crate list.

A format contract is the marker plus whatever type-level vocabulary the format needs. It is tiny and always-on. A concrete codec is a byte engine that implements the contract, and it is an optional dependency you can swap.

All four contract crates and all four codecs are sibling repos of their own; neither the Capi framework workspace nor the capi_wire workspace holds a format. (The full map is in How the Crates Fit Together.) What matters here is the shape of the relationship: a codec crate exposes one function, codec(), that returns its marker’s handle —

// capiw_serde_json
pub fn codec() -> Codec<Json> {
    Codec::new(&SERDE_JSON, "application/json")
}

— so a config depends on the tiny contract for the type (Codec<Json>) and on the concrete codec only to build one. That split is what lets a downstream user swap the JSON engine without your types changing, and it is why XML and protobuf are first-class formats rather than bolted-on special cases: they go through the same Codec<F> machinery as everything else.

How much vocabulary a contract carries varies with the format. JSON is self-describing, so capiw_ext_json is just the marker — no decoration macro, no attributes. XML needs to distinguish elements from attributes, protobuf needs field numbers, and URL forms need a list convention, so those three contracts carry extra vocabulary through the extension mechanism, each with a decoration macro behind an off-default model feature. The marker mechanism is identical either way. So is the feature story of the codecs: capiw_serde_json, capiw_urlencoded, and capiw_prost are no_std with no features to set; capiw_quick_xml links the standard library.

The same handle shape serves the header plane. capiw_wire_headers::WireHeaders encodes a wire model’s fields into header values and back with no marker at all — a header map is a header map — and is what an endpoint’s header proxy runs through.

Where a marker comes from

A marker looks like a triviality — an empty type whose only job is to be distinct. Its real work is coordination, and the problem it coordinates is worth seeing first.

The problem: annotations with nowhere to live

Some formats need per-field instructions that the format itself demands. XML is the standard example: a value can be a child element, an attribute, or the element’s text, and nothing about a Rust struct says which. Serde has no slot for that kind of format-specific knowledge, so serde-based XML libraries smuggle it through the one field that is free-form — the name. Placement gets encoded as magic prefixes inside #[serde(rename = "...")], with each library choosing its own sigils.

The consequence is that the annotations on your types belong to one library, not to XML. Switching engines means rewriting every type, because the second library reads different magic. Types and codec end up welded together — precisely what a swappable codec is supposed to prevent.

The marker as the anchor

The wire model gives those annotations a real home — the binding’s extension bag — and gives that vocabulary an identity: the format marker. The marker and its extension keys together are the contract. capiw_ext_xml doesn’t just say “this is XML”; it defines what XML annotations mean and provides the typed readers a codec uses to read them back.

Any library that wants to speak the format links against that marker crate and honors that vocabulary. Two independently written XML engines that both build on capiw_ext_xml are genuinely interchangeable: swapping one for the other is a one-line change to the codec constructor, and not one annotation on your types moves. The marker is what makes “swap the engine” mean something.

“Honors that vocabulary” is testable, and the family tests it. Each format’s conformance suite lives in the codec that runs it — capiw_serde_json/tests/conformance.rs, capiw_quick_xml/tests/conformance.rs, capiw_prost/tests/conformance.rs — and pins the contract crate’s semantics: presence rules, the omit-on-None default, flattening, the extension attributes. A second engine for a format proves itself by carrying the same suite. The unpublished capiw_conformance_tests holds only the properties that need several codecs at once.

So a marker does two jobs at once. As a type parameter it keeps formats from mixing — a Codec<Xml> cannot satisfy a Codec<Json> boundary. As a contract it lets separate implementations agree on what a format’s annotations mean.

Who defines them

Anyone. Format is unsealed and empty, so a marker is one line in your own crate, and there is no registry to petition and no blessed list. Capi ships four — Json, UrlForm, Xml, and Protobuf — which is enough to develop the mechanism and exercise it against real formats. They carry no special status beyond being first.

That openness also covers a format outgrowing its own vocabulary. Because markers are ordinary types, a new one coexists with the old rather than replacing it: if a future revision of a format needed annotations the current contract can’t express, it would ship as its own marker with an enriched vocabulary, and both would remain valid at once. Types and codecs migrate when they choose to; nothing breaks on the day the new contract appears. A versioned format is the natural case for this — a hypothetical Protobuf4 alongside Protobuf rather than in place of it.

One marker per format, please

The openness has a failure mode worth naming, because it is the community’s to avoid rather than the compiler’s.

If three crates each define their own Json marker, the compiler sees three unrelated formats. A Codec<TheirJson> will not satisfy a Codec<OurJson> boundary even though both are, in every way that matters, JSON. The mechanism that was meant to catch a real mistake — handing XML to a JSON slot — instead splits an ecosystem into incompatible islands.

So the convention is simple: one marker per format, depended on rather than duplicated. If a contract crate for your format exists, build on it. Mint a new marker when the vocabulary genuinely differs — a new format, or a version needing annotations the existing contract can’t express — not to avoid taking a dependency.

If you do end up holding a handle tagged with a rival marker, assume::<F>() re-tags it, and how well that works depends entirely on the format. For one with no annotation vocabulary at all, the assertion is nearly always true — a JSON engine is a JSON engine, and a re-tagged handle will behave. For XML or protobuf, where the engine must read specific extension keys to function, re-tagging a codec that doesn’t honor them compiles cleanly and then misbehaves at runtime. assume is a repair tool for a fragmentation that shouldn’t have happened; converging on one marker is the actual fix.

The object-safe ABI

Skip this section on a first read; it is the “why” under the hood, not something you call.

Runtime format selection needs the byte engine to be an object — a &'static dyn WireFormat — because the whole point is that the config, not the compiler, picks the format. Rust makes that awkward for a reason worth understanding, because it is exactly why Serde’s traits could not be used here directly.

A trait with a generic method is not object-safe: the compiler can’t build one vtable covering every monomorphization. Serde’s core trait has exactly that shape:

pub trait Serialize {
    fn serialize<S: Serializer>(&self, serializer: S) -> Result<S::Ok, S::Error>;
}

serialize is generic over S, so dyn Serialize is illegal and you cannot store “something serializable” behind a trait object. The wire model defines its own object-safe split instead:

pub trait ByteEncode {                                   // erase a value → bytes
    fn encode_u8(&self, value: &dyn DynEncode) -> Result<Vec<u8>, CodecError>;
    fn encode_into(&self, value: &dyn DynEncode, out: &mut Vec<u8>) -> Result<(), CodecError> { … }  // provided
}
pub trait ByteDecode {                                   // hand back an erased decoder over bytes
    fn decoder<'de>(&self, bytes: &'de [u8]) -> Result<Box<dyn ErasedDecoder<'de> + 'de>, CodecError>;
}

// The object-safe principal a `Codec` points at, blanket-combined over the two halves:
pub trait WireFormat: ByteEncode + ByteDecode + Send + Sync {}

The generic-method problem is dissolved rather than wrapped: DynEncode is the object-safe counterpart to the generic WireEncode, ErasedDecoder the counterpart to WireDecode, and the dynamic dispatch happens inside those trait objects’ methods while the Codec handle itself stays entirely static. There is no erased_serde and no unsafe anywhere on this path. The only cost is a vtable call per operation and one heap allocation for the boxed decoder — negligible next to an HTTP round trip.

This is also why the two planes coexist. WireEncode/WireDecode are generic, so the typed path monomorphizes with no dynamic dispatch; the erased path exists for the runtime-selected handle. A type derives once and serves both.

The accessor convention

A config exposes its codecs through hand-written accessors, by convention named json_lib() / form_lib() / xml_lib():

impl GrokConfig {
    pub fn json_lib(&self) -> Codec<Json>    { self.json }
    pub fn form_lib(&self) -> Codec<UrlForm> { self.form }
}

These are not methods on the core ApiConfig trait — a client that speaks only JSON has no reason to answer xml_lib(), and a codec-free client has no codecs at all. Each config declares exactly the accessors its endpoints need. Endpoint and decoder code then reads them at the point of use: config.json_lib().encode_to_vec(...), config.json_lib().decode_from_slice(...). Building that config — the two-constructor pattern (new(<codecs>) for swappability vs. new_with_defaults() for batteries included) you can already see in the GrokConfig above — is the subject of the config chapter.

Next: how a type tells a codec what it looks like — Describing a Type.