Seams

Structure cache

structure_cache

The structure-response cache seam.

The generic structure route can serve a previously-rendered response straight from a consumer-owned cache, skipping materialise + serialise + splice. The library does not know which responses are worth caching (that is a catalog-shaped speedup decision — e.g. a consumer may cache its largest codelists and hierarchies) nor where the bytes live; it only knows to ask the injected StructureCache before doing work and to offer the finished response back for storage.

A consumer implements StructureCache; the default NullStructureCache caches nothing, so a bare catalog just always does the work. The cache is carried on the CatalogRuntime.

StructureQuery

The resolved structure request a cache keys on.

version is post-latest-resolution and token is the runtime's effective_token: one edition, with the annotator's fingerprint folded in, so a new edition or an annotation change invalidates every slot.

StructureCache

Bases: Protocol

Consumer-owned cache for rendered structure responses.

lookup returns a ready response to short-circuit with, or None to proceed with the live render. store is offered the finished response (only 2xx, body-bearing responses are passed) for the consumer to cache if its own gating says the query is worth it.

lookup

lookup(query: StructureQuery) -> Response | None

Return a cached response for query, or None to render live.

store

store(query: StructureQuery, response: Response) -> None

Offer a freshly rendered response for caching (consumer decides).

NullStructureCache

Default no-op cache — never hits, never stores.

lookup

lookup(query: StructureQuery) -> Response | None

Never a hit.

store

store(query: StructureQuery, response: Response) -> None

No-op.

Availability values

availability_values

The precomputed-availability seam, which supplies value lists for the library to render.

/availability answers in two steps — compute the per-component distinct value sets and the observation count, then render them. This seam sits between the two. A consumer that already has the values, because its producer baked them into the edition or because it computed them once and kept them, hands them back and the scan never runs. The library renders the response either way, through ~sdmxlib_rest.responses.availability.render_availability.

The seam is consulted only for unfiltered requests, which keeps its key small. Precompute can only ever cover the unfiltered answer: the filter space is unbounded, so nobody can bake every combination of c[] values, partial keys and period bounds. Restricting the seam to that case means its key is just the build, the flow and the component — nothing to normalise, so no way for two spellings of one filter to land in different slots, or two different filters to share one. A filtered request always scans.

"Unfiltered" is judged after the ~sdmxlib_rest.filter_shaper.ComponentFilterShaper runs. A shaper may add a constraint the request never named, and the answer is then filtered whatever the URL said — so a precomputed unfiltered answer would be wrong for it. A shaper may only narrow requests that were already filtered, but the library cannot assume every shaper does.

The key omits mode, format, language and references. With no constraints, mode=available relaxes nothing and returns exactly what mode=exact does. Accept, Accept-Language and references= change only the rendering. So one entry per (build, flow, component) serves every client spelling — where the response-level cache this replaced needed a slot per accept variant x query suffix x language, and still missed ?lang=fr.

Repeat-caching a filtered answer is memoisation and belongs above the route. A consumer can install an HTTP response cache through create_app(outer_middleware=...). It sits outside compression there, so it must key on Accept-Encoding as well as Accept and Accept-Language.

AvailabilityKey

Identifies one precomputable availability answer.

Hashable and canonical — SdmxUrn compares on agency, id and version — so a consumer can key a plain dict on it, or rebuild one from a baked filename with Dataflow.urn_for(...) and get a matching key.

Attributes:

Name	Type	Description
token	str	Pins the entry to one edition, so a new edition invalidates every slot without any bookkeeping.
flow_urn	SdmxUrn[Dataflow]	The resolved dataflow. Keying on the flow lets the 1.5 and 2.2.2 surfaces share an entry, and keeps a forwarder's path shape from colliding two flows.
component_id	str	all or one component id, verbatim from the path. all is every key dimension; a specific id may also name an attribute.

ResolvedAvailability

The two things a scan produces, and everything rendering needs from it.

Attributes:

Name	Type	Description
key_values	tuple[KeyValue, ...]	One entry per enumerated component, in the order the route would scan them — the cube region preserves it.
obs_count	int	Rows in the flow. With no constraints this is the total, and it is the figure .Stat shows in its "(N observations)" badge. Zero renders as a 204.

AvailabilityValues

Bases: Protocol

Consumer-owned source of precomputed availability values.

lookup is asked before scanning and returns the values, or None to fall through to the scan, so a partial bake is a normal state. store is offered the scanned values afterwards, which lets a consumer with no producer-side bake still pay the scan once per build instead of once per request. Both are called only for unfiltered requests.

lookup

lookup(key: AvailabilityKey) -> ResolvedAvailability | None

Return the values for key, or None to scan.

store

store(
    key: AvailabilityKey, resolved: ResolvedAvailability
) -> None

Offer freshly scanned values for keeping (consumer decides).

NullAvailabilityValues

Default: knows nothing, keeps nothing — every request scans.

lookup

lookup(key: AvailabilityKey) -> ResolvedAvailability | None

Never a hit.

store

store(
    key: AvailabilityKey, resolved: ResolvedAvailability
) -> None

No-op.

Metadata store

metadata_store

The reference-metadata store seam.

The /metadata/* routes need to answer "which MetadataSets are attached to this structural artefact / metadataflow?" — a lookup over the edition's referential-metadata linkage. The consumer decides how that linkage is stored (a consumer's edition keeps it in its own DuckDB tables); the library needs only the set URNs back, which it then materialises through the catalog like any other artefact.

A consumer implements MetadataStore; the default NullMetadataStore returns nothing, so a catalogue without a metadata linkage simply reports no attached sets. The store is carried on the CatalogRuntime.

MetadataStore

Bases: Protocol

Consumer-owned lookup from structural artefacts to attached MetadataSet URNs.

The runtime owns the DuckDB connection (the metadata linkage lives in the same meta store as everything else), so it is passed in; the store owns only the queries.

sets_for_structure

sets_for_structure(
    conn: DuckDBPyConnection,
    *,
    structure_urn: str,
    provider_id: str | None,
    limit: int,
) -> list[str]

MetadataSet URNs whose declared structure is structure_urn.

provider_id (or None for all providers) narrows to sets from one data provider; limit caps the result. URNs come back sorted.

sets_for_target

sets_for_target(
    conn: DuckDBPyConnection,
    *,
    target_urn: str,
    item_pattern: str | None,
    limit: int,
) -> list[str]

MetadataSet URNs attached to target_urn (or item URNs matching item_pattern).

item_pattern is a URN LIKE-pattern for item-level targets under the artefact (e.g. Concepts of a ConceptScheme); None matches the exact target only. URNs come back distinct + sorted.

NullMetadataStore

Default store — no metadata linkage, so every artefact reports no attached sets.

sets_for_structure

sets_for_structure(
    conn: DuckDBPyConnection,
    *,
    structure_urn: str,
    provider_id: str | None,
    limit: int,
) -> list[str]

Always empty.

sets_for_target

sets_for_target(
    conn: DuckDBPyConnection,
    *,
    target_urn: str,
    item_pattern: str | None,
    limit: int,
) -> list[str]

Always empty.

Component filter shaper

filter_shaper

The component-filter shaper seam.

Between "parse the request into a ComponentFilter" and "turn that filter into WHERE clauses" there is a gap where catalogs legitimately want to intervene, for two different reasons:

Semantics. A vendor query parameter can widen the filter — a within[GEO]=<parent> descends a geographic hierarchy and rewrites the dimension's value list to every code beneath that parent. The client asked one question; the catalog answers a broader one because it knows the hierarchy.

Physics. A catalog that knows its own partition layout can add a redundant predicate that leaves the result unchanged and narrows the scan — e.g. a lake partitioned by geographic level can inject a GEO_LEVEL equality when REF_AREA is a short concrete list, which is what lets DuckDB prune partitions.

Neither belongs in a generic library: the first is a vendor extension, the second is knowledge of a specific lake layout. But both have to happen at the same point in every route that builds a filter, so the library defines the hook and the catalog fills it. The default NullComponentFilterShaper returns the filter untouched, so a catalog that models neither pays nothing.

The shaper receives the raw query items because the library does not know which parameters a given extension reads — parsing within[...] is the shaper's own business.

FilterContext

Everything a shaper needs to rewrite a filter, resolved once per request.

query_items is the full multi-valued query string as (name, value) pairs — repeated parameters are preserved, which bracketed vendor params like within[DIM] rely on.

ComponentFilterShaper

Bases: Protocol

Consumer hook to rewrite a parsed component filter before SQL generation.

Implementations must be pure with respect to the request: return a new filter and leave the input unchanged, and raise ValueError / KeyError for a malformed vendor parameter so the route can turn it into a 400 alongside the standard key-parse errors.

shape

shape(
    component_filter: ComponentFilter, ctx: FilterContext
) -> sl.ComponentFilter

Return the filter to generate SQL from — possibly the input unchanged.

NullComponentFilterShaper

Default no-op shaper — returns the filter exactly as parsed.

shape

shape(
    component_filter: ComponentFilter, ctx: FilterContext
) -> sl.ComponentFilter

Pass the filter through untouched.

Message annotator

message_annotator

The pre-serialise structure-message annotation seam.

Some consumers need to project extra annotations onto a StructureMessage before it is written — e.g. the .Stat Suite Data Explorer discovers a DSD's reference-metadata (MSD) by reading a type="METADATA" annotation on the DataStructure (its getMSDIdentifiers selector). Because this seam runs on the model, ahead of to_json / to_xml, the annotation reaches every wire format (SDMX-JSON 2.0, SDMX-ML 2.1 / 3.0) — unlike a post-serialise JSON splice, which only ever touches the JSON body.

The library does not know which annotations a given catalogue wants to emit, so a consumer implements MessageAnnotator; the default NullMessageAnnotator returns the message untouched. The annotator is carried on the CatalogRuntime and applied by the structure route right after it builds the message and before it serialises.

The runtime owns the DuckDB connection (structural + metadata linkage live in the same meta store), so it is passed in; the annotator owns only the projection.

An annotator whose output can change while the edition stays the same — one applying an annotation overlay that is edited on a running deployment — also implements SupportsFingerprint (GH-2). The runtime folds the fingerprint into CatalogRuntime.effective_token, which the structure cache and the ETag key on, so an annotation change is never served from a stale cache entry or answered with a 304.

MessageAnnotator

Bases: Protocol

Consumer-owned projection applied to a structure message before serialisation.

annotate

annotate(
    message: StructureMessage,
    *,
    registry: EditionRegistry,
    conn: DuckDBPyConnection,
    token: str,
) -> StructureMessage

Return message (or a copy) with the consumer's annotations applied.

registry + conn let the annotator derive whatever it needs from the edition; token changes with every edition, so the annotator can memoise derivations per edition.

SupportsFingerprint

Bases: Protocol

An annotator whose output can change while the edition's token does not.

fingerprint

fingerprint() -> str

A value that changes whenever annotate would produce different output.

Called on every request, by the ETag middleware and the structure route, so it must be cheap: a version counter or a digest computed when the annotator's inputs change, never I/O. "" means the output depends on the edition alone, which is the same as not implementing this.

CompositeMessageAnnotator

Several annotators applied in order, as one.

Its fingerprint joins its members': "" while none has one, so a composite of fingerprint-free annotators keys exactly as they would alone.

annotate

annotate(
    message: StructureMessage,
    *,
    registry: EditionRegistry,
    conn: DuckDBPyConnection,
    token: str,
) -> StructureMessage

Run each annotator on the previous one's output.

fingerprint

fingerprint() -> str

The members' fingerprints, by position; "" if every one is empty.

NullMessageAnnotator

Default annotator — leaves every message untouched.

annotate

annotate(
    message: StructureMessage,
    *,
    registry: EditionRegistry,
    conn: DuckDBPyConnection,
    token: str,
) -> StructureMessage

Return the message unchanged.

annotator_fingerprint

annotator_fingerprint(annotator: MessageAnnotator) -> str

annotator's fingerprint, or "" when it does not implement one.

Overlays

overlays

A MessageAnnotator that applies sdmxlib annotation overlays at serve time (GH-10).

An overlay (sdmxlib.edition.Overlay) is a versioned document of entries that set, add or remove annotations of one type on the artefacts and items a selector picks: annotations only, never ids, codes, structure or data. It lets a deployment change display hints without rebuilding its edition.

OverlayAnnotator applies its overlays to every structure message before it is serialised, so the change reaches every wire format. Its fingerprint is the hooks line and the overlays' digests as applied to its audience, so CatalogRuntime.effective_token comes out equal to sdmxlib's effective_token(manifest, overlays, merge=..., audience=...): the structure cache, the splices and the ETag move whenever an overlay is added, removed, reordered or edited, or a hook or the audience changes, and AvailabilityKey does not, since annotations cannot change availability.

replace swaps the overlays on a running deployment, for example from an admin endpoint or a file watcher the consumer owns. A request in flight when they are swapped may pair the new body with the old ETag; the next request carries the new ETag, so a client never holds a stale body for longer than one revalidation.

Applying overlays copies every artefact an entry selects, on every request the structure cache does not answer. A * or class-name selector that reaches a very large codelist's codes pays that on each such request; keep selectors specific, or settle a change into the edition with sdmxlib.edition.fold_overlays.

With an audience, an entry applies when its scope is that audience or null; without one, every entry applies. An overlay that changes a name or description (texts) is refused when it is set, since sdmxlib applies those only in fold_overlays, into a new edition.

OverlayAnnotator

Applies annotation overlays, in order, to every structure message.

Attributes:

Name	Type	Description
merge	Mapping[str, MergeHook] | None	Per annotation type, the MergeHook that decides how a set combines with the annotations already there; passed to apply_overlays. Replace by default.
audience	str | None	The audience whose scoped entries apply, with the unscoped ones; None applies every entry.

An overlay that changes a name or description raises OverlayError at construction.

overlays property

overlays: tuple[Overlay, ...]

The overlays applied, in order.

from_files classmethod

from_files(
    paths: Iterable[str | Path],
    *,
    merge: Mapping[str, MergeHook] | None = None,
    audience: str | None = None,
) -> Self

Build an annotator applying the overlay documents at paths, in order.

A file that is not an overlay document this sdmxlib can read raises sdmxlib.edition.OverlayError from read_overlay.

Parameters:

Name	Type	Description	Default
paths	Iterable[str | Path]	Overlay documents, applied in this order.	required
merge	Mapping[str, MergeHook] | None	Per annotation type, the MergeHook for a set. Replace by default.	None
audience	str | None	The audience whose scoped entries apply; None applies every entry.	None

Returns:

Type	Description
Self	The annotator.

replace

replace(overlays: Iterable[Overlay]) -> None

Apply overlays from the next request on, in place of the current ones.

An overlay that changes a name or description raises OverlayError, and the current overlays stay.

Parameters:

Name	Type	Description	Default
overlays	Iterable[Overlay]	The overlays to apply, in order.	required

annotate

annotate(
    message: StructureMessage,
    *,
    registry: EditionRegistry,
    conn: DuckDBPyConnection,
    token: str,
) -> StructureMessage

message with the overlays applied: changed copies, never in place.

fingerprint

fingerprint() -> str

The hooks line and the overlays' digests, computed when they were set; "" for none.

Splices

splice

Response-splice registry — the post-serialise extension seam.

sdmxlib's JSON writer emits a spec-clean structure body. Consumers routinely need to splice extra fields into it after serialisation — client-specific annotations, detail-aware codelist/concept hydration, .Stat compatibility trims. The library must not know about any of those; it only knows how to run an ordered list of splices in one parse/encode pass.

A splice is (parsed_json: dict, ctx: SpliceContext) -> bool (sync or async) that mutates parsed_json in place and returns whether it changed anything. Consumers register their splices in a SpliceRegistry; the generic route calls run_structure_splices, which runs them in order.

Instead of a hardcoded import-and-call list, the order is a registry the consumer owns, and the single orjson.loads → mutate-each → orjson.dumps cycle lives here.

SpliceContext

Everything a structure splice may need about the request being served.

The generic route builds this and hands it to every splice; a splice reads only the fields it cares about. extra carries surface-specific flags (e.g. inline_contentconstraint, strip_hierarchy_managed) so consumer sub-apps can vary splice behaviour without new context fields.

SpliceRegistry

Ordered splices a consumer installs for a given app surface.

structure runs on structure-message JSON bodies. Each app instance (canonical, /v1, /de) carries its own registry, so the same generic route serves different post-processing per surface without the library knowing which splices exist.

run_structure_splices async

run_structure_splices(
    response: Response,
    registry: SpliceRegistry,
    ctx: SpliceContext,
) -> Response

Run every registered structure splice in one parse/encode pass.

Skips non-JSON responses untouched. Splices may be sync or async; each is awaited if it returns an awaitable. Rebuilds the response only when something actually changed (dropping the stale ETag / Content-Length so the outer middleware recomputes them).