API Reference¶
REST registries¶
The entry point for connecting to an SDMX REST endpoint over HTTP.
RestRegistry ¶
RestRegistry(
endpoint: str | ProviderInfo,
*,
api: str = "2.1",
timeout: float = 30.0,
headers: dict[str, str] | None = None,
id: str | None = None,
store: LocalRegistry | None = None,
fetch_cache_size: int = 256,
latest_ttl: int = 0,
csv_media_type: str | None = None,
**httpx_kwargs: Unpack[HttpxOptions],
)
Connection to an SDMX REST endpoint.
Use as a context manager. All artefact access happens within the scope.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
endpoint
|
str | ProviderInfo
|
Base URL or |
required |
api
|
str
|
SDMX REST API version — |
'2.1'
|
id
|
str | None
|
Stable identifier for this registry, used as the
|
None
|
store
|
LocalRegistry | None
|
Optional |
None
|
fetch_cache_size
|
int
|
How many distinct questions to remember having fetched (default 256). Bounds the fetch cache; eviction only means a later identical query fetches again. |
256
|
latest_ttl
|
int
|
Seconds a |
0
|
csv_media_type
|
str | None
|
The |
None
|
Thread safety: concurrent reads are safe and do not duplicate work.
Stated because it was previously unstated, which is not the same as
absent — the intended shape of this API is a server (§4.1 of
docs/design/registry-and-catalog.md is a FastAPI facade), and an
unstated contract in that setting is one nobody can rely on and nobody
can violate. The commitment matches the local LocalRegistry's, and the
reasons are in docs/design/concurrency.md:
- The query store is built once under a double-checked lock. Two threads
previously built two
:memory:stores, one of which was leaked still open with a thread's fetched artefacts inside it — so a query could read an empty store and correctly report nothing. - Two threads asking the same question fetch once: the second waits
on the first rather than repeating the HTTP round-trip
(
_singleflight.SingleFlight). - Two threads asking one dataflow different observation questions take turns, because the table holds one question's rows at a time.
Mutating calls — load, and anything that writes to a store
store — inherit the local LocalRegistry's rule and remain single-threaded.
Example
Cache-mode federation: structural reads cached locally.¶
local = sl.LocalRegistry("cache.duckdb") with sl.RestRegistry(sl.Provider.ECB, store=local) as reg: dsd = reg.get(sl.DataStructure.urn_for(agency="ECB", id="ECB_EXR1"), resolve=True)
Single-source ad-hoc.¶
with sl.RestRegistry(sl.Provider.ECB) as reg: for flow_id in reg.query(sl.q.dataflow).filter( sl.q.dataflow.agency == "ECB" ).ids(): print(flow_id)
registry
property
¶
The InMemoryRegistry populated by every artefact fetched so far.
Artefacts from all get(), query(), fetch(), and
resolve() calls accumulate here, every Ref bound to it (#492). Cross-request
Ref resolution happens automatically — a codelist fetched in one
request resolves dimension Refs in a DSD fetched in another.
Example
with sl.RestRegistry(...) as rest: dsd = rest.get(sl.DataStructure.urn_for(agency="ECB", id="ECB_EXR1"), resolve=True) codelists = rest.registry.get_all(sl.Codelist)
refused
property
¶
Artefacts a provider served that this registry refused, each with why (#419).
A registry's document is read artefact by artefact: one that
contradicts the schema, or breaks a rule add enforces, is left out
and the rest are read, so one bad DSD does not cost a bulk fetch every
other. get of a refused artefact raises with its reason.
load ¶
Load artefacts from a local SDMX file into this registry.
Useful for pre-populating the registry with local files before making network requests, or for mixing local and remote artefacts. All loaded artefacts are treated as fully resolved — no follow-up HTTP requests are made.
References are bound to this registry: artefacts loaded from file resolve Refs in
artefacts already present (from earlier load() or HTTP calls), and
vice-versa.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
str | Path | bytes
|
File path ( |
required |
Returns:
| Type | Description |
|---|---|
StructureMessage
|
The parsed |
Example
with sl.RestRegistry("https://...") as reg: reg.load("local_constraints.xml") # pre-populate dsd = reg.get(sl.DataStructure.urn_for(agency="ESTAT", id="CPI"), resolve=True)
close ¶
Close the HTTP client, and the query store if this registry made one.
A store passed in belongs to the caller and is left open —
closing someone else's store on the way out is the kind of tidiness
that loses data.
__contains__ ¶
True if this artefact is already cached locally — no HTTP fetch.
Checks the in-process Session and the store local store
(when configured). Returns False for URNs that would require a
network round-trip; this matches the RegistryReader Protocol's
semantics for membership.
Takes an SdmxUrn or a Ref, matching get and the other two
registries (#370). Anything else is False rather than an error.
query ¶
A lazy query over the relation root names, served over HTTP (#312).
The same entry point and the same language as LocalRegistry.query. What
differs is where the rows come from: this plans the query against what
SDMX REST can answer, fetches exactly what it needs, materialises it,
and then runs the query locally.
reg.query(sl.q.dataflow).filter(
sl.q.dataflow.agency == "ECB",
sl.q.dataflow.dsd.agency == "ECB",
sl.q.dataflow.dsd.dimensions.any(sl.q.component.concept.id == "REF_AREA"),
).ids()
A scan that leaves anything to filter locally needs an agency, and
that is the one thing this surface asks for that the local one does
not. The URL is /structure/{resource}/{agency}/{id}/{version}, so
what REST can answer is a listing narrowed by those three; a predicate
it cannot evaluate runs over whatever came back. Without an agency that
is every agency's artefacts, and the planner refuses rather than fetch
a registry to filter three rows from it. A query REST answers whole —
a bare listing, or equality on id or version — needs none. A
dataflow's agency does not imply its DSD's — cross-agency structure
reuse is normal SDMX — so the example above states both. The refusal
names the column to add.
Where the rows are kept. store, if set, which makes the
fetch a one-off across the store's whole life; otherwise an in-memory
store living as long as this registry. Repeating a query costs nothing
either way — the plan is keyed by meaning, so filter(a).filter(b) and
filter(b).filter(a) share a fetch.
Nothing runs, and nothing is fetched, until a terminal. That is what
lets the planner see the whole question: reg.query(sl.q.dataflow)
names no agency, and fetching there would fetch everything.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
root
|
Cols[R, A]
|
A relation accessor from |
required |
Returns:
| Type | Description |
|---|---|
Query[R, A]
|
A |
Raises:
| Type | Description |
|---|---|
PushdownError
|
At the terminal, when the query cannot be served within the bound. The message names what could not be pushed and what would make it admissible. |
get ¶
get(
target: SdmxUrn[object] | Ref[object],
/,
*,
expect: type[T] | None = None,
resolve: bool | Sequence[type] = False,
partial: bool = True,
) -> object
Read one artefact by identity.
Read order: in-process Session → local-store cache (when a store is
set and the URN is pinned) → HTTP. So this makes zero or one
request, not one by definition — the same call against a
LocalRegistry makes none.
resolve is reference depth, and it collapses what used to be three
entry points — fetch(), resolve(), and the GetQuery returned by
get(type, agency=…, id=…):
reg.get(urn) # skeleton; refs unresolved
reg.get(urn, resolve=True) # every reference followed
reg.get(urn, resolve=[sl.Codelist]) # only refs of these types
reg.get(urn, resolve=True, partial=False) # and untrimmed
The default differs from LocalRegistry.get, and that is not an
inconsistency. Resolving is a join away locally and another round
trip here, so each states the default its cost implies rather than
agreeing on a number neither would pick.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
target
|
SdmxUrn[object] | Ref[object]
|
A URN, or a |
required |
resolve
|
bool | Sequence[type]
|
|
False
|
partial
|
bool
|
When resolving, trim referenced ItemSchemes by their
DataConstraints. |
True
|
expect
|
type[T] | None
|
The class the caller requires — the artefact is returned
when it is an instance, |
None
|
expect= names the class and gets that class or None (#406). A URN
parsed from a string carries no type a checker can use, and what one
checker infers for get(parse(...)) is not what another does (#399).
Passing the class states the expectation where the caller holds it:
reg.get(SdmxUrn.parse(text), expect=MetadataSet) # MetadataSet | None
The check is isinstance, the same operation the caller would write by
hand, so a response naming something else answers None rather than a
mistyped object.
Returns:
| Type | Description |
|---|---|
object
|
The artefact, or |
Raises:
| Type | Description |
|---|---|
SdmxConformanceError
|
If the provider served the artefact and it
was refused (#419) — see |
download ¶
download(
flow: Dataflow | Ref[Dataflow] | SdmxUrn[Dataflow],
*,
fmt: DataFormat,
key: str = "all",
start_period: str | None = None,
end_period: str | None = None,
to: Path | str | None = None,
) -> bytes | Path
Fetch a dataflow's data in fmt, exactly as the provider sent it.
Byte-faithful passthrough — nothing here parses, validates or reorders
the body, which is what separates it from obs. Use it to
keep a provider's own file, to feed a tool that wants SDMX-ML, or to
capture a payload for a bug report.
reg.download(flow_urn, fmt=sl.DataFormat.SDMX_ML_3_0, to="exr.xml")
Identity in, bytes out, and no structural fetch. flow names the
dataflow and nothing more; unlike obs this needs no DSD,
because there is no key to widen and no columns to interpret. A
download therefore costs exactly one request.
The format is asked for, never inferred. A path extension is
decorative on the SDMX-REST side and would be a silent trap here: a
file named .xml written with CSV bytes is worse than a refusal. So
fmt is required, to's extension is only checked against it, and
the response's Content-Type is checked too — a provider that
ignores Accept is caught rather than believed.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
flow
|
Dataflow | Ref[Dataflow] | SdmxUrn[Dataflow]
|
The dataflow, as an artefact, a |
required |
fmt
|
DataFormat
|
The format to request, sent as |
required |
key
|
str
|
SDMX key filter string, e.g. |
'all'
|
start_period
|
str | None
|
ISO period start, e.g. |
None
|
end_period
|
str | None
|
ISO period end. |
None
|
to
|
Path | str | None
|
Destination file. When given, the body is streamed there in chunks and never held whole — the only form suitable for a multi-GB flow. |
None
|
Returns:
| Type | Description |
|---|---|
bytes | Path
|
The response body when |
bytes | Path
|
written. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
obs ¶
A lazy query over this dataflow's observations, served over HTTP.
The same entry point and the same language as LocalRegistry.obs.
What differs is where the rows come from: this plans the query against
what SDMX REST's /data endpoint can answer, fetches exactly that,
materialises it, and then runs the original query locally — the
structural query() design, one plane over.
obs = reg.obs(flow_urn)
obs.filter(obs.col("CURRENCY").is_in(["USD", "GBP"])).frame()
Identity in, this registry's structure out. flow is used only to
name the dataflow; the DSD is resolved here, so a Dataflow resolved
from another registry cannot make this one interpret its rows with
foreign structure.
The binding is synthesised, not stored. SDMX-CSV names its columns
by component id, so DataflowBinding.from_dsd describes the fetched
table exactly — same artefact type as a producer's binding, different
provenance, and nothing downstream can tell them apart.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
flow
|
Dataflow | Ref[Dataflow] | SdmxUrn[Dataflow]
|
The dataflow, as an artefact, a |
required |
Returns:
| Type | Description |
|---|---|
ObsQuery
|
An |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the dataflow or its structure cannot be resolved — without a DSD there is no key order, and a key built on a guess queries a different slice silently. |
scan_csv ¶
An SDMX-CSV file's observations, as a query — the structure fetched from this registry.
Reads the file's DATAFLOW / STRUCTURE_ID column to name the
dataflow, fetches it with its structure, and reads the file with it:
the same ObsQuery obs(flow) returns, over the file's rows
instead of a fetch. A path is read in place on every query; bytes are
loaded once. See sdmxlib.local._scan.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
str | Path | bytes
|
File path or raw SDMX-CSV bytes. |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the |
Example
with sl.RestRegistry(sl.Provider.ECB) as reg: obs = reg.scan_csv("ecb_exr.csv") obs.frame(labels="en")
scan_json ¶
An SDMX-JSON data message's observations, as a query — the structure fetched from this registry.
Either version, told apart by its envelope. The dataflow its
rel="dataflow" link names is fetched with its structure, as
scan_csv does, and the message's rows are loaded once.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
str | Path | bytes
|
An SDMX-JSON data message, or its bytes. |
required |
Returns:
| Type | Description |
|---|---|
ObsQuery
|
An |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the message is not an SDMX-JSON data message, or names no dataflow. |
check_binding_drift ¶
Inspect binding for drift against this registry.
Performs at most one HTTP fetch (the bound Dataflow). Returns a
list of DriftIssue — empty if no drift is detected. Use this
when you want to enumerate or display drift; use
validate_binding() when you want to raise on any drift.
validate_binding ¶
Raise DriftError if binding has drifted against this registry.
For inspection without raising, use check_binding_drift().
Raises:
| Type | Description |
|---|---|
DriftError
|
If any drift is detected. |
FmrRegistry ¶
FmrRegistry(
endpoint: str = SDMX_GLOBAL_REGISTRY,
*,
auth: tuple[str, str] | BearerToken | None = None,
timeout: float = 30.0,
id: str | None = None,
store: LocalRegistry | None = None,
**kwargs: Unpack[HttpxTransportOptions],
)
Bases: RestRegistry
A RestRegistry configured for Fusion Metadata Registry endpoints.
Defaults to the SDMX Global Registry.
Example
with sl.FmrRegistry() as reg: ... with sl.FmrRegistry("https://my-fmr.example.com/sdmxapi/rest") as reg: ... with sl.FmrRegistry(auth=sl.BearerToken("eyJ...")) as reg: ... with sl.FmrRegistry(auth=("user", "pass")) as reg: ...
Provider ¶
Built-in SDMX REST endpoint URLs.
Each entry is a ProviderInfo carrying the endpoint URL and its
REST API version. Pass directly to RestRegistry::
with sl.RestRegistry(sl.Provider.ECB) as reg:
...
with sl.RestRegistry(sl.Provider.BIS) as reg:
...
Supports both attribute and subscript access::
sl.Provider.ECB # ProviderInfo(url="...", api="2.1")
sl.Provider["ECB"] # same
sl.Provider.ECB.url # "https://data-api.ecb.europa.eu/service"
sl.Provider.ECB.api # "2.1"
ResponseTooLargeError ¶
Bases: Exception
A provider refused a request as too large (SDMX-REST 413 / 414).
Distinct from PushdownError, which is our refusal to plan a fetch that
would be evaluated locally over everything. This one is the service's, it
arrives after the request was sent, and the fix is the same either way:
narrow the query.
Federation¶
Routing across multiple registries by agency. Every registry — REST, local, in-memory, federated — satisfies the RegistryReader protocol.
FederatedRegistry ¶
Read-side dispatcher across multiple Registry sources.
Routes each URN-keyed lookup by maintenance agency. Configure with a
routing map from agency id to a Registry source, plus
an optional default source for agencies absent from the map.
Satisfies the Registry Protocol —
a FederatedRegistry can itself be a source in another
FederatedRegistry (composition).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
routing
|
dict[str, RegistryReader]
|
Mapping of agency id (e.g. |
required |
default
|
RegistryReader | None
|
Optional fallback source for URNs whose agency is not
in |
None
|
Note
FederatedRegistry is a router, not a store. It has no
add() method — write to specific concrete registries
(typically the local one in your routing chain).
default
property
¶
The fallback source used when no routing entry matches.
source_for ¶
Public version of the routing decision — useful for diagnostics.
get ¶
get(
target: SdmxUrn[object] | Ref[object],
/,
*,
expect: type[T] | None = None,
resolve: bool | Sequence[type] = False,
partial: bool = True,
) -> object
Resolve target via the routed source.
Routing rules:
- URN agency in
routing→ dispatch to that source. - URN agency absent and
defaultis set → dispatch to default. - URN agency absent and no default → raise
KeyError.
None means the source answered "not here". An unreachable
source raises RegistryUnavailable naming it and the URN — §5.8.
There is no policy converting one into the other: they are different
facts, and the caller's response differs (retry a dead source; fall
back for a missing artefact).
resolve and partial mean what they mean on RestRegistry, and
are translated rather than forwarded blindly — see
_dispatch_kwargs for which source gets which, and why a local one
needs neither.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
target
|
SdmxUrn[object] | Ref[object]
|
Identity of the artefact to read — a URN, or a |
required |
resolve
|
bool | Sequence[type]
|
How deep to follow references. |
False
|
partial
|
bool
|
|
True
|
expect
|
type[T] | None
|
The class the caller requires. The artefact is returned
when it is an instance, |
None
|
Raises:
| Type | Description |
|---|---|
KeyError
|
If no source is routed for the URN's agency. |
TypeError
|
If |
RegistryUnavailable
|
If the routed source could not be reached. |
__contains__ ¶
True if the routed source already has urn cached.
Like the underlying Registry membership: never triggers a network
round-trip. False if no source is routed for the URN's agency.
check_binding_drift ¶
Inspect binding for drift against this federated registry.
Routes the bound Dataflow lookup through the configured sources;
only the routed source is touched. Returns a list of
DriftIssue — empty if no drift is detected. Use this when you
want to enumerate or display drift; use validate_binding()
when you want to raise on any drift.
validate_binding ¶
Raise DriftError if binding has drifted against this registry.
For inspection without raising, use check_binding_drift().
Raises:
| Type | Description |
|---|---|
DriftError
|
If any drift is detected. |
RegistryUnavailable ¶
Bases: Exception
A source could not be reached, so no read of it can be believed.
Wraps the underlying httpx / connection error so callers catch one type
without depending on the HTTP layer. Carries the source and what was
tried, because §5.8 asks the error to name both — a bare
ConnectError: connection refused names neither, and in a federation
the first question is always "which one?".
Record which source failed, doing what.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source_id
|
str | None
|
The registry's |
required |
cause
|
Exception
|
The underlying transport error, kept as |
required |
tried
|
str | None
|
What the read was after — a URN, or a description. Optional because a caller deep in the stack may not have it, and a message naming the source alone still beats naming neither. |
None
|
RegistryReader ¶
Bases: SupportsGet, Protocol
A URN-keyed source of SDMX artefacts.
Minimal Protocol — implementations may offer richer APIs (transitive resolution, bulk fetch, browse) but every RegistryReader must support these two operations so federation dispatchers can compose them uniformly.
__contains__ ¶
True if the artefact named by target is available from this source.
Takes an SdmxUrn or a Ref, matching get. object rather than
that union because in must answer no for an unrelated value
instead of raising — the narrowing happens inside (#370).
Positional-only, like get: in never calls it by keyword, and
without the marker every implementation has to agree on the parameter
name to satisfy this Protocol. They did not — RestRegistry said
urn — which is a naming constraint no caller can benefit from.
Local registry & storage¶
Persistent DuckDB-backed structural store, the in-memory registry, and the data plane that shares the store's connection.
LocalRegistry ¶
LocalRegistry(
uri: str | Path,
*,
source_registry: str | None = None,
resolve_cache_items: int = DEFAULT_RESOLVE_CACHE_ITEMS,
target: str | None = None,
)
Bases: SessionFactory
DuckDB-backed persistent SDMX artefact store.
Path-style construction. LocalRegistry(uri) is cheap — it just stores
the URI. The DuckDB connection is opened by open() (or implicitly
by __enter__ when used as a context manager) and closed on exit.
Provides the same read interface as the read-only RegistryReader
Protocol, plus mutations (add, drop, replace) and direct SQL
helpers (code_map, dsd_code_map) that avoid creating Python
objects.
Typical usage::
with LocalRegistry("store.duckdb") as reg:
reg.add(codelist)
cl = reg.get(codelist_urn) # LazyCodelist
Sharing an existing DuckDB connection (e.g. when co-located with a
DuckLake catalog the caller already opened) is done via
from_connection, which doesn't take ownership of the
connection's lifecycle.
Concurrency. Reads (get / get_all / codelist /
code_map / dsd_code_map / to_registry / __contains__
/ __len__) are safe to call from multiple threads against the
same LocalRegistry. Writes (add / drop / replace /
add_all / ingest) are single-threaded — serialise externally
if you need to mutate from multiple threads. See
docs/design/concurrency.md for the model and deployment
recipes.
Open or wrap a DuckDB-backed registry.
target declares which SDMX-ML version this store's contents are
meant to be written as — "2.1", "3.0" or "3.1". Adding an
artefact carrying something that format cannot represent then raises
rather than storing it, because a declared target is a statement of
intent and losing a field at write time is exactly what the declaration
was made to prevent.
None (the default) keeps the store format-agnostic, which is
correct for anything holding parsed data: a cache of a 2.1 registry
must accept what that registry publishes, and a target would refuse it.
Declare one when authoring; leave it unset when mirroring.
resolve_cache_items bounds the FIFO cache of resolved Ref
targets that get / get_all / to_registry
share across calls. Pass 0 to disable caching (matches
pre-v0.27.2 behaviour — one full read per Ref). The cache is
cleared on every mutation (add / drop / replace) for
invalidation correctness. FIFO eviction (not LRU) keeps cache
reads lock-free; see docs/design/concurrency.md.
connection
property
¶
The store's base DuckDB connection — stable, and safe to hold.
Use it for setup, DDL, and as the base another component derives its
own handles from (DuckDBDataStore.from_registry stores it in a
field, which is correct precisely because this object does not move).
Not the handle to execute on from more than one thread.
DuckDBPyConnection is not thread-safe, which is why every query here
runs on a per-thread handle instead (#75/#80). For custom SQL use
execute or session, which resolve that handle for you and cannot be
captured by accident (#300).
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If the registry hasn't been opened. |
closed
property
¶
True once close has run, or before open: a reference it bound can no longer resolve.
refused
property
¶
Every stored artefact a read refused since the store last changed (#492).
A store read checks each artefact it builds as add would. get raises
for one it refuses; every other read — get_all, a query's
collect(), resolving a reference — leaves it out, records it here,
and reads on, as parse(strict=False) does. A reference to it stays
unresolved. add, drop and replace clear it: what they changed may
be what was refused.
open ¶
Open the DuckDB connection and run any pending schema migrations.
Returns self so the call can be chained with a with::
with LocalRegistry("store.duckdb").open() as reg:
...
Idempotent within a single open/close cycle; calling open()
on an already-open registry is a no-op.
from_connection
classmethod
¶
from_connection(
conn: DuckDBPyConnection,
*,
source_registry: str | None = None,
resolve_cache_items: int = DEFAULT_RESOLVE_CACHE_ITEMS,
target: str | None = None,
) -> Self
Wrap an existing DuckDB connection.
The LocalRegistry does NOT take ownership of the connection — closing the LocalRegistry leaves the connection open. Use this when the connection is shared with another component (e.g. a co-located DuckLake catalog) that owns its lifecycle.
resolve_cache_items configures the resolver cache (FIFO), and target
declares an SDMX-ML version to hold the contents to; both are
described on __init__.
close ¶
Close the database connection (only if we own it).
After close(), the registry can be re-opened with open().
Drops the per-thread cursor cache — reopen starts fresh.
transaction ¶
Group writes into a single DuckDB transaction.
Wraps any sequence of add / drop / replace calls in
one BEGIN/COMMIT so DuckDB amortises WAL flush and index
maintenance across the batch instead of paying autocommit per
statement::
with reg.transaction():
for artefact in many_artefacts:
reg.add(artefact)
On exception the transaction rolls back and is re-raised. Either way, the resolver cache is cleared once at exit (a partial in-flight state may have left it stale).
Nestable as a no-op: re-entering transaction inside an
already-open block does not emit a second BEGIN — DuckDB
rejects nested transactions, and silently joining the outer
block is what callers usually want.
When the sdmxlib.local.registry logger is set to DEBUG,
a one-line transaction_summary is emitted at exit with the
artefact-count and total-wall-time breakdown by class.
add ¶
Add or replace an artefact in the store.
source_registry overrides the registry-level default for this
call. None (default) means "use the registry-level default."
Clears the resolver cache: any cached resolved target may now be
stale (e.g. a freshly-added Codelist replaces the one a previous
get had cached).
Single-threaded. Mutations are not safe under concurrent
callers — do not invoke add / drop / replace from
multiple threads against the same LocalRegistry. Concurrent reads
through get / get_all / code_map etc. are safe. See
docs/design/concurrency.md.
add_all ¶
add_all(
artefacts: Iterable[SupportsUrn],
*,
source_registry: str | None = None,
rewrite_version: VersionRewrite | None = None,
) -> tuple[RewrittenVersion, ...]
Add multiple artefacts in a single transaction.
Equivalent to::
with reg.transaction():
for a in artefacts:
reg.add(a, source_registry=source_registry)
except that every version in the batch is checked before anything is written, so a refusal names every problem at once.
Convenience wrapper for the common case where the caller has a
ready iterable. For interleaved or conditional writes, use
transaction() directly.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
artefacts
|
Iterable[SupportsUrn]
|
The batch. |
required |
source_registry
|
str | None
|
Overrides the registry-level default, as for |
None
|
rewrite_version
|
VersionRewrite | None
|
Rewrites versions SDMX 3.x cannot hold — a mapping
from published to ingested version, or a function such as
|
None
|
Returns:
| Type | Description |
|---|---|
tuple[RewrittenVersion, ...]
|
One |
Raises:
| Type | Description |
|---|---|
ValidationError
|
If a version only SDMX 2.1 accepts is left after the rewrite, if the rewrite makes two artefacts one or replaces one the store holds, or if an artefact outside the batch references a version the batch rewrites. |
codelist ¶
Return a LazyCodelist for tree navigation without materializing codes.
Faster than get() for hierarchy traversal — skips ref resolution
and only reads the codelist spine row::
cl = local.codelist(geo_urn)
cl.roots() # top-level codes — SQL, no Python objects
cl.narrower("1") # direct children — SQL, only rows needed
get ¶
get(
target: SdmxUrn[object] | Ref[object],
/,
*,
expect: type[T],
eager: bool = False,
) -> T | None
get(
target: SdmxUrn[Codelist] | Ref[Codelist],
/,
*,
eager: Literal[False] = False,
) -> LazyCodelist | None
get(
target: SdmxUrn[object] | Ref[object],
/,
*,
expect: type[T] | None = None,
eager: bool = False,
) -> object
Look up an artefact by URN, or by a Ref naming one.
For Codelists, returns a LazyCodelist by default (codes not
materialized). Pass eager=True to get a real Codelist.
An untyped URN answers object | None. SdmxUrn.parse(text) and
artefact_urn(x) are SdmxUrn[object] (#399): a URN string can name
anything, so this cannot say which class comes back, and the caller
narrows. A typed URN — Codelist.urn_for(...), Codelist.parse_urn(text),
artefact.urn — picks the matching overload and its class.
Until #399 those were SdmxUrn[Any], which made the call ambiguous
across these overloads; basedpyright then answered a confident, wrong
LazyCodelist | None and pyrefly an unchecked Any | None.
tests/typing/probe_urn_any.py pins what is declared now.
expect= names the class when the URN does not carry it (#406).
It names the class, so T comes from the argument rather than from the
URN's phantom, and the ambiguity above never arises:
local.get(SdmxUrn.parse(text), expect=MetadataSet) # MetadataSet | None
The check is isinstance at runtime, which makes it the same operation
the caller would otherwise write by hand — a URN read out of a store may
legitimately name something this call site cannot use, and None is how
that is said.
With the lazy roots, expect what comes back. expect=Codelist
answers None for a stored codelist, because a LazyCodelist is not a
Codelist. Ask for what you will get: expect=LazyCodelist,
expect=CodelistLike for either, or eager=True alongside
expect=Codelist.
The two lazy roots are declared, not implied (#397). A
SdmxUrn[Hierarchy] used to give Hierarchy | None here while a
LazyHierarchy came back — and that one has no codes, tree or
descendants at all, so local.get(h_urn).codes type-checked and
raised AttributeError. The overloads above say which class each URN
actually produces, in both eager= branches: the flag changes the
returned type, and one declaration could not describe both.
Every Ref on the returned artefact is bound to this store and
finds its target on access (#492), so standard traversal works::
dsd = local.get(dsd_urn)
dsd.dimensions["FREQ"].codelist # -> LazyCodelist
dsd.code_map(lang="en") # works
A Ref is accepted for the same reason RestRegistry.get accepts one
— traversal hands them out. It used to return None here instead,
silently; a str used to work here and nowhere else (#370).
reported_attributes ¶
A lazy query over reported reference-metadata values (#281).
The SQL counterpart to metadata_report, which projects one set at a
time. This filters across the whole catalogue in one statement — "every
QUALITY.COVERAGE statement", "everything said about this dataflow"::
reg.reported_attributes().filter(
sl.q.reported_attribute.metadata_set.targets.artefact.urn == flow_urn
).collect()
Exactly reg.query(sl.q.reported_attribute), kept as a named entry point
because "the reported values" is a question worth a name and because
metadata_report is its per-set counterpart. Since #306 it is the same
Query every other root returns — the bespoke MetadataValueQuery is
gone.
Returns:
| Type | Description |
|---|---|
Query[ReportedAttributeRow, MetadataSection]
|
A |
Query[ReportedAttributeRow, MetadataSection]
|
|
Query[ReportedAttributeRow, MetadataSection]
|
terminals when the declared type information is not needed. |
artefacts ¶
A lazy query over every maintainable artefact, narrowed by class.
The polymorphic counterpart to the named roots. reg.query(q.codelist)
is the spelling when the type is known at the call site; this is for
when it is a variable, which is the case get_all used to serve by
loading and resolving everything::
reg.artefacts(sl.Dataflow).filter(q.artefact.agency == "ESTAT").urns()
Since #306 it returns the same Query every other root does, so it
composes with filter / order_by / select / group_by and the
whole terminal surface. artefact_type applies the same discriminator
the named roots carry — q.artefact.type is that column, exposed.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
artefact_type
|
type[object] | None
|
Narrow to one SDMX class. Omit to sweep every maintainable artefact in the catalogue. |
None
|
Returns:
| Type | Description |
|---|---|
Query[ArtefactRow, Maintainable]
|
A |
Raises:
| Type | Description |
|---|---|
TypeError
|
If artefact_type is not a stored SDMX artefact type — naming the class, since the alternative is an empty result. |
query ¶
A lazy query over the relation root names (#306).
The generic entry point for the path query language: pass any accessor
from sl.q and compose with filter / limit / offset, then take a
terminal.
reg.query(sl.q.dsd).filter(
sl.q.dsd.dimensions.any(sl.q.component.concept.id == "REF_AREA")
).ids()
Generic rather than one method per relation because the accessor is the witness: it carries both the relation and its row kind, so a predicate built for another root is a type error rather than a wrong join.
Nothing runs until a terminal — ids(), urns(), count(),
exists(), collect(), frame(). See sdmxlib.query.query.Query.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
root
|
Cols[R, A]
|
A relation accessor from |
required |
Returns:
| Type | Description |
|---|---|
Query[R, A]
|
A |
Query[R, A]
|
and its batched resolver. |
sql ¶
Run raw SQL against this registry's layout — the escape hatch (#306).
Two things at once. A terminal:
reg.sql("SELECT agency_id, count(*) FROM sdmx.artefact GROUP BY 1").collect()
and a single-column relation the structural DSL can splice:
urns = reg.sql("SELECT urn FROM sdmx.artefact WHERE resource_id LIKE $p", p="CL_%")
reg.query(sl.q.dataflow).filter(sl.q.dataflow.urn.is_in(urns))
Deliberately a different method on a different object from filter.
The structural DSL still never accepts text; this is visibly not it at
the call site, which is the property that ban was protecting.
Values are parameters, never interpolation. Placeholders are
$name and every one needs a matching keyword argument. Name columns
through sl.schema rather than as literal strings, so a raw query is
coupled to the layout and a schema change is a loud failure.
Anything reached often through here is a candidate for the DSL — that is what the hatch is for.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
text
|
str
|
One SQL statement, with |
required |
params
|
object
|
A value per placeholder. |
{}
|
Returns:
| Type | Description |
|---|---|
RawSql
|
A lazy |
Raises:
| Type | Description |
|---|---|
RawSqlError
|
If the SQL will not parse, is more than one statement, or its placeholders and arguments do not correspond. |
RuntimeError
|
If the registry is closed. |
get_all ¶
All artefacts of the given Python type.
Resolution shares one call-scoped memo across the batch and is
backed by the registry's persistent _resolve_cache, so any
parent scheme referenced by multiple artefacts is loaded once,
not once per artefact.
applicable_hierarchies ¶
All Hierarchies applicable to dim_id on dataflow.
See InMemoryRegistry.applicable_hierarchies for semantics. Read off
sdmx.hierarchy_applicability, which is where the rule lives; the
in-memory registry walks the objects for the same answer, and
tests/unit/test_query_applicability.py compares them.
unique_hierarchy ¶
The single applicable Hierarchy, None if none, raises on >1.
See InMemoryRegistry.unique_hierarchy for semantics. "More than one"
means within the winning tier: a scoped association beating an
unscoped one is a resolution, not a conflict, while two scoped ones
are a conflict the caller asserted would not happen.
bundle
classmethod
¶
bundle(
path: str | Path,
*,
source_registry: str | None = None,
resolve_cache_items: int = DEFAULT_RESOLVE_CACHE_ITEMS,
) -> LocalRegistry
Open a published bundle — artefacts and observations together.
The standard layout, which a bundle builder writes and a downstream SDMX REST service serves::
<path>/meta.duckdb sdmx.* artefacts, including the bindings
<path>/lake.ducklake DuckLake catalog
<path>/parquet/ DuckLake-managed observation data
A constructor, not a type: a bundle answers exactly what any other local registry answers, so it differs in wiring rather than in method set — §9's rule. The lake is ATTACHed lazily, when something first asks for observations, so opening a bundle to read a codelist costs nothing.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str | Path
|
The bundle directory. |
required |
source_registry
|
str | None
|
Provenance tag for artefacts added later. |
None
|
resolve_cache_items
|
int
|
As the constructor. |
DEFAULT_RESOLVE_CACHE_ITEMS
|
Returns:
| Type | Description |
|---|---|
LocalRegistry
|
An open registry over the bundle. |
binding_for ¶
Where this dataflow's observations live, and what they mean.
A DataflowBinding is a registration in everything but name — the
record of where a provider's data for one dataflow actually sits — so
it is read from this registry like any other artefact.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
flow
|
Dataflow | Ref[Dataflow] | SdmxUrn[Dataflow]
|
The dataflow, as an artefact, a |
required |
Returns:
| Type | Description |
|---|---|
DataflowBinding | None
|
The binding, or |
dsd ¶
The data structure this dataflow is built on, fully resolved.
Read back by URN, not off the dataflow's ref. Locally the two
converge — get already resolves a stored artefact's refs, so
flow.structure() would usually answer too. By URN is the form that
holds when it does not: a dataflow whose structure ref arrived
unresolved still yields a resolved DSD here, and labelling skips any
component whose codelist is unresolved.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
flow
|
Dataflow | Ref[Dataflow] | SdmxUrn[Dataflow]
|
The dataflow, as an artefact, a |
required |
Returns:
| Type | Description |
|---|---|
DataStructure | None
|
The |
DataStructure | None
|
cannot be resolved here. |
obs ¶
A lazy query over this dataflow's observations.
The same language as query, rooted at a dataflow rather than a named
relation — because there is no singleton observation relation to name:
a DSD fixes each dataflow's columns, so the root is built per binding::
obs = reg.obs(flow)
(
obs.filter(obs.col("REF_AREA") == "ON")
.group_by(obs.col("SEX"))
.agg(total=obs.col("OBS_VALUE").sum())
.collect()
)
Identity in, this registry's structure out. flow names the
dataflow; the DSD is resolved here, so an artefact resolved from
another registry cannot make this one read its rows with foreign
structure.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
flow
|
Dataflow | Ref[Dataflow] | SdmxUrn[Dataflow]
|
The dataflow, as an artefact, a |
required |
Returns:
| Type | Description |
|---|---|
ObsQuery
|
An |
Raises:
| Type | Description |
|---|---|
KeyError
|
If this registry has no binding for the dataflow — which means it holds no rows for it, and naming the flow is the fix. |
scan_csv ¶
An SDMX-CSV file's observations, as a query — the structure this registry holds.
The file names its own structure (STRUCTURE / STRUCTURE_ID,
or 1.0's DATAFLOW), and it is looked up here, so what is needed
is the dataflow and its structure — not a binding, and not rows::
with sl.LocalRegistry(":memory:") as reg:
reg.add_all([codelists, concepts, dsd, flow])
obs = reg.scan_csv("exr.csv")
obs.pivot()
The same ObsQuery obs(flow) returns, over the file's
rows: filters, pivot, validate, the sinks. A path is read in place
on every query; bytes are loaded once. See sdmxlib.local._scan.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
str | Path | bytes
|
An SDMX-CSV file, or its bytes. |
required |
Returns:
| Type | Description |
|---|---|
ObsQuery
|
An |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the file names no structure, or one this registry cannot resolve; or if it names a data structure rather than a dataflow or provision agreement — an observation query is over a dataflow, and a structure alone does not say which. |
scan_json ¶
An SDMX-JSON data message's observations, as a query — the structure this registry holds.
Either version, 1.0 or 2.0, told apart by its envelope. The message
names its dataflow by a rel="dataflow" link, which is looked up
here, as scan_csv looks up a file's STRUCTURE_ID. JSON does not
stream, so the message is decoded and its rows loaded once.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
str | Path | bytes
|
An SDMX-JSON data message, or its bytes. |
required |
Returns:
| Type | Description |
|---|---|
ObsQuery
|
An |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the message is not an SDMX-JSON data message, names no dataflow, or names one this registry does not hold. |
browse ¶
Every dataflow this registry holds rows for.
Keyed on bindings, so it answers "which dataflows have observations
here" rather than "which dataflows exist" — the second is
query(sl.q.dataflow), and a remote provider has no cheap way to
answer the first at all, which is why this is a local method.
Returns:
| Type | Description |
|---|---|
list[DataflowEntry]
|
One |
check_binding_drift ¶
Inspect binding for drift against this registry.
Returns one DriftIssue per detected mismatch — empty list means
the pinned structural state matches the current registry. Use this
when you want to enumerate or display drift; use
validate_binding() when you want to raise on any drift.
validate_binding ¶
Raise DriftError if binding has drifted against this registry.
For inspection without raising, use check_binding_drift().
Raises:
| Type | Description |
|---|---|
DriftError
|
If any drift is detected. |
metadata_for_target ¶
metadata_for_target(
target_urn: str | SdmxUrn[object],
*,
structure_urn: str | SdmxUrn[object] | None = None,
) -> list[MetadataSet]
All MetadataSets whose targets include target_urn.
The reverse of MetadataSet.targets — given any artefact URN
(a Code, Codelist, Concept, Dataflow, …), return every reported
MetadataSet attached to it. This is the SDMX 3.0
/metadata/structure/{artefactType}/… by-target lookup realised
locally, and the primitive both the REST view and the search
indexer navigate.
Sets are returned without resolving their target/structure Refs (the URNs are all present) — cheap even for sets with many targets. Ordered by set URN for determinism.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
target_urn
|
str | SdmxUrn[object]
|
The artefact URN to find attached metadata for. |
required |
structure_urn
|
str | SdmxUrn[object] | None
|
Optional MSD or Metadataflow URN to filter by, e.g. return only reference-guide sets targeting the code. |
None
|
Returns:
| Type | Description |
|---|---|
list[MetadataSet]
|
The matching MetadataSets (empty list if none). |
metadata_report ¶
Flattened, MSD-typed view of a MetadataSet's reported values.
Returns one MetadataSection per metadata_value row,
each carrying the leaf attribute's declared text_type (e.g.
"String" / "XHTML") and is_presentational flag joined
from the MSD. Sections are ordered by (path, sibling_index).
This is the sectioned input a chunker / search indexer consumes — it turns the reported tree + schema into a flat, typed list without the caller re-joining the MSD. Returns an empty list if the set has no values (or does not exist).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
set_urn
|
str | SdmxUrn[object]
|
The MetadataSet URN to project. |
required |
__contains__ ¶
True if this store holds the artefact named by target.
Takes a Ref as well as an SdmxUrn, for the same reason get does
(#370) — ref in reg answered False for a present artefact.
Anything else is still False rather than an error: in is a
question, and the answer for an unrelated object is "no".
drop ¶
Remove an artefact by URN. Returns True if it existed.
Clears the resolver cache (cached resolved targets may reference the dropped artefact).
Single-threaded — see add.
replace ¶
Remove the old artefact and add the new one.
Each leg clears the resolver cache via add / drop;
the net effect is a single fully-cleared cache at the end.
Single-threaded — see add.
clear_resolve_cache ¶
Purge the resolver cache and reset hit/miss counters.
Useful for explicit invalidation hooks (e.g. a webhook telling the
service that upstream metadata changed without going through this
registry's mutation API). Cache also clears automatically on every
add / drop / replace.
resolve_cache_info ¶
Return resolver cache stats: entries, weight, max_items, hits, misses, refused.
Intended for ops dashboards and integration tests asserting cache
behaviour. entries and weight are 0 after construction or any
mutation and rise as get / get_all populate it; weight is
the one capped, at max_items.
weight is the number to watch, not entries — that is the
whole of #386. An entry is an artefact, and artefacts differ in size by
four orders of magnitude, so five of them can be 4.4 GB. entries
remains reported because a hit rate is per lookup, not per item.
refused counts artefacts too heavy for the bound to hold at all.
A non-zero value with a poor hit rate is the signature of a working set
that does not fit: raise resolve_cache_items, or accept the re-read.
code_map ¶
Return {code_id: label} for a codelist without creating Code objects.
dsd_code_map ¶
Return {component_id: {code_id: label}} for all enumerated dimensions.
ingest ¶
Copy all artefacts from an InMemoryRegistry into this store.
source_registry overrides the registry-level default for this
bulk import.
to_registry ¶
Load everything into an InMemoryRegistry, every reference bound to it.
Each artefact is built bound to the new registry (#492), so add
copies nothing, and every reference finds its target there once the
load is done — the whole store is in it, so nothing is resolved here.
InMemoryRegistry ¶
In-memory SDMX artefact store with URN identity and bound references.
Satisfies the Registry Protocol via get(urn) and
__contains__(urn).
Example
reg = InMemoryRegistry() reg.add_all([cl_freq, concept_scheme, dsd])
dsd = reg.get(DataStructure.urn_for(agency="ESTAT", id="CPI", version="1.0")) dims = reg.get_all(Dimension) # not typically used; prefer dsd.dimensions
add ¶
Add an artefact to the registry.
- Stores a copy whose every
Refis bound to this registry, so it finds its target here — now, or once the target is added (#492). The caller's object is left as built: read the stored one back withget. - Every reference already bound here finds this artefact at once.
A version only SDMX 2.1 accepts, on the artefact or on a reference
inside it, is refused: the model holds SDMX 3.x (#420). add_all
takes a rewrite for those.
Raises:
| Type | Description |
|---|---|
TypeError
|
If the artefact does not satisfy |
ValidationError
|
If a version only SDMX 2.1 accepts is present. |
add_all ¶
add_all(
artefacts: Iterable[SupportsUrn],
*,
rewrite_version: VersionRewrite | None = None,
) -> tuple[RewrittenVersion, ...]
Add multiple artefacts in any order, refusing the batch as a whole.
Every artefact is checked before any is added, so a refusal leaves the registry as it was and names every problem at once.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
artefacts
|
Iterable[SupportsUrn]
|
The batch. |
required |
rewrite_version
|
VersionRewrite | None
|
Rewrites versions SDMX 3.x cannot hold — a mapping
from published to ingested version, or a function such as
|
None
|
Returns:
| Type | Description |
|---|---|
tuple[RewrittenVersion, ...]
|
One |
Raises:
| Type | Description |
|---|---|
ValidationError
|
If a version only SDMX 2.1 accepts is left after the rewrite, if the rewrite makes two artefacts one or replaces one this registry holds, or if an artefact outside the batch references a version the batch rewrites. |
get ¶
Look up an artefact by exact URN, or by a Ref naming one.
Returns None if absent. A str raises TypeError — it used to
work here by accident and nowhere else (#370).
An item URN answers the item: get of
…codelist.Code=AG:CL(1.0).A is that Code, found inside the
codelist this registry holds. The store keeps maintainables, so this
used to answer None for every item URN — which made
reg.get(dim.concept.urn) a silent miss while
reg.get(dsd.urn) worked, and there is no reading of "look up an
artefact by URN" under which a Code is not one (#405). The lookup is
lazy: a miss, then one O(1) step inside the scheme the URN names, so
holding a 579,951-code codelist costs nothing until someone asks for a
code.
expect= asks for a class and gets that class or None. A URN
parsed from a string carries no type — SdmxUrn.parse answers
SdmxUrn[object], so this answers object | None (#399). Passing the
class states the expectation where the caller actually holds it:
reg.get(SdmxUrn.parse(text), expect=MetadataSet) # MetadataSet | None
The check is isinstance, which makes it the same operation a caller
writes by hand — a store row may legitimately name something other than
what this call site can use, and returning None is how that is said
(#406).
get_all ¶
All artefacts of the given Python type (exact match, not subclasses).
In insertion order, and one entry per URN: re-adding an artefact
replaces it here exactly as it does in get.
ref ¶
A Ref bound to this registry: it finds its target here, now or once it is added.
__contains__ ¶
True if an artefact with this URN is stored.
Takes a Ref as well as an SdmxUrn, for the same reason get does
— ref in reg answered False for a present artefact before
370. Anything else is still False rather than an error: in¶
is a question, and the answer for an unrelated object is "no".
Answers for an item URN whatever get answers. Kept in step
deliberately: urn in reg being False while reg.get(urn) returns
the code is not a smaller bug than #405, it is a stranger one.
applicable_hierarchies ¶
All Hierarchies whose HierarchyAssociations bind dim_id on dataflow.
Returns dataflow-scoped HAs first (sorted lex by HA URN), then
unscoped HAs (also lex-sorted). HAs whose context_object
names a different dataflow are skipped.
Per the SDMX 3.x IM the relation is many-to-many — this is the
canonical enumeration surface. Use unique_hierarchy when the
caller asserts at most one applies.
unique_hierarchy ¶
The single Hierarchy applicable to dim_id on dataflow, when exactly one applies.
Returns None if no HierarchyAssociation binds the dimension.
Returns the single applicable Hierarchy's Ref if exactly one
applies (dataflow-scoped wins over unscoped at the same dim).
Raises AmbiguousHierarchyError if more than one HA applies at
the chosen scope — call applicable_hierarchies to enumerate.
check_binding_drift ¶
Inspect binding for drift against this registry.
Returns one DriftIssue per detected mismatch — empty list means
the pinned structural state matches the current registry. Use this
when you want to enumerate or display drift; use
validate_binding() when you want to raise on any drift.
validate_binding ¶
Raise DriftError if binding has drifted against this registry.
For inspection without raising, use check_binding_drift().
Raises:
| Type | Description |
|---|---|
DriftError
|
If any drift is detected. |
Session ¶
A scoped execution handle. Valid only inside its with block.
Obtained from SessionFactory.session, normally as registry.session().
Do not store one: the point of the scope is that the handle's lifetime is
visible, and a Session outliving its block refers to a handle that may
have been released or handed to another user of the thread.
handle
property
¶
The raw DuckDB handle, for calls this class does not wrap.
Escape hatch within an escape hatch. Same rule as the Session itself:
use it inside the block, do not keep it.
execute ¶
Run sql on this session's handle.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
sql
|
str
|
The statement. |
required |
params
|
list[object] | None
|
Bind parameters. Values only — never interpolate them into sql, per the SQL style rule in CLAUDE.md. |
None
|
Returns:
| Type | Description |
|---|---|
DuckDBPyConnection
|
The handle, positioned on the result, so |
DuckDBPyConnection
|
friends chain as they would on a raw connection. |
value_set ¶
Register values as a relation for the duration of the block (#271).
An eager is_in binds the list as one parameter (= ANY(?)), and
DuckDB's Python client converts its elements one at a time. Registering hands it a pointer to
an Arrow buffer instead, so the bind path is linear in the number of
values and the registered path is very nearly flat::
with reg.session() as active, active.value_set(ids) as wanted:
codes.filter(sl.code.id.is_in(wanted)).collect()
How linear depends on whether pandas is installed. Converting each
bound value, DuckDB checks whether it is pandas.NaT or
pandas.NA, and with pandas absent that check retries
import pandas — twice per value, every time, because DuckDB does
not remember the failure (#411). Median of 7 on a 200k-row table,
duckdb 1.5.5:
N bound, no pandas bound, pandas registered 10 5.4 ms 4.1 ms 6.5 ms 100 13.7 ms 4.3 ms 5.4 ms 500 43.6 ms 5.0 ms 5.5 ms 1000 85.6 ms 5.4 ms 5.5 ms 5000 445 ms 10.0 ms 6.5 ms 20000 1635 ms 26.6 ms 10.1 ms 50000 4121 ms 59.1 ms 13.6 ms
Reach for this at about a hundred values without pandas, about a thousand with it. sdmxlib does not depend on pandas, so the first column is the default; below the crossover a plain list is as fast and simpler. The shape is the same in both and is what the feature rests on — linear against flat, so the gap keeps widening.
Re-measure before quoting these: they are properties of a dependency and its environment, not of this library. The history shows why —
271 measured ~48 us a value, a later re-measure ~1.9 us and called¶
DuckDB "25x faster", and the difference was pandas being importable in one run and not the other, not a DuckDB change.
Why the caller owns the window. A registered relation is session
state: it outlives the query, a fixed name would let two callers clobber
each other, and a lazily-built query separates "register" from "execute"
with no safe moment to free in between. Scoping it to a block removes
all three — the name is generated, the relation exists only inside, and
finally frees it even on error.
Inside the block, to_sql is self-contained. The relation is on
this session's handle, which is the handle the DSL uses, so a compiled
query naming it can be handed to another library and executed — the
case an automatic threshold could never serve, because the values would
be gone before the SQL was.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
values
|
Iterable[object]
|
The set. Order and duplicates are irrelevant; |
required |
Yields:
| Type | Description |
|---|---|
Generator[ValueSet]
|
A |
ValueSet ¶
A caller-supplied set of values, registered as a relation (#271).
Satisfies sdmxlib.sql.SqlRelation, so it goes straight into
is_in with no change to the DSL — the seam has accepted a relation since
261. What was missing was a way to turn a plain Python list into one.¶
Obtained from Session.value_set, and valid only inside that block.
to_sql ¶
The relation, with no binds at all.
That is the entire point: the values live in the registered Arrow
table, not in the parameter list. See Session.value_set.
DataStore ¶
Bases: Protocol
Write-side contract for the data plane.
Implementations must accept a DataflowBinding and materialise rows into
the storage they manage. Reading goes through
DuckDBDataStore.obs — see the module docstring for why that is
not stated here.
write_observations ¶
write_observations(
*,
schema: str,
table: str,
source: WriteSource,
partition_by: tuple[str, ...] = (),
mode: Literal["overwrite", "append"] = "overwrite",
lake_alias: str = "lake",
) -> None
Materialise observation rows into a backend-managed table.
Implementations route the write to whatever physical storage they manage (DuckLake, Iceberg, Postgres, raw Parquet). The caller side is uniform: schema + table + a source that can be turned into rows.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
schema
|
str
|
Target schema. |
required |
table
|
str
|
Target table name. |
required |
source
|
WriteSource
|
Rows to write. Accepted shapes vary by implementation but Polars frames and Arrow tables are the portable lowest common denominator. |
required |
partition_by
|
tuple[str, ...]
|
SDMX dimension ids the table is partitioned on.
Recorded on the matching |
()
|
mode
|
Literal['overwrite', 'append']
|
|
'overwrite'
|
lake_alias
|
str
|
For ATTACH-based backends, the alias of the target catalog. Implementations that don't use ATTACH may ignore this argument. |
'lake'
|
WriteSource ¶
What write_observations accepts as a source of rows.
Public and defined once, because it appears in the signature of a Protocol
method: anyone implementing DataStore outside this package has to name this
type, and until #299 it existed only under TYPE_CHECKING in two modules —
spelled identically in both, importable from neither.
Runtime rather than type-checking-only: duckdb, polars and pyarrow are
all baseline runtime dependencies, so there is nothing to defer. A comment in
local/data_store.py claimed pyarrow was optional; it has not been since the
dependency baseline was set.
DuckDBDataStore ¶
DuckDBDataStore(
uri: str | Path,
*,
attach: dict[str, str | tuple[str, str]] | None = None,
extensions: list[str] | None = None,
azure_credentials: dict[str, str] | None = None,
)
Bases: SessionFactory
A DataStore backed by DuckDB.
Two construction paths:
DuckDBDataStore(uri)— Path-style. Opens its own DuckDB connection atopen()/__enter__time. Use when the data store stands alone (no registry sharing).DuckDBDataStore.from_registry(reg, attach=, extensions=)— reuses aLocalRegistry's connection so the SDMX structural side and the observation side share a single DuckDB session. Lazy codelists keep working across queries, and queries can join across the two without cross-connection plumbing.
The attach / extensions / azure_credentials knobs
configure the DuckDB session at open time. attach maps
alias → ATTACH spec (each executed as
ATTACH <spec> AS <alias>); extensions INSTALL+LOAD the
listed extensions before any ATTACH. azure_credentials is a
dict that drives a DuckDB CREATE SECRET for ADLSv2 access — see
below.
Azure credential dicts accept one of three shapes:
{"connection_string": "..."}→PROVIDER 'config'withCONNECTION_STRING.{"account_name": ..., "tenant_id": ..., "client_id": ..., "client_secret": ...}→PROVIDER 'service_principal'.{"chain": "cli"}(or"env","managed_identity") →PROVIDER 'credential_chain'with the named chain.
When azure_credentials is set, or any ATTACH spec contains
abfss://, the azure extension is auto-INSTALL+LOADed so
DATA_PATH URIs like abfss://acct.dfs.core.windows.net/c/p/ work
out of the box.
Example
with sl.LocalRegistry("sdmx.duckdb") as reg: store = sl.DuckDBDataStore.from_registry( reg, attach={"lake": "ducklake:'catalog.duckdb' (READ_ONLY)"}, extensions=["ducklake"], ) obs = store.obs(binding) df = obs.filter(obs.col("REF_AREA") == "CA").frame()
connection
property
¶
The store's base DuckDB connection — stable, and safe to hold.
For setup, DDL, and as the object another component derives its own
handles from. Not the handle to run queries on: it is shared by
every thread and DuckDBPyConnection is not thread-safe, which is why
the library executes on per-thread handles (#75/#80). Use session or
execute for custom SQL — they resolve that handle, share its session
state with the DSL, and cannot be captured by accident (#300).
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If the data store hasn't been opened. |
registry
property
¶
The LocalRegistry whose connection is shared, or None.
Set when the data store was constructed via from_registry;
None for standalone Path-style construction.
open ¶
Open the DuckDB connection and apply extensions + attach.
Idempotent within a single open/close cycle.
from_registry
classmethod
¶
from_registry(
registry: LocalRegistry,
*,
attach: dict[str, str | tuple[str, str]] | None = None,
extensions: list[str] | None = None,
azure_credentials: dict[str, str] | None = None,
) -> Self
Wrap a LocalRegistry's open connection.
The data store does NOT take ownership of the connection — closing
the data store leaves the registry's connection open. The registry
must already be open (i.e. inside its with block).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
registry
|
LocalRegistry
|
An open |
required |
attach
|
dict[str, str | tuple[str, str]] | None
|
|
None
|
extensions
|
list[str] | None
|
DuckDB extensions to INSTALL+LOAD before any ATTACH. |
None
|
azure_credentials
|
dict[str, str] | None
|
Optional Azure secret config. See
|
None
|
inspect ¶
Return this store's SchemaInspector — the custom-SQL seam (#163).
The inspector reflects the live DuckDB session (cached DESCRIBE)
and turns a DataflowBinding into a QueryPlan, so consumers who
build their own SQL against connection reuse the store's table-name
convention and sidecar join-key derivation. The returned inspector is
the same instance the internal query path uses, so its column
cache is shared.
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If the data store hasn't been opened. |
obs ¶
A lazy query over this dataflow's observations (#297).
The same path language as LocalRegistry.query, rooted at a dataflow
instead of at a structural relation — so a filter is a predicate over
named components rather than a SQL string:
store.obs(binding).filter(
obs["REF_AREA"] == "CA", obs["TIME_PERIOD"] >= "2020"
).collect()
where obs is Observations(binding). Component ids come from the
binding's column maps, and an unknown one raises with the valid ones
listed — the check a downstream SDMX REST service hand-writes as a 400.
where= / columns= / order_by= on query remain, as the escape
hatch for what this cannot express (value formatting, PIVOT). They are
SQL text; this is not.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
binding
|
DataflowBinding
|
The dataflow's binding. |
required |
dsd
|
DataStructure | None
|
The dataflow's structure, for labelling. A store cannot
resolve one — that needs a structural registry — so this is
filled in by |
None
|
Returns:
| Type | Description |
|---|---|
ObsQuery
|
A |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the binding has a series-attributes sidecar, which this root does not join yet. |
KeyError
|
If the binding's |
ingest_csv ¶
ingest_csv(
*,
schema: str,
table: str,
path: Path,
types: Mapping[str, str],
lake_alias: str = "lake",
) -> None
Load a CSV file straight into a table, without materialising it here.
DuckDB reads the file itself, so the rows never pass through a Python object. Measured against the two alternatives on 2M rows — reading the bytes into polars, and scanning to Arrow IPC and back — this is both the fastest and the lightest, and by enough that there is no axis to trade off:
bytes -> polars -> duckdb 0.30 s 660 MB
file -> ipc -> duckdb 0.27 s 608 MB
duckdb reads the file 0.21 s 307 MB
types is a partial override, and that is deliberate. Only the
columns a caller names are forced; the rest are inferred. An SDMX-CSV
envelope differs between 1.0 (DATAFLOW, KEY) and 2.0 (STRUCTURE,
STRUCTURE_ID, ACTION), so a complete map would have to know which
one arrived — while the components are known from the structure
either way. Forcing those is what matters: a code column of 007 or
0100 inferred as an integer is silent corruption.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
schema
|
str
|
Target schema, created if absent. |
required |
table
|
str
|
Target table, replaced if it exists. |
required |
path
|
Path
|
The CSV file to read. |
required |
types
|
Mapping[str, str]
|
Column name to DuckDB type, for the columns worth forcing. |
required |
lake_alias
|
str
|
Catalog the table lives in. |
'lake'
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If any identifier is not a plain SQL identifier. |
write_observations ¶
write_observations(
*,
schema: str,
table: str,
source: WriteSource,
partition_by: tuple[str, ...] = (),
sort_by: tuple[str, ...] = (),
mode: Literal["overwrite", "append"] = "overwrite",
lake_alias: str = "lake",
) -> None
Materialise observation rows into a DuckLake-managed table.
Routes through the ATTACHed lake (lake_alias, default lake).
Schema is created if missing. mode="overwrite" uses
CREATE OR REPLACE TABLE; mode="append" uses INSERT INTO
and requires the target table to already exist with a compatible
schema. partition_by is currently informational — DuckLake
infers partition columns from table definition; we record the
layout on the matching DataflowBinding.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
schema
|
str
|
Target schema inside the lake (created if absent). |
required |
table
|
str
|
Target table name. Together with |
required |
source
|
WriteSource
|
Rows to write. Anything DuckDB can register as a virtual relation — Polars frames, Arrow tables, or an existing DuckDB relation. |
required |
partition_by
|
tuple[str, ...]
|
SDMX dimension ids the table is partitioned on. |
()
|
sort_by
|
tuple[str, ...]
|
Columns of Opt-in, and the reason is the cost. Measured on 4M rows of
three columns, sorting on the way in cost 82 ms against 33 ms
and 342 MB of peak RSS against 132 MB — roughly 2.5x both. It
buys a query-side Physical column names, not SDMX ids: this method holds a source
and a target, never a binding, so an id would have nothing to
resolve against. Record the SDMX ids on the binding's
|
()
|
mode
|
Literal['overwrite', 'append']
|
|
'overwrite'
|
lake_alias
|
str
|
ATTACH alias of the target lake. Defaults to
|
'lake'
|
DuckLakeVersionError ¶
Bases: Exception
A DuckLake catalog's format is newer than the installed extension.
Raised when ATTACH-ing a bundle whose DuckLake catalog format exceeds
what the loaded ducklake extension supports. The fix is to align
versions (upgrade duckdb >= 1.5.3 and/or the ducklake
extension), not to migrate the bundle in place. See issue #156.
SchemaInspector ¶
Cached schema reflection + binding→query planning for a DuckDB session.
Bound to a single DuckDB connection. DESCRIBE results are memoized per
table for the inspector's lifetime — the schema of a read table is stable,
and re-running DESCRIBE on every query is pure redundant work. The
memoisation is single-flight (a build lock, double-checked): concurrent
callers for the same table run DESCRIBE once, not once per thread —
the store's read path is concurrent (per-thread cursors, #75/#80). When a
table is rewritten (e.g. DuckDBDataStore.write_observations), call
invalidate so a stale column list isn't served.
Obtain one via store.inspect() so the internal query path and your
custom SQL share the same cache; constructing one directly is supported but
gets a private cache.
columns ¶
Column names for table via DuckDB DESCRIBE (cached).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
table
|
str
|
A table reference DuckDB can |
required |
column_types ¶
{column: DuckDB type} for table, from the same cached DESCRIBE as columns.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
table
|
str
|
As |
required |
shared_columns ¶
Columns present in both tables, in table_a order.
These are the join keys for a main⋈sidecar LEFT JOIN.
key_counts ¶
(rows, distinct key combinations) of table over keys (cached).
Rows with a NULL in any key are left out of both counts: USING never
matches NULL, so repeated NULL-keyed rows cannot multiply a join, and
refusing a table for them would be refusing a harmless one.
A scan of the table — ~5 ms on the 2021 Census Profile bundle's 619-row geography
sidecar, ~37 ms cold — so memoised single-flight beside columns, and
dropped by invalidate with it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
table
|
str
|
The table to count, as |
required |
keys
|
tuple[str, ...]
|
The columns whose combinations are counted. |
required |
invalidate ¶
Drop everything cached about table (e.g. after a rewrite).
plan ¶
Resolve binding's main/sidecar tables into a QueryPlan.
Mirrors what DuckDBDataStore.query builds for the FROM side:
<database_role>.<table_location> for the main table, and — when
binding.series_attributes_location is set — a LEFT JOIN onto the
sidecar keyed on their shared columns.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
binding
|
DataflowBinding
|
The dataflow binding to resolve. |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the binding names a sidecar that shares no columns with the main table (no derivable join keys). |
SidecarNotUniqueError
|
If the sidecar holds more than one row for some combination of those keys, which would multiply every observation the join matches (#412). |
QueryPlan ¶
The resolved FROM side of a DataflowBinding's query.
Produced by SchemaInspector.plan. Consumers build their custom SQL by
splicing a projection and WHERE onto from_clause:
f"SELECT {cols} FROM {plan.from_clause} WHERE {predicate}"
Attributes:
| Name | Type | Description |
|---|---|---|
main |
str
|
Fully-qualified main table ( |
sidecar |
str | None
|
Fully-qualified series-attributes table, or |
join_keys |
tuple[str, ...]
|
Columns common to main and sidecar (in main order) that the
|
from_clause
property
¶
The FROM fragment: main alone, or main LEFT JOIN sidecar USING (...).
versions ¶
Rewriting versions SDMX 3.x cannot hold, at ingest (#420, #424).
1.03 and 1.2.3.4 are valid SDMX 2.1 and have no SDMX 3.x form, so a
registry refuses them. add_all takes a rewrite to ingest them anyway:
reg.add_all(msg.all_artefacts(), rewrite_version=sl.versions.strip_leading_zeros)
reg.add_all(msg.all_artefacts(), rewrite_version={"1.2.3.4": "1.2.4"})
It returns one RewrittenVersion per artefact whose version changed. See
sdmxlib.model.version_rewrite for the rules.
RewrittenVersion ¶
One artefact whose version was rewritten at ingest.
Attributes:
| Name | Type | Description |
|---|---|---|
urn |
str
|
The artefact's URN as ingested, at the new version. |
original |
str
|
The version as published. |
version |
str
|
The version as ingested. |
strip_leading_zeros ¶
1.03 → 1.3: the one 2.1-only spelling with an exact 3.x equivalent.
SDMX-ML 2.1's VersionType documents that "a version of 1.3 and 1.03 would
be equivalent, as both parse to an integer value of 3", so this changes the
spelling and not the version. Anything else comes back unchanged —
including 1.2.3.4, which has no 3.x equivalent and stays refused.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
version
|
str
|
A version as published. |
required |
Returns:
| Type | Description |
|---|---|
str
|
The version with each numeric part's leading zeros removed. |
RewrittenVersion ¶
One artefact whose version was rewritten at ingest.
Attributes:
| Name | Type | Description |
|---|---|---|
urn |
str
|
The artefact's URN as ingested, at the new version. |
original |
str
|
The version as published. |
version |
str
|
The version as ingested. |
Catalog¶
One row of a LocalRegistry.browse() result — a bound dataflow with its binding and provenance.
DataflowEntry ¶
One row of a browse result.
Attributes:
| Name | Type | Description |
|---|---|---|
dataflow_urn |
SdmxUrn[Dataflow]
|
URN of the bound dataflow. |
binding |
DataflowBinding
|
The |
name |
InternationalString | None
|
The dataflow's |
source_registry |
str | None
|
Provenance label from the binding —
|
Structural queries¶
The path language. sl.q is the accessor — sl.q.code, sl.q.dsd,
sl.q.hcode, one lowercase singleton per relation — and reg.query(sl.q.code)
is the entry point. sl.schema names the physical tables and columns for raw
SQL, reflected from the DDL. See the
Structures as Frames and
Catalogue Discovery cookbook pages.
Query ¶
A lazy query over the relation root names.
A is what the object-building terminals return — collect(),
first(), one(), one_or_none() and Page.collect(). It comes off the
root accessor (see Cols) and defaults to object, which is what all
five returned for every root before #396.
A is invariant, so it has to be carried rather than dropped: the
set-op methods take Query[R, A], and Select / Grouped declare the
parameter even though they never read it, because they hold a query and a
Query[R, A] does not satisfy a Query[R]. That is the cost of the
parameter and it is worth naming — the alternative was 21 overloads on
each terminal, kept in step with _RESOLVERS by hand.
The root is stored as Cols[R], without A. The compiler never needs it:
A exists for the caller's benefit, and the runtime correspondence
between a root and what it builds lives in _RESOLVERS, guarded by
tests/unit/test_query_collect_types.py.
Construct through sdmxlib.query.registry.query, not directly: the
connection and resolver are wiring rather than part of the question.
order_keys
property
¶
The caller's sort keys, empty when nobody has sorted.
default_order
property
¶
The relation's natural order, when nobody has sorted.
Set by the collection a query came from rather than by a caller —
codelist.codes is in declaration order, matching the eager
ItemList it stands in for.
Separate from order_keys because the two are different claims. A
caller's order_by is a decision the refinement laws govern: it blocks
group_by, and #294 refuses to let a later call append to it. A
default is neither — it is what the rows mean when nobody has said, so
codelist.codes.group_by(...) must work, and the first order_by
replaces it outright rather than sorting within it.
unordered
property
¶
Whether the caller has declined a total order — order_by(None).
A third state, not the absence of one. order_keys empty still emits
the root-key tiebreak, so unsorted and unordered are different
claims and Django separates them too (QuerySet.ordered).
source
property
¶
The database this query reads — sdmxlib.sql.HasSource.
The registry's base connection, not a per-thread cursor, so two queries
from one registry report the same token from any thread. Compared with
is; source_label is what a message shows.
source_label
property
¶
How to name this query's database in an error.
A DuckDBPyConnection has no useful repr, so a refusal built from
the token alone reads "a different database (DuckDBPyConnection)"
twice — naming two sources and distinguishing neither, which is
exactly what §5.9 asks the message not to do.
columns
property
¶
Column names this query produces — the SqlRelation contract.
Every column of the root relation, because to_sql() with no argument
selects them all. So a bare Query is a multi-column relation and
is_in refuses it by name (#274) rather than letting the database
emit a column-count error. select(one_column) is the single-column
form, which is what splices into WHERE col IN (SELECT ...).
Read with DESCRIBE, which plans without running.
filter ¶
Narrow by AND-ing predicates onto the query.
Several predicates in one call and several calls are the same thing —
filter is AND, the one composition law unanimous across SQLAlchemy,
Django, ibis and polars.
Typed on the query's own row kind, so a predicate built for another
root is rejected here rather than compiling to a WHERE against a
relation absent from the FROM.
Raises:
| Type | Description |
|---|---|
SliceError
|
If the query has already been sliced. |
TypeError
|
If handed something that is not a predicate — a bare
column or an accessor. The row-kind phantoms catch a predicate
built for another root statically; they cannot catch a value
that is not a predicate at all, and untyped call sites are
where that arrives. |
CrossSourceError
|
If a predicate carries a relation from a
different database — see |
qualify ¶
Narrow by AND-ing windowed predicates — SQL's QUALIFY.
A separate clause from filter because a windowed test is not an
ordinary one: COUNT(*) OVER (...) is not valid in a WHERE, and the
two land in different parts of the statement.
Call order does not decide which set the window sees. Every clause
compiles into one SELECT, and SQL evaluates WHERE before the
window, so filter(p).qualify(w) and qualify(w).filter(p) are the
same query: the rank is computed over the filtered rows either way.
They commute, which is what the refinement principle asks of every
pair except limit/offset.
The other reading — rank across everything, then keep the rows
matching p — is a different query rather than a different call
order, and it is spelled by nesting, exactly as filter-after-slice is
(#310)::
ranked = reg.query(q.dataflow).qualify(w).select(urn=q.dataflow.urn)
reg.query(q.dataflow).filter(q.dataflow.urn.is_in(ranked), p)
This docstring previously claimed the two call orders expressed those two readings. They never did: both emitted byte-identical SQL, so a caller reaching for the second got the first with nothing to say so.
Why filter-after-qualify is not refused, unlike filter-after-slice: there the two readings return different rows, so picking one silently is a wrong answer and the guard prevents it. Here they return the same rows, so a refusal would fire on a correct, unambiguous query — breaking the ordinary case where a helper qualifies and its caller narrows — to prevent a misunderstanding rather than a defect.
The distinction is the type, not discipline. .over() returns a
Win, whose comparisons produce WinPred rather than Pred, so
filter(rank == 1) and qualify(id == "X") are both rejected by the
checker with the other clause named.
reg.query(q.dataflow).qualify(
q.dataflow.version.rank(descending=True)
.over(q.dataflow.agency, q.dataflow.id) == 1
)
Raises:
| Type | Description |
|---|---|
SliceError
|
If the query has already been sliced. |
TypeError
|
If handed something that is not a windowed comparison. The checker refuses it statically; an untyped call site reached the compiler, which said only "cannot compile Compare" (#511). |
order_by ¶
Set the sort order, replacing any already set. No arguments clears.
Replaces rather than appends, which is the one place this DSL
deliberately breaks the refinement principle, and the existing DSLs
already do the same (#294). Appending cannot be undone, so a helper
that sorted would silently outrank its caller; and "sort by A, then by
B" is one call with two keys, not two calls. Direction is per key —
q.dsd.version.desc() — because "newest first, then id ascending" is
the common shape.
Ordering by a path that crosses a to-many edge is refused: there is no single value per row to sort on.
order_by(None) declines ordering outright, which is SQLAlchemy's
spelling for it and a third state rather than the absence of one:
q.order_by(q.code.id) # sort by id, then the tiebreak
q.order_by() # clear the keys, keep the tiebreak
q.order_by(None) # no ORDER BY at all
The middle one drops the caller's keys and leaves the root-key
tiebreak, so an unsorted result is still stable between calls rather
than merely unspecified — paging depends on that, and #264 is what
happens without it. On a small relation the tiebreak is free. On 4.26 M
observations it was 5.4x the scan it decorates, for an order most
callers of frame() never look at (#389), and nothing could decline it
but stream().
So this is the caller saying they do not need one. It closes the query
to limit, offset and after, since a window over an unspecified
scan order silently skips and repeats rows between calls — the failure
the tiebreak exists to prevent, which is why declining it and paging
cannot both be allowed. A later order_by(...) reopens it.
Raises:
| Type | Description |
|---|---|
SliceError
|
If the query has already been sliced. A different order
over the same window is a different set of rows, so this is the
same violation |
TypeError
|
If a key is not a column, or if |
intersect ¶
Narrow to rows also matched by other — a semi-join.
A method because it narrows: the result is always a subset of this
query, so nothing other carries can undo what this one established.
union is a free function for exactly the opposite reason.
Typed on the row kind, so intersecting dataflows with codes is a type error rather than an empty result.
Raises:
| Type | Description |
|---|---|
SliceError
|
If this query has already been sliced. |
TypeError
|
If |
difference ¶
Narrow to rows not matched by other — an anti-join.
Narrows for the same reason intersect does, so it is likewise a
method.
Raises:
| Type | Description |
|---|---|
SliceError
|
If this query has already been sliced. |
TypeError
|
If |
after ¶
Narrow to rows after cursor — the keyset half of paging.
None returns the query unchanged, so the first page and every later
one are the same expression:
page = reg.query(q.dataflow).order_by(q.dataflow.id).after(cursor).page(100)
Why not offset. offset(n) makes the database walk and discard
n rows, so page 500 costs 500 pages of work; and a row inserted
before the window between calls shifts every later page, silently
repeating one row and skipping another. A keyset comparison does
neither.
The design left open whether this should refuse without an
order_by. It does not need to: every query here already carries the
root's key as a tiebreak, so the order is always total and a cursor
is always well defined. An unsorted query pages by URN, which is
arbitrary but stable — and stable is the whole requirement.
The cursor knows how many keys it was built from and in which directions, and is refused against a query that has since been sorted differently.
Raises:
| Type | Description |
|---|---|
SliceError
|
If the query has already been sliced. |
limit ¶
Cap the number of rows, never raising an existing cap.
Raises:
| Type | Description |
|---|---|
SliceError
|
If the query is unordered — see |
offset ¶
Raise the window's floor, holding its ceiling fixed.
Raises:
| Type | Description |
|---|---|
SliceError
|
If the query is unordered — see |
get ¶
The row with item_id, or None — ItemList.get over a relation.
The lazy and eager forms of a collection have to answer the same
questions or codelist.codes means two different things depending on
where the codelist came from, which is the whole premise of the lazy
artefacts.
__getitem__ ¶
Slicing refines; indexing does not. A string looks up by id.
The string form is ItemList.__getitem__ — codes["FREQ"] — and is
the reason indexing is not simply refused outright. It is unambiguous
against the other two because an id is never a position.
to_sql ¶
Compile without executing.
columns defaults to every column of the row — the root relation's,
plus any root join's, which is what makes a sidecar attribute appear in
an unprojected result. A terminal that needs one column passes it
rather than trimming afterwards.
ordered=False drops the ORDER BY entirely, for the two terminals
that reduce to a single row. count(*) sorts nothing, and a sort key
naming a column outside the aggregate is a binder error rather than a
wasted clause.
order_by(None) drops it too, for every terminal — this is where that
state reaches the compiler. The parameter and the flag are and-ed
rather than one overriding the other: both are reasons not to sort, and
neither is a reason to.
explain ¶
A human-readable plan. Runs nothing.
Says when rows are fetched first, because the SQL alone does not.
A query served over HTTP compiles to an ordinary SELECT over the
store it will be materialised into — so without this line the plan
reads as a local table scan, and the request that fills that table is
invisible. The distinction is worth a line: it is the difference
between reading rows that are already here and downloading them.
canonical_key ¶
A stable, hashable key identifying this query's meaning (#270).
Two queries built by different call sequences that mean the same thing
return the same key — filter(a).filter(b) and filter(b).filter(a)
differ as values and emit their clauses in different orders, and agree
here. That is what a downstream cache needs and what neither the SQL
text nor the query object offers.
Deterministic across processes: no id(), no hash(), no set
iteration. Built by the same canonical_key the four older DSLs use,
which walks attrs fields structurally — so the connection and the
resolver are excluded by their own eq=False, and a field added to
Query is in the key without anything here being updated.
The expression leaves are not attrs, and describe themselves through
__canonical__. That is load-bearing rather than tidy: the fallback
keys an object on its repr, and AnyCol.__repr__ omits the language,
so name["en"] and name["fr"] keyed identically until #306 — a cache
serving one language's rows for another.
Returns:
| Type | Description |
|---|---|
object
|
A tuple of tuples of primitives. Opaque; compare and hash it, do |
...
|
not parse it. |
count ¶
Number of matching rows — COUNT(*), fetching none of them.
Wrapped in SELECT count(*) FROM (...) when the query is sliced or
qualified, and only then — a bare count(*) is cheaper and is what
most calls get.
Both cases are wrong without the wrapper, in different ways. Applying a
slice to the aggregate counts every row and then limits the one-row
result, so q.limit(3).count() would return the full count — the
defect #264 found in __len__. A QUALIFY alongside count(*) is not
merely wrong but invalid: the window's partition columns are not in the
aggregate, and DuckDB refuses to bind it.
first ¶
The first matching artefact, or None.
First in this query's order — which after order_by(None) is no
order at all, so the row is an arbitrary match rather than a defined
one. That is what the caller asked for and what SQLAlchemy's .first()
does with an unordered statement; order_by(...) is how to mean a
particular row.
one ¶
The single matching artefact.
Raises:
| Type | Description |
|---|---|
ValueError
|
If the query matched anything other than one row. |
one_or_none ¶
The single matching artefact, or None if there were none.
Raises:
| Type | Description |
|---|---|
ValueError
|
If the query matched more than one row. |
collect ¶
Resolve and return the matching artefacts.
The expensive terminal: each match is loaded and its refs resolved. Use
urns, ids or frame when identifiers or columns are enough — that
is the 331x difference the milestone opens with.
annotations=False skips the annotation read and leaves every
.annotations empty. On a 579,951-code codelist that join and its
grouping are 32% of the collect — more than building the Code
objects — for annotations most callers never look at (#381).
It is off by default and should stay off by default: a silently empty
.annotations is wrong data, not fast data, and a caller who does not
know to ask must not get it. Passing it is the caller saying they know.
Supported on q.code and q.hcode, where annotations are a separate
query. Other roots build their objects through readers that always read
them, and raise rather than accept the flag and ignore it.
Raises:
| Type | Description |
|---|---|
TypeError
|
If nothing can build objects from this root's rows. |
ValueError
|
If |
page ¶
The next n rows, and a cursor for the ones after them.
The paging terminal. It selects the sort keys alongside the URNs
because a cursor is made of them — which is why this is its own
terminal rather than a flag on urns().
Page.cursor is None exactly when the page was short, meaning there
is nothing after it. A full page always yields a cursor, even when it
happened to be the last one; the alternative is fetching n + 1 rows
to find out, and a caller who cares learns it from the next empty page.
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
SliceError
|
If the query is unordered. A page is a window, and a
cursor is made of the sort keys — see |
frame ¶
Every column of the root relation, as a polars.DataFrame.
labels works as it does on a projection — see Select.collect. An
observation frame's columns are named by component id and typed from
the DSD (_typed): codes as their codelist's pl.Enum, values as
their Facet declares.
stream ¶
The same rows as frame, in Arrow batches instead of one frame.
For a result too large to hold. Measured, not assumed, on 4M rows of three columns:
collect() 441 MB peak RSS
stream(chunk_rows=1_000) 5 MB
stream(chunk_rows=50_000) 5 MB
stream(chunk_rows=500_000) ~8 MB
and the streamed figure does not move at 8M rows, where collect()
roughly doubles. That is what "bounded" means here, and it is the
reason to reach for this at all.
Measured against rows that are already in the store, which is the local case and is not every case. A query served over HTTP fetches first, so the figures above describe the stream and not the fetch:
| where the rows are | bounded? |
|---|---|
| a bundle, or any file-backed store | yes — the figures above |
fetched, into a file-backed store= |
yes, after a full download |
fetched, into a RestRegistry's own in-memory store |
no — materialised in RAM first |
The third row is the default when no store= is given, and explain()
is what tells the two apart: it says whether a query fetches. Streaming
a large remote result wants a store= on disk.
chunk_rows is not the knob it looks like. What decides bounded
from unbounded is whether the query sorts, because a sort has to
materialise the whole result before the first batch can leave. Every
query carries a key tiebreak by default — see _is_ordered — and while
a stream still did, this cost 210 MB at 4M rows and 368 MB at 8M
whatever chunk_rows said: 1,000 and 50,000 measured identically.
Chunk size only starts to cost anything once it is large enough to be a
buffer in its own right.
So size chunk_rows for the consumer — a writer's preferred batch,
an HTTP chunk — not for memory. 10k to 50k is a reasonable default and
anything under a few hundred thousand costs nothing measurable.
Both spellings work, and that is the reader's doing rather than a
choice made here. pa.RecordBatchReader is an iterator and a
context manager, so:
for batch in query.stream(): # simple
...
with query.stream() as batches: # deterministic close
for batch in batches:
...
The second is worth preferring for a stream abandoned part-way — a
break, an exception, an HTTP client that hangs up. Without it the
cursor is released whenever the reader is collected, which is soon but
not defined.
Arrow batches rather than polars frames because that is what a writer
wants — CSV, parquet, an HTTP body — and pl.from_arrow(batch) is a
cheap zero-copy step for a caller who wants the other thing (the
pl.DataFrame(batch) constructor is ~2x slower, #427). Converting
here would force it on everyone.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
chunk_rows
|
int
|
Target rows per batch. The last batch is short. |
10000
|
Returns:
| Type | Description |
|---|---|
RecordBatchReader
|
A reader over the result. Nothing has run when it is returned; |
RecordBatchReader
|
the first pull executes. |
preview ¶
A bounded head, for looking at a query in a REPL.
_window rather than limit, so this still works on an unordered
query: it reads once and shows what it got, which is what a REPL look
is, rather than a page anything continues from.
select ¶
Project columns and aggregates, returning a Select.
Positional columns take their own name; keyword arguments name theirs.
A path crossing a to-many edge has no single value per row, so it must
be collected — .list(), .count(), .n_unique() — and is refused
otherwise with the alternatives named.
Each aggregate compiles to a correlated subquery rather than a join, so
several in one projection cannot multiply rows the way several joins
would. They do each cost a pass, which is why list() takes several
columns and returns structs instead of asking for one call per field.
group_by ¶
Group rows by keys; call agg or len on the result.
Not a refinement, and returns a different type for that reason.
select changes what a row is; group_by changes it further, into one
row per distinct key. Neither is a narrowing, so neither belongs in the
chain the refinement laws govern — which is the same reasoning that
keeps union a free function.
Two stages rather than one, because the keys and the aggregates answer
different questions and the aggregates are only meaningful once the
keys are fixed. polars splits it the same way.
reg.query(q.dataflow).group_by(q.dataflow.agency).agg(
n=q.dataflow.id.count(),
).collect()
Raises:
| Type | Description |
|---|---|
TypeError
|
If no keys are given. |
SliceError
|
If the query is ordered or sliced. Neither survives grouping — a row sort key that is not a group key is not valid SQL, and a limit applied before grouping would silently aggregate a truncated set. Group first, then order and slice the grouped result, which can also order by an aggregate. |
agg ¶
Several aggregates over every row of the query, in one scan (#414).
One row out, one column per aggregate — group_by(...).agg(...) with
no keys, which is why it is not spelled that way: group_by() refuses
zero keys, and a row count alone is count(). The case this serves is
several answers from one pass, e.g. the distinct values each
dimension takes under a filter, plus the row count — what SDMX-REST's
/availability returns:
obs.agg(
SEX=obs.col("SEX").unique(),
TIME_PERIOD=obs.col("TIME_PERIOD").unique(),
n=sl.len(),
).collect()
Not always faster than one select(col).unique() per column. On
the 2021 Census Profile bundle (519M rows, six dimensions) one scan lost 2.8x unfiltered
and won 1.1-4.8x under a filter, so it is an option to take when a
filter narrows the scan, not a replacement.
An Agg here is a plain SQL aggregate over the query's rows — the same
expression in select() is a correlated subquery over a fan-out.
Raises:
| Type | Description |
|---|---|
TypeError
|
If no aggregates are given, or one is a plain column —
with no group keys there is no single value a column could
take. Project it with |
SliceError
|
If the query is ordered or sliced. An order means nothing once every row folds into one, and a limit applied first would silently aggregate a truncated set. |
Select ¶
A projected query. collect is the only executing call.
Deliberately not filterable. select is not a row refinement — it
changes what a row is — so a predicate added afterwards would be about
the projection rather than about the relation, and the refinement laws say
nothing about that. Filter first, then project. query-composition.md
records this as one of the two rows that test whether the principle is
real: both errors predate it and both fall out of it.
columns
property
¶
The projected names, in order — the SqlRelation contract.
Known without asking the database: a projection names its own columns.
source
property
¶
The database this frame reads — sdmxlib.sql.HasSource.
On Select rather than only on Query because a frame is what
is_in is handed: select() is what turns a query into a
single-column relation, so this is the object whose source a consumer
needs (§5.9).
order_by ¶
Sort the projection, replacing any order already set.
Keys must be projected. The top-N shape is what this is for:
reg.query(q.dataflow).group_by(q.dataflow.agency).agg(
n=q.dataflow.id.count(),
).order_by(...).limit(3)
Raises:
| Type | Description |
|---|---|
TypeError
|
If a key is not one of the projected items. |
unique ¶
Collapse duplicate rows — SELECT DISTINCT.
Full-row only, and takes no arguments. A subset form fails twice
over: filter(keep).unique(agency) and unique(agency).filter(keep)
return different rows, so it would not commute with filter, and it
leaves which row of each group survives unspecified. Excluding it
keeps unique in the commuting set rather than adding a carve-out to
the third composition law. A caller wanting one row per group has
group_by(...).agg(...), which says which one — and, since #306, that
is a promise the code keeps rather than a forward reference.
It lives on the projection rather than on Query because the rows of a
keyed relation are already distinct — unique() before select()
would always be a no-op.
having ¶
Keep only the groups satisfying predicates — SQL's HAVING.
The clause filter cannot be. filter asks about a row and
group_by has replaced rows with groups, so a row predicate is no
longer meaningful — which is what filter refuses and should keep
refusing. A predicate about the group is a different question and had
no spelling at all:
reg.query(q.dataflow).group_by(q.dataflow.agency).agg(
n=q.dataflow.id.count(),
).having(q.dataflow.id.count() > 10).collect()
Note the aggregate is written out again rather than referred to by its
output name. That is deliberate: n is a label on the result, while
HAVING runs before the result exists, and SQL dialects disagree
about whether the alias is visible there. Writing the aggregate says
what is computed without depending on that.
Only meaningful after group_by. An ungrouped projection has one
row per root row, and an aggregate over it is a correlated subquery the
root's own filter already accepts — `filter(q.codelist.codes.count()
3)
— so there is nothinghaving` would add and a second spelling would only be a way to get it wrong.
Terms accumulate and are ANDed, as filter does on Query, so
.having(a).having(b) and .having(a, b) are the same query.
filter ¶
Always raises. Filter before projecting.
having() is the grouped counterpart — see there for why the two are
different questions rather than two spellings of one.
to_sql ¶
Compile without executing.
ordered=False drops the ORDER BY, for stream — see there for why
an unordered stream is the difference between bounded and unbounded
memory.
explain ¶
A human-readable plan. Runs nothing.
Says when rows are fetched first, because the SQL alone does not.
A query served over HTTP compiles to an ordinary SELECT over the
store it will be materialised into — so without this line the plan
reads as a local table scan, and the request that fills that table is
invisible. The distinction is worth a line: it is the difference
between reading rows that are already here and downloading them.
collect ¶
Execute and return a polars.DataFrame. The only executing call.
labels attaches a human-readable column per enumerated component and
language — REF_AREA_label_en. The language is always in the name,
including for one, so adding a second later does not rename the first
and break whatever read it. label_suffix renames the middle part:
label_suffix="_name" gives REF_AREA_name_en.
A column projected under an alias is labelled under that alias — the name in the frame is the name a caller reads.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
labels
|
str | Sequence[str]
|
One or more BCP-47 tags. Empty attaches nothing. |
()
|
label_suffix
|
str
|
Between the column and the language. |
'_label'
|
Raises:
| Type | Description |
|---|---|
TypeError
|
If labels are asked of a query with no DSD — one built from a store rather than a catalog. |
ValueError
|
If no projected column is enumerated, so labelling
would add nothing. See |
stream ¶
The same rows as collect, in Arrow batches instead of one frame.
For a result too large to hold. Measured, not assumed, on 4M rows of three columns:
collect() 441 MB peak RSS
stream(chunk_rows=1_000) 5 MB
stream(chunk_rows=50_000) 5 MB
stream(chunk_rows=500_000) ~8 MB
and the streamed figure does not move at 8M rows, where collect()
roughly doubles. That is what "bounded" means here, and it is the
reason to reach for this at all.
Measured against rows that are already in the store, which is the local case and is not every case. A query served over HTTP fetches first, so the figures above describe the stream and not the fetch:
| where the rows are | bounded? |
|---|---|
| a bundle, or any file-backed store | yes — the figures above |
fetched, into a file-backed store= |
yes, after a full download |
fetched, into a RestRegistry's own in-memory store |
no — materialised in RAM first |
The third row is the default when no store= is given, and explain()
is what tells the two apart: it says whether a query fetches. Streaming
a large remote result wants a store= on disk.
chunk_rows is not the knob it looks like. What decides bounded
from unbounded is whether the query sorts, because a sort has to
materialise the whole result before the first batch can leave. Every
query carries a key tiebreak by default — see _is_ordered — and while
a stream still did, this cost 210 MB at 4M rows and 368 MB at 8M
whatever chunk_rows said: 1,000 and 50,000 measured identically.
Chunk size only starts to cost anything once it is large enough to be a
buffer in its own right.
So size chunk_rows for the consumer — a writer's preferred batch,
an HTTP chunk — not for memory. 10k to 50k is a reasonable default and
anything under a few hundred thousand costs nothing measurable.
Both spellings work, and that is the reader's doing rather than a
choice made here. pa.RecordBatchReader is an iterator and a
context manager, so:
for batch in query.stream(): # simple
...
with query.stream() as batches: # deterministic close
for batch in batches:
...
The second is worth preferring for a stream abandoned part-way — a
break, an exception, an HTTP client that hangs up. Without it the
cursor is released whenever the reader is collected, which is soon but
not defined.
Arrow batches rather than polars frames because that is what a writer
wants — CSV, parquet, an HTTP body — and pl.from_arrow(batch) is a
cheap zero-copy step for a caller who wants the other thing (the
pl.DataFrame(batch) constructor is ~2x slower, #427). Converting
here would force it on everyone.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
chunk_rows
|
int
|
Target rows per batch. The last batch is short. |
10000
|
Returns:
| Type | Description |
|---|---|
RecordBatchReader
|
A reader over the result. Nothing has run when it is returned; |
RecordBatchReader
|
the first pull executes. |
Grouped ¶
A query with its group keys fixed, waiting for aggregates.
Deliberately not runnable on its own: grouping without aggregating is
select(...).unique(), which already exists and says so. Every way out of
here produces a Select.
agg ¶
Aggregate each group, yielding the keys followed by the aggregates.
Keyword-only, because an aggregate has no name of its own to fall back
on the way a column does — count() over two different columns would
collide, and positionally-named output columns are how a projection
silently reorders under a caller.
An Agg here is a plain SQL aggregate over the group's rows, where the
same expression in select() is a correlated subquery over a fan-out.
The expression says what to compute and the clause says over what.
Raises:
| Type | Description |
|---|---|
TypeError
|
If no aggregates are given. |
len ¶
Count rows per group, in a column named len — as polars names it.
A method rather than a bare count() aggregate because it counts
rows, not values of any column, so there is nothing to name it after.
Page ¶
One page of results, and where the next one starts.
Not a query — it has already run. So it extracts rather than refines, which
is why it carries the dunders Query refuses: there is nothing left to
push down.
collect ¶
Resolve this page's rows to domain objects.
Each URN is handed over as a one-tuple, which is exactly right for
every root page() accepts: it needs a urn column, and on all of
those the key is that column. Pinned by a test, since the day a root
has a URN and a wider key this becomes silently wrong.
Raises:
| Type | Description |
|---|---|
TypeError
|
If nothing can build objects from this root's rows. |
union ¶
Every row matched by any of the given queries.
A free function, never a method, and that is the refinement principle
rather than style. Union widens: helper_scoped_to_estat.union(all_of_it)
would undo the helper's scoping, which is precisely what the principle
exists to prevent. As a function it composes its arguments into a third
query instead of refining either — the same reasoning that keeps group_by
off the refinement surface.
The result is a fresh query over the shared relation, carrying one membership predicate and none of the arguments' own clauses. Their predicates, orders and slices are inside the compiled subqueries, so each contributes exactly the rows it matched — including rows the others excluded, which a method could not express.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
first
|
Query[R, A]
|
The query whose root the result takes. |
required |
rest
|
Query[R, A]
|
Further queries over the same relation. Calling with none returns
|
()
|
Raises:
| Type | Description |
|---|---|
TypeError
|
If any query is rooted at a different relation. |
CrossSourceError
|
If any query reads a different database. |
len ¶
The number of rows, as an aggregate — count(*), named for polars' pl.len().
For Query.agg, which folds a whole query into one row, so the row count
can come out of the same scan as the other aggregates (#414):
obs.agg(SEX=obs.col("SEX").unique(), n=sl.len())
A function rather than a column method because it counts rows, not the
values of any column — the same reason group_by(...).len() is a method on
the grouping rather than on a column. The row kind is taken from where it is
used, so it fits any query's agg.
Shadows the builtin only as sl.len; this module never calls len()
itself, which is why it is defined here rather than re-exported from a
module that would need a second suppression.
CompiledQuery ¶
Bases: NamedTuple
A runnable statement and its binds, in placeholder order.
A NamedTuple so it is both compiled.sql and sql, params = compiled
— which is what lets Query satisfy sdmxlib.sql.SqlRelation structurally
rather than through an adapter. That protocol is how a lazy single-column
relation splices into WHERE col IN (SELECT ...), and it is implemented by
other libraries too, so conforming means a new-language query goes wherever
a CodeQuery already goes.
params is a list rather than a tuple to match the protocol exactly. A
tuple[object, ...] is not assignable to list[object], so the conformance
would have been structural in spirit and rejected by the checker.
RawSql ¶
A prepared raw statement. A terminal, and deliberately only a terminal.
Lazy like everything else here — sql() compiles and validates, and
nothing runs until collect().
Not a sdmxlib.sql.SqlRelation, which it was until the boundary was
drawn. It carried to_sql() for no reason but that protocol, so removing
the method removes the capability everywhere at once rather than at each
call site: is_in, the old DSLs, and rest.py all probe structurally.
The two are different contracts, and mixing them costs more than it reads:
- The DSL's safety property is that a caller supplies field references
and literals and nothing else. Splicing arbitrary SQL back in returns
what
query-composition.mdbans, via an object instead of a string. - A raw statement has no row kind, so nothing checks that the spliced relation yields urns, or codes, or anything related to the query it lands in. The static story the row-kind phantoms tell is bypassed silently — the failure is an empty result, not an error.
- It cannot be pushed down. §4 compiles this DSL to REST backends and a federation planner. An opaque DuckDB statement is not analysable, so one splice pins the whole query to DuckDB for good.
- It hides the evidence. This hatch exists to show what to build next — 113 hand-written SQL sites are why the milestone exists. Free splicing makes the gap painless and therefore unreported.
Crossing back is one call, and the point is that it is visible:
urns = reg.sql("SELECT urn FROM sdmx.artefact WHERE ...").collect()["urn"]
reg.query(q.dataflow).filter(q.dataflow.urn.is_in(urns)).collect()
That materialises where the subquery did not, which is the real cost of the rule: a large set becomes N binds. When that hurts, the answer is to make the DSL express the query — which is the hatch working as intended.
Composition between compiled relations is untouched: Query satisfies
SqlRelation, so is_in(another_query) still compiles to one
IN (SELECT ...), as does a producer from another library, for which the
protocol has no alternative.
columns
property
¶
Column names this statement produces, in order.
Read with DESCRIBE, which plans without running — so asking what a
relation yields does not fetch it.
RawSqlError ¶
Bases: ValueError
The escape hatch was handed SQL it will not run.
Distinct from a database error: these are the library's own rules — named parameters only, every one supplied, and no construct the rewriter cannot read.
Query refusals¶
What a query raises when it cannot be written, planned or served — each names the fix.
PathError ¶
Bases: ValueError
A path names a hop the edge graph does not declare.
SliceError ¶
Bases: ValueError
A clause was added after the query was sliced.
Every clause compiles into one SELECT, so a filter added after a
limit lands in the WHERE and runs before it — returning rows the
query being refined did not contain. Both readings are defensible, so the
DSL refuses rather than picking one silently. Django does the same, for the
same reason.
CrossSourceError ¶
Bases: ValueError
A relation from one database was spliced into a query on another (§5.9).
Its own type rather than a bare ValueError because the caller's remedy is
structural — give both sides one database — and a program that federates
may want to catch exactly this while letting other ValueErrors through.
PushdownError ¶
Bases: ValueError
A query cannot be run against this backend without fetching too much.
Raised rather than answered, per §5 and the whole of %11: a valid result or an error naming the problem, never a plausible-looking wrong one. The problem here is invisible in the result — the rows come back correct, just at the cost of having fetched every artefact in the registry to filter three of them locally.
SidecarNotUniqueError ¶
Bases: ValueError
A series-attributes sidecar holds more than one row per join key (#412).
The sidecar is joined as LEFT JOIN … USING (<keys>), so every extra row
per key multiplies each observation it matches. Nothing errors downstream:
count(), collect() and every aggregate over a sidecar column come back
inflated by the fan-out factor, and every row looks valid.
A ValueError because it is the same kind of refusal plan already makes
for a sidecar sharing no columns — the binding names a table that cannot
serve as its series attributes — so a caller catching that catches this.
Schema ¶
sl.schema — the sdmx namespace, resolved against the live DDL.
Expressions and row kinds¶
The typed pieces a path is built from. You rarely name these — sl.q hands them out — but they appear in signatures.
Col ¶
Bases: AnyCol
A column reference, carrying the path walked to reach it.
Not an attrs class: __eq__ is overridden to build a predicate rather than
to compare, which is the one place this API departs from Python's usual
meaning and the reason sl.q. exists as a visible marker.
list ¶
Collect this column across the fan-out its path crosses.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
where
|
AnyPred | None
|
Restrict which children are collected — or, on a column
that does not fan out, under |
None
|
count ¶
Count the children the fan-out reaches — count(*), not of values.
n_unique ¶
Count distinct values of this column across the fan-out.
sum ¶
Total this column across the fan-out.
Numeric columns only. The compiler carries no column types, so a
sum over text is refused by DuckDB — No function matches the given
name and argument types 'sum(VARCHAR)' — rather than here. That is
the honest position while the types are not available; #327 records
it as the open question.
mean ¶
Average this column across the fan-out — SQL's avg.
Named for polars, like every other aggregate here, and compiled to the
SQL spelling the way n_unique becomes count(DISTINCT …).
Nulls are skipped, not counted as zero, so the mean of [2, NULL]
is 2 rather than 1. SQL and polars agree on this, which is worth
stating because #326 records a case where they do not.
min ¶
Smallest value of this column across the fan-out.
Not restricted to numbers: SQL and polars both order text, so
min over a code id is its lexicographic first. That makes min/max
usable where sum/mean are not.
unique ¶
The distinct values of this column across the fan-out.
list keeps duplicates, n_unique counts them away and returns a
number; this is the third corner — which values, each once.
The two uniques in this language mean different things and are not
interchangeable, which is worth reading carefully once:
Projection.unique() deduplicates whole rows (SELECT DISTINCT),
and this deduplicates values inside one fan-out. Both names are
polars', where DataFrame.unique() and Expr.unique() divide exactly
the same way, so importing that split imports a distinction users of
polars already hold.
n_unique() is kept rather than left as unique().count() for the
reason polars keeps it: count(DISTINCT x) is a single pass, and
counting a materialised distinct list is not.
rank ¶
Rank rows by this column within a window — ties share a rank.
The framing is a separate call: .rank() says what is computed and
.over(...) says over which rows. "The latest version of each
artefact" is version.rank(descending=True).over(agency, id) == 1.
dense_rank ¶
Like rank, but without gaps after a tie.
row_number ¶
Number rows by this column within a window — ties broken arbitrarily.
Use rank when tied rows should share a number. row_number picks one
of them, and which one is not defined.
lag ¶
This column's value offset rows earlier in the window.
lead ¶
This column's value offset rows later in the window.
sort_by ¶
Order an aggregate over this column by key.
Ordering is a property of the expression rather than an argument to
the aggregate — polars' shape, where pl.col("x").sort_by("ord")
composes with whatever aggregates it afterwards.
codes.id.sort_by(codes.position).list()
key must cross the same fan-out as the column being collected; a key
from outside it has no value per collected row, and is refused where
the aggregate's own columns are checked.
Declared on Col[R] rather than on AnyCol, where the implementation
would otherwise sit, so the parameter can name the row kind. AnyCol
is the non-generic base the compiler reads and is not part of the
public surface — a signature mentioning it asks a caller to name a type
they cannot import (#299).
Without an order an aggregate returns whatever the scan produced, which
is not an order at all — the hazard __key__ already removes from
ORDER BY, where two rows equal on the caller's keys "come back in
whatever order the scan produced" (#327). The default is unchanged:
an unsorted list() is still scan-ordered, because adding an
ORDER BY to every aggregate costs a sort nobody asked for.
sort ¶
Order an aggregate over this column by the column itself.
codes.id.sort().list() # ['CA', 'MX', 'US', 'W']
The only ordering unique() accepts. A DISTINCT aggregate collapses
the rows any other key would have come from — which duplicate's key
would it use? — so SQL requires the sort key to be the distinct
expression, and DuckDB refuses anything else outright.
desc ¶
This column as a descending sort key.
Descending is the one the existing DSLs already use — direction
marked on the key rather than a descending=[...] list parallel to
the keys, as SQLAlchemy, ibis and Django all do. Shared rather than
restated so .desc() means one thing across every DSL here, which is
the whole point of the migration this milestone ends in.
There is no asc(): an unmarked key is ascending.
is_in ¶
Membership, in a collection of values or in a single-column relation.
The relation form takes anything satisfying sdmxlib.sql.SqlRelation —
a one-column Select (q.select(col)), or a producer from another
library — and compiles to one IN (SELECT ...). reg.sql(...) is
refused: raw text carries no row kind to check. Fetching its rows and passing them as a list
would work and is what a caller does without this, at the cost of N
binds and a round trip.
A bare str is rejected: is_in("CA") silently meaning
is_in(["C", "A"]) shipped once (#261), found only by placing the API
beside polars, which rejects it.
Raises:
| Type | Description |
|---|---|
TypeError
|
On a |
Cols ¶
Bases: AnyCols
A relation accessor, parameterised on the query root's row kind and artefact.
Instances are cheap and immutable; walking an edge builds a new one rather than mutating, so an accessor is safe to share as a module singleton without one query's path leaking into another's.
R is the row kind of the root, so a predicate built for one root cannot
be handed to a query over another. A is what collect() builds from this
relation's rows, and it is pinned by the accessor class rather than left
free — _Code always collects Code, at whatever root the path started
from. Roots that build nothing pin Never; that is 16 of the 37 (#396).
A defaults to object, which is what collect() returned for every root
before #396. That default is what let the map be filled in per accessor
rather than in one edit: an accessor that pins nothing behaves exactly as
it did, and Cols[R] keeps meaning what it meant.
Agg ¶
Bases: AnyAgg
A projected aggregate over the fan-out a column's path crosses.
kind is list, count or n_unique. columns is what is aggregated —
several only for list, which then produces a list of structs.
That multi-column form is not sugar. Two separate list() subqueries over
one child align only incidentally: list() has no defined order without an
ORDER BY, so labels[i] is not reliably the label of codes[i], and it
diverges the moment either side gains an order. A struct makes the
alignment structural rather than promised, and costs one subquery instead
of two. See docs/design/query-language-examples.md §4.
desc ¶
This aggregate as a descending sort key — the top-N shape.
group_by(...).agg(n=...).order_by(n.desc()).limit(1) is why a lazy
projection is worth having: sorting a materialised frame has already
fetched every row. Same Descending marker a column uses, so the
spelling does not change with what is being sorted.
Declared here rather than on AnyAgg — which is where it lived until
299 — so the key names the row kind. AnyAgg is the non-generic base¶
the compiler narrows to and is not exported, so a signature returning
Descending[AnyAgg] asked a caller to name a type they cannot import.
Moving it down rather than overriding it is what Col.desc already
does: Descending is @frozen over a single read-only field, so
covariance would be sound, but attrs declares key: K as a plain
mutable attribute and PEP 695 therefore infers K invariant — which
makes the narrowing an incompatible override. There is no base method
to conflict with once the declaration is here.
over ¶
Recompute this aggregate as a window function.
q.dsd.id.count() counts a fan-out's children; .over(q.dsd.agency)
instead counts the rows of the outer query sharing an agency. Same
SQL function, a different thing entirely — which is why the framing is
a separate call rather than an argument.
Win ¶
Win(
source: tuple[AnyCol, ...],
function: str,
partition: tuple[AnyCol, ...],
*,
descending: bool = False,
offset: int = 1,
)
Bases: Col[R]
A framed window expression: a function, its column, and its partition.
A Col[R], so it projects like any other column —
select(n=q.dsd.id.count().over(q.dsd.agency)) is fine. Only the filter
position is restricted, and by the return type of its comparisons rather
than by a runtime check.
__canonical__ ¶
The frame as well as the column.
Two ranks over the same column differing only in partition, direction or offset are different questions; the base key sees none of that.
WinFn ¶
A window function with no framing yet. .over() supplies that.
Unusable on its own — there is no Col here to project or compare — which
is the point: a window function without a partition is a question nobody
means to ask, and splitting the two stages makes that unspellable rather
than merely discouraged.
over ¶
Select this function over rows sharing the given columns.
No arguments means one partition covering every row — rank() over the
whole result rather than within a group. That is SQL's own meaning for
an empty PARTITION BY, so it is spelled by omission rather than
refused.
AnyPred ¶
Non-generic base for predicates, so the compiler can name one.
Pred[R] is invariant in its row kind — __and__ takes another Pred[R],
a contravariant position — so Pred[ComponentRow] does not satisfy a
Pred[RowKind] parameter. The compiler erases row kinds anyway: they exist
to stop a predicate reaching the wrong query, and by compile time that has
already been checked. So it takes this instead.
AggPred ¶
Bases: AnyPred
A test on an aggregate — the vocabulary having() is written in.
Exported, and abstract, for the reason WinPred is: a clause's signature
has to name what it takes, and naming the concrete node would publish
compiler detail. test_public_surface holds both halves of that — every
type in a public signature must be importable, and no concrete expression
node may be.
Non-generic for the reason AnyAgg is: AggCompare[R] is invariant
through Pred[R], so a frame's own predicate does not satisfy an
AggCompare[RowKind] parameter. Row kinds are erased by compile time and
having() has already checked the predicate belongs to this root.
The data lives here so the compiler can read it off whatever it narrowed
to — the same reasoning as AnyAgg.
WinPred ¶
Bases: AnyWinPred
A test on a windowed value. Not a Pred[R], and that is load-bearing.
A WHERE runs before the window is computed, so a rank compared there
would be a rank over a set that no longer exists by the time it is
filtered. Keeping the two predicate types apart means filter(rank == 1)
is a type error naming qualify rather than a wrong answer.
Cursor ¶
Bases: AnyCursor
An opaque position in one query's order — where the next page starts.
Carries the sort-key values of the last row of a page, in the order the
ORDER BY emitted them, plus the direction of each. after() turns that
into the lexicographic "strictly after this row" comparison, which is what
makes keyset paging skip no rows and repeat none even while rows are being
inserted underneath it — the failure offset has and cannot fix.
Parameterised on the row kind so a cursor from one query cannot be handed
to another. The directions travel with it because a cursor is only
meaningful against the order that produced it: after() checks them and
refuses a cursor whose shape no longer matches.
Do not construct one; take it from Query.page.
RowKind ¶
Marker for the relation a query is rooted at. Never instantiated.
A pure phantom: it carries no behaviour and exists so that a predicate
built for one root cannot be handed to a query over another. That check is
the only reason the parameter exists, and it is worth the noise — without
it, reg.query(q.component).filter(q.codelist.id == "X") type-checks and
then emits a WHERE against a table absent from the FROM.
Untyped ¶
Untyped(
path: tuple[str, ...],
table: str,
column: str,
component_id: str = "",
kind: ColumnKind | None = None,
)
Bases: Col[R]
A column whose type nothing has stated, and its SDMX name.
Every other column class knows what it holds and enforces it:
Str.__eq__ refuses an int literal, Num.__eq__ refuses a str, and that
check (_check_literal) is why they are separate classes rather than one
with a flag. A dataflow's observation column has no such declaration —
the physical column is whatever its producer wrote — so this one holds
anything and checks nothing.
It offers exactly the operations that mean one thing without knowing the type, and no more:
- comparison and ordering, which is what a time range needs;
is_in, withoutStr's member-type check;-,*,/, which are numeric whatever the operands were, and which is what a share needs;.str, where the caller says to read it as text.
+ is deliberately absent. It means addition on Num and
concatenation on Str, and with no declared type there is nothing to pick
by: inheriting Str's would make obs.col("A") + obs.col("B") compile to
||, silently concatenating two measures into "12". Absent, it is a
plain TypeError at the call.
Why not simply put ordering on Col. Version extends Str and
overrides only sort_column, so an inherited text >= would make
version >= "1.10" true for 1.9 — #311 as a predicate rather than as
an ORDER BY. Today that mistake is unspellable, and widening the base
would make it spellable everywhere in order to fix it in one place. A
column class that opts in leaves the structural surface exactly as tight as
it was.
The remaining cost is bounded and honest: obs.col("REF_AREA") * 2
type-checks and fails at DuckDB, because nothing here can know better.
§6.3 of docs/design/data-plane-query.md is how it learns — a component's
DSD representation says whether it is enumerated or carries a DataType,
so a typed accessor can return Str or Num exactly, and this class stays
for the dynamic case.
component_id is the SDMX id this column was asked for by.
kind, when known, is what the column holds (#511): a literal of the
other kind is refused as the predicate is built. Unknown — no DSD
type and no stored type to read — checks nothing, as before.
str
property
¶
Text operations, where the caller states the reading.
contains on an untyped column is a claim the caller makes — which
is the difference between this and an inherited +: one is asked for
by name, the other would be picked silently.
is_in ¶
As Col.is_in, plus the kind check on the first member — as Str.is_in checks its own.
Descending ¶
What ref.desc() returns — a sort key with its direction attached.
Direction is per key, not per call, because one order_by can carry
several keys and they routinely disagree: "newest version first, then id
ascending" is the common shape.
A marker on the key rather than a descending=[True, False] list
parallel to the keys. polars takes the list; SQLAlchemy, ibis and Django
all mark the key (col.desc(), t.a.desc(), "-a"). Parallel
arrays put the direction at a distance from the thing it describes and go
wrong silently when the two lengths drift, which is a failure mode this
codebase already carries an error message for.
Ascending is the bare ref, so there is no asc(): an unmarked key is
ascending, as in every one of those libraries.
Generic over the key rather than the row kind, so unwrapping gives the
ref type back: a row query gets a ColumnRef, a FrameQuery gets whatever
it accepts. Parameterising on the row instead would erase that and force a
cast at every use.
Observation queries¶
LocalRegistry.obs(flow) returns an ObsQuery — the path language rooted at a dataflow's observations.
ObsQuery ¶
A query over observations, which also names their columns.
Every structural query needs two objects — the accessor (sl.q.code, a
module singleton) and the query — and that is fine there, because the
accessor is importable and shared. An observation accessor is neither: it
is built per binding, so making a caller hold both means making them find
the binding first.
obs = store.obs(binding)
obs.filter(obs.col("REF_AREA") == "ON").select(v=obs.col("OBS_VALUE")).collect()
So the query carries the accessor and exposes its columns. Refinements
return Self and attrs.evolve keeps the concrete class, so column access
survives the whole chain rather than being lost at the first filter.
col() rather than [], which was the first cut: Query.__getitem__
already means look up a row by id for a string, and slicing for a slice.
Overriding it here would give one spelling two meanings depending on the
root — the LSP violation basedpyright rejected, and a genuine trap for
anyone reading a query without knowing which root it has.
This is also where §6.2's Query/KeyedQuery split will land: ids() and
urns() have nothing to return for an observation and should be absent
here rather than raising.
structure
property
¶
The dataflow's DataStructure, or None for a query a store built without one.
What the retired Dataset exposed piecemeal — dimensions,
measures, wide_dimension — is this structure's to answer (#516):
obs.structure.dimensions.ids(), obs.structure.measure_dimension.
A registry's obs(flow) always resolves it; only
DuckDBDataStore.obs(binding) without dsd= leaves it out.
col ¶
The column for one SDMX component. See Observations.__getitem__.
Declared Untyped, which is what it always returned (#413). It said
Col — the lie #347 corrected on __getitem__ and left here — so the
operators Untyped really offers were invisible to every checker and
every reader: the comparisons, and .str's contains, starts_with,
ends_with and caseless rung. Those four are SDMX-REST's co, nc
(negated), sw and ew, with the LIKE escaping done by
_label_match rather than by each consumer.
text ¶
The column for one component as text — exact Str[ObsRow] where col gives Untyped (#511).
col has one signature for every id, so it cannot promise a type; this
is where a caller states the expectation and the model checks it:
obs.text("REF_AREA").str.contains("C"). See Observations.text.
Raises:
| Type | Description |
|---|---|
KeyError
|
If component_id is not a component. |
TypeError
|
If the DSD or the store says it holds numbers. |
num ¶
The column for one component as a number — exact Num[ObsRow] (#511).
obs.num("OBS_VALUE") > 1.5, with Num's aggregates and arithmetic
visible to a checker. See Observations.num.
Raises:
| Type | Description |
|---|---|
KeyError
|
If component_id is not a component. |
TypeError
|
If the DSD or the store says it holds text — a code, a time period, a string. |
filter_components ¶
Narrow by an SDMX REST component filter — a key, structurally (#329).
ComponentFilter is what a dotted key plus c[DIM]=… parameters parse
into, and it was the one typed thing the SQL-text API took. It is
translated here so that API can go: each constrained component becomes
is_in, which is what its own SQL rendering did.
obs.filter_components(parse_components(key, params, dsd))
None means unconstrained and contributes nothing. [] means the
permitted set is empty and matches no row — is_in([]) compiles to
1 = 0, the reading polars and SQLAlchemy both take and the one #256
argued from first principles. A SqlRelation value stays lazy, as a
subquery.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
component_filter
|
ComponentFilter
|
Dimension and attribute constraints, by SDMX id. |
required |
Returns:
| Type | Description |
|---|---|
Self
|
The query, narrowed. Unchanged when the filter is a wildcard. |
Raises:
| Type | Description |
|---|---|
KeyError
|
If the filter names a component this dataflow has not got, listing the ones it has. The filter is parsed against a DSD and applied against a binding, so a mismatch between them surfaces here rather than as a missing column at the database. |
sink_parquet ¶
Write the observations to a Parquet file, a batch at a time.
Typed as frame() is — each batch of stream() cast from the DSD —
so the file holds what a frame would, without the frame. stream()
is the raw rows, for a store whose values do not all fit.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str | Path
|
Destination file. |
required |
chunk_rows
|
int
|
Rows per batch, and so per row group. |
10000
|
sink_csv ¶
Write the observations to a CSV file, a batch at a time.
Plain CSV, a column per component, as frame().write_csv would write;
sdmxlib.formats.sdmx_csv.write_sdmx_csv is the SDMX-CSV message.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str | Path
|
Destination file. |
required |
chunk_rows
|
int
|
Rows per batch. |
10000
|
sink_json ¶
sink_json(
path: str | Path,
*,
lang: str,
version: Literal["1.0", "2.0"] = "2.0",
message_id: str | None = None,
prepared: datetime | None = None,
sender_id: str = "sdmxlib",
chunk_rows: int = 10000,
) -> None
Write the observations as an SDMX-JSON data message, a batch at a time.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str | Path
|
Destination file. |
required |
lang
|
str
|
The language of the names the message carries — each
dimension's, value's and the dataflow's — and its
|
required |
version
|
Literal['1.0', '2.0']
|
|
'2.0'
|
message_id
|
str | None
|
|
None
|
prepared
|
datetime | None
|
When the message was prepared; timezone-aware, since the schemas require a zone. Defaults to UTC now. |
None
|
sender_id
|
str
|
The sender's id. |
'sdmxlib'
|
chunk_rows
|
int
|
Rows per batch. |
10000
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If the query has no |
validate ¶
Every value its component's codelist does not list, and how often.
The report frame() cannot give: a typed read stops at the first bad
value, and this one names them all —
┌──────────┬───────────────┬───────┐
│ column │ invalid_value │ count │
└──────────┴───────────────┴───────┘
empty when every value fits. It reads the raw rows (stream()) and
reports a batch at a time (sdmxlib.polars.validation_report), so a
dataflow is checked without being held.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
chunk_rows
|
int
|
Rows per batch. |
100000
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
The violations, components in DSD order, values sorted within one. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the query has no |
pivot ¶
The observations wide: one column per value of a dimension.
The axis is on, or the DSD's MeasureDimension — SDMX 2.1's formal
signal that a dataflow arrives long, one OBS_VALUE per measure per
row. Its values become columns, in the order its concept scheme or
codelist lists them (in sorted order, read from the data, when that is
not resolved), and every other dimension, with the time dimension,
forms the index. One measure fills the cells; several give a column
per measure per value. Attributes are not carried.
A pl.DataFrame, not a LazyFrame (#516), as every other terminal's
result is: the query has run by the time a pivot can start, so
laziness after it would push nothing back into the fetch, and when
the axis's enumeration is unresolved its columns can only be read
from the rows. .lazy() on the result costs nothing.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
on
|
str | None
|
The dimension to spread into columns. Needed when the DSD declares no measure dimension. |
None
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
The wide frame, rows in the order their series first appear in |
DataFrame
|
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If there is no axis — |
ObsRow ¶
Bases: RowKind
One observation of one dataflow.
One kind for every dataflow, not one per dataflow. It catches the error
that matters — an observation predicate handed to a structural query, or
the reverse — and permits the rarer one, predicates from two different
dataflows, which the binder catches anyway because it resolves every column
against a binding. The alternative, ObsRow[Flow] keyed on a marker type,
buys static separation between flows at the cost of an annotation nobody
writing a script will write.
Domain model¶
Every domain object is an attrs frozen class: a change is an attrs.evolve copy, and every rule an artefact can check on its own runs as it is built.
Data structures¶
DataStructure ¶
Bases: MaintainableArtefact
Defines the shape of a dataset — dimensions, attributes, measures.
Example
dsd.dimensions["FREQ"].codes # ItemList[Code] if resolved dsd.code_map(lang="en") # {dim_id: {code_id: label}} dsd.all_codelists # {dim_id: Codelist} dsd.column_labels(lang="en") # {dim_id: concept label} dsd.enumerated_dimensions # ItemList[Dimension]
components
property
¶
All components: dimensions + measure_dimension (if any) + attributes + measures.
key_dimensions
property
¶
Dimension components in DSD order — regular dimensions, then time.
Not, on its own, the shape of a REST key, which is what an earlier
version of this docstring claimed (#337). Whether the time dimension
occupies a segment depends on the API version, so ask
key_slot_ids and say which. This property is the raw ordering both
answers are built from.
enumerated_dimensions
property
¶
Dimensions that are constrained to a codelist.
free_text_dimensions
property
¶
Dimensions that are not enumerated.
all_codelists
property
¶
All resolved codelists keyed by component id.
ref_to ¶
Build a resolved Ref to a component of this structure, by id.
Dispatches on which list holds the id — dimensions, the time or
measure dimension, measures, attributes or groups — so the caller
states the id and nothing else. That is unambiguous rather than merely
tidy: storage.schema declares PRIMARY KEY (dsd_urn, component_id)
on dsd_component, so a component id is already unique across the
whole structure in this library, whichever list it sits in.
validation enforces the same thing on the model (#407).
Ref[object], not a six-way union: the class varies, every field that
consumes one of these is declared Ref[object]
(HierarchyAssociation.linked_object, Categorisation.target), and the
URN carries the real class, so nothing is lost at runtime.
Raises:
| Type | Description |
|---|---|
KeyError
|
If no component has this id, naming the lists searched. |
Example
HierarchyAssociation(..., linked_object=dsd.ref_to("REF_AREA"))
find_component ¶
find_component(
component_id: str,
) -> (
Dimension
| TimeDimension
| MeasureDimension
| Attribute
| Measure
| None
)
The component with this id, whatever kind it is, or None.
components deliberately omits the time dimension — it is not part of
the ordinary component lists — so it cannot answer "does this structure
define X". This can, which is what a caller holding only an id needs.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
component_id
|
str
|
Id to look for. |
required |
Returns:
| Type | Description |
|---|---|
Dimension | TimeDimension | MeasureDimension | Attribute | Measure | None
|
The component, or None if the structure defines no such id. |
key_slot_ids ¶
Positional slots of a REST data key, in order, for one API version.
The two SDMX-REST generations disagree about the time dimension, and the disagreement is silent — a key of the wrong length is either rejected outright or, worse, read with every value one slot out (#337):
| API | slots | time filtered by |
|---|---|---|
"2.1" (REST 1.5) |
dimensions, without time | startPeriod / endPeriod |
"3.0" (REST 2.2.2) |
dimensions, then time | a segment, or the periods |
ECB's canonical M.USD.EUR.SP00.A is the 2.1 shape: five segments for
a six-dimension structure. sdmx1 excludes time for the same reason,
commenting "TimeDimensions handled by query parameters".
A formal MeasureDimension occupies a slot too, and it goes after
the regular dimensions and before time — the order the 2.1
DimensionList declares. The slot exists whether or not the caller
selects any measures; unconstrained, it is simply an empty segment.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
api
|
ApiVersion
|
The REST API version, required and never inferred. A default would build a wrong-shaped key silently, which is the failure this method exists to prevent. |
required |
Returns:
| Type | Description |
|---|---|
list[str]
|
Component ids, in the order their values appear in the key. |
role_of ¶
What part of the shape component_id plays, or None if unknown.
The question a backend asks before deciding whether it can filter on a
column. See ComponentRole for why the roles are cut this way.
None means this structure does not define that component — not
"no role". A caller must treat it as unpushable rather than guessing,
because the alternative is building a positional key with a wrong slot,
which queries a different slice silently instead of failing.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
component_id
|
str
|
The SDMX component id, not a physical column name. |
required |
Returns:
| Type | Description |
|---|---|
ComponentRole | None
|
The role, or |
code_map ¶
Build {component_id: {code_id: label}} for all resolved enumerations.
A ValueList-enumerated component is included, keyed by its values.
A LazyCodelist answers this with one SQL projection, cached and
single-flight at the source; iterating its codes instead would build a
Code and an InternationalString per row to read one string off each
(CL_GEOGRAPHY: 78k codes, ~16s). sdmxlib.polars has taken that fast
path for a while — this took the slow one until #306 removed the
implicit iteration that was hiding it.
column_labels ¶
Map component ids to human labels from their concepts.
add ¶
add(
items: Dimension
| MeasureDimension
| Attribute
| Measure
| list[
Dimension | MeasureDimension | Attribute | Measure
],
) -> DataStructure
Return a new DataStructure with items added to the appropriate list(s).
Dispatches by type — dimensions go to dimensions, attributes to attributes, measures to measures. Accepts a single component or a mixed list.
Example
dsd = dsd.add(sl.Dimension(id="PARTNER", representation=cl)) dsd = dsd.add([sl.Dimension(id="PARTNER", representation=cl), sl.Attribute(id="OBS_STATUS")])
drop ¶
Return a new DataStructure with the given component(s) removed.
Searches dimensions, attributes, and measures by id. Accepts a single id string or a list of id strings. Raises KeyError if any id is not found in any component list.
Example
dsd = dsd.drop("SECTOR") dsd = dsd.drop(["SECTOR", "OBS_STATUS"])
Dimension ¶
Bases: ConceptIdentityMixin, RepresentationMixin
An identifying component of a DataStructure.
text_type
property
¶
Text type if the representation is a Facet, else None.
Declared as Facet.type is, which is what it returns: a DataType for
the types SDMX names, a plain str for one it does not. It said
str | None, which is true — DataType is a StrEnum — but hid the
enum from a checker narrowing case DataType.Double:.
enumeration_urn
property
¶
URN of the referenced codelist or value list, or None.
TimeDimension ¶
Bases: ConceptIdentityMixin
The time dimension of a DataStructure (always id=TIME_PERIOD).
text_type
property
¶
Text type from the representation, as Facet.type declares it.
MeasureDimension ¶
Bases: ConceptIdentityMixin
The measure dimension in an SDMX 2.1 cross-sectional DSD.
Acts as a regular dimension whose values name the measures present in the
dataset (e.g. EMPLOYMENT, UNEMPLOYMENT for an LFS flow). Its presence is
the formal SDMX 2.1 signal that data arrives in long format with one
OBS_VALUE per row — a pivot on this dimension produces the wide
(multi-measure) view analysts expect.
Unlike every other component, this one is enumerated by a ConceptScheme,
not a Codelist: the values naming the measures are concepts. The XSD types
the enumeration ConceptSchemeReferenceType. That is why this class does
not use RepresentationMixin — the mixin's representation is typed
for a Codelist, and narrowing a mutable attribute in a subclass is not
sound. It carries its own field and its own accessors, named for what they
actually hold: concept_scheme and concepts. There is deliberately no
codelist/codes alias — those names were wrong here.
SDMX 3.0 removes MeasureDimension entirely. Multi-measure data is instead
modelled with multiple Measure components in the MeasureDescriptor,
giving each observation all measure values as columns (wide format). The
two patterns produce different data shapes: MeasureDimension is long
(one value per row, pivot to get wide), while 3.0 measures are natively
columnar.
is_enumerated
property
¶
True if this component is constrained to a ConceptScheme.
resolved_concept_scheme
property
¶
The resolved ConceptScheme, or None for free-text/unresolved.
Narrowing helper mirroring RepresentationMixin.resolved_codelist:
non-None exactly when is_enumerated and the Ref has been resolved.
concept_scheme
property
¶
The resolved ConceptScheme enumerating this measure dimension.
Raises:
| Type | Description |
|---|---|
TypeError
|
The representation is a |
UnresolvedRef
|
The ConceptScheme has not been fetched. |
concepts
property
¶
Concepts from the resolved ConceptScheme, or None if unresolved.
Attribute ¶
Bases: ConceptIdentityMixin, RepresentationMixin
A descriptive component attached to an observation, series, or dataset.
Measure ¶
Bases: ConceptIdentityMixin, RepresentationMixin
An observed value component (PrimaryMeasure in v2.1, Measure in v3.0).
In SDMX 2.1, a DSD has exactly one measure (OBS_VALUE). In SDMX 3.0+,
a DSD may declare multiple measures — each observation then carries all
measure values as separate columns. This covers both multi-valued
observations (e.g. a value with confidence bounds) and wide microdata
(e.g. census records with many variables per person).
Group ¶
Bases: ParsesUrn
A dimension grouping (v2.1). None for v3.0 sources.
AttachmentLevel ¶
Bases: Enum
Where an attribute attaches within the observation structure.
AttributeUsage ¶
Bases: Enum
Whether an attribute must be reported.
SDMX 3.0 / 3.1 UsageType (per SDMXStructureDataStructure.xsd).
SDMX 2.1 used a larger enum (AssignmentStatusType: Mandatory /
Conditional); the reader collapses Conditional → OPTIONAL when
parsing 2.1 inputs.
parse
classmethod
¶
Parse any SDMX-flavoured usage string into the canonical enum.
Accepts the 3.0/3.1 lowercase form (mandatory/optional)
and the 2.1 capitalised form (Mandatory/Conditional);
Conditional maps to OPTIONAL. Returns None for
None input. An already-typed AttributeUsage passes
through unchanged (so callers can construct an Attribute
either way without a manual coercion).
ComponentRole ¶
Bases: Enum
What part of a dataflow's shape a component plays.
The question a backend asks before deciding whether it can filter on a component.
The SDMX REST data endpoint constrains different components by different
means — the positional key, startPeriod/endPeriod, measures=,
c[<id>]= — so "which component is this" is not the useful question and
"what can be done with it" is. A capability declares itself over these
rather than over column names, because an observation's columns come from a
DSD: column="REF_AREA" is a fact about one dataflow, not about a backend.
KEY_DIMENSION and TIME_DIMENSION are split because the time dimension is
the one component reachable two ways — as a key slot, which takes equality
or membership, and as startPeriod/endPeriod, which takes a range. A
backend that can serve a range on time can serve nothing of the sort on any
other dimension, and collapsing them would claim otherwise.
MetadataAttributeUsage ¶
A metadata attribute a DataStructure uses, and what it attaches to (#422).
SDMX 3.0 lets a DSD report reference metadata alongside data: it names an
MSD (DataStructure.metadata) and, in its AttributeList, uses some of that
MSD's attributes, each attached like a data attribute. The usage has no id
of its own — the schema prohibits one — and is known by the attribute it
uses: id is that attribute's id, local to DataStructure.metadata
(3.0's MetadataAttributeReference).
Attributes:
| Name | Type | Description |
|---|---|---|
id |
str
|
The id of the MSD attribute used. |
attachment |
AttachmentLevel
|
What it attaches to; as |
related_dimensions |
list[str] | None
|
As |
optional_dimensions |
frozenset[str]
|
As |
related_group |
str | None
|
As |
annotations |
Annotations
|
The usage's annotations. |
Codelists¶
Codelist ¶
Bases: MaintainableArtefact
A finite set of valid codes for an enumerated component.
Constructor accepts convenient Python types alongside the strict stored
types — dict[str, str] for name/description, list[Code] for
codes. After construction the stored types are always strict.
Example
cl = Codelist( id="CL_FREQ", maintainer="SDMX", name={"en": "Frequency"}, codes=[Code(id="A", name={"en": "Annual"})], ) cl.codes["A"] # Code by id cl.roots() # top-level codes cl.narrower("A") # direct children of code A (SKOS-aligned)
ref_to ¶
Build a resolved Ref to a code within this codelist.
The URN is built from this scheme's agency, id and version, so it
cannot name a parent that does not exist — which is what six hand-written
keyword arguments to SdmxUrn.make invite, and what
validate_references now correctly reports as broken (#407).
Raises:
| Type | Description |
|---|---|
KeyError
|
If no code with this id is in this codelist. |
Example
HierarchicalCode(id="CA", code=cl.ref_to("CA"))
broader ¶
Direct parent (SKOS broader) of code_id. None for roots.
narrower_transitive ¶
narrower_transitive(
code_id: str,
*,
depth: int | None = None,
include_self: bool = False,
) -> ItemList[Code]
All descendants of code_id. depth bounds the hop count.
include_self defaults off because this is a closure, and SKOS
specifies narrowerTransitive as non-reflexive. It matches
LazyCodelist.narrower_transitive, which is the same walk over a
store.
broader_transitive ¶
broader_transitive(
code_id: str,
*,
depth: int | None = None,
include_self: bool = False,
) -> ItemList[Code]
All ancestors of code_id. depth bounds the hop count.
include_self as in narrower_transitive: off by default, and the
node itself comes first when asked for, so the result stays
leaf-to-root.
code_ids ¶
Every code id, in order.
The eager half of CodelistLike.code_ids; on a LazyCodelist the same
call projects one column out of SQL rather than building 579,951
Code objects to read their ids (#398).
codes_for ¶
{code_id: Code} for exactly ids; unknown ids are absent.
The eager half of CodelistLike.codes_for — an ItemList.get per id.
codes_for_each
classmethod
¶
The eager half of CodelistLike.codes_for_each: each codes_for in turn, there being nothing to batch.
materialize ¶
This codelist — it is already materialised.
The eager half of CodelistLike.materialize. Returning self rather
than a copy: the protocol promises the codes are built, not that the
result is fresh, and copying a 579,951-code codelist to honour a call
that changes nothing would be the cost this whole issue is about.
expand_wildcards ¶
This codelist with each wildcarded member value of its extensions spelled out (#451).
SDMX-ML 3.x lets a code selection name codes by pattern: A% is
every code of the extended codelist whose id starts with A.
SDMX-JSON's member value is a plain id, so a pattern is written as
the ids it matches now, each with the member's cascade. A code added
to the extended codelist later is then not selected.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
available
|
Iterable[Codelist]
|
Codelists to find an extended codelist in when the extension's reference is not resolved — a message's own, say. |
()
|
Returns:
| Type | Description |
|---|---|
Codelist
|
A new codelist, or this one when no member value is wildcarded. |
Raises:
| Type | Description |
|---|---|
LookupError
|
If a wildcarded member cannot be expanded;
|
apply_extensions ¶
This codelist with the codes its extensions inherit, and no extensions (#421).
SDMX 3.x lets a codelist extend others: each CodelistExtension
contributes the extended codelist's codes, all of them or a selection,
each id behind an optional prefix. This is the codelist a reader of
the 3.x document means — what SDMX-ML 2.1, which has no extensions,
has to be given instead.
Precedence follows the IM (§4.3.3.2: "the latest Codelist used overrides any previous Codelist") and FMR, which flattens the same way (#486): the codelist's own codes win over inherited ones, and a later extension over an earlier one. An inherited code keeps the place it first appeared in. A selected code whose parent was not selected becomes a root, as in the extending codelist itself. An extended codelist that extends others has its own extensions applied first.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
available
|
Iterable[Codelist]
|
Codelists to find an extended codelist in when the extension's reference is not resolved — a message's own, say. A resolved reference is used as it stands. |
()
|
Returns:
| Type | Description |
|---|---|
Codelist
|
A new codelist, or this one when it extends nothing. |
Raises:
| Type | Description |
|---|---|
LookupError
|
If an extended codelist is neither resolved nor in
|
ValueError
|
If the extensions form a cycle. |
TypeError
|
If a geographic codelist would inherit codes of another kind, which 3.x allows and the class cannot hold. SDMX-ML 2.1 writes such a codelist as a plain one, where they can. |
__len__ ¶
How many codes this codelist holds.
LazyCodelist has had this since it existed — it is one COUNT(*)
there rather than a materialised list — and its absence here was the
reason a caller counting codes had to know which class it held (#398).
code_map ¶
{code_id: label} in lang, falling back to the id.
The one member LazyCodelist had and this class did not, which is why
callers wanting labels had to ask which class they were holding — see
sdmxlib.model.protocols.CodelistLike (#398). The lazy side projects
it straight out of SQL; here it is a dict comprehension over codes that
are already in memory, so neither needs a cache
(docs/design/caching.md).
Falls back to the id rather than omitting the code, because a label map with holes in it silently drops rows when a caller joins on it.
broader_map ¶
{code_id: parent_id} for every code; None for roots.
narrower_map ¶
{code_id: (child_id, ...)} for every code, in declaration order.
ancestors_map ¶
{code_id: (ancestor_id, ...)} root-first, excluding self.
descendants_map ¶
{code_id: (descendant_id, ...)} in pre-order DFS, excluding self.
inherit_along_broader ¶
Propagate values down the parent chain.
For each code, walk the broader chain and assign the first
value encountered. Codes whose chain has no explicit value are
omitted from the result.
add ¶
Return a new Codelist with codes added.
Accepts a single Code or a list of Codes.
drop ¶
Return a new Codelist with the given code(s) removed.
Accepts a single id string or a list of id strings. Raises KeyError if any id is not present.
Code ¶
Bases: ParsesUrn
A single code within a Codelist.
CodelistExtension ¶
One extended codelist composed into an extending codelist.
SDMX 3.0 / 3.1 CodelistType carries a (possibly empty) ordered
sequence of CodelistExtension entries. Order matters — the
XSD documents it as precedence order for conflict resolution.
An extension contributes either all codes of the source codelist (no
selection) or a filtered subset (one CodeSelection with an
include or exclude mode). The optional prefix is applied to all
inherited code ids in the extending codelist, useful when extending
codelists that would otherwise collide on code ids.
Not modelled in SDMX-ML 2.1; the 2.1 serializer silently drops this field if the extending codelist is also written as 2.1 output.
CodeSelection ¶
An ordered list of code references with an inclusion mode.
Maps to the SDMX 3.0 / 3.1 CodeSelectionType. Wrapping the mode
on the selection (rather than splitting into two parallel fields)
keeps the XSD's mutual-exclusion invariant from being a runtime
check the caller has to remember.
MemberValue ¶
A single code reference in a CodeSelection.
The value is a code id within the referenced (extended) codelist,
optionally containing the % wildcard character per
WildcardedMemberValueType. cascade is the per-member cascade
behaviour; None means accept the XSD default (SELF_ONLY).
CascadeSelection ¶
Bases: Enum
How a MemberValue cascades into a hierarchical codelist.
The SDMX 3.1 CascadeSelectionType is a string union of xs:boolean
plus the literal excluderoot. Three semantic values; we keep them
as a small enum.
InclusionMode ¶
Bases: Enum
Whether a CodeSelection lists codes to include or to exclude.
SDMX 3.0 / 3.1 CodelistExtensionType uses a <xs:choice> between
<InclusiveCodeSelection> and <ExclusiveCodeSelection>, both of
type CodeSelectionType. Semantically the two are one selection with
a mode flag — we model the flag explicitly so a caller can't
accidentally populate both branches.
GeographicCodelist ¶
Bases: Codelist
A codelist of geographic feature sets (SDMX 3.x).
A Codelist in every respect SDMX gives it — its URN, its references, the
components it enumerates — whose codes are each a GeoFeatureSetCode.
SDMX-ML 2.1 has no such class and writes it as a plain codelist; see
sdmxlib.unrepresentable.
Example
GeographicCodelist( id="CL_REGION", maintainer="ESTAT", name={"en": "Regions"}, codes=[GeoFeatureSetCode(id="BE1", value="POLYGON((...))")], )
GeoFeatureSetCode ¶
Bases: Code
A code standing for a set of geographic features (SDMX 3.x).
Its URN is a plain Code's: SDMX gives geographic codes no URN class of
their own, so this is a Code with one more field rather than a new kind
of item.
GeoGridCodelist ¶
Bases: Codelist
A codelist of geographic grid cells (SDMX 3.x).
A Codelist, as GeographicCodelist is, whose codes are each a
GeoGridCode and which defines the grid they divide.
Example
GeoGridCodelist( id="CL_GRID", maintainer="ESTAT", name={"en": "1 km grid"}, grid_definition="EPSG:3035;1km", codes=[GeoGridCode(id="C1", geo_cell="1kmN2599E4498")], )
GeoGridCode ¶
Bases: Code
A code standing for one cell of a geographic grid (SDMX 3.x).
A Code with the cell it stands for; see GeoFeatureSetCode for why it
is not its own kind of item.
CodelistLike ¶
Bases: EnumerationLike, Protocol
What a resolved codelist offers, whether it is eager or lazy (#398).
Dimension.resolved_codelist follows the component's Ref. When the DSD
came from a LocalRegistry that ref resolves to a LazyCodelist, not a
Codelist — so declaring Codelist was a lie, and one that reached
users: polars.schema, polars.validation_report,
polars.structure_frame, Dataset.pivot and Dataset.label all raised
TypeError on a DSD read from a local registry, because they iterated or
took len() of something that had become a Query.
The two classes already shared 28 public members. This declares the part
callers use, and the gaps it exposed were closed rather than narrowed
around: Codelist gained code_ids, code_map, materialize and
__len__, each of which LazyCodelist already had in substance. So there
is no isinstance branch anywhere — every member answers on its own
cheapest path, a SQL projection on one side and a comprehension on the
other.
Materialising instead was measured and rejected: callers want ids or a
count, and on CL_GEOGRAPHY those cost 92 ms and 22 ms against 2,854 ms
to materialise — 31x and 129x, which is the cost #378 removed. That option
survives as materialize(), for the two callers that genuinely need each
Code and should say so in one visible call.
Methods rather than the fields they wrap, which is not a style choice.
code_ids() exists because a protocol member declared as
codes: ItemList[Code] | Query[...] does not check: basedpyright reads
Codelist.codes as ItemList[Code] for attribute access and as
__converterDescriptor_codes for protocol matching, so the same member
satisfies one and not the other. name and description are absent for
the identical reason and no other — both are real, both are on both
classes, and neither can be declared here until that is fixed. Minimal
repro in tests/typing/probe_codelist_like.py; reach for the concrete
class meanwhile.
codes_for ¶
{code_id: Code} for exactly ids, each fully built.
Ids the codelist does not hold are absent from the result rather than an error — a caller resolving observed values meets values outside the enumeration, and the id is then its own label.
The targeted counterpart to materialize: one batched read on a lazy
codelist, whose cost follows len(ids) rather than the codelist. The
SDMX-JSON data writer called materialize per component per message
and paid for every code in a 579,951-code geography to label the
handful a response observed (#410).
codes_for_each
classmethod
¶
codes_for for several codelists of this class at once, answered in request order.
A lazy codelist answers them in one read per store, where asking each
in turn cost three statements apiece (#441). codes_for_all groups a
caller's requests by class and asks each class once.
materialize ¶
This codelist with every Code built, for callers needing more.
Free on an eager Codelist, which returns itself; a full read on a
lazy one, whose cost is the whole codelist. A caller that needs some
codes' annotations or parent_id rather than all of them wants
codes_for instead — that was the SDMX-JSON data writer (#410).
EnumerationLike ¶
Bases: Protocol
What a resolved enumeration offers: a codelist, eager or lazy, or a ValueList.
A component's representation may enumerate its values by either kind in
SDMX 3.x (#422). This is the part both answer — the ids, their labels and
how many there are — so the label maps and the value checks built on a
component work whichever it holds. Reach for CodelistLike when you need
Code objects, which a value list does not have.
LazyCodelist ¶
LazyCodelist(
*,
urn: str,
conn: DuckDBPyConnection,
id: str,
maintainer: Agency,
version: str,
name: InternationalString,
description: InternationalString,
legacy_is_final: bool | None,
uri: str | None = None,
is_external_reference: bool = False,
is_partial_language: bool = False,
service_url: str | None = None,
structure_url: str | None = None,
is_partial: bool = False,
valid_from: datetime | None = None,
valid_to: datetime | None = None,
)
Codelist facade backed by DuckDB.
Duck-types as Codelist for its scalar read surface — id, name,
agency, version and the rest.
codes is where it deliberately differs. It returns a Query,
a lazy plan for SQL-pushed filtering and hierarchy navigation, where a
plain Codelist returns a materialised ItemList. Since #306 that plan is
not iterable, has no len() and answers no in: those ran SQL without
saying so, which is what stops a planner pushing anything down.
So for c in cl.codes is for c in cl.codes.collect() here, and
len(cl.codes) is cl.codes.count() — the second being the point
rather than the cost, since a length used to build every Code in order to
throw them all away. On CL_GEOGRAPHY that is 78k objects for one integer.
urn
property
¶
This artefact's URN — its identity, as on an eager Codelist.
Parsed rather than held typed because the row stores it as text. Added
when the upgrade guide's own cl.urn example — on the artefact a
LocalRegistry hands back — turned out to raise AttributeError.
agency_id
property
¶
Identifier of the maintaining agency. Shorthand for maintainer.id.
description
property
¶
Longer human-readable description, keyed by language code.
is_final
property
¶
Whether the version names a final artefact. See MaintainableArtefact.is_final.
legacy_is_final
property
¶
SDMX 2.1's isFinal where it disagrees with the version. See Codelist.legacy_is_final.
uri
property
¶
External link describing this artefact, or None. See MaintainableArtefact.uri.
is_external_reference
property
¶
Whether this is a stub for an artefact held elsewhere. See MaintainableArtefact.
is_partial_language
property
¶
Whether some text lacks a language the rest carries. See MaintainableArtefact.
service_url
property
¶
Where a stub's full artefact can be queried. See MaintainableArtefact.
structure_url
property
¶
Where a stub's full artefact is written out. See MaintainableArtefact.
is_partial
property
¶
Whether this codelist holds a subset of its codes. See Codelist.is_partial.
valid_from
property
¶
Start of the artefact's validity period, or None if unbounded.
valid_to
property
¶
End of the artefact's validity period, or None if unbounded.
codes
property
¶
This codelist's codes, as a lazy query (#306).
Ordered by declaration position, as the default order rather than as
a caller's order_by. The path language's own default is its __key__
tiebreak — code_urn, under which C10 precedes C2 — and a codelist
has a stated order a caller reading cl.codes is entitled to. Setting
it as the default keeps cl.codes.group_by(...) legal, which a real
order_by would refuse, and the first order_by replaces it.
annotations
property
¶
This artefact's annotations, read on demand.
A property rather than a constructor argument because the proxy exists
to avoid loading what nobody asked for, and most callers never ask.
Uncached for the reason code_map is cached and this is not: one small
query, and a cache would need invalidating on every write.
Its absence was half of #322 — the attribute did not exist at all, so
materialize() could only return an artefact with none.
links
property
¶
This artefact's links (#422), read on demand for the reason annotations is.
extensions
property
¶
The codelists this one extends, read on demand.
On the proxy for the same reason annotations is (#322): if only the
eager reader carried them, get() and get(eager=True) would answer
differently about what codes this list is composed of.
__contains__ ¶
Whether this codelist holds a code with that id.
Its own SQL rather than self.codes.__contains__, which #306 removed:
on a query in ignored the query's own filters. On the artefact
there are no filters to ignore, so the question is well posed — this is
the one place the old behaviour was correct.
Note plain Codelist has no __contains__ at all, so this is an
addition of the lazy facade rather than part of the duck-typing.
Left as it was; narrowing that asymmetry is its own change.
ref_to ¶
Build a resolved Ref to a code in this codelist — see Codelist.ref_to.
Works without materialising: the existence check goes through this class's own subscript, which is one targeted query (#407).
Raises:
| Type | Description |
|---|---|
KeyError
|
If no code with this id is in the codelist. |
__getitem__ ¶
One code, by id or by position in declaration order.
Its own lookup rather than self.codes[key], which #306 changed under
it: on a Query the subscript is a slice, because a query is not a
sequence. CodeQuery accepted an id here and a Query cannot, without
q[0:10] and q["FREQ"] meaning two different things.
broader ¶
Direct parent of code_id — at most one. None for roots.
Read as "the code whose children include this one", which is the
children edge rather than the raw self-join this used to spell.
narrower_transitive ¶
narrower_transitive(
code_id: str,
*,
depth: int | None = None,
include_self: bool = False,
) -> Query[CodeRow, Code]
All descendants of code_id.
Ancestry is a stored fact — the writer bakes ancestor_ids — so this
is a containment test rather than recursion, and depth is an ordinary
conjunction on the baked depth column rather than anything the
compiler has to know about. The starting code's depth is fetched once
so the comparison is against a literal.
broader_transitive ¶
broader_transitive(
code_id: str,
*,
depth: int | None = None,
include_self: bool = False,
) -> Query[CodeRow, Code]
All ancestors of code_id. depth bounds hops.
broader_map ¶
{code_id: parent_id} for every code. One SQL query.
narrower_map ¶
{code_id: (child_id, ...)} for every code, ordered by position.
ancestors_map ¶
{code_id: (ancestor_id, ...)} root-first. One SQL query.
Reads the precomputed ancestor_ids column.
descendants_map ¶
{code_id: (descendant_id, ...)} in pre-order DFS, excluding self.
inherit_along_broader ¶
Propagate values down the parent chain. One SQL query.
For each code, walk the broader chain (using ancestor_ids) and
assign the first explicit value encountered. Codes with no explicit
value anywhere in their chain are omitted from the result.
code_ids ¶
Direct SQL: every code id, without creating Code objects.
The lazy half of CodelistLike.code_ids — 92 ms against 2,854 ms to
materialise, on a 579,951-code codelist (#398).
codes_for ¶
Direct SQL: {code_id: Code} for exactly ids; unknown ids are absent.
The lazy half of CodelistLike.codes_for — one batched read whose cost
follows len(ids), where materialize builds every code (#410).
Not through self.codes.filter(...is_in(ids)): that binds the ids,
converted one at a time — linear, and ~80 us each without pandas
installed (#411) — where resolve.codes_by_id matches against a
registered Arrow buffer.
codes_for_each
classmethod
¶
Direct SQL: every request's codes in one batched read per store (#441).
The lazy half of CodelistLike.codes_for_each. Requests are grouped by
connection, and two asking the same codelist share its branch of
resolve.codes_by_ids.
code_map ¶
Direct SQL: {code_id: label} without creating Code objects.
Memoised per instance per lang with a single-flight build lock
(#155): the first caller runs the SQL projection, concurrent
callers for the same lang block on the lock and then return the
hot cache rather than each re-running the query. The returned dict
is the cached instance — treat it as read-only.
__getattr__ ¶
Refuse a materialised-only member by name, not with a bare miss.
The same reasoning as LazyHierarchy.__getattr__, for a smaller gap:
codes exists here and differs only in type, so the loud refusal a
caller meets for it comes from Query itself. These two do not exist
at all, and used to say so without saying what to do about it.
Raises:
| Type | Description |
|---|---|
AttributeError
|
Always — naming the counterpart, or the ordinary message for a genuine typo. |
materialize ¶
Load all codes and return a real Codelist.
Every field the proxy holds is passed on. Omitting one made the two
read paths disagree — lazy.uri returned the stored value and
lazy.materialize().uri returned None — so materialising for an
unrelated reason silently changed the answer (#322). valid_from,
valid_to, uri and annotations were all missing.
Checked as get(eager=True) is (#492): a codelist that breaks a rule
Registry.add enforces raises SdmxConformanceError.
Value lists¶
SDMX 3.x's enumeration that is not a codelist; a component's enumerated representation is a Ref[Codelist | ValueList].
ValueList ¶
Bases: MaintainableArtefact
A closed set of values a component may take (SDMX 3.x).
What a component's representation enumerates when the values need no
hierarchy and no identity of their own — a Codelist without either. Its
URN class is ValueList, so a component enumerated by one holds a
Ref[ValueList]; RepresentationMixin.resolved_enumeration answers for
either kind. SDMX-ML 2.1 has no such class, so a 2.1 write leaves it out,
with everything that references it (sdmxlib.dropped).
Example
ValueList( id="VL_UNIT", maintainer="ESTAT", name={"en": "Units"}, items=[ValueItem(id="kg", name={"en": "Kilogram"})], )
ValueItem ¶
One value of a ValueList: the value, optionally named and described.
Not identifiable, as a Code is: SDMX gives a value item no URN and no
@uri, and its id is any string rather than an SDMX identifier — it
is the value, which may hold characters an id may not.
Hierarchies¶
Hierarchy ¶
Bases: MaintainableArtefact, PathNavigation[HierarchicalCode]
A standalone hierarchy of codes from one or more codelists.
Nodes are identified by path, as SDMX identifies them: a node id may repeat in another branch, never among siblings (#439).
tree: top-level entry points (roots) withchildrennavigationhierarchy["CANADA.ON.OTTAWA"]: one node by path; a bare id is a roothierarchy.by_id("OTTAWA"): every node with that leaf idcodes: every node, in pre-order
Example
hierarchy.tree["CANADA"].children["ON"] # tree navigation hierarchy["CANADA.ON.OTTAWA"].code() # Ref[Code] unwrap → Code
codes
property
¶
Every node in the hierarchy, in pre-order.
A tuple, not an id-keyed list: a node id may repeat in another branch
(#439), so there is no single node for a leaf id. Look a node up by
path with hierarchy["CANADA.ON"], or by leaf with
hierarchy.by_id("ON").
descendants ¶
Code ids of all descendants of root_code_id in this hierarchy.
Pure-Python counterpart to a stored hierarchy's
narrower_transitive(path) — use this when you want the descendant
set without a SQL round-trip (e.g. to feed a small id list into a
ComponentFilter's dimensions). Keyed by code id, not by path:
it finds every placement of the code.
Matches nodes by underlying code URN's item id (or by
node.id if a node has no code ref). If multiple nodes
reference the same root code, walks every subtree and dedupes
while preserving first-seen order. Returns [] if no node in
this hierarchy references root_code_id.
HierarchicalCode ¶
Bases: ParsesUrn
A node in a Hierarchy tree, referencing a code from a Codelist.
Provides tree navigation via children; the parent Hierarchy looks a
node up by path, or every node sharing a leaf id with by_id.
Example
node = hierarchy.tree["CANADA"] node.children["ON"] # tree navigation hierarchy["CANADA.ON"] # by path hierarchy.by_id("ON") # every node whose id is ON node.code() # Ref[Code] unwrap → Code
Level ¶
Bases: ParsesUrn
A named level in a Hierarchy (SDMX 3.1).
Hierarchies that declare levels allow each HierarchicalCode node
to explicitly state which structural level it belongs to — rather
than inferring level from tree depth or code ID patterns.
The optional coding_format records the character/length
constraints that codes at this level must follow (for example,
"ISO-3166-2 subdivision codes are 1 to 3 alphanumeric chars"). It
reuses the existing Facet value type so callers have one
constraint vocabulary across the model.
Example
hierarchy.levels["SECTOR"] # Level lookup by id node.level # Level | None
HierarchyAssociation ¶
Bases: MaintainableArtefact
Binds a Hierarchy to an identifiable artefact within an optional context.
SDMX 3.0 / 3.1 HierarchyAssociationType. The Hierarchy itself is
structure-agnostic — it just defines a code tree. The association
says which dimension (or other identifiable) the hierarchy applies
to, optionally within a particular context (e.g. a Dataflow).
Example
sl.HierarchyAssociation( id="HA_GEO_REF_AREA", maintainer="ESTAT", linked_hierarchy=hierarchy_ref, linked_object=dim_ref, context_object=dataflow_ref, )
SDMX-ML 2.1 has none, and binds a hierarchy to nothing, so a 2.1 write
leaves it out (formats.gaps.dropped, #421).
HierarchicalCodelist ¶
Bases: MaintainableArtefact
A 2.1 container grouping related hierarchies.
SDMX-ML 2.1 nests every <str:Hierarchy> inside a
<str:HierarchicalCodelist>, which is a maintainable in its own right —
id, agency, version, validity dates, name — and asserts that the
hierarchies inside it belong together. SDMX 3.0 removed it and promoted
Hierarchy to a top-level artefact.
sdmxlib follows 3.0 for the hierarchies themselves and models the
container as a peer that references them, for the same reason
StructureSet is modelled that way: one copy of the grouping, rather than
the container's identity denormalised onto each member.
Before this existed, reading a 2.1 document kept the inner hierarchy's name and silently discarded the container's — so a codelist named "Geographic hierarchy" containing a hierarchy named "Canada → provinces → cities" came back with only the second, and re-emitting produced a different document (#239).
Written only to SDMX-ML 2.1. The other targets leave it out and write its
hierarchies (formats.gaps.dropped, #421); a 2.1 reader names each one
for it (model._container.member_id).
AmbiguousHierarchyError ¶
Bases: ValueError
Raised when unique_hierarchy finds more than one applicable Hierarchy.
Carries the list of conflicting HierarchyAssociation URNs. The
caller asserted "at most one Hierarchy applies" (via the
unique_hierarchy accessor); the IM permits many, so this signals
that the caller's assumption doesn't hold for this registry.
Use applicable_hierarchies instead when the IM-canonical many-case
is the intended question.
LazyHierarchy ¶
LazyHierarchy(
*,
urn: str,
conn: DuckDBPyConnection,
id: str,
maintainer: Agency,
version: str,
name: InternationalString,
description: InternationalString,
uri: str | None = None,
is_external_reference: bool = False,
is_partial_language: bool = False,
service_url: str | None = None,
structure_url: str | None = None,
has_formal_levels: bool,
levels: tuple[Level, ...],
valid_from: datetime | None = None,
valid_to: datetime | None = None,
)
Hierarchy facade backed by DuckDB.
Duck-types as Hierarchy for read operations. nodes returns
a Query for SQL-pushed filtering and tree
navigation. SKOS verbs (roots/leaves/narrower/broader/
*_transitive/depth_of) and bulk projections (depths/
broader_map/...) mirror LazyCodelist exactly.
urn
property
¶
This artefact's URN — its identity, as on an eager Hierarchy.
Parsed rather than held typed because the row stores it as text. Added
when the upgrade guide's own cl.urn example — on the artefact a
LocalRegistry hands back — turned out to raise AttributeError.
agency_id
property
¶
Identifier of the maintaining agency. Shorthand for maintainer.id.
description
property
¶
Longer human-readable description, keyed by language code.
is_final
property
¶
Whether the version names a final artefact. See MaintainableArtefact.is_final.
uri
property
¶
External link describing this artefact, or None. See MaintainableArtefact.uri.
is_external_reference
property
¶
Whether this is a stub for an artefact held elsewhere. See MaintainableArtefact.
is_partial_language
property
¶
Whether some text lacks a language the rest carries. See MaintainableArtefact.
service_url
property
¶
Where a stub's full artefact can be queried. See MaintainableArtefact.
structure_url
property
¶
Where a stub's full artefact is written out. See MaintainableArtefact.
valid_from
property
¶
Start of the artefact's validity period, or None if unbounded.
valid_to
property
¶
End of the artefact's validity period, or None if unbounded.
has_formal_levels
property
¶
Whether the hierarchy declares a formal level structure (SDMX @hasFormalLevels).
levels
property
¶
The Level definitions declared by this hierarchy, in stored order.
nodes
property
¶
This hierarchy's placements, as a lazy query (#306).
Ordered by the path language's node_urn tiebreak, which is total and
stable across calls — what keyset paging needs — but not declaration
order. That is stored now (position, #439) and materialize restores
it; order by q.hcode.position here to ask for it.
annotations
property
¶
This artefact's annotations, read on demand.
A property rather than a constructor argument because the proxy exists
to avoid loading what nobody asked for, and most callers never ask.
Uncached for the reason code_map is cached and this is not: one small
query, and a cache would need invalidating on every write.
Its absence was half of #322 — the attribute did not exist at all, so
materialize() could only return an artefact with none.
links
property
¶
This artefact's links (#422), read on demand for the reason annotations is.
__contains__ ¶
Whether this hierarchy has a node at path key — a bare id is a root.
By path, as Hierarchy.__contains__ is: a node id may repeat in another
branch (#439). A node object answers False, as on the eager class — it
does not carry the path it sits at.
__getitem__ ¶
One placement, by path or by position. See Hierarchy.__getitem__.
by_id ¶
Every placement whose own id is node_id, at any depth. See Hierarchy.by_id.
leaves ¶
Placements that no other placement names as parent.
narrower ¶
Direct children of path. A node's id rather than its path is refused (#514).
broader ¶
Direct parent of path. None for roots; a node's id rather than its path is refused (#514).
narrower_transitive ¶
narrower_transitive(
path: str,
*,
depth: int | None = None,
include_self: bool = False,
) -> Query[HCodeRow, HierarchicalCode]
All descendants of path. depth bounds hops. A node's id rather than its path is refused (#514).
broader_transitive ¶
broader_transitive(
path: str,
*,
depth: int | None = None,
include_self: bool = False,
) -> Query[HCodeRow, HierarchicalCode]
All ancestors of path. depth bounds hops. A node's id rather than its path is refused (#514).
ancestor_at ¶
Map every node to its nearest ancestor at level.
The join input for hierarchical aggregation — totalling observations by province needs each geography mapped to its province. Compiled, not materialised: the mapping stays in the database and composes into a join, where applications previously traversed the tree in Python and registered the result as a view (#274).
A node at the level maps to itself; a node with no ancestor there is
absent rather than NULL. Same semantics as
Hierarchy.inherit_along_broader, which the property tests use as the
oracle.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
level
|
str | Level
|
The |
required |
include_self
|
bool
|
Whether a node already at the level maps to itself. Defaults to True, unlike the transitive walks, because this is a mapping rather than a closure — excluding the identity leaves every node at the level with no image, so a province would have no province (#274). |
True
|
Returns:
| Type | Description |
|---|---|
AncestorMap
|
A two-column |
broader_map ¶
{path: parent_path} for every node. One SQL query.
narrower_map ¶
{path: (child_path, ...)} for every node, in declaration order.
ancestors_map ¶
{path: (ancestor_path, ...)} root-first. One SQL query.
descendants_map ¶
{path: (descendant_path, ...)} in pre-order, excluding self.
inherit_along_broader ¶
Propagate values (keyed by path) down the parent chain. One SQL query.
__getattr__ ¶
Refuse a materialised-only member by name, not with a bare miss.
LazyHierarchy duck-types as Hierarchy and is what
LocalRegistry.get returns, so h.codes used to raise a plain
AttributeError: 'LazyHierarchy' object has no attribute 'codes' —
naming nothing and reading like a bug in this library rather than a
deliberate difference. #397 declared the return type so a checker now
catches it; this is the same answer for a caller who has no checker
running, and it mirrors what a Query already does for len().
Only reached when normal lookup fails, so it costs nothing on the attributes that do exist.
Raises:
| Type | Description |
|---|---|
AttributeError
|
Always — either naming the lazy counterpart, or with the ordinary message for a genuine typo. |
materialize ¶
Load the full hierarchy tree and return a real Hierarchy.
Builds the tree bottom-up: leaves first, then parents. Each
HierarchicalCode gets its children list at construction
time — no post-construction mutation.
Checked as get(eager=True) is (#492): a hierarchy that breaks a rule
Registry.add enforces raises SdmxConformanceError.
AncestorMap ¶
Every node mapped to its nearest ancestor at a Level, as a relation.
The central query in hierarchical analysis: to total observations by province you need each geography mapped to the province containing it. Applications were building this by hand — traverse the tree in Python, materialise a dict, register it as a DuckDB view, join, unregister — with their own cache around it (#274). It is one query over the ancestor paths the storage writer already bakes.
Keyed by path, (path, node_id, rollup_path, rollup_id): a node id may
repeat in another branch (#439), so a mapping keyed by id would give one
node two images. The ids ride along because they are what a join against
observation codes uses. A join input rather than an IN list;
ancestors narrows it to the rollup ids for the membership case.
Semantics follow Hierarchy.inherit_along_broader, which is the in-memory
equivalent and the oracle the property tests compare against:
- a node at the level maps to itself,
- a node below maps to its nearest ancestor at that level,
- a node with no such ancestor is absent, not mapped to
NULL.
Example
rollup = h.ancestor_at("PR") rollup.to_dict() # {"CA.ON.ON-01": "CA.ON", "CA.ON": "CA.ON", ...} rollup.collect() # pl.DataFrame
columns
property
¶
(path, node_id, rollup_path, rollup_id) — see sdmxlib.sql.SqlRelation.
to_sql ¶
Compile without executing, for use as a join input or subquery.
Self and ancestors are unioned and the greatest depth wins, which
is what makes the match the nearest one — and makes a node at the
level beat its own ancestors, since it is deepest.
ancestors ¶
The distinct rollup targets, as a one-column relation.
For the membership case — code.id.is_in(h.ancestor_at("PR").ancestors())
— where the two-column mapping is not what IN wants. Typed as the
public sdmxlib.SqlRelation it is, not the private class behind it,
so a caller can annotate what they hold (#513).
to_dict ¶
Execute and return {path: rollup_path}.
The eager form, matching what Hierarchy.inherit_along_broader
returns — both keyed by path. Prefer to_sql or ancestors where the
mapping can stay in the database.
Dataflows¶
Dataflow ¶
Bases: MaintainableArtefact
A published data product backed by a DataStructure.
Example
df.structure # Ref[DataStructure] df.structure() # DataStructure (if resolved) df.structure.dimensions # convenience via resolved structure
Concepts¶
ConceptScheme ¶
Bases: MaintainableArtefact
An ordered collection of Concepts maintained by an agency.
add ¶
Return a new ConceptScheme with concepts added.
Accepts a single Concept or a list of Concepts.
drop ¶
Return a new ConceptScheme with the given concept(s) removed.
Accepts a single id string or a list of id strings. Raises KeyError if any id is not present.
ref_to ¶
Build a resolved Ref to a concept within this scheme.
Raises KeyError if concept_id is not present.
One of seven now — Codelist, CategoryScheme, the three organisation
schemes, MetadataStructure and DataStructure all have one, on the
shared _items.item_ref path rather than a copy of SdmxUrn.make each
(#407). This was the only one for a long time, which is why authoring a
3.0 Hierarchy meant building item URNs by hand.
Example
cs = ConceptScheme(id="CS_COMMON", maintainer="SDMX", version="1.0", concepts=ItemList([Concept(id="FREQ")])) ref = cs.ref_to("FREQ") ref() # Concept(id="FREQ", ...)
Concept ¶
Bases: ParsesUrn
The semantic meaning of a DSD component (e.g. "Frequency", "Reference Area").
urn_for
classmethod
¶
Build an item-level URN for a Concept within a ConceptScheme.
Hides the SDMX package/class internals. Maps the caller's intuitive
scheme / id parameters to the URN's id / item_id fields.
Example
Concept.urn_for(agency="SDMX", scheme="CROSS_DOMAIN_CONCEPTS", id="FREQ")
urn:sdmx:org.sdmx.infomodel.conceptscheme.Concept=SDMX:CROSS_DOMAIN_CONCEPTS(latest).FREQ¶
ISOConceptReference ¶
A concept's counterpart in an ISO 11179 registry — SDMX's ISOConceptReference (#422).
Three plain strings, as the schemas spell them: a reference out of SDMX,
so not a Ref, which names SDMX artefacts. sdmx1 models it the same way.
Categories¶
CategoryScheme ¶
Bases: MaintainableArtefact, PathNavigation[Category]
A hierarchical classification scheme.
ref_to ¶
Build a resolved Ref to a category within this scheme, by path.
A nested category is named by its dotted path, which is also the item
part of its URN: cs.ref_to("ECON.GDP") gives
Category=AG:CS(1.0).ECON.GDP.
The URN is built from this scheme's agency, id and version, so it
cannot name a parent that does not exist — which is what six hand-written
keyword arguments to SdmxUrn.make invite, and what
validate_references now correctly reports as broken (#407).
Raises:
| Type | Description |
|---|---|
KeyError
|
If no category has this path in this scheme. |
Example
Categorisation(..., target=cs.ref_to("ECON.GDP"))
add ¶
Return a new CategoryScheme with categories added.
Accepts a single Category or a list of Categories.
drop ¶
Return a new CategoryScheme with the given category(ies) removed.
Accepts a single id string or a list of id strings. Raises KeyError if any id is not present.
Category ¶
Bases: ParsesUrn
A node in a classification hierarchy.
Categorisation ¶
Constraints¶
DataConstraint ¶
Bases: MaintainableArtefact
A constraint that limits the valid key combinations for a dataflow.
allows ¶
Return True if key satisfies all CubeRegions in this constraint.
availability_misfits ¶
Why this constraint has no SDMX 3.1 AvailabilityConstraint form; empty when it has one.
3.1's availability is one included region of one structure, flow or agreement, and nothing else: no role (it is Actual by being what it is), no key set, no other attachment.
Returns:
| Type | Description |
|---|---|
list[str]
|
One phrase per reason, for a refusal to name. |
to_availability_constraint ¶
This Actual constraint as SDMX 3.1 states availability (#421).
3.1 removed the role from DataConstraint and gave Actual content an
artefact of its own, which has no identity: no id, agency, version,
name, description, validity, uri or links. Those are lost, and what
reads back is an AvailabilityConstraint. The attachment, region and
annotations are kept.
Raises:
| Type | Description |
|---|---|
ValueError
|
If the constraint does not fit; |
AvailabilityConstraint ¶
SDMX 3.1's statement of what data exists for one structure, flow or agreement.
3.1 split Actual content out of DataConstraint into this. It is not a
maintainable artefact: AvailabilityConstraintType extends
AnnotableType, with no id, agency, version or name. It is the answer to
an availability query — one included cube region, and optionally how many
series and observations that query would return — rather than an artefact
a registry governs. So it is held in StructureMessage.availability_constraints
and in no artefact list, has no URN, and is neither stored nor written to
any target but SDMX-ML 3.1; the others leave it out (formats.gaps.dropped).
A 3.0 or 2.1 DataConstraint with the Actual role is written to 3.1 as one
when it fits (DataConstraint.to_availability_constraint).
Attributes:
| Name | Type | Description |
|---|---|---|
attachment |
ConstraintAttachment
|
The one data structure, dataflow or provision agreement. |
region |
CubeRegion
|
The included cube region with data. |
series_count |
int | None
|
How many series the availability query would return. |
obs_count |
int | None
|
How many observations it would return. |
annotations |
Annotations
|
The constraint's annotations. |
target
property
¶
What it states availability for, when the attachment names exactly one thing.
AttachmentConstraint ¶
Bases: MaintainableArtefact
An SDMX-ML 2.1 attachment constraint.
A 2.1-only artefact that pairs a set of data keys with the artefacts to
which they are attached. Sibling of DataConstraint inside
<str:Constraints>.
SDMX 3.0 did not delete the concept — it unified it, folding both of
2.1's constraint artefacts into one DataConstraint with a role. So
the 3.x and SDMX-JSON writers convert rather than refuse; see
to_data_constraint.
to_data_constraint ¶
The SDMX 3.0 form of this constraint.
Every field of an AttachmentConstraint is also a field of a
DataConstraint — attachment, key set, identity, validity — so the
conversion loses nothing. The 3.x and SDMX-JSON writers apply it
instead of refusing, which is what the old refusal advised ("use a
DataConstraint") and could not be done until DataConstraint gained
key sets (#241).
The return trip is not exact. A 3.0 DataConstraint carrying only a key set could have been either 2.1 artefact, and the document does not record which, so writing it back as 2.1 yields a ContentConstraint. The content survives; the artefact kind does not.
That asymmetry is asserted by _CONVERTED_ON_WRITE in the round-trip
properties, and belongs to the cross-version harness (#246) rather
than to KNOWN_GAPS — the latter records a field lost on one target,
where this is an artefact kind that changes across a version hop
(#244).
Returns:
| Type | Description |
|---|---|
DataConstraint
|
An equivalent |
ConstraintType ¶
Bases: Enum
What kind of statement a DataConstraint makes.
Per ContentConstraintTypeCodeType (2.1) / ConstraintRoleType
(3.0 / 3.1). The 2.1 element <str:ContentConstraint type=...>
maps to the same two values as the 3.0+ <str:DataConstraint role=...>.
parse
classmethod
¶
Parse an SDMX constraint-type string into the enum.
Returns ALLOWED for None (matches the historical default
when a constraint omits the attribute). An already-typed value
passes through.
ConstraintAttachment ¶
What a constraint is attached to.
SDMX makes this a choice, in every format: one data provider, or data
sets, or simple data sources, or one or more artefacts of a single kind
— data structures, dataflows or provision agreements — which may carry
queryable data sources beside them. The fields hold each alternative;
validate refuses more than one set at once (#422). Until #422 each kind
held one artefact, and a reader kept the first.
Attributes:
| Name | Type | Description |
|---|---|---|
data_provider |
Ref[object] | None
|
The provider whose data is constrained. |
data_sets |
tuple[DataSetReference, ...]
|
SDMX-ML 2.1's data sets; 3.x has none. |
simple_data_sources |
tuple[str, ...]
|
URLs of SDMX-ML data messages. SDMX-ML 3.1 has none. |
data_structures |
tuple[Ref[object], ...]
|
Data structures the constraint applies to. |
dataflows |
tuple[Ref[object], ...]
|
Dataflows the constraint applies to. |
provision_agreements |
tuple[Ref[object], ...]
|
Provision agreements the constraint applies to. |
queryable_data_sources |
tuple[QueryableDataSource, ...]
|
Services answering for the structures, flows or agreements. SDMX-ML 3.1 has none. |
is_empty
property
¶
True when the attachment names nothing.
The default DataConstraint.attachment is an empty one, so this is a
normal state for a partially-assembled constraint. It matters on the
wire because <ConstraintAttachment> is minOccurs="0" in every
SDMX-ML version and requires one of its target elements when
present — so an empty attachment must be omitted rather than emitted
as a bare wrapper, which is a document the schema rejects (#229).
alternatives
property
¶
The names of the alternatives this attachment sets — one when well-formed.
Queryable data sources are not an alternative of their own: they sit beside the structures, flows or agreements they answer for.
CubeRegion ¶
A region in the hypercube: per-dimension selections, and per-component ones.
allows ¶
Return True if key is permitted by this region.
A dimension the key does not mention is not checked. A time range
constrains only when the key's value and the range's bounds all read
as ISO dates or date-times; a reporting period (2020-Q1) is not
interpreted, so the range does not exclude it. Components are not
checked: a key names dimensions.
A value set with include=False turns its own test around: the key
is inside the region on that dimension when its value is not one
of them (#422).
A value's RegionValue.cascade is not expanded — selecting a code's
descendants needs the codelist, which a key does not carry — and its
validity, and the set's, are not checked against any date.
KeyValue ¶
A dimension constraint: the allowed values for one dimension, or a time range.
SDMX's CubeRegionKeyType is a choice of Value+ or one TimeRange, so
a key holds exactly one of the two. Until #433 the time range had nowhere
to go and a key read from OECD came back with neither, which every
SDMX-ML version rejects.
ComponentValueSet ¶
The values allowed for one non-dimension component in a cube region.
SDMX-ML 2.1 <Attribute>, 3.0 <Component>, SDMX-JSON components: an
attribute's or measure's values, or a time range — and, unlike a
KeyValue, possibly neither.
RegionValue ¶
One value a cube region's value set lists, with what qualifies it (#422).
SDMX-ML 3.x's SimpleComponentValueType (2.1's SimpleValueType), and
SDMX-JSON's object form of a region value. A bare string is still accepted
wherever a value set is built, and stands for a RegionValue with nothing
set — the common case, and what every qualifier defaults to.
Attributes:
| Name | Type | Description |
|---|---|---|
value |
str
|
The value — usually a code id. |
cascade |
CascadeSelection
|
Whether the code's descendants are selected too: by default
the code alone, the |
lang |
str | None
|
The language of a multilingual text value. SDMX 3.x prohibits it on a dimension's value, so only a component's may carry one. |
valid_from |
str | None
|
When the value starts to apply, as written
( |
valid_to |
str | None
|
When the value stops applying, likewise. |
TimeRange ¶
A time selection in a cube region: before, after, or between periods.
SDMX's TimeRangeValueType is a choice of BeforePeriod, AfterPeriod,
or StartPeriod and EndPeriod, so the three shapes map one to one:
only end is "before", only start is "after", both is a range. One
with neither has no form in any format and is refused as it is built.
TimeBound ¶
One end of a TimeRange: an observational time period, as written.
period is the wire text, not a datetime: SDMX types it
ObservationalTimePeriodType, which takes reporting periods (2020-Q1,
2020-A1) that have no faithful datetime, and parsing would also
decide a time zone the source never gave (#440).
DataKeySet ¶
Whole observation keys a constraint includes, or excludes (#479).
SDMX's DataKeySet: isIncluded is required, and says whether its
keys are the ones allowed or the ones ruled out. The flag belongs to the
set, not to a key, so the set is a class of its own, as in the
information model, pysdmx (KeySet) and sdmx1 (DataKeySet).
Example
DataKeySet(is_included=False, keys=[DataKey(values=[("FREQ", "A"), ("REF_AREA", "XX")])])
DataKey ¶
A single full or partial data key: one value per dimension.
Differs from KeyValue (sets of allowed values per dimension):
a DataKey enumerates one concrete key. Dimensions omitted from
values are treated as wildcarded per the SDMX 2.1 spec.
DataSetReference ¶
A data set, named by its provider and id — SDMX-ML 2.1's SetReferenceType (#422).
2.1 only: 3.x attaches a constraint to no data set.
QueryableDataSource ¶
A data source that answers queries — SDMX QueryableDataSourceType (#422).
Attributes:
| Name | Type | Description |
|---|---|---|
data_url |
str
|
Where the data is queried ( |
is_rest_datasource |
bool
|
Whether it is a REST service ( |
is_web_service_datasource |
bool
|
Whether it is a SOAP service
( |
wsdl_url |
str | None
|
Its WSDL description ( |
wadl_url |
str | None
|
Its WADL description ( |
ReferencePeriod ¶
Time period bounds for a constraint's data coverage.
ReleaseCalendar ¶
Release schedule information for a constraint.
All fields are ISO 8601 duration strings (e.g. "P1M", "P1M15D").
Provision agreements¶
ProvisionAgreement ¶
Bases: MaintainableArtefact
Binds a data provider to a dataflow.
Example
pa.dataflow() # Dataflow (if resolved) pa.dataflow().structure() # DataStructure pa.dataflow().structure.dimensions # ItemList[Dimension]
MetadataProvisionAgreement ¶
Bases: MaintainableArtefact
Binds a metadata provider to a metadataflow (SDMX 3.0).
Example
mpa.metadataflow() # Metadataflow (if resolved) mpa.metadataflow().structure() # MetadataStructure
Organisation schemes¶
AgencyScheme ¶
Bases: MaintainableArtefact
Maintainable container for Agency items.
ref_to ¶
Build a resolved Ref to a agency within this scheme.
The URN is built from this scheme's agency, id and version, so it
cannot name a parent that does not exist — which is what six hand-written
keyword arguments to SdmxUrn.make invite, and what
validate_references now correctly reports as broken (#407).
Raises:
| Type | Description |
|---|---|
KeyError
|
If no agency with this id is in this scheme. |
Example
scheme.ref_to("ESTAT")
add ¶
Return a new AgencyScheme with agencies added.
Accepts a single Agency or a list of Agencies.
drop ¶
Return a new AgencyScheme with the given agency(ies) removed.
Accepts a single id string or a list of id strings. Raises KeyError if any id is not present.
Agency ¶
Bases: ParsesUrn
An organisation that maintains SDMX artefacts.
DataProviderScheme ¶
Bases: MaintainableArtefact
Maintainable container for DataProvider items.
ref_to ¶
Build a resolved Ref to a provider within this scheme.
The URN is built from this scheme's agency, id and version, so it
cannot name a parent that does not exist — which is what six hand-written
keyword arguments to SdmxUrn.make invite, and what
validate_references now correctly reports as broken (#407).
Raises:
| Type | Description |
|---|---|
KeyError
|
If no provider with this id is in this scheme. |
Example
scheme.ref_to("ECB")
add ¶
Return a new DataProviderScheme with providers added.
Accepts a single DataProvider or a list of DataProviders.
drop ¶
Return a new DataProviderScheme with the given provider(s) removed.
Accepts a single id string or a list of id strings. Raises KeyError if any id is not present.
DataProvider ¶
Bases: ParsesUrn
An organisation that provides data under a ProvisionAgreement.
urn_for
classmethod
¶
urn_for(
*,
agency: str,
id: str,
scheme: str = "DATA_PROVIDERS",
version: str = "latest",
) -> SdmxUrn[DataProvider]
Build an item-level URN for a DataProvider within its scheme.
Hides the package/class internals. Building one by concatenating
.id onto the scheme's urn produces a urn whose class is
DataProviderScheme, which then resolves against nothing —
an easy mistake, and a silent one (#315).
scheme defaults to "DATA_PROVIDERS" because the XSD pins the
scheme's id to that one literal — see
validation._FIXED_SCHEME_IDS — so the caller almost never has a
choice to make.
Example
DataProvider.urn_for(agency="SDMX", id="P1")
DataConsumerScheme ¶
Bases: MaintainableArtefact
Maintainable container for DataConsumer items (#469). Its id is fixed to DATA_CONSUMERS.
ref_to ¶
Build a resolved Ref to a consumer within this scheme.
The URN is built from this scheme's agency, id and version, so it cannot name a parent that does not exist (#407).
Raises:
| Type | Description |
|---|---|
KeyError
|
If no consumer with this id is in this scheme. |
add ¶
Return a new DataConsumerScheme with consumers added.
Accepts a single DataConsumer or a list of them.
drop ¶
Return a new DataConsumerScheme with the given consumer(s) removed.
Accepts a single id string or a list of id strings. Raises KeyError if any id is not present.
DataConsumer ¶
Bases: ParsesUrn
An organisation that consumes data: the receiving counterpart of DataProvider (#469).
Identical in shape to DataProvider, as both are Organisation in the IM.
Held in a DataConsumerScheme, whose id every format pins to
DATA_CONSUMERS.
urn_for
classmethod
¶
urn_for(
*,
agency: str,
id: str,
scheme: str = "DATA_CONSUMERS",
version: str = "latest",
) -> SdmxUrn[DataConsumer]
Build an item-level URN for a DataConsumer within its scheme.
The same trap as DataProvider.urn_for: concatenating .id onto the
scheme's urn yields class DataConsumerScheme, which resolves
against nothing (#315).
Example
DataConsumer.urn_for(agency="SDMX", id="C1")
MetadataProviderScheme ¶
Bases: MaintainableArtefact
Maintainable container for MetadataProvider items — #315.
Mirrors DataProviderScheme. Until this existed,
MetadataProvisionAgreement.metadata_provider was a Ref[object] and the
stored metadata_provider_urn column pointed at a relation that did not
exist: populated, and unresolvable by anything.
ref_to ¶
Build a resolved Ref to a provider within this scheme.
The URN is built from this scheme's agency, id and version, so it
cannot name a parent that does not exist — which is what six hand-written
keyword arguments to SdmxUrn.make invite, and what
validate_references now correctly reports as broken (#407).
Raises:
| Type | Description |
|---|---|
KeyError
|
If no provider with this id is in this scheme. |
Example
scheme.ref_to("ECB")
add ¶
Return a new MetadataProviderScheme with providers added.
Accepts a single MetadataProvider or a list of them.
drop ¶
Return a new MetadataProviderScheme with the given provider(s) removed.
Accepts a single id string or a list of id strings. Raises KeyError if any id is not present.
MetadataProvider ¶
Bases: ParsesUrn
An organisation that provides reference metadata under an MPA — #315.
The metadata twin of DataProvider, and identical in shape: both are
Organisation in the IM, so both carry contacts.
SDMX 3.0 surface. 2.1 has no MetadataProviderScheme at all — verified
against the vendored schemas, where it appears in the 3.0 and 3.1
SDMXStructureOrganisation.xsd and in neither 2.1 file — so writing 2.1
refuses the scheme rather than dropping it (validation._UNWRITABLE).
urn_for
classmethod
¶
urn_for(
*,
agency: str,
id: str,
scheme: str = "METADATA_PROVIDERS",
version: str = "latest",
) -> SdmxUrn[MetadataProvider]
Build an item-level URN for a MetadataProvider within its scheme.
The metadata twin. Same trap it exists to avoid: concatenating .id
onto the scheme's urn yields class MetadataProviderScheme,
and the join then matches no rows without erroring (#315).
scheme defaults to "METADATA_PROVIDERS" because the XSD pins the
scheme's id to that one literal — see
validation._FIXED_SCHEME_IDS — so the caller almost never has a
choice to make.
Example
MetadataProvider.urn_for(agency="SDMX", id="P1")
OrganisationUnitScheme ¶
Bases: MaintainableArtefact
Maintainable container for OrganisationUnit items, hierarchical through OrganisationUnit.parent_id (#469).
ref_to ¶
Build a resolved Ref to an organisation unit within this scheme.
The URN is built from this scheme's agency, id and version, so it cannot name a parent that does not exist (#407).
Raises:
| Type | Description |
|---|---|
KeyError
|
If no organisation unit with this id is in this scheme. |
add ¶
Return a new OrganisationUnitScheme with organisation_units added.
Accepts a single OrganisationUnit or a list of them.
drop ¶
Return a new OrganisationUnitScheme with the given organisation unit(s) removed.
Accepts a single id string or a list of id strings. Raises KeyError if any id is not present.
OrganisationUnit ¶
Bases: ParsesUrn
A unit within an organisation, such as a department or division (#469).
Hierarchical within its scheme: parent_id names another unit of the same
OrganisationUnitScheme, as Code.parent_id names a code. That is how
every format writes it: 2.1 as <Parent><Ref id>, 3.x as <Parent> text,
and SDMX-JSON as parent.
Contact ¶
Contact information for an agency or data provider.
Structure maps¶
StructureSet ¶
Bases: MaintainableArtefact
A 2.1 container grouping related maps.
SDMX-ML 2.1 nests every map inside a <str:StructureSet>, which is a
maintainable in its own right: it has an id, an agency, a version, validity
dates and a name, and it asserts that the maps inside it belong together.
SDMX 3.0 removed it and promoted each map kind to a top-level container.
sdmxlib follows 3.0 for the maps themselves — they remain top-level artefacts — and models the container as a peer that references them. That keeps one copy of the grouping rather than denormalising the container's identity onto each map, where nothing would stop the copies disagreeing.
Written only to SDMX-ML 2.1. The 3.0, 3.1 and SDMX-JSON targets have no
such concept, so they leave it out and write its maps
(formats.gaps.dropped, #421); a 2.1 reader names each map for it
(model._container.member_id).
Example
sl.StructureSet( id="SS_TEST", maintainer="TEST", version="1.0", name={"en": "Test Structure Set"}, maps=[sl.Ref.to(codelist_map)], )
StructureMap ¶
Bases: MaintainableArtefact
Maps components of one structure to components of another.
Supersedes the 2.1 DataStructureMap and DataflowMap. Source and
target may be either DataStructure or Dataflow refs (3.0's
StructureOrUsageReferenceType). The 2.1 serializer dispatches on
the source ref class to emit either <str:DataStructureMap> or
<str:DataflowMap>.
ComponentMap ¶
Single component-level mapping inside a StructureMap.
SDMX 3.x lets one map several source components to several target ones (#422); 2.1's is one to one.
FixedValueMap ¶
A fixed value for one source or target component — SDMX 3.x's FixedValueMap (#422).
The schema's choice of Source or Target, so exactly one of
source_id / target_id.
DatePatternMap ¶
Bases: ParsesUrn
Maps a formatted date to a time period — SDMX 3.x's DatePatternMap (#422).
The source components hold dates written in source_pattern for
locale. Its other fields are an EpochMap's, which shares the schema's
DateMapType.
EpochMap ¶
Bases: ParsesUrn
Maps an epoch count to a time period — SDMX 3.x's EpochMap (#422).
The source components hold a number of epoch_periods since
base_period; the target, a time period. Frequency comes from a
frequency_dimension (whose codes the mapped_frequencies —
FrequencyFormatMapping ids — say how to format), or is the one
target_frequency_id: the schema's choice, so exactly one of the two.
EpochPeriod ¶
Bases: Enum
The unit an EpochMap counts in — SDMX's EpochPeriodType.
FrequencyFormatMapping ¶
Bases: ParsesUrn
How a frequency's periods are written — SDMX 3.x's FrequencyFormatMapping (#422).
A date map's mapped_frequencies name these by id.
ResolvePeriod ¶
Bases: Enum
Where in a lower-frequency period a date map resolves — SDMX's ResolvePeriodType.
RepresentationMap ¶
Bases: MaintainableArtefact
Maps representations to others.
Supersedes the 2.1 CodelistMap. Each endpoint is either a codelist
Ref or a MapDataType literal (#131), and there may be several on
each side (#422): a mapping then holds one source value per source and one
target value per target. The 2.1 serializer round-trips the one-codelist-
to-one-codelist case as <str:CodelistMap> with <str:CodeMap>
entries.
The 3.x-only shapes — several endpoints, a dataType endpoint, and regex, index or N:M mappings — have no 2.1 form and are refused there rather than dropped. They used to be written as plain literal pairs, which restated a regex pattern as a code id: a document asserting a mapping the source never made (#242).
translate ¶
Return the target value for a source value, or None if unmapped.
Only reads 1:1 mappings whose source is a literal — a regex, index or N:M mapping needs the caller to apply its semantics themselves.
RepresentationMapping ¶
One mapping of a RepresentationMap: source values → target values.
SDMX 3.x's RepresentationMapping (ValueMappingType). It is N:M:
one source value per source endpoint of the map, one target value per
target endpoint. No target maps the source values to nothing.
SourceValue ¶
One source value of a RepresentationMapping — SDMX's MappedValueType.
A literal unless is_regex makes value a regular expression, or
start_index / end_index take the substring it is compared with. The
indexes are the wire's integers, passed through without reindexing.
ToValueType ¶
Bases: Enum
What of a source value a 2.1 component map carries into an uncoded target — ToValueTypeType.
ItemMap ¶
Single item-level mapping: source item id → target item id.
Used by ConceptSchemeMap, CategorySchemeMap. Matches the SDMX
3.0 SingleValueMappingType (id-to-id, with its validity dates; the
1:N variants are not modelled).
ConceptSchemeMap ¶
CategorySchemeMap ¶
OrganisationSchemeMap ¶
Bases: MaintainableArtefact
Maps one organisation scheme to another, organisation by organisation (#448).
Its ends may be any organisation scheme — agencies, data or metadata
providers, or the data consumer and organisation unit schemes sdmxlib
does not model, whose references then stay unresolved. SDMX calls each
pair an OrganisationMap; it is an ItemMap as for the other two
scheme maps.
Annotations and links¶
Annotation ¶
A single SDMX annotation attached to an artefact or item.
Annotations ¶
Bases: Sequence[Annotation]
Immutable collection of Annotation with type-based querying.
Example
"DEPRECATED" in annotations # checks annotation type annotations.by_type("CONTACT") # list[Annotation] annotations.first("CONTACT") # Annotation | None annotations.types # list[str]
first ¶
First annotation with the given type, or None.
AnnotationURL ¶
One external resource an annotation points at, optionally in one language.
SDMX 3.x's AnnotationURLType: a URI with an optional xml:lang
naming the language of the resource, absent when it is not localised.
SDMX-JSON spells it a link with href and hreflang, and gives
that link a urn and a type besides, which SDMX-ML's
AnnotationURLType has no place for: an SDMX-ML write drops them
(declared in formats/gaps.py).
Attributes:
| Name | Type | Description |
|---|---|---|
url |
str
|
The resource's URL. |
lang |
str | None
|
The language of the resource, None when it is not localised. |
urn |
str | None
|
The SDMX registry URN of what the link points at — SDMX-JSON's alone (#422). |
type |
str | None
|
The kind of resource, such as |
Link ¶
One <com:Link>: a resource an identifiable artefact points at.
SDMX 3.x's LinkType, which IdentifiableType allows any number of —
"if there is reference metadata associated with a structure, a link to the
metadata report can be dynamically inserted in the structure metadata".
SDMX-ML 2.1 has no such element, so a 2.1 write drops links (declared in
formats/gaps.py). SDMX-JSON spells one a links entry, with url as
href.
Distinct from @uri, which is the artefact's single descriptive
pointer, and from Ref: urn is an xs:anyURI, "an SDMX registry
urn of the object being linked (if applicable)", so it is held as text
and is not an edge of the artefact graph.
Attributes:
| Name | Type | Description |
|---|---|---|
rel |
str
|
The type of object linked to — |
url |
str
|
The linked resource's URL. |
urn |
str | None
|
The linked object's SDMX registry URN, where it has one. |
type |
str | None
|
The kind of link, such as |
hreflang |
str | None
|
The language of the linked resource. SDMX-JSON's alone:
SDMX-ML's |
Shared bases¶
MaintainableArtefact ¶
Bases: ParsesUrn
Mixin for top-level SDMX artefacts that have a globally unique URN.
Subclasses must declare sdmx_package and sdmx_class as
class variables. The .urn_for() classmethod then constructs a typed
SdmxUrn[Self] without repeating the logic in every class.
NOTE: Subclasses must redeclare the standard fields (id, maintainer,
version, name, description, annotations) because slotted
dataclasses cannot inherit slots from a non-dataclass parent.
Python requires that all __slots__ are declared on the class that
uses them.
Example
Codelist.urn_for(agency="SDMX", id="CL_FREQ") # SdmxUrn[Codelist] DataStructure.urn_for(agency="ESTAT", id="CPI", version="2.0")
is_external_reference
property
¶
@isExternalReference: this is a stub of an artefact held elsewhere.
service_url
property
¶
@serviceURL: a registry or REST service holding the full artefact.
structure_url
property
¶
@structureURL: an SDMX-ML document holding the full artefact.
is_partial_language
property
¶
SDMX 3.1's @isPartialLanguage: the text is not in every language it uses.
is_final
property
¶
Whether this artefact is final, read from its version as SDMX 3.0 does.
Derived rather than stored (#420, #424): 3.0 dropped isFinal and made
finality a fact about the version, so 1.0.0 is final and 1.0 and
1.0.0-draft are not. A 2.1 document's isFinal, where it says
otherwise, is kept on legacy_is_final by the classes 2.1 lets carry
it, and only the 2.1 writer reads it — the convention FMR follows.
agency_id
property
¶
Maintainer agency identifier. Derived from the maintainer field.
urn
property
¶
This artefact's canonical URN.
A property because Ref[T].urn is one, and a reader holding either
should not have to remember which. It used to be the builder — a
classmethod — so flow.urn on an instance gave back a bound method,
and ref.py's own docstring offered Ref.unresolved(cl_freq.urn) as
an example, which raised. The builder is urn_for.
Recomputed, not cached. This was because artefacts were mutable and
a stored URN went stale the moment someone assigned version; they are
frozen since #492, so a cache would now be sound. It is still not one
until a profile names this line — see docs/design/caching.md.
Typed SdmxUrn[Self], which artefact_urn() cannot be: that takes a
SupportsUrn and has no concrete type to return, so it hands back
SdmxUrn[object] (#399). This keeps the type, so
Ref.unresolved(flow.urn) is a Ref[Dataflow].
urn_for
classmethod
¶
A typed SdmxUrn for this artefact class, from coordinates.
The building half of what urn used to be both of. Named apart from
the property because the two differ in more than receiver: this
defaults version to "latest", which is right for an identity you
are looking up, while an artefact you hold has whatever version it
has.
parse_rest_ref
classmethod
¶
Parse an SDMX REST comma-separated artefact reference.
SDMX REST URLs identify a maintainable artefact as
AGENCY,ID,VERSION. Symmetric with urn.
Example
Dataflow.parse_rest_ref("ESTAT,STS_INPR_M,1.0")
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ref
|
str
|
Comma-separated |
required |
Returns:
| Type | Description |
|---|---|
SdmxUrn[Self]
|
A typed |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
InternationalString ¶
A multilingual string where language access is always explicit.
Stores key-value pairs of language code → text. Immutable after
construction (no __setitem__).
Example
s = InternationalString(en="Annual", fr="Annuel") s["en"] # "Annual" s.get("de", "en") # "Annual" (fallback) "fr" in s # True s.langs # ["en", "fr"]
of
classmethod
¶
Build from an existing dict.
Returns the shared EMPTY for an empty mapping — the common case on a
real bundle, and the one that used to allocate (#377). Only for this
exact class: a subclass gets its own fresh instance, since EMPTY is
an InternationalString and returning it would be a type error the
annotation cannot see.
get ¶
Return the first available language in priority order, or None.
as_dict ¶
The language→text mapping.
Fast path for serialisers that need the whole map: a direct dict
copy avoids the items() iterator + dict(generator) overhead on
hot per-item paths (annotation/name/description emit).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
copy
|
bool
|
When True (default) return a shallow copy that is safe to
mutate. When False return the backing map directly — no copy —
for the serialise hot path, where the result feeds a transient,
immediately-discarded output structure and is never mutated.
Callers passing |
True
|
emit_pair ¶
(best-match string, full lang→text map) for the SDMX-JSON split.
The serialise hot path's single-call primitive: collapses the
empty-check + best-match pick + backing-map fetch into one method with
direct field access, avoiding the separate __bool__ / as_dict /
iter/next calls that dominate emit_istr at scale. Returns
(None, None) when empty so omit_defaults drops both wire fields.
The map is the backing store (not copied) — read-only for the caller.
Reference metadata (MSD)¶
MetadataStructure ¶
Bases: MaintainableArtefact, PathNavigation[MetadataAttribute]
The structural definition of a reference-metadata schema (SDMX MSD).
Holds a tree of MetadataAttribute components. Each MetadataAttribute declares one reportable slot — a concept, a representation (codelist or facet), and cardinality bounds.
Example
msd = sl.MetadataStructure( id="MSD_FOOTNOTES", maintainer="STATCAN", attributes=[sl.MetadataAttribute(id="FOOTNOTE_TEXT", ...)], ) msd.attributes["FOOTNOTE_TEXT"] # a root attribute by id msd["CONTACT.EMAIL"] # any attribute by path
ref_to ¶
Build a resolved Ref to a metadata attribute within this structure, by path.
A nested attribute is named by its dotted path, which is also the item
part of its URN: msd.ref_to("CONTACT.EMAIL").
The URN is built from this scheme's agency, id and version, so it
cannot name a parent that does not exist — which is what six hand-written
keyword arguments to SdmxUrn.make invite, and what
validate_references now correctly reports as broken (#407).
Raises:
| Type | Description |
|---|---|
KeyError
|
If no metadata attribute has this path in this structure. |
Example
msd.ref_to("CONTACT.EMAIL")
add ¶
Return a new MetadataStructure with attributes added.
drop ¶
Return a new MetadataStructure with the given attribute(s) removed.
Searches only the top-level attributes by id (children are not flattened). Raises KeyError if any id is not present.
MetadataAttribute ¶
Bases: ConceptIdentityMixin, RepresentationMixin
A structural component of a MetadataStructure.
Defines one slot in the metadata schema — what can be reported, with which representation (codelist or text), and how often.
Children form a recursive tree (no parent back-reference). Cardinality is expressed via min_occurs and max_occurs (None = unbounded).
Example
sl.MetadataAttribute( id="FOOTNOTE_TEXT", concept=cs.ref_to("FOOTNOTE_TEXT"), representation=Facet(type=DataType.String), min_occurs=0, max_occurs=1, )
Metadataflow ¶
Bases: MaintainableArtefact
A published reference-metadata product backed by a MetadataStructure.
Mirrors Dataflow: binds a maintained id/version to one MSD so that
metadata sets can reference the flow rather than the bare structure.
Per the SDMX 3.0 XSD, a Metadataflow also carries one or more
targets — URN references to the identifiable artefacts the
reported metadata applies to. URNs may include wildcards per
WildcardUrnType so we model them as raw strings rather than
resolved Refs.
Example
mdf = sl.Metadataflow( id="MDF_FOOTNOTES", maintainer="STATCAN", structure=msd, targets=("urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=ESTAT:CPI(*)",), name={"en": "Footnotes flow"}, )
target_misfits ¶
Why with_derived_targets cannot give this flow targets, or nothing when it can (#421).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
available
|
Iterable[MetadataStructure]
|
Metadata structures to find this flow's in when its reference is not resolved — a message's own, say. |
()
|
Returns:
| Type | Description |
|---|---|
tuple[str, ...]
|
One reason per obstacle; empty when the targets can be derived. |
with_derived_targets ¶
This flow with the targets its 2.1 metadata structure implies (#421).
SDMX 3.x requires a metadataflow to name at least one target — what
the metadata reported against it may describe. SDMX-ML 2.1 put that
in the metadata structure instead, as MetadataTarget s, so a flow
read from 2.1 has none. Each IdentifiableObjectTarget there names a
class of object and becomes the wildcard URN naming every object of
that class: Dataflow gives
urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=*:*(*). The
other target kinds — a data key, a data set, a reporting period, a
constraint — describe something within data, which no URN names, so
they contribute nothing.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
available
|
Iterable[MetadataStructure]
|
Metadata structures to find this flow's in when its reference is not resolved — a message's own, say. |
()
|
Returns:
| Type | Description |
|---|---|
Metadataflow
|
A new flow, or this one when it already has targets. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the targets cannot be derived; |
MetadataSet ¶
Bases: MaintainableArtefact
A populated reference-metadata payload describing a list of targets.
All attributes apply to all targets — there is no per-target scoping. This matches the SDMX 3.0 information model, which collapsed v2.1's MetadataReport layer.
The structure field references a Metadataflow or a
MetadataProvisionAgreement — the two structure-ref options the SDMX 3.0
model permits. (There is no direct-MSD reference; wrap an MSD in a
Metadataflow.)
Note that MetadataSet does NOT belong in a Structure message envelope —
it is carried in a GenericMetadata message (<mes:GenericMetadata> in
SDMX-ML 3.0; data.metadataSets keyed on the metadata-schema $schema
in SDMX-JSON 2.0). Use a MetadataMessage to serialise it.
Example
ms = sl.MetadataSet( id="FOOTNOTE_A", maintainer="STATCAN", structure=mdf, targets=[char_1_urn, char_5_urn, char_12_urn], attributes=[sl.ReportedAttribute(id="FOOTNOTE_TEXT", value={"en": "..."})], )
ReportedAttribute ¶
One populated metadata attribute within a MetadataSet.
The id field references a MetadataAttribute in the MSD. Either the value is set (leaf) or children is non-empty (presentational container). Repetition is handled by multiple sibling instances with the same id.
Example
sl.ReportedAttribute(id="FOOTNOTE_TEXT", value={"en": "..."}) sl.ReportedAttribute(id="FREQ", value="A") sl.ReportedAttribute(id="QUALITY", children=[ sl.ReportedAttribute(id="COVERAGE", value="full"), ])
ReportedAttributeValue ¶
MetadataSection ¶
A flattened, MSD-typed view of one reported metadata value.
The retrieval projection of a MetadataSet: one MetadataSection per
metadata_value row, with the leaf attribute's declared data type
(text_type, e.g. "String" / "XHTML") joined in from the
MSD. This is the sectioned input a chunker / search indexer wants —
path gives the dotted hierarchical location (with [n] sibling
suffixes preserved), attribute_id the leaf id, and value the
scalar or InternationalString content.
Produced by Registry.metadata_report.
Messages¶
What a read returns and a write takes. A StructureMessage carries definitions; a MetadataMessage carries populated reference metadata.
StructureMessage ¶
Wire-format envelope for SDMX structural definitions.
Carries Codelists, DSDs, MSDs, Metadataflows, etc. Cross-references within the message are auto-resolved by the reader.
Example
msg = parse(xml_bytes) dsd = msg.data_structures[0] cl = msg.codelists["SDMX:CL_FREQ(1.0)"]
refused
property
¶
Artefacts the reader refused, each with the reason (#435).
A document is read artefact by artefact: one that contradicts the
schema is recorded here and the rest are read. Empty for a message
built in code, or read from a document with nothing wrong in it.
parse() raises on a non-empty one unless called with
strict=False.
lost
property
¶
What the reader dropped from artefacts it built, per artefact and construct (#422).
SDMX 3.x carries constructs the model has no field for yet — a
com:Link, a DSD's MetadataAttributeUsage. The artefact is built
without them and each loss is recorded here, so a read is never
quietly smaller than its document. Empty for a message built in code.
parse(..., lossless=True) raises on a non-empty one.
An SDMX-JSON read works its losses out here, on first access, rather
than during the read (#461). Finding them walks the whole document, and
most callers never ask, so a read that walked eagerly cost them 40-90%
more. The price is memory: until lost is first read, the message
holds its source document's bytes, at most one copy, the size of what
was parsed (#473). Reading lost, or parsing with
parse(..., lossless=True), which reads it, releases them.
assumed
property
¶
Values the reader supplied where the document omitted something the schema requires (#474).
Beside refused (artefacts not built) and lost (content dropped), the
third account of a read: content filled in. IMF omits the required
AttributeRelationship on thousands of attribute usages, and the
reader takes each to be dataset-level, a tolerance chosen in #421; this
is where that choice shows. Empty for a message built in code.
parse(..., lossless=True) raises on a non-empty one.
header
property
¶
Who sent the message, to whom, when and what about; None when it has none (#480).
all_artefacts ¶
Flat list of every artefact in the message — all 24 slots.
Every slot, which it did not used to be: structure_sets and
hierarchical_codelists were absent, and the two callers of this
method want opposite things from it (#394).
Validation wants everything, and silently got 21 of 23 — so
validate_references never checked a StructureSet or a
HierarchicalCodelist, which between them carry four of the five
container-of-Ref fields in the model.
Storage wants only what it can persist, and this is not that list:
four of the 21 it already yielded have no ArtefactKind (#395), so
add_all(msg.all_artefacts()) refuses them. Completing the list makes
that six.
Completing it is still right, because the alternative was worse than a
refusal. Ingesting a message through parse(xml, registry=…) dropped
the two absent artefacts silently — a HierarchicalCodelist named
"Geographic hierarchy" simply ceased to exist, which is the name #239
added the container to stop losing. formats/gaps.py states the rule
this follows: dropping is the one that burns an author; a refusal is
visible, a drop is not.
tests/unit/test_domain_model.py compares this against the class's
own fields, so a new slot cannot be added without a decision. One is
left out on purpose: availability_constraints, SDMX 3.1's
availability statements, which have no identity and so are not
artefacts — nothing can store, reference or validate one by URN (#421).
containing
classmethod
¶
Wrap a single structural artefact in a fresh StructureMessage.
Dispatches on the artefact's runtime type to the matching
collection slot (Codelist → codelists, Dataflow →
dataflows, …) using the artefact class's msg_field
ClassVar. Saves REST handlers from a long isinstance chain.
Example
return StructureMessage.containing(dataflow)
equivalent to: StructureMessage(dataflows=[dataflow])¶
A lazy artefact is materialised first. LocalRegistry.get returns a
LazyCodelist, so a /structure handler reading from a local store
hands one straight to this method — and a message holding a lazy
object would go stale the moment its registry closed, which is after
the handler returns and before the bytes are written.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
artefact
|
object
|
A maintainable artefact whose type maps to one of this message's slots. |
required |
Raises:
| Type | Description |
|---|---|
TypeError
|
If the artefact's class has no |
MetadataMessage ¶
Wire-format envelope for populated reference metadata.
Carries MetadataSet instances. Maps to <mes:GenericMetadata> in
SDMX-ML 3.0 and to the metadata $schema envelope in SDMX-JSON 2.0.
Example
msg = parse(metadata_xml_bytes) for ms in msg.metadata_sets: for target in ms.targets: print(target.urn, ms.attributes.ids())
Header ¶
A structure message's header (StructureHeaderType).
Attributes:
| Name | Type | Description |
|---|---|---|
id |
str
|
The message's id, an SDMX |
prepared |
datetime
|
When the message was prepared, tagged UTC as every model
datetime is. A document's |
sender |
Sender
|
Who sent it. |
test |
bool
|
Whether it is a test message. |
receivers |
tuple[Party, ...]
|
Who it is for. |
name |
InternationalString
|
The message's names, by language. |
source |
InternationalString
|
Where its content comes from, by language; SDMX-JSON has no place for it. |
Example
header = Header(id="REQ1", prepared=datetime(2026, 10, 20, tzinfo=UTC), sender=Sender(id="ESTAT")) msg = StructureMessage(header=header, codelists=[...])
MetadataHeader ¶
Bases: Header
A reference-metadata message's header (GenericMetadataHeaderType): a Header and four more.
SDMX-JSON has a place for none of the four.
Attributes:
| Name | Type | Description |
|---|---|---|
metadata_provider |
Ref[MetadataProvider] | None
|
The provider the metadata sets come from. |
data_set_action |
ActionType | None
|
What the sets do to what the receiver holds. |
data_set_ids |
tuple[str, ...]
|
The sets' ids. |
extracted |
datetime | None
|
When the metadata was extracted from its source, tagged UTC. |
Party ¶
A party to a message, a receiver or the sender (PartyType).
Attributes:
| Name | Type | Description |
|---|---|---|
id |
str
|
The party's id, an SDMX |
name |
InternationalString
|
Its names, by language. |
contacts |
tuple[Contact, ...]
|
How to reach it. SDMX-ML's message contact has no id, which SDMX-JSON's has; SDMX-ML writes one without it. |
Sender ¶
Bases: Party
The party that sent a message (SenderType): a Party and its time zone.
Attributes:
| Name | Type | Description |
|---|---|---|
timezone |
str | None
|
|
ActionType ¶
Bases: Enum
What a message's data or metadata sets do to what the receiver holds (ActionType).
Collections¶
ItemList ¶
Bases: Sequence[T]
Ordered collection of child artefacts (codes, dimensions, etc.).
Items have an id attribute (T is bound to SupportsId). Lookup is by
bare id.
Example
dims["FREQ"] # by id dims.filter(lambda d: d.is_enumerated) # lambda predicate dims.filter(item("is_enumerated")) # expression predicate dims.sort_by(lambda d: d.id) # sort → new ItemList dims.ids() # ["FREQ", "REF_AREA", ...]
filter ¶
Return new ItemList with items satisfying predicate.
Accepts a lambda or an item() expression: dims.filter(lambda d: d.id == "FREQ") dims.filter(item("id") == "FREQ") dims.filter(item("is_enumerated")) dims.filter(~item("is_enumerated"))
first_where ¶
Return first item satisfying the predicate, or None.
partition ¶
Split into (matching, non-matching) ItemLists.
sort_by ¶
Return new ItemList sorted by key function.
to_dict ¶
Convert to dict keyed by attribute name or key function.
A key function recovers its return type: to_dict(lambda c: c.id)
is dict[str, T]. The attribute-name form keys by that attribute's
value (typed str for the common id/agency_id cases).
ArtefactList ¶
Bases: Sequence[T]
Ordered collection of maintainable SDMX artefacts.
Supports multiple key formats for lookup:
"CL_FREQ"— bare id (KeyError if ambiguous)"SDMX:CL_FREQ"— agency-qualified"SDMX:CL_FREQ(1.0)"— fully qualifiedSdmxUrn[T]— typed URN (version="latest"matches any)SdmxRef(...)— partial identity match
Example
msg.codelists["CL_FREQ"] # bare id msg.codelists["SDMX:CL_FREQ"] # agency-qualified msg.codelists["SDMX:CL_FREQ(1.0)"] # fully qualified msg.codelists[dim.codelist.urn] # SdmxUrn msg.codelists.ids(qualified=True) # ["SDMX:CL_FREQ", ...]
get ¶
Return artefact by key, or None on miss or ambiguity.
filter ¶
Return new ArtefactList with items satisfying predicate.
Accepts a lambda or an item() expression: codelists.filter(lambda cl: cl.agency_id == "SDMX") codelists.filter(item("agency_id") == "SDMX")
first_where ¶
Return first item satisfying the predicate, or None.
partition ¶
Split into (matching, non-matching) ArtefactLists.
sort_by ¶
Return new ArtefactList sorted by key function.
to_dict ¶
Convert to dict keyed by attribute name or key function.
A key function recovers its return type: to_dict(lambda cl: cl.id)
is dict[str, T]. The attribute-name form keys by that attribute's
value (typed str for the common id/agency_id cases).
Maintainable ¶
References and identity¶
A Ref always carries its target's URN and finds the target on access, through the registry or document that bound it. Calling one whose target is not there raises UnresolvedRef.
Ref ¶
Bases: Generic[T_co]
A reference to an SDMX artefact.
Always carries the URN. Finds its target on access (#492): first in the
Resolver of the registry that bound it — so a target added after the
reference was made is found — then in the object it was built with, if
Ref.of / Ref.to built it. Nothing is ever written to it: two Refs
with one URN are equal, and hash alike, whatever either can find.
None on a parent field means "no reference exists."
Ref[T] means "a reference exists, resolved or not."
Access tiers:
ref.urn,ref.id,ref.agency— URN metadata, always availableref.resolved— bool checkref.codes,ref.nameetc — proxy to resolved object, raisesUnresolvedRefif not yet resolved
Construction:
Ref.to(cl_freq) # from domain object — resolved
Ref(cl_freq.urn) # from SdmxUrn — unresolved
Ref.unresolved(urn) # explicit unresolved factory
Ref.of(urn, obj) # explicit resolved factory
Example
ref = Ref(cl_freq_urn) ref.id # "CL_FREQ" ref.resolved # False ref.codes # raises UnresolvedRef
reg.add(cl_freq) reg.ref(cl_freq_urn).resolved # True — bound to the registry reg.ref(cl_freq_urn).codes # ItemList[Code]
resolved
property
¶
True if the target can be found now. False, too, if its registry is closed.
bound
property
¶
True if a registry binds this reference, whether or not it holds the target yet.
item_id
property
¶
Item id from the URN (e.g. the concept id within a ConceptScheme).
__attrs_post_init__ ¶
Reject anything that is not a SdmxUrn.
Validation only — the constructor takes a URN and nothing else. It used
to also accept a MaintainableArtefact and rewrite self.urn in
place, which no type checker can model: the field said SdmxUrn[T],
the constructor accepted a union, and the difference was carried by
three suppressions in this file. Both forms were identical at runtime,
so the only thing the artefact form added was an expression that could
not typecheck at any call site. Ref.to is now the one way in (#219).
__call__ ¶
Unwrap to the target.
Raises:
| Type | Description |
|---|---|
UnresolvedRef
|
If it cannot be found, or its registry is closed. |
__getattr__ ¶
Proxy attribute access to the resolved object.
The Any return is the one legitimate one in model/: this forwards
an arbitrary attribute off the resolved artefact, so its type is only
knowable at the call site (ref.codes, ref.dimensions, …).
Returning object would force a cast at every one of those.
Allows ref.codes instead of unwrapping manually.
Raises UnresolvedRef if the ref has not been resolved.
unresolved
classmethod
¶
Create an unresolved Ref carrying the URN only.
Built as Ref(...) rather than cls(...): on a class whose
parameter is covariant, cls is bound to T_co and a factory
generic in its own U cannot return through it. Ref is not
subclassed, so nothing is lost (#289).
of
classmethod
¶
Create a resolved Ref. Useful in tests and format readers.
to
classmethod
¶
Create a resolved Ref from a MaintainableArtefact instance.
The one way to build a Ref from an artefact. Derives the URN and
holds the object, so the attribute proxy works immediately.
The object must be a MaintainableArtefact (Codelist, DataStructure,
Dataflow, …). Child items like Dimension, Code, or Concept
are not maintainable — their identity lives inside the parent's URN as
an item_id fragment, so this raises TypeError on them. Construct
their URN explicitly via SdmxUrn.make(...) and pass it to
Ref.unresolved or Ref.of instead.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
obj
|
U
|
The maintainable artefact to reference. |
required |
Returns:
| Type | Description |
|---|---|
Ref[U]
|
A resolved |
Raises:
| Type | Description |
|---|---|
TypeError
|
If obj is not a |
Example
cl = Codelist(id="CL_FREQ", maintainer="SDMX", version="1.0", ...) ref = Ref.to(cl) ref.resolved # True
SdmxRef ¶
Partial artefact identity for filtering and lookup.
All fields optional. None means "match any."
Used as ArtefactList key and in Query.
Example
SdmxRef(agency="SDMX", id="CL_FREQ") SdmxRef(id="CL_FREQ", version="1.0")
UnresolvedRef ¶
Bases: Exception
Raised when accessing an unresolved Ref.
SupportsUrn ¶
Bases: Protocol
Protocol for SDMX artefacts that can be referenced by URN.
Any MaintainableArtefact satisfies this protocol. Accepts anywhere
a Ref target is needed — allows Ref.to(cl_freq) instead of
Ref.unresolved(cl_freq.urn).
That second spelling now works, and did not when this docstring was
written: urn was the classmethod builder, so cl_freq.urn on an
instance was a bound method and Ref.unresolved refused it. It is a
property, and the builder is urn_for.
artefact_urn ¶
Canonical SdmxUrn for a maintainable artefact.
The returned URN matches the primary key written by
writers, so external code sharing the same
DuckDB connection can compute foreign keys without re-implementing
the URN convention.
Prefer artefact.urn where the type is known. This exists for code
holding a SupportsUrn — a writer walking a heterogeneous list — and pays
for that generality in the return type below. The property recovers
SdmxUrn[Self] instead, so it carries no Any.
Example
str(artefact_urn(codelist)) 'urn:sdmx:org.sdmx.infomodel.codelist.Codelist=SDMX:CL_FREQ(1.0)'
Returns SdmxUrn[object] (#399): a SupportsUrn can be any artefact, so
the URN can name any. It was SdmxUrn[Any], which let a caller recover
whatever Ref[T] it assigned the result to; a caller that knows the type
has artefact.urn, which is SdmxUrn[Self].
SdmxUrn ¶
Bases: Generic[T_co]
Globally unique identifier for an SDMX artefact.
T is a phantom type — not stored at runtime, used by the type
checker to propagate the target type through
reg.fetch(urn: SdmxUrn[T]) -> T.
Example
SdmxUrn.parse("urn:sdmx:org.sdmx.infomodel.codelist.Codelist=SDMX:CL_FREQ(1.0)") SdmxUrn.make( Codelist, package="codelist", artefact_class="Codelist", agency="SDMX", id="CL_FREQ", version="1.0", )
is_item_urn
property
¶
True if this URN points to an item within a scheme (e.g. a Code).
parse
classmethod
¶
Parse a full URN string, naming no particular artefact type.
SdmxUrn[object]: a URN string can name anything, so nothing
downstream may assume what. get(SdmxUrn.parse(text)) answers
object | None and the caller narrows, or passes expect=. A caller
that knows the type — a reader filling a Ref[Concept] from a stored
string — parses on the class instead: Concept.parse_urn(text), which
is SdmxUrn[Concept] (see ParsesUrn).
It was SdmxUrn[Any] until #399, so that those readers' refs could
become whatever Ref[T] they were assigned to. The same Any made
get(SdmxUrn.parse(text)) answer a confident, wrong LazyCodelist
under basedpyright and an unchecked Any under pyrefly.
make
classmethod
¶
make(
artefact_type: type[U],
/,
*,
package: str,
artefact_class: str,
agency: str,
id: str,
version: str = "latest",
item_id: str | None = None,
) -> SdmxUrn[U]
Low-level factory with basic validation.
artefact_type is a phantom type witness — unused at runtime,
used by the type checker to bind U and propagate the artefact
type through session.get and session.extract.
parse_item
classmethod
¶
Parse an item URN, reusing the scheme half across a batch.
Every item URN in one scheme shares everything but its item_id, and
the readers parse them in batches of hundreds of thousands — 499,138
on the 2021 Census Profile's H_CP_GEO_FED, one per node. Re-running the full
regex for each is re-deriving a constant.
This splits at ").", caches the scheme half, and builds the rest
directly: 0.85 us to 0.55 us, a 1.53x saving measured over 500,000
URNs drawn from two interleaved codelists (so a single hoisted prefix
would not have been enough).
Falls back to parse for anything without that separator — a
versionless or non-item URN — so the result is parse's for every
input, which tests/unit/test_urn_parse_item.py checks differentially.
The cache is keyed on the scheme prefix, so it holds one small tuple per codelist or concept scheme in play. Bounding it by count is right here for the reason it is wrong in #386: the entries are uniform and tiny.
Representation¶
A component's representation is Ref[Codelist | ValueList] | Facet | None — enumerated, free text with bounds, or absent.
DataType ¶
Bases: StrEnum
SDMX data types for non-enumerated representations.
Values match the SDMX-ML textType attribute strings exactly, so
DataType.String == "String" and existing string-passing code continues
to work unchanged.
Example
Facet(type=DataType.String, max_length=100) Facet(type=DataType.ObservationalTimePeriod) Facet(type="String") # str also accepted
Facet ¶
Non-enumerated representation with optional constraints.
type accepts a DataType member or a raw string (for
non-standard values from registry responses).
Example
Facet(type=DataType.String, max_length=100) Facet(type=DataType.ObservationalTimePeriod) Facet(type="String") # str accepted; DataType.String preferred
SentinelValue ¶
A value with a special meaning in a text format: -1 for "not applicable" (#422).
SDMX 3.x's SentinelValueType: the value as written — any simple type,
held as its text, since SDMX-JSON may send a number — and its required
multilingual name and optional description.
Formats¶
Reading and writing SDMX-ML 2.1, 3.0 and 3.1 and SDMX-JSON.
read ¶
read(
source: str | Path | bytes,
*,
version: str | None = None,
registry: SupportsAddAll | None = None,
strict: bool = True,
lossless: bool = False,
) -> StructureMessage | MetadataMessage
Read an SDMX file (or raw bytes) into a Structure or Metadata message.
Format and message type are auto-detected. The result is a
StructureMessage for definitions or a MetadataMessage for
populated reference-metadata payloads.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
str | Path | bytes
|
File path ( |
required |
version
|
str | None
|
Optional format hint: |
None
|
registry
|
SupportsAddAll | None
|
Optional registry to add the artefacts to after parsing. Only honoured for StructureMessages — MetadataMessages do not interact with the registry today. |
None
|
strict
|
bool
|
Raise if the reader refused any artefact. See |
True
|
lossless
|
bool
|
Raise if the read could not hold everything. See |
False
|
Returns:
| Type | Description |
|---|---|
StructureMessage | MetadataMessage
|
The parsed message. |
parse ¶
parse(
data: bytes,
*,
version: str | None = None,
registry: SupportsAddAll | None = None,
strict: bool = True,
lossless: bool = False,
) -> StructureMessage | MetadataMessage
Parse SDMX bytes into a StructureMessage or MetadataMessage.
Auto-detects the format and message type from the payload:
- JSON envelopes are routed by
$schemaURL. - XML envelopes are routed by root element (
<mes:Structure>vs<mes:GenericMetadata>).
A structure document is read artefact by artefact (#435): one that
contradicts the schema is refused — recorded in
StructureMessage.refused, never partly built — and the others are read.
So is one that breaks a rule Registry.add enforces (#419), and a
reference another artefact holds into it is left unresolved.
By default a refusal still fails the call, so nothing wrong is accepted
silently; strict=False returns what was read, refusals attached.
Content inside an artefact that the model has no field for yet — SDMX-ML
2.1's RepresentationMapping in a component map — is a different case
(#422): the artefact is built without it and the loss recorded in
StructureMessage.lost. The read succeeds unless lossless=True.
SDMX-ML only, so far; the SDMX-JSON reader does not report losses yet.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
bytes
|
Raw bytes — JSON or XML. |
required |
version
|
str | None
|
Optional hint: |
None
|
registry
|
SupportsAddAll | None
|
Optional registry to add the artefacts to. Honoured only for StructureMessages. Nothing is interned if the call raises. |
None
|
strict
|
bool
|
When True (the default), any refusal raises
|
True
|
lossless
|
bool
|
When True, any loss, or any value the reader had to supply
for required content the document omitted, raises
|
False
|
Returns:
| Type | Description |
|---|---|
StructureMessage | MetadataMessage
|
The parsed message. |
Raises:
| Type | Description |
|---|---|
SdmxParseError
|
If the payload cannot be read as SDMX — the format
cannot be detected, the bytes are not well-formed XML or JSON, or
the document contradicts the schema ( |
serialize ¶
serialize(
msg: StructureMessage | MetadataMessage,
*,
version: str = "2.1",
pretty_print: bool = False,
validate: bool = True,
) -> bytes
Serialise a Structure or Metadata message to bytes.
Dispatches on the message type:
StructureMessage→<mes:Structure>envelope (XML) or structure$schema(JSON).MetadataMessage→<mes:GenericMetadata>envelope (XML) or metadata$schema(JSON).
Validates before emitting. check_serialisable runs first and raises
ValidationError — with every issue accumulated, not just the first —
so a message the target format cannot carry never reaches the wire. What
every format requires of an artefact alone is refused as it is built
(#492), so there is nothing of that left to check here.
This used to default to False, reasoning that "messages built through
the typed model or read back by the parser are already well-formed". That
proved untrue often enough to matter — #205 found five distinct ways to
build a message the SDMX schema rejects — and the cost argument does not
hold either: validation measures 4-9% of serialisation, the lower figure
being the larger payload, because the item-id checks vectorise above 64
items.
Pass validate=False to skip deliberately — e.g. round-tripping a
message you know is non-conformant in order to repair it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
msg
|
StructureMessage | MetadataMessage
|
The message to serialise. |
required |
version
|
str
|
Wire format — |
'2.1'
|
pretty_print
|
bool
|
If True, indent the output. |
False
|
validate
|
bool
|
Run |
True
|
to_xml ¶
to_xml(
msg: StructureMessage | MetadataMessage,
*,
version: str = "2.1",
pretty_print: bool = False,
validate: bool = True,
) -> bytes
Serialise a Structure or Metadata message to SDMX-ML XML bytes.
MetadataMessage requires version="3.0" (SDMX-ML 2.1 GenericMetadata
is intentionally not supported). See serialize for validate.
to_json ¶
to_json(
msg: StructureMessage | MetadataMessage,
*,
pretty_print: bool = False,
validate: bool = True,
) -> bytes
Serialise a Structure or Metadata message to SDMX-JSON 2.0 bytes.
See serialize for validate.
DataFormat ¶
Bases: StrEnum
A data format a provider can serve, named by its media type.
The value is the exact Accept header to send. Attributes derived from
it — base, version, suffix — are parsed from that one string, so
there is no second place for a member to be described inconsistently.
A StrEnum, so a member is its media type: it can go straight into a
headers dict and compares equal to the string. That is also what keeps
value typed as str rather than Any, which every property here would
otherwise launder out through a -> str.
base
property
¶
The underlying serialisation: "csv", "json" or "xml".
Read out of the media type's +suffix, per RFC 6839 — the same
place a server reads it.
version
property
¶
The format version, as the media type states it.
Note this is the format's version, not the API's, and not always
three parts: SDMX-ML 2.1's media types say version=2.1 while
SDMX-CSV's say version=2.0.0. Reported as written rather than
normalised, because it is echoed back in Accept.
suffix
property
¶
The conventional file extension, leading dot included.
Advisory only. RestRegistry.download never infers a format from a
path — it uses this to refuse a path that contradicts the format asked
for, which is a different thing and the safer half of the two.
What a format cannot carry¶
unrepresentable(artefact, target) names what writing one artefact to a format would silently lose; dropped names what a message would leave out.
unrepresentable ¶
What writing artefact to target would silently lose.
Answers only for this artefact: a Codelist with legacy_is_final unset loses
nothing to 3.0, so nothing is reported. That is what makes the result worth
reading rather than a restatement of the table.
Does not report what would be refused — an unrepresentable version,
or a 2.1 DataStructure with no measures. A draft into 2.1 is not refused
but left out, so it is reported here; what else in a message goes with
it is dropped's to say, since one artefact cannot see the rest. Those raise at write with a
message of their own, and a caller who sees them cannot miss them. This is
for the losses that make no sound.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
artefact
|
object
|
A maintainable, or anything with the fields below. |
required |
target
|
str
|
One of |
required |
Returns:
| Type | Description |
|---|---|
tuple[Gap, ...]
|
The gaps, in table order. Empty when the artefact writes losslessly. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Example
for gap in sl.unrepresentable(codelist, "3.0"): print(gap)
dropped ¶
The artefacts writing msg to target leaves out, each with its cause.
SDMX-ML 2.1's version grammar has no form for an extension, so a draft
(1.0.0-draft) cannot be written. Stripping the suffix would make a draft
look final and could collide with a real 1.0.0, so the artefact is left
out whole (#424).
An artefact of a class the target lacks is left out too (_NO_SUCH_CLASS):
a ValueList or a HierarchyAssociation from 2.1 (#422, #421), a
HierarchicalCodelist or a StructureSet from 3.x and SDMX-JSON (#421),
whose hierarchies or maps are written without it. Every target but
SDMX-ML 3.1, the store included, leaves out an AvailabilityConstraint.
The rest of the message follows. An artefact that references a draft, or any value list, cannot name it in 2.1 and is left out too, and so — repeatedly, until nothing changes — is one that references an artefact the write leaves out, so the output never points at something it does not contain. A reference to an artefact that was never in the message is left alone: whether it exists is not this write's to say.
For a caller that serves artefacts: one asked for by name and left out is
a refusal (a 404 naming reason), and one left out of a listing is a note
on it. serialize cannot tell the two apart; this is how the caller does.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
msg
|
StructureMessage
|
The message about to be written. |
required |
target
|
str
|
One of |
required |
Returns:
| Type | Description |
|---|---|
DroppedArtefact
|
The artefacts left out, in message order, then the availability |
...
|
constraints. Empty for a message with nothing the target lacks. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Gap ¶
One piece of information a target format cannot carry.
Attributes:
| Name | Type | Description |
|---|---|---|
target |
str
|
The format, one of |
field |
str
|
Dotted path as a reader of the model would write it. |
reason |
str
|
Why the format cannot carry it, in terms of the format. |
DroppedArtefact ¶
One artefact a write to a target leaves out, and why.
Attributes:
| Name | Type | Description |
|---|---|---|
urn |
str
|
The artefact left out — for an |
cause |
str
|
The artefact that caused it: |
reason |
str
|
Why, in terms of the format. |
Read errors and outcomes¶
What a read refused, could not hold, or assumed — reported on the message, and raised under strict= / lossless=.
SdmxParseError ¶
Bases: ValueError
Raised when a payload cannot be read as SDMX.
The one exception type a caller needs to catch to handle a bad payload.
Before this existed, parse documented ValueError and actually raised
five types, four of them from the untyped shell — lxml.etree
.XMLSyntaxError, msgspec.DecodeError, msgspec.ValidationError
and UnicodeDecodeError. Handling a malformed provider response
therefore meant importing lxml and msgspec to name their exceptions, which
is the opposite of the boundary this library is supposed to draw: parse,
validate, hand typed objects inward.
The underlying failure is preserved as __cause__, so nothing is lost —
a caller that genuinely wants the lxml line and column can still reach it.
Subclasses ValueError because that is what parse already promised and
what every wrapped exception except lxml's already was, so existing
except ValueError handlers keep working and the docstring stays true.
Example
try: msg = sl.parse(response.content) except sl.SdmxParseError as e: log.warning("provider sent unreadable SDMX: %s", e)
SdmxConformanceError ¶
SdmxConformanceError(
element: str,
detail: str,
*,
refused: tuple[Refusal, ...] = (),
message: StructureMessage | None = None,
)
Bases: SdmxParseError
Raised by the format readers when a message contradicts the SDMX schema.
Distinct from ReferenceValidationError, which reports a reference that
simply could not be resolved — that is a normal state for a partial
message, and stays a non-fatal Ref.unresolved. This one means the input
is wrong: a URN whose artefact class contradicts the element carrying
it, or an element the schema restricts to a URN holding something that is
not one. Neither can be produced by a schema-valid message, so the reader
fails rather than reifying semantics the input does not support.
A SdmxParseError, because it is still "this payload is not usable SDMX" —
the narrower case where the document parsed but says something impossible,
rather than not parsing at all. Catching the base handles both.
Raised two ways. A reader raises it about one element, and the reader one
level up catches it and refuses that artefact (#435). parse then raises
it about the whole document via refusing when anything was refused,
carrying the refusals and the message of everything that was read — so
a caller can recover the rest instead of re-parsing with
strict=False.
Attributes:
| Name | Type | Description |
|---|---|---|
element |
str
|
The XML element name that carried the bad value. |
detail |
str
|
What was expected versus what was found. |
refused |
tuple[Refusal, ...]
|
Every artefact refused, when raised by |
message |
StructureMessage | None
|
What was read despite the refusals, when raised by |
refusing
classmethod
¶
The whole-document error for a message that refused artefacts.
Names the count and the first few, like ValidationError — a
registry document can refuse dozens, and the message is for reading.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
message
|
StructureMessage
|
A read message whose |
required |
Returns:
| Type | Description |
|---|---|
SdmxConformanceError
|
The error, carrying |
LossyReadError ¶
Bases: SdmxParseError
Raised by parse(..., lossless=True) when the message is not exactly the document (#422, #474).
The document is valid, but the model either has nowhere to put some of it
(StructureMessage.lost) or had to supply a value the document omitted
(StructureMessage.assumed). By default both are recorded and the read
succeeds. A caller that would rather stop than continue on something other
than what the document said, a pipeline that round-trips documents, asks
for this instead.
Attributes:
| Name | Type | Description |
|---|---|---|
lost |
tuple[ReadGap, ...]
|
Every loss, per artefact and construct. |
assumed |
tuple[ReadAssumption, ...]
|
Every value the reader supplied, per artefact and construct. |
message |
StructureMessage
|
Everything that was read, losses and assumptions recorded. |
SdmxErrorMessage ¶
Bases: SdmxParseError
Raised by parse when the document is an SDMX error message (GH-31).
A registry answers a query it cannot serve with an error message —
<mes:Error> in SDMX-ML, {"errors": [...]} in SDMX-JSON — rather
than a structure or metadata message. parse used to read one as an empty
StructureMessage, so a saved "100 No Results Found" was indistinguishable
from a query that matched nothing. The REST client never gets this far: it
raises on the HTTP status first. This is for bytes read from a file.
A SdmxParseError, because the payload is still not the message asked
for; catching the base handles it with every other unusable document.
Attributes:
| Name | Type | Description |
|---|---|---|
errors |
tuple[tuple[str, str], ...]
|
One |
Refusal ¶
One artefact a reader declined to build, and why (#435).
A document is a batch of independent maintainable artefacts, and one of
them contradicting the schema used to lose all of them: registry.sdmx.org's
3.0 datastructure document carries 23 DSDs, three enumerate a
dimension with a ConceptScheme, and the reader returned none. The unit of
failure is now the artefact — the reader records a Refusal and reads on.
Plain data, so it can be logged, tallied and compared. It carries no partial artefact: a refused artefact is never constructed.
Attributes:
| Name | Type | Description |
|---|---|---|
artefact_class |
str
|
The SDMX class the element would have built, e.g.
|
urn |
str
|
Best-effort URN text, from what the element itself says — its agency, id and version. The element was not read, so nothing in it was checked; this names it, it does not vouch for it. |
reason |
str
|
Why it was refused — the |
ReadGap ¶
Content a reader met and the model has nowhere to put (#422).
The artefact is built without it and the loss is recorded here, rather
than the artefact refused: a DataStructure carrying one com:Link is still
the DataStructure, and refusing it would cost everything else it says. A
Refusal is for an artefact that cannot be built at all.
Attributes:
| Name | Type | Description |
|---|---|---|
urn |
str
|
The artefact that lost it. |
construct |
str
|
What was lost, as the format spells it — |
count |
int
|
How many occurrences in this artefact. |
reason |
str
|
What the model holds instead, and why that is less. |
ReadAssumption ¶
A value a reader filled in because the document omitted something the schema requires (#474).
A ReadGap records what a reader dropped; this records what it supplied.
The artefact holds the assumed value exactly as if the document had stated
it, so without this a caller cannot tell the two apart. A value the schema
itself defaults is the standard's own answer, and is never one of these.
Attributes:
| Name | Type | Description |
|---|---|---|
urn |
str
|
The artefact holding the assumed value. |
construct |
str
|
What was missing, as the format spells it, e.g.
|
assumed |
str
|
The value used in its place, e.g. |
count |
int
|
How many occurrences in this artefact. |
reason |
str
|
Why that value: the tolerance it implements. |
Polars integration¶
schema ¶
Return a Polars schema dict for all DSD components.
Enumerated components become pl.Enum; text components are mapped
from their Facet.type; everything else defaults to
pl.String.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dsd
|
DataStructure
|
The resolved |
required |
coded_as
|
CodedAs
|
How to type components with a resolved codelist.
|
'enum'
|
Example
df = pl.read_csv("data.csv", schema_overrides=slpl.schema(dsd))
"what exactly is wrong with this file?"¶
lf = pl.scan_csv(path, schema_overrides=slpl.schema(dsd, coded_as="string")) slpl.validation_report(dsd, lf)
validation_report ¶
Validate a DataFrame against enumerated DSD components.
Returns a DataFrame of violations::
┌──────────┬───────────────┬───────┐
│ column │ invalid_value │ count │
└──────────┴───────────────┴───────┘
Only columns present in df are checked. Non-enumerated components
and unresolved codelists are skipped.
Example
report = slpl.validation_report(dsd, df) if report.is_empty(): print("All values valid")
label ¶
label(
df: LazyFrame,
dsd: DataStructure,
*,
lang: str | Sequence[str],
suffix: str = "_label",
columns: Mapping[str, str] | None = None,
) -> pl.LazyFrame
Add human-readable label columns for every enumerated component.
For each enumerated dimension/attribute/measure that exists in df,
appends a {col}{suffix}_{lang} column with the code label. Codes with
no label fall back to the code id.
The one labelling path: Query.frame(labels=) and
Select.collect(labels=) both come here (#517), so a result is
labelled the same way whichever terminal produced it.
columns maps a component id to the column carrying it, for a frame whose
columns are named something else — a projection under an alias. The label
then takes the frame's name: {"REF_AREA": "area"} labels area as
area_label_en, because the name a caller reads is the one they chose.
Without it, ids and column names are assumed to match.
The language is always in the name, including for a single one:
REF_AREA_label_en, never REF_AREA_label. Otherwise adding a second
language later renames the first column, and a purely additive change
breaks whatever read it. lang accepts one or several.
Label maps are cached per (codelist instance, lang) at the source
(LazyCodelist.code_map) with a single-flight build lock, so the
first caller builds the map and concurrent callers share it rather
than each re-querying. A fresh ingest produces a new codelist
instance with a fresh cache, so re-ingestion is automatically correct.
Columns are matched by SDMX component id, so the frame's columns have
to be named the way the DSD names them. A frame carrying physical column
names — GEO where the DSD says REF_AREA — matches nothing.
That used to happen silently and return the frame unchanged, which is how
catalog.query(with_labels=True) (deleted in #329) came to be a no-op for
any binding whose
columns are not already SDMX ids. Nobody noticed because nothing failed.
So a call that matches no component at all now raises: labelling
nothing is never what a caller meant, and a frame with some matching
components is a different case — extra columns are ordinary, and those are
still passed through untouched.
Raises:
| Type | Description |
|---|---|
ValueError
|
If not one of the DSD's components names a column of |
Example
labelled = slpl.label(pl.scan_csv("data.csv"), dsd, lang="en").collect()
structure_frame ¶
Export the DSD structure as a tidy DataFrame.
Returns one row per component::
┌────────────┬───────────┬──────────┬───────────┬──────────┬─────────┐
│ component │ role │ position │ concept │ codelist │ n_codes │
└────────────┴───────────┴──────────┴───────────┴──────────┴─────────┘
role is one of "dimension", "time_dimension",
"attribute", "measure".
Example
meta = slpl.structure_frame(dsd, lang="en")
Dataflow bindings¶
A DataflowBinding ties a Dataflow to a concrete data-plane table
(local file, DuckDB table, or DuckLake catalog) so a registry can answer
observation queries. Drift detection is exposed on every registry
(see RestRegistry, FederatedRegistry, InMemoryRegistry above) as
two complementary methods:
check_binding_drift(binding) -> list[DriftIssue]— inspection; returns the list of mismatches (empty if clean).validate_binding(binding) -> None— enforcement; raisesDriftError(withissuesattached) if any drift is detected.
DataflowBinding ¶
Bases: ParsesUrn
Binding between an SDMX dataflow and a concrete data table.
Attributes:
| Name | Type | Description |
|---|---|---|
dataflow |
Ref[Dataflow]
|
Ref to the bound dataflow (may be remote). |
database_role |
str
|
ATTACH alias of the catalog database that owns the
table. Defaults to |
table_location |
str
|
|
partition_layout |
tuple[str, ...]
|
SDMX dimension ids the table is partitioned on, in partition order. Drives partition pruning at query time. |
sort_order |
tuple[str, ...]
|
SDMX component ids the rows are stored sorted by, outermost first. Empty means the layout says nothing, not that the rows are unsorted. Declared for the reason It does not make a query's |
dimension_column_map |
dict[str, str]
|
SDMX dimension id → table column name. Identity mapping is the common case. |
measure_column_map |
dict[str, str]
|
SDMX measure id → table column name. |
attribute_column_map |
dict[str, str]
|
SDMX attribute id → table column name. |
structural_version |
str
|
|
source_registry |
str
|
Provenance label for the structural side
(e.g. |
materialized_at |
datetime
|
When the data was last refreshed. Must be tz-aware; storage normalises to UTC. |
bound_at |
datetime
|
When this binding record was created. Must be tz-aware; storage normalises to UTC. |
series_attributes_location |
str | None
|
Optional sidecar table for series-level
attributes that are constant across (REF_AREA, TIME_PERIOD)
slices and would otherwise bloat the fact table 5x if
denormalised in. Same shape as |
component_of ¶
The SDMX component id a physical column carries, or None.
The inverse of the three column maps, and the only route from a compiled predicate — which carries a physical name and nothing else — back to the component a DSD talks about.
First mapping wins if two ids share a column, which is malformed rather
than meaningful: check_binding_drift is where that should surface,
not here.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
column
|
str
|
A physical column name, as it appears on the table. |
required |
Returns:
| Type | Description |
|---|---|
str | None
|
The component id, or |
urn_for
classmethod
¶
Construct a typed SdmxUrn[DataflowBinding].
The binding's URN mirrors the bound dataflow's identity — one binding per dataflow per version.
from_dsd
classmethod
¶
from_dsd(
dsd: DataStructure,
*,
dataflow: Ref[Dataflow],
schema: str,
table: str,
partition_layout: tuple[str, ...] = (),
source_registry: str = "local",
materialized_at: datetime | None = None,
bound_at: datetime | None = None,
database_role: str = "lake",
series_attributes_location: str | None = None,
) -> DataflowBinding
Build a DataflowBinding from a DSD with identity column mapping.
Derives dimension_column_map / measure_column_map /
attribute_column_map from the DSD's components (SDMX id →
SDMX id), pins structural_version as
f"{dsd.agency_id}:{dsd.id}({dsd.version})", and addresses the
target table via database_role + schema.table (no
database alias prefix is written into table_location).
materialized_at and bound_at default to datetime.now(UTC).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dsd
|
DataStructure
|
The |
required |
dataflow
|
Ref[Dataflow]
|
Ref to the bound dataflow. |
required |
schema
|
str
|
Schema portion of |
required |
table
|
str
|
Table portion of |
required |
partition_layout
|
tuple[str, ...]
|
SDMX dimension ids the table is partitioned on, in partition order. |
()
|
source_registry
|
str
|
Provenance label for the structural side. |
'local'
|
materialized_at
|
datetime | None
|
When the data was last refreshed. Defaults
to |
None
|
bound_at
|
datetime | None
|
When this binding record was created. Defaults to
|
None
|
database_role
|
str
|
ATTACH alias of the database that owns the
table. Defaults to |
'lake'
|
series_attributes_location
|
str | None
|
Optional sidecar table for
series-level attributes (same shape as
|
None
|
Returns:
| Type | Description |
|---|---|
DataflowBinding
|
A |
DriftIssue ¶
A single drift between a binding's pinned structural state and the live registry.
Attributes:
| Name | Type | Description |
|---|---|---|
kind |
str
|
Stable identifier for the kind of drift. v1 vocabulary:
|
binding_urn |
str
|
URN of the binding the issue relates to. |
description |
str
|
Human-readable explanation including before/after values where relevant. |
DriftError ¶
Bases: Exception
Raised by Registry.validate_binding() when any drift is detected.
issues carries every drift found — not just the first — so a single
catch surfaces the full set. For inspection without raising (the
expected pattern when you want to display drift in a UI or log it),
use Registry.check_binding_drift() instead.
REST facade kit¶
For services publishing SDMX REST endpoints — parses inbound URL slices into typed filters. See the REST Facade cookbook.
parse_components ¶
parse_components(
key: str,
component_filters: dict[str, list[str] | SqlRelation],
key_dimension_ids: list[str],
) -> ComponentFilter
Parse SDMX REST component filters into a ComponentFilter.
Combines the two filter mechanisms of SDMX REST 2.0:
- The dotted-key path segment that constrains key dimensions.
- The
c[<id>]=<vals>query params that constrain any other component — typically attributes that don't appear in the key.
The HTTP layer is expected to have already split the URL slice into
these two parts before calling this function. component_filters
carries the parsed c[...]=... dict; the web framework handles
URL-decoding.
Filter values may be eager list[str] (the URL form, after
splitting on +) or a SqlRelation (lazy one-column subquery —
typically LazyCodelist.narrower_transitive(...).select(sl.q.code.id)
for hierarchy slices).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key
|
str
|
The dotted-key fragment from the URL. |
required |
component_filters
|
dict[str, list[str] | SqlRelation]
|
|
required |
key_dimension_ids
|
list[str]
|
Dimension ids in DSD key order. Used both to size the result and to validate the segment count. |
required |
Returns:
| Type | Description |
|---|---|
ComponentFilter
|
A |
ComponentFilter
|
to |
ComponentFilter
|
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If the key has a different number of segments than
|
ComponentFilter ¶
Structured form of an SDMX REST component filter.
Self-describing: carries the dimension ids it was built against
(parallel to dimensions) so downstream translators don't have
to second-guess key order. Attribute filters are a flat id →
values dict — order is irrelevant for attributes.
Each filter value may be either an eager list[str] (OR set of
allowed codes) or a SqlRelation
(lazy single-column subquery). The SQL builder dispatches on the
type — lists become IN (?, ?, ...); relations become
IN (<subquery>) with the relation's params spliced into the
outer bind list.
None and [] are different, deliberately (#256). None is
"this component is unconstrained"; [] is "the set of permitted values
is empty", which no observation can satisfy, so the query returns nothing.
Reading [] as "no constraint" would make a dynamically-computed code
list that came back empty silently return the entire dataset, and it would
give the library two spellings for one meaning while the literal reading of
one of them is the opposite. polars (is_in([]) → empty) and SQLAlchemy
(in_([]) → a 1 != 1 contradiction) both resolve it this way, as does
the relational definition: x IN S is ∃y∈S: x = y, false for empty
S.
Attributes:
| Name | Type | Description |
|---|---|---|
dimension_ids |
list[str]
|
Dimension ids in DSD-key order. |
dimensions |
list[list[str] | SqlRelation | None]
|
Parallel list — |
attributes |
dict[str, list[str] | SqlRelation]
|
Per-attribute allowed values. Each value is either
a list (OR semantics, |
is_wildcard
property
¶
True if no dimension or attribute is constrained.
An empty list is a constraint — the one that admits nothing — so a filter carrying one is not a wildcard even though it selects no rows.
to_where_clauses ¶
to_where_clauses(
dimension_column_map: dict[str, str],
attribute_column_map: dict[str, str] | None = None,
) -> tuple[list[str], list[object]]
Render this filter as (where_fragments, params).
Each constrained dimension or attribute becomes one
"col IN (...)" fragment with values appended to params
in order. Wildcard dimensions are skipped. Eager-list values
emit IN (?, ?, ...); SqlRelation values emit
IN (<subquery>) with the relation's bind params spliced
in at the right position.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dimension_column_map
|
dict[str, str]
|
SDMX dimension id → physical column
name. Usually |
required |
attribute_column_map
|
dict[str, str] | None
|
SDMX attribute id → physical column
name. Usually |
None
|
Returns:
| Type | Description |
|---|---|
list[str]
|
A pair of |
list[object]
|
|
Raises:
| Type | Description |
|---|---|
KeyError
|
If a constrained dimension is missing from
|
widen_key ¶
Accept a key written for the other REST generation, where that is safe.
Real clients disagree about the time dimension. Fusion Registry, pandasdmx and SDMX-RI all send 1.5-shaped keys — time omitted — and a 2.2.2 endpoint expects one segment more (#337). A facade that refuses them is refusing conformant requests from most of the ecosystem, so this widens a short key by appending the empty (wildcard) time segment.
Only that one direction, and only when it is unambiguous. A 2.2.2 key
given to a 2.1 endpoint is not narrowed: dropping a trailing segment
would silently discard a filter the caller wrote. And a key of any other
length is returned untouched, so parse_components still refuses it with
a message naming both counts rather than this guessing.
Deliberately not applied inside parse_components. Leniency is a
policy a facade chooses; a parser that quietly reinterprets its input has
no way to tell a caller they sent the wrong thing. Call this first when you
want it:
key = sl.widen_key(raw_key, dsd, "3.0")
cf = sl.parse_components(key, params, dsd.key_slot_ids("3.0"))
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key
|
str
|
The dotted key as received. |
required |
dsd
|
DataStructure
|
The structure the key is positional over. |
required |
api
|
ApiVersion
|
The generation this endpoint serves. |
required |
Returns:
| Type | Description |
|---|---|
str
|
The key, widened when it was one short of this version's shape and the |
str
|
missing slot is the time dimension. Otherwise unchanged. |
Validation¶
Every rule an artefact can check on its own runs as it is built, so a bad
artefact is refused at construction. validate() keeps what only a message
can know — that its references resolve within it and no artefact appears
twice.
validate ¶
Check what only a message can know: its references resolve, and no artefact is in it twice.
An artefact is checked as it is built (#492): every rule that depends on
it alone runs in its constructor and in attrs.evolve, so none can fail
here, and passing one says so rather than appear to check it. What the
target format can carry is serialize's question (check_serialisable).
Closed-world, as validate_references is: a reference must name an
artefact in the message, so a partial bundle — a dataflow without its DSD —
fails, and is resolved against a richer registry instead.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
msg
|
StructureMessage | MetadataMessage
|
A |
required |
Raises:
| Type | Description |
|---|---|
ValidationError
|
If a maintainable is in the message twice. |
ReferenceValidationError
|
If any reference fails to resolve. |
TypeError
|
If msg is an artefact or anything else. |
Example
validate(msg) # no return; raises on any issue
validate_references ¶
Check that all Refs within a message resolve to artefacts in that message.
This is a closed-world check — every outgoing Ref must point to a URN of an artefact also present in the message. Partial bundles (e.g. a dataflow without its DSD) will fail and must use ref-resolution against a richer registry instead.
An item reference resolves against the items of a scheme in the
message: a HierarchicalCode pointing at a Code, a Categorisation at a
Category, a HierarchyAssociation at the Dimension it organises. Three
spellings of the same target are accepted, because all three are written in
practice — a nested item by dotted path or by leaf id, a component under
either the 2.1 or the 3.0 class name, and any artefact with its version
omitted, which SDMX reads as latest.
Raises:
| Type | Description |
|---|---|
ReferenceValidationError
|
If any Ref fails to resolve.
|
ValidationIssue ¶
A single validation problem found on an artefact.
Attributes:
| Name | Type | Description |
|---|---|---|
path |
str
|
Dotted path to the offending field (e.g. |
message |
str
|
Human-readable description of the problem. |
ValidationError ¶
Bases: Exception
Raised when an artefact breaks a rule as it is built, or a message or format check finds an issue.
All issues are accumulated — issues holds the full list, not just
the first problem. The exception message renders the first few inline
with a count of the rest.
Example
try: codelist = Codelist(id="CL", maintainer="AG", name={"en": "c"}, codes=codes) except ValidationError as e: for issue in e.issues: fix(issue)
ReferenceIssue ¶
A broken cross-artefact reference within a StructureMessage.
ReferenceValidationError ¶
Bases: Exception
Raised by validate_references() when any Ref fails to resolve.
issues carries every broken reference in the message — not just
the first — so a single catch surfaces the full set.
SdmxWarning ¶
Bases: UserWarning
The base of every warning sdmxlib gives.
warnings.simplefilter("error", sl.SdmxWarning) turns them all into
errors, and "ignore" silences them.
ImRuleWarning ¶
Bases: SdmxWarning
An artefact breaks a rule the information model states and no wire schema enforces (#481).
The artefact is still built: every SDMX format can carry it, and refusing would break documents and builders that work today. One warning per artefact, carrying every rule it breaks.
Example
with warnings.catch_warnings(record=True) as caught: warnings.simplefilter("always", sl.ImRuleWarning) msg = sl.parse(document) for warning in caught: if isinstance(warning.message, sl.ImRuleWarning): for issue in warning.message.issues: print(issue.path, issue.message)
Dict conversion¶
Round-trip artefacts through plain Python dicts (JSON-friendly, no SDMX wire format).
to_dict ¶
Convert a domain object to a nested plain dict.
InternationalString→dict[str, str]Ref[T]→ URN string (or{"kind": "Enumeration", "urn": "..."}for representation fields)Facet→{"kind": "Facet", ...}ItemList/ArtefactList→list[dict]Annotations→list[dict](omitted when empty)Agencymaintainer →"agency_id": str(serialized as plain ID string)- Enums → string value
Nonefields are omitted
Example
d = to_dict(dsd) json.dumps(d, indent=2)
from_dict ¶
Construct a domain object from a nested plain dict.
Handles:
- dict name/description fields → coercion via __post_init__
- URN strings for Ref fields → Ref.unresolved
- Representation dicts → Ref[Codelist] or Facet
- List of child dicts → recursive from_dict per element type
- Attachment strings → AttachmentLevel
- Annotation dicts → Annotation instances
The resulting Refs are unresolved (URN only). To resolve them, use a Registry or pass the object through a StructureMessage parser.
Example
cl = from_dict(Codelist, {"id": "CL_FREQ", "agency_id": "SDMX", "name": {"en": "Frequency"}, "codes": [{"id": "A"}]}) cl.codes["A"] # Code(id="A")
data is dict[str, Any] rather than dict[str, object] because the
payload's shape depends on cls — every value flows into a differently
typed constructor parameter, so there is no single narrowing that would
serve. This is the generic deserialiser; the typed entry points are the
per-format readers built over it (#285).
SQL surface¶
The protocol a one-column projection satisfies, so it splices into another query (or a ComponentFilter) as a subquery.
SqlRelation ¶
Bases: Protocol
A lazy relation usable inside WHERE col IN (...) or as a join input.
Producers — sdmxlib.query.query.Select, AncestorMap, and the statcatlib
query builders — expose their compiled SELECT as a (sql, params) pair.
The consumer splices the SQL inline as a subquery and concatenates the
params with the outer query's binds, so an N-row traversal becomes one
IN (SELECT ...) clause with no parameter explosion.
The subquery's tables must be visible on the consumer's connection. For
DuckDBDataStore (which shares its connection with Registry), codelist
relations compose automatically because both sides see the same
sdmx.code table. For statcatlib's Catalog, the same applies when the
bundle uses Topology B (one DuckDB file holding sdmx., statcat., and the
DuckLake catalog).
columns
property
¶
Names of the columns to_sql produces, in order.
Declared so arity is checkable rather than promised in prose. A
single-column relation is what IN (<subquery>) needs; a rollup is
two columns and is used as a join. Consumers that require one column
should say so and reject the rest, instead of emitting SQL that fails
at the database with a column-count error (#274).
Filter expressions¶
item() builds predicates for ItemList.filter().
item ¶
Build a filter expression for an item attribute.
Example
dims.filter(item("id") == "FREQ") dims.filter(item("is_enumerated")) dims.filter(~item("is_enumerated")) dims.filter(item("id").is_in(["FREQ", "GEO"])) dims.filter(item("codelist").is_null())
ItemExpr ¶
Expression builder for a single item attribute.
Created via item("attr_name"). Calling the expression directly
performs a truthy check; comparison operators and methods return
composable ItemPredicate objects.
Example
item("is_enumerated") # truthy check ~item("is_enumerated") # negated truthy item("id") == "FREQ" # equality item("id") != "FREQ" # inequality item("id").is_in(["FREQ", "GEO"]) # membership item("codelist").is_null() # None check
ItemPredicate ¶
A composable predicate returned by ItemExpr operations.
Supports &, | and ~. The operands may be any filter
ItemList.filter() would accept — another predicate, a bare ItemExpr, or
a plain lambda:
~item("is_enumerated") # not enumerated
(item("id") == "FREQ") & item("is_enumerated") # both
(item("id") == "FREQ") | (item("id") == "TIME") # either
These are the same three operators the SQL-backed DSLs grew in #262, spelled
the same way. The implementation has nothing in common — this side composes
Python callables and there is no NULL, so unlike the SQL predicates these do
obey excluded middle: p | ~p holds for every item.