Skip to content

Catalog

Generated from the docstrings. Catalog API walks through the same surface with examples.

The catalog model

Catalog

Catalog(
    *args: object,
    _opening: _Opening | None = None,
    **kwargs: object,
)

A directory of products, each a chain of immutable editions (dcat:Catalog).

Create one with create, which also reopens an existing catalog for writing, or open one read-only with open. A catalog reads its Inventory through sdmxlib.inventory.LiveInventory, so it sees live editions only; refresh re-reads it. Every view it returns shares one DuckDB engine, and close releases them all.

Queries run on a product's view: cat.product(id).view().query(...). Search and facets across products are not yet supported.

Refuse construction from outside, with a TypeError naming open, create and temporary_catalog.

writable property

writable: bool

Whether the catalog came from create, so its products can be written.

catalog_id property

catalog_id: str

The catalog's identity, a UUIDv7 from catalog.json.

description property

description: CatalogDescription | None

The catalog's description, as of the last refresh; None when none was recorded.

CatalogAdmin.describe records it in the Inventory root's extensions.statcat, and every later version carries it. It holds the catalog-level fields only; the datasets come from the products' live editions.

create classmethod

create(path: str | Path, *, exist_ok: bool = False) -> Self

Create a catalog at path, or with exist_ok reopen the one there for writing.

A missing or empty directory gets catalog.json, holding a UUIDv7 catalog_id and restore_epoch 0, and nothing else: the first seal writes Inventory version 1. The path created is logged at INFO, since a mistyped path with exist_ok=True creates a new, empty catalog.

Parameters:

Name Type Description Default
path str | Path

The catalog directory.

required
exist_ok bool

Reopen an existing catalog for writing, writing nothing.

False

Returns:

Type Description
Self

The catalog, writable: its products' Product.working_copy works.

Raises:

Type Description
FileExistsError

path is a catalog and exist_ok is false.

A legacy layout or other files at path, or a filesystem without hard links, raise IncompatibleCatalogError.

open classmethod

open(path: str | Path) -> Self

Open the catalog at path read-only.

Reads catalog.json and the Inventory, and writes nothing: no repair, no DDL. A catalog with no Inventory version yet has no products.

Parameters:

Name Type Description Default
path str | Path

The catalog directory.

required

Returns:

Type Description
Self

The catalog. Its products' Product.working_copy raises; open it

Self

with create to write.

Raises:

Type Description
FileNotFoundError

path is not a catalog.

A legacy layout or other files at path raise IncompatibleCatalogError.

refresh

refresh() -> None

Re-read the Inventory, so product and products see the newest live editions.

A view taken before stays on its edition; call product(id).view() again to follow the live edition.

products

products() -> list[str]

The products with a live edition, sorted.

product

product(product_id: str) -> Product

The product product_id, which need not exist yet.

Parameters:

Name Type Description Default
product_id str

The product's id. It names a directory: letters, digits, ., _ and -, starting with a letter or digit.

required

Returns:

Type Description
Product

The product. An invalid product_id raises ValueError.

seal

seal(
    working_copies: Iterable[WorkingCopy],
    *,
    checks: Iterable[
        Callable[[EditionView], Iterable[Finding]]
    ] = (),
) -> SealOutcomes

Seal several products' working copies and record them in one Inventory version.

It runs WorkingCopy.seal's steps in four passes. The structure pass pre-checks each structure product's working copy (journal, checks, content, fast-forward) and seals each that passed. The structure-check pass runs each sealed structure edition's publish check, given the batch's other structure editions in place of their products' recorded ones, and drops each edition with a finding until a pass drops none. The data pass pre-checks every data product's working copy before any of them is sealed, then seals each that passed, and runs the publish check of each data edition with partials, given the batch's structure editions in place of their products' recorded ones. Last it records every sealed edition in one Inventory commit, leaving out each product whose record sdmxlib's rules refuse, and each edition whose publish check was given a left-out one. A refusal in one product never stops another's seal.

A crash during the batch can leave its products recorded across several Inventory versions: each sealed, unrecorded edition keeps its journal entry, and its product's next working_copy() records it.

Parameters:

Name Type Description Default
working_copies Iterable[WorkingCopy]

The working copies, one per product, each from this catalog.

required
checks Iterable[Callable[[EditionView], Iterable[Finding]]]

Seal checks (statcatlib.checks.SealCheck), run against every working copy's view.

()

Returns:

Type Description
SealOutcomes

Each product's outcome. An empty batch writes nothing.

Before anything is sealed, a working copy of another catalog, or two of one product, raise ValueError, and a sealed, discarded or closed working copy raises WorkingCopyClosedError. A catalog opened read-only with Catalog.open raises RuntimeError.

close

close() -> None

Close every view this catalog returned, and its engine.

search

search(*args: object, **kwargs: object) -> NoReturn

Not yet supported across products: raises NotImplementedError; search one product's view.

facets

facets(*args: object, **kwargs: object) -> NoReturn

Not yet supported across products: raises NotImplementedError; count one product's view.

__getattr__

__getattr__(name: str) -> NoReturn

Name the replacement of a query method that moved to EditionView.

Parameters:

Name Type Description Default
name str

The attribute asked for.

required

Raises:

Type Description
AttributeError

Always; for a moved method, naming product(id).view().<name>.

Product

Product(
    *args: object,
    _of: tuple[Catalog, str] | None = None,
    **kwargs: object,
)

One product of a catalog: its live edition, and its working copy.

Catalog.product returns one.

Refuse construction from outside, with a TypeError: Catalog.product returns products.

product_id property

product_id: str

The product's id.

view

view(
    *,
    embedding_model: EmbeddingModel | None = None,
    query_prefix: str = "",
) -> EditionView

A read-only view of the product's live edition, as of the catalog's last Catalog.refresh.

It attaches the edition to the catalog's engine with LocalRegistry.edition, which reads manifest.json once, checks its SHA-256 against the one the Inventory records, and checks its store schema. The view stays on this edition after a refresh or a rollback.

Parameters:

Name Type Description Default
embedding_model EmbeddingModel | None

The encoder the index was built with, for vector and hybrid search. BM25 search needs none.

None
query_prefix str

Prepended to every query before embedding; the one the index recorded when empty.

''

Returns:

Type Description
EditionView

The view.

Raises:

Type Description
IncompatibleCatalogError

The edition's manifest.json is missing, unreadable, or not the one the Inventory recorded (sdmxlib's ManifestError, chained). Its statcat schema at another version, or a search index this build does not read, raise it too, and a product with no live edition raises NoLiveEditionError.

editions

editions() -> list[Edition]

The product's live editions, each with the previous its manifest records.

The public catalog holds live editions only, one per store schema; CatalogAdmin.editions lists every state.

working_copy

working_copy(
    *,
    kind: Kind | None = None,
    lifecycle: Lifecycle = "revisable",
    resume: bool = False,
    embedding_model: EmbeddingModel | None = None,
    document_prefix: str = "",
    query_prefix: str = "",
) -> WorkingCopy

Start the product's working copy on its live edition, or resume the unsealed one.

It settles the product's seal journal first (the journal rule, in statcatlib.store.working_copy), so an edition a crash left sealed and unrecorded is recorded before the new one picks its base.

Parameters:

Name Type Description Default
kind Kind | None

data, or structure for a product of full artefacts only (WorkingCopy.fill_structures). None takes the kind the product is recorded with, or the one a resumed working copy was started with, and data for a new product. A product keeps the kind it was first recorded with.

None
lifecycle Lifecycle

revisable or final. A product keeps the lifecycle it was first recorded with.

'revisable'
resume bool

Reopen the product's unsealed edition in place of starting one.

False
embedding_model EmbeddingModel | None

The encoder WorkingCopy.seal embeds with.

None
document_prefix str

Prepended to every document before embedding.

''
query_prefix str

Prepended to every query before embedding.

''

Returns:

Type Description
WorkingCopy

The working copy.

A catalog opened read-only with Catalog.open raises RuntimeError. A product recorded with another lifecycle raises InventoryRuleError (rule="lifecycle"); one recorded with another kind, or resumed with another kind than it was started with, raises it with rule="kind" before anything is built; and one with an unsealed edition raises WorkingCopyExistsError unless resume is true.

rollback

rollback(*args: object, **kwargs: object) -> NoReturn

Rollback is an operational change: raises AttributeError naming CatalogAdmin.rollback.

Edition

A sealed edition, as the Inventory records it.

Attributes:

Name Type Description
edition_id str

The edition's id, a UUIDv7.

product_id str

Its product.

previous str | None

The edition it was built on, or None for the product's first.

manifest_sha256 str

The SHA-256 of its manifest.json.

SealOutcomes

What Catalog.seal did with each working copy, keyed by product id.

Every product of the batch is in exactly one of the two mappings.

Attributes:

Name Type Description
sealed Mapping[str, Edition]

Each recorded product's new edition. All of them were recorded in one Inventory version.

refused Mapping[str, Exception]

Each refused product's error. A SealCheckError, a NonFastForwardError from the pre-check, or another InventoryRuleError from sdmxlib's rules leaves the working copy open; a ManifestError closes it, and working_copy(resume=True) reopens it. An EditionCheckError from the publish check of a structure edition or of an edition with partials, or an InventoryRuleError at record time, leaves the sealed edition on disk unrecorded, and an InventoryContentionError at record time leaves it for the product's next working_copy() to record.

WorkingCopy

WorkingCopy(
    *args: object,
    _parts: _Parts | None = None,
    **kwargs: object,
)

A product's one unsealed edition, open for writing.

Product.working_copy returns one. Write datasets with put and the other write methods, and SDMX structures and observations through registry; read them back through view; then seal it, or discard it. Once sealed or discarded, every method on it, and on its views, raises WorkingCopyClosedError.

Refuse construction from outside, with a TypeError: Product.working_copy returns working copies.

product_id property

product_id: str

The product this working copy belongs to.

edition_id property

edition_id: str

The id of the edition it seals into, a UUIDv7.

previous property

previous: str | None

The edition it was built on, or None for the product's first.

kind property

kind: Kind

The product's kind: data, or structure for a product of full artefacts only.

registry property

registry: LocalRegistry

The writable sdmxlib registry over the edition: SDMX structures and observations.

wc.registry.ingest(source) adds structures, and wc.registry.write_observations(...) writes observations into the edition's lake.

view

view() -> EditionView

A read-only view of this working copy, which sees its writes as they are made.

fill_structures

fill_structures(
    source: EditionRegistry,
    *,
    families: Collection[type[MaintainableArtefact]] = (),
    agency: str | None = None,
    ids: Collection[str] = (),
    artefacts: Collection[
        SdmxUrn[MaintainableArtefact] | str
    ] = (),
) -> tuple[str, ...]

Copy full artefacts from source into a structure product's working copy.

It calls sdmxlib.edition.fill_structures. An artefact is selected when artefacts names its URN, or when its class is one of families, its agency is agency (any when None) and its id matches one of the ids patterns (SGC*; any id when empty). Nothing is added for what a selected artefact references, so a reference into another structure product stays open; the seal's publish check resolves it there. An artefact the working copy holds at a selected URN is replaced.

Parameters:

Name Type Description Default
source EditionRegistry

The registry holding the full artefacts, such as a scratch sdmxlib.LocalRegistry an FMR's structures were ingested into.

required
families Collection[type[MaintainableArtefact]]

The artefact classes to select, such as sdmxlib.Codelist or sdmxlib.ConceptScheme.

()
agency str | None

The maintenance agency of the artefacts families selects.

None
ids Collection[str]

Patterns the id of an artefact families selects must match.

()
artefacts Collection[SdmxUrn[MaintainableArtefact] | str]

More artefacts, by URN.

()

Returns:

Type Description
tuple[str, ...]

The URNs copied, sorted.

Raises:

Type Description
TypeError

The working copy is a data product's.

sdmxlib raises TypeError for a family that is Dataflow, DataflowBinding or a class the store does not hold, and ValueError for a URN that names a dataflow, a binding, a partial or nothing source holds, or when nothing is selected.

fill_dataflows

fill_dataflows(
    source: RestRegistry | EditionRegistry,
    dataflows: Sequence[SdmxUrn[Dataflow] | str],
    *,
    observations: Mapping[str, WriteSource] | None = None,
    partials: bool = True,
) -> dict[str, str]

Add dataflows' closures and observations to a data product's working copy, and cut its codelists.

It calls sdmxlib.edition.fill_edition, which keeps what the working copy holds: on a republish the other dataflows' bindings stay, and a codelist they share is cut to the codes every binding uses. The map it returns is merged into statcat._meta.partial_of, a later fill's value winning for a URN, so a seal after working_copy(resume=True) in another process records each partial's partial_of. statcat._meta.fill_in_progress is set while sdmxlib fills; a fill that stops before the merge leaves it, and the seal refuses the working copy until a fill completes.

Fill a data product through this method: a direct fill_edition(wc.registry, ...) leaves its cuts out of the map.

Parameters:

Name Type Description Default
source RestRegistry | EditionRegistry

Where the structures are: a sdmxlib.RestRegistry, or a sealed edition's or a scratch sdmxlib.EditionRegistry.

required
dataflows Sequence[SdmxUrn[Dataflow] | str]

The dataflows, by URN. A latest URN is filled at the version it resolves to.

required
observations Mapping[str, WriteSource] | None

The rows of each dataflow that has any, by URN string, in any form sdmxlib.LocalRegistry.write_observations reads.

None
partials bool

Whether to cut codelists to partials.

True

Returns:

Type Description
dict[str, str]

The fingerprint of the full codelist each partial cut here was

dict[str, str]

cut from, by URN.

Raises:

Type Description
TypeError

The working copy is a structure product's.

sdmxlib raises TypeError for a URN that names no dataflow at source, and ValueError for a dataflow with no data structure, for observations naming a dataflow not in dataflows, and for a coded column holding a value outside its codelist.

put

put(
    obj: Dataset
    | ConceptScheme
    | Sequence[Dataset | ConceptScheme],
) -> None

Add or update a dataset, a concept scheme, or a sequence of either.

Parameters:

Name Type Description Default
obj Dataset | ConceptScheme | Sequence[Dataset | ConceptScheme]

What to write.

required

put_concept

put_concept(scheme_uri: str, concept: Concept) -> None

Add or update one concept of a scheme.

Parameters:

Name Type Description Default
scheme_uri str

The scheme's URI.

required
concept Concept

The concept.

required

add_lineage

add_lineage(
    source_id: str,
    target_id: str,
    relationship: str,
    *,
    activity_label: str | None = None,
    activity_type: str | None = None,
    agent_id: str | None = None,
    timestamp: datetime | None = None,
    confidence: str = "asserted",
    notes: str | None = None,
) -> None

Record a PROV-O-aligned lineage edge between two datasets.

Parameters:

Name Type Description Default
source_id str

The source dataset.

required
target_id str

The target dataset.

required
relationship str

One of LINEAGE_RELATIONSHIPS.

required
activity_label str | None

The activity's label.

None
activity_type str | None

The activity's type.

None
agent_id str | None

The agent responsible.

None
timestamp datetime | None

When.

None
confidence str

One of LINEAGE_CONFIDENCE.

'asserted'
notes str | None

Free text.

None

add_language_alternates

add_language_alternates(
    pairs: Mapping[str, Sequence[str]], *, source: str
) -> None

Record language tags a source declares as alternates of a primary tag.

Parameters:

Name Type Description Default
pairs Mapping[str, Sequence[str]]

Each primary tag mapped to its alternate tags.

required
source str

Where the pairs come from.

required

bulk_load

bulk_load() -> Generator[Self]

Hold DuckDB's write-ahead log until the block ends, then checkpoint once.

transaction

transaction() -> Generator[Self]

Wrap the writes in the block in one DuckDB transaction.

build_index

build_index(
    embedding_model: EmbeddingModel | None = None,
    progress: Callable[[int, int], None] | None = None,
    *,
    include_codes: bool = True,
    embed_codes: bool = False,
    max_codes_per_codelist: int = 500,
    codelist_urns: Sequence[str] | None = None,
    codelist_policies: Sequence[CodelistIndexPolicy] = (),
    include_alt_labels: bool = True,
    carry_hierarchy_fields: bool = True,
    reference_metadata_blends: Sequence[
        ReferenceMetadataBlend
    ] = (),
    context_annotation_types: Sequence[str] | None = None,
    exclude_annotation_types: Sequence[str] | None = None,
    include_concepts: bool = True,
    embed_concepts: bool = True,
    concept_scheme_urns: Sequence[str] | None = None,
    include_reference_metadata: bool = False,
    embed_reference_metadata: bool = True,
    reference_metadataflow_urns: Sequence[str]
    | None = None,
    document_prefix: str | None = None,
    query_prefix: str | None = None,
    model_name: str | None = None,
) -> int

Build the search index now, so view can search before the seal.

seal rebuilds the index from the edition's rows with the arguments of the last call here (the defaults when there was none), so the sealed index always holds the sealed rows.

With model_name, a text whose text_sha256 (model name, document prefix, text) matches a stored vector takes that vector, so only new or changed texts reach the encoder, and a seal with unchanged text encodes nothing. Without model_name every text is encoded, since the encoder may be a new model.

Parameters:

Name Type Description Default
embedding_model EmbeddingModel | None

The encoder; the working copy's when None.

None
progress Callable[[int, int], None] | None

Called with (current, total) while texts are encoded.

None
include_codes bool

Index SDMX code rows.

True
embed_codes bool

Embed code rows too (slow on large catalogs).

False
max_codes_per_codelist int

Leave out codelists larger than this.

500
codelist_urns Sequence[str] | None

Index only these codelists' codes.

None
codelist_policies Sequence[CodelistIndexPolicy]

Per-codelist overrides (CodelistIndexPolicy).

()
include_alt_labels bool

Fold skos:altLabel annotations into code text.

True
carry_hierarchy_fields bool

Record each code's parent and depth.

True
reference_metadata_blends Sequence[ReferenceMetadataBlend]

Reference-metadata attributes folded into code text.

()
context_annotation_types Sequence[str] | None

Annotation types folded into code text; statcat:searchContext when None.

None
exclude_annotation_types Sequence[str] | None

Annotation types that keep a code out; statcat:searchExclude when None.

None
include_concepts bool

Index SDMX concept rows.

True
embed_concepts bool

Embed concept rows.

True
concept_scheme_urns Sequence[str] | None

Index only these schemes' concepts.

None
include_reference_metadata bool

Index reference-metadata sections as rows.

False
embed_reference_metadata bool

Embed reference-metadata rows.

True
reference_metadataflow_urns Sequence[str] | None

Index only sets of these structures.

None
document_prefix str | None

Prepended to every document before embedding.

None
query_prefix str | None

Prepended to every query before embedding.

None
model_name str | None

The model id recorded in embeddings.json and hashed into each row's text_sha256. Carried vectors are reused only when it is given; when None, every text is encoded and hashed under the carried name.

None

Returns:

Type Description
int

The number of datasets indexed.

close

close() -> None

Close the working copy and leave its edition unsealed, for working_copy(resume=True).

A no-op once it is sealed, discarded or closed.

discard

discard() -> None

Delete the unsealed edition (sdmxlib.edition.discard) and close the working copy.

seal

seal(
    checks: Iterable[
        Callable[[EditionView], Iterable[Finding]]
    ] = (),
) -> Edition

Seal the working copy into an edition and record it in the Inventory.

The steps: settle the journal; run checks; refuse when the base is no longer live; rebuild the search index; compute the statcat section; close; seal with sdmxlib (journal entry, then manifest), passing the partial_of of each partial statcat._meta names and the store still holds; run the publish check on a structure edition, or on an edition whose partials record a partial_of, with every other structure product's live and embargoed editions; record the edition, live when the catalog's embargo is off; settle the journal entry. Catalog.seal runs the same steps for several working copies and records them in one Inventory version.

Parameters:

Name Type Description Default
checks Iterable[Callable[[EditionView], Iterable[Finding]]]

Seal checks (statcatlib.checks.SealCheck), run against view before anything is written. Any finding refuses the seal and leaves the working copy open.

()

Returns:

Type Description
Edition

The edition.

Raises:

Type Description
EditionCheckError

The publish check found something, such as a reference no other structure product's live or embargoed edition resolves, or a partial whose codes its full codelist lacks, or one with no structure edition in the catalog to check it against; the sealed edition stays on disk unrecorded.

InventoryRuleError

sdmxlib's rules refused the record after the manifest was written, such as NonFastForwardError when a rollback lands during the seal; the sealed edition stays on disk unrecorded.

InventoryContentionError

The record lost too many races; the product's next working_copy() records the edition.

InventoryWriteRefusedError

The catalog has opted in to expiry, or the newest Inventory version's time is more than 5 minutes ahead of this machine's clock. Raised from the pre-check, it leaves the working copy open; raised from the record, it keeps the journal entry for a later working_copy().

Before anything is written, a check's finding raises SealCheckError (for a structure product, also each dataflow, binding or partial it holds; for a data product, a fill that stopped before it recorded its cuts, or a partial cut outside fill_dataflows that would carry the previous edition's partial_of), and a base that is no longer the live edition raises NonFastForwardError; both leave the working copy open. A ManifestError means sdmxlib refused the edition, such as for a coded value outside its codelist, before writing the journal entry; the working copy is closed, and working_copy(resume=True) reopens it.

EditionView

EditionView(
    *args: object,
    _store: _Store | None = None,
    **kwargs: object,
)

A read-only view of one edition: its datasets, codes, concepts, search and facets.

Product.view returns one over the product's live edition, and WorkingCopy.view one over the working copy, which sees the working copy's writes as they are made. It has the query methods, scopes and projections, and no write method.

A view stays on its edition. A process that follows the live edition calls product.view() again after Catalog.refresh.

Refuse construction from outside, with a TypeError: Product.view and WorkingCopy.view return views.

registry property

registry: EditionRegistry

The sdmxlib read surface over the same edition: get, query, obs.

A sealed edition's sdmxlib.EditionRegistry; for a working copy's view, its build registry.

index_annotations property

index_annotations: IndexAnnotations | None

The annotation types the search index was built from, and their fingerprint.

index_layout property

index_layout: dict[str, LanguageLayout]

What each language's search table was built with: rows and stemmer.

embedding_metadata property

embedding_metadata: EmbeddingMetadata | None

The embedding model and settings the search index was built with, or None.

embedding_model_name property

embedding_model_name: str | None

The model id that produced the vector index, if known.

document_prefix property

document_prefix: str

Prefix prepended to documents at index time (e.g. "passage: ").

query_prefix property

query_prefix: str

Prefix prepended to queries at search time (e.g. "query: ").

themes property

themes: ThemesScope

Lazy scope over theme classification schemes: view.themes["subjects"].

concept_schemes property

concept_schemes: ConceptSchemesScope

Lazy scope over SKOS concept schemes: view.concept_schemes["uri"].

codes property

codes: CodesScope

Typed search and lookup over SDMX codes: view.codes.find("Toronto").

concepts property

concepts: ConceptsScope

Typed search and lookup over SDMX concepts: view.concepts.find("Gini").

datasets property

datasets: DatasetsScope

Typed dataset search: view.datasets.find(q, mode="hybrid").

reference_metadata property

reference_metadata: ReferenceMetadataScope

Typed search over indexed SDMX reference-metadata sections.

codelists property

codelists: CodelistsScope

Every codelist the edition holds.

check_index_compatible

check_index_compatible(
    *,
    model_name: str | None = None,
    embedding_dim: int | None = None,
) -> None

Raise IncompatibleIndexError when the index was built with another model or width.

Parameters:

Name Type Description Default
model_name str | None

The encoder id the caller loaded.

None
embedding_dim int | None

The width of the caller's vectors.

None

attach_embedding_model

attach_embedding_model(model: EmbeddingModel) -> None

Attach the encoder embedding_model_name names, for vector and hybrid search.

Parameters:

Name Type Description Default
model EmbeddingModel

The encoder.

required

dataset

dataset(identifier: str) -> DatasetScope

Scope for one dataset: its codelists, themes and lineage.

Parameters:

Name Type Description Default
identifier str

The dataset's identifier.

required

Returns:

Type Description
DatasetScope

The scope.

codelist

codelist(urn: str) -> CodelistScope

Scope for one codelist, by URN.

Parameters:

Name Type Description Default
urn str

The codelist's URN.

required

Returns:

Type Description
CodelistScope

The scope.

dimensions

dimensions(ds: AggregateDataset) -> DimensionsScope

Scope over a dataset's dimensions.

Parameters:

Name Type Description Default
ds AggregateDataset

The dataset.

required

Returns:

Type Description
DimensionsScope

The scope.

get

get(cls: type[T], identifier: str) -> T | None

The dataset identifier of type cls, with every relationship loaded, or None.

Parameters:

Name Type Description Default
cls type[T]

The dataset class to narrow to.

required
identifier str

The dataset's identifier.

required

Returns:

Type Description
T | None

The dataset, or None.

query

query(cls: type[T], **filters: object) -> QueryBuilder[T]

A structured query over datasets of cls, filtered by keyword.

Parameters:

Name Type Description Default
cls type[T]

The dataset class to narrow to.

required
**filters object

Field filters, as QueryBuilder.filter takes them.

{}

Returns:

Type Description
QueryBuilder[T]

The query builder, for chaining.

search

search(
    text: str,
    *,
    kinds: Sequence[str] = ("dataset", "concept", "code"),
    lang: str = "en",
    limit: int = 50,
    offset: int = 0,
    operator: str = "or",
    fuzzy: int = 0,
    rrf_k: int = 60,
    per_kind_oversample: int = 100,
) -> SearchResult[AnyHit]

Search every indexed entity kind, RRF-merged.

Parameters:

Name Type Description Default
text str

The query text.

required
kinds Sequence[str]

The entity kinds to search.

('dataset', 'concept', 'code')
lang str

The language whose table each kind is searched in.

'en'
limit int

Hits to return.

50
offset int

Hits to skip.

0
operator str

"or" or "and" between query terms.

'or'
fuzzy int

Edit distance for fuzzy term matching.

0
rrf_k int

The reciprocal-rank-fusion constant.

60
per_kind_oversample int

Hits fetched per kind before the merge.

100

Returns:

Type Description
SearchResult[AnyHit]

The merged hits, best first.

facets

facets(*fields: str) -> dict[str, dict[str, int]]

Counts per value of each field, across every dataset.

Parameters:

Name Type Description Default
*fields str

The facet fields.

()

Returns:

Type Description
dict[str, dict[str, int]]

{field: {value: count}}.

agent_labels

agent_labels(lang: str = 'en') -> dict[str, str]

Each agent's name in lang, by agent id.

Parameters:

Name Type Description Default
lang str

The language tag.

'en'

Returns:

Type Description
dict[str, str]

The labels, by agent id.

concept_labels

concept_labels(
    lang: str = "en", *, scheme_uri: str | None = None
) -> dict[str, str]

Each concept's label in lang, by concept URI.

Parameters:

Name Type Description Default
lang str

The language tag.

'en'
scheme_uri str | None

Restrict to one concept scheme.

None

Returns:

Type Description
dict[str, str]

The labels, by concept URI.

spatial_labels

spatial_labels(lang: str = 'en') -> dict[str, str]

Each spatial facet key's label in lang.

Parameters:

Name Type Description Default
lang str

The language tag.

'en'

Returns:

Type Description
dict[str, str]

The labels, by facet key.

get_concept_scheme

get_concept_scheme(uri: str) -> ConceptScheme | None

A concept scheme with its concepts, or None.

Parameters:

Name Type Description Default
uri str

The scheme's URI.

required

Returns:

Type Description
ConceptScheme | None

The scheme, or None.

get_concept

get_concept(uri: str) -> Concept | None

One concept, or None.

Parameters:

Name Type Description Default
uri str

The concept's URI.

required

Returns:

Type Description
Concept | None

The concept, or None.

code_map

code_map(
    dataset_id: str, lang: str = "en"
) -> dict[str, dict[str, str]]

{dimension_id: {code_id: label}} for a dataset.

Parameters:

Name Type Description Default
dataset_id str

The dataset's identifier.

required
lang str

The language of the labels.

'en'

Returns:

Type Description
dict[str, dict[str, str]]

The labels, by dimension and code.

codelist_for

codelist_for(
    dataset_id: str, dimension: str
) -> sdmxlib.model.codelist.Codelist | None

The codelist of one dimension of a dataset, or None.

Parameters:

Name Type Description Default
dataset_id str

The dataset's identifier.

required
dimension str

The dimension's id.

required

Returns:

Type Description
Codelist | None

The codelist, or None.

get_structure

get_structure(
    ds: AggregateDataset,
) -> sdmxlib.model.datastructure.DataStructure | None

The data structure of an AggregateDataset, or None.

Parameters:

Name Type Description Default
ds AggregateDataset

The dataset.

required

Returns:

Type Description
DataStructure | None

The data structure, or None.

metadata_for

metadata_for(
    target_urn: str | SdmxUrn[object],
    *,
    structure_urn: str | SdmxUrn[object] | None = None,
) -> list[sdmxlib.MetadataSet]

Every reference-metadata set attached to target_urn.

Parameters:

Name Type Description Default
target_urn str | SdmxUrn[object]

The artefact URN.

required
structure_urn str | SdmxUrn[object] | None

Restrict to sets of this MSD or metadataflow.

None

Returns:

Type Description
list[MetadataSet]

The sets, in insertion order.

metadata_report

metadata_report(
    set_urn: str | SdmxUrn[object],
) -> list[sdmxlib.MetadataSection]

The sections of a reference-metadata set, typed by its MSD.

Parameters:

Name Type Description Default
set_urn str | SdmxUrn[object]

The set's URN.

required

Returns:

Type Description
list[MetadataSection]

One section per reported value.

metadata_targets

metadata_targets(
    set_urn: str | SdmxUrn[object],
) -> list[str]

The artefact URNs a reference-metadata set describes.

Parameters:

Name Type Description Default
set_urn str | SdmxUrn[object]

The set's URN.

required

Returns:

Type Description
list[str]

The URNs, in order.

language_alternates

language_alternates() -> list[Row]

The language tags recorded as alternates of a primary tag.

Queries

QueryBuilder

QueryBuilder(
    store: _Store,
    cls: type[T],
    *,
    filters: dict[str, object] | None = None,
    exists_filters: list[PositionalFilter] | None = None,
    map_filters: list[MapFilter] | None = None,
    from_ids: list[str] | None = None,
    text: str | None = None,
    hybrid_text: str | None = None,
    search_operator: str = "or",
    search_fuzziness: int = 0,
    search_lang: str = "en",
    sort_fields: list[_SortSpec] | None = None,
    limit: int | None = None,
    offset: int = 0,
    facet_fields: list[str] | None = None,
    select_cols: tuple[str, ...] | None = None,
    select_lang: str | None = None,
    scalar_col: str | None = None,
    include_rels: frozenset[str] | None = None,
)

Chainable query builder. Terminal method: .execute() or .all().

filter

filter(
    *pos_filters: PositionalFilter | MapFilter,
    **kw_filters: object,
) -> QueryBuilder[T]

Add filter criteria. Returns a new builder (immutable chain).

from_ids

from_ids(ids: list[str]) -> QueryBuilder[T]

Restrict results to a pre-computed list of identifiers.

search

search(
    text: str,
    *,
    lang: str = "en",
    operator: str = "or",
    fuzzy: int = 0,
) -> QueryBuilder[T]

Add full-text search. Requires LanceDB.

Parameters:

Name Type Description Default
text str

The search text. Tokenised and stemmed with lang's rules before matching.

required
lang str

The language whose search table answers the query. Each language has its own table, so a French query needs lang="fr". A language with no table matches nothing.

'en'
operator str

How to join multi-term queries: "or" (default) matches any term; "and" requires all terms.

'or'
fuzzy int

Levenshtein edit distance for fuzzy matching: 0 (default, exact match), 1, or 2.

0

Returns:

Type Description
QueryBuilder[T]

A new QueryBuilder with the search text applied. The original is

QueryBuilder[T]

unchanged — builders are immutable.

hybrid_search(
    text: str, *, lang: str = "en"
) -> QueryBuilder[T]

Add hybrid BM25 + semantic search. Requires LanceDB.

Parameters:

Name Type Description Default
text str

The search text.

required
lang str

The language whose search table answers both the BM25 and the vector query, as for search.

'en'

Returns:

Type Description
QueryBuilder[T]

A new QueryBuilder with the search text applied.

sort_by

sort_by(
    field: str,
    *,
    desc: bool = False,
    nulls: str | None = None,
    lang: str | None = None,
) -> QueryBuilder[T]

Add a sort criterion.

For MAP columns like title, pass lang to specify the language key (e.g. lang="en"). Raises ValueError if lang is omitted for a MAP column.

limit

limit(n: int) -> QueryBuilder[T]

Set maximum number of results.

offset

offset(n: int) -> QueryBuilder[T]

Set pagination offset.

with_facets

with_facets(*fields: str) -> QueryBuilder[T]

Request facet counts alongside the results.

include

include(*relationships: str) -> QueryBuilder[T]

Declare which relationships to batch-load during hydration.

By default all relationships are loaded (~17 queries). Calling include() switches to selective loading: only the named relationships are fetched; everything else stays at its model default (None / []).

Subsequent include() calls accumulate — they do not replace the previous set. Use full() to revert to loading everything.

Valid names mirror the model field names::

store.query(BaseDataset)
    .include("publisher", "themes", "keywords")
    .collect()   # 3 batch loads instead of 17

Raises ValueError for unrecognised relationship names.

full

full() -> QueryBuilder[T]

Load all relationships (the default behaviour).

Useful to explicitly opt into full hydration after a partial include() call, or to make the cost visible at the call site::

store.query(BaseDataset).filter(...).full().collect()

select

select(
    *columns: str, lang: str | None = None
) -> QueryBuilder[T]

Set dict projection mode. Terminals return dicts instead of objects.

lang is required when selecting MAP columns (title, description). Raises ValueError for unknown column names.

scalars

scalars(
    column: str, *, lang: str | None = None
) -> QueryBuilder[T]

Set single-column flat projection mode.

Terminals return bare values instead of objects or dicts.

execute

execute() -> SearchResult[T]

Execute the query and return a SearchResult with metadata.

Only available in entity mode (no .select() / .scalars()).

collect

collect() -> list[T]

Execute and return domain objects.

This is the preferred terminal for entity results — consistent with scope terminals (.collect() on SchemeScope, CodelistScope, etc.). Use .rows() for Row projections or .values() for scalar projections.

all

all() -> list[T]

Alias for .collect().

first

first() -> T | None

Execute with limit=1 and return the first domain object or None.

one

one() -> T

Execute and return exactly one domain object; raise on 0 or >1.

rows

rows() -> list[Row]

Execute and return Row objects (requires .select()).

row

row() -> Row | None

Execute with limit=1 and return the first Row or None.

values

values() -> list[object]

Execute and return flat scalar values (requires .scalars()).

value

value() -> object | None

Execute with limit=1 and return the first scalar value or None.

count

count() -> int

Return the total count without fetching items.

exists

exists() -> bool

Return True if at least one result matches the filters.

SearchResult

Container for query results with pagination metadata.

Search hits

Hit

A search hit — payload plus per-match ranking metadata.

The discriminator kind lets a heterogeneous result list (e.g. EditionView.search over all entity kinds) be narrowed at runtime by a tagged match; the generic parameter T lets type-scoped accessors (cat.codes.find etc.) narrow statically.

Per-hit metadata that depends on the query (not the entity) lives here, not on payload:

  • score — backend-native score (BM25 relevance, vector similarity, or fused RRF).
  • source — which retrieval path produced this hit; useful for hybrid-search explainability and telemetry.
  • highlights — per-field snippet extracts when available.
  • matched_field — which indexed column drove the rank.

CodeHit

Payload for a code-search hit.

Score, language, and ranking metadata live on the enclosing Hit; this type is the pure structural projection of one code row — exactly what an LLM agent translating a phrase like "Ontario" into a (dimension_id, code_id) filter needs.

ConceptHit

Payload for a concept-search hit.

SDMX Concepts carry definitional text (description) which is where vector search earns its cost — a query like "income inequality" finds "Gini coefficient" through its definition, not its label. The hit exposes both the structural identifiers (URNs, scheme, id) and the localised text.

ReferenceMetadataHit

Payload for a reference-metadata (methodology-document) search hit.

One indexed chunk of a sectioned SDMX MetadataSet — a reference guide, a methodology report, a data-quality note. Unlike a code hit (a short controlled-vocabulary label), this is prose: the vector/BM25 target is the section text itself. The hit carries the identifiers needed to navigate back to the structured graph — set_urn (the document) and attribute_path (the section) — so a caller can fetch the full, typed section via EditionView.metadata_report or resolve the artefacts the document describes via the set's targets.

ReferenceMetadataBlend

Blend a reference-metadata attribute into a code's search text.

For each code, statcatlib joins sdmx.metadata_target → sdmx.metadata_value to pull the localised value of attribute_path and append it to the code's search text. Improves BM25 + vector recall on definitional queries when the catalog publishes concept definitions or footnotes via SDMX MetadataSets (e.g. an MSD carrying a FOOTNOTE_TEXT attribute).

Attributes:

Name Type Description
attribute_path str

Dotted MSD attribute id, e.g. "FOOTNOTE_TEXT".

max_chars int

Per-code truncation applied after joining multiple values with a single space. Defaults to 240 — long enough to carry one or two sentences without overwhelming the BM25 term mix or the embedding.

Projections and selective loading

Row

Bases: dict[str, object]

A dict subclass returned by .select() projections.

Supports both dict-style and attribute-style field access::

row = store.query(BaseDataset).select("identifier", "status").first()
row["identifier"]  # dict-style
row.identifier  # attribute-style

Row is a plain dict subclass — isinstance(row, dict) is True, JSON serialisation works as normal, and all dict methods (keys(), items(), values(), etc.) are inherited.

is_unloaded

is_unloaded(value: object) -> bool

Return True if value is an UNLOADED sentinel.

NotLoadedError

NotLoadedError(field: str)

Bases: AttributeError

Raised when accessing a relationship that was not loaded in this query.

Parameters:

Name Type Description Default
field str

The relationship that was not loaded, as include names it.

required
Example

ds = store.query(BaseDataset).include("publisher").first() ds.themes # NotLoadedError: 'themes' was not loaded.

Add 'themes' to .include() or call .full().

Search configuration

EmbeddingMetadata

Read-side view of an edition's search/embeddings.json sidecar.

Returned by EditionView.embedding_metadata. Fields that were not present in the on-disk sidecar (e.g. when reading a v2 file with a v3 reader) come back as None rather than guesses.

CodelistIndexPolicy

Per-codelist override for WorkingCopy.build_index.

Lets a caller opt large codelists into the Lance search index even when their size would otherwise trip max_codes_per_codelist, and decouple "should this codelist be indexed" from "should its codes get vector embeddings". Short proper-noun labels (geography, population characteristics) score better with Tantivy FTS than with vector similarity over short strings, and skipping the encode is what makes indexing a geography codelist of tens of thousands of codes take seconds rather than minutes, at a fraction of the disk.

Attributes:

Name Type Description
urns tuple[str, ...]

Codelist URNs this policy applies to.

mode str

How the listed codelists should be indexed:

  • "fts_only" — bypass max_codes_per_codelist; rows are written to Lance with embedding=None regardless of the global embed_codes flag, so Tantivy FTS still finds them.

Errors

IncompatibleCatalogError

Bases: RuntimeError

Raised when a catalog or an edition is in a layout or at a version this build does not read.

The cases: a directory holding a statcatlib 0.11 catalog, a 0.72 bundle or other files (statcatlib.catalog); an edition whose manifest.json is not the one the Inventory recorded; a statcat schema written by a statcatlib with another schema.SCHEMA_VERSION (_refuse_statcat_schema_mismatch); a search/index.json at another manifest version (_refuse_unreadable_index); and a filesystem without hard links under Catalog.create. An edition on another sdmxlib store schema is refused by sdmxlib itself, with sdmxlib.StoreSchemaError.

IncompatibleIndexError

Bases: ValueError

Raised by EditionView.check_index_compatible on mismatch.

Subclasses ValueError so existing broad-catch sites still work.

NoLiveEditionError

Bases: LookupError

The product has no live edition, so there is nothing for a public reader to view.

WorkingCopyExistsError

Bases: Exception

The product already has an unsealed edition; a product has one at a time.

WorkingCopyClosedError

Bases: RuntimeError

The working copy was sealed or discarded, so neither it nor its views answer.

SealCheckError

SealCheckError(findings: Sequence[Finding])

Bases: Exception

A seal check found something wrong, so the edition was not sealed.

Attributes:

Name Type Description
findings tuple[Finding, ...]

Everything the checks found.

List the findings.

EditionCheckError

EditionCheckError(
    findings: Sequence[Finding],
    *,
    product: str,
    edition: str,
)

Bases: Exception

The publish check found something wrong with a sealed edition, so it was not recorded.

The edition stays on disk, sealed and unrecorded, and statcat doctor reports it. Fix the source, then start a new working copy from the live edition with working_copy().

Attributes:

Name Type Description
findings tuple[Finding, ...]

Everything sdmxlib.edition.check_edition found.

product str

The product.

edition str

The sealed edition.

List the findings, and name the product and the edition.

JournalError

Bases: Exception

A seal journal entry cannot be read, so the product's seals cannot be settled.

InventoryRuleError

InventoryRuleError(
    message: str,
    *,
    rule: Rule | Literal["lifecycle"] | CrossRule,
    product: str,
    edition: str | None,
)

Bases: Exception

A change sdmxlib's Inventory rules refuse, or a lifecycle change statcatlib refuses.

Attributes:

Name Type Description
rule Rule | Literal['lifecycle'] | CrossRule

The rule broken: one of sdmxlib's Rule values, "kind" also for a product reopened with another kind, "lifecycle" for a product reopened with another lifecycle, or a cross-product rule (CrossProductRuleError): "full-artefacts", "structure-home" or "publish-order".

product str

The product.

edition str | None

The edition, where one is concerned.

Say why, and name the rule, the product and the edition.

wrapping classmethod

wrapping(
    exc: TransitionError,
    *,
    product: str,
    edition: str | None,
) -> InventoryRuleError

The error for sdmxlib's refusal exc, which becomes its __cause__ when raised.

A fast-forward refusal becomes a NonFastForwardError.

NonFastForwardError

NonFastForwardError(
    message: str,
    *,
    rule: Rule | Literal["lifecycle"] | CrossRule,
    product: str,
    edition: str | None,
)

Bases: InventoryRuleError

An edition built on an edition that is no longer live.

The message names the edition, its base, the live edition, and the fix: discard the working copy, then start one from the live edition.

CrossProductRuleError

CrossProductRuleError(
    message: str,
    *,
    rule: CrossRule,
    item: Literal[4, 7, 8],
    urn: str,
    editions: Sequence[tuple[str, str]],
    product: str,
    edition: str | None,
)

Bases: InventoryRuleError

A version that breaks a cross-product rule, specification §6.5 item 4, 7 or 8.

product and edition name the incoming entry, the one that becomes live or embargoed in the version. The message names the item, the URN, the editions compared and the fix. rule is "full-artefacts" (item 4: a data edition's full artefact or partial matches no live or embargoed edition of its structure home, a cut codelist has no home, or a full artefact with no home differs from another data product's live copy), "structure-home" (item 7: a second structure home among live editions) or "publish-order" (item 8: a data edition entering live before its structure edition).

Attributes:

Name Type Description
item Literal[4, 7, 8]

The specification §6.5 item: 4, 7 or 8.

urn str

The artefact concerned.

editions tuple[tuple[str, str], ...]

(product, edition) pairs: the incoming entry first, then each edition it was compared with.

Say why, and name the rule, the item, the URN and the editions compared.

of classmethod

of(refusal: Refusal) -> CrossProductRuleError

The error for refusal, from statcatlib.store.cross_product.refusals.

InventoryConflictError

Bases: Exception

Another writer created the Inventory version this one tried to create.

InventoryWriter.try_commit raises it; InventoryWriter.commit catches it, re-reads the head and tries again.

InventoryContentionError

Bases: Exception

A commit lost MAX_RACES races in a row to other writers, and gave up.

InventoryWriteRefusedError

Bases: InventoryError

This writer must not write the next Inventory version, whatever the version holds.

The catalog has opted in to expiry (catalog.json 2.x), which needs a writer with the catalog lock, or the base version's created_at is more than 5 minutes ahead of this machine's clock. The message names the format, or the base version's time and this machine's. It stays refused until the catalog or the clock changes, so a seal refused after its manifest keeps its journal entry, as InventoryContentionError does.