Catalog¶
Generated from the docstrings. Catalog API walks through the same surface with examples.
The catalog model¶
Catalog ¶
A directory of products, each a chain of immutable editions (dcat:Catalog).
Create one with create, which also reopens an existing catalog for
writing, or open one read-only with open. A catalog reads its Inventory
through sdmxlib.inventory.LiveInventory, so it sees live editions only;
refresh re-reads it. Every view it returns shares one DuckDB engine,
and close releases them all.
Queries run on a product's view: cat.product(id).view().query(...).
Search and facets across products are not yet supported.
Refuse construction from outside, with a TypeError naming open, create and temporary_catalog.
writable
property
¶
Whether the catalog came from create, so its products can be written.
description
property
¶
The catalog's description, as of the last refresh; None when none was recorded.
CatalogAdmin.describe records it in the Inventory root's
extensions.statcat, and every later version carries it. It holds
the catalog-level fields only; the datasets come from the products'
live editions.
create
classmethod
¶
Create a catalog at path, or with exist_ok reopen the one there for writing.
A missing or empty directory gets catalog.json, holding a UUIDv7
catalog_id and restore_epoch 0, and nothing else: the first
seal writes Inventory version 1. The path created is logged at INFO,
since a mistyped path with exist_ok=True creates a new, empty
catalog.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str | Path
|
The catalog directory. |
required |
exist_ok
|
bool
|
Reopen an existing catalog for writing, writing nothing. |
False
|
Returns:
| Type | Description |
|---|---|
Self
|
The catalog, writable: its products' |
Raises:
| Type | Description |
|---|---|
FileExistsError
|
path is a catalog and exist_ok is false. |
A legacy layout or other files at path, or a filesystem without
hard links, raise IncompatibleCatalogError.
open
classmethod
¶
Open the catalog at path read-only.
Reads catalog.json and the Inventory, and writes nothing: no
repair, no DDL. A catalog with no Inventory version yet has no
products.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str | Path
|
The catalog directory. |
required |
Returns:
| Type | Description |
|---|---|
Self
|
The catalog. Its products' |
Self
|
with |
Raises:
| Type | Description |
|---|---|
FileNotFoundError
|
path is not a catalog. |
A legacy layout or other files at path raise IncompatibleCatalogError.
refresh ¶
Re-read the Inventory, so product and products see the newest live editions.
A view taken before stays on its edition; call product(id).view()
again to follow the live edition.
product ¶
The product product_id, which need not exist yet.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
product_id
|
str
|
The product's id. It names a directory: letters,
digits, |
required |
Returns:
| Type | Description |
|---|---|
Product
|
The product. An invalid product_id raises |
seal ¶
seal(
working_copies: Iterable[WorkingCopy],
*,
checks: Iterable[
Callable[[EditionView], Iterable[Finding]]
] = (),
) -> SealOutcomes
Seal several products' working copies and record them in one Inventory version.
It runs WorkingCopy.seal's steps in four passes. The structure pass
pre-checks each structure product's working copy (journal, checks,
content, fast-forward) and seals each that passed. The
structure-check pass runs each sealed structure edition's publish
check, given the batch's other structure editions in place of their
products' recorded ones, and drops each edition with a finding until
a pass drops none. The data pass pre-checks every data product's
working copy before any of them is sealed, then seals each that
passed, and runs the publish check of each data edition with
partials, given the batch's structure editions in place of their
products' recorded ones. Last it records every sealed edition in one
Inventory commit, leaving out each product whose record sdmxlib's
rules refuse, and each edition whose publish check was given a
left-out one. A refusal in one product never stops another's seal.
A crash during the batch can leave its products recorded across
several Inventory versions: each sealed, unrecorded edition keeps its
journal entry, and its product's next working_copy() records it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
working_copies
|
Iterable[WorkingCopy]
|
The working copies, one per product, each from this catalog. |
required |
checks
|
Iterable[Callable[[EditionView], Iterable[Finding]]]
|
Seal checks ( |
()
|
Returns:
| Type | Description |
|---|---|
SealOutcomes
|
Each product's outcome. An empty batch writes nothing. |
Before anything is sealed, a working copy of another catalog, or two
of one product, raise ValueError, and a sealed, discarded or closed
working copy raises WorkingCopyClosedError. A catalog opened
read-only with Catalog.open raises RuntimeError.
search ¶
Not yet supported across products: raises NotImplementedError; search one product's view.
facets ¶
Not yet supported across products: raises NotImplementedError; count one product's view.
__getattr__ ¶
Name the replacement of a query method that moved to EditionView.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
The attribute asked for. |
required |
Raises:
| Type | Description |
|---|---|
AttributeError
|
Always; for a moved method, naming
|
Product ¶
One product of a catalog: its live edition, and its working copy.
Catalog.product returns one.
Refuse construction from outside, with a TypeError: Catalog.product returns products.
view ¶
A read-only view of the product's live edition, as of the catalog's last Catalog.refresh.
It attaches the edition to the catalog's engine with
LocalRegistry.edition, which reads manifest.json once, checks
its SHA-256 against the one the Inventory records, and checks its
store schema. The view stays on this edition after a refresh or a
rollback.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
embedding_model
|
EmbeddingModel | None
|
The encoder the index was built with, for vector and hybrid search. BM25 search needs none. |
None
|
query_prefix
|
str
|
Prepended to every query before embedding; the one the index recorded when empty. |
''
|
Returns:
| Type | Description |
|---|---|
EditionView
|
The view. |
Raises:
| Type | Description |
|---|---|
IncompatibleCatalogError
|
The edition's |
editions ¶
The product's live editions, each with the previous its manifest records.
The public catalog holds live editions only, one per store schema;
CatalogAdmin.editions lists every state.
working_copy ¶
working_copy(
*,
kind: Kind | None = None,
lifecycle: Lifecycle = "revisable",
resume: bool = False,
embedding_model: EmbeddingModel | None = None,
document_prefix: str = "",
query_prefix: str = "",
) -> WorkingCopy
Start the product's working copy on its live edition, or resume the unsealed one.
It settles the product's seal journal first (the journal rule, in
statcatlib.store.working_copy), so an edition a crash left sealed
and unrecorded is recorded before the new one picks its base.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
kind
|
Kind | None
|
|
None
|
lifecycle
|
Lifecycle
|
|
'revisable'
|
resume
|
bool
|
Reopen the product's unsealed edition in place of starting one. |
False
|
embedding_model
|
EmbeddingModel | None
|
The encoder |
None
|
document_prefix
|
str
|
Prepended to every document before embedding. |
''
|
query_prefix
|
str
|
Prepended to every query before embedding. |
''
|
Returns:
| Type | Description |
|---|---|
WorkingCopy
|
The working copy. |
A catalog opened read-only with Catalog.open raises RuntimeError.
A product recorded with another lifecycle raises InventoryRuleError
(rule="lifecycle"); one recorded with another kind, or resumed
with another kind than it was started with, raises it with
rule="kind" before anything is built; and one with an unsealed
edition raises WorkingCopyExistsError unless resume is true.
rollback ¶
Rollback is an operational change: raises AttributeError naming CatalogAdmin.rollback.
Edition ¶
A sealed edition, as the Inventory records it.
Attributes:
| Name | Type | Description |
|---|---|---|
edition_id |
str
|
The edition's id, a UUIDv7. |
product_id |
str
|
Its product. |
previous |
str | None
|
The edition it was built on, or |
manifest_sha256 |
str
|
The SHA-256 of its |
SealOutcomes ¶
What Catalog.seal did with each working copy, keyed by product id.
Every product of the batch is in exactly one of the two mappings.
Attributes:
| Name | Type | Description |
|---|---|---|
sealed |
Mapping[str, Edition]
|
Each recorded product's new edition. All of them were recorded in one Inventory version. |
refused |
Mapping[str, Exception]
|
Each refused product's error. A |
WorkingCopy ¶
A product's one unsealed edition, open for writing.
Product.working_copy returns one. Write datasets with put and the
other write methods, and SDMX structures and observations through
registry; read them back through view; then seal it, or discard it.
Once sealed or discarded, every method on it, and on its views, raises
WorkingCopyClosedError.
Refuse construction from outside, with a TypeError: Product.working_copy returns working copies.
previous
property
¶
The edition it was built on, or None for the product's first.
kind
property
¶
The product's kind: data, or structure for a product of full artefacts only.
registry
property
¶
The writable sdmxlib registry over the edition: SDMX structures and observations.
wc.registry.ingest(source) adds structures, and
wc.registry.write_observations(...) writes observations into the
edition's lake.
view ¶
A read-only view of this working copy, which sees its writes as they are made.
fill_structures ¶
fill_structures(
source: EditionRegistry,
*,
families: Collection[type[MaintainableArtefact]] = (),
agency: str | None = None,
ids: Collection[str] = (),
artefacts: Collection[
SdmxUrn[MaintainableArtefact] | str
] = (),
) -> tuple[str, ...]
Copy full artefacts from source into a structure product's working copy.
It calls sdmxlib.edition.fill_structures. An artefact is selected
when artefacts names its URN, or when its class is one of
families, its agency is agency (any when None) and its id
matches one of the ids patterns (SGC*; any id when empty).
Nothing is added for what a selected artefact references, so a
reference into another structure product stays open; the seal's
publish check resolves it there. An artefact the working copy holds at
a selected URN is replaced.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
EditionRegistry
|
The registry holding the full artefacts, such as a
scratch |
required |
families
|
Collection[type[MaintainableArtefact]]
|
The artefact classes to select, such as
|
()
|
agency
|
str | None
|
The maintenance agency of the artefacts families selects. |
None
|
ids
|
Collection[str]
|
Patterns the id of an artefact families selects must match. |
()
|
artefacts
|
Collection[SdmxUrn[MaintainableArtefact] | str]
|
More artefacts, by URN. |
()
|
Returns:
| Type | Description |
|---|---|
tuple[str, ...]
|
The URNs copied, sorted. |
Raises:
| Type | Description |
|---|---|
TypeError
|
The working copy is a data product's. |
sdmxlib raises TypeError for a family that is Dataflow,
DataflowBinding or a class the store does not hold, and
ValueError for a URN that names a dataflow, a binding, a partial or
nothing source holds, or when nothing is selected.
fill_dataflows ¶
fill_dataflows(
source: RestRegistry | EditionRegistry,
dataflows: Sequence[SdmxUrn[Dataflow] | str],
*,
observations: Mapping[str, WriteSource] | None = None,
partials: bool = True,
) -> dict[str, str]
Add dataflows' closures and observations to a data product's working copy, and cut its codelists.
It calls sdmxlib.edition.fill_edition, which keeps what the working
copy holds: on a republish the other dataflows' bindings stay, and a
codelist they share is cut to the codes every binding uses. The map
it returns is merged into statcat._meta.partial_of, a later
fill's value winning for a URN, so a seal after
working_copy(resume=True) in another process records each
partial's partial_of. statcat._meta.fill_in_progress is set
while sdmxlib fills; a fill that stops before the merge leaves it, and
the seal refuses the working copy until a fill completes.
Fill a data product through this method: a direct
fill_edition(wc.registry, ...) leaves its cuts out of the map.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
RestRegistry | EditionRegistry
|
Where the structures are: a |
required |
dataflows
|
Sequence[SdmxUrn[Dataflow] | str]
|
The dataflows, by URN. A |
required |
observations
|
Mapping[str, WriteSource] | None
|
The rows of each dataflow that has any, by URN
string, in any form |
None
|
partials
|
bool
|
Whether to cut codelists to partials. |
True
|
Returns:
| Type | Description |
|---|---|
dict[str, str]
|
The fingerprint of the full codelist each partial cut here was |
dict[str, str]
|
cut from, by URN. |
Raises:
| Type | Description |
|---|---|
TypeError
|
The working copy is a structure product's. |
sdmxlib raises TypeError for a URN that names no dataflow at
source, and ValueError for a dataflow with no data structure, for
observations naming a dataflow not in dataflows, and for a coded
column holding a value outside its codelist.
put ¶
Add or update a dataset, a concept scheme, or a sequence of either.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
obj
|
Dataset | ConceptScheme | Sequence[Dataset | ConceptScheme]
|
What to write. |
required |
put_concept ¶
Add or update one concept of a scheme.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
scheme_uri
|
str
|
The scheme's URI. |
required |
concept
|
Concept
|
The concept. |
required |
add_lineage ¶
add_lineage(
source_id: str,
target_id: str,
relationship: str,
*,
activity_label: str | None = None,
activity_type: str | None = None,
agent_id: str | None = None,
timestamp: datetime | None = None,
confidence: str = "asserted",
notes: str | None = None,
) -> None
Record a PROV-O-aligned lineage edge between two datasets.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source_id
|
str
|
The source dataset. |
required |
target_id
|
str
|
The target dataset. |
required |
relationship
|
str
|
One of |
required |
activity_label
|
str | None
|
The activity's label. |
None
|
activity_type
|
str | None
|
The activity's type. |
None
|
agent_id
|
str | None
|
The agent responsible. |
None
|
timestamp
|
datetime | None
|
When. |
None
|
confidence
|
str
|
One of |
'asserted'
|
notes
|
str | None
|
Free text. |
None
|
add_language_alternates ¶
Record language tags a source declares as alternates of a primary tag.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pairs
|
Mapping[str, Sequence[str]]
|
Each primary tag mapped to its alternate tags. |
required |
source
|
str
|
Where the pairs come from. |
required |
bulk_load ¶
Hold DuckDB's write-ahead log until the block ends, then checkpoint once.
transaction ¶
Wrap the writes in the block in one DuckDB transaction.
build_index ¶
build_index(
embedding_model: EmbeddingModel | None = None,
progress: Callable[[int, int], None] | None = None,
*,
include_codes: bool = True,
embed_codes: bool = False,
max_codes_per_codelist: int = 500,
codelist_urns: Sequence[str] | None = None,
codelist_policies: Sequence[CodelistIndexPolicy] = (),
include_alt_labels: bool = True,
carry_hierarchy_fields: bool = True,
reference_metadata_blends: Sequence[
ReferenceMetadataBlend
] = (),
context_annotation_types: Sequence[str] | None = None,
exclude_annotation_types: Sequence[str] | None = None,
include_concepts: bool = True,
embed_concepts: bool = True,
concept_scheme_urns: Sequence[str] | None = None,
include_reference_metadata: bool = False,
embed_reference_metadata: bool = True,
reference_metadataflow_urns: Sequence[str]
| None = None,
document_prefix: str | None = None,
query_prefix: str | None = None,
model_name: str | None = None,
) -> int
Build the search index now, so view can search before the seal.
seal rebuilds the index from the edition's rows with the arguments
of the last call here (the defaults when there was none), so the
sealed index always holds the sealed rows.
With model_name, a text whose text_sha256 (model name, document
prefix, text) matches a stored vector takes that vector, so only new
or changed texts reach the encoder, and a seal with unchanged text
encodes nothing. Without model_name every text is encoded, since
the encoder may be a new model.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
embedding_model
|
EmbeddingModel | None
|
The encoder; the working copy's when |
None
|
progress
|
Callable[[int, int], None] | None
|
Called with |
None
|
include_codes
|
bool
|
Index SDMX code rows. |
True
|
embed_codes
|
bool
|
Embed code rows too (slow on large catalogs). |
False
|
max_codes_per_codelist
|
int
|
Leave out codelists larger than this. |
500
|
codelist_urns
|
Sequence[str] | None
|
Index only these codelists' codes. |
None
|
codelist_policies
|
Sequence[CodelistIndexPolicy]
|
Per-codelist overrides ( |
()
|
include_alt_labels
|
bool
|
Fold |
True
|
carry_hierarchy_fields
|
bool
|
Record each code's parent and depth. |
True
|
reference_metadata_blends
|
Sequence[ReferenceMetadataBlend]
|
Reference-metadata attributes folded into code text. |
()
|
context_annotation_types
|
Sequence[str] | None
|
Annotation types folded into code text;
|
None
|
exclude_annotation_types
|
Sequence[str] | None
|
Annotation types that keep a code out;
|
None
|
include_concepts
|
bool
|
Index SDMX concept rows. |
True
|
embed_concepts
|
bool
|
Embed concept rows. |
True
|
concept_scheme_urns
|
Sequence[str] | None
|
Index only these schemes' concepts. |
None
|
include_reference_metadata
|
bool
|
Index reference-metadata sections as rows. |
False
|
embed_reference_metadata
|
bool
|
Embed reference-metadata rows. |
True
|
reference_metadataflow_urns
|
Sequence[str] | None
|
Index only sets of these structures. |
None
|
document_prefix
|
str | None
|
Prepended to every document before embedding. |
None
|
query_prefix
|
str | None
|
Prepended to every query before embedding. |
None
|
model_name
|
str | None
|
The model id recorded in |
None
|
Returns:
| Type | Description |
|---|---|
int
|
The number of datasets indexed. |
close ¶
Close the working copy and leave its edition unsealed, for working_copy(resume=True).
A no-op once it is sealed, discarded or closed.
discard ¶
Delete the unsealed edition (sdmxlib.edition.discard) and close the working copy.
seal ¶
Seal the working copy into an edition and record it in the Inventory.
The steps: settle the journal; run checks; refuse when the base is
no longer live; rebuild the search index; compute the statcat
section; close; seal with sdmxlib (journal entry, then manifest),
passing the partial_of of each partial statcat._meta names
and the store still holds; run the publish check on a structure
edition, or on an edition whose partials record a partial_of, with
every other structure product's live and embargoed editions; record
the edition, live when the catalog's embargo is off; settle the
journal entry.
Catalog.seal runs the same steps for several working copies and
records them in one Inventory version.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
checks
|
Iterable[Callable[[EditionView], Iterable[Finding]]]
|
Seal checks ( |
()
|
Returns:
| Type | Description |
|---|---|
Edition
|
The edition. |
Raises:
| Type | Description |
|---|---|
EditionCheckError
|
The publish check found something, such as a reference no other structure product's live or embargoed edition resolves, or a partial whose codes its full codelist lacks, or one with no structure edition in the catalog to check it against; the sealed edition stays on disk unrecorded. |
InventoryRuleError
|
sdmxlib's rules refused the record after the
manifest was written, such as |
InventoryContentionError
|
The record lost too many races; the
product's next |
InventoryWriteRefusedError
|
The catalog has opted in to expiry, or
the newest Inventory version's time is more than 5 minutes
ahead of this machine's clock. Raised from the pre-check, it
leaves the working copy open; raised from the record, it keeps
the journal entry for a later |
Before anything is written, a check's finding raises SealCheckError
(for a structure product, also each dataflow, binding or partial it
holds; for a data product, a fill that stopped before it recorded its
cuts, or a partial cut outside fill_dataflows that would carry the
previous edition's partial_of), and a base that is no longer the
live edition raises
NonFastForwardError; both leave the working copy open. A
ManifestError means sdmxlib refused the edition, such as for a coded
value outside its codelist, before writing the journal entry; the
working copy is closed, and working_copy(resume=True) reopens it.
EditionView ¶
A read-only view of one edition: its datasets, codes, concepts, search and facets.
Product.view returns one over the product's live edition, and
WorkingCopy.view one over the working copy, which sees the working
copy's writes as they are made. It has the query methods, scopes and
projections, and no write method.
A view stays on its edition. A process that follows the live edition calls
product.view() again after Catalog.refresh.
Refuse construction from outside, with a TypeError: Product.view and WorkingCopy.view return views.
registry
property
¶
The sdmxlib read surface over the same edition: get, query, obs.
A sealed edition's sdmxlib.EditionRegistry; for a working copy's
view, its build registry.
index_annotations
property
¶
The annotation types the search index was built from, and their fingerprint.
index_layout
property
¶
What each language's search table was built with: rows and stemmer.
embedding_metadata
property
¶
The embedding model and settings the search index was built with, or None.
embedding_model_name
property
¶
The model id that produced the vector index, if known.
document_prefix
property
¶
Prefix prepended to documents at index time (e.g. "passage: ").
query_prefix
property
¶
Prefix prepended to queries at search time (e.g. "query: ").
themes
property
¶
Lazy scope over theme classification schemes: view.themes["subjects"].
concept_schemes
property
¶
Lazy scope over SKOS concept schemes: view.concept_schemes["uri"].
codes
property
¶
Typed search and lookup over SDMX codes: view.codes.find("Toronto").
concepts
property
¶
Typed search and lookup over SDMX concepts: view.concepts.find("Gini").
datasets
property
¶
Typed dataset search: view.datasets.find(q, mode="hybrid").
reference_metadata
property
¶
Typed search over indexed SDMX reference-metadata sections.
check_index_compatible ¶
check_index_compatible(
*,
model_name: str | None = None,
embedding_dim: int | None = None,
) -> None
Raise IncompatibleIndexError when the index was built with another model or width.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model_name
|
str | None
|
The encoder id the caller loaded. |
None
|
embedding_dim
|
int | None
|
The width of the caller's vectors. |
None
|
attach_embedding_model ¶
Attach the encoder embedding_model_name names, for vector and hybrid search.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
EmbeddingModel
|
The encoder. |
required |
dataset ¶
Scope for one dataset: its codelists, themes and lineage.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
identifier
|
str
|
The dataset's identifier. |
required |
Returns:
| Type | Description |
|---|---|
DatasetScope
|
The scope. |
codelist ¶
Scope for one codelist, by URN.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
urn
|
str
|
The codelist's URN. |
required |
Returns:
| Type | Description |
|---|---|
CodelistScope
|
The scope. |
dimensions ¶
Scope over a dataset's dimensions.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ds
|
AggregateDataset
|
The dataset. |
required |
Returns:
| Type | Description |
|---|---|
DimensionsScope
|
The scope. |
get ¶
The dataset identifier of type cls, with every relationship loaded, or None.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
cls
|
type[T]
|
The dataset class to narrow to. |
required |
identifier
|
str
|
The dataset's identifier. |
required |
Returns:
| Type | Description |
|---|---|
T | None
|
The dataset, or |
query ¶
A structured query over datasets of cls, filtered by keyword.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
cls
|
type[T]
|
The dataset class to narrow to. |
required |
**filters
|
object
|
Field filters, as |
{}
|
Returns:
| Type | Description |
|---|---|
QueryBuilder[T]
|
The query builder, for chaining. |
search ¶
search(
text: str,
*,
kinds: Sequence[str] = ("dataset", "concept", "code"),
lang: str = "en",
limit: int = 50,
offset: int = 0,
operator: str = "or",
fuzzy: int = 0,
rrf_k: int = 60,
per_kind_oversample: int = 100,
) -> SearchResult[AnyHit]
Search every indexed entity kind, RRF-merged.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
text
|
str
|
The query text. |
required |
kinds
|
Sequence[str]
|
The entity kinds to search. |
('dataset', 'concept', 'code')
|
lang
|
str
|
The language whose table each kind is searched in. |
'en'
|
limit
|
int
|
Hits to return. |
50
|
offset
|
int
|
Hits to skip. |
0
|
operator
|
str
|
|
'or'
|
fuzzy
|
int
|
Edit distance for fuzzy term matching. |
0
|
rrf_k
|
int
|
The reciprocal-rank-fusion constant. |
60
|
per_kind_oversample
|
int
|
Hits fetched per kind before the merge. |
100
|
Returns:
| Type | Description |
|---|---|
SearchResult[AnyHit]
|
The merged hits, best first. |
facets ¶
Counts per value of each field, across every dataset.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
*fields
|
str
|
The facet fields. |
()
|
Returns:
| Type | Description |
|---|---|
dict[str, dict[str, int]]
|
|
agent_labels ¶
Each agent's name in lang, by agent id.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
lang
|
str
|
The language tag. |
'en'
|
Returns:
| Type | Description |
|---|---|
dict[str, str]
|
The labels, by agent id. |
concept_labels ¶
Each concept's label in lang, by concept URI.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
lang
|
str
|
The language tag. |
'en'
|
scheme_uri
|
str | None
|
Restrict to one concept scheme. |
None
|
Returns:
| Type | Description |
|---|---|
dict[str, str]
|
The labels, by concept URI. |
spatial_labels ¶
Each spatial facet key's label in lang.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
lang
|
str
|
The language tag. |
'en'
|
Returns:
| Type | Description |
|---|---|
dict[str, str]
|
The labels, by facet key. |
get_concept_scheme ¶
A concept scheme with its concepts, or None.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
uri
|
str
|
The scheme's URI. |
required |
Returns:
| Type | Description |
|---|---|
ConceptScheme | None
|
The scheme, or |
get_concept ¶
One concept, or None.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
uri
|
str
|
The concept's URI. |
required |
Returns:
| Type | Description |
|---|---|
Concept | None
|
The concept, or |
code_map ¶
{dimension_id: {code_id: label}} for a dataset.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_id
|
str
|
The dataset's identifier. |
required |
lang
|
str
|
The language of the labels. |
'en'
|
Returns:
| Type | Description |
|---|---|
dict[str, dict[str, str]]
|
The labels, by dimension and code. |
codelist_for ¶
The codelist of one dimension of a dataset, or None.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_id
|
str
|
The dataset's identifier. |
required |
dimension
|
str
|
The dimension's id. |
required |
Returns:
| Type | Description |
|---|---|
Codelist | None
|
The codelist, or |
get_structure ¶
The data structure of an AggregateDataset, or None.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ds
|
AggregateDataset
|
The dataset. |
required |
Returns:
| Type | Description |
|---|---|
DataStructure | None
|
The data structure, or |
metadata_for ¶
metadata_for(
target_urn: str | SdmxUrn[object],
*,
structure_urn: str | SdmxUrn[object] | None = None,
) -> list[sdmxlib.MetadataSet]
Every reference-metadata set attached to target_urn.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
target_urn
|
str | SdmxUrn[object]
|
The artefact URN. |
required |
structure_urn
|
str | SdmxUrn[object] | None
|
Restrict to sets of this MSD or metadataflow. |
None
|
Returns:
| Type | Description |
|---|---|
list[MetadataSet]
|
The sets, in insertion order. |
metadata_report ¶
The sections of a reference-metadata set, typed by its MSD.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
set_urn
|
str | SdmxUrn[object]
|
The set's URN. |
required |
Returns:
| Type | Description |
|---|---|
list[MetadataSection]
|
One section per reported value. |
metadata_targets ¶
The artefact URNs a reference-metadata set describes.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
set_urn
|
str | SdmxUrn[object]
|
The set's URN. |
required |
Returns:
| Type | Description |
|---|---|
list[str]
|
The URNs, in order. |
language_alternates ¶
The language tags recorded as alternates of a primary tag.
Queries¶
QueryBuilder ¶
QueryBuilder(
store: _Store,
cls: type[T],
*,
filters: dict[str, object] | None = None,
exists_filters: list[PositionalFilter] | None = None,
map_filters: list[MapFilter] | None = None,
from_ids: list[str] | None = None,
text: str | None = None,
hybrid_text: str | None = None,
search_operator: str = "or",
search_fuzziness: int = 0,
search_lang: str = "en",
sort_fields: list[_SortSpec] | None = None,
limit: int | None = None,
offset: int = 0,
facet_fields: list[str] | None = None,
select_cols: tuple[str, ...] | None = None,
select_lang: str | None = None,
scalar_col: str | None = None,
include_rels: frozenset[str] | None = None,
)
Chainable query builder. Terminal method: .execute() or .all().
filter ¶
Add filter criteria. Returns a new builder (immutable chain).
from_ids ¶
Restrict results to a pre-computed list of identifiers.
search ¶
Add full-text search. Requires LanceDB.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
text
|
str
|
The search text. Tokenised and stemmed with lang's rules before matching. |
required |
lang
|
str
|
The language whose search table answers the query. Each
language has its own table, so a French query needs
|
'en'
|
operator
|
str
|
How to join multi-term queries: |
'or'
|
fuzzy
|
int
|
Levenshtein edit distance for fuzzy matching: |
0
|
Returns:
| Type | Description |
|---|---|
QueryBuilder[T]
|
A new |
QueryBuilder[T]
|
unchanged — builders are immutable. |
hybrid_search ¶
Add hybrid BM25 + semantic search. Requires LanceDB.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
text
|
str
|
The search text. |
required |
lang
|
str
|
The language whose search table answers both the BM25 and
the vector query, as for |
'en'
|
Returns:
| Type | Description |
|---|---|
QueryBuilder[T]
|
A new |
sort_by ¶
sort_by(
field: str,
*,
desc: bool = False,
nulls: str | None = None,
lang: str | None = None,
) -> QueryBuilder[T]
Add a sort criterion.
For MAP columns like title, pass lang to specify the
language key (e.g. lang="en"). Raises ValueError if
lang is omitted for a MAP column.
with_facets ¶
Request facet counts alongside the results.
include ¶
Declare which relationships to batch-load during hydration.
By default all relationships are loaded (~17 queries). Calling
include() switches to selective loading: only the named
relationships are fetched; everything else stays at its model
default (None / []).
Subsequent include() calls accumulate — they do not replace
the previous set. Use full() to revert to loading everything.
Valid names mirror the model field names::
store.query(BaseDataset)
.include("publisher", "themes", "keywords")
.collect() # 3 batch loads instead of 17
Raises ValueError for unrecognised relationship names.
full ¶
Load all relationships (the default behaviour).
Useful to explicitly opt into full hydration after a partial
include() call, or to make the cost visible at the call site::
store.query(BaseDataset).filter(...).full().collect()
select ¶
Set dict projection mode. Terminals return dicts instead of objects.
lang is required when selecting MAP columns (title,
description). Raises ValueError for unknown column names.
scalars ¶
Set single-column flat projection mode.
Terminals return bare values instead of objects or dicts.
execute ¶
Execute the query and return a SearchResult with metadata.
Only available in entity mode (no .select() / .scalars()).
collect ¶
Execute and return domain objects.
This is the preferred terminal for entity results — consistent with
scope terminals (.collect() on SchemeScope, CodelistScope,
etc.). Use .rows() for Row projections or .values() for
scalar projections.
SearchResult ¶
Container for query results with pagination metadata.
Search hits¶
Hit ¶
A search hit — payload plus per-match ranking metadata.
The discriminator kind lets a heterogeneous result list (e.g.
EditionView.search over all entity kinds) be narrowed at runtime
by a tagged match; the generic parameter T lets type-scoped
accessors (cat.codes.find etc.) narrow statically.
Per-hit metadata that depends on the query (not the entity) lives
here, not on payload:
score— backend-native score (BM25 relevance, vector similarity, or fused RRF).source— which retrieval path produced this hit; useful for hybrid-search explainability and telemetry.highlights— per-field snippet extracts when available.matched_field— which indexed column drove the rank.
CodeHit ¶
Payload for a code-search hit.
Score, language, and ranking metadata live on the enclosing
Hit; this type is the pure structural projection of one
code row — exactly what an LLM agent translating a phrase like
"Ontario" into a (dimension_id, code_id) filter needs.
ConceptHit ¶
Payload for a concept-search hit.
SDMX Concepts carry definitional text (description) which is
where vector search earns its cost — a query like "income
inequality" finds "Gini coefficient" through its definition, not
its label. The hit exposes both the structural identifiers
(URNs, scheme, id) and the localised text.
ReferenceMetadataHit ¶
Payload for a reference-metadata (methodology-document) search hit.
One indexed chunk of a sectioned SDMX MetadataSet — a reference guide,
a methodology report, a data-quality note. Unlike a code hit (a short
controlled-vocabulary label), this is prose: the vector/BM25 target is
the section text itself. The hit carries the identifiers needed to
navigate back to the structured graph — set_urn (the document) and
attribute_path (the section) — so a caller can fetch the full,
typed section via EditionView.metadata_report or resolve the
artefacts the document describes via the set's targets.
ReferenceMetadataBlend ¶
Blend a reference-metadata attribute into a code's search text.
For each code, statcatlib joins sdmx.metadata_target →
sdmx.metadata_value to pull the localised value of
attribute_path and append it to the code's search text. Improves
BM25 + vector recall on definitional queries when the catalog
publishes concept definitions or footnotes via SDMX MetadataSets
(e.g. an MSD carrying a FOOTNOTE_TEXT attribute).
Attributes:
| Name | Type | Description |
|---|---|---|
attribute_path |
str
|
Dotted MSD attribute id, e.g. |
max_chars |
int
|
Per-code truncation applied after joining multiple values with a single space. Defaults to 240 — long enough to carry one or two sentences without overwhelming the BM25 term mix or the embedding. |
Projections and selective loading¶
Row ¶
Bases: dict[str, object]
A dict subclass returned by .select() projections.
Supports both dict-style and attribute-style field access::
row = store.query(BaseDataset).select("identifier", "status").first()
row["identifier"] # dict-style
row.identifier # attribute-style
Row is a plain dict subclass — isinstance(row, dict) is
True, JSON serialisation works as normal, and all dict methods
(keys(), items(), values(), etc.) are inherited.
NotLoadedError ¶
Bases: AttributeError
Raised when accessing a relationship that was not loaded in this query.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
field
|
str
|
The relationship that was not loaded, as |
required |
Example
ds = store.query(BaseDataset).include("publisher").first() ds.themes # NotLoadedError: 'themes' was not loaded.
Add 'themes' to .include() or call .full().¶
Search configuration¶
EmbeddingMetadata ¶
Read-side view of an edition's search/embeddings.json sidecar.
Returned by EditionView.embedding_metadata. Fields that were not
present in the on-disk sidecar (e.g. when reading a v2 file with a v3
reader) come back as None rather than guesses.
CodelistIndexPolicy ¶
Per-codelist override for WorkingCopy.build_index.
Lets a caller opt large codelists into the Lance search index even
when their size would otherwise trip max_codes_per_codelist, and
decouple "should this codelist be indexed" from "should its codes get
vector embeddings". Short proper-noun labels (geography, population
characteristics) score better with Tantivy FTS than with vector
similarity over short strings, and skipping the encode is what makes
indexing a geography codelist of tens of thousands of codes take
seconds rather than minutes, at a fraction of the disk.
Attributes:
| Name | Type | Description |
|---|---|---|
urns |
tuple[str, ...]
|
Codelist URNs this policy applies to. |
mode |
str
|
How the listed codelists should be indexed:
|
Errors¶
IncompatibleCatalogError ¶
Bases: RuntimeError
Raised when a catalog or an edition is in a layout or at a version this build does not read.
The cases: a directory holding a statcatlib 0.11 catalog, a 0.72 bundle or
other files (statcatlib.catalog); an edition whose manifest.json is
not the one the Inventory recorded; a statcat schema written by a
statcatlib with another schema.SCHEMA_VERSION
(_refuse_statcat_schema_mismatch); a search/index.json at another
manifest version (_refuse_unreadable_index); and a filesystem without
hard links under Catalog.create. An edition on another sdmxlib store
schema is refused by sdmxlib itself, with sdmxlib.StoreSchemaError.
IncompatibleIndexError ¶
Bases: ValueError
Raised by EditionView.check_index_compatible on mismatch.
Subclasses ValueError so existing broad-catch sites still work.
NoLiveEditionError ¶
Bases: LookupError
The product has no live edition, so there is nothing for a public reader to view.
WorkingCopyExistsError ¶
Bases: Exception
The product already has an unsealed edition; a product has one at a time.
WorkingCopyClosedError ¶
Bases: RuntimeError
The working copy was sealed or discarded, so neither it nor its views answer.
SealCheckError ¶
Bases: Exception
A seal check found something wrong, so the edition was not sealed.
Attributes:
| Name | Type | Description |
|---|---|---|
findings |
tuple[Finding, ...]
|
Everything the checks found. |
List the findings.
EditionCheckError ¶
Bases: Exception
The publish check found something wrong with a sealed edition, so it was not recorded.
The edition stays on disk, sealed and unrecorded, and statcat doctor
reports it. Fix the source, then start a new working copy from the live
edition with working_copy().
Attributes:
| Name | Type | Description |
|---|---|---|
findings |
tuple[Finding, ...]
|
Everything |
product |
str
|
The product. |
edition |
str
|
The sealed edition. |
List the findings, and name the product and the edition.
JournalError ¶
Bases: Exception
A seal journal entry cannot be read, so the product's seals cannot be settled.
InventoryRuleError ¶
InventoryRuleError(
message: str,
*,
rule: Rule | Literal["lifecycle"] | CrossRule,
product: str,
edition: str | None,
)
Bases: Exception
A change sdmxlib's Inventory rules refuse, or a lifecycle change statcatlib refuses.
Attributes:
| Name | Type | Description |
|---|---|---|
rule |
Rule | Literal['lifecycle'] | CrossRule
|
The rule broken: one of sdmxlib's |
product |
str
|
The product. |
edition |
str | None
|
The edition, where one is concerned. |
Say why, and name the rule, the product and the edition.
wrapping
classmethod
¶
The error for sdmxlib's refusal exc, which becomes its __cause__ when raised.
A fast-forward refusal becomes a NonFastForwardError.
NonFastForwardError ¶
NonFastForwardError(
message: str,
*,
rule: Rule | Literal["lifecycle"] | CrossRule,
product: str,
edition: str | None,
)
Bases: InventoryRuleError
An edition built on an edition that is no longer live.
The message names the edition, its base, the live edition, and the fix: discard the working copy, then start one from the live edition.
CrossProductRuleError ¶
CrossProductRuleError(
message: str,
*,
rule: CrossRule,
item: Literal[4, 7, 8],
urn: str,
editions: Sequence[tuple[str, str]],
product: str,
edition: str | None,
)
Bases: InventoryRuleError
A version that breaks a cross-product rule, specification §6.5 item 4, 7 or 8.
product and edition name the incoming entry, the one that becomes
live or embargoed in the version. The message names the item, the URN,
the editions compared and the fix. rule is "full-artefacts" (item
4: a data edition's full artefact or partial matches no live or
embargoed edition of its structure home, a cut codelist has no home, or
a full artefact with no home differs from another data product's live
copy), "structure-home" (item 7: a second structure home among live
editions) or "publish-order" (item 8: a data edition entering live
before its structure edition).
Attributes:
| Name | Type | Description |
|---|---|---|
item |
Literal[4, 7, 8]
|
The specification §6.5 item: 4, 7 or 8. |
urn |
str
|
The artefact concerned. |
editions |
tuple[tuple[str, str], ...]
|
|
Say why, and name the rule, the item, the URN and the editions compared.
of
classmethod
¶
The error for refusal, from statcatlib.store.cross_product.refusals.
InventoryConflictError ¶
Bases: Exception
Another writer created the Inventory version this one tried to create.
InventoryWriter.try_commit raises it; InventoryWriter.commit catches
it, re-reads the head and tries again.
InventoryContentionError ¶
Bases: Exception
A commit lost MAX_RACES races in a row to other writers, and gave up.
InventoryWriteRefusedError ¶
Bases: InventoryError
This writer must not write the next Inventory version, whatever the version holds.
The catalog has opted in to expiry (catalog.json 2.x), which needs a
writer with the catalog lock, or the base version's created_at is more
than 5 minutes ahead of this machine's clock. The message names the
format, or the base version's time and this machine's. It stays refused
until the catalog or the clock changes, so a seal refused after its
manifest keeps its journal entry, as InventoryContentionError does.