statcatlib.testing

Generated from the docstrings.

testing

Throwaway catalogs and synthetic statcat editions, for tests and benchmarks (GH-3, GH-51).

temporary_catalog creates a catalog in a temporary directory, which replaces statcatlib 0.11's Catalog(":memory:"). synthetic_catalog builds a sealed edition of any shape: sdmxlib.testing.synthetic_edition builds a conforming sdmxlib edition from a sdmxlib.testing.Shape and a seed, and statcatlib adds the catalog, a dataset per dataflow and a search index over it. The same seed builds the same structures and observations, so a failure reproduces.

codr_structures and codr_product build a catalog shaped like Statistics Canada's CODR tables: three structure products (STRUCTURE_PRODUCTS) holding the full codelists, the geography hierarchy and the concept scheme, and data products of one or more tables whose codelists are cut to partials. A CodrShape sets the sizes, and one seed gives one set of artefact fingerprints, so products generated in separate calls match the structure products.

Example

from sdmxlib.testing import Dim, Shape

from statcatlib import Catalog from statcatlib.testing import synthetic_catalog

shape = Shape(dimensions=(Dim(codes=5000, depth=4), Dim(codes=12)), observations=100_000, languages=("en", "fr")) edition = synthetic_catalog("out/stress", shape, seed=7) with Catalog.open("out/stress") as cat: view = cat.product(edition.product_id).view()

CodrShape

The sizes of a generated CODR catalog.

Attributes:

Name	Type	Description
geography_codes	int	The codes of CL_GEO.
geography_depth	int	Its levels, from the country down.
classification_codes	tuple[int, ...]	The codes of each classification codelist, CL_CLASS_1 onward; one dimension per entry. The first two have three levels, the others one.
observations	int	The rows of each table.

CorpusGroup

One group of a codr_corpus: products data products of tables tables each.

Attributes:

Name	Type	Description
products	int	The data products in the group.
tables	int	The tables of each, one dataflow each.

__attrs_post_init__

__attrs_post_init__() -> None

Refuse a negative product count and a product of no table.

Raises:

Type	Description
ValueError	products is negative or tables is below 1.

temporary_catalog

temporary_catalog() -> Generator[Catalog]

A writable catalog in a temporary directory, deleted when the block ends.

Yields:

Type	Description
Generator[Catalog]	The catalog, from Catalog.create.

codr_structures

codr_structures(
    catalog: Catalog,
    *,
    seed: int = 0,
    shape: CodrShape = _DEFAULT_SHAPE,
) -> dict[str, Edition]

Seal the three CODR structure products into catalog, in one Catalog.seal.

The artefacts depend on seed and shape alone: agency CODR, version 1.0, names "<id> en" and "<id> fr", no validity dates and no annotations. structure-geo holds CL_GEO and HCL_GEO, a hierarchy over all its codes. structure-classifications holds CL_CLASS_1 onward, CL_UNIT and the SDMX-agency CL_OBS_STATUS, so every codelist a table cuts has a structure home. structure-concepts holds CS_CODR, whose coded concepts name those codelists, so its publish check resolves references in the other two.

Parameters:

Name	Type	Description	Default
catalog	Catalog	A writable catalog.	required
seed	int	Chooses the codelists' trees.	0
shape	CodrShape	The sizes.	_DEFAULT_SHAPE

Returns:

Type	Description
dict[str, Edition]	Each structure product's new edition.

Raises:

Type	Description
RuntimeError	Catalog.seal refused a structure product; the message names each refusal, and the first is the cause.

codr_working_copy

codr_working_copy(
    catalog: Catalog,
    product_id: str,
    *,
    tables: int = 1,
    seed: int = 0,
    shape: CodrShape = _DEFAULT_SHAPE,
) -> WorkingCopy

Fill a CODR-shaped data product's working copy of tables tables, with its codelists cut to partials.

codr_product seals what this returns; a caller sealing several products in one Catalog.seal takes the working copies from here. It seals the three structure products first (codr_structures) unless all three are live. Each table n gets a data structure, DSD_<product_id>_<n> (dimensions GEO and CLASS_1 onward, then TIME_PERIOD; attributes UNIT_MEASURE and OBS_STATUS; measure OBS_VALUE), a dataflow DF_<product_id>_<n>, and shape.observations rows over a share of each codelist's codes: 5 % of the geography's leaves with their ancestors, a quarter of each classification, two units and two statuses. ingest_dataflows fills the product's working copy with partials=True, so the seal records each partial's partial_of. No hierarchy association names a table, so CL_GEO is cut. When lancedb is installed, the search index takes the geography's codes with CodelistIndexPolicy in fts_only mode.

The structures depend on seed alone, and the rows on seed and product_id. Generate every product of a catalog with the seed and the shape its structure products were generated with; another seed gives other fingerprints, whose partials the publish check refuses.

Parameters:

Name	Type	Description	Default
catalog	Catalog	A writable catalog.	required
product_id	str	The data product.	required
tables	int	The tables, one dataflow each.	1
seed	int	Chooses the structures and the rows.	0
shape	CodrShape	The sizes.	_DEFAULT_SHAPE

Returns:

Type	Description
WorkingCopy	The filled working copy, unsealed.

codr_product

codr_product(
    catalog: Catalog,
    product_id: str,
    *,
    tables: int = 1,
    seed: int = 0,
    shape: CodrShape = _DEFAULT_SHAPE,
) -> Edition

Seal a CODR-shaped data product of tables tables into catalog, with its codelists cut to partials.

codr_working_copy fills the working copy, and this seals it.

Parameters:

Name	Type	Description	Default
catalog	Catalog	A writable catalog.	required
product_id	str	The data product.	required
tables	int	The tables, one dataflow each.	1
seed	int	Chooses the structures and the rows.	0
shape	CodrShape	The sizes.	_DEFAULT_SHAPE

Returns:

Type	Description
Edition	The sealed edition.

corpus_product_id

corpus_product_id(group: CorpusGroup, index: int) -> str

The id of the index-th product of group: codr-<tables>t-<index, five digits>.

Parameters:

Name	Type	Description	Default
group	CorpusGroup	The group.	required
index	int	The product's place in the group, from 0.	required

Returns:

Type	Description
str	The product id.

corpus_embargoed

corpus_embargoed(
    product_id: str,
    *,
    seed: int = 0,
    embargoed: float = 0.1,
) -> bool

Whether codr_corpus seals product_id under embargo: a seeded draw below the share embargoed.

Parameters:

Name	Type	Description	Default
product_id	str	The product.	required
seed	int	The corpus seed.	0
embargoed	float	The share of products sealed under embargo.	0.1

Returns:

Type	Description
bool	Whether the product seals under embargo.

codr_corpus

codr_corpus(
    catalog: Catalog,
    groups: Sequence[CorpusGroup] = (CorpusGroup(7000),),
    *,
    seed: int = 0,
    shape: CodrShape = _DEFAULT_SHAPE,
    embargoed: float = 0.1,
    on_sealed: Callable[[str, Edition], None] | None = None,
) -> list[str]

Seal a corpus of CODR data products into catalog, with the structure products they are cut from.

Each group's products are named by corpus_product_id and built by codr_product with seed and shape, so every product cuts its codelists from the same structure editions, and each product's rows come from its id. A product the Inventory records already, in any state, is skipped, so a call with larger groups grows a corpus a smaller call built, and a call after a crash resumes it, discarding the unsealed working copy the crash left. A share embargoed of the products, chosen by corpus_embargoed, seals with the catalog's embargo on, at the cost of two embargo-change versions each, so the corpus holds live and embargoed data entries. The embargo is off when the call returns.

Parameters:

Name	Type	Description	Default
catalog	Catalog	A writable catalog.	required
groups	Sequence[CorpusGroup]	The groups, sealed in order.	(CorpusGroup(7000),)
seed	int	Chooses the structures and every product's rows.	0
shape	CodrShape	The sizes.	_DEFAULT_SHAPE
embargoed	float	The share of products sealed under embargo, from 0 to 1.	0.1
on_sealed	Callable[[str, Edition], None] | None	Called with each product id and edition after its seal.	None

Returns:

Type	Description
list[str]	The products this call sealed, in order.

Raises:

Type	Description
ValueError	embargoed is outside 0 to 1.

synthetic_catalog

synthetic_catalog(
    path: str | Path,
    shape: Shape,
    *,
    product_id: str = "synthetic",
    seed: int = 0,
) -> Edition

Create a catalog at path holding one sealed synthetic edition of the given shape.

The structures and observations are sdmxlib.testing.synthetic_edition's, built straight into the product's unsealed edition, which a working copy then resumes. Each dataflow becomes an AggregateDataset. The seal builds the search index when statcatlib[search] is installed, and records the edition live.

Parameters:

Name	Type	Description	Default
path	str | Path	The catalog directory to create. It must not exist or must be empty.	required
shape	Shape	Codelist sizes, hierarchy depth, observation count and languages.	required
product_id	str	The product to build.	'synthetic'
seed	int	What makes the edition reproducible.	0

Returns:

Type	Description
Edition	The sealed edition.