Diagnostics

Generated from the docstrings. Catalog administration describes the statcat verify-catalog and statcat doctor commands these functions run.

diagnostics

Integrity and support checks over a catalog directory (Phase 2a S4).

Two functions, each run by a statcat command:

• verify_catalog (statcat verify-catalog PATH) reads the newest valid Inventory version with sdmxlib.inventory.FullInventory. It checks every root the read protocol skipped, every leaf, and, for every entry the leaves hold, the manifest_sha256 the Inventory records and the routing fields and artefact records it holds. It then runs sdmxlib.edition.check_edition at the publish profile with statcatlib.bundle.check_catalog, which checks every file the manifest lists, the shared data/ files included, each partial against the structure edition it was cut from, and each structure edition's references in the other structure products. It also lists the stale data editions (specification §6.5 item 4), which are no finding. A catalog it cannot read at all raises CatalogUnreadableError.
• doctor (statcat doctor PATH) reports on a catalog for support: the layout, catalog.json, CURRENT against the newest root, root and leaf validation, stale data editions, partials with no structure home, unsealed editions, seal journal entries, sealed editions no version names, files a writer staged and left behind, and a synced-folder path.

Both only read. Neither repairs, records or deletes anything: the journal rule (statcatlib.store.working_copy.settle_journal) repairs a crashed seal when the product's next working copy starts.

StaleEdition

A live or embargoed data edition holding an old copy of a structure (specification §6.5 item 4).

Attributes:

Name	Type	Description
product_id	str	The data product.
edition_id	str	The stale edition.
state	State	live, which keeps serving its own closure, or embargoed, whose release is refused.
urn	str	The artefact.
fingerprint	str	The fingerprint the edition holds: a full artefact's own, or a partial's partial_of.
home	str	The structure product that holds the URN.
home_live	str | None	The home's live edition, or None when only an embargoed edition holds the URN.

CatalogUnreadableError

Bases: Exception

The directory is not a catalog this build can read, so nothing in it can be verified.

statcat verify-catalog exits 2 on it. Its message says why: no directory, a legacy layout, no or a malformed catalog.json, or no valid Inventory version.

VerifyFinding

One thing verify_catalog found wrong.

Attributes:

Name	Type	Description
file	str	What it is about, as a /-separated path from the catalog directory: the damaged file where a check names one, or else the edition's directory.
detail	str	What is wrong with it.

VerifyReport

What verify_catalog checked, and what it found.

Attributes:

Name	Type	Description
root	Path	The catalog directory.
seq	int | None	The Inventory version read, or None for a catalog with no version yet, which has nothing to verify.
editions	tuple[str, ...]	Each edition checked, as <product>/<edition_id>.
findings	tuple[VerifyFinding, ...]	Everything found wrong, in the order found.
stale	tuple[StaleEdition, ...]	Each stale data edition, one item per edition and URN: a live or embargoed data edition holding a copy of a structure that no live or embargoed edition of its structure home holds (specification §6.5 item 4). A stale edition is a state the specification allows until its product is republished, so it is no finding and leaves ok alone.

ok property

ok: bool

Whether nothing was found wrong.

DoctorItem

One line of a doctor report.

Attributes:

Name	Type	Description
topic	str	What it is about: layout, synced folder, catalog.json, CURRENT, inventory, stale data editions, partials with no structure home, working copy, journal, unrecorded edition or leftover file.
level	Literal['info', 'warning']	warning for something support acts on, info for state worth knowing.
detail	str	The report.

DoctorReport

What doctor found.

Attributes:

Name	Type	Description
root	Path	The directory examined.
layout	Literal['catalog', 'legacy', 'unknown']	catalog for a directory holding catalog.json, legacy for a statcatlib 0.11 catalog or a 0.72 bundle, and unknown for anything else.
items	tuple[DoctorItem, ...]	The report, in the order doctor examined it.

warnings property

warnings: tuple[DoctorItem, ...]

The items at level warning.

about

about(topic: str) -> tuple[DoctorItem, ...]

The items about topic.

Parameters:

Name	Type	Description	Default
topic	str	One of DoctorItem.topic's values.	required

Returns:

Type	Description
tuple[DoctorItem, ...]	Those items, in report order.

verify_catalog

verify_catalog(path: str | Path) -> VerifyReport

Check a catalog's integrity: its Inventory, and every edition the Inventory records.

It reads the newest valid Inventory version, as a reader does. A root the read protocol skipped is a finding. For each product it checks the leaf against its name and sdmxlib's leaf rules; for each entry of the leaf, in every state but retired, it compares manifest.json with the manifest_sha256 the Inventory records, compares the entry's filled dataflows, pull_bytes and artefacts with the manifest's (sdmxlib.inventory.verify_entry; an entry without them is skipped), then runs check_edition(at="publish", extensions={"statcat": check_catalog}), which checks the size and digests of every file the manifest lists, data/ files included, the lake, the bindings and the statcat section. It gives the check the catalog's structure editions: for a data edition every recorded structure edition, so each partial is checked against the edition it was cut from; for a structure edition the other structure products' live and embargoed editions, where its references resolve. Last it lists the stale data editions (VerifyReport.stale). It writes nothing.

Parameters:

Name	Type	Description	Default
path	str | Path	The catalog directory.	required

Returns:

Type	Description
VerifyReport	The report. A catalog with no Inventory version yet has nothing to
VerifyReport	verify, and its report has no seq.

A path that is not a directory, holds a legacy layout, has no or a malformed catalog.json, or has Inventory versions and no valid one raises CatalogUnreadableError.

synced_folder

synced_folder(path: str | Path) -> str | None

The sync client whose folder holds path, or None.

It matches the folder names Dropbox, OneDrive, iCloud Drive and Google Drive create, in path and in path with symbolic links resolved.

Parameters:

Name	Type	Description	Default
path	str | Path	The path.	required

Returns:

Type	Description
str | None	The client's name, such as "Dropbox".

doctor

doctor(path: str | Path) -> DoctorReport

Report on a catalog for support. It writes nothing.

The report covers, in order: a synced-folder path (Dropbox, OneDrive, iCloud Drive or Google Drive), the layout, catalog.json, CURRENT against the newest valid root, root and leaf validation, each stale data edition (specification §6.5 item 4, with the fix: republish the product; info for a live edition, warning for an embargoed one), each partial of a live or embargoed data edition whose URN no live or embargoed structure edition holds full (a warning, with the fix), and for each product its unsealed editions with their start times, its seal journal entries (empty ones included), and its sealed editions that the newest version neither names nor tombstones. Last come files in inventory/ or the catalog root that a writer staged and left behind.

Parameters:

Name	Type	Description	Default
path	str | Path	The directory.	required

Returns:

Type	Description
DoctorReport	The report. A directory that is not a catalog gets a report on its
DoctorReport	layout only.