The query language: evidence¶

Evidence record: peer surveys (polars 1.x, SQLAlchemy 2.x, ibis 12, Django 6.x, DuckDB), schema and path measurements, benchmarks and prototypes from the query-language design pass for #306/#280, August 2026, with in-place corrections through October 2026 (#411, #425, #511). Not maintained against the code; the design is query-language.md.

Companion to query-language.md, which carries the design and the decisions. This file carries what was run: peer surveys, schema measurements, benchmarks, and the prototype log. Each entry states what it tested and what it showed, so a decision in the design file can be checked against the thing that justified it rather than restated.

Two conventions from the parent file are kept. Anything recalled rather than run says so. And a measurement that later turned out to answer the wrong question is corrected in place rather than deleted, because the correction is usually the more useful record.

Peer surveys¶

Peer survey: column naming¶

Run rather than recalled; each row was executed and its output inspected.

library	naming a column	bound to
ibis	`a = con.table("artefact"); a.agency_id`	a table handle
SQLAlchemy	`Dsd.agency` → `InstrumentedAttribute`	a mapped class
polars	`pl.col("agency_id")`	nothing — a string
Django	`"dsd__agency"`	nothing — a string path
pandas / PySpark	`df.col`	the frame
sdmxlib	`sl.artefact.agency_id`	a module-level singleton

sdmxlib is the only library that binds a column to a module-level singleton. Predicates here are built standalone and handed to .filter(), so the accessor has nothing to be bound to. Every other library binds the column to the relation it came from, which is what lets them check joins and infer scope.

Peer survey: join ambiguity in a fixed schema¶

This survey bears more directly on sdmxlib, whose schema is fixed; the peers' schemas mostly are not.

library	spelling	ambiguity resolved by
SQLAlchemy	`select(Component).join(Component.dsd)`	the relationship attribute
Django	`filter(dsd__agency="ESTAT")`	the FK path
GraphQL	nested selection along named fields	impossible — every hop is named
Cypher	`MATCH (f)-[:HAS_COMPONENT]->(c)`	the named relationship type

All four name the edge they join along. Verified for SQLAlchemy by constructing the ambiguous case — dsd reachable from both Component and Flow — where join(Component.dsd) and join(Flow.dsd) produce different, unambiguous SQL.

A prototype that joined by table and picked the first matching edge was written and works, but it silently chooses a path when two exist. That is the one design in the survey with no precedent.

What `in` means elsewhere, measured¶

The peers give __contains__ a different meaning, beyond executing eagerly. Run, not recalled:

library	`x in relation`	what it means	row-existence idiom
polars `DataFrame`	supported	column name — `"a" in df` True, `1 in df` False	filter + `.height`
polars `LazyFrame`	supported, warns	column name, and says resolving the schema is expensive	`.filter(...).collect()`
ibis `Table`	supported	column name — `"id" in t` True, `"CA" in t` False	`col.isin([...]).any()`
SQLAlchemy `Select`	not implemented	`TypeError: argument of type 'Select' is not iterable`	`select(q.exists())`
sdmxlib today	supported	row id	—

None of the three uses in for row membership. polars and ibis use it for column names, and SQLAlchemy refuses it. A polars or ibis reader would take "CA" in cl.codes as a column-name test.

That makes it a fifth instance of the hazard query-composition.md names — borrowing polars' vocabulary without its laws, the seam behind #222, #255, #261 and #292 — caught before it shipped.

.exists() is SQLAlchemy's answer and the only one that suits a lazy query. The design does not repurpose in to column membership for parity: columns here are sl.q.code.* references rather than strings, so the polars meaning would check nothing useful.

Ref.__getattr__ resolves eagerly, and keeps doing so. It belongs to the object graph, which queries do not cover, and attribute access on that graph is CLAUDE.md's headline promise. The design names it as the one eager subsystem.

The schema, measured¶

The Ref graph and the join graph differ¶

domain Ref fields          : 45
storage *_urn join columns : 65

They overlap and neither contains the other:

Storage has containment edges the model expresses as lists. code.codelist_urn, concept.scheme_urn, category.scheme_urn are Codelist.codes / ConceptScheme.concepts in the model — ItemList membership, with no Ref to derive from. These are the most common joins.
Storage has side tables with no domain counterpart: annotation .owner_urn, localized_text.owner_urn, artefact_ref.source/target_urn.
Names differ where they do correspond: Dimension.representation is dsd_component.codelist_urn.

So edges derived from Ref fields would be wrong. Declaring the storage edges does not declare the Ref graph twice, because the two graphs only overlap. The query layer needs the storage graph. The storage schema is ours and changes (three tables arrived in #276/#281), and a DESCRIBE guard covers that.

Every consumer shape, against the design¶

The 113 hand-written sites from #275/#276/#281, by shape:

sites	shape	verdict
29	projection of two columns, no objects	works today (#275)
22	discovery: flows matching a concept	works today (#276)
21	dimensions of a flow, at catalogue scale	needs this design — nested fan-out
13	reference metadata for an artefact	works today (#281)
12	labels / spine / annotations	works today
8	concept-driven discovery	works with the design
—	aggregation (`GROUP BY`, counts)	works today (#275)
—	array aggregation (`list(...)`)	now possible — see above
—	label fallback (`COALESCE`)	still open — `.fill_null()` / `when-then` were proposed and never built; fall back in polars after `collect()`
—	facet counts over a caller table	still missing: cross-schema join

Two gaps survive: joins over tables we do not own, and expression-valued projection. The second was recorded here as closed by the polars branch, on the reasoning that fill_null / when-then-otherwise and .over() come free with an expression language rather than needing a feature. That was wrong as a statement of what shipped: .over() was built (windows, through qualify), but fill_null and sl.when never were. A label fallback is done in polars on the collected frame. (Corrected in place; the row above said "closed".)

Path uniqueness, measured three ways¶

The same question asked at three granularities, over time. Consolidated here because the three numbers were quoted in different places and read as though they measured one thing.

granularity	edges	reachable pairs	unique	note
tables	11 curated	32	0	`DataStructure`, `Dataflow`, `Codelist`, `Concept` are all rows of `artefact`, so everything collapses
query-model types, with hubs	15	99	24 (24 %)	`annotation.owner_urn` / `localized_text.owner_urn` point at any artefact and connect everything to everything
query-model types, no hubs	15	33	23 (70 %)
minimal FK-only, no hubs, no shortcut edges	13	31	26 (84 %)	what an implementation would actually declare

The 0-of-32 figure was the wrong measurement for the question it was used to answer. It is at table granularity, while the query model's namespace is finer. At that granularity most pairs are unique, and some of the remaining ambiguity in the middle rows came from declaring convenience edges alongside the real FK ones.

The irreducible residue, under the minimal model:

component -> codelist   ['codelist', 'concept.codelist']
component -> code       ['codelist.codes', 'concept.codelist.codes']
dataflow  -> codelist   ['dsd.components.codelist', 'dsd.components.concept.codelist']
dsd       -> codelist   ['components.codelist', 'components.concept.codelist']
dsd       -> code       ['components.codelist.codes', 'components.concept.codelist.codes']

All five trace to one root cause: dsd_component.codelist_urn (a component's own representation) versus concept.repr_codelist_urn (its concept's default). SDMX lets a component override its concept's representation, so these are two different codelists and two legitimate questions. Demonstrated with rows under "Two routes to a codelist" below.

What these numbers do and do not decide. They bear on form A (bare columns after a join()) and on any contextual resolution scheme where a bare reference auto-joins. They do not bear on form B, where every reference is rooted and the row-kind phantom keeps roots apart structurally.

Two routes to a codelist, with rows¶

Fixture: concept REF_AREA defaults to CL_GEO; CPI uses it as a dimension but overrides the codelist to CL_AREA; TRADE uses it as an attribute, keeping CL_GEO.

dsd -> codelist, asking "which DSDs reference CL_GEO"
  components.codelist          (component's own representation)  -> ['TRADE']
  components.concept.codelist  (the concept's default)           -> ['CPI', 'TRADE']

The answers differ only on the structure that overrides. Drop the override from the fixture and both return ['TRADE']. Test data without overrides never shows the difference; production data with overrides gets a wrong answer.

A second case is contingent rather than inherent:

dsd -> concept, asking "which DSDs use REF_AREA"
  dimensions.concept  -> ['CPI']
  attributes.concept  -> ['TRADE']
  components.concept  -> ['CPI', 'TRADE']

Three answers only if dimensions / attributes / measures are declared as separate edges. With a single components edge it resolves uniquely. Three is the likely choice — DataStructure carries three ItemLists and the consumer survey's largest bucket is role-specific — but it remains a design choice.

Performance¶

Cost of the path machinery¶

Node.__getattribute__ runs on every attribute access, which looked like a hot-path risk worth checking before adopting. Measured:

plain Col == 'x'  (no Node)                      76 ns
Component.id == 'x'           (0 hops)           82 ns
Component.dsd.agency == 'x'   (1 hop)           201 ns
Component.flow.dsd.spine...   (3 hops)          788 ns

for scale
  today's `code.id == "C1"` predicate build      388 ns
  one compiled + executed query              223,000 ns

The cost is negligible. A three-hop path is 0.35 % of one query, and the current DSL's predicate build is slower than the new one-hop path. The cost is superlinear — ~293 ns per extra hop, because each hop rebuilds child nodes — which is irrelevant at SDMX's one-to-three hops and would matter on deep paths.

Fan-out and N+1 in consumer code¶

The common client shape is a fan-out the caller iterates: for each codelist, its codes. Written as N+1 today. Measured, 60 codelists x 40 codes:

N+1  (resolve each, then .codes.ids())    39.8 ms    120 queries
one nested query                           3.3 ms      1 query      12x

It scales with N, so production is worse than the fixture. The spelling:

(
    reg.codelists()
    .filter(sl.q.codelist.agency == "ESTAT")
    .select(
        codelist=sl.q.codelist.id,
        codes=sl.q.codelist.codes.list(sl.q.code.id),
        n=sl.q.codelist.codes.count(),
    )
)

and the caller iterates without returning to the database, because codes is a list[str] already in memory:

for urn, codes, n in frame.iter_rows(): ...

.list() and .count() compile to correlated subqueries, so asking for two fan-outs at once (codes and annotations) cannot produce a cartesian product the way two joins would. That makes fan-outs composable.

The peer sweep across the proposed surface¶

query-composition.md says: when a DSL decision has a counterpart in polars or SQLAlchemy, check it. Four of the five borrowed-names defects on record (#222, #255, #261, #292) were found by tests after shipping; the in finding above was found by running the peers side by side. This section does that sweep across the operators the design proposes but had not checked.

Everything below was run. polars 1.x, SQLAlchemy 2.x, ibis 12 (duckdb), Django 6.x, DuckDB. Where a row says the peers disagree, they disagreed when executed.

Terminals¶

ours	SQLAlchemy	Django	ibis	polars
`collect()`	`.all()` / `.scalars().all()`	iterate / `list()`	`.execute()` / `.to_pyarrow()`	`.collect()`
`one()`	`.one()`	`.get()`	—	—
`one_or_none()`	`.one_or_none()`	—	—	—
`first()`	`.first()`	`.first()`	`.head(1)`	`.first()`
`count()`	`func.count()`	`.count()`	`.count()`	`.count()`
`exists()`	`select(q.exists())`	`.exists()`	`.isin(...).any()`	—
`explain()`	`str(stmt)` / `.compile()`	`.explain()`	`.compile()` / `ibis.to_sql`	`.explain()`
`preview(n)`	—	—	`.head(n)`	`.head(n)`

one / one_or_none / first are SQLAlchemy's names exactly, and exists / count / explain are Django's. Every terminal the design proposed exists under that name in at least one peer, most in two.

`[]` and `in`, confirmed across four¶

	`q[slice]`	`q[0]`	`x in q`
polars `LazyFrame`	lazy refine	`TypeError`	column name (warns)
Django `QuerySet`	lazy refine (`QuerySet`)	executes → model	row membership
ibis `Table`	`.limit()`	—	column name
SQLAlchemy	`.limit()/.offset()`	—	not implemented

polars and Django agree independently: a slice refines lazily, an index extracts. Two peers support the design's terminal rule (as built: q[1:3] refines, q["FR"] looks a row up by id, and in raises). Django is the lone peer whose in means row membership, and it is also the one whose in is documented to fall back on a full fetch.

Nullable edges and the join-kind rule¶

Django, nullable FK, join kind by context:

filter(cl__name='Areas')   -> INNER JOIN       orphan dropped
values('cid','cl__name')   -> LEFT OUTER JOIN  orphan kept, cl__name = None
exclude(cl__name='Areas')  -> LEFT OUTER JOIN  orphan RETURNED

The first two confirm the design's candidate rule. The third corrects it. A negative filter takes LEFT, because excluding rows that match must still return rows that cannot match — under INNER the orphan would vanish from a query asking for everything that is not "Areas".

So the candidate rule, "filter → INNER, projection → LEFT", becomes:

The join kind follows from whether a null-extended row can satisfy the predicate. Positive filter: it cannot, so INNER. Projection or negation: it can, so LEFT.

SQLAlchemy does not infer the join kind — join(...) is INNER and isouter=True is explicit, verified by the orphan appearing only in the second:

select(Code).join(Code.cl)                -> ['x', 'y']
select(Code).join(Code.cl, isouter=True)  -> ['x', 'y', 'orphan']

The peers split here: Django infers per context, SQLAlchemy makes the caller say. Inference fits a path-expression DSL, which has no join clause to hang isouter on, but it meant the design had to specify the negation case, which no draft did. (Since resolved; the rule as built is query-language.md §3.3 and §3.6.)

`list()` alignment in projections¶

Asserted earlier, verified here. Two correlated list() subqueries over the same child table, DuckDB:

SELECT (SELECT list(id)  FROM code WHERE cl=cl.id) AS ids,
       (SELECT list(lab) FROM code WHERE cl=cl.id) AS labs FROM cl
-> ids [x, y, z]   labs [X, Y, Z]      aligned

They align only because of the plan. list() has no defined order without ORDER BY, so the correspondence depends on the plan. Change one subquery's order and they misalign with no error:

       (SELECT list(lab ORDER BY pos) FROM code WHERE cl=cl.id)
-> ids [x, y, z]   labs [Y, X, Z]      MISALIGNED, no error

The struct form cannot misalign, because there is one list:

list({'id': id, 'lab': lab} ORDER BY pos)
-> [{'id': y, 'lab': Y}, {'id': x, 'lab': X}, {'id': z, 'lab': Z}]

Both peers with an equivalent return the aligned form. polars group_by().agg(pl.col("id"), pl.col("lab")) and ibis group_by().agg(ids=t.id.collect(), labs=t.lab.collect()) each keep the two lists element-aligned, because each builds both from a single grouping pass. Two independent correlated subqueries have no such guarantee. A struct gives SQL the same single-pass construction.

Where the peers already agree with us¶

Checked because a divergence here would change results without an error:

behaviour	polars	SQLAlchemy	Django	ibis	ours
`x != v` drops nulls	yes	yes (SQL)	yes (SQL)	yes	yes
`is_in([])` → no rows	yes	yes — emits `1 != 1`	yes	yes	yes (#273)
`select(name=expr)` kwargs	yes	`.label()`	`.annotate()`	yes	proposed (built; `query-language.md` §6)
`.any()` on a to-many → `EXISTS`	—	yes	no — `INNER JOIN` (dupes)	—	yes

Two notes. select(alias=expr) as keyword arguments is polars' and ibis's spelling and validates the design's select(flow=…, dims=…). And the .any() row restates the measured Django footgun in query-language.md §3.3 — SQLAlchemy is the peer we match, deliberately.

Names for array aggregation¶

list() (DuckDB / SQL array_agg), collect() (ibis), agg() / implode() (polars), ArrayAgg (Django, PostgreSQL only). No convergence to borrow, so the name is ours to choose; list() matches the SQL we emit and the DuckDB function, which is the tiebreak used elsewhere in this file.

Sweep scope¶

PIVOT and window functions, then unimplemented gaps listed in query-composition.md. (Since answered: query-language.md §7.3 and §13, item 11.)
Streaming / chunked collection. polars has collect(engine="streaming") and ibis .to_pyarrow_batches(); nothing in this design said what happens when a result does not fit in memory, and its terminal list was silent on it. (Since built: query-language.md §5.)
group_by / agg shapes beyond naming.
The data plane. Its operator mapping to c[...] was checked against a downstream SDMX REST service, but not against these peers, and its namespace is dynamic so most rows above would not apply.

Typing hazards found while prototyping¶

The `.str` namespace shadows the builtin¶

Found while building the prototype. It applies to any namespace named after a builtin.

class A:
    @property
    def str(self) -> StrNS: ...
    def f(self, x: list[str]) -> None: ...   # `str` is now the property

Confirmed at runtime rather than argued:

typing.get_type_hints(A.f)  ->  {'x': list[<property object at 0x...>], ...}
basedpyright                ->  0 errors
pyrefly                     ->  2 errors

The annotation resolves to the property. basedpyright misses it; pyrefly catches it. It was the third such case in this work, and a data point for what later made pyrefly the only type checker (#425; CLAUDE.md, Typing conventions).

It also breaks at runtime: anything introspecting through get_type_hints sees a property object where a type belongs, and attrs does that. The fix is import builtins and builtins.str in any class carrying a .str namespace — which is what the prototype does.

Hostile-review measurements¶

Three things checked while trying to break the chosen design. The first found a hole; the second contradicted a claim the design was leaning on; the third corrected a claim about polars.

The conjunction ambiguity, with rows¶

reg.query(q.dsd).filter(q.dsd.dimensions.concept.id == "REF_AREA",
                        q.dsd.dimensions.codelist.id == "CL_GEO")

Fixture: DSD A has one dimension carrying both values; DSD B has two dimensions carrying one each.

(a) two EXISTS  — any dimensions, possibly different  -> ['A', 'B']
(b) one EXISTS  — the same dimension satisfies both   -> ['A']

Different answers, and a bare dot-through path cannot say which is meant.

Django distinguishes them by chaining, measured:

filter(a=1, b=2)         SAME related object   -> ['A']       1 JOIN
filter(a=1).filter(b=2)  INDEPENDENT objects   -> ['A', 'B']  2 JOINs

That resolution is unavailable here: query-composition.md states, and generated property tests assert, that filter chaining is AND and identical to passing both at once — which is also polars' behaviour. Buying Django's distinction would mean breaking a law with tests behind it.

Cost of correlated subqueries per fan-out column¶

The design relies on correlated subqueries composing where joins do not. That holds for correctness. The cost, over 2,000 codelists × 50 codes = 100k child rows:

1 correlated list() subquery      9.7 ms
2                                21.4 ms
3                                29.8 ms
4                                40.6 ms        ~10 ms each, linear
1 struct list() with 4 fields    56.6 ms        <- the alignment-safe form
GROUP BY equivalent (4 lists)    31.8 ms        <- what polars and ibis do

Three findings. Cost is linear in the number of fan-out columns, so a five-column projection pays five passes. The struct form recommended for alignment safety is the slowest. And GROUP BY — rejected because it changes row count — is faster than either.

The design stands, and the cost falls on the compiler. Multiple fan-outs over one edge must be coalesced into a single pass rather than emitted independently, and at the time of measurement that shared-prefix deduplication had not been prototyped.

polars can express the same-child conjunction¶

An earlier claim that polars cannot do this was wrong. It can, and the mechanism is pl.element() — a global bound by the .list.eval() context:

pl.col("dims").list.eval(
    (pl.element().struct.field("concept") == "REF_AREA")
    & (pl.element().struct.field("codelist") == "CL_GEO")
).list.any()                                        # -> ['A']  same element

# two separate .list.eval(...).list.any() conjoined  -> ['A','B']  independent

polars ships a binder, so a binder here is consistent with polars. pl.element() is untyped, while a child-rooted accessor carries its row kind and is checked.

Gap-analysis probes¶

Three claims about what the design could not express checked rather than asserted (the gaps still open are query-language.md §13).

Any-language label search. name is MAP(VARCHAR, VARCHAR), and name['en'] picks one language. Matching in any language needs map traversal:

SELECT id FROM code
 WHERE len(list_filter(map_values(name), x -> x ILIKE '%lindo%')) > 0
-> ['MX']      matched on the Spanish label alone

The project rule that no API has a default language makes this common: a search box over a multilingual catalogue is the normal case.

Array-column predicates. ancestor_ids is TEXT[]. descendants_of(x) covers a single ancestor; "under any of these" is a different operator:

SELECT id FROM code WHERE list_has_any(ancestor_ids, ['EUR','AFR'])
-> ['FR']

"The latest version of each artefact" needs a window function and a filter on its result. Window framing with .over() alone cannot express it:

SELECT agency,id,version FROM (
  SELECT *, row_number() OVER (PARTITION BY agency,id ORDER BY version DESC) rn
  FROM art) WHERE rn = 1
-> ESTAT/GDP 1.0, ESTAT/CPI 2.0

So row_number / rank / lag are a separate requirement from .over(), and filtering on a windowed column is a third.

Cross-schema and cross-plane probes¶

A downstream accessor with an edge into the sdmx graph type-checks. The caller declares its own row kind and accessor; edge() threads the caller's row kind through our classes because they are generic over it:

my.dataset.code.name["en"]        -> Str[DatasetRow]
my.dataset.code.codelist.id       -> Str[DatasetRow]
reg.query(my.dataset).filter(q.code.id == "CA")
    error: Argument of type "Pred[CodeRow]"

basedpyright 0/0/0, pyrefly 0 errors. It needs no change in sdmxlib; the RowKind extension point covers it.

Labelling observations is one statement. The DSD is known when a data query is bound, so each dimension's codelist URN is a constant:

SELECT o.REF_AREA, ra.name['en'] AS REF_AREA_label,
       o.FREQ,     fr.name['en'] AS FREQ_label, o.value
FROM obs o
LEFT JOIN sdmx.code ra ON ra.codelist_urn='u:cl:AREA' AND ra.code_id=o.REF_AREA
LEFT JOIN sdmx.code fr ON fr.codelist_urn='u:cl:FREQ' AND fr.code_id=o.FREQ

LEFT JOIN, so an unmapped code keeps its row with a null label; no lookup and no N+1, because the URN is resolved at bind time rather than per row.

Which set operations narrow, measured against polars' equivalents:

union       [1,2,3] u [3,4] -> [1,2,3,3,4]   subset of a? False   WIDENS
intersect   semi-join       -> [3]           subset of a? True    narrows
difference  anti-join       -> [1,2]         subset of a? True    narrows

So intersect and difference are refinements and may be methods on a query; union is not, and a union method would let a later caller undo an earlier helper's scoping — the exact failure query-composition.md's principle exists to prevent. Hence sl.union(a, b) as a free function.

Filtering on a window expression¶

polars permits it directly:

df.filter(pl.col("version").rank("ordinal", descending=True).over("agency","id") == 1)
-> ESTAT/CPI 2.0, ESTAT/GDP 1.0

SQL needs a subquery wrapper, so the compiler must emit one:

SELECT * FROM (SELECT *, rank() OVER (PARTITION BY agency,id ORDER BY version DESC) rn
               FROM art) WHERE rn = 1

Two new features that do not commute¶

Checked against the third composition law ("every ordering either commutes or is refused"). Both fail it, and neither was refused:

filter(p).filter(rank().over(...) == 1)   -> 3 rows
filter(rank().over(...) == 1).filter(p)   -> 2 rows       commute? False

filter(keep).unique(agency)               -> 1 row
unique(agency).filter(keep)               -> 0 rows       commute? False

The first is inherent — a rank is computed over whatever set precedes it, and SQL runs WHERE before window functions. DuckDB supports QUALIFY, which names the stage so the order is fixed by vocabulary rather than by call order:

SELECT * FROM art WHERE version <> '2.0'
 QUALIFY rank() OVER (PARTITION BY agency,id ORDER BY version DESC) = 1

The second has a further problem: unique(subset) also leaves which row survives unspecified. Full-row unique() has neither problem, so the subset form is excluded rather than given a precondition.

Types enforce both fixes. .over() returns Win[R] whose comparisons give WinPred[R], which is deliberately not a Pred[R]:

filter(<WinPred>)      error: Argument of type "WinPred[DsdRow]"
qualify(<Pred>)        error: Argument of type "Pred[DsdRow]"
unique(q.dsd.agency)   error: Expected 0 positional

basedpyright 0/0/0, pyrefly 0 errors. pyrefly additionally required @override on Win.__eq__ / Win.__ne__, which override Col's — a real base class rather than object, which is the case CLAUDE.md keeps missing-override-decorator for. basedpyright did not flag it.

Prototype log¶

Prototype scope¶

The prototype covered less than the design:

The join compiler handles depth-1 to-one paths against one hard-coded edge. Multi-hop paths, shared-prefix deduplication and alias collision were unimplemented in the prototype; query/_joins.py built them later.
The codelist->code fan-out timing above is of hand-written SQL in the shape the design would emit, not of generated output.
No order_by / limit / offset / group_by, so none of query-composition.md's refinement laws are exercised over a joined query.
Hierarchy fan-out is unexplored and is the harder case. "Each node with its descendants" is recursive, and hierarchy_node bakes depth / ancestor_ids precisely so descendants need no recursion. Whether .descendants.list(...) maps onto the baked columns or needs a recursive CTE is open — and narrower_transitive already exists as a scoped verb, which is the replace-or-coexist question in concrete form. (Since answered: query-language.md §2.7 and query-tree-paths.md.)
Binding a URN list with unnest(?) is the ~48 us/value path #271 measured. (That rate was duckdb with pandas absent; with pandas importable it is ~1-2 us — DuckDB retries import pandas per bound value when it is missing (#411). The shape holds either way. See tests/unit/test_value_set.py.) Fine at 60 URNs; at 5,000 it is what value_set exists for.

The three throwaway prototypes from the original pass¶

Three throwaway prototypes were written and run while producing this document: join-by-table with declared edges; ibis-style handles with inferred joins; and a component query with is_in composition. None is committed. They exist only as evidence that the compiled SQL and the ergonomics were observed rather than imagined, and the SQL each produced is quoted above.

The typed prototypes, and what each settled¶

Roughly twenty throwaway prototypes were written and type-checked while producing these documents. None is committed; they exist as evidence that the typing was observed rather than imagined. Every result below is from a run.

probe	question	result
standalone query model	does a separate query model type where instrumenting attrs does not?	basedpyright 0/0/0; instrumenting loses all static checking
`edge()` witness	can a typed factory record a path with no annotation and no suppression?	yes; one `cast` inside the helper, 0 errors
lowercase singletons	does `sl.q.component` type as well as `sl.q.Component`?	both checkers clean, 9 `assert_type` pass; closed the "pyrefly unverified" caveat
two entry points	can `artefacts(T)` and `dataflows()` coexist?	yes; all three cross-combinations rejected
root-bound row kind	does `edge()` return the target's row kind or the root's?	the root's — the first version failed to compile
`Many` witness	`many("x", ComponentCols)` vs `many("x", ComponentRow)`	pyrefly caught the accessor/row-kind mismatch basedpyright accepted
generic helper	can one helper scope any root?	yes, at the cost of threading the accessor as an argument
path-is-proof	does an undeclared hop fail?	`q.dataflow.concept` → Cannot access attribute "concept" for class "DataflowCols"
parent-level `.any()`	does it type?	yes — vacuously; accepts `q.dsd.any(q.metadataflow.id == …)`
reachability union	can the root check be recovered?	yes — then killed by `Pred.__and__` making `Pred` invariant
`Fan[R]` marker	can the type force EXISTS/list?	dead — `Pred[Fan[Fan[R]]]` on nested fan-out
dot-through	child accessor at the parent's row kind	both checkers clean; nests to any depth; cross-root still rejected
`.every()` modifier	does `Pred[R].every()` type?	trivially; the SQL is where the trouble is
HOAS binder	does a callback give fresh aliases?	yes; self-join works by going up then down, not by nesting
polars-shaped surface	namespaces + expression methods, typed	both checkers clean
`.str` shadowing	does a `.str` namespace shadow the builtin?	yes, at runtime; basedpyright silent, pyrefly catches it

pyrefly caught three things basedpyright did not across this work — the Many witness mismatch, the .str shadowing, and the empty-body stubs. That was a data point for what later made pyrefly the only type checker (#425; CLAUDE.md, Typing conventions), and at the time an argument for running both on any prototype before trusting it.