The query language: naming relations and joining them¶
Status: built. This is the design of the path language as it ships in
src/sdmxlib/query/ — what it is, the rules it follows, and the reasons for
them. The surface is sl.q (one accessor per stored relation, 48 of them),
reg.query(root), and the Query / Select / Grouped / Page objects it
returns. It compiles to one DuckDB statement, serves RestRegistry by fetching
and then compiling, and reaches a dataflow's observations through
reg.obs(flow).
The query-language pages. This one carries the design — the rules, the arguments for them, and what is still open.
query-language-examples.mdis the surface construct by construct, andquery-language-cookbook.mdapplies it to the questions consumers were hand-writing SQL for.query-language-evidence.mdcarries what was run while the design was argued — peer surveys, schema measurements, benchmarks, the prototype log.query-language-explorations.mdcarries branches that were prototyped and rejected, kept because each failure is why a live choice is right.Neighbours:
query-composition.mdowns how clauses compose within one relation (the refinement laws);query-pushdown.mdowns where a query runs;query-tree-paths.mdowns the four tree edges;query-naming.mdowns what things are called;data-plane-query.mdanddata-plane-pushdown.mdown the observation side. For a guided introduction, read the query tutorial first.
Claims here cite the other files rather than restating them; a measurement appears inline only where the argument turns on the number.
import sdmxlib as sl
q = sl.q
frame = (
reg.query(q.dataflow) # a plan; nothing has run
.filter(q.dataflow.dsd.dimensions.concept.id == "REF_AREA")
.select(flow=q.dataflow.id, dims=q.dataflow.dsd.dimensions.id.list())
.collect() # one SELECT, a pl.DataFrame
)
1. What problem it solves¶
Before #306 there were four query DSLs — cl.codes over sl.code, h.nodes
over sl.node, reg.artefacts(T) over sl.artefact, and
reg.reported_attributes() over sl.meta — sharing ~2,400 lines of
_query_common. Each was roughly 300 statements of filter / order_by /
limit / select repeated per relation, and none could cross from one
relation to another. When #276 needed another table it produced a predicate
compiling to a correlated EXISTS (uses_concept, uses_codelist,
in_category), leaving the row kind unchanged.
The question that broke it (#280) is the one 21 hand-written consumer sites ask: every dataflow × dimension × codelist as one frame. A projection could not span relations because the row kind was singular, and resolving each structure instead cost 265 ms against 1 ms as one statement for 200 DSDs × 12 dimensions — 331x. Across the surveyed consumers (#275 / #276 / #281) there were 113 hand-written SQL sites; the shapes they fall into are the cookbook's chapters.
The answer is one language over the whole storage graph:
- one entry point,
reg.query(root), taking anysl.qaccessor; - paths as column references —
q.dataflow.dsd.dimensions.concept.idis a column of the dataflow query, reached through declared edges; - one compiler, producing a single SQL statement whatever the paths cross;
- one plan, which a second backend (SDMX REST) consumes without the language changing.
The four DSLs are deleted. What survived of _query_common is
query/_common.py — the refinement laws and canonical_key, the part that was
never about emitting SQL. sdmxlib.sql still publishes SqlRelation, which is
the composition that had external consumers; it no longer publishes
Predicate / CompileContext / RowKind, so a downstream library can inject a
relation but not a predicate.
2. The rules¶
Stated compactly; several were reached by correcting an earlier draft, and the
corrections are no longer interesting — the branches tried and rejected are in
query-language-explorations.md.
2.1 A path appears exactly where there is a decision¶
Where the query model can reach a column without choosing anything, there is no
path. The 1:1 identity split between an extension table and sdmx.artefact is
physical layout, not a join — sdmx.dataflow is urn PRIMARY KEY against the
same urn, and there is no codelist table at all. So the maintainable
surface — agency, id, version, is_final, valid_from, valid_to,
name, description, uri, urn, annotations — is on every maintainable
accessor with no prefix, by inheritance from a _Maintainable[R] base. Every
accessor roots at the table holding its columns, which for a maintainable is
always sdmx.artefact; an edge that lives on the extension is declared as a
compound hop across the split (_Dataflow.dsd is ("dataflow", "dsd")).
A reference edge — component.dsd vs component.concept vs
component.codelist — is a real choice and is always named.
Guard: hoisting is sound only while each identity split stays PK-to-PK and
mandatory. tests/unit/test_query_edges.py::test_the_identity_split_is_one_to_one
fails loudly if one becomes optional or non-unique, because hoisted columns
would otherwise silently multiply or drop rows.
The validity columns are worth naming because they were stored from the
schema's first version and not reachable for a while: valid_from / valid_to
are Temporal columns, so as-of filtering is an ordinary comparison
(q.codelist.valid_to < cutoff, q.codelist.valid_to.is_null()).
2.2 Name the edge, not the table¶
Both peer surveys in query-language-evidence.md agree: nobody joins tables;
everyone names the edge — SQLAlchemy via the relationship attribute, Django
via the FK path, Cypher via the relationship type, GraphQL by making every hop a
field. A prototype that joined by table and picked the first matching edge
worked, and silently chose a route when two existed — the one design in the
survey with no precedent. The schema has two routes to almost everything (a
component reaches a codelist directly and through its concept), so
q.component.codelist and q.component.concept.codelist are two spellings for
two questions and neither is guessed.
2.3 Accessors are lowercase singletons, and sl.q. compiles¶
sl.q.component, not sl.Component. The accessor does four jobs — namespace
for column names, target of an edge, root anchor for reg.query(), and
row-kind witness — and only the last needs a type, which is kept separate
(ComponentRow, CodeRow, …). The accessor classes are private and only the
singletons are exported, so "never instantiated by a caller" needs no guard.
That spelling carries the law separating the two traversals of this graph.
Ref.__getattr__ resolves eagerly and returns domain objects; sl.q compiles to
SQL. The discriminator is lexical, present at every call site:
sl.q.compiles. Everything else resolves.
dim.codelist.codes # resolves now, returns Code objects
sl.q.component.codelist.codes # compiles to SQL
An earlier draft used uppercase classes and made class-versus-instance the
discriminator, SQLAlchemy's rule (User.name is an expression, user.name a
value). That is a real rule but an implicit one, and implicit rules are what
222, #255, #261 and #292 cost here. The prefix is visible. What it cannot do¶
is survive aliasing — c = sl.q.component removes it from the call site — the
same exposure pl.col aliasing has, accepted.
2.4 The root fixes the grain¶
reg.query(q.component) is one row per component; reg.query(q.dataflow) is
one row per dataflow. Edges up are to-one and denormalise onto each row; edges
down are to-many and never multiply rows (§3.3). Rooting at the child is what
reaches the flat (dsd, component) grain sl.polars.structure_frame returns,
which no single base relation expresses otherwise.
This is the only place in the library that traverses upward:
dsd.dimensions, Ref.__getattr__ and GraphQL all descend.
2.5 The row kind belongs to the root, not to the column's relation¶
A predicate is about a row of the query's root, reached through joins. So
q.component.dsd.agency is a Str[ComponentRow] even though agency_id lives
on artefact, and every accessor is generic over the row kind with the root
binding it:
q.codelist.id Str[CodelistRow] root
q.component.codelist.id Str[ComponentRow] the component's own representation
q.component.concept.codelist.id Str[ComponentRow] the concept's default
q.codelist and q.component.codelist are the same accessor class at different
row-kind parameters, which is the precise reason they cannot be confused. The
first prototype typed edges as returning the target's row kind, and the
positive cases failed to compile — a component query rejected its own path. So
this is a hard requirement on the typing, not a preference.
A predicate built for one root and handed to another is refused by the checker
(Pred[CodelistRow] is not a Pred[ComponentRow]). At runtime filter refuses
a predicate over a different table — filter() got a predicate about
'artefact', but this query is rooted at 'dsd_component'. Two maintainable roots
share sdmx.artefact, so reg.query(q.dataflow).filter(q.codelist.agency ==
...) is caught only statically; the column it names is the same one, so the
answer is not wrong, but the code says something it does not mean.
2.6 These are path expressions, not patterns¶
sl.q.component is a closed nominal type: the dsd_component relation. A path
is a single statically-known route compiling to a fixed join tree; nothing is
matched and the result is single-valued.
| deterministic path expression | pattern / path-matching | |
|---|---|---|
| examples | JPQL c.dsd.agency, EF Core c => c.Dsd.Agency, Django dsd__agency, GraphQL selection |
XPath, JSONPath, Cypher MATCH, SPARQL property paths, Gremlin, jq |
| wildcards / alternation / recursion | none | *, //, p1\|p2, [:R*1..3] |
| a path denotes | one route, known statically | a set of routes, resolved at match time |
We are the left column; the nearest precedents are JPQL, whose spec term for
c.dsd.agency is literally path expression, and EF Core navigation paths.
(That table is recalled rather than run, unlike the surveys.) The right column
is forbidden rather than deferred, by the same argument as §2.2: a wildcard
path over a graph with two routes to everything resolves to an ambiguous union,
and a SQL row shape would have to pick, union, or refuse.
2.7 Hierarchy fan-out and recursion: a named verb, never path syntax¶
A hierarchy's nodes, a codelist's codes and a scheme's categories are fan-outs
like any other. Recursion is not. Variable-length paths — Cypher's
[:NARROWER*1..2], SPARQL's narrower+ — never enter path syntax; the
transitive questions stay named domain verbs on the lazy artefact
(cl.narrower_transitive("EU"), cl.roots(), cl.leaves()), and the compiler
never emits a recursive CTE.
That costs almost nothing, because the transitive closure is baked at write
time: every tree relation stores depth and an ancestor_ids list, so
q.code.ancestors / q.code.descendants are ordinary to-many edges matched by
membership in a stored list — one containment test, not a recursion. The four
tree edges (parent, children, ancestors, descendants) are the subject of
query-tree-paths.md §5. The corollary is the property
the compiler rests on: every path is finite and compiles to a fixed join
tree.
2.8 A separate query model, not an instrumented domain¶
The domain model already declares edges — 45 Ref-typed fields. The first
instinct was to make the query language be that graph, with
SQLAlchemy-style descriptors so sl.Dataflow.agency == "ESTAT" built a
predicate. Both were prototyped:
- Instrumenting attrs works at runtime, but a bare descriptor loses static
checking (
Dataflow.version == "1.0"infers asbool), and recovering it needsMapped[T]-style annotations on every queryable field — query types in the domain model's signatures. - A standalone query model types cleanly and can mirror storage honestly. That
matters because the
Refgraph and the storage join graph overlap without either containing the other: storage has containment edges the model expresses asItemListmembership, side tables with no domain counterpart (annotations, localised text), and different names where the two do correspond. Deriving edges fromReffields would be wrong, not merely redundant.
This is SQLAlchemy Core's split — Table / Column independent of any mapped
class — and it is why sdmxlib.query imports from the model but the model
imports nothing from it.
3. Edges, fan-outs, and how a path compiles¶
3.1 Edges are declared once and read both ways¶
query/_edges.py declares one Edge per storage foreign key — 110 of them.
dsd.components (down) and component.dsd (up) are the same
dsd_component.dsd_urn, declared once with an inverse name (SQLAlchemy's
back_populates, Django's related_name), so the compiler keeps a single edge
table and the two directions cannot drift.
Cardinality is derived, not declared. Forward is to-one exactly when the
target column is unique; backward is to-one exactly when the source column is.
tests/unit/test_query_edges.py checks both, and optionality, against the live
schema — because a mis-declared cardinality fails silently, multiplying or
dropping rows, where a missing column fails loudly. The case that motivated it
is in query-language-examples.md §5: "each
component with its dataflow" was written as a to-one path in seven design
examples, and three real rows showed it duplicating one component and dropping
another. The same file also requires every reference-shaped column to be an
edge or carry a written reason for not being one.
An accessor attribute is the user-facing side of an edge: edge(...) for a
to-one hop, many(...) for a to-many one, whose target class mixes in FanOut
so the cardinality change is visible in the type. Some to-many attributes are
facets of one edge — dsd.dimensions, dsd.attributes, dsd.measures are
dsd.components restricted by component_type, and dimensions excludes the
time dimension.
3.2 To-one hops: one join tree, shared prefixes¶
A path is a tuple of attribute names; _joins.py resolves it against the edge
graph into hops, which is also where an undeclared route is caught — a path is a
proof the route exists. All the paths one scope needs become one join tree:
two paths off one root, dimensions.concept.id and dimensions.codelist.id,
produce one join to dsd_component, not two. That deduplication is a
requirement rather than an optimisation — the hostile review measured
correlated-subquery cost as linear in the number of fan-out columns
(query-language-evidence.md, "Correlated subqueries: linear in fan-out
columns").
3.3 To-many hops: a correlated subquery, never a join¶
A path that crosses a to-many hop compiles to a correlated subquery, never a join.
EXISTS in a filter, an aggregate subquery in a projection. That is what makes
q.dsd.dimensions.concept.id == "REF_AREA" mean some dimension while leaving
the row count alone — the property #254 chose EXISTS for — and the reason two
fan-outs in one filter cannot produce a cartesian product the way two joins
would.
Django is the counter-example: Codelist.objects.filter(codes__cid__in=["A",
"B"]) is an INNER JOIN returning the codelist twice, and the documented fix
is .distinct(). Adopting Django's implicit spelling for to-many would import a
defect class we do not have.
So a to-many column has no single value per row, and the language says so
rather than picking one: projecting q.dsd.dimensions.id bare, grouping by it,
or using it inside a grouped aggregate raises PathError naming the fix —
collect it (.list(), .count(), .n_unique()) or root the query at the
child.
3.4 Dot-through and .any(): independent children versus one child¶
Dotting through a to-many hop is the common case and pays no ceremony. But two predicates over one to-many edge are ambiguous in every path language:
reg.query(q.dsd).filter(
q.dsd.dimensions.concept.id == "REF_AREA",
q.dsd.dimensions.codelist.id == "CL_GEO",
)
means either "some dimension has that concept and some dimension has that
codelist" or "one dimension has both", and they return different rows. Django
distinguishes them by chaining (filter(a, b) vs filter(a).filter(b)), which
is unavailable here because chained filter is AND by a law with property tests
behind it. The rule:
Predicates inside one
.any()call land on one child. Predicates outside it are independent.
reg.query(q.dsd).filter(
q.dsd.dimensions.any(
q.component.concept.id == "REF_AREA",
q.component.codelist.id == "CL_GEO",
)
)
.any(*preds) takes child-rooted predicates (Pred[ComponentRow], which is
what q.component.* produces); a parent-rooted one is a type error (and, today,
a database binder error at runtime rather than a refusal naming the fix). The child
accessor is noise when a fan-out holds one predicate and is exactly what marks
the grouping when it holds several. polars makes the same distinction with
pl.element() bound by .list.eval(); ours differs only in being typed.
.any() is a binder by position: nothing in the text connects
q.component to dimensions except being that call's argument —
SQLAlchemy's shape (Dsd.dimensions.any(Component.x == 1)). Nested binders of
different relations compose, each .any() binding its own child. Two bindings
of the same relation at once are not expressible (§12).
3.5 .every(): null-safe, and vacuously true¶
FanOut.every(*preds) is the universal and takes the same child-rooted
predicates as .any():
Its SQL has two traps, found on a fixture of A (all dimensions CL_GEO), B
(mixed), C (one dimension with a NULL codelist) and D (no dimensions):
some -> ['A', 'B']
every -> ['A', 'C', 'D'] naive NOT EXISTS (... AND NOT pred)
every -> ['A', 'D'] null-safe
C is wrong under the naive form: NOT(NULL) is NULL, so a non-enumerated
dimension is not counted as a violation and silently satisfies "every dimension
is CL_GEO". The compiler negates with IS NOT TRUE, so a child whose column
is NULL counts as a violation. D is vacuous truth — standard logic, a
standard surprise, documented rather than fixed.
every composes freely: inside another every, inside any, and across nested
to-many edges (q.codelist.codes.every(q.code.annotations.type == "NOTE")
crosses two). An earlier draft predicted refusals for zero or several to-many
hops; the compiler that shipped needs neither, and
tests/unit/test_query_compile.py::test_every_composes_across_nested_to_many_edges
keeps it so.
3.6 Nullable to-one edges: the join kind follows the predicate¶
component.codelist is absent for a non-enumerated component, so this is the
common case. Measured against Django:
filter(cl__name='Areas') -> INNER JOIN orphan dropped
values('cid','cl__name') -> LEFT OUTER JOIN orphan kept, value None
exclude(cl__name='Areas') -> LEFT OUTER JOIN orphan RETURNED
The compiler joins a mandatory to-one hop with JOIN and an optional one with
LEFT JOIN (optionality, like cardinality, is read off the schema), so a
projection keeps the row with a null value, and a positive filter drops it
because a null cannot satisfy it. Negation follows SQL's three-valued logic: ~(q.component.codelist.id ==
"CL_FREQ") and != are both NULL for a component with no codelist, so
neither returns it. That is the documented reading of ~ (SQL's NOT) — a row
whose column is NULL satisfies neither p nor ~p — and differs from
Django's exclude. To include the orphans, say so:
(q.component.codelist.id != "CL_FREQ") | q.component.codelist.id.is_null().
~ over a to-many edge is NOT EXISTS, a true complement.
3.7 An aggregate over a fan-out is also a predicate¶
q.code.children.count() == 0 — "how many children, and is that zero" —
compiles to a correlated scalar subquery in the WHERE, row-preserving like
every other fan-out. This is the affirmative way to ask about an empty fan-out,
the one the peers converged on (polars list.len() == 0, ibis length() == 0,
Django annotate(Count(...)).filter(n=0)); Django's other spelling,
children__isnull=True, is a LEFT JOIN plus IS NULL and is deliberately not
copied. Such a comparison composes with & / | like any predicate.
4. Pushdown: one language, several backends¶
The goal¶
Where a query runs is a property of the backend, not of the language. A
caller writing reg.query(q.dataflow).filter(...).select(...) should not have
to know whether the structures live in a local DuckDB or behind an SDMX REST
endpoint.
The plan nodes¶
The design proposed a small node set — Scan, Filter, Project,
Join(edge), Nest / Explode, Aggregate, Sort, Limit — small because
the schema is fixed. When the second backend was built, every one of them
turned out to exist already, and that measurement is why there is no
separate IR (query-pushdown.md §2):
| node | where it lives |
|---|---|
Scan |
Plan.root — the accessor, carrying its table and discriminator |
Filter |
Plan.predicates; windowed filters in Plan.qualifiers |
Join(edge) |
not stored — derived from the paths, because a path names its route |
Sort |
Plan.order_keys / default_order |
Limit |
Plan.limit_n / offset_n / cursor |
Project, Aggregate, Nest |
Select — its items and group keys |
Explode |
not built: one .explode() call in polars (§6) |
query/_plan.py's Plan[R] is the question with nothing about answering it;
Query is a Plan plus its wiring (a connection, a resolver, and for REST a
prepare hook that fetches before the first statement). Every clause evolves a
new frozen Plan, and canonical_key keys it by meaning, so
filter(a).filter(b) and filter(b).filter(a) are one question.
Capability, program, and the REST backend¶
query/_pushdown.py puts a Plan and a backend's Capability together and
produces a Program — an ordered sequence of Push and Residual steps, not a
(pushed, residual) pair, because REST answers some questions in two calls
with a local step between. DuckDB's capability is total, so its program is one
Push and the emitted SQL is unchanged.
SDMX REST's structure plane filters on agency / id / version only, plus
detail= and references=. So RestRegistry.query(...) does not interpret a
query remotely at all: it fetches what the plan needs, materialises it into a
local store, and runs the original query through the compiler that already
exists (query/_execute.py). The load-bearing part is the fetch's closure
(query/_closure.py): the fetch must contain every relation the query's paths
reach, counting crossings into other artefacts rather than hops, because a
fetch one relation short does not fail — it returns nothing. references=
comes from there (query/_rest.py).
Refuse rather than silently fetch¶
Partial pushdown has one catastrophic failure mode: a filter REST cannot
evaluate, over an unbounded scan, means downloading a registry to keep three
rows — the 800-resolutions problem, arrived at silently. The planner refuses
instead: a scan that leaves anything to filter locally must be bounded by an
agency, or the terminal raises PushdownError naming the column to add. A bare
listing, or equality on id or version, needs none. The rule, the per-scan bound
and why limit cannot be pushed past a residual are
query-pushdown.md §5.
explain() runs nothing and shows the SQL and its binds; for a query that will
fetch first it says so on its own line, because the SQL alone reads as a local
scan.
Why the earlier no-IR decision does not apply¶
query-composition.md decided against an intermediate representation for the
structural DSLs. Its reasoning was about expressiveness — 13 of 18 collected
shapes were expressible, and the other five were feature decisions an IR would
not deliver — and that measurement stands. But it was taken with one backend,
so the only question an IR could answer was what a query can say. Two
backends ask where a query runs, which is orthogonal. In the event the IR was
the Plan the query already carried, so the conclusion "build no node set"
survived while its premise did not.
Observations are the same plan, one plane over¶
reg.obs(flow) roots the same Query machinery at a dataflow's
observations (§9). Over REST it pushes key-dimension equality and membership
into the data key and attribute filters into SDMX 3.0's c[...], and evaluates
the rest locally on what came back — the data plane is a strong pushdown target
where the structure plane is a weak one. See
data-plane-pushdown.md.
5. Strict laziness¶
Nothing executes until a terminal. reg.query(...), filter, select,
order_by and every expression build descriptions; the terminals run them:
| terminal | returns |
|---|---|
collect(annotations=True) |
artefacts / items (Query), or a pl.DataFrame (Select) |
ids(), urns() |
list[str], no object construction |
count(), exists() |
COUNT(*) / EXISTS, fetching no rows |
first(), one(), one_or_none() |
one object, or None, or a raise |
frame(), stream() |
the relation's stored columns as polars / Arrow batches |
preview(n) |
a bounded head, for a REPL |
page(n) |
a Page of urns plus a cursor |
explain(), to_sql() and columns run nothing. Every terminal leaves the
query unchanged: a plan is a value, refined and run as often as wanted.
No implicit execution, anywhere¶
polars refuses len(lf), iter(lf), lf[0] and bool(lf) deliberately; the
old DSLs implemented all four, and each was an un-announced query. for c in
cl.codes inside a loop over codelists is the N+1 shape measured at 12x in
query-language-evidence.md ("Fan-out and N+1"). So a Query has no
__iter__, __len__ or __contains__, and each raises naming the terminal to
use instead. The middle position — keep the dunders where a receiver bounds the
scope (cl.codes) — does not hold, because bounded-per-receiver is exactly the
shape that produces N+1 across receivers.
What it buys: uniform laziness is the precondition for pushdown — a planner
cannot push what has already been evaluated, and the REST prepare hook can
fetch at the terminal precisely because reg.query(q.dataflow) alone has not
yet said which agency it means. And explain() is total: the plan a caller
inspects is the whole of what will happen.
in was wrong on its own terms, too. Measured, polars, LazyFrame and ibis all
implement __contains__ as column-name membership, and SQLAlchemy's Select
not at all, so "CA" in cl.codes would read to a polars user as "is there a
column called CA". And the old CodeQuery.__contains__ looked the id up against
the table, ignoring the query's own filters. q.filter(...).exists() cannot
bypass them.
[]: a slice refines, a position is refused, a string looks up¶
q[1:3]isoffset(1).limit(2)and inherits their laws — polars' and Django's rule,[]may refine.q[0]raises: positional extraction needs a total order to mean anything and is a terminal wearing an operator.q["FR"]looks the row up by id and returns the object (KeyErrorif absent);q.get("FR")returnsNoneinstead. This isItemListparity —codelist.codeshas to answer the same questions whether the codelist is aCodelistor aLazyCodelist— and it is unambiguous against the other two because an id is never a position.
The design at one point proposed the string form as lazy sugar for
filter(id == ...). It shipped as an extracting lookup instead, because the
docs and callers use codes["FREQ"] for ItemList parity throughout; the
lazy reading is one filter away.
Collections extract, plans refine¶
| family | operates on | filter | sort | terminal |
|---|---|---|---|---|
| query — lazy | a plan | filter(predicate) |
order_by |
collect() |
| collection — materialised | objects in memory | filter(lambda) or sl.item(...) |
sort_by |
— |
Lambdas are for materialised collections; predicates are for queries. This
is the sort_by / order_by split query-composition.md already calls
deliberate — the name is the signal — extended to filter.
That retired the REST-backed Query[T] (reg.query(sl.Dataflow, ...)). Its
only distinguishing feature was filter(lambda) evaluated in Python over
everything fetched, and strict laziness bans exactly that — so once it was lazy
and compiled it was the path language under another name. The measurement was
worse than the argument: its filter / limit ran after the fetch, so
query(Dataflow, agency="ECB").limit(5).fetch() read as a bounded request and
was a full catalogue download trimmed in Python; and filter appending to the
receiver and returning self was #291. The project breaks cleanly pre-stability
rather than carrying both, and RestRegistry.query(sl.q.X) is the replacement.
Kept beside reg.query: reg.artefacts(T) (a class-to-discriminator mapping
over the shared maintainable surface, for helpers that cannot name a root) and
reg.reported_attributes() (every reported metadata value) — both return the
same Query. And the lazy artefacts' named verbs (cl.narrower(...),
cl.roots(), …) return Querys too.
What it costs¶
Discoverability: a lazy object prints a plan, not data — the standing
complaint against ibis and polars, mitigated the same way, with a repr that
shows the plan, explain(), and preview(). One word at every site that used
to iterate: .collect().
6. Projection and aggregation¶
select(...) turns a Query into a Select, whose collect() returns a
pl.DataFrame and whose columns are known without running anything.
- Names. A positional column projects under its own name; a localised one
carries its language (
name["en"]→name_en), so two languages cannot collide. Two columns that would share a name —q.dataflow.idandq.dataflow.dsd.id— are refused with a message asking for keywords. An aggregate must be named. Selecthas nofilter.selectchanges what a row is, so a predicate afterwards would be about the projection; it raises "filter first". ASelectrefines withorder_by(by column, aggregate or projected name),limit,offset,uniqueand, when grouped,having.- A one-column
Selectis anSqlRelation, so it splices intois_inas a lazy subquery (§7.6).
Nesting is the default; exploding is polars'¶
Graph APIs split two ways on fan-out. Cypher, Gremlin and SPARQL return tables
and multiply rows because the pattern says so; GraphQL returns trees. Cypher has
both, via collect(), and that maps onto this stack exactly:
So a fan-out in a projection nests: nesting cannot silently multiply anyone's
row count, and exploding is an explicit request made in polars, where the caller
already knows the verb (frame.explode("dims")). Measured on 300 dataflows × 12
dimensions: exploded 0.7 ms / 3,600 rows; nested 0.9 ms / 300 rows. No
server-side explode is built.
The array-aggregation refusal was wrong¶
query-composition.md once recorded list(annotation.title) per code as a
refusal: aggregating side-table rows "would mean the join that EXISTS exists
to avoid". The premise was false. A correlated list subquery aggregates with no
join and no grouping:
SELECT c.code_id,
(SELECT list(an.title) FROM sdmx.annotation an
WHERE an.owner_urn = c.code_urn AND an.annotation_type = ?) AS notes
FROM sdmx.code c
Row count preserved — the exact property EXISTS was chosen for in #254. That
is what every projected fan-out compiles to.
list() of several columns returns structs¶
Two list() calls over one child are two independent correlated subqueries, and
DuckDB's list() has no defined order without ORDER BY. So labels[i] is not
reliably the label of codes[i], and the two diverge silently the moment either
gains an order (measured in query-language-evidence.md, "list() alignment").
The fix makes alignment structural rather than promised: the fan-out's own
list() takes several columns and returns one list of structs, in one
subquery.
polars' group_by().agg() and ibis's collect() keep two lists aligned only
because each comes from a single grouping pass — evidence that everyone gets
alignment from construction, not that two separate calls are safe.
Order inside a fan-out is a property of the expression, polars' shape:
q.codelist.codes.id.sort_by(q.codelist.codes.position).list(), or .sort() to
order by the value itself (the only order unique() accepts, since DISTINCT
collapses the rows any other key came from). Unsorted, an aggregate is
scan-ordered; adding ORDER BY to every aggregate would cost a sort nobody
asked for.
One aggregate, three readings¶
The aggregates are polars' names — list, unique, count, n_unique, sum,
mean, min, max, and sl.len() for rows. The same expression means
different things in different clauses, decided by the clause that receives it:
| where it appears | reading | SQL |
|---|---|---|
select(...) / filter(...), over a path crossing a to-many hop |
per row, over that row's children | correlated subquery |
group_by(...).agg(...), or Query.agg(...) |
over the group's rows | GROUP BY aggregate |
followed by .over(...) |
over the outer rows sharing a partition | window function |
That context-dependence is the polars-pipeline hazard query-composition.md
warns about, adopted deliberately: polars' Expr layer has it too (§7.5).
where= restricts what one aggregate reads, in both readings — SQL's
FILTER (WHERE ...) (#373). Over a fan-out it chooses which children
(c.count(where=q.code.depth == 0)); in a grouped aggregate it chooses which of
the group's rows. It is evaluated per candidate row, so the
same-child / different-child question .any() answers does not arise, and AND /
OR are ordinary & / |. A tuple is refused — a variadic form would only be a
second spelling of &. The fan-out's own list(..., where=) / count(where=)
types the predicate to the child's row kind, so prefer it to the column's.
Nothing matching gives 0 or [].
Grouping¶
group_by(*keys) returns a Grouped; .agg(**named) or .len() makes it a
Select. It is not a refinement — it changes shape — so it is refused after
order_by or a slice (a row sort key that is not a group key is not valid SQL,
and a limit before grouping aggregates a truncated set). A to-many path can be
neither a group key nor the column of a grouped aggregate (§3.3): a correlated
subquery has no value to aggregate across a group, and the language says so
rather than inventing a join. having(...) takes aggregate predicates on a
grouped Select and is refused on an ungrouped one, naming filter instead.
unique() is full-row only¶
Select.unique() is SELECT DISTINCT over the projected row. The subset form,
unique(agency), is excluded: measured, filter(keep).unique(agency) and
unique(agency).filter(keep) return different rows, and it leaves which row
survives unspecified. A caller wanting one row per group has
group_by(...).agg(...), which says which. Excluding it keeps unique in the
commuting set, so the third composition law needs no carve-out.
The aggregate Col.unique() / FanOut.unique(...) is a different thing — the
distinct values inside one fan-out — and both names are polars', where
DataFrame.unique() and Expr.unique() divide the same way.
7. Composition: laws, set operations, windows, paging¶
7.1 The refinement laws hold across paths¶
Everything query-composition.md guarantees holds unchanged: filter ANDs,
limit takes the minimum, offset raises the floor, order_by replaces, and
filter / qualify / order_by after a slice raise SliceError rather than
silently running before the slice. The generated properties in
tests/unit/test_properties_query_laws.py — monotonicity, idempotence, and
every ordering either commutes or is refused — run over path predicates like
any other.
7.2 Set operations: union is a free function¶
sl.union(a, b) # a FREE FUNCTION, never a method
a.intersect(b) # semi-join — a method
a.difference(b) # anti-join — a method
The split is the refinement principle, not style: nothing a later caller adds
can undo what an earlier call established. Measured against polars,
intersect and difference return a subset of a; union does not.
helper_scoped_to_estat.union(everything) as a method would undo the helper's
scoping, which is exactly what the principle forbids; as a free function it
composes two queries into a third rather than refining either. All three
require one root — cannot combine a query over dsd with one over codelist —
including two maintainable roots (q.dsd and q.dataflow), whose rows are
different relations even where their columns are shared.
7.3 Windows and qualify¶
.over() supplies the framing; the window functions are separate —
rank, dense_rank, row_number, lag, lead — and any aggregate can be
framed too (q.component.id.count().over(q.component.dsd.id)). "The latest
version of each artefact" needs both a window function and a filter on its
result, and that filter is a separate clause:
reg.query(q.dataflow).qualify(
q.dataflow.version.rank(descending=True).over(q.dataflow.agency, q.dataflow.id) == 1
)
qualify is SQL's own name for a post-window filter, and it exists because a
windowed predicate does not commute with an ordinary one: a rank is computed
over whatever set precedes it.
filter(p).filter(rank().over(...) == 1) -> 3 rows
filter(rank().over(...) == 1).filter(p) -> 2 rows commute? False
Naming the stage moves the order-dependence into the vocabulary, so filter
commutes with filter and qualify is a later stage whatever the call order.
The type enforces it: .over() returns Win[R], whose comparisons produce
WinPred[R], deliberately not a Pred[R] — the checker rejects a WinPred in
filter and a Pred in qualify. A Win stays a Col, so a windowed column
is projectable; only the filter position is restricted. == / != build the
predicate; the orderings are methods (.le(3), .lt, .ge, .gt) — "the three
newest" is row_number(...).over(...).le(3). Version columns order by the
version sort key baked beside them, so 1.10 ranks after 1.9 (#311).
7.4 Keyset paging¶
limit / offset degrade at depth. page(n) returns a Page of n urns plus
a Cursor[R], and after(cursor) resumes from it; after narrows, so it is a
refinement like offset. Cursor[R] is parameterised on the row kind, so a
cursor from one root cannot be handed to another.
after() needs no precondition about order, because every query already emits
the root's key as a tiebreak whether or not the caller sorted — so the order is
always total and an unsorted query pages by key: arbitrary but stable, which is
the whole requirement. What after() refuses is a cursor from a differently
shaped order (a different number of keys, or reversed), which would compare
the wrong columns and skip rows silently. order_by(None) declines the
tiebreak — for stream(), which reads everything once — and is therefore
refused together with limit or a cursor (#389).
The comparison is nested rather than SQL's row constructor: (a, b) > (?, ?)
cannot express keys whose directions disagree, and "newest version first, then
id ascending" is the common shape. Nulls follow the emitted NULLS LAST.
7.5 Have we become a pipeline?¶
A fair worry, since query-composition.md names polars as the standing hazard:
"its vocabulary is borrowed here, but it is a pipeline". The answer is no,
because polars has two layers with different laws:
| polars layer | semantics | what we took |
|---|---|---|
Expr — pl.col("a") > 5 |
declarative; construction order irrelevant, the receiving context gives meaning | this — paths as expressions, namespaces, column methods |
LazyFrame — .head(3).filter(p) |
a pipeline; order matters throughout | not this — the refinement laws stay |
So filter still ANDs, limit still takes the minimum, and filter-after-slice
is still refused. The vocabulary is polars'; the composition laws are ours, and
that division is deliberate rather than the "names without the laws" failure
that produced #222, #255, #261 and #292.
The two features that did not commute — windowed filters and unique(subset) —
were caught by the third law before either was settled, and each was fixed the
cheaper of two ways: the window by naming its stage (qualify), the subset
unique by excluding it. What it does share with polars' Expr is
context-dependent meaning (§6, "One aggregate, three readings") — taken
knowingly, and the reason the receiving clause is always visible.
7.6 Relations inside is_in, and "matched by either of two routes"¶
is_in takes a list of literals or any one-column SqlRelation, compiled to
one IN (SELECT ...). A Select of one column is one, so a structural query
filters another lazily without leaving the language:
european = cl.narrower_transitive("EU").select(q.code.id)
cl.codes.filter(q.code.id.is_in(european))
Both sides must read one database: a relation from another registry is refused
with CrossSourceError, because the subquery would silently read this
database under the other one's names.
The same mechanism settles how "matched by either of two routes" is spelled —
uses_concept matched a DSD carrying a component and a dataflow whose DSD
did. Checked against the peers, SQLAlchemy writes or_(exists(...),
exists(...)), ibis t.c.isin(a) | t.c.isin(b), polars an is_in over two
sets; none reaches for a set operation, because union is for stacking result
sets, not for one set matched two ways. So it is an | of two is_in
sub-queries, which composes because it is a predicate:
dsds = reg.query(q.dsd).filter(q.dsd.dimensions.codelist.id == "CL_UNIT").select(q.dsd.urn)
flows = reg.query(q.dataflow).filter(q.dataflow.dsd.dimensions.codelist.id == "CL_UNIT").select(q.dataflow.urn)
reg.query(q.artefact).filter(q.artefact.urn.is_in(dsds) | q.artefact.urn.is_in(flows))
8. Columns, literals and text¶
Column types¶
Each column is typed for what it can do, and the operators follow:
| type | e.g. | carries |
|---|---|---|
Str |
q.code.id |
==, !=, is_in, .str, .len(), + (concatenation) |
Label |
q.code.name["en"] |
as Str, plus the unaccented / folded rungs |
Vocabulary |
q.component.role |
as Str, with the legal values checked |
Version |
q.dataflow.version |
compares as text, orders by version sort key |
Num |
q.code.depth, q.code.position |
orderings and arithmetic |
Temporal |
q.codelist.valid_to |
orderings; projects as a UTC instant |
Bool |
q.dsd.is_final |
==, != |
Arr |
q.component.concept_roles |
.has(value) |
Untyped |
obs.col("REF_AREA") |
comparisons and .str, for columns known only at runtime |
Every column has is_null(), is_not_null(), is_in(...), .desc(), the
aggregates and the window functions.
Literals are checked; comparisons are column against literal¶
q.code.id == 1 raises code_id holds str, got int. Did you mean "1"?; ==
None raises naming is_null() (= NULL is never true); is_in("CA") raises,
because a string is an iterable of characters (#261 — found only by placing the
API beside polars, which rejects it). A Vocabulary column refuses a value it
can never hold — q.component.role == "dimension" names 'Dimension' and the
legal values — because == "Codlist" matches nothing and != "Codlist"
matches everything, and neither is the question.
A comparison's right-hand side is a literal: comparing two columns
(q.code.id == q.code.parent.id) is refused by the same literal check. Computed
columns are columns, so q.code.position + 1 == 2 and q.code.id.len() == 2
work, as does projecting q.code.codelist.id + ":" + q.code.id. Column-to-
column comparison is open (§13).
Operators that cannot be overloaded¶
and / or / not cannot be overloaded in Python, so on an expression they
would silently drop an operand; bool() on a predicate raises instead, polars'
precedent. Predicates combine with &, |, ~, and since Python binds & /
| tighter than ==, (a == 1) | b == 2 parses as ((pred) | column) == 2 —
refused on both arms, including the reflected one ("A" | ... calls
str.__or__, so a guard on Pred alone never sees it). Equality on a
predicate is refused too (it used to report two different columns equal); query
identity is canonical_key().
Text matching is a ladder, under .str¶
caseless, unaccented and folded were paid for by #222 and #255:
q.code.name["en"].str.contains("region") # exact
q.code.name["en"].str.caseless.contains("region") # ignore case
q.code.name["en"].str.unaccented.contains("Montreal") # + diacritics: finds "Montréal"
q.code.name["en"].str.folded.contains("Trois Rivieres") # + case, punctuation
q.code.id.str.caseless == "cl_freq" # == only at this rung
Three things are kept exactly:
caselesscarries==and!=;unaccentedandfoldeddo not. The differential oracles encode it: both fold case forcontains/starts_with/ends_withwhile==stays case-sensitive (#222).caselessapplies to ids, not only labels — a caller typing an id in the wrong case is where #222 begins, and SDMXIDTypetreatsFandfas distinct codes that can coexist.unaccentedandfoldedexist on labels only.- They are properties, not methods, matching polars' namespace convention.
LIKE metacharacters in a needle are escaped (#222): "50%" matches the text,
not a pattern.
A hazard found prototyping this, worth knowing for any namespace named after a
builtin: a str property shadows the builtin inside the class body, so
def f(self, x: list[str]) annotates list[<property>]. The fix is
builtins.str (query-language-evidence.md, "The .str namespace shadows the
builtin").
9. Observations¶
reg.obs(flow) returns an ObsQuery — the same Query machinery
rooted at one dataflow's observations, on both LocalRegistry (via a
DataflowBinding that says which table holds them) and RestRegistry (fetched
as SDMX-CSV into a synthesised binding).
The two planes share operators and the plan, not the naming mechanism.
Structural column names are fixed and declarable, so q.component.id is an
attribute and type-checks. Observation column names come from the DSD at
runtime, so they are named by string — obs.col("REF_AREA") — and typed
Untyped: comparisons, orderings, aggregates and the .str rungs work, and a
typo is a KeyError listing the components that exist. The accessor is built
per binding, so the query carries it: obs.col(...), obs.components.
col() rather than obs["REF_AREA"], which was the first cut: Query.__getitem__
already means look up a row by id for a string, and overriding it on one root
would give one spelling two meanings depending on the root. An attribute form,
obs.REF_AREA, was rejected for looking statically checked without being so.
Everything else is the structural surface: filter, qualify, group_by,
select, order_by, the terminals. Labels are the one observation-specific
terminal option — obs.frame(labels="en") adds <COMPONENT>_label_en columns
for coded components, resolved in one statement against the stored codelists,
with the language always named. filter_components(...) applies a parsed SDMX
REST component filter, for code serving SDMX REST.
The design reasoning for the data plane — what consumers' hand-written SQL
needed, the binding, what pushes into a REST data key — is in
data-plane-query.md and
data-plane-pushdown.md.
10. The escape hatch¶
Removing raw SQL was never the intent. The ban in query-composition.md is
narrow and still correct: the structural DSL must not accept SQL text, because
its safety property is that a caller supplies field references and literals and
nothing else. That is an argument against merging the two representations, not
against a separate, clearly-marked escape. And an escape hatch is what tells us
what to build next — closing it removes the evidence of a need, not the need.
reg.sql("SELECT agency_id, count(*) FROM sdmx.artefact GROUP BY 1").collect()
reg.sql("SELECT code_id FROM sdmx.code WHERE code_id LIKE $p", p="E%").collect()
sl.schema.code.code_id # 'sdmx.code.code_id' — checked against the DDL
The rules it carries:
- A different method on a different object.
reg.sqlis visibly notfilterat the call site, which is the property the ban protects. - Values are bound, never interpolated.
$nameparameters are rewritten to?and checked against DuckDB's own parse.reg.sql(...)is lazy;.collect()runs it. - It is a terminal. It briefly also spliced into
is_inas aSqlRelation; that was withdrawn. A raw statement carries no row kind to check against the column it lands beside (the failure is an empty result, not an error); it cannot be pushed to a non-DuckDB backend, so one splice would pin a whole query to DuckDB; and a painless crossing hides the gap the hatch exists to report.RawSqldoes not satisfySqlRelation, andis_innames both ways across: materialise (reg.sql(...).collect()["col"]), or use a query, which composes lazily. - Schema constants (
sl.schema) reflect the DDL, so a stale table or column is anAttributeErrorat the point of use listing what is there, rather than a SQL error naming neither the call site nor the fix.
SqlRelation itself is untouched, because it was never the raw-SQL seam: a
Query / Select satisfies it, and so do external producers (statcatlib's query
builders). Both are compiled relations. Raw text is not.
11. The row kind: what it is for, and what it costs¶
RowKind is a pure phantom — a never-instantiated marker per relation
(CodeRow, ComponentRow, …). Its only job is to make a predicate built for one
root un-handable to another, and it is the price of module-level accessors.
Every peer binds a column to a relation handle — ibis t = con.table(...) then
t.col; SQLAlchemy User.name — and a free-floating predicate has no handle to
carry its relation, so the relation rides in the type instead. The alternative,
binding the accessor to the query (c = reg.components(); c.filter(c.dsd.agency
== ...)), removes the phantom and costs standalone predicates, which costs
helper composition.
Before #306 the row kind was a Protocol with behaviour (column(), emit(),
urn_column); the query model moved that onto the accessor, which already names
its table and columns, leaving the bare marker. Verified before the change that
nothing was lost: the check comes from the parameter, not the Protocol.
Dropping the parameter would be a different proposition — Pred[R] becoming
Pred would let a cross-root predicate type-check and emit a WHERE against a
table absent from the FROM.
What it costs is that a shared-surface predicate cannot be written once and applied to two roots. A root-generic helper takes the accessor as an argument:
def scope_to_agency(query, root, agency):
return query.filter(root.agency == agency)
scope_to_agency(reg.query(q.dataflow), q.dataflow, "ESTAT")
scope_to_agency(reg.query(q.codelist), q.codelist, "ESTAT")
That extra parameter is the price. Annotating the helper is where it pinches:
the shared maintainable accessor class is private, so root has no public type
to name, and an unannotated helper is checked only at runtime — where a
mismatched pair of maintainable roots shares a table and is not caught (§2.5). A q.cols on the query was
considered and not built — a second spelling for every column reference,
drifting toward the bound-accessor model.
Query[R, A] carries a second parameter, A, for what the object-building
terminals return — reg.query(q.dataflow).collect() is list[Dataflow]. It
comes off the root accessor (Cols[R, A]) and is invariant, which is why set
operations take Query[R, A].
12. Risks carried¶
The design was attacked deliberately once it looked settled
(query-language-evidence.md, "Hostile-review measurements"). One objection
changed the shape — the same-child conjunction, answered by .any(*preds)
(§3.4). These are carried rather than resolved:
- Context-dependent meaning is structural. One expression is
EXISTSin a filter, a correlated aggregate in a projection, aGROUP BYaggregate inagg, and a window after.over(). Deliberate (§7.5), and the receiving clause is always visible — but it is the hazardquery-composition.mdnames. - Predicates cannot be reused across roots (§11). A shared predicate library is impossible by construction.
- Correlated subqueries are linear in fan-out columns. Correct always; cost
grows with the number of fan-out columns projected. Shared-prefix
deduplication (§3.2) is what bounds it, and
GROUP BYwould beat it where row count may change. - Pushdown correctness is safety-critical. Refuse-rather-than-fetch is only
as good as the capability declaration; a permissive error silently restores
the 800-resolution problem.
query-pushdown.md§8 lists the gates. - Self-joins have no general spelling. There is one module singleton per
relation, so two rows of one relation cannot be bound at once (§13). The
common cases are aggregates in disguise — "a DSD with two dimensions sharing a
concept" is a group of components with
len() > 1— and the cookbook works them.
13. Open questions¶
Questions the implementation answered are stated as decisions in the sections above. These remain open.
- Column-to-column comparison.
q.code.id == q.code.parent.idandq.component.codelist.id != q.component.concept.codelist.idare refused by the literal check, so "a component overriding its concept's representation" and "a code sharing its parent's label" have no spelling. Comparing two aggregates (count() > n_unique()) is not refused cleanly either — it fails at the database. Whether to admit a column operand, and with what null semantics, is open; it meets item 4. - Self-joins. Two bindings of one relation at once (a DSD with two
dimensions sharing a concept, answered here only via grouping). A
higher-order binder —
dimensions.any(lambda a: a.dsd.dimensions.any(lambda b: ...))— was prototyped and type-checked, and is not built; it would change.any()'s signature and needs both edge directions declared, which they now are. - Cross-schema joins over caller tables. A caller's own relation can filter
ours through
is_inwith any one-columnSqlRelation, or throughreg.sql. A caller declaring an accessor with an edge into our graph — so their dataset rows could dot through tocode.name["en"]— was prototyped and type-checked, and is not public: the column and edge factories are not exported, andEDGESis a closed table. Reverse edges (ours to theirs) are further off still. - Null-safe equality as an operator.
IS DISTINCT FROMis used internally forevery; callers comparing nullable values hit three-valued logic with no opt-out beyond writing theis_null()arm themselves (§3.6). - Any-language label search.
name["en"]picks one language; "matches in any language", the natural behaviour for a search box, needsmap_valuesand has no spelling — sharper given the rule that there is no default language. - Regex matching. Facets carry
repr_pattern; matching ids against one, or searching by pattern, has no operator. - Non-equi and range joins. "Artefacts whose validity overlaps this one" is a range predicate between two relations; every declared edge is an equi-join.
- As-of queries across relations.
valid_from/valid_tofilter one relation; "the catalogue as it stood on date X", consistently across several, is a different thing. - Value-column coverage. Reference-shaped columns are guarded — each is an
edge or carries a written reason. Plain value columns are not: an accessor
can omit a stored column and nothing fails. Report-only, or a ratchet like
tools/count_any.py? - Federation.
FederatedRegistryroutes by agency, which is also §5 ofquery-pushdown.md's bounding constraint and its natural materialisation unit; per-agency pushdown across registries is not prototyped (query-pushdown.md§14). - Struct access and
PIVOT. Reading back into alist(struct)column from an earlier stage has no spelling;PIVOTis a reshaping of a materialised frame, which polars does in one call — probably left there. - Whether the
sl.q.rule can be checked. It holds at every call site that writes the prefix; nothing stopsc = sl.q.component. Probably acceptable, aspl.colaliasing is.
14. Superseded and never built¶
Earlier versions of this page, the examples page and the cookbook presented these as designed or answered. None exists; the reasoning behind the shipped alternative is in the sections cited.
| was proposed | what exists instead |
|---|---|
named roots — reg.dataflows(), reg.components(), reg.codes() |
reg.query(q.dataflow) — one entry, the accessor is the witness (§5) |
four accessors sl.code / sl.artefact / sl.node / sl.meta, FrameQuery |
sl.q.<relation>, Query, Select |
form A — join(edge) then bare columns |
paths in the reference (form B); A needs aliased() the moment two routes are live, and lets a helper's bare predicate silently mean another join — query-language-explorations.md |
Many with verbs only (.any(lambda c: ...), HOAS binders) |
dot-through plus .any(*preds) / .every(*preds) on FanOut (§3.4) |
a postfix universal (pred).every() |
FanOut.every(*preds) (§3.5) |
.hier (descendants_of, narrower_transitive on a column) |
the four tree edges, plus the lazy artefact's named verbs (§2.7) |
.ann["NOTE"] |
annotations is a to-many edge like any other |
fill_null, sl.when(...).then(...).otherwise(...) |
not built; coalesce in polars after collect() |
q.obs[...] / q.obs.REF_AREA, flow.data() |
reg.obs(flow) and obs.col(...) (§9) |
.labelled(lang=) on an observation query |
obs.frame(labels="en") (§9) |
obs.text(...) / obs.num(...), a KeyedQuery split |
not built; observation columns are Untyped |
cl.codes["CA"] as lazy refinement |
it looks up and returns the row (§5) |
public sl.col / sl.edge / sl.Str / sl.CodeCols for caller accessors |
not public (§13, item 3) |
FederatedCatalog.from_parts(...).query(...) |
RestRegistry.query with store=; federation is open (§13, item 10) |
list() of separate calls, aligned "incidentally" |
list(*columns) returns structs (§6) |
Rejected branches with their full arguments — bare and contextual paths, a
parent-level .any(), the Fan[R] marker, a strict path with a bare inner
predicate — are in
query-language-explorations.md.