API Reference#
Transpilation#
- giql.transpile.transpile(giql, tables=None, *, dialect=None)[source]#
Transpile a GIQL query to SQL.
Parses the GIQL syntax and converts it to portable SQL for the active target (uses LATERAL joins where needed for operations like NEAREST). The output is not strictly SQL-92: depending on the target it may use engine extensions such as
LATERALorSELECT * EXCEPT(see thedialectparameter).Parameters#
- giqlstr
The GIQL query string containing genomic extensions like INTERSECTS, CONTAINS, WITHIN, CLUSTER, MERGE, NEAREST, or DISJOIN.
- tableslist[str |
Table] | None Table configurations. Strings use default column mappings (chrom, start, end, strand).
Tableobjects provide custom column name mappings.- dialectDialectName | str | Target | None
Optional target engine, as either a
giql.targets.Targetor the name of one. A Target carries the engine’s capability set;Noneselects the generic portable target,"duckdb"/"datafusion"the built-in targets, and any other name a customgiql.targets.Targetregistered on the plugin hub (seegiql.expander.register()/giql.expander.ExpanderRegistry.register_target()). Passing the instance directly is the way to use a one-off Target without registering it under a name, since registration is process-global and outlives the call.giql.targets.GenericTargetis accepted as an instance but has no selectable name –Noneis the one public spelling for it. When set to"duckdb", column-to-columnINTERSECTSjoins (INNER, SEMI, ANTI, and LEFT / RIGHT outer) are transpiled into a per-chromosome dynamic-SQL pattern (SET VARIABLE+query(getvariable(...))) that DuckDB plans through its range-join family (IE_JOIN/PIECEWISE_MERGE_JOIN). A LEFT or RIGHT outer join is decomposed into an INNER half, aNOT EXISTShalf carrying the preserved side’s unmatched rows, and a third branch carrying its NULL-chromosome rows, combined withUNION ALL; the first two reach the fast operator and the third is a filtered scan. Because the per-chromosome form emits one branch per distinct chromosome, it degrades to a single non-partitioned query above a partition-count ceiling, decided at execution time: the partitioned form wins by orders of magnitude on a primary assembly and loses by orders of magnitude on a scaffold-level one. Shapes the IEJoin path declines fall through to the generic naive overlap predicate; the performance guide enumerates them, and that enumeration is deliberately not repeated here, because a second copy drifts."datafusion"andNonealways emit that naive predicate — a plainON a.chrom = b.chrom AND a.start < b.end AND b.start < a.endcondition the engine’s own optimizer plans as a range join — for both inner and outer column-to-column INTERSECTS joins. For the INNER / SEMI / ANTI shapes, a projection the rewrite cannot attribute to a side raisesValueErrorat transpile time. The outer-join form instead declines silently to the naive predicate for every projection it cannot split, sodialect="duckdb"never changes an outer join’s result — only its plan. See the performance guide for the full enumeration.The target’s capabilities also choose the coordinate-canonicalization emit form for a non-canonically-encoded table:
"duckdb"emitsSELECT * REPLACE (...), while the generic (None) and"datafusion"targets emit the portableSELECT * EXCEPT (...), <start>, <end>form, which runs on* EXCEPT-capable engines (the DataFusion family) but not on DuckDB — passdialect="duckdb"for DuckDB-runnable output. Tables in the canonical 0-based half-open encoding are unaffected (they emit portable SQL on every target).
Returns#
- str
The transpiled SQL. Every target but
"duckdb"returns a single statement, as does"duckdb"for any shape its IEJoin path declines. An accelerated column-to-columnINTERSECTSjoin instead returns a multi-statement script: oneSET VARIABLEstatement, or two for a decomposed outer join, followed by the finalSELECT. Execute it on a single connection – a driver that splits statements, or forwards only the last, drops the variable theSELECTreads and yields empty or NULL results. Variable names are derived from the content they hold, so re-running one query shape rebinds its own names rather than leaving a fresh pair behind per call.
Raises#
- ValueError
If the query cannot be parsed or transpiled, or if
dialectis unknown.- TypeError
If an entry in
tablesis neither astrnor agiql.table.Table.
Examples#
Basic usage with default column mappings:
sql = transpile( "SELECT * FROM peaks WHERE interval INTERSECTS 'chr1:1000-2000'", tables=["peaks"], )
Custom
Tableconfiguration:sql = transpile( "SELECT * FROM peaks WHERE interval INTERSECTS 'chr1:1000-2000'", tables=[ Table( "peaks", genomic_col="interval", chrom_col="chrom", start_col="start", end_col="end", ) ], )
Column-to-column INTERSECTS join (naive overlap predicate; inner or outer, planned as a range join by the target engine):
sql = transpile( "SELECT a.*, b.* FROM peaks a JOIN genes b " "ON a.interval INTERSECTS b.interval", tables=["peaks", "genes"], )
DuckDB IEJoin dialect (column-to-column INNER/SEMI/ANTI/LEFT/RIGHT JOIN). INNER, SEMI and ANTI require qualified projections and raise otherwise; the outer-join forms decline to the naive predicate instead:
sql = transpile( "SELECT a.chrom, a.start, b.start " "FROM peaks a JOIN genes b ON a.interval INTERSECTS b.interval", tables=["peaks", "genes"], dialect="duckdb", )
- class giql.table.Table(name, genomic_col='interval', chrom_col='chrom', start_col='start', end_col='end', strand_col='strand', coordinate_system='0based', interval_type='half_open')[source]#
Genomic table configuration for transpilation.
This class defines how genomic intervals are stored in a database table, mapping a pseudo-column name (genomic_col) to the physical columns that store chromosome, start, end, and optionally strand information.
Instances are immutable.
coordinate_systemandinterval_typeselect the coordinate translation the emitted SQL performs, so a value outside the documented pair would produce a predicate matching no coordinate system at all. Validation therefore has to hold for the object’s whole lifetime, not only at construction – freezing is what makes__post_init__the single way in. Build a variant withdataclasses.replace()rather than by assignment. Immutability also makes the value hashable and value-equal.Parameters#
- namestr
The table name.
- genomic_colstr
The pseudo-column name used in GIQL queries to reference the genomic interval (default: “interval”).
- chrom_colstr
The physical column name storing chromosome/contig (default: “chrom”).
- start_colstr
The physical column name storing interval start position (default: “start”).
- end_colstr
The physical column name storing interval end position (default: “end”).
- strand_colstr | None
The physical column name storing strand information, or None if the table has no strand column (default: “strand”).
- coordinate_systemLiteral[“0based”, “1based”]
The coordinate system used for positions (default: “0based”).
- interval_typeLiteral[“half_open”, “closed”]
The interval endpoint convention (default: “half_open”).
Examples#
Using default column names (via transpile):
sql = transpile(query, tables=["peaks"])
Mixing default and custom table configurations:
sql = transpile( query, tables=[ "peaks", Table( "variants", genomic_col="position", chrom_col="chr", start_col="pos_start", end_col="pos_end", strand_col=None, # No strand column coordinate_system="1based", interval_type="closed", ), ], )
- __init__(name, genomic_col='interval', chrom_col='chrom', start_col='start', end_col='end', strand_col='strand', coordinate_system='0based', interval_type='half_open')#
Extension hook#
The registry-based extension point for custom targets and operator expanders
(see Extending GIQL: custom targets and operators). Every symbol below is re-exported from the top-level
giql package.
- giql.expander.register(target, operator)[source]#
Register an operator expander for
(target, operator)— the extension hook.The public decorator by which a built-in or user-supplied expander is added to the process-wide
REGISTRY:@register(DuckDBTarget, GIQLDisjoin) def expand_disjoin(node, ctx): ...
Parameters#
- targettype[Target] | Target
The target the expander handles, given as the target class (the usual form — it is instantiated with its default capabilities) or an explicit instance. Pass
GenericTargetfor the portable expander every target falls back to.- operatortype
The GIQL operator expression class the expander handles.
Returns#
- Callable
A decorator that registers its argument and returns it unchanged, so the decorated object stays usable on its own.
- giql.expander.REGISTRY = <giql.expander.ExpanderRegistry object>#
The process-wide registry the
register()decorator writes to and theExpandOperatorspass reads from. The built-in expanders register into it at import time viagiql.expanders; the pass rewrites every GIQL operator node by resolving its expander here.
- class giql.expander.ExpanderRegistry[source]#
Maps
(target, operator_type)to the expander that handles it.Resolution follows the epic’s fallback chain so a generic expander written once covers every target, and a per-target entry overrides it only where the engine genuinely differs:
(target, op)— an expander registered for the exact active target;(generic, op)— the portable expander registered forGenericTarget;None— no expander resolves. Every built-in operator registers a(generic, op)expander, so this only arises for a cleared registry or an operator with no generic fallback; the pass raises rather than emitting (there is no legacy*_sqlfallback).
The registry keys on the frozen, value-equal
Targetinstance (twoDuckDBTarget()compare and hash alike), so registering againstDuckDBTargetand resolving against a freshly builtDuckDBTarget()hit the same entry. Registering against the classDuckDBTargetis a convenience the decorator instantiates.- register(target, operator, expander)[source]#
Register expander for
(target, operator), overriding any prior.Parameters#
- targetTarget
The target instance the expander handles. Use
GenericTargetfor the portable fallback.- operatortype
The GIQL operator expression class (e.g.
GIQLDisjoin).- expanderOperatorExpander | ExpanderFn
The expander object or function. A later registration for the same key replaces an earlier one (last-write-wins override).
Notes#
Registering an expander also declares target by name (see
register_target()), so a custom target becomes selectable viatranspile(dialect=target.name)as soon as any expander is registered against it — a user overriding one operator need not also declare the target separately.A later registration for the same
(target, operator)key replaces the earlier one (last-write-wins), so a user override of a built-in target-specific expander — notably the DuckDB IEJoin(DuckDBTarget, Intersects)join rewrite (giql.expanders.intersects_duckdb) — simply supersedes it in the registry; the pass then dispatches the operator node to the user’s expander instead. (has_override()reports whether such an exact non-generic entry exists for introspection.)
- register_target(target)[source]#
Declare target so
transpile(dialect=target.name)resolves it.The registry doubles as the target plugin hub: a custom
Targetregistered here (directly, or as a side effect ofregister()) is resolvable by name throughgiql.targets.resolve_target(). This is the path for a capability-only target — one that overrides no operators (so it never callsregister()) but declares a distinctCapabilitiesset /sqlglot_dialect.The built-in
GenericTargetis intentionally not declared: it is the portable fallback, selected only viadialect=None(seegiql.targets.resolve_target()), never by name. The skip is by value (target == GenericTarget()), so registering an expander againstGenericTargetis a no-op here — but a customTargetsubclass is declared under its ownnameeven if that name happens to be"generic"(it does not compare equal to the built-in). A later registration for the same name replaces an earlier one.
- target(name)[source]#
Return the registered custom target named name, or
None.Consulted by
giql.targets.resolve_target()for any dialect name that is not a built-in, so a registered custom target is selectable viatranspile(dialect=name).
- resolve(target, operator)[source]#
Return the expander for
(target, operator)via the fallback chain.Tries the exact
(target, op)entry, then the(GenericTarget(), op)fallback, thenNone.Nonemeans no expander is registered; theExpandOperatorspass treats that as an internal invariant violation and raises (there is no legacy emitter).Resolution is the single dispatch path for every operator, including the DuckDB IEJoin join rewrite: on
duckdbthis returns the built-in(DuckDBTarget, Intersects)override (giql.expanders.intersects_duckdb), and a user registration for the same key replaces it (last-write-wins) so this returns the user’s expander instead (#169). There is no separate pre-pass for INTERSECTS anymore.
- has_override(target, operator)[source]#
Whether an exact non-generic
(target, operator)entry is registered.Returns
Trueonly when target is notGenericTargetand an exact(target, operator)entry is registered; the portable(GenericTarget(), operator)fallback is not an override and does not count here.Such an entry marks a target-specific override that supersedes the
(GenericTarget, operator)fallback — for example the built-in DuckDB IEJoin(DuckDBTarget, Intersects)join rewrite (giql.expanders.intersects_duckdb), or a plugin’s own override of it. This is an introspection helper (the dispatch itself flows throughresolve()); it is not a gate on any pre-pass, since INTERSECTS join strategy is fully registry-driven (#169) with no pre-pass to bypass.
- unregister(target, operator)[source]#
Drop the
(target, operator)entry if present.Part of the registry’s public teardown seam: a caller that registered a custom expander (a plugin, or a test fixture) undoes one registration with this rather than mutating private state. Resolving the same key afterward falls back through the chain to
None. Dropping an absent key is a no-op.
- clear()[source]#
Drop every registration — both expanders and declared targets.
The bulk form of
unregister(), and the public reset for the process-wideREGISTRY— e.g. a test fixture that saves and restores the registry around a body that registers custom expanders or targets.
- snapshot()[source]#
Return a shallow copy of the current registrations and targets.
The save half of a save/restore seam that supports test baseline-isolation: capture the baseline with this and hand it back to
restore()afterward, so the built-in registrations survive an isolating fixture that would otherwiseclear()them permanently.The returned
RegistrySnapshotis a fresh, opaque value bundling both the(target, operator)expander map and the declared-target map; mutating the registry afterward does not affect it, and two snapshots compare equal when the registry is in the same state.
- restore(snapshot)[source]#
Replace all registrations with those captured by
snapshot().The restore half of the save/restore seam that supports test baseline-isolation. Drops every current entry and re-installs exactly the snapshot contents (expanders and declared targets), so a fixture can return the registry to a previously captured baseline regardless of what its body registered or cleared.
- __contains__(key)[source]#
Whether an exact
(target, operator)entry is registered.Tests the exact key only — it does not walk the generic fallback chain
resolve()follows ((GenericTarget(), op) in registryis the only way(target, op)reports present via the generic entry). Part of the public teardown seam, for asserting a registration landed or was torn down.
- class giql.expander.RegistrySnapshot(expanders, targets)[source]#
An opaque save/restore token for
ExpanderRegistrystate.Bundles the
(target, operator)expander map and the declared-target map captured byExpanderRegistry.snapshot(). Hand it back toExpanderRegistry.restore()to return the registry to that state, or compare two snapshots for equality to assert the registry returned to a baseline (the leak-guard use).A snapshot is compared by equality and handed to
restore()only; it is not hashable (its twodictfields are mutable), sohash(snapshot)raises — never use one as a set member or dict key.- __init__(expanders, targets)#
- class giql.expander.ExpansionContext(node, resolution, target, tables, alias_seq=None, registry=None, finalizers=None)[source]#
Everything an
OperatorExpanderneeds to expand one operator.A fresh context is built per operator node by
ExpandOperators. It bundles the node’s pass-1 resolution metadata together with the active target, collision-safe alias minting, and the registered tables, so an expander never has to reach back into the pass or re-resolve anything.Attributes#
- nodeexp.Expression
The GIQL operator node being expanded.
- resolutionOperatorResolution | None
The metadata
giql.resolver.resolve_operator_refs()attached to node (its resolved reference slots, column operands, deferrals, and de-canonicalization output tables).Noneonly if the node was never annotated — an internal invariant violation an expander may assert on.- targetTarget
The active
Target, exposing itscapabilitiesandsqlglot_dialect.- tablesTables
The registered
Tablescontainer.- registryExpanderRegistry | None
The registry the pass is resolving against. Carried so a whole-query rewrite (CLUSTER / MERGE) can re-enter
expand_operators()over the SELECT it just restructured and expand sibling operators it copied into it, honoring a custom-registry pass run.Nonefor a standalone context built outside the pass.
A node-local expander that needs to rewrite the enclosing statement (rather than just replace its own node) registers a
StatementFinalizerviaadd_statement_finalizer(); the pass applies every finalizer to the statement after all node-local replacements. This is the query-level seam for a target whose expansion must reshape the enclosing statement — for example to project internal helper columns out of a surfacingSELECT *.- node#
- resolution#
- target#
- tables#
- registry#
- property capabilities#
The active target’s
Capabilities.
- add_statement_finalizer(finalizer)[source]#
Register a query-level
StatementFinalizerfor this run.The query-level seam: an expander is node-local — its return value replaces only its own node — so a target that must rewrite the enclosing statement (for example to project internal helper columns out of a surfacing
SELECT *) registers a finalizer here instead.expand_operators()applies every registered finalizer to the statement, in registration order, after all node-local replacements complete; each receives the current statement root and returns the (possibly new) root.A finalizer’s returned root is emitted verbatim — beyond a type check that it is an
Expression, it is not semantically re-validated — so it must not reference columns or relations absent from the projection it rewrites: a wrapper over an absent column builds without error but fails at engine runtime. Finalizers registered on a standalone context (one built outside the pass) are collected but never applied.
- class giql.expander.OperatorExpander(*args, **kwargs)[source]#
Turns one GIQL operator node into standard sqlglot AST for a target.
An expander is any callable-bearing object whose
expand()takes the operator node and itsExpansionContextand returns the sqlglot AST the node expands to for the active target. The returned expression replaces the operator node in the tree; the stock per-target serializer renders it.The protocol is
runtime_checkable(), so the registry can assert a registered object satisfies it. A plain function is not anOperatorExpander(it has noexpandmethod); register one by wrapping it (seeregister(), which accepts either form).An expander’s return value is node-local:
expand(node, ctx) -> exp.Expressionreturns the one expression that replaces the operator node in place. When a target additionally needs to rewrite the enclosing statement — for example to project internal helper columns away from a surfacingSELECT *— the expander registers a query-levelStatementFinalizerviaExpansionContext.add_statement_finalizer(), applied to the statement after every node-local replacement. The DuckDB INTERSECTS IEJoin whole-query fold uses exactly this seam: its(DuckDBTarget, Intersects)expander (giql.expanders.intersects_duckdb, #169) registers a finalizer that replaces the statement root with the per-chromosome dynamic-SQL rewrite.- __init__(*args, **kwargs)#
Targets and capabilities#
- class giql.targets.Target(name, sqlglot_dialect, capabilities)[source]#
A SQL target engine.
Subclasses declare the engine
name, thesqlglot_dialectused to serialize AST for that engine (Noneselects sqlglot’s default generic serialization), and the enginecapabilities.Targets are frozen, value-equal, and hashable: two
DuckDBTarget()instances compare equal and hash alike, so the operator-expander registry (#138) can key on a resolved target by value. Equality is class-scoped —GenericTarget() != DataFusionTarget()even where their fields overlap.- capabilities: Capabilities#
- __init__(name, sqlglot_dialect, capabilities)#
- class giql.targets.Capabilities(supports_lateral, supports_star_replace, supports_qualify)[source]#
Feature set of a SQL target engine.
Each field is a portable choice that later steps of epic #137 turn into a capability lookup instead of a hardcoded dialect branch.
Parameters#
- supports_lateralbool
Whether the engine supports
LATERAL/ correlated joins. Drives the NEAREST LATERAL-vs-window-function strategy (#142): a correlated NEAREST expands to a portable correlatedLATERALsubquery where this holds and to a decorrelated window-function form where it does not. This capability is the single source of truth — the former generator-levelSUPPORTS_LATERALattribute has been removed.- supports_star_replacebool
Whether the engine supports
SELECT * REPLACE (...). Drives every coordinate-canonicalization site — the canonicalizer wrapper CTE and the DISJOIN / NEAREST passthroughs (#143/#145):* REPLACEwhere supported, the portable* EXCEPTprojection otherwise. Supported by DuckDB / BigQuery / Snowflake / ClickHouse; not by PostgreSQL, SQLite, or DataFusion.- supports_qualifybool
Whether the engine supports the
QUALIFYclause. Reserved: no emission path consumes it yet (a future window-function operator port would).
- __init__(supports_lateral, supports_star_replace, supports_qualify)#
- class giql.targets.GenericTarget(name='generic', sqlglot_dialect=None, capabilities=Capabilities(supports_lateral=True, supports_star_replace=False, supports_qualify=False))[source]#
Portable SQL-92-ish target with no engine-specific features.
This is the default target (
dialect=None). Its capabilities are the conservative, maximally portable baseline, serialized through sqlglot’s dialect-less default (sqlglot_dialect = None).“SQL-92-ish”, not strict SQL-92: because
supports_star_replace=False, every coordinate-canonicalization site over a non-canonical target falls back to aSELECT * EXCEPT (...)projection (re-appending the recomputed interval columns) — the canonicalizer wrapper CTE and the DISJOIN / NEAREST passthroughs alike. A star-projected CLUSTER is a further* EXCEPTsite (independent of canonicalization): it hides its synthesized__giql_is_new_clusterflag with* EXCEPT(#184).* EXCEPTis not SQL-92 and is not DuckDB-runnable — it is a DataFusion-family extension — so the generic target’s output from any of these sites runs only on an* EXCEPT-capable engine (transpile withdialect="duckdb"to execute on DuckDB, where it spells the exclusionEXCLUDE). A canonical (0-based half-open) target with no star-projected CLUSTER passes the row through as a plain, fully portableSELECT *.- capabilities: Capabilities = Capabilities(supports_lateral=True, supports_star_replace=False, supports_qualify=False)#
- __init__(name='generic', sqlglot_dialect=None, capabilities=Capabilities(supports_lateral=True, supports_star_replace=False, supports_qualify=False))#
- class giql.targets.DuckDBTarget(name='duckdb', sqlglot_dialect='duckdb', capabilities=Capabilities(supports_lateral=True, supports_star_replace=True, supports_qualify=True))[source]#
DuckDB target.
Serializes through sqlglot’s
duckdbdialect and uses the IEJoin per-partition plan for column-to-column INTERSECTS joins, registered as the(DuckDBTarget, Intersects)override in the operator-expander registry (giql.expanders.intersects_duckdb, #169).Serializing through the
duckdbdialect makes null ordering explicit onORDER BY/ window terms, emittingNULLS FIRSTwhere the generic (dialect-less) serialization leaves it implicit. This is result-preserving for every migrated operator because each sorts on a non-null coordinate key (chromosome / position, or a same-chromosome-filtered distance), so the explicitNULLS FIRSTnever reorders a NULL into or out of the top rows. A future operator ordering on a nullable key would need to re-verify this before relying on the DuckDB serialization being a semantic no-op.- capabilities: Capabilities = Capabilities(supports_lateral=True, supports_star_replace=True, supports_qualify=True)#
- __init__(name='duckdb', sqlglot_dialect='duckdb', capabilities=Capabilities(supports_lateral=True, supports_star_replace=True, supports_qualify=True))#
- class giql.targets.DataFusionTarget(name='datafusion', sqlglot_dialect=None, capabilities=Capabilities(supports_lateral=False, supports_star_replace=False, supports_qualify=False))[source]#
Apache DataFusion target.
sqlglot has no DataFusion dialect, so serialization uses the generic form (
sqlglot_dialect = None); this is the finalized strategy (#145) — the portable SQL the generic generator emits runs on DataFusion, verified end-to-end by the cross-target oracle (tests/integration/datafusion/): every operator at the default encoding, and the canonicalizing operators (DISJOIN / NEAREST) across all four coordinate encodings plus custom-column and strand schemas.The capability values below are validated against a real DataFusion engine by that oracle:
supports_lateral=False(no correlated-LATERAL physical plan, so NEAREST takes the decorrelated window fallback),supports_star_replace= False(DataFusion supports* EXCEPT/* EXCLUDEbut not* REPLACE, so the canonicalizer and the NEAREST/DISJOIN passthroughs emit the portable* EXCEPTform), andsupports_qualify=False. DataFusion registers no INTERSECTS override, so a column-to-column INTERSECTS join falls back to the generic naive overlap predicate — a plainONcondition DataFusion plans as a hash join keyed onchromwith the position inequalities as a residual join filter (#167).A
SELECT */SELECT b.*over a correlated NEAREST would otherwise expose the decorrelated fallback’s reserved__giql_x_*rank/key columns; a statement finalizer wraps the enclosingSELECTinSELECT * EXCEPT (...)on the fallback path (#160), so the cross-target identity claim holds for star projections over a correlated NEAREST unconditionally. The finalizer re-locates its target join by a reservedmetatag and mints its reserved column names per fallback (#172), so the claim now holds through the two former residual compositions too: a correlated NEAREST re-surfaced by an enclosingSELECT *outside its own SELECT (e.g. a wrappingCLUSTER, whosecopy()+transplantthe tag survives) and two correlated NEAREST fallbacks in one query (whose now-distinct per-run reserved names each* EXCEPTindependently).- capabilities: Capabilities = Capabilities(supports_lateral=False, supports_star_replace=False, supports_qualify=False)#
- __init__(name='datafusion', sqlglot_dialect=None, capabilities=Capabilities(supports_lateral=False, supports_star_replace=False, supports_qualify=False))#