Repository navigation
feat(format): add the listelement scalar index specification (experimental) - #10949
Conversation
|
Important Format specification voteThis PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer), at least one of them on the latest commit, and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process. Approvals carry over across pushes, so a rebase or a typo fix does not send everyone back to re-vote. Whoever approves the latest commit is vouching that nothing substantive has changed since the earlier approvals; if something has, ask for fresh votes. Status: ❌ Blocked — 0 of 3 required approvals
Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the |
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The field-identity and NULL-semantics findings are fixed. The contract now preserves child identity across renames and requires data evaluation for NOT when NULL rows cannot be determined exactly. I found no remaining blocker in this experimental format specification.
|
Gatekeeper's two findings are fixed, and it now recommends approval. This spec is marked experimental. Under the experimental rules, it can merge with a maintainer's review before its stabilization vote, as #9076 did. The design discussion is #10948. The stabilization PR, which removes the experimental markers, will carry the PMC vote. The implementation, #10950, also has Gatekeeper's approval. I'll rebase it as soon as this merges. @BubbleCal, could you review and merge this when you have a moment? |
…ental) Specify the persisted contract of the listelement scalar index, which indexes fields inside the elements of a list<struct> column and answers any_element filters: the ListElementIndexDetails message, the target BTree's files over Binary keys, the Arrow row key encoding with a worked example, and the queries a reader may answer from the index. Marked experimental in the proto and the docs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…valued NULL rule - Store the key fields' Lance field IDs (key_field_ids); keys is informational. A reader resolves each ID among the direct children of the indexed column's element struct, checks its logical type, and declines the index when a key field was dropped, moved or retyped. - Define an element's predicate as UNKNOWN when no conjunct is FALSE and at least one is UNKNOWN, found over every non-empty null subset of the constrained keys. Residual conjuncts, NULL literals (including inside IN), null-sensitive tests and an exceeded enumeration limit leave the NULL set undetermined, and NOT must then evaluate the call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
7012c7d to
c9d3adc
Compare
… elements (#10950) ## Summary Adds `listelement`, an experimental built-in scalar index over fields inside the elements of a `list<struct>` column. It answers `any_element` filters (#10901) from the index, with every condition on the same element: ```python dataset.create_scalar_index( "detections", IndexConfig(index_type="listelement", parameters={"keys": ["label", "confidence"]}), ) dataset.to_table(filter="any_element(detections, 'label = ''car'' AND confidence > 0.9')") ``` This is the second part of #10900. The format is specified in #10949, and the design discussion is #10948. **Depends on #10949.** This PR carries the identical `protos/index.proto` change and generated source so that it compiles. Those changes drop out when this branch is rebased after #10949 merges. ## Design - **Training:** one entry per element. - The key is the element's `keys` fields as a tuple, encoded with Arrow's row format (ascending, nulls first), so the bytes sort like the tuples. - The posting is the row address. - Entries are sorted with spilling, and an ordinary BTREE is trained over the `Binary` keys. The index wraps the BTREE the way the JSON index wraps its target. - **Search:** `visit_scalar_function` claims `any_element(<indexed column>, '<literal>')`. The predicate is planned exactly as `any_element` plans it, then with the tuple planner (`plan_tuple_predicate`) over the keys. - **Seeks:** an `=`, `IN` or `IS NULL` prefix, then one range on the next key, becomes byte-range seeks in the BTREE. - **Superset and recheck:** conditions the keys can't express, such as a field that isn't a key, make the answer a superset. The scan then rechecks it with `any_element`. - **No first key, no index:** without a condition on the first key, the index declines. - **NULLs:** NULL lists, and elements on which the predicate is NULL, are tracked, so `NOT` is exact. - **Maintenance:** update encodes the new fragments the same way, then defers to the BTREE. Remap defers to the BTREE. - **Row addresses:** in `lance-table`, the index joins the row-address and eager-remap lists. On stable-row-ID tables, the #10352 writer fence applies and compaction remaps the index. - **`lance-datafusion` API:** it exports `ANY_ELEMENT`, `any_element_predicate` and `plan_element_predicate`, so the index plans the same expression the scan evaluates. `lance-index` now depends on `lance-datafusion` as a normal dependency; there's no cycle. ## Fix included On a stable-row-ID table, `optimize_indices` dropped old entries of a row-address index outside fragment 0. The generic update arm in `merge_scalar_indices` only treated the JSON index as storing row addresses, so it filtered row addresses against row IDs. That arm now also covers this index. JSON behavior is unchanged. ## Tests - **Index unit tests** (13): - same-element trap, every range operator, `IN`, label-only prefix, recheck; - NULL rows for `NOT`; - shapes the index declines; - details round trip, and invalid parameters (no keys, unknown key, repeated key, nested key type, unknown parameter, non-struct list); - segment key mismatch, an all-null column, and the spec's encoding bytes. - **`lance-table`:** row-address and eager-remap classification. - **Dataset tests:** #10901's fixture is extended rather than duplicated. - Every filter case, plus the edge-row and invalid-predicate tests, runs with and without the index, checking the plan and whether it rechecks. - New: declining when the first key isn't constrained; `fast_search` plus `optimize_indices`; delete plus compaction remap, with and without stable row IDs. ## Docs The read and write guide covers creating the index and which filters it answers, and the scalar indexing guide has a row for it. Both mark it experimental. ## Not in this PR - **Training memory:** there's no explicit budget. Like the JSON index, training uses the default spill pool. At 100M elements the prototype needed `LANCE_MEM_POOL_SIZE=8GiB`. - **Merging segments** for distributed builds. - **Fragment reuse index:** batch row-ID remapping isn't supported. Like JSON, the index is reported incompatible on stable-row-ID tables that use a fragment reuse index. Closes #10900. This conflicts with #10947 in the dataset scanner tests and the guide. Whichever merges second will rebase. Lint: `cargo fmt --all`, and `cargo clippy --no-deps -p lance-datafusion -p lance-index -p lance-table -p lance --tests --benches --features lance-datafusion/substrait -- -D warnings`. With dependencies included, clippy on arm64 currently fails in `lance-encoding` (`unreachable_code` from #10905), which this PR doesn't touch. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Add the persisted contract of the
listelementscalar index, which indexes fields inside the elements of alist<struct>column and answersany_elementfilters (#10901). It is marked experimental. Design discussion: #10948. Issue: #10900.Details:
ListElementIndexDetailshas these fields:keys: direct children of the element struct, at least one, no repeats.key_types: Lance logical types. A reader declines the index when a key field is gone or its type changed.key_encoding: an enum.ARROW_ROW_V1is the only defined value, and UNSPECIFIED must not be used.target_detailsandtarget_index_version: the details and version of the inner BTREE.All segments of one index share keys, types and encoding.
Files: none of its own. The files are a standard BTREE over
Binary, given as PyArrow schemas, with one row per list element:The posting is the row address.
Key encoding: Arrow row format, every key ascending, nulls first. The admitted key types are the same as the
arrow_rowv1 set in docs(format): specify composite B-tree keys #10358. The byte layout is spelled out for fixed-width and variable-width types, with a worked example,("car", 0.5), that an implementation must reproduce.Queries:
=,INorIS NULLon the first key, then equalities on further keys in order, then one range on the next key.any_element.NOT.Row addresses: the index stores row addresses, so it joins the row-address and eager-remap lists. On stable-row-ID tables the fix(index): fence older writers from row-address indices #10352 writer fence applies.
Library edits are limited to the generated protobuf source. The implementation follows in a separate PR.
Experimental:
🤖 Generated with Claude Code