Repository navigation
feat: add the experimental listelement scalar index over list<struct> elements - #10950
Conversation
|
Important Format specification voteThis PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer), at least one of them on the latest commit, and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process. Approvals carry over across pushes, so a rebase or a typo fix does not send everyone back to re-vote. Whoever approves the latest commit is vouching that nothing substantive has changed since the earlier approvals; if something has, ask for fresh votes. Status: ❌ Blocked — 0 of 3 required approvals
Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the |
2a4e1a9 to
bb2f416
Compare
… elements
Index chosen fields of each element of a list<struct> column, for
example {"keys": ["label", "confidence"]}, and answer any_element
filters from the index with every condition on the same element. Each
element's key tuple is encoded with Arrow's row format and stored in a
BTree over Binary keys, posting the row address; conditions the keys
can't express are rechecked by the scan.
Also fixes optimize_indices on stable-row-ID tables dropping old entries
of this row-address index outside fragment 0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he overlay rescan domain - Record the key fields' IDs after training, and resolve them against the current schema before planning, opening or updating the index, so it follows renames and is skipped when a key field is dropped or retyped. Plugins only see Arrow fields, so this lives beside the other per-type details handling in the lance crate. - Mark the plan for recheck when the NULL set can't be determined (residual conjuncts, NULL literals including inside IN, null-sensitive tests), so NOT evaluates the call instead of trusting the index. - Merge old index data and rescan overlays in the index's actual row domain (old_data_address_domain), separately from the legacy-migration classification, so optimize after a column overlay no longer keeps stale postings on stable-row-ID tables. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bb2f416 to
854700c
Compare
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The rebase preserves the reviewed implementation and the fix for stale overlay matches. No further implementation changes requested.
Keep the declared dependency sequence: merge the separate format specification in #10949, then rebase this implementation.
Summary
Adds
listelement, an experimental built-in scalar index over fields inside the elements of alist<struct>column. It answersany_elementfilters (#10901) from the index, with every condition on the same element:This is the second part of #10900. The format is specified in #10949, and the design discussion is #10948.
Depends on #10949. This PR carries the identical
protos/index.protochange and generated source so that it compiles. Those changes drop out when this branch is rebased after #10949 merges.Design
keysfields as a tuple, encoded with Arrow's row format (ascending, nulls first), so the bytes sort like the tuples.Binarykeys. The index wraps the BTREE the way the JSON index wraps its target.visit_scalar_functionclaimsany_element(<indexed column>, '<literal>'). The predicate is planned exactly asany_elementplans it, then with the tuple planner (plan_tuple_predicate) over the keys.=,INorIS NULLprefix, then one range on the next key, becomes byte-range seeks in the BTREE.any_element.NOTis exact.lance-table, the index joins the row-address and eager-remap lists. On stable-row-ID tables, the fix(index): fence older writers from row-address indices #10352 writer fence applies and compaction remaps the index.lance-datafusionAPI: it exportsANY_ELEMENT,any_element_predicateandplan_element_predicate, so the index plans the same expression the scan evaluates.lance-indexnow depends onlance-datafusionas a normal dependency; there's no cycle.Fix included
On a stable-row-ID table,
optimize_indicesdropped old entries of a row-address index outside fragment 0. The generic update arm inmerge_scalar_indicesonly treated the JSON index as storing row addresses, so it filtered row addresses against row IDs. That arm now also covers this index. JSON behavior is unchanged.Tests
IN, label-only prefix, recheck;NOT;lance-table: row-address and eager-remap classification.fast_searchplusoptimize_indices; delete plus compaction remap, with and without stable row IDs.Docs
The read and write guide covers creating the index and which filters it answers, and the scalar indexing guide has a row for it. Both mark it experimental.
Not in this PR
LANCE_MEM_POOL_SIZE=8GiB.Closes #10900.
This conflicts with #10947 in the dataset scanner tests and the guide. Whichever merges second will rebase.
Lint:
cargo fmt --all, andcargo clippy --no-deps -p lance-datafusion -p lance-index -p lance-table -p lance --tests --benches --features lance-datafusion/substrait -- -D warnings. With dependencies included, clippy on arm64 currently fails inlance-encoding(unreachable_codefrom #10905), which this PR doesn't touch.🤖 Generated with Claude Code