Skip to content

feat: add the experimental listelement scalar index over list<struct> elements - #10950

Merged
BubbleCal merged 2 commits into
lance-format:mainfrom
sarwarbhuiyan:feat/listelement-index
Nov 12, 2026
Merged

BubbleCal merged 2 commits into
lance-format:mainfrom
sarwarbhuiyan:feat/listelement-index

Conversation

@sarwarbhuiyan

Copy link
Copy Markdown
Contributor

Summary

Adds listelement, an experimental built-in scalar index over fields inside the elements of a list<struct> column. It answers any_element filters (#10901) from the index, with every condition on the same element:

dataset.create_scalar_index(
    "detections",
    IndexConfig(index_type="listelement", parameters={"keys": ["label", "confidence"]}),
)
dataset.to_table(filter="any_element(detections, 'label = ''car'' AND confidence > 0.9')")

This is the second part of #10900. The format is specified in #10949, and the design discussion is #10948.

Depends on #10949. This PR carries the identical protos/index.proto change and generated source so that it compiles. Those changes drop out when this branch is rebased after #10949 merges.

Design

  • Training: one entry per element.
    • The key is the element's keys fields as a tuple, encoded with Arrow's row format (ascending, nulls first), so the bytes sort like the tuples.
    • The posting is the row address.
    • Entries are sorted with spilling, and an ordinary BTREE is trained over the Binary keys. The index wraps the BTREE the way the JSON index wraps its target.
  • Search: visit_scalar_function claims any_element(<indexed column>, '<literal>'). The predicate is planned exactly as any_element plans it, then with the tuple planner (plan_tuple_predicate) over the keys.
    • Seeks: an =, IN or IS NULL prefix, then one range on the next key, becomes byte-range seeks in the BTREE.
    • Superset and recheck: conditions the keys can't express, such as a field that isn't a key, make the answer a superset. The scan then rechecks it with any_element.
    • No first key, no index: without a condition on the first key, the index declines.
  • NULLs: NULL lists, and elements on which the predicate is NULL, are tracked, so NOT is exact.
  • Maintenance: update encodes the new fragments the same way, then defers to the BTREE. Remap defers to the BTREE.
  • Row addresses: in lance-table, the index joins the row-address and eager-remap lists. On stable-row-ID tables, the fix(index): fence older writers from row-address indices #10352 writer fence applies and compaction remaps the index.
  • lance-datafusion API: it exports ANY_ELEMENT, any_element_predicate and plan_element_predicate, so the index plans the same expression the scan evaluates. lance-index now depends on lance-datafusion as a normal dependency; there's no cycle.

Fix included

On a stable-row-ID table, optimize_indices dropped old entries of a row-address index outside fragment 0. The generic update arm in merge_scalar_indices only treated the JSON index as storing row addresses, so it filtered row addresses against row IDs. That arm now also covers this index. JSON behavior is unchanged.

Tests

  • Index unit tests (13):
    • same-element trap, every range operator, IN, label-only prefix, recheck;
    • NULL rows for NOT;
    • shapes the index declines;
    • details round trip, and invalid parameters (no keys, unknown key, repeated key, nested key type, unknown parameter, non-struct list);
    • segment key mismatch, an all-null column, and the spec's encoding bytes.
  • lance-table: row-address and eager-remap classification.
  • Dataset tests: feat: add any_element filter for per-element predicates on list<struct> columns #10901's fixture is extended rather than duplicated.
    • Every filter case, plus the edge-row and invalid-predicate tests, runs with and without the index, checking the plan and whether it rechecks.
    • New: declining when the first key isn't constrained; fast_search plus optimize_indices; delete plus compaction remap, with and without stable row IDs.

Docs

The read and write guide covers creating the index and which filters it answers, and the scalar indexing guide has a row for it. Both mark it experimental.

Not in this PR

  • Training memory: there's no explicit budget. Like the JSON index, training uses the default spill pool. At 100M elements the prototype needed LANCE_MEM_POOL_SIZE=8GiB.
  • Merging segments for distributed builds.
  • Fragment reuse index: batch row-ID remapping isn't supported. Like JSON, the index is reported incompatible on stable-row-ID tables that use a fragment reuse index.

Closes #10900.

This conflicts with #10947 in the dataset scanner tests and the guide. Whichever merges second will rebase.

Lint: cargo fmt --all, and cargo clippy --no-deps -p lance-datafusion -p lance-index -p lance-table -p lance --tests --benches --features lance-datafusion/substrait -- -D warnings. With dependencies included, clippy on arm64 currently fails in lance-encoding (unreachable_code from #10905), which this PR doesn't touch.

🤖 Generated with Claude Code

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions github-actions Bot added A-python Python bindings A-index Vector index, linalg, tokenizer A-java Java bindings + JNI A-deps Dependency updates A-format On-disk format: protos and format spec docs A-docs Documentation format-change A change to the format spec, which requires a vote. Remove if minor (e.g. fixing typo). enhancement New feature or request labels Nov 11, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Important

Format specification vote

This PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer), at least one of them on the latest commit, and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process.

Approvals carry over across pushes, so a rebase or a typo fix does not send everyone back to re-vote. Whoever approves the latest commit is vouching that nothing substantive has changed since the earlier approvals; if something has, ask for fresh votes.

Status: ❌ Blocked — 0 of 3 required approvals

Approvals none (0/3)
Latest commit approved by none — one PMC member must approve the latest commit
Vetoes none
Voting period ends Mon 2026-11-16 14:04 UTC (06:04 PST)

Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the format-waived label to waive the vote for a trivial edit (typo, wording, formatting).

lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Nov 11, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Nov 11, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 11, 2026
sarwarbhuiyan and others added 2 commits November 11, 2026 11:58
… elements

Index chosen fields of each element of a list<struct> column, for
example {"keys": ["label", "confidence"]}, and answer any_element
filters from the index with every condition on the same element. Each
element's key tuple is encoded with Arrow's row format and stored in a
BTree over Binary keys, posting the row address; conditions the keys
can't express are rechecked by the scan.

Also fixes optimize_indices on stable-row-ID tables dropping old entries
of this row-address index outside fragment 0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he overlay rescan domain

- Record the key fields' IDs after training, and resolve them against
  the current schema before planning, opening or updating the index, so
  it follows renames and is skipped when a key field is dropped or
  retyped. Plugins only see Arrow fields, so this lives beside the other
  per-type details handling in the lance crate.
- Mark the plan for recheck when the NULL set can't be determined
  (residual conjuncts, NULL literals including inside IN, null-sensitive
  tests), so NOT evaluates the call instead of trusting the index.
- Merge old index data and rescan overlays in the index's actual row
  domain (old_data_address_domain), separately from the legacy-migration
  classification, so optimize after a column overlay no longer keeps
  stale postings on stable-row-ID tables.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 11, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

The rebase preserves the reviewed implementation and the fix for stale overlay matches. No further implementation changes requested.

Keep the declared dependency sequence: merge the separate format specification in #10949, then rebase this implementation.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-deps Dependency updates A-docs Documentation A-format On-disk format: protos and format spec docs A-index Vector index, linalg, tokenizer A-java Java bindings + JNI A-python Python bindings enhancement New feature or request format-change A change to the format spec, which requires a vote. Remove if minor (e.g. fixing typo). K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feature: index fields inside list<struct> elements (listelement index and any_element filter)

2 participants