Skip to content

feat(format): add the listelement scalar index specification (experimental) - #10949

Merged
BubbleCal merged 2 commits into
lance-format:mainfrom
sarwarbhuiyan:docs/listelement-index-format
Nov 12, 2026
Merged

BubbleCal merged 2 commits into
lance-format:mainfrom
sarwarbhuiyan:docs/listelement-index-format

Conversation

@sarwarbhuiyan

Copy link
Copy Markdown
Contributor

Add the persisted contract of the listelement scalar index, which indexes fields inside the elements of a list<struct> column and answers any_element filters (#10901). It is marked experimental. Design discussion: #10948. Issue: #10900.

  • Details: ListElementIndexDetails has these fields:

    • keys: direct children of the element struct, at least one, no repeats.
    • key_types: Lance logical types. A reader declines the index when a key field is gone or its type changed.
    • key_encoding: an enum. ARROW_ROW_V1 is the only defined value, and UNSPECIFIED must not be used.
    • target_details and target_index_version: the details and version of the inner BTREE.

    All segments of one index share keys, types and encoding.

  • Files: none of its own. The files are a standard BTREE over Binary, given as PyArrow schemas, with one row per list element:

    • A null element is an all-null key tuple.
    • A null list is a single null key.
    • An empty list has no rows.

    The posting is the row address.

  • Key encoding: Arrow row format, every key ascending, nulls first. The admitted key types are the same as the arrow_row v1 set in docs(format): specify composite B-tree keys #10358. The byte layout is spelled out for fixed-width and variable-width types, with a worked example, ("car", 0.5), that an implementation must reproduce.

  • Queries:

    • What the index can answer: =, IN or IS NULL on the first key, then equalities on further keys in order, then one range on the next key.
    • Exact or rechecked: the answer is exact when the index covers every condition. Otherwise it's a superset that the scan rechecks with any_element.
    • NULL rows: they are found for NOT.
  • Row addresses: the index stores row addresses, so it joins the row-address and eager-remap lists. On stable-row-ID tables the fix(index): fence older writers from row-address indices #10352 writer fence applies.

Library edits are limited to the generated protobuf source. The implementation follows in a separate PR.

Experimental:

  • Forward compatible: readers that don't know the type ignore the index and scan.
  • Backward compatible: nothing existing changes.
  • Droppable: dropping the index never rewrites data.

🤖 Generated with Claude Code

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions github-actions Bot added A-index Vector index, linalg, tokenizer A-format On-disk format: protos and format spec docs format-change A change to the format spec, which requires a vote. Remove if minor (e.g. fixing typo). enhancement New feature or request labels Nov 11, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Important

Format specification vote

This PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer), at least one of them on the latest commit, and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process.

Approvals carry over across pushes, so a rebase or a typo fix does not send everyone back to re-vote. Whoever approves the latest commit is vouching that nothing substantive has changed since the earlier approvals; if something has, ask for fresh votes.

Status: ❌ Blocked — 0 of 3 required approvals

Approvals none (0/3)
Latest commit approved by none — one PMC member must approve the latest commit
Vetoes none
Voting period ends Mon 2026-11-16 14:03 UTC (06:03 PST)

Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the format-waived label to waive the vote for a trivial edit (typo, wording, formatting).

lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added K-changes Latest Gatekeeper recommendation requests changes. and removed K-changes Latest Gatekeeper recommendation requests changes. labels Nov 11, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

The field-identity and NULL-semantics findings are fixed. The contract now preserves child identity across renames and requires data evaluation for NOT when NULL rows cannot be determined exactly. I found no remaining blocker in this experimental format specification.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 11, 2026
@sarwarbhuiyan

Copy link
Copy Markdown
Contributor Author

Gatekeeper's two findings are fixed, and it now recommends approval.

This spec is marked experimental. Under the experimental rules, it can merge with a maintainer's review before its stabilization vote, as #9076 did. The design discussion is #10948. The stabilization PR, which removes the experimental markers, will carry the PMC vote.

The implementation, #10950, also has Gatekeeper's approval. I'll rebase it as soon as this merges.

@BubbleCal, could you review and merge this when you have a moment?

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. and removed K-approved Latest Gatekeeper recommendation permits acceptance. labels Nov 11, 2026
sarwarbhuiyan and others added 2 commits November 11, 2026 11:58
…ental)

Specify the persisted contract of the listelement scalar index, which
indexes fields inside the elements of a list<struct> column and answers
any_element filters: the ListElementIndexDetails message, the target
BTree's files over Binary keys, the Arrow row key encoding with a worked
example, and the queries a reader may answer from the index. Marked
experimental in the proto and the docs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…valued NULL rule

- Store the key fields' Lance field IDs (key_field_ids); keys is
  informational. A reader resolves each ID among the direct children of
  the indexed column's element struct, checks its logical type, and
  declines the index when a key field was dropped, moved or retyped.
- Define an element's predicate as UNKNOWN when no conjunct is FALSE and
  at least one is UNKNOWN, found over every non-empty null subset of the
  constrained keys. Residual conjuncts, NULL literals (including inside
  IN), null-sensitive tests and an exceeded enumeration limit leave the
  NULL set undetermined, and NOT must then evaluate the call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sarwarbhuiyan
sarwarbhuiyan force-pushed the docs/listelement-index-format branch from 7012c7d to c9d3adc Compare November 11, 2026 17:11
@BubbleCal BubbleCal added the format-waived Author attests no format vote is needed label Nov 12, 2026
@BubbleCal
BubbleCal merged commit d144dda into lance-format:main Nov 12, 2026
43 checks passed
BubbleCal pushed a commit that referenced this pull request Nov 12, 2026
… elements (#10950)

## Summary

Adds `listelement`, an experimental built-in scalar index over fields
inside the elements of a `list<struct>` column. It answers `any_element`
filters (#10901) from the index, with every condition on the same
element:

```python
dataset.create_scalar_index(
    "detections",
    IndexConfig(index_type="listelement", parameters={"keys": ["label", "confidence"]}),
)
dataset.to_table(filter="any_element(detections, 'label = ''car'' AND confidence > 0.9')")
```

This is the second part of #10900. The format is specified in #10949,
and the design discussion is #10948.

**Depends on #10949.** This PR carries the identical
`protos/index.proto` change and generated source so that it compiles.
Those changes drop out when this branch is rebased after #10949 merges.

## Design

- **Training:** one entry per element.
- The key is the element's `keys` fields as a tuple, encoded with
Arrow's row format (ascending, nulls first), so the bytes sort like the
tuples.
    - The posting is the row address.
- Entries are sorted with spilling, and an ordinary BTREE is trained
over the `Binary` keys. The index wraps the BTREE the way the JSON index
wraps its target.
- **Search:** `visit_scalar_function` claims `any_element(<indexed
column>, '<literal>')`. The predicate is planned exactly as
`any_element` plans it, then with the tuple planner
(`plan_tuple_predicate`) over the keys.
- **Seeks:** an `=`, `IN` or `IS NULL` prefix, then one range on the
next key, becomes byte-range seeks in the BTREE.
- **Superset and recheck:** conditions the keys can't express, such as a
field that isn't a key, make the answer a superset. The scan then
rechecks it with `any_element`.
- **No first key, no index:** without a condition on the first key, the
index declines.
- **NULLs:** NULL lists, and elements on which the predicate is NULL,
are tracked, so `NOT` is exact.
- **Maintenance:** update encodes the new fragments the same way, then
defers to the BTREE. Remap defers to the BTREE.
- **Row addresses:** in `lance-table`, the index joins the row-address
and eager-remap lists. On stable-row-ID tables, the #10352 writer fence
applies and compaction remaps the index.
- **`lance-datafusion` API:** it exports `ANY_ELEMENT`,
`any_element_predicate` and `plan_element_predicate`, so the index plans
the same expression the scan evaluates. `lance-index` now depends on
`lance-datafusion` as a normal dependency; there's no cycle.

## Fix included

On a stable-row-ID table, `optimize_indices` dropped old entries of a
row-address index outside fragment 0. The generic update arm in
`merge_scalar_indices` only treated the JSON index as storing row
addresses, so it filtered row addresses against row IDs. That arm now
also covers this index. JSON behavior is unchanged.

## Tests

- **Index unit tests** (13):
- same-element trap, every range operator, `IN`, label-only prefix,
recheck;
    - NULL rows for `NOT`;
    - shapes the index declines;
- details round trip, and invalid parameters (no keys, unknown key,
repeated key, nested key type, unknown parameter, non-struct list);
- segment key mismatch, an all-null column, and the spec's encoding
bytes.
- **`lance-table`:** row-address and eager-remap classification.
- **Dataset tests:** #10901's fixture is extended rather than
duplicated.
- Every filter case, plus the edge-row and invalid-predicate tests, runs
with and without the index, checking the plan and whether it rechecks.
- New: declining when the first key isn't constrained; `fast_search`
plus `optimize_indices`; delete plus compaction remap, with and without
stable row IDs.

## Docs

The read and write guide covers creating the index and which filters it
answers, and the scalar indexing guide has a row for it. Both mark it
experimental.

## Not in this PR

- **Training memory:** there's no explicit budget. Like the JSON index,
training uses the default spill pool. At 100M elements the prototype
needed `LANCE_MEM_POOL_SIZE=8GiB`.
- **Merging segments** for distributed builds.
- **Fragment reuse index:** batch row-ID remapping isn't supported. Like
JSON, the index is reported incompatible on stable-row-ID tables that
use a fragment reuse index.

Closes #10900.

This conflicts with #10947 in the dataset scanner tests and the guide.
Whichever merges second will rebase.

Lint: `cargo fmt --all`, and `cargo clippy --no-deps -p lance-datafusion
-p lance-index -p lance-table -p lance --tests --benches --features
lance-datafusion/substrait -- -D warnings`. With dependencies included,
clippy on arm64 currently fails in `lance-encoding` (`unreachable_code`
from #10905), which this PR doesn't touch.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-format On-disk format: protos and format spec docs A-index Vector index, linalg, tokenizer enhancement New feature or request format-change A change to the format spec, which requires a vote. Remove if minor (e.g. fixing typo). format-waived Author attests no format vote is needed K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants