Skip to content

docs(format): specify composite B-tree keys - #10358

Merged
Xuanwo merged 16 commits into
lance-format:mainfrom
Ali2Arslan:docs/composite-btree-keys
Nov 20, 2026
Merged

Xuanwo merged 16 commits into
lance-format:mainfrom
Ali2Arslan:docs/composite-btree-keys

Conversation

@Ali2Arslan

@Ali2Arslan Ali2Arslan commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Why

This PR is the format vote for the composite key track in #10250. It specifies how a BTree index stores encoded keys, for two or more columns or for one column with an explicit direction and null placement, so that later PRs can build the codec, training, and search against a voted contract.

What changed

  • BTreeIndexDetails gains an optional key_layout. It sets the key encoding and the key ordering, each a oneof with one message per kind, and gives each key its direction and null placement. The arrow_row message carries its version.
  • Index version 2 is a superset of version 1. A version 2 segment without key_layout has the version 1 layout, and a reader picks the page layout from whether key_layout is present, not from the number of fields.
  • The BTree spec gains an experimental Key Layout section. It covers one-key and composite layouts, the Binary page layout, the arrow_row version 1 bytes with 52 examples from the Arrow ordering fixtures, the lexicographic ordering, the reader rules, and the writer rules. A short compatibility section gives the reason for each fence: the index version fences released readers, and the flag fences the v12.0.0 and v13.0.0 writers that would otherwise extend fragment_bitmap.
  • The writer rules state as musts when a writer sets bit 22, that it drops a segment with a key_layout from a manifest without the bit, and how it maintains a segment it cannot use. Compaction may rewrite the fragments such a segment covers. The same commit removes them from its fragment_bitmap, as an update does, also when compaction defers the remap to a fragment reuse index, and readers scan them. By default, a writer builds a single-column index without a key_layout. One-key layouts stay allowed.
  • Writer feature flag bit 22, FLAG_BTREE_KEY_LAYOUT, is reserved and unsupported, and lance-tools names it btree_key_layout. Its doc comment says what the bit means and links to the writer rules. Bits 19, 20, and 21 went to FLAG_INLINE_DELETIONS, FLAG_LOCAL_FRAGMENT_ALIASES (feat(format)!: add local_fragment_aliases to IndexMetadata #10780), and FLAG_DELETION_FILE_PATHS (feat(table): store explicit deletion file path #10914) on main. The bit 22 fence re-check passed 173 checks: 104 table and chain checks matched the expected outcomes, and all 69 rows shared with the bit 20 runs matched cell for cell. It gives the table below.
  • main checks in generated protobuf code, so rust/lance-index/src/pb/lance.table.rs is regenerated for key_layout.
  • Encoding Stability now pins arrow_row version 1 to arrow-row 59.3.3, the version main builds with, and the arrow_ordering.rs fixtures pass on it.

Scope

  • Spec, proto, the regenerated protobuf code, and the reserved flag with its lance-tools name. No table writer emits a key_layout or derives bit 22 yet, and no BTree query path interprets key_layout. Generated protobuf code can decode the field, the feature-flag gate rejects bit 22 as unsupported, and lance-tools renders its name.
  • Bit 22 covers only segments with a key_layout. fix(index): fence older writers from row-address indices #10352 added FLAG_ROW_ADDR_INDICES at bit 18 for stable row ID tables with a BTree at index version 1 or above, but FLAG_ROW_ADDR_INDICES cannot replace FLAG_BTREE_KEY_LAYOUT: writers that support FLAG_ROW_ADDR_INDICES without key_layout support (such as 15.0.0b7) recompute bit 18 from index_version >= 1 on restore, clone, and commits without checking the source manifest's flags or dropping a damaged key_layout segment after a 10.0.0 or 11.0.0 restore clears bit 18 and a 12.0.0 or 13.0.0 update extends fragment_bitmap. Keeping FLAG_BTREE_KEY_LAYOUT as an independent bit that pre-key_layout writers cannot set ensures a damaged segment never becomes usable again.
  • Covering columns on an index with a key_layout are left to a later vote.

Blast Radius

Every BTree written today has empty details, so it decodes as before. No writer sets bit 22 yet. Index version 2 is free, because #10150 kept stable row ID BTrees on version 1 and #10152 does not change the index version.

Verification

  • A script implements arrow_row version 1 from the spec text alone. It reproduces all 113 fixtures in rust/lance/tests/arrow_ordering.rs and all 52 examples on the page. On 3,000 random tuples, byte order equals the lexicographic order. Three planted errors each fail it.
  • Fence probe. The head Rust API writes a table with a composite index at index version 2 and sets FLAG_BTREE_KEY_LAYOUT, once with stable row IDs and once without. Each pylance release runs each write first on its own copy, then queries a = 1 and b = 1.
  • A one-key key_layout meets the same fences, from reading 10.0.0 through 14.0.0. Every release drops a BTree segment above its maximum index version from the index list that reads and query planning use, before any code counts fields. The commits of 10.0.0 and 11.0.0 also use that filtered list, which is why they drop the index. The commits of 12.0.0 through 14.0.0 keep the segment, and an update counts its fields, but FLAG_BTREE_KEY_LAYOUT refuses those commits. Once a 10.0.0 or 11.0.0 restore clears the bit, the reader rule makes the segment unusable whatever its key count, even if a writer that supports FLAG_ROW_ADDR_INDICES (2^18) later sets bit 18 again.
  • cargo run --locked --manifest-path protos/codegen/Cargo.toml -- --check, cargo test -p lance-table --lib feature_flags, cargo test -p lance-tools format_feature_flags, cargo test -p lance --test arrow_ordering, cargo fmt --all -- --check, workspace clippy, ci/check_proto_comments.py, ci/check_module_size.py, and make build in docs/ pass, and every link and anchor on the changed pages resolves.

The index version fences readers. In 10.0.0 through 14.0.0, a = 1 and b = 1 return the same rows as a full scan after every write, and no plan uses the composite index. Bit 22 fences writers from 12.0.0 on. "Stable" and "plain" name the table kind where the two differ.

Release Insert Update, delete, create index, add column Restore, shallow clone, create branch optimize_indices() compact_files()
14.0.0 refused refused refused refused stable refused, plain commits nothing
13.0.0, 12.0.0 refused refused refused stable commits nothing, plain refused stable refused, plain commits nothing
11.0.0, 10.0.0 refused drops the index, clears bit 22 keeps the index, clears bit 22 stable commits nothing, plain drops the index and clears bit 22 drops the index, clears bit 22
head, stable only refused refused refused refused refused

After an 11.0.0 restore, shallow clone, or branch clears bit 22, a 13.0.0 update moves the composite index's fragment_bitmap from [0, 1, 2, 3] to [0, 1, 2, 3, 5], and a 15.0.0b7 restore sets bit 18 (FLAG_ROW_ADDR_INDICES) again while leaving bit 22 (FLAG_BTREE_KEY_LAYOUT) clear. That result is why an index with a key_layout from a manifest without FLAG_BTREE_KEY_LAYOUT is unusable.

Add BTreeKeyLayout to BTreeIndexDetails and a Composite Keys section to
the BTree spec. A composite index is keyed on two or more columns. Each
page key is the arrow_row version 1 encoding of the row's key tuple,
stored in a Binary column, so the existing page lookup compares bytes.
The index uses BTree index version 2, which released readers skip.

The section is marked experimental. It defines the descriptor, the page
layout, the bytes for each admitted key type with the arrow ordering
fixtures as examples, the lexicographic ordering, the reader rules, the
rule that keeps version 1 bytes fixed across Arrow upgrades, and what
releases v10.0.0 through v14.0.0 do with a composite index.

Reserve writer feature flag bit 18, FLAG_COMPOSITE_BTREE_KEYS. Without
it, a v12.0.0 or v13.0.0 update on a table with stable row ids adds a
fragment to a composite index's fragment bitmap that the index does not
cover. A writer that supports the bit sets it exactly when an index
segment has a key_layout. Restore, shallow clone, and branch creation
by v10.0.0 and v11.0.0 clear the bit and keep the index, so a reader
does not use a composite index from a manifest without the bit, and a
supporting writer never carries one out of such a manifest. This build
masks the bit out, as it does bit 17.

This is the format vote for composite keys. No code reads or writes
the new fields or the bit yet.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions github-actions Bot added A-format On-disk format: protos and format spec docs format-change A change to the format spec, which requires a vote. Remove if minor (e.g. fixing typo). documentation Improvements or additions to documentation labels Oct 27, 2026
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Important

Format specification vote

This PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer), at least one of them on the latest commit, and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process.

Approvals carry over across pushes, so a rebase or a typo fix does not send everyone back to re-vote. Whoever approves the latest commit is vouching that nothing substantive has changed since the earlier approvals; if something has, ask for fresh votes.

Status: ✅ Vote passed — 3 PMC approvals, voting period elapsed

Approvals @westonpace, @Xuanwo, @wjones127 (3/3)
Latest commit approved by @wjones127
Vetoes none
Voting period elapsed — ended Fri 2026-10-30 01:14 UTC (Thu 18:14 PDT)

Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the format-waived label to waive the vote for a trivial edit (typo, wording, formatting).

lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Oct 27, 2026
Co-authored-by: Cursor <cursoragent@cursor.com>
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Oct 28, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Oct 28, 2026

@westonpace westonpace left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some initial questions, nothing blocking. Probably my biggest question at the moment is why we need a feature flag.

Comment thread docs/src/format/index/scalar/btree.md Outdated
Comment on lines +86 to +91
A composite BTree index is keyed on two or more columns. It orders rows by the
tuple of their key values, compared one key at a time. Each page key is one
byte string that encodes the whole tuple, and byte order equals tuple order, so
the page lookup above finds pages by comparing encoded keys as bytes. Nulls are
part of the encoded tuple, so the null handling in
[Accelerated Queries](#accelerated-queries) does not apply to a composite index.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there any instances where the individual order is different than an individual's contribution to the composite order? In other words...

Assume we have an index on field X, which has values A and B, and A < B.

For every composite index on fields Y,X, where Y has a value C, can we assume that (C, A) < (C,B) and (A,C) < (A,B)?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there any instances where the individual order is different than an individual's contribution to the composite order?

Not if we fix the direction and null placement. But since each key sets these on its own, a descending key reverses its order and a nulls-last key moves its nulls to the end.

Comment thread docs/src/format/index/scalar/btree.md Outdated
Comment on lines +99 to +100
- `index_version` is 2. Only a composite index uses version 2. A single-column
index keeps version 0 or 1.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why can't a single-column index use version 2? It would be nice if each version is a superset of the previous version.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could have single-column indexes use version 2, but my thought was that we currently tie version 2/composite to a codec and the old ones stored raw values and didn't need to worry about mem-comparable ordering, I could try to change this though, if you want, and accommodate single columns as well.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could try to change this though, if you want, and accommodate single columns as well.

What is there to accommodate? I guess I'm thinking a reader will look at the number of fields declared on the index first. If there is one then it expects values to match the data type and makes a standard btree index. If there are more than one then it expects values to be a binary column and makes a composite btree index.

We don't need the version to tell us what kind of index we are dealing with do we?

Index versions are primarily used to mark compatibility breaks. If older readers will not be able to process an index created by newer writers then we need to bump the version so older readers know to ignore those indexes and don't attempt to open them.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm I guess I was thinking we were going to allow specifying (part of) the layout for single column btree's as well, w.r.t. direction and null placement, but I guess we should still do that and just give those single column btree's version 2 as well. looking at the number of fields feels brittle tho, maybe we use the key_layout presence instead?

Comment thread docs/src/format/index/scalar/btree.md Outdated
Comment on lines +103 to +104
A composite index declares no covering fields, so `fields` holds only key
columns under either [covering-field contract](../index.md#covering-field-contracts).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this unique to composite btree indexes? Or was it already a property of regular btree indexes? Why do we need this?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, single-column B-trees don't carry columns either, it's mostly here because under the legacy covering contract, carried columns are the trailing entries of fields. This pins every entry as a key under either contract. I can reword to make that clearer

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. I think there might be cases where it makes sense to cover columns in a btree (single or composite) index and so we want to leave the door open for that.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, definitely agree

Comment thread docs/src/format/index/scalar/btree.md Outdated
Comment on lines +108 to +109
`key_layout` does not repeat it. Every key stores both `descending` and
`nulls_first`, and the format has no default. The protobuf zero values mean

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every key stores both descending and
nulls_first, and the format has no default.

What does this sentence mean?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, that's AI slop :(, mostly means that each is an explicit decision now, no assumed defaults, I'll reword it :)

Comment thread protos/index_old.proto Outdated
message KeyOrdering {
// The ordering name, such as "lexicographic".
string name = 1;
uint32 version = 2;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why freeform string names and not an enum?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just decouples the encodings for the proto, but we can type it in the proto if you would like

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we are defining the rules for each ordering as part of the spec I think it is ok to be part of the proto. Probably the most future proof thing would be...

message Lexicographic {}
message KeyOrdering {
  oneof ordering {
    Lexicographic lexicographic = 0
  }
}

This approach would even leave the door open for an ordering to require its own parameterization. However, I can't think of an example where that might happen so 🤷 maybe I'm just being paranoid.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like your example better, I'll update it to use that :)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

though I still think have a version associated with the type of ordering would be good, in-case we ever have to change codecs for that logical ordering?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

err ignore that, the encoding variants already carry the version

Comment on lines +134 to +135
Rows sort by `values` compared as unsigned byte strings, and rows with equal
`values` sort by `ids` in ascending order. Two byte strings compare at their

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the id secondary sort to keep things stable. Do we have a similar concept in the single-column btree index? we probably should.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

they don't currently do this, though I do plan to support it, up to you whether the format vote for that should be coupled? maybe adding a field to the proto indicating whether training provides this (new) guarantee now?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to couple it together. I was mostly just pondering out loud.

Comment thread docs/src/format/index/scalar/btree.md Outdated
Comment on lines +150 to +151
`min` and `max` are the first and the last `values` of the page. `null_count`
is always 0, because `values` is never null.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If null_count is always 0 then why have the column? Just for consistency with non-composite indexes?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yup mostly to keep the same page_lookup schema the same

Comment thread docs/src/format/index/scalar/btree.md Outdated
Comment on lines +136 to +137
first differing byte. A byte string sorts before every longer byte string that
starts with it.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does this last sentence mean? Are you saying 0xFF will always come before 0x0000?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's for the prefix case, where one encoded byte string is a prefix of another, not that shorter ones sort first

Comment thread docs/src/format/index/scalar/btree.md Outdated
Comment on lines +160 to +163
The encoding admits these key types: Boolean, Int8, Int16, Int32, Int64,
UInt8, UInt16, UInt32, UInt64, Float32, Float64, Date32, Date64, Timestamp with
any unit and with or without a time zone, Utf8, LargeUtf8, Binary, and
LargeBinary.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What types are we not including (structs? lists? binary?) and why?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

binary is fine, but yeah complex types were excluded in our initial implementation, which has support for multiple codecs, and our codec did not have encoding for those, but arrow's does, and if we will always use arrow's then no reason to exclude them :)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, it's fine to leave them out for now.

Comment thread docs/src/format/index/scalar/btree.md Outdated

The encoding of a key depends on its type, on `descending`, and on
`nulls_first`. The null sentinel of a key is the byte `0x00` when `nulls_first`
is true and the byte `0xFF` when it is false.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will 0xFF always be last if other value encode into multiple bytes?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, nulls sorted before or after every non-null value, an artifact of arrow's codec

@Ali2Arslan

Copy link
Copy Markdown
Contributor Author

Some initial questions, nothing blocking. Probably my biggest question at the moment is why we need a feature flag.

Mostly because of this bug :( #10351

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. and removed K-approved Latest Gatekeeper recommendation permits acceptance. labels Oct 28, 2026
Address review of the composite B-tree key spec.

- Index version 2 is a superset of version 1. A version 2 segment without
  key_layout has the version 1 layout, and a reader picks the page layout
  from whether key_layout is present, not from the number of fields.
- A key_layout may have one key. That gives a single-column index with an
  explicit direction and null placement, stored in the same encoded pages
  as a composite index.
- Rename FLAG_COMPOSITE_BTREE_KEYS to FLAG_BTREE_KEY_LAYOUT, because it
  also fences one-key layouts, and open its docs with why writers need it.
- KeyEncoding and KeyOrdering become oneofs of one message per kind.
  arrow_row keeps its version as a field, and lexicographic has none.
- This version defines no covering columns for a key_layout index, and a
  later format change may add them.
- Say that each key keeps its own order inside the tuple order, how
  differing bytes and prefixes compare, why null_count stays, which types
  arrow_row leaves out, and why a null sorts before or after every value.

Co-authored-by: Cursor <cursoragent@cursor.com>
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Oct 29, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. and removed K-approved Latest Gatekeeper recommendation permits acceptance. labels Oct 29, 2026
Ali2Arslan and others added 2 commits October 29, 2026 13:18
Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	rust/lance-table/src/feature_flags.rs
lance-format#10352 added writer feature flag bit 18 as FLAG_ROW_ADDR_INDICES for
stable-row-ID tables that carry row-address indices, including any
BTree segment at version 1 or above.

- Drop the separate FLAG_BTREE_KEY_LAYOUT constant and document
  FLAG_ROW_ADDR_INDICES in the feature flags table.
- Scope the BTree key_layout reader and writer flag rules to manifests
  that set FLAG_STABLE_ROW_IDS, where v12.0.0 and v13.0.0 updates
  extend fragment_bitmap to rewritten fragments unless fenced.
- Document how v10.0.0 through v14.0.0 handle a key_layout segment on
  tables with and without FLAG_STABLE_ROW_IDS.

Co-authored-by: Cursor <cursoragent@cursor.com>
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Oct 29, 2026
@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Oct 30, 2026
Ali2Arslan and others added 2 commits November 2, 2026 08:44
Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	docs/src/format/table/versioning.md
#	rust/lance-table/src/feature_flags.rs
Main gave writer flag bit 19 to FLAG_INLINE_DELETIONS. The merge moved
FLAG_BTREE_KEY_LAYOUT to bit 20 in feature_flags.rs and versioning.md.
This commit updates the B-tree spec's compatibility section to match.

Co-authored-by: Cursor <cursoragent@cursor.com>
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 2, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

No blocking issues found after moving the independent key-layout fence to bit 20. The experimental format contract can land ahead of runtime support while that bit remains unsupported.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 2, 2026

@westonpace westonpace left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After some external discussions on the need for the feature flag (fixing bugs in older btree readers where index version was not properly respected) I have been convinced. Thanks!

BubbleCal pushed a commit that referenced this pull request Nov 12, 2026
…ental) (#10949)

Add the persisted contract of the `listelement` scalar index, which
indexes fields inside the elements of a `list<struct>` column and
answers `any_element` filters (#10901). It is marked experimental.
Design discussion: #10948. Issue: #10900.

- **Details:** `ListElementIndexDetails` has these fields:
- `keys`: direct children of the element struct, at least one, no
repeats.
- `key_types`: Lance logical types. A reader declines the index when a
key field is gone or its type changed.
- `key_encoding`: an enum. `ARROW_ROW_V1` is the only defined value, and
UNSPECIFIED must not be used.
- `target_details` and `target_index_version`: the details and version
of the inner BTREE.

  All segments of one index share keys, types and encoding.
- **Files:** none of its own. The files are a standard BTREE over
`Binary`, given as PyArrow schemas, with one row per list element:
  - A null element is an all-null key tuple.
  - A null list is a single null key.
  - An empty list has no rows.

  The posting is the row address.
- **Key encoding:** Arrow row format, every key ascending, nulls first.
The admitted key types are the same as the `arrow_row` v1 set in #10358.
The byte layout is spelled out for fixed-width and variable-width types,
with a worked example, `("car", 0.5)`, that an implementation must
reproduce.
- **Queries:**
- **What the index can answer:** `=`, `IN` or `IS NULL` on the first
key, then equalities on further keys in order, then one range on the
next key.
- **Exact or rechecked:** the answer is exact when the index covers
every condition. Otherwise it's a superset that the scan rechecks with
`any_element`.
  - **NULL rows:** they are found for `NOT`.
- **Row addresses:** the index stores row addresses, so it joins the
row-address and eager-remap lists. On stable-row-ID tables the #10352
writer fence applies.

Library edits are limited to the generated protobuf source. The
implementation follows in a separate PR.

**Experimental:**
- **Forward compatible:** readers that don't know the type ignore the
index and scan.
- **Backward compatible:** nothing existing changes.
- **Droppable:** dropping the index never rewrites data.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@Ali2Arslan

Ali2Arslan commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

@wjones127 or @BubbleCal could you cast the third PMC vote on this format change when you have time? @westonpace and @Xuanwo have approved, the voting period has ended, and it still merges cleanly with main. #10459 also claims bit 20, so whichever of the two lands second will move to bit 21.

@wjones127 wjones127 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, and for the careful compatibility testing against released versions. Nothing here blocks, so I'm approving as the third vote. I would like answers to the questions below, though, and there's some cleanup that could happen here or in a follow-up:

  • Whether a single key needs a key_layout at all, given the raw layout can already be read in either direction (#11124).
  • Why unusable segments block compaction instead of shrinking fragment_bitmap.
  • Making the compatibility section state rules rather than release history, and trimming the versioning table row to match.

I also filed #11238 so future index format changes don't each need their own writer flag.


A BTree index with a `key_layout` has one or more keys and stores them in the
[encoded page layout](#encoded-page-layout). With two or more keys it is a
composite index. With one key it is a single-column index whose direction and

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question(non-blocking): What does a one-key key_layout buy over the existing raw-value layout? With one key, every direction and null placement is the reverse of another, and #11124 already serves ORDER BY ts DESC NULLS LAST from a raw single-column B-tree. Per-key direction seems to matter only for composite keys with mixed directions, e.g. a ASC, b DESC.

Unless there's a case I'm missing, I wonder if we should require at least two keys for now, so there's only one way to write a single-column B-tree. At minimum, I think writers should default to the existing layout for a single key.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Honestly, the only thing I can think of, for having a one-key key_layout would be if we wanted to support different collations, which affect both equality and sorting, even on single-keys.
I would like to keep it even for single-keys as we technically also lose nothing if we support it, and if it's missing it will use the raw-value layout anyways? what do you think?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that seems acceptable. I don't have a strong opinion.

Comment thread docs/src/format/index/scalar/btree.md Resolved
tuples are equal exactly when their encodings are equal. A reader therefore
orders and searches pages by comparing bytes, and never decodes a key to do so.

### Encoding Examples

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

praise: Love the worked byte examples here. Combined with your from-the-spec reimplementation that reproduces the ordering fixtures, this makes "the page, not the crate, defines version 1" a real promise.

Comment thread docs/src/format/index/scalar/btree.md Outdated
`FLAG_BTREE_KEY_LAYOUT`, a writer that supports key layouts drops the
index (see
[Compatibility With Released Versions](#compatibility-with-released-versions)).
Otherwise the engine does not add rows or fragments to the index, and its

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question(non-blocking): Why does an unusable segment stop compaction from rewriting the fragments it covers, rather than letting compaction drop those fragments from fragment_bitmap the way update does?

Take a writer that understands arrow_row version 1 sharing a table with a newer writer's version 2 segment. The older writer would leave those fragments uncompacted indefinitely. Shrinking coverage keeps results correct (the reader just scans those fragments), so I'd lean toward that unless there's a reason I'm missing.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed as suggested. A writer that supports key layouts may now compact the fragments that an unusable key layout segment covers. The commit that rewrites them removes them from the segment's fragment_bitmap, as an update does, and readers scan the new fragments. The same commit removes them when compaction defers the remap to a fragment reuse index, so the segment covers none of the fragments that the reuse index maps them to. The rule is in the new Key Layout Writer Rules section. Released versions keep their behavior, because v12.0.0 through v14.0.0 refuse the flag and the compaction of v10.0.0 and v11.0.0 drops the index.

Comment thread docs/src/format/index/scalar/btree.md Outdated
whose writer feature flags lack `FLAG_BTREE_KEY_LAYOUT` (see
[Key Layout Reader Rules](#key-layout-reader-rules)).

A writer that supports key layouts never carries an index with a `key_layout`

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: This paragraph is the rule writers have to follow, but it comes after ~25 lines on what v10.0.0 through v14.0.0 do (L382-L404). Could we move it next to Key Layout Reader Rules as "Key Layout Writer Rules", written as musts?

Then we could trim the release-by-release narrative to a sentence or two of rationale. The same narrative is repeated in the FLAG_BTREE_KEY_LAYOUT doc comment and in the versioning table. The spec should say what implementations must do, not what past releases did.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. A new Key Layout Writer Rules section follows the reader rules and states as musts when a writer sets FLAG_BTREE_KEY_LAYOUT, that it drops a key layout segment from a manifest without the bit, and how it maintains a segment it cannot use. Compatibility With Released Versions now gives only the reason for each fence. The FLAG_BTREE_KEY_LAYOUT doc comment now says what the bit means and links to the writer rules.

Comment thread docs/src/format/table/versioning.md Outdated
| `2^17` | `FLAG_OBJECT_METADATA` | No | Yes | The snapshot contains [clustering definitions, references, or a nonzero allocation high-water mark](clustering.md), nonempty [fragment metadata](index.md#fragment-metadata), or nonempty [index segment metadata](../index/index.md#index-segment-metadata). Reserved, unsupported until writers implement the maintenance rules. Readers may ignore this metadata. |
| `2^18` | `FLAG_ROW_ADDR_INDICES` | No | Yes | A stable-row-ID table contains an index that stores physical row addresses. Older writers could treat its entries as stable row IDs and silently extend coverage without indexing rewritten fragments. |
| `2^19` | `FLAG_INLINE_DELETIONS` | Yes | Yes | Some fragment stores its deletion vector in the manifest (`DeletionFile.inline`) rather than in a file under `_deletions/`. A reader without this flag would look for a file that does not exist. |
| `2^20` | `FLAG_BTREE_KEY_LAYOUT` | No | Yes | Some BTree index segment has a [key layout](../index/scalar/btree.md#key-layout) (`BTreeIndexDetails.key_layout`). Without the bit, an update by v12.0.0 or v13.0.0 on a table with stable row IDs adds fragments to the segment's coverage without indexing their rows, and a later writer that supports `FLAG_ROW_ADDR_INDICES` (`2^18`) but not key layouts can re-set `FLAG_ROW_ADDR_INDICES` without repairing the segment. A writer that supports this bit sets it on every published manifest, including by clone and restore, exactly when such a segment exists, and never carries such a segment out of a manifest that lacks it. Readers do not use such a segment from a manifest that lacks the bit. Reserved, unsupported until key layouts are implemented. |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question(non-blocking): Why do we need a table-level flag for an index feature? I'd expect the index version to be enough. As I read the compatibility section, released writers update fragment_bitmap without checking whether they support the index version, so this is the only fence for v12/v13. That makes sense for this PR.

I filed #11238 so future index changes don't each need their own writer flag.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Released writers update fragment_bitmap without checking the index version, so for v12.0.0 and v13.0.0 the flag is the only writer fence. Thanks for filing #11238.

Comment thread docs/src/format/table/versioning.md Resolved
Ali2Arslan and others added 2 commits November 19, 2026 05:50
Co-authored-by: Will Jones <willjones127@gmail.com>
Co-authored-by: Will Jones <willjones127@gmail.com>
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 19, 2026
Ali2Arslan and others added 3 commits November 19, 2026 05:55
Main checks in the generated protobuf sources, so regenerate
rust/lance-index/src/pb/lance.table.rs for BTreeIndexDetails.key_layout
and its messages.

Name FLAG_BTREE_KEY_LAYOUT "btree_key_layout" in the lance-tools feature
flag table and in the test that lists every known flag.

Co-authored-by: Cursor <cursoragent@cursor.com>
…overage

Collect the writer rules for BTree key layouts in a Key Layout Writer Rules
section next to the reader rules: when to set FLAG_BTREE_KEY_LAYOUT, dropping
a key layout segment carried from a manifest without the bit, and maintaining
a segment the writer cannot use.

A writer that supports key layouts may now compact the fragments that an
unusable key layout segment covers. The compacting commit removes them from
the segment's fragment_bitmap, as an update does, including when compaction
defers the remap to a fragment reuse index. Readers then scan the new
fragments.

By default, a writer builds a single-column index without a key_layout.

Cut Compatibility With Released Versions to the rationale for the two
fences, and trim the FLAG_BTREE_KEY_LAYOUT doc comment to what the bit means
and a link to the writer rules.

Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions github-actions Bot added the A-index Vector index, linalg, tokenizer label Nov 19, 2026

@wjones127 wjones127 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for turning these around. The new Key Layout Writer Rules section reads as rules now, compaction shrinking fragment_bitmap instead of being blocked is what I was hoping for, and the compatibility section and versioning row are much tighter. Re-approving on the latest commit so the vote gate clears.

One optional note inline: the Encoding Stability section still pins version 1 to arrow-row 58.4.0, but the workspace is on 59.2.0.

FYI, #10459 also claims bit 20. Whichever lands second needs to move to bit 21, including the FLAG_UNKNOWN bump and the lance-tools name.

Comment thread docs/src/format/index/scalar/btree.md Outdated
### Encoding Stability

The bytes of `arrow_row` version 1 are the bytes that the `RowConverter` of the
Arrow Rust crate `arrow-row` 58.4.0 writes for the admitted types. Arrow does

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: Update the pinned arrow-row version to the one we build with.

The workspace is on arrow-row 59.2.0 now (Cargo.toml), so this pins version 1 to a crate version we no longer build with. Since the page defines the bytes, the version here is just provenance, but I'd update it (assuming the arrow_ordering.rs fixtures still pass on 59.2.0) so it doesn't look like the upgrade skipped a check.

Suggested change
Arrow Rust crate `arrow-row` 58.4.0 writes for the admitted types. Arrow does
Arrow Rust crate `arrow-row` 59.2.0 writes for the admitted types. Arrow does

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated to 59.3.3 rather than 59.2.0. The 59.2.0 in Cargo.toml is the minimum it accepts. After merging main, Cargo.lock resolves 59.3.3, so that is the crate the build links and the one the fixtures ran on. cargo test -p lance --test arrow_ordering passes on 59.3.3, all 46 tests, including the 29 row_converter_bytes_match_baseline cases. I did not run them on 59.2.0.

Ali2Arslan and others added 2 commits November 19, 2026 15:50
Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	docs/src/format/table/versioning.md
#	rust/lance-table/src/feature_flags.rs
#	rust/lance-tools/src/manifest.rs
….3.3

Main took bits 20 and 21 for FLAG_LOCAL_FRAGMENT_ALIASES and
FLAG_DELETION_FILE_PATHS, so the writer rules name FLAG_BTREE_KEY_LAYOUT
as 2^22. Encoding Stability now names arrow-row 59.3.3, the version
Cargo.lock resolves; the arrow_ordering fixtures pass on it.

Co-authored-by: Cursor <cursoragent@cursor.com>
@Ali2Arslan

Ali2Arslan commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor Author

@wjones127 @westonpace @Xuanwo thanks, the vote cleared on a6d61b2. Since then main took bits 20 and 21 (#10780 and #10914), so this PR conflicted. I merged main and pushed aca8f04:

  • FLAG_BTREE_KEY_LAYOUT moves to bit 22 and FLAG_UNKNOWN to bit 23. lance-tools keeps the btree_key_layout name, and the protobuf code is regenerated for main's proto changes.
  • Encoding Stability pins arrow-row 59.3.3, per the inline note. The arrow_ordering.rs fixtures pass on it.
  • No rule changed. The bit 22 fence re-check passed 173 checks; all 69 rows shared with the bit 20 runs matched cell for cell.

The vote gate needs one approval on the new head. Could one of you re-approve, and could a maintainer merge once CI is green? Merging soon would avoid another bit collision, since #10459 also needs a new bit.

The Java jobs fail while Maven resolves spotless's dependencies (google-java-format 1.22.0 and others reported absent), the same failure as main's latest Java run (https://github.com/lance-format/lance/actions/runs/44633319178). This PR changes no Java code.

The Java jobs on aca8f04 failed to resolve maven-jar-plugin 3.0.2 from Maven Central. No content change.

Co-authored-by: Cursor <cursoragent@cursor.com>
@Xuanwo
Xuanwo merged commit 179c5cd into lance-format:main Nov 20, 2026
48 of 52 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-format On-disk format: protos and format spec docs A-index Vector index, linalg, tokenizer documentation Improvements or additions to documentation format-change A change to the format spec, which requires a vote. Remove if minor (e.g. fixing typo).

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants