Repository navigation
docs(format): specify composite B-tree keys - #10358
Conversation
Add BTreeKeyLayout to BTreeIndexDetails and a Composite Keys section to the BTree spec. A composite index is keyed on two or more columns. Each page key is the arrow_row version 1 encoding of the row's key tuple, stored in a Binary column, so the existing page lookup compares bytes. The index uses BTree index version 2, which released readers skip. The section is marked experimental. It defines the descriptor, the page layout, the bytes for each admitted key type with the arrow ordering fixtures as examples, the lexicographic ordering, the reader rules, the rule that keeps version 1 bytes fixed across Arrow upgrades, and what releases v10.0.0 through v14.0.0 do with a composite index. Reserve writer feature flag bit 18, FLAG_COMPOSITE_BTREE_KEYS. Without it, a v12.0.0 or v13.0.0 update on a table with stable row ids adds a fragment to a composite index's fragment bitmap that the index does not cover. A writer that supports the bit sets it exactly when an index segment has a key_layout. Restore, shallow clone, and branch creation by v10.0.0 and v11.0.0 clear the bit and keep the index, so a reader does not use a composite index from a manifest without the bit, and a supporting writer never carries one out of such a manifest. This build masks the bit out, as it does bit 17. This is the format vote for composite keys. No code reads or writes the new fields or the bit yet.
|
Important Format specification voteThis PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer), at least one of them on the latest commit, and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process. Approvals carry over across pushes, so a rebase or a typo fix does not send everyone back to re-vote. Whoever approves the latest commit is vouching that nothing substantive has changed since the earlier approvals; if something has, ask for fresh votes. Status: ✅ Vote passed — 3 PMC approvals, voting period elapsed
Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the |
Co-authored-by: Cursor <cursoragent@cursor.com>
westonpace
left a comment
There was a problem hiding this comment.
Some initial questions, nothing blocking. Probably my biggest question at the moment is why we need a feature flag.
| A composite BTree index is keyed on two or more columns. It orders rows by the | ||
| tuple of their key values, compared one key at a time. Each page key is one | ||
| byte string that encodes the whole tuple, and byte order equals tuple order, so | ||
| the page lookup above finds pages by comparing encoded keys as bytes. Nulls are | ||
| part of the encoded tuple, so the null handling in | ||
| [Accelerated Queries](#accelerated-queries) does not apply to a composite index. |
There was a problem hiding this comment.
Are there any instances where the individual order is different than an individual's contribution to the composite order? In other words...
Assume we have an index on field X, which has values A and B, and A < B.
For every composite index on fields Y,X, where Y has a value C, can we assume that (C, A) < (C,B) and (A,C) < (A,B)?
There was a problem hiding this comment.
Are there any instances where the individual order is different than an individual's contribution to the composite order?
Not if we fix the direction and null placement. But since each key sets these on its own, a descending key reverses its order and a nulls-last key moves its nulls to the end.
| - `index_version` is 2. Only a composite index uses version 2. A single-column | ||
| index keeps version 0 or 1. |
There was a problem hiding this comment.
Why can't a single-column index use version 2? It would be nice if each version is a superset of the previous version.
There was a problem hiding this comment.
We could have single-column indexes use version 2, but my thought was that we currently tie version 2/composite to a codec and the old ones stored raw values and didn't need to worry about mem-comparable ordering, I could try to change this though, if you want, and accommodate single columns as well.
There was a problem hiding this comment.
I could try to change this though, if you want, and accommodate single columns as well.
What is there to accommodate? I guess I'm thinking a reader will look at the number of fields declared on the index first. If there is one then it expects values to match the data type and makes a standard btree index. If there are more than one then it expects values to be a binary column and makes a composite btree index.
We don't need the version to tell us what kind of index we are dealing with do we?
Index versions are primarily used to mark compatibility breaks. If older readers will not be able to process an index created by newer writers then we need to bump the version so older readers know to ignore those indexes and don't attempt to open them.
There was a problem hiding this comment.
hmm I guess I was thinking we were going to allow specifying (part of) the layout for single column btree's as well, w.r.t. direction and null placement, but I guess we should still do that and just give those single column btree's version 2 as well. looking at the number of fields feels brittle tho, maybe we use the key_layout presence instead?
| A composite index declares no covering fields, so `fields` holds only key | ||
| columns under either [covering-field contract](../index.md#covering-field-contracts). |
There was a problem hiding this comment.
Is this unique to composite btree indexes? Or was it already a property of regular btree indexes? Why do we need this?
There was a problem hiding this comment.
no, single-column B-trees don't carry columns either, it's mostly here because under the legacy covering contract, carried columns are the trailing entries of fields. This pins every entry as a key under either contract. I can reword to make that clearer
There was a problem hiding this comment.
Ok. I think there might be cases where it makes sense to cover columns in a btree (single or composite) index and so we want to leave the door open for that.
There was a problem hiding this comment.
yeah, definitely agree
| `key_layout` does not repeat it. Every key stores both `descending` and | ||
| `nulls_first`, and the format has no default. The protobuf zero values mean |
There was a problem hiding this comment.
Every key stores both
descendingand
nulls_first, and the format has no default.
What does this sentence mean?
There was a problem hiding this comment.
yeah, that's AI slop :(, mostly means that each is an explicit decision now, no assumed defaults, I'll reword it :)
| message KeyOrdering { | ||
| // The ordering name, such as "lexicographic". | ||
| string name = 1; | ||
| uint32 version = 2; |
There was a problem hiding this comment.
Why freeform string names and not an enum?
There was a problem hiding this comment.
just decouples the encodings for the proto, but we can type it in the proto if you would like
There was a problem hiding this comment.
Since we are defining the rules for each ordering as part of the spec I think it is ok to be part of the proto. Probably the most future proof thing would be...
message Lexicographic {}
message KeyOrdering {
oneof ordering {
Lexicographic lexicographic = 0
}
}
This approach would even leave the door open for an ordering to require its own parameterization. However, I can't think of an example where that might happen so 🤷 maybe I'm just being paranoid.
There was a problem hiding this comment.
I like your example better, I'll update it to use that :)
There was a problem hiding this comment.
though I still think have a version associated with the type of ordering would be good, in-case we ever have to change codecs for that logical ordering?
There was a problem hiding this comment.
err ignore that, the encoding variants already carry the version
| Rows sort by `values` compared as unsigned byte strings, and rows with equal | ||
| `values` sort by `ids` in ascending order. Two byte strings compare at their |
There was a problem hiding this comment.
I like the id secondary sort to keep things stable. Do we have a similar concept in the single-column btree index? we probably should.
There was a problem hiding this comment.
they don't currently do this, though I do plan to support it, up to you whether the format vote for that should be coupled? maybe adding a field to the proto indicating whether training provides this (new) guarantee now?
There was a problem hiding this comment.
No need to couple it together. I was mostly just pondering out loud.
| `min` and `max` are the first and the last `values` of the page. `null_count` | ||
| is always 0, because `values` is never null. |
There was a problem hiding this comment.
If null_count is always 0 then why have the column? Just for consistency with non-composite indexes?
There was a problem hiding this comment.
yup mostly to keep the same page_lookup schema the same
| first differing byte. A byte string sorts before every longer byte string that | ||
| starts with it. |
There was a problem hiding this comment.
What does this last sentence mean? Are you saying 0xFF will always come before 0x0000?
There was a problem hiding this comment.
it's for the prefix case, where one encoded byte string is a prefix of another, not that shorter ones sort first
| The encoding admits these key types: Boolean, Int8, Int16, Int32, Int64, | ||
| UInt8, UInt16, UInt32, UInt64, Float32, Float64, Date32, Date64, Timestamp with | ||
| any unit and with or without a time zone, Utf8, LargeUtf8, Binary, and | ||
| LargeBinary. |
There was a problem hiding this comment.
What types are we not including (structs? lists? binary?) and why?
There was a problem hiding this comment.
binary is fine, but yeah complex types were excluded in our initial implementation, which has support for multiple codecs, and our codec did not have encoding for those, but arrow's does, and if we will always use arrow's then no reason to exclude them :)
There was a problem hiding this comment.
Ok, it's fine to leave them out for now.
|
|
||
| The encoding of a key depends on its type, on `descending`, and on | ||
| `nulls_first`. The null sentinel of a key is the byte `0x00` when `nulls_first` | ||
| is true and the byte `0xFF` when it is false. |
There was a problem hiding this comment.
Will 0xFF always be last if other value encode into multiple bytes?
There was a problem hiding this comment.
yes, nulls sorted before or after every non-null value, an artifact of arrow's codec
Mostly because of this bug :( #10351 |
Address review of the composite B-tree key spec. - Index version 2 is a superset of version 1. A version 2 segment without key_layout has the version 1 layout, and a reader picks the page layout from whether key_layout is present, not from the number of fields. - A key_layout may have one key. That gives a single-column index with an explicit direction and null placement, stored in the same encoded pages as a composite index. - Rename FLAG_COMPOSITE_BTREE_KEYS to FLAG_BTREE_KEY_LAYOUT, because it also fences one-key layouts, and open its docs with why writers need it. - KeyEncoding and KeyOrdering become oneofs of one message per kind. arrow_row keeps its version as a field, and lexicographic has none. - This version defines no covering columns for a key_layout index, and a later format change may add them. - Say that each key keeps its own order inside the tuple order, how differing bytes and prefixes compare, why null_count stays, which types arrow_row leaves out, and why a null sorts before or after every value. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # rust/lance-table/src/feature_flags.rs
lance-format#10352 added writer feature flag bit 18 as FLAG_ROW_ADDR_INDICES for stable-row-ID tables that carry row-address indices, including any BTree segment at version 1 or above. - Drop the separate FLAG_BTREE_KEY_LAYOUT constant and document FLAG_ROW_ADDR_INDICES in the feature flags table. - Scope the BTree key_layout reader and writer flag rules to manifests that set FLAG_STABLE_ROW_IDS, where v12.0.0 and v13.0.0 updates extend fragment_bitmap to rewritten fragments unless fenced. - Document how v10.0.0 through v14.0.0 handle a key_layout segment on tables with and without FLAG_STABLE_ROW_IDS. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # docs/src/format/table/versioning.md # rust/lance-table/src/feature_flags.rs
Main gave writer flag bit 19 to FLAG_INLINE_DELETIONS. The merge moved FLAG_BTREE_KEY_LAYOUT to bit 20 in feature_flags.rs and versioning.md. This commit updates the B-tree spec's compatibility section to match. Co-authored-by: Cursor <cursoragent@cursor.com>
westonpace
left a comment
There was a problem hiding this comment.
After some external discussions on the need for the feature flag (fixing bugs in older btree readers where index version was not properly respected) I have been convinced. Thanks!
…ental) (#10949) Add the persisted contract of the `listelement` scalar index, which indexes fields inside the elements of a `list<struct>` column and answers `any_element` filters (#10901). It is marked experimental. Design discussion: #10948. Issue: #10900. - **Details:** `ListElementIndexDetails` has these fields: - `keys`: direct children of the element struct, at least one, no repeats. - `key_types`: Lance logical types. A reader declines the index when a key field is gone or its type changed. - `key_encoding`: an enum. `ARROW_ROW_V1` is the only defined value, and UNSPECIFIED must not be used. - `target_details` and `target_index_version`: the details and version of the inner BTREE. All segments of one index share keys, types and encoding. - **Files:** none of its own. The files are a standard BTREE over `Binary`, given as PyArrow schemas, with one row per list element: - A null element is an all-null key tuple. - A null list is a single null key. - An empty list has no rows. The posting is the row address. - **Key encoding:** Arrow row format, every key ascending, nulls first. The admitted key types are the same as the `arrow_row` v1 set in #10358. The byte layout is spelled out for fixed-width and variable-width types, with a worked example, `("car", 0.5)`, that an implementation must reproduce. - **Queries:** - **What the index can answer:** `=`, `IN` or `IS NULL` on the first key, then equalities on further keys in order, then one range on the next key. - **Exact or rechecked:** the answer is exact when the index covers every condition. Otherwise it's a superset that the scan rechecks with `any_element`. - **NULL rows:** they are found for `NOT`. - **Row addresses:** the index stores row addresses, so it joins the row-address and eager-remap lists. On stable-row-ID tables the #10352 writer fence applies. Library edits are limited to the generated protobuf source. The implementation follows in a separate PR. **Experimental:** - **Forward compatible:** readers that don't know the type ignore the index and scan. - **Backward compatible:** nothing existing changes. - **Droppable:** dropping the index never rewrites data. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
@wjones127 or @BubbleCal could you cast the third PMC vote on this format change when you have time? @westonpace and @Xuanwo have approved, the voting period has ended, and it still merges cleanly with main. #10459 also claims bit 20, so whichever of the two lands second will move to bit 21. |
wjones127
left a comment
There was a problem hiding this comment.
Thanks for this, and for the careful compatibility testing against released versions. Nothing here blocks, so I'm approving as the third vote. I would like answers to the questions below, though, and there's some cleanup that could happen here or in a follow-up:
- Whether a single key needs a
key_layoutat all, given the raw layout can already be read in either direction (#11124). - Why unusable segments block compaction instead of shrinking
fragment_bitmap. - Making the compatibility section state rules rather than release history, and trimming the versioning table row to match.
I also filed #11238 so future index format changes don't each need their own writer flag.
|
|
||
| A BTree index with a `key_layout` has one or more keys and stores them in the | ||
| [encoded page layout](#encoded-page-layout). With two or more keys it is a | ||
| composite index. With one key it is a single-column index whose direction and |
There was a problem hiding this comment.
question(non-blocking): What does a one-key key_layout buy over the existing raw-value layout? With one key, every direction and null placement is the reverse of another, and #11124 already serves ORDER BY ts DESC NULLS LAST from a raw single-column B-tree. Per-key direction seems to matter only for composite keys with mixed directions, e.g. a ASC, b DESC.
Unless there's a case I'm missing, I wonder if we should require at least two keys for now, so there's only one way to write a single-column B-tree. At minimum, I think writers should default to the existing layout for a single key.
There was a problem hiding this comment.
Honestly, the only thing I can think of, for having a one-key key_layout would be if we wanted to support different collations, which affect both equality and sorting, even on single-keys.
I would like to keep it even for single-keys as we technically also lose nothing if we support it, and if it's missing it will use the raw-value layout anyways? what do you think?
There was a problem hiding this comment.
Yeah, that seems acceptable. I don't have a strong opinion.
| tuples are equal exactly when their encodings are equal. A reader therefore | ||
| orders and searches pages by comparing bytes, and never decodes a key to do so. | ||
|
|
||
| ### Encoding Examples |
There was a problem hiding this comment.
praise: Love the worked byte examples here. Combined with your from-the-spec reimplementation that reproduces the ordering fixtures, this makes "the page, not the crate, defines version 1" a real promise.
| `FLAG_BTREE_KEY_LAYOUT`, a writer that supports key layouts drops the | ||
| index (see | ||
| [Compatibility With Released Versions](#compatibility-with-released-versions)). | ||
| Otherwise the engine does not add rows or fragments to the index, and its |
There was a problem hiding this comment.
question(non-blocking): Why does an unusable segment stop compaction from rewriting the fragments it covers, rather than letting compaction drop those fragments from fragment_bitmap the way update does?
Take a writer that understands arrow_row version 1 sharing a table with a newer writer's version 2 segment. The older writer would leave those fragments uncompacted indefinitely. Shrinking coverage keeps results correct (the reader just scans those fragments), so I'd lean toward that unless there's a reason I'm missing.
There was a problem hiding this comment.
Changed as suggested. A writer that supports key layouts may now compact the fragments that an unusable key layout segment covers. The commit that rewrites them removes them from the segment's fragment_bitmap, as an update does, and readers scan the new fragments. The same commit removes them when compaction defers the remap to a fragment reuse index, so the segment covers none of the fragments that the reuse index maps them to. The rule is in the new Key Layout Writer Rules section. Released versions keep their behavior, because v12.0.0 through v14.0.0 refuse the flag and the compaction of v10.0.0 and v11.0.0 drops the index.
| whose writer feature flags lack `FLAG_BTREE_KEY_LAYOUT` (see | ||
| [Key Layout Reader Rules](#key-layout-reader-rules)). | ||
|
|
||
| A writer that supports key layouts never carries an index with a `key_layout` |
There was a problem hiding this comment.
suggestion: This paragraph is the rule writers have to follow, but it comes after ~25 lines on what v10.0.0 through v14.0.0 do (L382-L404). Could we move it next to Key Layout Reader Rules as "Key Layout Writer Rules", written as musts?
Then we could trim the release-by-release narrative to a sentence or two of rationale. The same narrative is repeated in the FLAG_BTREE_KEY_LAYOUT doc comment and in the versioning table. The spec should say what implementations must do, not what past releases did.
There was a problem hiding this comment.
Done. A new Key Layout Writer Rules section follows the reader rules and states as musts when a writer sets FLAG_BTREE_KEY_LAYOUT, that it drops a key layout segment from a manifest without the bit, and how it maintains a segment it cannot use. Compatibility With Released Versions now gives only the reason for each fence. The FLAG_BTREE_KEY_LAYOUT doc comment now says what the bit means and links to the writer rules.
| | `2^17` | `FLAG_OBJECT_METADATA` | No | Yes | The snapshot contains [clustering definitions, references, or a nonzero allocation high-water mark](clustering.md), nonempty [fragment metadata](index.md#fragment-metadata), or nonempty [index segment metadata](../index/index.md#index-segment-metadata). Reserved, unsupported until writers implement the maintenance rules. Readers may ignore this metadata. | | ||
| | `2^18` | `FLAG_ROW_ADDR_INDICES` | No | Yes | A stable-row-ID table contains an index that stores physical row addresses. Older writers could treat its entries as stable row IDs and silently extend coverage without indexing rewritten fragments. | | ||
| | `2^19` | `FLAG_INLINE_DELETIONS` | Yes | Yes | Some fragment stores its deletion vector in the manifest (`DeletionFile.inline`) rather than in a file under `_deletions/`. A reader without this flag would look for a file that does not exist. | | ||
| | `2^20` | `FLAG_BTREE_KEY_LAYOUT` | No | Yes | Some BTree index segment has a [key layout](../index/scalar/btree.md#key-layout) (`BTreeIndexDetails.key_layout`). Without the bit, an update by v12.0.0 or v13.0.0 on a table with stable row IDs adds fragments to the segment's coverage without indexing their rows, and a later writer that supports `FLAG_ROW_ADDR_INDICES` (`2^18`) but not key layouts can re-set `FLAG_ROW_ADDR_INDICES` without repairing the segment. A writer that supports this bit sets it on every published manifest, including by clone and restore, exactly when such a segment exists, and never carries such a segment out of a manifest that lacks it. Readers do not use such a segment from a manifest that lacks the bit. Reserved, unsupported until key layouts are implemented. | |
There was a problem hiding this comment.
question(non-blocking): Why do we need a table-level flag for an index feature? I'd expect the index version to be enough. As I read the compatibility section, released writers update fragment_bitmap without checking whether they support the index version, so this is the only fence for v12/v13. That makes sense for this PR.
I filed #11238 so future index changes don't each need their own writer flag.
There was a problem hiding this comment.
Yes. Released writers update fragment_bitmap without checking the index version, so for v12.0.0 and v13.0.0 the flag is the only writer fence. Thanks for filing #11238.
Co-authored-by: Will Jones <willjones127@gmail.com>
Co-authored-by: Will Jones <willjones127@gmail.com>
Main checks in the generated protobuf sources, so regenerate rust/lance-index/src/pb/lance.table.rs for BTreeIndexDetails.key_layout and its messages. Name FLAG_BTREE_KEY_LAYOUT "btree_key_layout" in the lance-tools feature flag table and in the test that lists every known flag. Co-authored-by: Cursor <cursoragent@cursor.com>
…overage Collect the writer rules for BTree key layouts in a Key Layout Writer Rules section next to the reader rules: when to set FLAG_BTREE_KEY_LAYOUT, dropping a key layout segment carried from a manifest without the bit, and maintaining a segment the writer cannot use. A writer that supports key layouts may now compact the fragments that an unusable key layout segment covers. The compacting commit removes them from the segment's fragment_bitmap, as an update does, including when compaction defers the remap to a fragment reuse index. Readers then scan the new fragments. By default, a writer builds a single-column index without a key_layout. Cut Compatibility With Released Versions to the rationale for the two fences, and trim the FLAG_BTREE_KEY_LAYOUT doc comment to what the bit means and a link to the writer rules. Co-authored-by: Cursor <cursoragent@cursor.com>
wjones127
left a comment
There was a problem hiding this comment.
Thanks for turning these around. The new Key Layout Writer Rules section reads as rules now, compaction shrinking fragment_bitmap instead of being blocked is what I was hoping for, and the compatibility section and versioning row are much tighter. Re-approving on the latest commit so the vote gate clears.
One optional note inline: the Encoding Stability section still pins version 1 to arrow-row 58.4.0, but the workspace is on 59.2.0.
FYI, #10459 also claims bit 20. Whichever lands second needs to move to bit 21, including the FLAG_UNKNOWN bump and the lance-tools name.
| ### Encoding Stability | ||
|
|
||
| The bytes of `arrow_row` version 1 are the bytes that the `RowConverter` of the | ||
| Arrow Rust crate `arrow-row` 58.4.0 writes for the admitted types. Arrow does |
There was a problem hiding this comment.
suggestion: Update the pinned arrow-row version to the one we build with.
The workspace is on arrow-row 59.2.0 now (Cargo.toml), so this pins version 1 to a crate version we no longer build with. Since the page defines the bytes, the version here is just provenance, but I'd update it (assuming the arrow_ordering.rs fixtures still pass on 59.2.0) so it doesn't look like the upgrade skipped a check.
| Arrow Rust crate `arrow-row` 58.4.0 writes for the admitted types. Arrow does | |
| Arrow Rust crate `arrow-row` 59.2.0 writes for the admitted types. Arrow does |
There was a problem hiding this comment.
Updated to 59.3.3 rather than 59.2.0. The 59.2.0 in Cargo.toml is the minimum it accepts. After merging main, Cargo.lock resolves 59.3.3, so that is the crate the build links and the one the fixtures ran on. cargo test -p lance --test arrow_ordering passes on 59.3.3, all 46 tests, including the 29 row_converter_bytes_match_baseline cases. I did not run them on 59.2.0.
Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # docs/src/format/table/versioning.md # rust/lance-table/src/feature_flags.rs # rust/lance-tools/src/manifest.rs
….3.3 Main took bits 20 and 21 for FLAG_LOCAL_FRAGMENT_ALIASES and FLAG_DELETION_FILE_PATHS, so the writer rules name FLAG_BTREE_KEY_LAYOUT as 2^22. Encoding Stability now names arrow-row 59.3.3, the version Cargo.lock resolves; the arrow_ordering fixtures pass on it. Co-authored-by: Cursor <cursoragent@cursor.com>
|
@wjones127 @westonpace @Xuanwo thanks, the vote cleared on a6d61b2. Since then
The vote gate needs one approval on the new head. Could one of you re-approve, and could a maintainer merge once CI is green? Merging soon would avoid another bit collision, since #10459 also needs a new bit. The Java jobs fail while Maven resolves spotless's dependencies ( |
The Java jobs on aca8f04 failed to resolve maven-jar-plugin 3.0.2 from Maven Central. No content change. Co-authored-by: Cursor <cursoragent@cursor.com>
Why
This PR is the format vote for the composite key track in #10250. It specifies how a BTree index stores encoded keys, for two or more columns or for one column with an explicit direction and null placement, so that later PRs can build the codec, training, and search against a voted contract.
What changed
BTreeIndexDetailsgains an optionalkey_layout. It sets the key encoding and the key ordering, each aoneofwith one message per kind, and gives each key its direction and null placement. Thearrow_rowmessage carries its version.key_layouthas the version 1 layout, and a reader picks the page layout from whetherkey_layoutis present, not from the number of fields.arrow_rowversion 1 bytes with 52 examples from the Arrow ordering fixtures, thelexicographicordering, the reader rules, and the writer rules. A short compatibility section gives the reason for each fence: the index version fences released readers, and the flag fences the v12.0.0 and v13.0.0 writers that would otherwise extendfragment_bitmap.key_layoutfrom a manifest without the bit, and how it maintains a segment it cannot use. Compaction may rewrite the fragments such a segment covers. The same commit removes them from itsfragment_bitmap, as an update does, also when compaction defers the remap to a fragment reuse index, and readers scan them. By default, a writer builds a single-column index without akey_layout. One-key layouts stay allowed.FLAG_BTREE_KEY_LAYOUT, is reserved and unsupported, andlance-toolsnames itbtree_key_layout. Its doc comment says what the bit means and links to the writer rules. Bits 19, 20, and 21 went toFLAG_INLINE_DELETIONS,FLAG_LOCAL_FRAGMENT_ALIASES(feat(format)!: add local_fragment_aliases to IndexMetadata #10780), andFLAG_DELETION_FILE_PATHS(feat(table): store explicit deletion file path #10914) onmain. The bit 22 fence re-check passed 173 checks: 104 table and chain checks matched the expected outcomes, and all 69 rows shared with the bit 20 runs matched cell for cell. It gives the table below.mainchecks in generated protobuf code, sorust/lance-index/src/pb/lance.table.rsis regenerated forkey_layout.arrow_rowversion 1 toarrow-row59.3.3, the versionmainbuilds with, and thearrow_ordering.rsfixtures pass on it.Scope
lance-toolsname. No table writer emits akey_layoutor derives bit 22 yet, and no BTree query path interpretskey_layout. Generated protobuf code can decode the field, the feature-flag gate rejects bit 22 as unsupported, andlance-toolsrenders its name.key_layout. fix(index): fence older writers from row-address indices #10352 addedFLAG_ROW_ADDR_INDICESat bit 18 for stable row ID tables with a BTree at index version 1 or above, butFLAG_ROW_ADDR_INDICEScannot replaceFLAG_BTREE_KEY_LAYOUT: writers that supportFLAG_ROW_ADDR_INDICESwithoutkey_layoutsupport (such as 15.0.0b7) recompute bit 18 fromindex_version >= 1on restore, clone, and commits without checking the source manifest's flags or dropping a damagedkey_layoutsegment after a 10.0.0 or 11.0.0 restore clears bit 18 and a 12.0.0 or 13.0.0 update extendsfragment_bitmap. KeepingFLAG_BTREE_KEY_LAYOUTas an independent bit that pre-key_layoutwriters cannot set ensures a damaged segment never becomes usable again.key_layoutare left to a later vote.Blast Radius
Every BTree written today has empty details, so it decodes as before. No writer sets bit 22 yet. Index version 2 is free, because #10150 kept stable row ID BTrees on version 1 and #10152 does not change the index version.
Verification
arrow_rowversion 1 from the spec text alone. It reproduces all 113 fixtures inrust/lance/tests/arrow_ordering.rsand all 52 examples on the page. On 3,000 random tuples, byte order equals thelexicographicorder. Three planted errors each fail it.FLAG_BTREE_KEY_LAYOUT, once with stable row IDs and once without. Each pylance release runs each write first on its own copy, then queriesa = 1andb = 1.key_layoutmeets the same fences, from reading 10.0.0 through 14.0.0. Every release drops a BTree segment above its maximum index version from the index list that reads and query planning use, before any code countsfields. The commits of 10.0.0 and 11.0.0 also use that filtered list, which is why they drop the index. The commits of 12.0.0 through 14.0.0 keep the segment, and an update counts itsfields, butFLAG_BTREE_KEY_LAYOUTrefuses those commits. Once a 10.0.0 or 11.0.0 restore clears the bit, the reader rule makes the segment unusable whatever its key count, even if a writer that supportsFLAG_ROW_ADDR_INDICES(2^18) later sets bit 18 again.cargo run --locked --manifest-path protos/codegen/Cargo.toml -- --check,cargo test -p lance-table --lib feature_flags,cargo test -p lance-tools format_feature_flags,cargo test -p lance --test arrow_ordering,cargo fmt --all -- --check, workspace clippy,ci/check_proto_comments.py,ci/check_module_size.py, andmake buildindocs/pass, and every link and anchor on the changed pages resolves.The index version fences readers. In 10.0.0 through 14.0.0,
a = 1andb = 1return the same rows as a full scan after every write, and no plan uses the composite index. Bit 22 fences writers from 12.0.0 on. "Stable" and "plain" name the table kind where the two differ.optimize_indices()compact_files()After an 11.0.0 restore, shallow clone, or branch clears bit 22, a 13.0.0 update moves the composite index's
fragment_bitmapfrom[0, 1, 2, 3]to[0, 1, 2, 3, 5], and a 15.0.0b7 restore sets bit 18 (FLAG_ROW_ADDR_INDICES) again while leaving bit 22 (FLAG_BTREE_KEY_LAYOUT) clear. That result is why an index with akey_layoutfrom a manifest withoutFLAG_BTREE_KEY_LAYOUTis unusable.