Repository navigation
Conversation
❌ 1 Tests Failed:
View the top 1 failed test(s) by shortest run time
To view more test analytics, go to the Test Analytics Dashboard |
geruh
force-pushed
the
feat/local-fragment-aliases-runtime
branch
from
November 20, 2026 01:43
aa91774 to
842dffb
Compare
…tenance Co-authored-by: Drew <geruh@users.noreply.github.com>
geruh
force-pushed
the
feat/local-fragment-aliases-runtime
branch
from
November 20, 2026 03:12
842dffb to
3b7b7b6
Compare
Contributor
There was a problem hiding this comment.
✅ Gate recommendation: approve.
All previously reported maintenance defects are fixed. Rewrites now preserve committed row addresses, and operations retaining index files retain their aliases.
Please mark this PR with the breaking-change label: lance_table::format::IndexMetadata.local_fragment_aliases is a new public field, so Rust callers using exhaustive struct literals must initialize it.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow up to #10780. This wires up
local_fragment_aliasesin the readers and index maintenance paths so that once staged transactions start writing aliases, they dont get lost or break queries, compaction, FRI, or index merges.When an index is built before commit, the index file has whatever fragment ids it saw during training (say frag 0). On commit or rebase, the dataset might assign that fragment a different id (say frag 42). Instead of rewriting the index file, we keep
{0: 42}inIndexMetadata.local_fragment_aliasesand translate on read before FRI runs:flowchart LR A["Index file (frag 0)"] --> B["local_fragment_aliases {0: 42}"] B --> C["Committed frag 42"] C --> D["FRI (if compacted)"] D --> E["Current frag"]When compaction, remap, or
optimize_indicesrewrites the index files, the new files have the real fragment ids andlocal_fragment_aliases(plus bit 20) gets cleared. Metadata only updates keep the map.What changed:
local_fragment_aliaseson commit and read, and set or clearFLAG_LOCAL_FRAGMENT_ALIASES(bit 20) inassemble_manifestandwrite_manifest_filesupports_batch_row_id_remapping), and namespace the index cache by the alias map (alias_cache_namespace)calculate_included_fragsfor BTree, ZoneMap, BloomFilter, and RTree so bitmap recovery doesnt write old training ids back intofragment_bitmapremap_streaming(prepend_fragment_aliases) for raw page readers during compaction while leaving already translated Bitmap and LabelList postings untouched, wire per source remappers into NGram merges, and rebuild aliased ZoneMap and BloomFilter segments duringoptimize_indicesCreateIndexround trips that would silently drop aliasesBreaking change (Rust):
lance_table::format::IndexMetadatanow has a publiclocal_fragment_aliases: BTreeMap<u32, u32>field, so external Rust callers constructingIndexMetadatastruct literals directly will need to initialize it.