Skip to content

feat(index): support local_fragment_aliases in readers and index maintenance - #11280

Open
geruh wants to merge 1 commit into
lance-format:mainfrom
geruh:feat/local-fragment-aliases-runtime
Open

geruh wants to merge 1 commit into
lance-format:mainfrom
geruh:feat/local-fragment-aliases-runtime

Conversation

@geruh

@geruh geruh commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Follow up to #10780. This wires up local_fragment_aliases in the readers and index maintenance paths so that once staged transactions start writing aliases, they dont get lost or break queries, compaction, FRI, or index merges.

When an index is built before commit, the index file has whatever fragment ids it saw during training (say frag 0). On commit or rebase, the dataset might assign that fragment a different id (say frag 42). Instead of rewriting the index file, we keep {0: 42} in IndexMetadata.local_fragment_aliases and translate on read before FRI runs:

flowchart LR
    A["Index file (frag 0)"] --> B["local_fragment_aliases {0: 42}"]
    B --> C["Committed frag 42"]
    C --> D["FRI (if compacted)"]
    D --> E["Current frag"]
Loading

When compaction, remap, or optimize_indices rewrites the index files, the new files have the real fragment ids and local_fragment_aliases (plus bit 20) gets cleared. Metadata only updates keep the map.

What changed:

  • Validate local_fragment_aliases on commit and read, and set or clear FLAG_LOCAL_FRAGMENT_ALIASES (bit 20) in assemble_manifest and write_manifest_file
  • Run alias translation ahead of both v0 FRI and tagged FRI when opening scalar indexes (anything with supports_batch_row_id_remapping), and namespace the index cache by the alias map (alias_cache_namespace)
  • Translate through the active remapper in calculate_included_frags for BTree, ZoneMap, BloomFilter, and RTree so bitmap recovery doesnt write old training ids back into fragment_bitmap
  • Prepend aliases ahead of remap_streaming (prepend_fragment_aliases) for raw page readers during compaction while leaving already translated Bitmap and LabelList postings untouched, wire per source remappers into NGram merges, and rebuild aliased ZoneMap and BloomFilter segments during optimize_indices
  • Reject logical row id segments on stable row id tables, vector indexes, and Java CreateIndex round trips that would silently drop aliases
  • Left format docs alone so we dont trigger another format vote

Breaking change (Rust): lance_table::format::IndexMetadata now has a public local_fragment_aliases: BTreeMap<u32, u32> field, so external Rust callers constructing IndexMetadata struct literals directly will need to initialize it.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions github-actions Bot added A-python Python bindings A-index Vector index, linalg, tokenizer A-java Java bindings + JNI A-namespace Namespace impls enhancement New feature or request labels Nov 20, 2026
@codecov

codecov Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

❌ 1 Tests Failed:

Tests completed Failed Passed Skipped
23296 1 23295 0
View the top 1 failed test(s) by shortest run time
lance-table::feature_flags::tests::test_deletion_file_paths_flag
Stack Traces | 0.296s run time
thread 'feature_flags::tests::test_deletion_file_paths_flag' (132206) panicked at .../lance-table/src/feature_flags.rs:1676:9:
assertion failed: !can_read_dataset(FLAG_LOCAL_FRAGMENT_ALIASES)
stack backtrace:
   0: __rustc::rust_begin_unwind
             at .../std/src/panicking.rs:679:5
   1: core::panicking::panic_fmt
             at .../core/src/panicking.rs:80:14
   2: core::panicking::panic
             at .../core/src/panicking.rs:150:5
   3: lance_table::feature_flags::tests::test_deletion_file_paths_flag
             at ./src/feature_flags.rs:1676:9
   4: lance_table::feature_flags::tests::test_deletion_file_paths_flag::{closure#0}
             at ./src/feature_flags.rs:1674:39
   5: <lance_table::feature_flags::tests::test_deletion_file_paths_flag::{closure#0} as core::ops::function::FnOnce<()>>::call_once
             at .../src/ops/function.rs:250:5
   6: <fn() -> core::result::Result<(), alloc::string::String> as core::ops::function::FnOnce<()>>::call_once
             at .../src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Nov 20, 2026
@geruh
geruh force-pushed the feat/local-fragment-aliases-runtime branch from aa91774 to 842dffb Compare November 20, 2026 01:43
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Nov 20, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Nov 20, 2026
…tenance

Co-authored-by: Drew <geruh@users.noreply.github.com>
@geruh
geruh force-pushed the feat/local-fragment-aliases-runtime branch from 842dffb to 3b7b7b6 Compare November 20, 2026 03:12
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Nov 20, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

All previously reported maintenance defects are fixed. Rewrites now preserve committed row addresses, and operations retaining index files retain their aliases.

Please mark this PR with the breaking-change label: lance_table::format::IndexMetadata.local_fragment_aliases is a new public field, so Rust callers using exhaustive struct literals must initialize it.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-index Vector index, linalg, tokenizer A-java Java bindings + JNI A-namespace Namespace impls A-python Python bindings enhancement New feature or request K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant