Skip to content

feat(index): incorporate data overlays when merging scalar index segments - #11273

Draft
ddupg wants to merge 6 commits into
lance-format:mainfrom
ddupg:fix/fts-merge-overlay
Draft

ddupg wants to merge 6 commits into
lance-format:mainfrom
ddupg:fix/fts-merge-overlay

Conversation

@ddupg

@ddupg ddupg commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Closes #11272. This addresses the full-text search (FTS) part of #11180.

Summary

A scalar index merge can combine segments built before and after the same data overlay. This draft refreshes each source against the snapshot used for the merge. It uses the source’s fragment coverage, indexed fields and build version to decide which entries need replacing. Once the indexed values and row references are current, the output segment can record the merge snapshot’s dataset_version.

  • Ordinary optimize_indices, explicit segment merges, and distributed optimize share the overlay refresh logic.
  • Row-oriented indices discard stale entries and train replacements from current values. FTS retains already-indexed hits under fast_search and indexes updated documents from older inputs. ZoneMap and BloomFilter can rebuild summaries for affected fragments.
  • The merged segment can serve refreshed rows without a query-time overlay recheck. Advancing its build version also allows later cleanup to trim old compaction mappings once no index needs them.

Validation

  • cargo fmt --all and git diff --check passed.
  • Ten targeted tests passed in the preceding verification of this change, including ZoneMap physical merges with and without stable row IDs. The whole-fragment stable-ID translation test and two FTS merge regressions passed locally. The BloomFilter version assertion is covered by the Linux CI rerun. Local workspace Clippy on macOS arm64 stops on existing unreachable expressions in lance-encoding/src/encodings/physical/byte_stream_split.rs (lines 358 and 393).

@github-actions github-actions Bot added A-index Vector index, linalg, tokenizer enhancement New feature or request labels Nov 19, 2026
@ddupg
ddupg force-pushed the fix/fts-merge-overlay branch from 81ad0f9 to be95d12 Compare November 19, 2026 14:21
@codecov

codecov Bot commented Nov 19, 2026

Copy link
Copy Markdown

❌ 1 Tests Failed:

Tests completed Failed Passed Skipped
7370 1 7369 0
View the top 1 failed test(s) by shortest run time
lance::index::create::tests::test_bloomfilter_distributed_segments_merge_and_query
Stack Traces | 0.422s run time
thread 'index::create::tests::test_bloomfilter_distributed_segments_merge_and_query' (60012) panicked at ....../src/index/create.rs:3234:9:
assertion `left == right` failed
  left: 2
 right: 1
stack backtrace:
   0: __rustc::rust_begin_unwind
             at .../std/src/panicking.rs:679:5
   1: core::panicking::panic_fmt
             at .../core/src/panicking.rs:80:14
   2: core::panicking::assert_failed_inner
             at .../core/src/panicking.rs:439:17
   3: core::panicking::assert_failed::<u64, u64>
             at .../core/src/panicking.rs:394:5
   4: {async_block#0}
             at ..../src/index/create.rs:3234:9
   5: poll<&mut dyn core::future::future::Future<Output=()>>
             at .../src/future/future.rs:133:9
   6: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:830:70
   7: with_budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../task/coop/mod.rs:167:5
   8: budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../task/coop/mod.rs:133:5
   9: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:830:25
  10: <tokio::runtime::scheduler::current_thread::Context>::enter::<core::task::poll::Poll<()>, <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}::{closure#0}>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:486:19
  11: <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on::<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:829:44
  12: <tokio::runtime::scheduler::current_thread::CoreGuard>::enter::<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:906:68
  13: <tokio::runtime::context::scoped::Scoped<tokio::runtime::scheduler::Context>>::set::<<tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>)>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../runtime/context/scoped.rs:40:9
  14: tokio::runtime::context::set_scheduler::<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}>::{closure#0}
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../src/runtime/context.rs:186:38
  15: try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>), tokio::runtime::scheduler::current_thread::{impl#9}::enter::{closure_env#0}<tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure_env#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>, core::option::Option<()>>>, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>)>
             at .../src/thread/local.rs:463:12
  16: <std::thread::local::LocalKey<tokio::runtime::context::Context>>::with::<tokio::runtime::context::set_scheduler<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}>::{closure#0}, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>)>
             at .../src/thread/local.rs:427:20
  17: tokio::runtime::context::set_scheduler::<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../src/runtime/context.rs:186:17
  18: <tokio::runtime::scheduler::current_thread::CoreGuard>::enter::<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:906:27
  19: <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on::<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:819:24
  20: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:216:33
  21: tokio::runtime::context::runtime::enter_runtime::<<tokio::runtime::scheduler::current_thread::CurrentThread>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, ()>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../runtime/context/runtime.rs:65:16
  22: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:204:9
  23: <tokio::runtime::runtime::Runtime>::block_on_inner::<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../src/runtime/runtime.rs:379:52
  24: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../src/runtime/runtime.rs:348:18
  25: test_bloomfilter_distributed_segments_merge_and_query
             at ..../src/index/create.rs:3289:77
  26: lance::index::create::tests::test_bloomfilter_distributed_segments_merge_and_query::{closure#0}
             at ..../src/index/create.rs:3129:69
  27: <lance::index::create::tests::test_bloomfilter_distributed_segments_merge_and_query::{closure#0} as core::ops::function::FnOnce<()>>::call_once
             at .../src/ops/function.rs:250:5
  28: <fn() -> core::result::Result<(), alloc::string::String> as core::ops::function::FnOnce<()>>::call_once
             at .../src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@ddupg
ddupg force-pushed the fix/fts-merge-overlay branch from 7690b0c to 8683af9 Compare November 20, 2026 03:27
@wombatu-kun

Copy link
Copy Markdown
Contributor

It looks like our work crossed in flight: #11260 and #11261 also address #11180, covering vector and FTS merges by retaining per-source overlay freshness. Your PR takes the other approach discussed in #10884: refreshing stale rows against the merge snapshot. It also goes beyond #11180 by bringing the other scalar indices up to date.

There is direct overlap in the FTS and shared merge lifecycle code, so I would like to coordinate rather than have us maintain competing FTS fixes. One possible split would be for this PR to own the scalar/FTS refresh, while I narrow #11260 and #11261 to the vector case. #11180 could then remain open until both the vector and FTS parts land.

What do you think? We can move the design discussion to #11180 so that everyone involved has the full context.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-index Vector index, linalg, tokenizer enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feature: incorporate data overlays when merging scalar index segments

2 participants