Skip to content

perf(vector): share batch scans for adaptive IVF_RQ - #11122

Open
sezruby wants to merge 4 commits into
lance-format:mainfrom
sezruby:perf/batch-adaptive-rq
Open

sezruby wants to merge 4 commits into
lance-format:mainfrom
sezruby:perf/batch-adaptive-rq

Conversation

@sezruby

@sezruby sezruby commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

What

Batch vector search with adaptive nprobes on an IVF_RQ index now takes the shared-scan path (ANNIvfBatchExec) instead of one indexed search per query: 5.4 to 8.5 times faster in a local benchmark, with the same recall.

Why

#10443 kept adaptive IVF_RQ on the per-query fallback, because the shared node's late search returned other distances than single-query search for it. Every IVF_RQ batch with the default (adaptive) probe bounds, or with an explicit search_effort, therefore searched each query separately.

The difference comes from the late search. Multi-bit RQ computes a row's full distance only when a probabilistic bound on it beats the current k-th distance of the query's top-k, so its results depend on the heap it scores against. Single-query search scores a query's late partitions with one search_partitions call, sharing one heap across them. The shared node scores each (query, partition) pair as its own virtual query in search_partitions_batch, with a heap of its own. With the fallback removed and nothing else changed, a filter that sends queries into late search (i % 97 = 0) returned the same rows as single-query search with different distances on 4 to 5 of 6 queries, in every run; fixed-probe RQ and single-query reruns matched exactly.

How

  • batch_probe::late_search checks whether a delta's scorer depends on the top-k (QuantizationType::Rabit). If so, each query that still needs rows searches its late range in every delta it joins with search_partitions, as single-query search does, with a PartitionSearchControl that stops it at the same budget. Deltas are searched in order and queries in parallel, and each partition's rows go straight into the query's bounded top-k as they arrive, so the late search holds at most k rows per query. The early search stays shared.
  • batch_index_search_supported no longer excludes adaptive IVF_RQ.
  • partitions_searched counts a query's late partitions once per query on this path, like single-query search, and the shared early scan once.

Testing

test_batch_knn_adaptive_nprobes_matches_single now runs IVF_RQ on the shared node and checks that rows and distances equal single-query search, for every case including late search with and without a capped maximum_nprobes. Without the per-query late search, the late-search cases fail on distances. test_sequential_late_search_stops_at_the_cut and test_sequential_late_search_over_empty_partitions check the per-query late search directly: it stops at the same partition as the shared one, keeps the same rows, and counts every partition it probes while keeping only bounded candidates. test_batch_knn_adaptive_rq_recall_matches_single runs adaptive IVF_RQ batches on 20,000 vectors, where the shared early scan can keep other rows than single-query search, and checks that their recall@10 against exact search stays within 0.02 of one search per query, with and without a filter that sends queries into late search. The Python test_vector_search_effort batch assertion now expects the shared count for IVF_RQ too.

Results compared with single-query search

The late search now matches single-query search: on the test dataset, adaptive IVF_RQ batches with a filter that sends queries into late search return the same rows and distances, where they returned other distances on 4 to 5 of 6 queries without the per-query late search.

The shared early scan, which fixed-probe IVF_RQ already uses on main, can keep other rows than single-query search on a larger index. On 200,000 random 128-d vectors (IVF_RQ, 256 partitions, 5 bits, k = 10), 3 to 5% of queries returned a different row among their 10, and recall@10 against exact search stayed within 0.0005 of per-query search:

Filter Queries Per-query search Shared scans Recall@10, per-query / shared
none 256 34 ms 6 ms 0.4449 / 0.4453
none 1,024 145 ms 17 ms 0.4337 / 0.4334
i % 50 = 0 256 65 ms 12 ms 0.2840 / 0.2844
i % 50 = 0 1,024 275 ms 38 ms 0.2743 / 0.2742

(Local bench profile build, best of 3, default adaptive bounds. Recall is low because the vectors are random; the comparison is between the two modes on the same index.)

Gates

On this branch with main at 3f2b154 merged in:

  • dataset::tests::dataset_scanner_knn, io::exec::knn and index::vector::ivf: 2,002 tests pass (1 ignored).
  • The same tests with LANCE_CPU_THREADS=1: 2,001 pass, and test_unowned_row_does_not_fill_the_shared_budget::query_parallelism_2_2 fails the same way on main at 3f2b154 ("the parallel path must not prepare partitions").
  • cargo clippy -p lance --tests -- -D warnings and cargo fmt --check are clean.
  • Python test_vector_index.py tests matching search, batch, multivector or rq: 106 pass.

🤖 Generated with Claude Code

sezruby and others added 2 commits November 15, 2026 18:56
Adaptive IVF_RQ batches fell back to one indexed search per query,
because the shared node's late search returned other distances: multi-bit
RQ computes a row's full distance only when a probabilistic bound beats
the query's current k-th distance, and the shared late search scored each
(query, partition) with a heap of its own, while single-query search
keeps one heap across a query's late partitions.

For a scorer whose results depend on the top-k (RabitQ), each query that
still needs rows now searches its late range with search_partitions, as
single-query search does, while the early search stays shared, and
batch_index_search_supported no longer excludes adaptive IVF_RQ.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions github-actions Bot added A-python Python bindings performance and removed A-python Python bindings labels Nov 16, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Nov 16, 2026
The per-query late search kept every partition's result batch, empty
ones included, and every finished query's batches until the whole batch
finished, so its memory grew as queries x late partitions even when
nothing matched. Each query now appends a partition's rows to its own
bounded top-k as they arrive and drops empty batches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added the A-python Python bindings label Nov 16, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Nov 16, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 16, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 18, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

The bounded late search remains intact after the merge with main, and the adaptive IVF_RQ regressions pass. I found no issues requiring changes.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. and removed K-approved Latest Gatekeeper recommendation permits acceptance. labels Nov 18, 2026
@sezruby

sezruby commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

@Xuanwo @wjones127 @BubbleCal could you review this when you have time? It moves adaptive IVF_RQ batches from the per-query fallback that #10443 kept onto the shared-scan path: 5.4 to 8.5 times faster in a local benchmark, with the same recall.

The late search runs per query through search_partitions, so it returns the same rows and distances as single-query search, and it holds at most k rows per query. CI is green with today's main merged in.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. and removed K-approved Latest Gatekeeper recommendation permits acceptance. labels Nov 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-python Python bindings K-approved Latest Gatekeeper recommendation permits acceptance. performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant