Repository navigation
Conversation
Adaptive IVF_RQ batches fell back to one indexed search per query, because the shared node's late search returned other distances: multi-bit RQ computes a row's full distance only when a probabilistic bound beats the query's current k-th distance, and the shared late search scored each (query, partition) with a heap of its own, while single-query search keeps one heap across a query's late partitions. For a scorer whose results depend on the top-k (RabitQ), each query that still needs rows now searches its late range with search_partitions, as single-query search does, while the early search stays shared, and batch_index_search_supported no longer excludes adaptive IVF_RQ. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The per-query late search kept every partition's result batch, empty ones included, and every finished query's batches until the whole batch finished, so its memory grew as queries x late partitions even when nothing matched. Each query now appends a partition's rows to its own bounded top-k as they arrive and drops empty batches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
@Xuanwo @wjones127 @BubbleCal could you review this when you have time? It moves adaptive IVF_RQ batches from the per-query fallback that #10443 kept onto the shared-scan path: 5.4 to 8.5 times faster in a local benchmark, with the same recall. The late search runs per query through |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Batch vector search with adaptive nprobes on an IVF_RQ index now takes the shared-scan path (
ANNIvfBatchExec) instead of one indexed search per query: 5.4 to 8.5 times faster in a local benchmark, with the same recall.Why
#10443 kept adaptive IVF_RQ on the per-query fallback, because the shared node's late search returned other distances than single-query search for it. Every IVF_RQ batch with the default (adaptive) probe bounds, or with an explicit
search_effort, therefore searched each query separately.The difference comes from the late search. Multi-bit RQ computes a row's full distance only when a probabilistic bound on it beats the current k-th distance of the query's top-k, so its results depend on the heap it scores against. Single-query search scores a query's late partitions with one
search_partitionscall, sharing one heap across them. The shared node scores each (query, partition) pair as its own virtual query insearch_partitions_batch, with a heap of its own. With the fallback removed and nothing else changed, a filter that sends queries into late search (i % 97 = 0) returned the same rows as single-query search with different distances on 4 to 5 of 6 queries, in every run; fixed-probe RQ and single-query reruns matched exactly.How
batch_probe::late_searchchecks whether a delta's scorer depends on the top-k (QuantizationType::Rabit). If so, each query that still needs rows searches its late range in every delta it joins withsearch_partitions, as single-query search does, with aPartitionSearchControlthat stops it at the same budget. Deltas are searched in order and queries in parallel, and each partition's rows go straight into the query's bounded top-k as they arrive, so the late search holds at mostkrows per query. The early search stays shared.batch_index_search_supportedno longer excludes adaptive IVF_RQ.partitions_searchedcounts a query's late partitions once per query on this path, like single-query search, and the shared early scan once.Testing
test_batch_knn_adaptive_nprobes_matches_singlenow runs IVF_RQ on the shared node and checks that rows and distances equal single-query search, for every case including late search with and without a cappedmaximum_nprobes. Without the per-query late search, the late-search cases fail on distances.test_sequential_late_search_stops_at_the_cutandtest_sequential_late_search_over_empty_partitionscheck the per-query late search directly: it stops at the same partition as the shared one, keeps the same rows, and counts every partition it probes while keeping only bounded candidates.test_batch_knn_adaptive_rq_recall_matches_singleruns adaptive IVF_RQ batches on 20,000 vectors, where the shared early scan can keep other rows than single-query search, and checks that their recall@10 against exact search stays within 0.02 of one search per query, with and without a filter that sends queries into late search. The Pythontest_vector_search_effortbatch assertion now expects the shared count for IVF_RQ too.Results compared with single-query search
The late search now matches single-query search: on the test dataset, adaptive IVF_RQ batches with a filter that sends queries into late search return the same rows and distances, where they returned other distances on 4 to 5 of 6 queries without the per-query late search.
The shared early scan, which fixed-probe IVF_RQ already uses on
main, can keep other rows than single-query search on a larger index. On 200,000 random 128-d vectors (IVF_RQ, 256 partitions, 5 bits,k = 10), 3 to 5% of queries returned a different row among their 10, and recall@10 against exact search stayed within 0.0005 of per-query search:i % 50 = 0i % 50 = 0(Local
benchprofile build, best of 3, default adaptive bounds. Recall is low because the vectors are random; the comparison is between the two modes on the same index.)Gates
On this branch with
mainat 3f2b154 merged in:dataset::tests::dataset_scanner_knn,io::exec::knnandindex::vector::ivf: 2,002 tests pass (1 ignored).LANCE_CPU_THREADS=1: 2,001 pass, andtest_unowned_row_does_not_fill_the_shared_budget::query_parallelism_2_2fails the same way onmainat 3f2b154 ("the parallel path must not prepare partitions").cargo clippy -p lance --tests -- -D warningsandcargo fmt --checkare clean.test_vector_index.pytests matchingsearch,batch,multivectororrq: 106 pass.🤖 Generated with Claude Code