Repository navigation
refactor(index): move IVF optimize tests into their own module - #11264
Merged
Xuanwo merged 1 commit intoNov 19, 2026
Merged
Conversation
Contributor
Author
|
cc @Xuanwo |
Xuanwo
approved these changes
Nov 19, 2026
Xuanwo
added a commit
that referenced
this pull request
Nov 20, 2026
…xes without panicking (#11263) Closes #11262 The new tests go in `ivf/tests/optimize.rs`, which #11264 split out of `ivf.rs` to keep it under the module size limit. ## What changes - The legacy `IVFIndex` reports `quantizer()` and `sub_index_type()` from its sub-index, and `PQIndex::quantizer()` returns its codebook. The shared-model check in `optimize_vector_indices` calls both before choosing a writer, so appending to or retraining a legacy index panicked. - `optimize_vector_indices_v2` gets a `(Hnsw, FlatBin)` arm. Every pair now has an arm, so the `unimplemented!` catch-all is gone. - Steady-state rebalancing skips legacy segments: the legacy writer never splits or joins, so it would rewrite the same file on every call. - On a stable-row-id table a legacy index no longer merges. The legacy writer copies every posting, so an updated row kept its old entry. Default optimize appends instead; an explicit merge returns `NotSupported`. - A legacy IVF_HNSW_PQ index returns `NotSupported` suggesting retrain. ## Tests - `test_optimize_ivf_hnsw_binary_vectors`: appends to a uint8 IVF_HNSW_FLAT index; the appended row comes back through ANN, recall ≥ 0.5. - `test_optimize_ivf_pq_legacy_segments`: merges into a legacy IVF_PQ index, which stays legacy with recall ≥ 0.5; steady state commits nothing; retrain rebuilds in the current format. - `test_optimize_legacy_ivf_pq_with_stable_row_ids_skips_merge`: after an update, the row's old vector no longer finds it; an explicit merge returns `NotSupported`. - `test_steady_state_splits_v3_segment_beside_legacy_segment`: beside a legacy segment, an oversized current-format segment is still split. - `test_optimize_legacy_v1_hnsw_pq_refuses_incremental_optimize_and_retrains`: a (0,2) IVF_HNSW_PQ segment returns `NotSupported`, and retrain rebuilds it. 🤖 Generated with Claude Opus --------- Co-authored-by: Xuanwo <github@xuanwo.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
rust/lance/src/index/vector/ivf.rsis 399,655 bytes on main, andci/check_module_size.pyfails a PR that grows a Rust file past 400,000 bytes. #11263 adds tests to it and fails that check.This moves the three optimize tests into
ivf/tests/optimize.rs:test_optimize_builds_merge_filters_only_when_merging,test_optimize_ivf_flat_binary_vectorsandtest_optimize_ivf_hnsw_records_actual_file_sizes.ivf.rsdrops to 390,869 bytes. The moved code is unchanged apart from one level of indentation, and the new module reaches the helpers throughuse super::*.