Repository navigation
fix(index): optimize legacy IVF segments and uint8 IVF_HNSW_FLAT indexes without panicking - #11263
Merged
Xuanwo merged 3 commits intoNov 20, 2026
Conversation
This was referenced Nov 19, 2026
LuciferYang
force-pushed
the
fix/optimize-legacy-ivf-and-hnsw-flatbin
branch
from
November 19, 2026 08:06
721ea98 to
bb60c19
Compare
Xuanwo
pushed a commit
that referenced
this pull request
Nov 19, 2026
`rust/lance/src/index/vector/ivf.rs` is 399,655 bytes on main, and `ci/check_module_size.py` fails a PR that grows a Rust file past 400,000 bytes. #11263 adds tests to it and fails that check. This moves the three optimize tests into `ivf/tests/optimize.rs`: `test_optimize_builds_merge_filters_only_when_merging`, `test_optimize_ivf_flat_binary_vectors` and `test_optimize_ivf_hnsw_records_actual_file_sizes`. `ivf.rs` drops to 390,869 bytes. The moved code is unchanged apart from one level of indentation, and the new module reaches the helpers through `use super::*`.
…xes without panicking
LuciferYang
force-pushed
the
fix/optimize-legacy-ivf-and-hnsw-flatbin
branch
from
November 19, 2026 09:23
bb60c19 to
1ccdc03
Compare
The legacy optimize guard admitted HNSW<ScalarQuantizer>, but the v1 HNSW partition writer only supports product quantization. It created the index and auxiliary writers before failing, and no version was published. Narrow the legacy eligibility to IVF_PQ before output allocation so HNSW_SQ returns the existing NotSupported retrain guidance without leaving empty files. Remove the now-unreachable legacy HNSW optimize path. Keep the current-format HNSW_SQ file-size test on its real v2 path, and add a released 0.11.0 IVF_HNSW_SQ fixture regression that verifies old reads, the (0,2) format, NotSupported guidance and no publication.
Retraining a released 0.11.0 HNSW_SQ segment fails before rebuilding: its missing index details make the planner treat the legacy vector index as a scalar one and look for page_lookup.lance. Point the rejection at dropping the segment and rebuilding from the data with create_index instead, while keeping retrain guidance for the legacy HNSW_PQ path that tests cover. Extend the released-fixture test to verify the rejected optimize preserves the legacy file and that the drop/create recovery indexes appended rows in the current format and survives reopen.
Xuanwo
approved these changes
Nov 20, 2026
Xuanwo
pushed a commit
that referenced
this pull request
Nov 20, 2026
Closes #11265 ## What changes - `optimize_options()` raises `ValueError: Invalid optimize_indices option: <key>` for a key it does not know, as `parse_compaction_options` does. `DatasetOptimizer.optimize_indices`, `OptimizeIndices.plan` and `OptimizeIndices.commit` all parse through it. - `DatasetOptimizer.optimize_indices` takes its six options as keyword-only parameters, like `compact_files`, so a typo raises `TypeError` before reaching the binding. - A known option set to `None` means unset in every entry point. Before, `retrain=None` and `index_names=None` raised `TypeError` or `ValueError`. - The `retrain` docs in Rust, Python and Java say that only vector indices are retrained. The Rust note "only supported for v3 vector indices" predates the current rebuild-from-source retrain. A legacy-format vector index panics on retrain until #11263 lands. A caller that passed an option name `optimize_indices` does not know now gets an error instead of a default optimize. No option in the repository or in lance-ray does. ## Tests `test_optimize_indices_rejects_unknown_options` passes a misspelled `index_name` to all four entry points (the Python method, the binding, `plan`, `commit`) and checks that each raises and nothing is committed. It then checks that `None` values are accepted by `plan` and by `optimize_indices`, and that the correctly spelled option optimizes. 🤖 Generated with Claude Opus
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #11262
The new tests go in
ivf/tests/optimize.rs, which #11264 split out ofivf.rsto keep it under the module size limit.What changes
IVFIndexreportsquantizer()andsub_index_type()from its sub-index, andPQIndex::quantizer()returns its codebook. The shared-model check inoptimize_vector_indicescalls both before choosing a writer, so appending to or retraining a legacy index panicked.optimize_vector_indices_v2gets a(Hnsw, FlatBin)arm. Every pair now has an arm, so theunimplemented!catch-all is gone.NotSupported.NotSupportedsuggesting retrain.Tests
test_optimize_ivf_hnsw_binary_vectors: appends to a uint8 IVF_HNSW_FLAT index; the appended row comes back through ANN, recall ≥ 0.5.test_optimize_ivf_pq_legacy_segments: merges into a legacy IVF_PQ index, which stays legacy with recall ≥ 0.5; steady state commits nothing; retrain rebuilds in the current format.test_optimize_legacy_ivf_pq_with_stable_row_ids_skips_merge: after an update, the row's old vector no longer finds it; an explicit merge returnsNotSupported.test_steady_state_splits_v3_segment_beside_legacy_segment: beside a legacy segment, an oversized current-format segment is still split.test_optimize_legacy_v1_hnsw_pq_refuses_incremental_optimize_and_retrains: a (0,2) IVF_HNSW_PQ segment returnsNotSupported, and retrain rebuilds it.🤖 Generated with Claude Opus