Skip to content

refactor(index): move IVF optimize tests into their own module - #11264

Merged
Xuanwo merged 1 commit into
lance-format:mainfrom
LuciferYang:refactor/ivf-optimize-tests-module
Nov 19, 2026
Merged

Xuanwo merged 1 commit into
lance-format:mainfrom
LuciferYang:refactor/ivf-optimize-tests-module

Conversation

@LuciferYang

@LuciferYang LuciferYang commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

rust/lance/src/index/vector/ivf.rs is 399,655 bytes on main, and ci/check_module_size.py fails a PR that grows a Rust file past 400,000 bytes. #11263 adds tests to it and fails that check.

This moves the three optimize tests into ivf/tests/optimize.rs: test_optimize_builds_merge_filters_only_when_merging, test_optimize_ivf_flat_binary_vectors and test_optimize_ivf_hnsw_records_actual_file_sizes. ivf.rs drops to 390,869 bytes. The moved code is unchanged apart from one level of indentation, and the new module reaches the helpers through use super::*.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

No blocking issues found. The test-only split addresses the file-size constraint while preserving existing coverage and production behavior.

@LuciferYang

Copy link
Copy Markdown
Contributor Author

cc @Xuanwo

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. and removed K-approved Latest Gatekeeper recommendation permits acceptance. labels Nov 19, 2026
@Xuanwo
Xuanwo merged commit f063327 into lance-format:main Nov 19, 2026
47 checks passed
Xuanwo added a commit that referenced this pull request Nov 20, 2026
…xes without panicking (#11263)

Closes #11262

The new tests go in `ivf/tests/optimize.rs`, which #11264 split out of
`ivf.rs` to keep it under the module size limit.

## What changes

- The legacy `IVFIndex` reports `quantizer()` and `sub_index_type()`
from its sub-index, and `PQIndex::quantizer()` returns its codebook. The
shared-model check in `optimize_vector_indices` calls both before
choosing a writer, so appending to or retraining a legacy index
panicked.
- `optimize_vector_indices_v2` gets a `(Hnsw, FlatBin)` arm. Every pair
now has an arm, so the `unimplemented!` catch-all is gone.
- Steady-state rebalancing skips legacy segments: the legacy writer
never splits or joins, so it would rewrite the same file on every call.
- On a stable-row-id table a legacy index no longer merges. The legacy
writer copies every posting, so an updated row kept its old entry.
Default optimize appends instead; an explicit merge returns
`NotSupported`.
- A legacy IVF_HNSW_PQ index returns `NotSupported` suggesting retrain.

## Tests

- `test_optimize_ivf_hnsw_binary_vectors`: appends to a uint8
IVF_HNSW_FLAT index; the appended row comes back through ANN, recall ≥
0.5.
- `test_optimize_ivf_pq_legacy_segments`: merges into a legacy IVF_PQ
index, which stays legacy with recall ≥ 0.5; steady state commits
nothing; retrain rebuilds in the current format.
- `test_optimize_legacy_ivf_pq_with_stable_row_ids_skips_merge`: after
an update, the row's old vector no longer finds it; an explicit merge
returns `NotSupported`.
- `test_steady_state_splits_v3_segment_beside_legacy_segment`: beside a
legacy segment, an oversized current-format segment is still split.
-
`test_optimize_legacy_v1_hnsw_pq_refuses_incremental_optimize_and_retrains`:
a (0,2) IVF_HNSW_PQ segment returns `NotSupported`, and retrain rebuilds
it.

🤖 Generated with Claude Opus

---------

Co-authored-by: Xuanwo <github@xuanwo.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants