Skip to content

fix(index): optimize legacy IVF segments and uint8 IVF_HNSW_FLAT indexes without panicking - #11263

Merged
Xuanwo merged 3 commits into
lance-format:mainfrom
LuciferYang:fix/optimize-legacy-ivf-and-hnsw-flatbin
Nov 20, 2026
Merged

Xuanwo merged 3 commits into
lance-format:mainfrom
LuciferYang:fix/optimize-legacy-ivf-and-hnsw-flatbin

Conversation

@LuciferYang

@LuciferYang LuciferYang commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Closes #11262

The new tests go in ivf/tests/optimize.rs, which #11264 split out of ivf.rs to keep it under the module size limit.

What changes

  • The legacy IVFIndex reports quantizer() and sub_index_type() from its sub-index, and PQIndex::quantizer() returns its codebook. The shared-model check in optimize_vector_indices calls both before choosing a writer, so appending to or retraining a legacy index panicked.
  • optimize_vector_indices_v2 gets a (Hnsw, FlatBin) arm. Every pair now has an arm, so the unimplemented! catch-all is gone.
  • Steady-state rebalancing skips legacy segments: the legacy writer never splits or joins, so it would rewrite the same file on every call.
  • On a stable-row-id table a legacy index no longer merges. The legacy writer copies every posting, so an updated row kept its old entry. Default optimize appends instead; an explicit merge returns NotSupported.
  • A legacy IVF_HNSW_PQ index returns NotSupported suggesting retrain.

Tests

  • test_optimize_ivf_hnsw_binary_vectors: appends to a uint8 IVF_HNSW_FLAT index; the appended row comes back through ANN, recall ≥ 0.5.
  • test_optimize_ivf_pq_legacy_segments: merges into a legacy IVF_PQ index, which stays legacy with recall ≥ 0.5; steady state commits nothing; retrain rebuilds in the current format.
  • test_optimize_legacy_ivf_pq_with_stable_row_ids_skips_merge: after an update, the row's old vector no longer finds it; an explicit merge returns NotSupported.
  • test_steady_state_splits_v3_segment_beside_legacy_segment: beside a legacy segment, an oversized current-format segment is still split.
  • test_optimize_legacy_v1_hnsw_pq_refuses_incremental_optimize_and_retrains: a (0,2) IVF_HNSW_PQ segment returns NotSupported, and retrain rebuilds it.

🤖 Generated with Claude Opus

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions github-actions Bot added the bug Something isn't working label Nov 19, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 19, 2026
@LuciferYang
LuciferYang force-pushed the fix/optimize-legacy-ivf-and-hnsw-flatbin branch from 721ea98 to bb60c19 Compare November 19, 2026 08:06
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 19, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 19, 2026
Xuanwo pushed a commit that referenced this pull request Nov 19, 2026
`rust/lance/src/index/vector/ivf.rs` is 399,655 bytes on main, and
`ci/check_module_size.py` fails a PR that grows a Rust file past 400,000
bytes. #11263 adds tests to it and fails that check.

This moves the three optimize tests into `ivf/tests/optimize.rs`:
`test_optimize_builds_merge_filters_only_when_merging`,
`test_optimize_ivf_flat_binary_vectors` and
`test_optimize_ivf_hnsw_records_actual_file_sizes`. `ivf.rs` drops to
390,869 bytes. The moved code is unchanged apart from one level of
indentation, and the new module reaches the helpers through `use
super::*`.
@LuciferYang
LuciferYang force-pushed the fix/optimize-legacy-ivf-and-hnsw-flatbin branch from bb60c19 to 1ccdc03 Compare November 19, 2026 09:23
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 19, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 19, 2026
@wjones127 wjones127 added the critical-fix Bugs that cause crashes, security vulnerabilities, or incorrect data. label Nov 19, 2026
The legacy optimize guard admitted HNSW<ScalarQuantizer>, but the v1 HNSW partition writer only supports product quantization. It created the index and auxiliary writers before failing, and no version was published.

Narrow the legacy eligibility to IVF_PQ before output allocation so HNSW_SQ returns the existing NotSupported retrain guidance without leaving empty files. Remove the now-unreachable legacy HNSW optimize path.

Keep the current-format HNSW_SQ file-size test on its real v2 path, and add a released 0.11.0 IVF_HNSW_SQ fixture regression that verifies old reads, the (0,2) format, NotSupported guidance and no publication.
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 20, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Nov 20, 2026
Retraining a released 0.11.0 HNSW_SQ segment fails before rebuilding: its missing index details make the planner treat the legacy vector index as a scalar one and look for page_lookup.lance.

Point the rejection at dropping the segment and rebuilding from the data with create_index instead, while keeping retrain guidance for the legacy HNSW_PQ path that tests cover. Extend the released-fixture test to verify the rejected optimize preserves the legacy file and that the drop/create recovery indexes appended rows in the current format and survives reopen.
@lance-gatekeeper lance-gatekeeper Bot removed K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Nov 20, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

No blocking issues found. Nine focused local regressions passed, including rebuilding the released HNSW_SQ fixture in the current format and querying appended rows through ANN after reopening.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 20, 2026
@Xuanwo
Xuanwo merged commit 40ae95f into lance-format:main Nov 20, 2026
44 of 49 checks passed
Xuanwo pushed a commit that referenced this pull request Nov 20, 2026
Closes #11265

## What changes

- `optimize_options()` raises `ValueError: Invalid optimize_indices
option: <key>` for a key it does not know, as `parse_compaction_options`
does. `DatasetOptimizer.optimize_indices`, `OptimizeIndices.plan` and
`OptimizeIndices.commit` all parse through it.
- `DatasetOptimizer.optimize_indices` takes its six options as
keyword-only parameters, like `compact_files`, so a typo raises
`TypeError` before reaching the binding.
- A known option set to `None` means unset in every entry point. Before,
`retrain=None` and `index_names=None` raised `TypeError` or
`ValueError`.
- The `retrain` docs in Rust, Python and Java say that only vector
indices are retrained. The Rust note "only supported for v3 vector
indices" predates the current rebuild-from-source retrain. A
legacy-format vector index panics on retrain until #11263 lands.

A caller that passed an option name `optimize_indices` does not know now
gets an error instead of a default optimize. No option in the repository
or in lance-ray does.

## Tests

`test_optimize_indices_rejects_unknown_options` passes a misspelled
`index_name` to all four entry points (the Python method, the binding,
`plan`, `commit`) and checks that each raises and nothing is committed.
It then checks that `None` values are accepted by `plan` and by
`optimize_indices`, and that the correctly spelled option optimizes.

🤖 Generated with Claude Opus
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working critical-fix Bugs that cause crashes, security vulnerabilities, or incorrect data. K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: optimize_indices panics on legacy-format IVF indexes and on uint8 IVF_HNSW_FLAT indexes

3 participants