Skip to content

fix(index): prevent float16 k-means centroid overflow - #11285

Open
lance-gatefixer[bot] wants to merge 1 commit into
mainfrom
gatekeeper/fix-8607-1
Open

lance-gatefixer[bot] wants to merge 1 commit into
mainfrom
gatekeeper/fix-8607-1

Conversation

@lance-gatefixer

@lance-gatefixer lance-gatefixer Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #8607.

K-means summed cluster members and converted cluster counts in the vector storage type. Float16 sums can overflow even when the mean is representable, and large cluster counts can become infinity. Although main already bounds empty-cluster donor selection, these updates can still produce invalid or zero centroids.

Accumulate both update strategies and average in f64 before converting the means back to the original type. Account for the wider accumulators in the existing partial-table memory budget. Keep split perturbations finite near the storage limits, and return a descriptive error when fewer than k training rows can be assigned.

Regression coverage includes large float16 values and counts, small values, both summation strategies, split perturbations at float16 limits, and unassignable training data. The IVF regression uses two fragments and checks persisted centroids, finite distances, and recall against a full scan.

Validation:

A direct reproduction using the original summation helper returned infinity for 128 values of 1000; the repaired helper returned 128,000.

  • cargo test -p lance-index --lib vector::kmeans::tests::: 59 passed; one existing benchmark ignored.
  • cargo test -p lance-index --lib vector::pq::tests::: 8 passed.
  • cargo test -p lance --lib test_create_ivf_flat_f16: 2 passed.
  • cargo fmt --all
  • cargo clippy --all --tests --benches -- -D warnings

Change walkthrough

flowchart LR
    A[Assign training rows] --> B{At least k assigned}
    B -->|No| C[Return input error]
    B -->|Yes| D[Sum and average f64]
    D --> E[Convert centroid type]
    E --> F[Split with finite perturbations]
Loading

The accumulator width determines which summation strategy fits the partial-table memory budget.

Read: Assignment guard → Centroid update → IVF regression.

Snapshot: 6fc4091

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

@github-actions github-actions Bot added A-index Vector index, linalg, tokenizer bug Something isn't working labels Nov 20, 2026
@lance-gatefixer

Copy link
Copy Markdown
Contributor Author

Blocked: Build and Test with Java 17 remains failed because Maven could not resolve the formatter dependency before Java tests ran.

Head: 6fc409110a365cab29cf6d91362f608d5304e895. The failed Java 17 job exits during mvn spotless:check: com.google.googlejavaformat:google-java-format:jar:1.22.0 could not be found in Maven Central. Java tests were skipped; the other Java matrix jobs were cancelled.

The repair changes only two Rust source files, and the Java configuration and workflow match current main. The native JNI build/tests and Rust Clippy/format checks passed; Java 11 also passed Spotless in this run. Direct checks now return HTTP 200 for both the formatter POM and JAR. This supports a transient dependency-resolution failure unrelated to the repair. No code change or base merge is warranted.

Please rerun the failed and cancelled jobs in workflow run 44663483083 on this same head. If resolution fails again, invalidate the Maven dependency cache and rerun with working Maven Central access.

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

The change addresses Float16 sum and count overflow during centroid updates while preserving the stored centroid type. Local k-means, PQ, and two-fragment IVF regression tests passed.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-index Vector index, linalg, tokenizer bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: creating an IVF index on a float16 vector column hangs forever when values are large

0 participants