Repository navigation
fix(index): prevent float16 k-means centroid overflow - #11285
lance-gatefixer[bot] wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
|
Blocked: Build and Test with Java 17 remains failed because Maven could not resolve the formatter dependency before Java tests ran. Head: The repair changes only two Rust source files, and the Java configuration and workflow match current Please rerun the failed and cancelled jobs in workflow run 44663483083 on this same head. If resolution fails again, invalidate the Maven dependency cache and rerun with working Maven Central access. |
Fixes #8607.
K-means summed cluster members and converted cluster counts in the vector storage type. Float16 sums can overflow even when the mean is representable, and large cluster counts can become infinity. Although
mainalready bounds empty-cluster donor selection, these updates can still produce invalid or zero centroids.Accumulate both update strategies and average in f64 before converting the means back to the original type. Account for the wider accumulators in the existing partial-table memory budget. Keep split perturbations finite near the storage limits, and return a descriptive error when fewer than
ktraining rows can be assigned.Regression coverage includes large float16 values and counts, small values, both summation strategies, split perturbations at float16 limits, and unassignable training data. The IVF regression uses two fragments and checks persisted centroids, finite distances, and recall against a full scan.
Validation:
A direct reproduction using the original summation helper returned infinity for 128 values of 1000; the repaired helper returned 128,000.
cargo test -p lance-index --lib vector::kmeans::tests::: 59 passed; one existing benchmark ignored.cargo test -p lance-index --lib vector::pq::tests::: 8 passed.cargo test -p lance --lib test_create_ivf_flat_f16: 2 passed.cargo fmt --allcargo clippy --all --tests --benches -- -D warningsChange walkthrough
flowchart LR A[Assign training rows] --> B{At least k assigned} B -->|No| C[Return input error] B -->|Yes| D[Sum and average f64] D --> E[Convert centroid type] E --> F[Split with finite perturbations]The accumulator width determines which summation strategy fits the partial-table memory budget.
Read: Assignment guard → Centroid update → IVF regression.
Snapshot: 6fc4091