Skip to content

fix: record the actual FTS posting format after index maintenance - #11284

Open
ddupg wants to merge 1 commit into
lance-format:mainfrom
ddupg:fix/fts-maintenance-posting-format
Open

ddupg wants to merge 1 commit into
lance-format:mainfrom
ddupg:fix/fts-maintenance-posting-format

Conversation

@ddupg

@ddupg ddupg commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Split out from #11273.

What problem does this solve?

FTS merge, update, and remap can write one posting format to disk and report another in the dataset manifest. For example, merging a V3 text index with 128-document blocks preserves V3 in the files but reports V2. The same mismatch can make a V1 segment pass the query planner’s V2/V3 check for cross-column FTS.

What changed?

Build the returned index details from the writer’s resolved posting format. Previously, maintenance reconstructed the format from stored tokenizer parameters, which omit the explicit version and default to V2 for text with 128-document blocks.

How was this tested?

Regression tests verify that merge, update, and remap report the posting format read back from the output files.

All nine targeted tests passed. Workspace Clippy stopped on two existing unreachable expressions in lance-encoding/src/encodings/physical/byte_stream_split.rs on macOS arm64.

@github-actions github-actions Bot added A-index Vector index, linalg, tokenizer bug Something isn't working labels Nov 20, 2026
@ddupg
ddupg marked this pull request as ready for review November 20, 2026 02:32

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

Using the writer’s resolved posting format fixes the metadata mismatch across merge, update, and remap while preserving the separate index-version contract.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-index Vector index, linalg, tokenizer bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant