Repository navigation
Conversation
81ad0f9 to
be95d12
Compare
❌ 1 Tests Failed:
View the top 1 failed test(s) by shortest run time
To view more test analytics, go to the Test Analytics Dashboard |
1d12a6e to
db6f582
Compare
7690b0c to
8683af9
Compare
|
It looks like our work crossed in flight: #11260 and #11261 also address #11180, covering vector and FTS merges by retaining per-source overlay freshness. Your PR takes the other approach discussed in #10884: refreshing stale rows against the merge snapshot. It also goes beyond #11180 by bringing the other scalar indices up to date. There is direct overlap in the FTS and shared merge lifecycle code, so I would like to coordinate rather than have us maintain competing FTS fixes. One possible split would be for this PR to own the scalar/FTS refresh, while I narrow #11260 and #11261 to the vector case. #11180 could then remain open until both the vector and FTS parts land. What do you think? We can move the design discussion to #11180 so that everyone involved has the full context. |
Closes #11272. This addresses the full-text search (FTS) part of #11180.
Summary
A scalar index merge can combine segments built before and after the same data overlay. This draft refreshes each source against the snapshot used for the merge. It uses the source’s fragment coverage, indexed fields and build version to decide which entries need replacing. Once the indexed values and row references are current, the output segment can record the merge snapshot’s
dataset_version.optimize_indices, explicit segment merges, and distributed optimize share the overlay refresh logic.fast_searchand indexes updated documents from older inputs. ZoneMap and BloomFilter can rebuild summaries for affected fragments.Validation
cargo fmt --allandgit diff --checkpassed.lance-encoding/src/encodings/physical/byte_stream_split.rs(lines 358 and 393).