Repository navigation
fix(scan): stop pushing a limit past AtLeast candidates - #11092
Merged
Merged
Conversation
…idates A limited read of an AtLeast index result must return the first matches in scan order, including rows outside the lower bound that only a recheck confirms. These cases fail: the guaranteed matches that come later are returned in place of earlier candidates.
A limited scan of an AtLeast index result (some rows guaranteed, the rest still to recheck) read only the guaranteed rows once they could fill offset + limit. Matching candidates that come earlier in scan order were skipped, so paging returned some rows twice and others never. Serve the range from the guaranteed rows only before the first candidate. Otherwise the fragment gets no push-down mirror and the regular plan, which rechecks candidates in scan order, is kept.
Contributor
There was a problem hiding this comment.
✅ Gate recommendation: approve.
No blocking issues found. The guaranteed-prefix fix and Rust dependencies are unchanged by this merge, so the prior verification remains applicable: 53 focused regression cases, all 62 pagination windows of the 139-match dataset, and 13,107,200 range-oracle checks. The fix preserves pagination order and safe limit pushdown.
Xuanwo
approved these changes
Nov 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A filtered scan with
limit/offsetreturns the wrong rows when anORmixes a ZoneMap or BloomFilter predicate with a negated BTree, Bitmap or LabelList predicate. Found while differential-testing scalar-index scans against full scans.Before (pylance 14.0.2 and 15.0.0b21):
[5, 6, 7, 8, 9] [5, 6, 8, 9, 11]. The window skips row 7, which matches only through the zone map. Paging through the 139 matches 10 at a time never returns 6 of them and returns 6 twice. After: on the same data, every page thatDataset::scanreturns, at 7 or 10 rows per page, equals the matching slice of the full result, and the pages hold all 139 matches once.The index result is
AtLeast: rows withy != 1are guaranteed matches, and every other row is a candidate that needs a recheck.FilteredReadStream::apply_index_to_fragmentmirrored only the guaranteed ranges into the pushed-down plan.plan_scanswitched to that plan, which skips the recheck, as soon as the guaranteed rows filledoffset + limit. A matching candidate earlier in scan order was never read, so a later guaranteed row took its place. Now the guaranteed ranges serve the window only up to the first candidate in each fragment. Otherwise the fragment gets no mirror, and the regular plan, which rechecks candidates in scan order, is kept. #10859 added this fallback for fragments the index does not cover; this change applies it to rows outside the lower bound. The push-down is still used when guaranteed rows come before every candidate in the window. Exact and AtMost results, such asx = 5on any index, plan as before. Datasets with stable row ids are not affected end to end, because the scanner does not push limits into them.Testing: with
cargo test --profile ci -p lance --lib -- io::exec::filtered_read::tests::test_at_least dataset::tests::dataset_scanner_scalar_index::test_not_or_zone_map, 11 of the 53 selected tests fail before the change and all pass after.test_at_least_index_preserves_candidatesadds lower bounds that guarantee only later matches. The newtest_at_least_limit_pushdown_needs_guaranteed_prefixchecks that the plan keeps the push-down when guaranteed rows come first and drops it when candidates come first.test_not_or_zone_mapaddslimit = 1. The fulllancelib suite passes, andcargo fmtandcargo clippy -p lance --testsare clean. An index-vs-full-scan differential over 288,000 limit/offset windows went from 9,654 wrong windows to 0. It covered ZoneMap, BloomFilter, BTree, Bitmap and LabelList indices, deletions, unindexed appends, one or five fragments, and stable row ids on and off. The cost is the fallback read: on 200,000 rows in 20 fragments,x = 5 OR y != 1withLIMIT 10now reads 11,808 rows, where it used to read 10 and return the wrong ones.