Skip to content

fix(scan): stop pushing a limit past AtLeast candidates - #11092

Merged
Xuanwo merged 4 commits into
lance-format:mainfrom
dayan02dev:fix/atleast-limit-pushdown
Nov 19, 2026
Merged

Xuanwo merged 4 commits into
lance-format:mainfrom
dayan02dev:fix/atleast-limit-pushdown

Conversation

@dayan02dev

Copy link
Copy Markdown
Contributor

A filtered scan with limit/offset returns the wrong rows when an OR mixes a ZoneMap or BloomFilter predicate with a negated BTree, Bitmap or LabelList predicate. Found while differential-testing scalar-index scans against full scans.

import lance
import pyarrow as pa

n = 200
tbl = pa.table({
    "id": pa.array(range(n), pa.int32()),
    "x": pa.array([(i * 7) % 11 for i in range(n)], pa.int32()),
    "y": pa.array([i % 3 for i in range(n)], pa.int32()),
})
ds = lance.write_dataset(tbl, "/tmp/repro.lance", max_rows_per_file=60)
ds.create_scalar_index("x", "ZONEMAP")
ds.create_scalar_index("y", "BTREE")
ids = lambda **kw: ds.to_table(columns=["id"], filter="x = 5 OR y != 1", **kw).column("id").to_pylist()
print(ids()[3:8], ids(limit=5, offset=3))

Before (pylance 14.0.2 and 15.0.0b21): [5, 6, 7, 8, 9] [5, 6, 8, 9, 11]. The window skips row 7, which matches only through the zone map. Paging through the 139 matches 10 at a time never returns 6 of them and returns 6 twice. After: on the same data, every page that Dataset::scan returns, at 7 or 10 rows per page, equals the matching slice of the full result, and the pages hold all 139 matches once.

The index result is AtLeast: rows with y != 1 are guaranteed matches, and every other row is a candidate that needs a recheck. FilteredReadStream::apply_index_to_fragment mirrored only the guaranteed ranges into the pushed-down plan. plan_scan switched to that plan, which skips the recheck, as soon as the guaranteed rows filled offset + limit. A matching candidate earlier in scan order was never read, so a later guaranteed row took its place. Now the guaranteed ranges serve the window only up to the first candidate in each fragment. Otherwise the fragment gets no mirror, and the regular plan, which rechecks candidates in scan order, is kept. #10859 added this fallback for fragments the index does not cover; this change applies it to rows outside the lower bound. The push-down is still used when guaranteed rows come before every candidate in the window. Exact and AtMost results, such as x = 5 on any index, plan as before. Datasets with stable row ids are not affected end to end, because the scanner does not push limits into them.

Testing: with cargo test --profile ci -p lance --lib -- io::exec::filtered_read::tests::test_at_least dataset::tests::dataset_scanner_scalar_index::test_not_or_zone_map, 11 of the 53 selected tests fail before the change and all pass after. test_at_least_index_preserves_candidates adds lower bounds that guarantee only later matches. The new test_at_least_limit_pushdown_needs_guaranteed_prefix checks that the plan keeps the push-down when guaranteed rows come first and drops it when candidates come first. test_not_or_zone_map adds limit = 1. The full lance lib suite passes, and cargo fmt and cargo clippy -p lance --tests are clean. An index-vs-full-scan differential over 288,000 limit/offset windows went from 9,654 wrong windows to 0. It covered ZoneMap, BloomFilter, BTree, Bitmap and LabelList indices, deletions, unindexed appends, one or five fragments, and stable row ids on and off. The cost is the fallback read: on 200,000 rows in 20 fragments, x = 5 OR y != 1 with LIMIT 10 now reads 11,808 rows, where it used to read 10 and return the wrong ones.

…idates

A limited read of an AtLeast index result must return the first matches
in scan order, including rows outside the lower bound that only a recheck
confirms. These cases fail: the guaranteed matches that come later are
returned in place of earlier candidates.
A limited scan of an AtLeast index result (some rows guaranteed, the rest
still to recheck) read only the guaranteed rows once they could fill
offset + limit. Matching candidates that come earlier in scan order were
skipped, so paging returned some rows twice and others never.

Serve the range from the guaranteed rows only before the first candidate.
Otherwise the fragment gets no push-down mirror and the regular plan,
which rechecks candidates in scan order, is kept.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions github-actions Bot added the bug Something isn't working label Nov 15, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 15, 2026
@github-actions github-actions Bot added A-python Python bindings A-index Vector index, linalg, tokenizer A-java Java bindings + JNI A-deps Dependency updates A-encoding Encoding, IO, file reader/writer A-docs Documentation A-ci CI / build workflows labels Nov 19, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 19, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 19, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 19, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

No blocking issues found. The guaranteed-prefix fix and Rust dependencies are unchanged by this merge, so the prior verification remains applicable: 53 focused regression cases, all 62 pagination windows of the 139-match dataset, and 13,107,200 range-oracle checks. The fix preserves pagination order and safe limit pushdown.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 19, 2026
@Xuanwo
Xuanwo merged commit d16a575 into lance-format:main Nov 19, 2026
46 of 54 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-ci CI / build workflows A-deps Dependency updates A-docs Documentation A-encoding Encoding, IO, file reader/writer A-index Vector index, linalg, tokenizer A-java Java bindings + JNI A-python Python bindings bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants