Skip to content

fix: reject schema-dependent commits across cleaned-up history - #11079

Merged
Xuanwo merged 6 commits into
lance-format:mainfrom
maharanay22:fix-missing-versions-conflicts
Nov 18, 2026
Merged

Xuanwo merged 6 commits into
lance-format:mainfrom
maharanay22:fix-missing-versions-conflicts

Conversation

@maharanay22

Copy link
Copy Markdown
Contributor

When cleanup_old_versions removes versions between a writer's read version and the latest version, the commit path cannot see those transactions. #10608 and #10665 made the rebase refuse that gap for operations that modify existing fragments or insert primary keys, but operations that only depend on the read version's schema still commit blind.

A stale drop_columns (or any other Project) replaces the table schema with one derived from its read version and prunes the data files of every field missing from it. If the gap hides an add_columns, the added column and its files silently leave the table. With intact history check_project_txn returns a retryable conflict instead.

import tempfile
from datetime import timedelta
import lance
import pyarrow as pa

path = tempfile.mkdtemp()
lance.write_dataset(pa.table({"id": pa.array([0, 1, 2, 3], pa.int32()), "x": pa.array([0, 1, 2, 3], pa.int32())}), path)
stale = lance.dataset(path)  # stays on version 1
lance.dataset(path).add_columns({"extra": "id * 2"})  # version 2
lance.write_dataset(pa.table({"id": pa.array([4], pa.int32()), "x": pa.array([4], pa.int32()), "extra": pa.array([8], pa.int32())}), path, mode="append")
lance.dataset(path).cleanup_old_versions(older_than=timedelta(0), delete_unverified=True)
stale.drop_columns(["x"])
print(lance.dataset(path).schema.names)  # ['id'], "extra" is gone

Staged appends and insert-only merge-inserts have the same blind spot. Their files keep the field ids they were written with and their rows were only validated against the read schema. If the gap hides a cast, a staged append of x = 5 reads back as x = null. If it hides an overwrite that reuses field ids, the appended files are read with the wrong types. If it hides alter_columns(nullable=False), a null lands in the non-nullable column and every later scan of the table fails with Found unmasked nulls for non-nullable StructArray field "x". With intact history the Merge, Overwrite and nullability checks in check_txn reject all of these.

This change makes check_missing_versions also return a retryable conflict for a Project across a gap, and for an Append or Update with new fragments when the latest schema no longer has a written field with the type the writer saw, or has a non-nullable field that was nullable or absent in the read version. Appends whose schema did not change keep committing across a gap, as test_safe_commit_survives_cleaned_history requires.

How was this patch tested

test_schema_dependent_commit_rejects_cleaned_history covers a stale drop_columns after add_columns, and staged appends after a cast, after setting a column non-nullable, and after an overwrite that reuses field ids, each with the versions in between cleaned up. All four cases commit on main and return a retryable conflict with this change. The existing io::commit tests still pass.

When cleanup removes versions between a writer's read version and the
latest version, the rebase cannot see their transactions. A Project
built from the read schema then replaces the latest schema and drops any
column a hidden merge added, and staged appends or insert-only
merge-inserts keep field ids and nullability the hidden transaction
changed, so their values read back as null, with the wrong type, or as
nulls in a non-nullable column.

Return a retryable conflict for these operations when the gap could hide
such a change.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions github-actions Bot added the bug Something isn't working label Nov 14, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Nov 14, 2026
An overwrite on a table with reusable field ids can give an id to another
column of the same type. Comparing types alone let a stale append commit
with its columns swapped, so also compare the field name.
@lance-gatekeeper lance-gatekeeper Bot removed K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Nov 14, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Nov 14, 2026
…a gap

Data files record leaf field ids, so an overwrite that swaps two struct
columns of the same shape leaves each written leaf with the same name and
type. Compare the leaf's whole path from the root instead.
@lance-gatekeeper lance-gatekeeper Bot removed K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Nov 14, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Nov 14, 2026
A struct and a list can both reach a leaf through the same names with the
same leaf type, so an overwrite between them kept the path unchanged and a
stale append committed files the new layout cannot read. Compare each
ancestor's logical type as well as its name.
@lance-gatekeeper lance-gatekeeper Bot removed K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Nov 14, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 14, 2026
…conflicts

Signed-off-by: Maha Yadavalli <271375718+maharanay22@users.noreply.github.com>

# Conflicts:
#	rust/lance/src/io/commit.rs
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 16, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 16, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 18, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

The schema guard preserves safe appends and index builds while rejecting incompatible commits across cleaned history.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 18, 2026
@Xuanwo
Xuanwo merged commit aef737d into lance-format:main Nov 18, 2026
44 of 48 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants