Repository navigation
fix: reject schema-dependent commits across cleaned-up history - #11079
Merged
Xuanwo merged 6 commits intoNov 18, 2026
Merged
Conversation
When cleanup removes versions between a writer's read version and the latest version, the rebase cannot see their transactions. A Project built from the read schema then replaces the latest schema and drops any column a hidden merge added, and staged appends or insert-only merge-inserts keep field ids and nullability the hidden transaction changed, so their values read back as null, with the wrong type, or as nulls in a non-nullable column. Return a retryable conflict for these operations when the gap could hide such a change.
An overwrite on a table with reusable field ids can give an id to another column of the same type. Comparing types alone let a stale append commit with its columns swapped, so also compare the field name.
…a gap Data files record leaf field ids, so an overwrite that swaps two struct columns of the same shape leaves each written leaf with the same name and type. Compare the leaf's whole path from the root instead.
A struct and a list can both reach a leaf through the same names with the same leaf type, so an overwrite between them kept the path unchanged and a stale append committed files the new layout cannot read. Compare each ancestor's logical type as well as its name.
…conflicts Signed-off-by: Maha Yadavalli <271375718+maharanay22@users.noreply.github.com> # Conflicts: # rust/lance/src/io/commit.rs
Xuanwo
approved these changes
Nov 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When
cleanup_old_versionsremoves versions between a writer's read version and the latest version, the commit path cannot see those transactions. #10608 and #10665 made the rebase refuse that gap for operations that modify existing fragments or insert primary keys, but operations that only depend on the read version's schema still commit blind.A stale
drop_columns(or any other Project) replaces the table schema with one derived from its read version and prunes the data files of every field missing from it. If the gap hides anadd_columns, the added column and its files silently leave the table. With intact historycheck_project_txnreturns a retryable conflict instead.Staged appends and insert-only merge-inserts have the same blind spot. Their files keep the field ids they were written with and their rows were only validated against the read schema. If the gap hides a cast, a staged append of
x = 5reads back asx = null. If it hides an overwrite that reuses field ids, the appended files are read with the wrong types. If it hidesalter_columns(nullable=False), a null lands in the non-nullable column and every later scan of the table fails withFound unmasked nulls for non-nullable StructArray field "x". With intact history the Merge, Overwrite and nullability checks incheck_txnreject all of these.This change makes
check_missing_versionsalso return a retryable conflict for a Project across a gap, and for an Append or Update with new fragments when the latest schema no longer has a written field with the type the writer saw, or has a non-nullable field that was nullable or absent in the read version. Appends whose schema did not change keep committing across a gap, astest_safe_commit_survives_cleaned_historyrequires.How was this patch tested
test_schema_dependent_commit_rejects_cleaned_historycovers a staledrop_columnsafteradd_columns, and staged appends after a cast, after setting a column non-nullable, and after an overwrite that reuses field ids, each with the versions in between cleaned up. All four cases commit on main and return a retryable conflict with this change. The existingio::committests still pass.