Skip to content

feat(transaction): resolve schema-evolution action sets vs legacy ops - #10474

Open
wjones127 wants to merge 1 commit into
will/oss-2597-conflict-parity-append-configfrom
will/oss-2598-conflict-parity-schema-evolution
Open

wjones127 wants to merge 1 commit into
will/oss-2597-conflict-parity-append-configfrom
will/oss-2598-conflict-parity-schema-evolution

Conversation

@wjones127

@wjones127 wjones127 commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

No wire-format change. One apply change: TombstoneFieldData now only tombstones overlays committed at or before its read version (plus ones its own set added).

Summary

Conflict parity 2/5 (OSS-2598). Schema-evolution action sets (AddField + AddDataFile, TombstoneFieldData + AddDataFile, AlterField rename) resolve at least as permissively as legacy Merge / DataReplacement / Project against every legacy operation, in both directions, except the gaps below. Legacy Merge and Project no longer fail closed against action sets.

  • Pending legacy Merge/Project get footprints. Both read Part::Schema and check the nullability they tighten (Footprint::tightenings). Merge also reads Part::Fragments, restates the fragments it read, and conflicts with rows inserted since. Committed, Merge records the values it wrote (wrote_values) and the fields it replaced by a same-named field with a new id, as a cast does (replaced_fields). Two new parts, Part::Schema and Part::Fragments, are looked up in the Window by comparing read and latest.
  • A drop landing before an append is adapted to, not a conflict. An action set that mints fragments no longer requires the committed fields its new files back. After the check, CompositeOperation::forget_dropped_fields (called from the new finish_action_set) leaves fields dropped since out of the new files and drops a file left binding nothing, as legacy Append over Project does. A V1 file is not adapted, since its reader places columns by field position; it still requires every field it backs. A drop paired with an add (a replacement or cast) is recorded as replaced_fields, and a writer binding that field in a new fragment conflicts (Forbid::FieldReplaced) rather than being adapted, because its values would read back as null.
  • A nullable child added under an existing struct conflicts with a stale append, whose packed rows would not decode. A nullable top-level column still lands.
  • Column rewrite vs overlay: TombstoneFieldData keeps an overlay committed after its read version, as legacy DataReplacement does. The AddDataFile that installs the new values still tombstones every overlay on the column, so an action column replacement and an overlay conflict both ways. That is a parity gap (below).

Stand-in removed: writes into minted fragments no longer require their committed fields (now adapted to, or FieldReplaced).

Evidence

  • conflict_matrix::cases::MATRIX gains action_add_column (vn, analogue Merge), action_replace_column (vr, DataReplacement) and action_rename_column (vm, Project). The project and action_append/action_* rows now land against each other where the old matrix had R/R*; legacy-vs-legacy cells are unchanged. Each landed cell runs the invariant battery and oracle.
  • Known gap, PARITY_GAPS / REWRITE_DROPS_NEWER_OVERLAY: action_replace_column over data_overlay, and data_overlay over it, is R* where legacy DataReplacement lands beside DataOverlay.
  • oracle::EXEMPT gains the action_add_column over append / action_append / action_append_and_config cells: rows appended concurrently read null in a nullable new column, which snapshot isolation permits. The existing data-replacement-over-overlay exemption is reworded.
  • add_data_file::tests::test_adapting_to_a_drop_leaves_the_field_out_of_new_files: error without adapting, V1 file left whole and still errors, adapted file drops the field and its column, empty file removed.
  • tombstone_field_data::tests::test_tombstone_field_data_keeps_overlays_committed_after_the_read_version.
  • footprint::tests cases: a drop and an append (now lands, also for a child under the dropped struct), a tightening and a merge rewriting it, a projection and an append, a projection and a schema metadata edit, a merge and an append, a merge and an index; test_an_append_over_a_merge_adding_a_nullable_field (struct child conflicts, top-level lands); test_a_write_into_a_new_fragment_over_a_cast (action append, named append, over legacy or action cast conflicts; over a plain drop lands).
  • conflict_resolver test_a_legacy_operation_against_an_action_txn_by_order gains merge and project cases.

Merge Risk

Door: two-way. Transaction V2 is experimental and gated behind with_experimental_composite_operations, and footprints are never persisted. The overlay bound in TombstoneFieldData apply only matters for action sets.
Blast Radius: concurrency (mixed action/legacy pairs; finish_action_set only rewrites action sets, legacy-vs-legacy dispatch untouched).

🤖 Generated with Claude Code

@wjones127

Copy link
Copy Markdown
Contributor Author

Decisions I made on my own, and questions for review:

Decisions

  • Four facets, not two. The ticket says to split the definition into type vs name/nullability. But appends must require nullability (a tightening stops a stale append) and must not require the name (a rename lands beside an append, as legacy Append beside Project does). So the split is Existence / Type / Nullability / Name.
  • A plain drop only writes Existence. As a result, an action append landing after an action DropField now lands, leaving the dropped field's column unread in the new fragment's file. This matches legacy Append over Project, and keeps action-vs-action consistent with action-vs-legacy. A drop+add replacement still writes Type. I changed the existing an_append_landing_after_a_drop case to expect this. dropping_a_field_collides_with_altering_it is now directional: an alter after a drop conflicts, a drop after an alter lands.
  • Overlays get their own coordinate (FieldOverlays) instead of WritesPart on FieldData. A column rewrite and an overlay now commute. This rests on the read-version bound I added to TombstoneFieldData apply, which also tombstones overlays its own set added.
  • FragmentFiles is mostly a committed-side coordinate. Pending Update/Delete/Merge keep Region::Fragment, which already contains it; I found no parity cell where that coarseness hurt. On the committed side, an Update that moves rows (it has new fragments) writes FragmentFiles of the fragments it moved rows out of. Without that, a column added or replaced concurrently would land and miss the moved rows (the oracle caught this).
  • Pending legacy DataOverlay now has the AddOverlays footprint, not Region::Fragment. It appends to the fragment as it is at commit. This closes data_overlay over action_replace_column.
  • Diff "at least as strong as direct". I read this as covering the operation's own named delta: coordinates, removals, value writes, rows, keys, and indices. Post-image regions are excluded, since they are what makes the pending side stricter.
  • Kept one gap for soundness. project over a schema-metadata action edit is R where legacy is L, because legacy Project silently discards the edit. Reason: SCHEMA_METADATA_REVERTED.
  • One PR. About 650 non-test lines.

Open questions

  • Is the metadata gap acceptable, or do you want parity there, which means losing the edit as legacy does?
  • Should an action AddDataFile apply tombstone columns for committed fields that no longer exist, so that append-after-drop doesn't leave dead columns? I left apply as is to match legacy.
  • Add-column over a concurrent append lands, and the appended rows read null. I added this to oracle EXEMPT as allowed under snapshot isolation. Legacy is R here. OK?

@github-actions github-actions Bot added the enhancement New feature or request label Oct 30, 2026
@wjones127
wjones127 force-pushed the will/oss-2598-conflict-parity-schema-evolution branch from 88ccf3e to b7a2eae Compare October 30, 2026 05:58
@wjones127

Copy link
Copy Markdown
Contributor Author

Rebased onto current #8646 head (8a5f8fd). Substantive changes:

@wjones127

Copy link
Copy Markdown
Contributor Author

Closed both gaps this PR picked up in the #8646 rebase. Format/wire: unchanged.

  • DROPPED_FIELD_REJECTED: an AddDataFile in a fragment the set mints no longer requires the existence of the committed fields it binds. A drop that landed first is now Overlap::FieldDropped { field }. The rebase removes the field from that file's field_ids and column_indices, and drops a file that binds nothing else, so the column stays in the file unread. That is what legacy Append over Project leaves. feat(transaction): implement the remaining drafted actions #8646's apply check is unchanged: a field the schema doesn't have is still rejected. Data for a dropped field in a committed fragment still conflicts.
  • OVERLAY_RESTAMP_CLAIMED: the diff of a committed legacy op no longer counts a last-updated restamp as a write of the row versions when the fragment gained overlays. This matches AddOverlays, which restamps without claiming.
  • Cells flipped: va/v+ over pj/pa/mg/ma R() → L, vr over ov R → L, vn over ov R → L. pa/ma/vn are beyond the bar, as they were before the rebase.

Commits: 993e03f18 (test), 4fa1d86d7 (fix), 8a89af12f (test), 5bd99b8f3 (fix).

@wjones127
wjones127 force-pushed the will/oss-2598-conflict-parity-schema-evolution branch from 5bd99b8 to 21b25ac Compare November 4, 2026 00:45
@wjones127
wjones127 force-pushed the will/oss-2598-conflict-parity-schema-evolution branch from 21b25ac to a1a0141 Compare November 12, 2026 23:14
@wjones127
wjones127 added this pull request to stack #8647 November 12, 2026 23:17
@wjones127
wjones127 marked this pull request as ready for review November 16, 2026 22:29

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Nov 16, 2026
@wjones127
wjones127 force-pushed the will/oss-2598-conflict-parity-schema-evolution branch from a1a0141 to f7544d9 Compare November 16, 2026 23:26
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Nov 16, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Nov 16, 2026
@wjones127
wjones127 force-pushed the will/oss-2598-conflict-parity-schema-evolution branch from f7544d9 to f059008 Compare November 17, 2026 19:25
@codecov

codecov Bot commented Nov 17, 2026

Copy link
Copy Markdown

⚠️ JUnit XML file not found

The CLI was unable to find any JUnit XML files to upload.
For more help, visit our troubleshooting guide.

@wjones127
wjones127 force-pushed the will/oss-2598-conflict-parity-schema-evolution branch from f059008 to a672b32 Compare November 17, 2026 20:08
@wjones127
wjones127 force-pushed the will/oss-2598-conflict-parity-schema-evolution branch from a672b32 to ed820a8 Compare November 17, 2026 22:49
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Nov 17, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 17, 2026
@wjones127
wjones127 force-pushed the will/oss-2598-conflict-parity-schema-evolution branch from ed820a8 to ceab40c Compare November 18, 2026 01:46
Adds footprints for legacy Merge and Project, detects casts, and adapts an append to a field dropped since it read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wjones127
wjones127 force-pushed the will/oss-2598-conflict-parity-schema-evolution branch from ceab40c to 594e8bc Compare November 19, 2026 01:58
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 19, 2026
@wjones127
wjones127 removed this pull request from stack #8647 November 19, 2026 02:01
@wjones127
wjones127 added this pull request to stack #11242 November 19, 2026 02:02

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❌ Gate recommendation: request changes.

All four earlier findings remain fixed. One additional data integrity gap lets a stale cast overwrite a nullable-column change committed by a legacy merge. Keep these merges conflicting until value changes represented by removed backing data are included in the conflict check.

merged
.files
.iter()
.filter(move |file| !was.files.contains(file))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A Merge may remove a nullable field’s backing while keeping it in the schema; it passes commit validation and the field reads as null (#3041). This helper sees no added file and records no value write, so a stale action cast can commit over it and restore the old values. The dataset test below reads [null, null] after the merge, then [11, 12] after the stale cast. Include removed field bindings and overlay changes in the facts about values written to existing fields, or conservatively conflict with those merges. A separate manifest test also reproduces the same admission when an overlay is removed.

Reproducer

Add this as rust/lance/tests/gate_merge_omission.rs:

use std::sync::Arc;

use arrow_array::{Int64Array, RecordBatch, record_batch};
use arrow_schema::{DataType, Field, Schema};
use lance::dataset::NewColumnTransform;
use lance::dataset::write::{CommitBuilder, InsertBuilder, WriteParams};
use lance_core::utils::tempfile::TempDir;
use lance_file::version::LanceFileVersion;
use lance_table::transaction::action::{
    Action, AddDataFile, AddField, CompositeOperation, DropField, FieldRef, Ref, UserStep,
};
use lance_table::transaction::{Operation, Transaction};

#[tokio::test]
async fn a_stale_cast_must_not_restore_values_a_merge_removed() {
    let dir = TempDir::default();
    let mut read = InsertBuilder::new(&dir.path_str())
        .with_params(&WriteParams {
            data_storage_version: Some(LanceFileVersion::V2_0),
            ..Default::default()
        })
        .execute(vec![record_batch!(("a", Int32, [1, 2])).unwrap()])
        .await
        .unwrap();
    read.add_columns(
        NewColumnTransform::SqlExpressions(vec![("c".into(), "a + 10".into())]),
        None,
        None,
    )
    .await
    .unwrap();
    let field_id = read.schema().field("c").unwrap().id;
    assert!(read.schema().field("c").unwrap().nullable);
    assert_eq!(read.manifest().fragments[0].files.len(), 2);
    let read_version = read.version().version;

    let staged_dir = TempDir::default();
    let cast_batch = RecordBatch::try_new(
        Arc::new(Schema::new(vec![Field::new("c", DataType::Int64, true)])),
        vec![Arc::new(Int64Array::from(vec![11, 12]))],
    )
    .unwrap();
    let staged = InsertBuilder::new(&staged_dir.path_str())
        .with_params(&WriteParams {
            data_storage_version: Some(LanceFileVersion::V2_0),
            ..Default::default()
        })
        .execute(vec![cast_batch])
        .await
        .unwrap();
    let mut file = staged.manifest().fragments[0].files[0].clone();
    std::fs::copy(
        staged_dir.std_path().join("data").join(&file.path),
        dir.std_path().join("data").join(&file.path),
    )
    .unwrap();
    file.fields = Arc::from([]);
    let cast = Transaction::new(
        read_version,
        Operation::CompositeOperation(CompositeOperation::new(vec![UserStep::new(
            "cast",
            vec![
                Action::DropField(DropField::new(FieldRef::Committed(field_id))),
                Action::AddField(AddField::new(0, staged.schema().fields[0].clone())),
                Action::AddDataFile(
                    AddDataFile::try_new_unchecked(
                        Ref::Committed(read.manifest().fragments[0].id),
                        file,
                        vec![FieldRef::Local(0)],
                        None,
                        true,
                    )
                    .unwrap(),
                ),
            ],
        )])),
        None,
    );
    let mut fragments = read.manifest().fragments.as_ref().clone();
    fragments[0]
        .files
        .retain(|file| !file.fields.contains(&field_id));
    let schema = read.schema().clone();
    let latest = CommitBuilder::new(Arc::new(read))
        .execute(Transaction::new(
            read_version,
            Operation::Merge {
                fragments,
                schema,
                preserves_nullability: true,
            },
            None,
        ))
        .await
        .unwrap();
    let merged = latest.scan().try_into_batch().await.unwrap();
    assert_eq!(
        merged["c"].null_count(),
        2,
        "the merge wrote nulls by omitting nullable c"
    );
    match CommitBuilder::new(Arc::new(latest))
        .with_experimental_composite_operations(true)
        .execute(cast)
        .await
    {
        Err(lance_core::Error::RetryableCommitConflict { .. }) => {}
        Err(error) => panic!("unexpected error: {error}"),
        Ok(landed) => {
            let actual = landed.scan().try_into_batch().await.unwrap();
            assert_eq!(
                actual["c"].null_count(),
                2,
                "the stale cast restored snapshot values over the merge's nulls: {:?}",
                actual["c"]
            );
        }
    }
}

Ran cargo test -p lance --test gate_merge_omission against 594e8bc69d85b4d97560047acd3f277f82fea2db. The merge commits and its null assertion passes. The stale cast also commits, and the final assertion fails with PrimitiveArray<Int64> [11, 12], left: 0, right: 2.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking-change enhancement New feature or request K-changes Latest Gatekeeper recommendation requests changes.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant