Repository navigation
Conversation
|
Decisions I made on my own, and questions for review: Decisions
Open questions
|
88ccf3e to
b7a2eae
Compare
|
Rebased onto current #8646 head (8a5f8fd). Substantive changes:
|
|
Closed both gaps this PR picked up in the #8646 rebase. Format/wire: unchanged.
Commits: |
5bd99b8 to
21b25ac
Compare
21b25ac to
a1a0141
Compare
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
a1a0141 to
f7544d9
Compare
f7544d9 to
f059008
Compare
|
f059008 to
a672b32
Compare
a672b32 to
ed820a8
Compare
ed820a8 to
ceab40c
Compare
Adds footprints for legacy Merge and Project, detects casts, and adapts an append to a field dropped since it read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ceab40c to
594e8bc
Compare
There was a problem hiding this comment.
❌ Gate recommendation: request changes.
All four earlier findings remain fixed. One additional data integrity gap lets a stale cast overwrite a nullable-column change committed by a legacy merge. Keep these merges conflicting until value changes represented by removed backing data are included in the conflict check.
| merged | ||
| .files | ||
| .iter() | ||
| .filter(move |file| !was.files.contains(file)) |
There was a problem hiding this comment.
A Merge may remove a nullable field’s backing while keeping it in the schema; it passes commit validation and the field reads as null (#3041). This helper sees no added file and records no value write, so a stale action cast can commit over it and restore the old values. The dataset test below reads [null, null] after the merge, then [11, 12] after the stale cast. Include removed field bindings and overlay changes in the facts about values written to existing fields, or conservatively conflict with those merges. A separate manifest test also reproduces the same admission when an overlay is removed.
Reproducer
Add this as rust/lance/tests/gate_merge_omission.rs:
use std::sync::Arc;
use arrow_array::{Int64Array, RecordBatch, record_batch};
use arrow_schema::{DataType, Field, Schema};
use lance::dataset::NewColumnTransform;
use lance::dataset::write::{CommitBuilder, InsertBuilder, WriteParams};
use lance_core::utils::tempfile::TempDir;
use lance_file::version::LanceFileVersion;
use lance_table::transaction::action::{
Action, AddDataFile, AddField, CompositeOperation, DropField, FieldRef, Ref, UserStep,
};
use lance_table::transaction::{Operation, Transaction};
#[tokio::test]
async fn a_stale_cast_must_not_restore_values_a_merge_removed() {
let dir = TempDir::default();
let mut read = InsertBuilder::new(&dir.path_str())
.with_params(&WriteParams {
data_storage_version: Some(LanceFileVersion::V2_0),
..Default::default()
})
.execute(vec![record_batch!(("a", Int32, [1, 2])).unwrap()])
.await
.unwrap();
read.add_columns(
NewColumnTransform::SqlExpressions(vec![("c".into(), "a + 10".into())]),
None,
None,
)
.await
.unwrap();
let field_id = read.schema().field("c").unwrap().id;
assert!(read.schema().field("c").unwrap().nullable);
assert_eq!(read.manifest().fragments[0].files.len(), 2);
let read_version = read.version().version;
let staged_dir = TempDir::default();
let cast_batch = RecordBatch::try_new(
Arc::new(Schema::new(vec![Field::new("c", DataType::Int64, true)])),
vec![Arc::new(Int64Array::from(vec![11, 12]))],
)
.unwrap();
let staged = InsertBuilder::new(&staged_dir.path_str())
.with_params(&WriteParams {
data_storage_version: Some(LanceFileVersion::V2_0),
..Default::default()
})
.execute(vec![cast_batch])
.await
.unwrap();
let mut file = staged.manifest().fragments[0].files[0].clone();
std::fs::copy(
staged_dir.std_path().join("data").join(&file.path),
dir.std_path().join("data").join(&file.path),
)
.unwrap();
file.fields = Arc::from([]);
let cast = Transaction::new(
read_version,
Operation::CompositeOperation(CompositeOperation::new(vec![UserStep::new(
"cast",
vec![
Action::DropField(DropField::new(FieldRef::Committed(field_id))),
Action::AddField(AddField::new(0, staged.schema().fields[0].clone())),
Action::AddDataFile(
AddDataFile::try_new_unchecked(
Ref::Committed(read.manifest().fragments[0].id),
file,
vec![FieldRef::Local(0)],
None,
true,
)
.unwrap(),
),
],
)])),
None,
);
let mut fragments = read.manifest().fragments.as_ref().clone();
fragments[0]
.files
.retain(|file| !file.fields.contains(&field_id));
let schema = read.schema().clone();
let latest = CommitBuilder::new(Arc::new(read))
.execute(Transaction::new(
read_version,
Operation::Merge {
fragments,
schema,
preserves_nullability: true,
},
None,
))
.await
.unwrap();
let merged = latest.scan().try_into_batch().await.unwrap();
assert_eq!(
merged["c"].null_count(),
2,
"the merge wrote nulls by omitting nullable c"
);
match CommitBuilder::new(Arc::new(latest))
.with_experimental_composite_operations(true)
.execute(cast)
.await
{
Err(lance_core::Error::RetryableCommitConflict { .. }) => {}
Err(error) => panic!("unexpected error: {error}"),
Ok(landed) => {
let actual = landed.scan().try_into_batch().await.unwrap();
assert_eq!(
actual["c"].null_count(),
2,
"the stale cast restored snapshot values over the merge's nulls: {:?}",
actual["c"]
);
}
}
}Ran cargo test -p lance --test gate_merge_omission against 594e8bc69d85b4d97560047acd3f277f82fea2db. The merge commits and its null assertion passes. The stale cast also commits, and the final assertion fails with PrimitiveArray<Int64> [11, 12], left: 0, right: 2.
No wire-format change. One apply change:
TombstoneFieldDatanow only tombstones overlays committed at or before its read version (plus ones its own set added).Summary
Conflict parity 2/5 (OSS-2598). Schema-evolution action sets (AddField + AddDataFile, TombstoneFieldData + AddDataFile, AlterField rename) resolve at least as permissively as legacy Merge / DataReplacement / Project against every legacy operation, in both directions, except the gaps below. Legacy Merge and Project no longer fail closed against action sets.
Part::Schemaand check the nullability they tighten (Footprint::tightenings). Merge also readsPart::Fragments, restates the fragments it read, and conflicts with rows inserted since. Committed, Merge records the values it wrote (wrote_values) and the fields it replaced by a same-named field with a new id, as a cast does (replaced_fields). Two new parts,Part::SchemaandPart::Fragments, are looked up in theWindowby comparing read and latest.CompositeOperation::forget_dropped_fields(called from the newfinish_action_set) leaves fields dropped since out of the new files and drops a file left binding nothing, as legacyAppendoverProjectdoes. A V1 file is not adapted, since its reader places columns by field position; it still requires every field it backs. A drop paired with an add (a replacement or cast) is recorded asreplaced_fields, and a writer binding that field in a new fragment conflicts (Forbid::FieldReplaced) rather than being adapted, because its values would read back as null.TombstoneFieldDatakeeps an overlay committed after its read version, as legacyDataReplacementdoes. TheAddDataFilethat installs the new values still tombstones every overlay on the column, so an action column replacement and an overlay conflict both ways. That is a parity gap (below).Stand-in removed: writes into minted fragments no longer require their committed fields (now adapted to, or
FieldReplaced).Evidence
conflict_matrix::cases::MATRIXgainsaction_add_column(vn, analogue Merge),action_replace_column(vr, DataReplacement) andaction_rename_column(vm, Project). Theprojectandaction_append/action_*rows now land against each other where the old matrix hadR/R*; legacy-vs-legacy cells are unchanged. Each landed cell runs the invariant battery and oracle.PARITY_GAPS/REWRITE_DROPS_NEWER_OVERLAY:action_replace_columnoverdata_overlay, anddata_overlayover it, isR*where legacyDataReplacementlands besideDataOverlay.oracle::EXEMPTgains theaction_add_columnoverappend/action_append/action_append_and_configcells: rows appended concurrently read null in a nullable new column, which snapshot isolation permits. The existing data-replacement-over-overlay exemption is reworded.add_data_file::tests::test_adapting_to_a_drop_leaves_the_field_out_of_new_files: error without adapting, V1 file left whole and still errors, adapted file drops the field and its column, empty file removed.tombstone_field_data::tests::test_tombstone_field_data_keeps_overlays_committed_after_the_read_version.footprint::testscases: a drop and an append (now lands, also for a child under the dropped struct), a tightening and a merge rewriting it, a projection and an append, a projection and a schema metadata edit, a merge and an append, a merge and an index;test_an_append_over_a_merge_adding_a_nullable_field(struct child conflicts, top-level lands);test_a_write_into_a_new_fragment_over_a_cast(action append, named append, over legacy or action cast conflicts; over a plain drop lands).conflict_resolvertest_a_legacy_operation_against_an_action_txn_by_ordergainsmergeandprojectcases.Merge Risk
Door: two-way. Transaction V2 is experimental and gated behind
with_experimental_composite_operations, and footprints are never persisted. The overlay bound inTombstoneFieldDataapply only matters for action sets.Blast Radius: concurrency (mixed action/legacy pairs;
finish_action_setonly rewrites action sets, legacy-vs-legacy dispatch untouched).🤖 Generated with Claude Code