Repository navigation
Conversation
7125dc4 to
0d863ec
Compare
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Drew <geruh@users.noreply.github.com>
Co-authored-by: Drew <geruh@users.noreply.github.com>
0d863ec to
fa9eb26
Compare
❌ 1 Tests Failed:
View the full list of 1 ❄️ flaky test(s)
To view more test analytics, go to the Test Analytics Dashboard |
Appends go through the existing insert writer and land as AddFragment and AddDataFile on the reusable ApplyState. A scan pins publication to the base version. Appends that were not scanned can rebase. The uncommitted snapshot uses u64::MAX so commit and maintenance reject it. Co-authored-by: Drew <geruh@users.noreply.github.com>
fa9eb26 to
e01d601
Compare
| /// Staged snapshots use `u64::MAX` instead. It sits at the top of the detached | ||
| /// range, and detached commits draw again when they land on it. A transaction | ||
| /// or manifest copied from the snapshot keeps this version, so commit rejects it. | ||
| pub const STAGED_VERSION: u64 = u64::MAX; |
There was a problem hiding this comment.
used u64::MAX instead of base_version + 1 so a concurrent commit at base_version + 1 doesn't collide with an open staged scan. detached_version redraws if it ever lands on it.
| }; | ||
| let mut dataset = self.clone(); | ||
| dataset.set_manifest(Arc::new(manifest), location); | ||
| dataset.staged_indices = Some(Arc::new(indices)); |
There was a problem hiding this comment.
reusing Dataset for the staged snapshot so tx.scan() gets the existing query and index path for free. the ensure_published() guards are the price of doing that for now, can split a smaller reader snapshot trait out later.
| } | ||
|
|
||
| fn append_dataset(&mut self) -> Result<Arc<Dataset>> { | ||
| if !self.state.append_context_dirty { |
There was a problem hiding this comment.
blind appends skip building a manifest and reuse append_base. only time the schema changes mid transaction is when a 2.2+ file promotes a legacy blob column, so we rebuild once there and clear the dirty flag.
|
|
||
| // Batch merge takes the minimum read_version, which would mask a | ||
| // STAGED_VERSION transaction mixed with committed ones. | ||
| transactions |
There was a problem hiding this comment.
InsertBuilder::execute_uncommitted needs to run on append_base (which can be a STAGED_VERSION snapshot after scan() or blob promotion), so we block STAGED_VERSION at commit time instead. checking before min(read_version) in execute_batch so it can't hide behind an older committed txn.
| // would remove. | ||
| let commit = CommitBuilder::new(self.base.clone()) | ||
| .with_experimental_composite_operations(true) | ||
| .with_skip_auto_cleanup(true); |
There was a problem hiding this comment.
skipping auto cleanup on staged commits so earlier scanners don't lose files under them. uncommitted files aren't pinned against explicit cleanup though.
|
Important Format specification voteThis PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer), at least one of them on the latest commit, and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process. Approvals carry over across pushes, so a rebase or a typo fix does not send everyone back to re-vote. Whoever approves the latest commit is vouching that nothing substantive has changed since the earlier approvals; if something has, ask for fresh votes. Status: ❌ Blocked — 0 of 3 required approvals
Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the |
There was a problem hiding this comment.
The documented cleanup limitation means cleanup must be coordinated with active transactions and scanners; it can otherwise leave a later commit referencing missing files or borrowed blob objects. Staged commits also skip automatic cleanup, so workloads using only this API need scheduled explicit cleanup.
Please mark this PR with the breaking-change label: fields of the public lance_table::transaction::action structs are now private and AddOverlays::data_change was removed, so Rust callers must use constructors and accessors.
Summary
Adds an experimental staged transaction API using Transaction V2.
This adds
Dataset::begin_transaction()andStagedTransactionfor staging multiple appends into a single atomic dataset version. Eachappend/append_streamwrites uncommitted data files viaInsertBuilder::execute_uncommitted*and applies the resultingAddFragment/AddDataFileactions to an in-memoryApplyState. Failed or rejected appends leave the previous working state intact, andabort(self)consumes the handle without deleting files so any already-created scanners can finish reading.Blind appends never build an intermediate manifest snapshot and can rebase over concurrent commits at
commit(). Callingscan()lazily builds an uncommitted snapshot tagged withSTAGED_VERSION(u64::MAX), holds its ownArc<Dataset>so earlier scanners survive subsequent appends, and pins publication tobase_versionso row IDs and addresses stay validStacked on #11203.
Example
How staging and publication work
Implementation
InsertBuilder::execute_uncommittedto write files without committing.CommitBuilderto publish the accumulated actions as one version.Failed statements leave the previous snapshot intact. Abort is logical, with uncommitted files left for normal cleanup.
Current scope
cleanup_old_versions/delete_unverifiedcalls (files older than 7 days can still be swept). Staged commits skip auto-cleanup so outstanding scanners keep working.Testing
git diff --check feat/reusable-apply-state..HEADcargo fmt --all -- --checkCARGO_INCREMENTAL=0 cargo clippy -p lance --tests --benches --no-deps -- -D warningsCARGO_INCREMENTAL=0 cargo test -p lance --lib dataset::stagedCARGO_INCREMENTAL=0 cargo test -p lance --lib dataset::tests::dataset_blob::test_stagedCARGO_INCREMENTAL=0 cargo test -p lance --lib dataset::tests::dataset_transactions::test_stagedCARGO_INCREMENTAL=0 cargo test -p lance --lib io::exec::table_identifier::testsCARGO_INCREMENTAL=0 cargo test -p lance --lib io::commit::tests::test_detached_commit_draws_again_on_the_staged_versionCARGO_INCREMENTAL=0 cargo test -p lance --doc dataset::staged