Skip to content

feat(dataset): add append-only staged transactions - #10722

Open
geruh wants to merge 4 commits into
lance-format:mainfrom
geruh:mutable-dataset
Open

geruh wants to merge 4 commits into
lance-format:mainfrom
geruh:mutable-dataset

Conversation

@geruh

@geruh geruh commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds an experimental staged transaction API using Transaction V2.

This adds Dataset::begin_transaction() and StagedTransaction for staging multiple appends into a single atomic dataset version. Each append / append_stream writes uncommitted data files via InsertBuilder::execute_uncommitted* and applies the resulting AddFragment / AddDataFile actions to an in-memory ApplyState. Failed or rejected appends leave the previous working state intact, and abort(self) consumes the handle without deleting files so any already-created scanners can finish reading.

Blind appends never build an intermediate manifest snapshot and can rebase over concurrent commits at commit(). Calling scan() lazily builds an uncommitted snapshot tagged with STAGED_VERSION (u64::MAX), holds its own Arc<Dataset> so earlier scanners survive subsequent appends, and pins publication to base_version so row IDs and addresses stay valid

Stacked on #11203.

Example

let mut tx = dataset.begin_transaction().await?;

tx.append(batch1).await?;
tx.append(batch2).await?;

// Reads include both appends, but nothing has been committed yet.
let rows = tx.scan()?.try_into_batch().await?;

let committed = tx.commit().await?;

How staging and publication work

flowchart LR
  B["Dataset::begin_transaction()"] -->|"ApplyState (at base_version)"| A["append / append_stream"]
  A -->|"blind append (no snapshot build)"| A
  A -->|"2.2+ file promotes legacy blob"| R["Rebuild append_base on next append"]
  R --> A
  A -->|"scan()"| S["Build STAGED_VERSION snapshot + pin base"]
  S -->|"independent Arc<Dataset>"| A
  A -->|"commit() (unpinned)"| C1["CompositeOperation (rebases on conflict)"]
  S -->|"commit() (pinned)"| C2["CompositeOperation (expected_latest_version = base)"]
Loading

Implementation

  • Reuses InsertBuilder::execute_uncommitted to write files without committing.
  • Converts the prepared fragments into V2 actions and keeps them in the transaction.
  • Applies those actions to an unpublished manifest so subsequent statements and scans see earlier writes.
  • Uses the existing CommitBuilder to publish the accumulated actions as one version.
  • Keeps earlier scanners bound to their original staged snapshots.

Failed statements leave the previous snapshot intact. Abort is logical, with uncommitted files left for normal cleanup.

Current scope

  • Append-only: Deletes, updates, schema evolution, and index mutations are left for follow-up PRs.
  • Unpinned staged files: Staged files and borrowed managed blobs are not pinned against explicit cleanup_old_versions / delete_unverified calls (files older than 7 days can still be swept). Staged commits skip auto-cleanup so outstanding scanners keep working.

Testing

  • git diff --check feat/reusable-apply-state..HEAD
  • cargo fmt --all -- --check
  • CARGO_INCREMENTAL=0 cargo clippy -p lance --tests --benches --no-deps -- -D warnings
  • CARGO_INCREMENTAL=0 cargo test -p lance --lib dataset::staged
  • CARGO_INCREMENTAL=0 cargo test -p lance --lib dataset::tests::dataset_blob::test_staged
  • CARGO_INCREMENTAL=0 cargo test -p lance --lib dataset::tests::dataset_transactions::test_staged
  • CARGO_INCREMENTAL=0 cargo test -p lance --lib io::exec::table_identifier::tests
  • CARGO_INCREMENTAL=0 cargo test -p lance --lib io::commit::tests::test_detached_commit_draws_again_on_the_staged_version
  • CARGO_INCREMENTAL=0 cargo test -p lance --doc dataset::staged

@github-actions github-actions Bot added A-java Java bindings + JNI A-format On-disk format: protos and format spec docs A-ci CI / build workflows format-change A change to the format spec, which requires a vote. Remove if minor (e.g. fixing typo). enhancement New feature or request labels Nov 4, 2026
wjones127 and others added 2 commits November 17, 2026 17:46
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Drew <geruh@users.noreply.github.com>
Co-authored-by: Drew <geruh@users.noreply.github.com>
@codecov

codecov Bot commented Nov 18, 2026

Copy link
Copy Markdown

❌ 1 Tests Failed:

Tests completed Failed Passed Skipped
21322 1 21321 0
View the full list of 1 ❄️ flaky test(s)
lance-io::scheduler::tests::test_object_store_selects_scheduler

Flake rate in main: 100.00% (Passed 0 times, Failed 2 times)

Stack Traces | 0.143s run time
thread 'scheduler::tests::test_object_store_selects_scheduler' (125251) panicked at .../lance-io/src/scheduler.rs:3227:9:
assertion failed: uring_store.prefers_lite_scheduler()
stack backtrace:
   0: __rustc::rust_begin_unwind
             at .../std/src/panicking.rs:679:5
   1: core::panicking::panic_fmt
             at .../core/src/panicking.rs:80:14
   2: core::panicking::panic
             at .../core/src/panicking.rs:150:5
   3: {async_block#0}
             at ./src/scheduler.rs:3227:9
   4: poll<&mut dyn core::future::future::Future<Output=()>>
             at .../src/future/future.rs:133:9
   5: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:830:70
   6: with_budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../task/coop/mod.rs:167:5
   7: budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../task/coop/mod.rs:133:5
   8: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:830:25
   9: <tokio::runtime::scheduler::current_thread::Context>::enter::<core::task::poll::Poll<()>, <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}::{closure#0}>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:486:19
  10: <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on::<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:829:44
  11: <tokio::runtime::scheduler::current_thread::CoreGuard>::enter::<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:906:68
  12: <tokio::runtime::context::scoped::Scoped<tokio::runtime::scheduler::Context>>::set::<<tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>)>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../runtime/context/scoped.rs:40:9
  13: tokio::runtime::context::set_scheduler::<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}>::{closure#0}
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../src/runtime/context.rs:186:38
  14: try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>), tokio::runtime::scheduler::current_thread::{impl#9}::enter::{closure_env#0}<tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure_env#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>, core::option::Option<()>>>, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>)>
             at .../src/thread/local.rs:463:12
  15: <std::thread::local::LocalKey<tokio::runtime::context::Context>>::with::<tokio::runtime::context::set_scheduler<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}>::{closure#0}, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>)>
             at .../src/thread/local.rs:427:20
  16: tokio::runtime::context::set_scheduler::<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../src/runtime/context.rs:186:17
  17: <tokio::runtime::scheduler::current_thread::CoreGuard>::enter::<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:906:27
  18: <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on::<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:819:24
  19: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:216:33
  20: tokio::runtime::context::runtime::enter_runtime::<<tokio::runtime::scheduler::current_thread::CurrentThread>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, ()>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../runtime/context/runtime.rs:65:16
  21: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:204:9
  22: <tokio::runtime::runtime::Runtime>::block_on_inner::<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../src/runtime/runtime.rs:379:52
  23: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../src/runtime/runtime.rs:348:18
  24: test_object_store_selects_scheduler
             at ./src/scheduler.rs:3249:49
  25: lance_io::scheduler::tests::test_object_store_selects_scheduler::{closure#0}
             at ./src/scheduler.rs:3204:51
  26: <lance_io::scheduler::tests::test_object_store_selects_scheduler::{closure#0} as core::ops::function::FnOnce<()>>::call_once
             at .../src/ops/function.rs:250:5
  27: <fn() -> core::result::Result<(), alloc::string::String> as core::ops::function::FnOnce<()>>::call_once
             at .../src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

Appends go through the existing insert writer and land as AddFragment
and AddDataFile on the reusable ApplyState. A scan pins publication to
the base version. Appends that were not scanned can rebase. The
uncommitted snapshot uses u64::MAX so commit and maintenance reject it.

Co-authored-by: Drew <geruh@users.noreply.github.com>
@geruh geruh changed the title feat(transaction): add append-only staged transactions feat(dataset): add append-only staged transactions Nov 18, 2026
/// Staged snapshots use `u64::MAX` instead. It sits at the top of the detached
/// range, and detached commits draw again when they land on it. A transaction
/// or manifest copied from the snapshot keeps this version, so commit rejects it.
pub const STAGED_VERSION: u64 = u64::MAX;

@geruh geruh Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

used u64::MAX instead of base_version + 1 so a concurrent commit at base_version + 1 doesn't collide with an open staged scan. detached_version redraws if it ever lands on it.

};
let mut dataset = self.clone();
dataset.set_manifest(Arc::new(manifest), location);
dataset.staged_indices = Some(Arc::new(indices));

@geruh geruh Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reusing Dataset for the staged snapshot so tx.scan() gets the existing query and index path for free. the ensure_published() guards are the price of doing that for now, can split a smaller reader snapshot trait out later.

}

fn append_dataset(&mut self) -> Result<Arc<Dataset>> {
if !self.state.append_context_dirty {

@geruh geruh Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blind appends skip building a manifest and reuse append_base. only time the schema changes mid transaction is when a 2.2+ file promotes a legacy blob column, so we rebuild once there and clear the dirty flag.


// Batch merge takes the minimum read_version, which would mask a
// STAGED_VERSION transaction mixed with committed ones.
transactions

@geruh geruh Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

InsertBuilder::execute_uncommitted needs to run on append_base (which can be a STAGED_VERSION snapshot after scan() or blob promotion), so we block STAGED_VERSION at commit time instead. checking before min(read_version) in execute_batch so it can't hide behind an older committed txn.

// would remove.
let commit = CommitBuilder::new(self.base.clone())
.with_experimental_composite_operations(true)
.with_skip_auto_cleanup(true);

@geruh geruh Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

skipping auto cleanup on staged commits so earlier scanners don't lose files under them. uncommitted files aren't pinned against explicit cleanup though.

@geruh
geruh marked this pull request as ready for review November 18, 2026 21:53

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions

Copy link
Copy Markdown
Contributor

Important

Format specification vote

This PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer), at least one of them on the latest commit, and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process.

Approvals carry over across pushes, so a rebase or a typo fix does not send everyone back to re-vote. Whoever approves the latest commit is vouching that nothing substantive has changed since the earlier approvals; if something has, ask for fresh votes.

Status: ❌ Blocked — 0 of 3 required approvals

Approvals none (0/3)
Latest commit approved by none — one PMC member must approve the latest commit
Vetoes none
Voting period ends Mon 2026-11-23 21:53 UTC (13:53 PST)

Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the format-waived label to waive the vote for a trivial edit (typo, wording, formatting).

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Gate recommendation: approve with a non-blocking risk.

The documented cleanup limitation means cleanup must be coordinated with active transactions and scanners; it can otherwise leave a later commit referencing missing files or borrowed blob objects. Staged commits also skip automatic cleanup, so workloads using only this API need scheduled explicit cleanup.

Please mark this PR with the breaking-change label: fields of the public lance_table::transaction::action structs are now private and AddOverlays::data_change was removed, so Rust callers must use constructors and accessors.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Nov 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-ci CI / build workflows A-format On-disk format: protos and format spec docs A-java Java bindings + JNI enhancement New feature or request format-change A change to the format spec, which requires a vote. Remove if minor (e.g. fixing typo). K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants