Skip to content

feat(dataset): add strict expected-head commit mode - #11021

Open
lichuang wants to merge 1 commit into
lance-format:mainfrom
lichuang:issue-8699
Open

lichuang wants to merge 1 commit into
lance-format:mainfrom
lichuang:issue-8699

Conversation

@lichuang

Copy link
Copy Markdown
Contributor

Summary

Add a strict expected-head commit mode for callers that validate a complete
snapshot-dependent basis before publishing output. Fixes #8699.

A normal non-overwrite commit may rebase a transaction before its first attempt.
That can accept a concurrent compatible change which landed after the caller's
validation. with_max_retries(0) does not help because the rebase occurs before
the retry loop.

What changed

  • Add CommitBuilder::with_expected_latest_version(expected).
    • The transaction's read_version must equal expected.
    • The current dataset head must be exactly expected.
    • A changed head returns Error::RetryableCommitConflict.
    • Strict commits use one attempt regardless of with_max_retries.
    • No TransactionRebase and no compatibility-based acceptance.
    • New datasets, detached commits, and mismatched read versions are rejected
      with InvalidInput.
  • Add CommitHandler::supports_expected_head_commit.
    • ConditionalPutCommitHandler: true
    • RenameCommitHandler: true
    • CommitLock: true
    • ExternalManifestCommitHandler: true
    • UnsafeCommitHandler: false
    • Custom handlers default to false and must opt in.
    • Strict mode returns NotSupported for handlers that cannot reject a
      concurrent write for the same manifest version.
  • Validate execute_batch transactions against the expected version.
  • Expose expected_latest_version in Python LanceDataset.commit and
    expectedLatestVersion in Java CommitBuilder.
  • Add Rust, Python, and Java tests and API documentation.

Semantics

The target version is expected + 1. A safe commit handler's conditional write
closes the race between the initial latest-version check and manifest
publication. If another writer occupies expected + 1, the handler reports a
conflict and strict mode does not retry.

After a strict conflict, the caller must reload and revalidate the dataset
before submitting a new transaction with a new expected version.

The current head witness is the version number. A version plus manifest identity
witness remains a possible future extension for ABA cases where a dataset is
deleted and recreated at the same version.

No protobuf or file-format change is required.

Tests

Added/updated:

  • test_strict_expected_head_commits_when_unchanged
  • test_strict_expected_head_rejects_changed_head
  • test_strict_expected_head_does_not_retry_after_concurrent_commit
  • test_strict_expected_head_requires_a_conflict_safe_handler
  • test_strict_expected_head_requires_matching_read_version
  • test_strict_expected_head_supports_rename_handler
  • test_commit_with_expected_latest_version (Python)
  • CommitBuilderExpectedHeadTest (Java)
  • Existing restore/expected-head regression test remains green.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions github-actions Bot added A-python Python bindings A-java Java bindings + JNI enhancement New feature or request labels Nov 13, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❌ Gate recommendation: request changes.

The publication condition needs to remain valid across history cleanup. Fence the durable head/high-watermark atomically, or coordinate cleanup with the full strict publication window, before advertising these handlers as strict expected-head capable.

}

fn supports_expected_head_commit(&self) -> bool {
true

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checking that expected + 1 is unoccupied does not preserve the expected-head guarantee when cleanup runs concurrently. After the strict preflight reads version 1, two compatible commits can publish versions 2 and 3, and ordinary cleanup can remove version 2. RenameCommitHandler then lets the strict writer recreate version 2 and return success while the actual head remains 3. Snapshot-dependent publishers can consequently receive success for output that never became the current dataset head, violating #8699's no-intervening-commit contract. Conditional PUT and lock-backed commits rely on the same removable manifest slot.

The publication fence must survive removal of that slot. A durable head/high-watermark condition or coordination with cleanup is needed; another unlocked head check leaves the race open.

Executed regression test

Add this to the existing rust/lance/src/io/commit.rs test module, using this PR's strict_dataset and empty_append helpers. Run cargo test -p lance --lib test_strict_expected_head. The six added PR tests pass; this test fails with strict commit returned success at Ok(2) after head advanced to 3. Cleanup uses its default verification settings; the dataset is never deleted or recreated.

#[derive(Debug)]
struct GateCleanupRaceHandler {
    advanced: Mutex<Option<Dataset>>,
}

#[async_trait::async_trait]
impl CommitHandler for GateCleanupRaceHandler {
    fn supports_expected_head_commit(&self) -> bool {
        true
    }

    async fn commit(
        &self,
        manifest: &mut Manifest,
        indices: Option<Vec<IndexMetadata>>,
        base_path: &Path,
        object_store: &ObjectStore,
        manifest_writer: ManifestWriter,
        naming_scheme: ManifestNamingScheme,
        transaction: Option<lance_table::format::Transaction>,
    ) -> std::result::Result<ManifestLocation, CommitError> {
        let mut advanced = self.advanced.lock().unwrap().take().unwrap();
        for _ in 0..2 {
            let read_version = advanced.version_id();
            advanced = CommitBuilder::new(Arc::new(advanced))
                .execute(empty_append(read_version))
                .await?;
        }
        let policy = CleanupPolicyBuilder::default()
            .versions(vec![manifest.version])?
            .build();
        crate::dataset::cleanup::cleanup_old_versions(&advanced, policy).await?;
        assert_eq!(advanced.latest_version_id().await?, manifest.version + 1);
        assert!(object_store.inner.head(&naming_scheme.manifest_path(base_path, manifest.version)).await.is_err());
        RenameCommitHandler
            .commit(manifest, indices, base_path, object_store, manifest_writer, naming_scheme, transaction)
            .await
    }
}

#[tokio::test]
async fn test_strict_expected_head_gate_cleanup_race() {
    let tmp = TempStrDir::default();
    let dataset = strict_dataset(tmp.as_str()).await;
    let expected = dataset.version_id();
    let handler = Arc::new(GateCleanupRaceHandler {
        advanced: Mutex::new(Some(dataset.clone())),
    });
    let result = CommitBuilder::new(Arc::new(dataset))
        .with_commit_handler(handler)
        .with_expected_latest_version(expected)
        .execute(empty_append(expected))
        .await;
    let actual_head = Dataset::open(tmp.as_str()).await.unwrap().version_id();
    assert_eq!(actual_head, expected + 2);
    assert!(result.is_err(), "strict commit returned success at {:?} after head advanced to {actual_head}", result.as_ref().map(|ds| ds.version_id()));
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The added post-publication check catches the original successful-return reproducer, but the same cleanup race still reports success after an ambiguous write error. Err(CommitError::OtherError(_)) → CommitOutcome::Ours bypasses strict_head_is_published, so a lost response can acknowledge version 2 after the head advanced to 3 and cleanup freed version 2. Built-in rename and conditional-PUT handlers suppress such errors after readback. This still violates #8699's strict publication contract. The publication fence must apply independently of the handler's response.

Reproducer

This wraps the PR's GateCleanupRaceHandler to model a lost response after its manifest write. Add it to the same test module and run cargo test -p lance --lib test_strict_expected_head_gate_cleanup_ambiguous.

    #[derive(Debug)]
    struct GateCleanupAmbiguousHandler {
        inner: GateCleanupRaceHandler,
    }

    #[async_trait::async_trait]
    impl CommitHandler for GateCleanupAmbiguousHandler {
        fn supports_expected_head_commit(&self) -> bool { true }
        fn propagate_commit_error_after_success(&self) -> bool { false }
        async fn commit(
            &self,
            manifest: &mut Manifest,
            indices: Option<Vec<IndexMetadata>>,
            base_path: &Path,
            object_store: &ObjectStore,
            manifest_writer: ManifestWriter,
            naming_scheme: ManifestNamingScheme,
            transaction: Option<lance_table::format::Transaction>,
        ) -> std::result::Result<ManifestLocation, CommitError> {
            self.inner.commit(manifest, indices, base_path, object_store,
                manifest_writer, naming_scheme, transaction).await?;
            Err(CommitError::OtherError(Error::io("injected lost response")))
        }
    }

    #[tokio::test]
    async fn test_strict_expected_head_gate_cleanup_ambiguous() {
        let tmp = TempStrDir::default();
        let dataset = strict_dataset(tmp.as_str()).await;
        let expected = dataset.version_id();
        let handler = Arc::new(GateCleanupAmbiguousHandler {
            inner: GateCleanupRaceHandler {
                advanced: Mutex::new(Some(dataset.clone())),
            },
        });
        let result = CommitBuilder::new(Arc::new(dataset))
            .with_commit_handler(handler)
            .with_expected_latest_version(expected)
            .execute(empty_append(expected)).await;
        let actual_head = Dataset::open(tmp.as_str()).await.unwrap().version_id();
        assert_eq!(actual_head, expected + 2);
        assert!(matches!(result, Err(Error::RetryableCommitConflict { .. })),
            "strict result {:?} while actual head is {actual_head}",
            result.as_ref().map(|ds| ds.version_id()));
    }

Executed on 2de2bb1: fails with strict result Ok(2) while actual head is 3. The seven PR strict-mode tests pass.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Nov 13, 2026
A normal non-overwrite commit can rebase a transaction before its first
attempt, accepting a concurrent compatible change that the caller never
validated. Disabling retries does not close this window because the initial
rebase happens before the retry loop.

Expose the existing conditional commit path through
CommitBuilder::with_expected_latest_version. The transaction read version must
equal the expected version; if the current head differs, the first attempt
returns a retryable conflict. Strict commits use one attempt, skip the
TransactionRebase path, and therefore cannot replace the original expected-head
witness. New datasets, detached commits, and mismatched read versions are
rejected before publication.

Strict mode also requires a commit handler that can reject a concurrent write
for the same manifest version. Add CommitHandler::supports_expected_head_commit
and enable it for conditional-put, rename, commit-lock, and external-manifest
handlers. UnsafeCommitHandler and other handlers that do not opt in return an
unsupported error instead of claiming a guarantee they cannot provide.

Expose the mode through the Python LanceDataset.commit API and the Java
CommitBuilder API, including tests and documentation.

Add tests for unchanged and changed heads, an injected concurrent commit before
publication, retry configuration being ignored, rename handlers, unsafe
handler rejection, and mismatched read versions.
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Nov 13, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❌ Gate recommendation: request changes.

Two correctness issues remain in the strict publication contract. Enforce the expected-head condition at publication with a fence that survives cleanup, and preserve successfully published manifests and transactions when a subsequent writer advances the head. A later head read cannot safely decide whether this attempt lost.

For the cleanup-exclusion approach, land #10504 first, then rebase and reuse its shared history guard across the full strict publication window.

match strict_head_is_published(&dataset, target_version).await {
Ok(true) => {}
Ok(false) => {
rollback_unpublished_manifest(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A successful strict commit is deleted if another writer publishes its successor before this check. The writer can correctly publish version 2 against expected head 1, and a normal writer can then publish version 3 incorporating its changes. This branch deletes version 2 and its transaction and returns a retryable conflict even though the change remains applied in version 3. That loses readable version history and invites retries of an already-applied operation. The matching CommitConflict/Ours branch has the same problem.

Keep successfully published artifacts intact and decide the expected-head condition atomically when publishing; observing a successor afterward does not establish that this attempt lost.

Reproducer

Add this to the existing rust/lance/src/io/commit.rs test module, using the PR's strict_dataset and empty_append helpers. Run cargo test -p lance --lib test_strict_expected_head_gate_successor_keeps_publication.

    #[derive(Debug)]
    struct GateAdvanceAfterPublishHandler {
        uri: String,
    }

    #[async_trait::async_trait]
    impl CommitHandler for GateAdvanceAfterPublishHandler {
        fn supports_expected_head_commit(&self) -> bool { true }
        async fn commit(
            &self,
            manifest: &mut Manifest,
            indices: Option<Vec<IndexMetadata>>,
            base_path: &Path,
            object_store: &ObjectStore,
            manifest_writer: ManifestWriter,
            naming_scheme: ManifestNamingScheme,
            transaction: Option<lance_table::format::Transaction>,
        ) -> std::result::Result<ManifestLocation, CommitError> {
            let location = RenameCommitHandler.commit(manifest, indices, base_path,
                object_store, manifest_writer, naming_scheme, transaction).await?;
            let published = Dataset::open(&self.uri).await?;
            assert_eq!(published.version_id(), manifest.version);
            CommitBuilder::new(Arc::new(published))
                .execute(empty_append(manifest.version)).await?;
            Ok(location)
        }
    }

    #[tokio::test]
    async fn test_strict_expected_head_gate_successor_keeps_publication() {
        let tmp = TempStrDir::default();
        let dataset = strict_dataset(tmp.as_str()).await;
        let expected = dataset.version_id();
        let transaction = Transaction::new(expected, Operation::UpdateConfig {
            config_updates: None,
            table_metadata_updates: Some(lance_table::transaction::UpdateMap {
                update_entries: vec![("strict-output", "published").into()],
                replace: false,
            }),
            schema_metadata_updates: None,
            field_metadata_updates: HashMap::new(),
        }, None);
        let result = CommitBuilder::new(Arc::new(dataset))
            .with_commit_handler(Arc::new(GateAdvanceAfterPublishHandler {
                uri: tmp.as_str().to_owned(),
            }))
            .with_expected_latest_version(expected)
            .execute(transaction).await;
        let latest = Dataset::open(tmp.as_str()).await.unwrap();
        assert_eq!(latest.version_id(), expected + 2);
        assert_eq!(latest.manifest.table_metadata.get("strict-output").map(String::as_str),
            Some("published"));
        let historical = latest.checkout_version(expected + 1).await;
        assert!(result.is_ok() && historical.is_ok(),
            "valid publication returned {:?}; version {} readable: {} (error: {:?}); its output remains in version {}",
            result.as_ref().map(|ds| ds.version_id()), expected + 1, historical.is_ok(),
            historical.as_ref().err(), latest.version_id());
    }

Executed on 2de2bb1: the assertion fails with RetryableCommitConflict, version 2 readable: false (DatasetNotFound), and its output remains in version 3. No cleanup or dataset deletion is performed in this reproducer.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Nov 13, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-java Java bindings + JNI A-python Python bindings enhancement New feature or request K-changes Latest Gatekeeper recommendation requests changes.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a strict expected-head dataset commit mode

1 participant