Skip to content

fix(cleanup): guard history references against concurrent deletion - #10504

Open
lance-gatefixer[bot] wants to merge 20 commits into
mainfrom
gatekeeper/fix-10499-1
Open

lance-gatefixer[bot] wants to merge 20 commits into
mainfrom
gatekeeper/fix-10499-1

Conversation

@lance-gatefixer

@lance-gatefixer lance-gatefixer Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Cleanup snapshots tags, branch lineage, and manifests before deleting files. A tag, branch, or restore published after that snapshot can therefore succeed while cleanup removes the version or data it references.

This lock complements the retention interval needed by writes and indexing jobs by protecting publication of references to versions already eligible for cleanup.

Coordinate cleanup and history reference publication with a storage lock at the table root, shared across branches and held throughout cascading cleanup. Tag creation, tag updates, metadata rewrites, branch creation, and attached or detached restore acquire the same guard. Combined branch creation holds it across reservation and cloning; standalone reservation and completion each guard their own phase. Overlapping operations return a reference conflict and can be retried. Remote lock acquisition and release use conditional writes bound to a unique owner; local stores use atomic create and delete. On a listing error, cleanup finishes the deletion sweep and waits for every sibling lineage before returning the first error and releasing the lock.

The regression pauses cleanup before its first deletion and attempts each affected reference operation, including standalone branch reservation and completion, against a multi-fragment source, on memory and local stores and on the default and a child branch. Additional tests cover cancellation during lock I/O, interrupted protected operations, nested-branch contention, and delayed release from an earlier owner. Ten error regressions cover partial listing batches at default and maximum delete concurrency, bulk requests, standalone cascades, and failing sibling lineages on both stores, including backend deletes that survive cancellation of their waiting futures. Restore regressions also verify that deleting a branch during normal or ambiguous publication returns an error and releases history exclusion.

The existing fragment-reuse garbage collection tests now live in a child module, preserving their module paths and assertions while keeping cleanup.rs below the 400 KB CI limit.

Validation passed after merging main at aef737d36:

  • cargo fmt --all and cargo fmt --all --check
  • cargo clippy --all --tests --benches --locked -- -D warnings
  • cargo test -p lance --lib --locked --features tokio/test-util -- dataset::cleanup:: dataset::refs:: dataset::tests::dataset_versioning io::commit:: dataset::blob::managed::tests dataset::tests::dataset_transactions::: 1,256 tests passed; two discovery utilities were ignored. Coverage includes the upstream large-delete-concurrency and cleaned-history schema-conflict regressions, maximum-concurrency error drainage on memory and local stores, history-reference races, restore, commit conflict resolution, and transactions.
  • python3 ci/check_module_size.py origin/main HEAD and python3 ci/check_module_size.py 4d85d7b6c35adaae627d76f8e474e0d2412128ad HEAD: passed with advisory size warnings.
  • python3 ci/check_advisory_ignores.py: passed.

Full cargo-deny could not run locally because it is not installed; dependency validation remains for CI on the pushed head.

The lock deliberately has no expiry: a paused cleanup must retain its exclusion when it resumes. Cancelling a protected operation also retains the lock because submitted storage requests may still complete. After cancellation, a process crash, or failed release, an operator must stop cleanup and history reference mutations before removing the active lock marker. All participating clients must implement this protocol, and remote stores must support reliable conditional updates. These constraints and recovery steps are documented in the tags and branches guide.

Fixes #10499

Change walkthrough

flowchart LR
    A[Acquire history lock] --> B[Grow candidate batch]
    B --> C[Drain deletes]
    C --> D{Stream exhausted?}
    D -->|No| B
    D -->|Yes| E[Finish cleanup and unlock]
Loading

Listing errors are recorded while remaining batches and sibling lineages finish; allocation grows with actual candidates even at maximum concurrency.

Read: History exclusion → Incremental batches and error drainage → Combined regression cases.

Snapshot: 7d7b5c2

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

@github-actions github-actions Bot added A-docs Documentation bug Something isn't working labels Oct 30, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Oct 30, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Oct 30, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Oct 30, 2026
@lance-gatekeeper lance-gatekeeper Bot removed K-risk Latest Gatekeeper recommendation includes a non-blocking risk. K-approved Latest Gatekeeper recommendation permits acceptance. labels Oct 30, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Oct 30, 2026
@lance-gatekeeper lance-gatekeeper Bot removed K-risk Latest Gatekeeper recommendation includes a non-blocking risk. K-approved Latest Gatekeeper recommendation permits acceptance. labels Oct 30, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Oct 30, 2026
@lance-gatekeeper lance-gatekeeper Bot removed K-risk Latest Gatekeeper recommendation includes a non-blocking risk. K-approved Latest Gatekeeper recommendation permits acceptance. labels Oct 30, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Oct 30, 2026
Preserve bulk cleanup batching while draining partial batches and sibling sweeps before releasing history exclusion. Cover listing errors in both single-file and bulk delete paths.
@lance-gatekeeper lance-gatekeeper Bot removed K-risk Latest Gatekeeper recommendation includes a non-blocking risk. K-approved Latest Gatekeeper recommendation permits acceptance. labels Nov 14, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-decision Latest Gatekeeper review requires a maintainer decision. label Nov 14, 2026
@lance-gatefixer

Copy link
Copy Markdown
Contributor Author

Blocked: the outstanding publication and mixed-version review requests depend on an ownership-safe rollback contract for the proposed retention-rule replacement.

The verified repair head is a7da2c651013dd2bbe2c8843935abd0db434cc61. It still holds the table-wide lock throughout cleanup, so the maintainer’s long-cleanup publication and mixed-version protection requests remain unsatisfied. The retention-rule proposal supplies the replacement direction, but its rejected-reference removal still needs the contract identified in the prior analysis.

I refreshed the Issue and complete review conversations and checked the current locking and deletion paths. Deletion still takes only a path: after publisher A writes a rejected reference, publisher B can successfully replace it before A’s rollback deletes B’s update. Checking ownership immediately before deletion leaves the same race. No code was changed or pushed this Turn. The previously addressed cleanup-error drainage remains intact, and all review threads remain unresolved.

To continue, specify whether rejected publications may use ownership-checked tombstones with defined reader behavior, or short per-reference coordination with defined interruption and recovery behavior; alternatively, supply another ownership-safe removal mechanism. Either choice can avoid holding the table-wide lock throughout cleanup.

@lance-gatekeeper lance-gatekeeper Bot removed the K-decision Latest Gatekeeper review requires a maintainer decision. label Nov 15, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-decision Latest Gatekeeper review requires a maintainer decision. label Nov 15, 2026
@lance-gatefixer

Copy link
Copy Markdown
Contributor Author

Blocked: the outstanding reference-publication and mixed-version requests require a decision on ownership-safe rejection in the proposed retention protocol.

At verified repair head a87f819cf0d45d4ecc7e2bbe773ff8cc6733463d, cleanup still holds the table-wide lock throughout its run. The maintainer’s long-cleanup publication and mixed-version protection requests therefore remain unsatisfied. The retention-rule proposal supplies the replacement direction, but does not yet specify safe rollback when another publisher updates the same reference.

I refreshed the Issue, all Reviews and threads, and the current publication, deletion, and cleanup paths. Deletion still accepts only a path: publisher A can write an expired reference, publisher B can replace it with an allowed reference and succeed, and A’s rejection can then delete B’s update. Reading ownership immediately before deleting leaves the same race. Cleanup-error drainage and its prior acknowledged disposition remain intact. No code changed, no tests were rerun, and no push or thread-state change was made this Turn.

To continue, select ownership-checked tombstones with defined reader behavior, or short per-reference coordination with defined interruption and recovery behavior; alternatively, supply another ownership-safe rejection mechanism. Either can allow publication during long cleanup without retaining the current table-wide exclusion.

@lance-gatekeeper lance-gatekeeper Bot removed the K-decision Latest Gatekeeper review requires a maintainer decision. label Nov 17, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-decision Latest Gatekeeper review requires a maintainer decision. label Nov 17, 2026
@lance-gatefixer

Copy link
Copy Markdown
Contributor Author

Blocked: the outstanding publication and mixed-version review requests depend on an ownership-safe rejection contract for the proposed retention protocol.

At verified repair head dde965ca87b48d96b1923a5fd4f7550c415dc1c4, cleanup still holds the table-wide lock throughout its run. The maintainer’s long-cleanup publication and mixed-version protection requests remain unsatisfied. The retention-rule proposal supplies the replacement direction, but its rejected-reference removal can delete a later successful update to the same reference.

I refreshed the Issue and all review conversations and inspected the current cleanup, publication, and deletion paths. Deletion still accepts only a path: publisher A can publish an expired reference, publisher B can replace it with an allowed reference and succeed, and A’s rollback can then delete B’s update. Reading ownership immediately before deletion leaves the same race. No code changed, no tests were rerun, and no push was made.

To continue, select ownership-checked tombstones with defined reader behavior, or short per-reference coordination with defined interruption and recovery behavior; alternatively, supply another ownership-safe rejection mechanism. Either can allow publication during long cleanup without table-wide exclusion.

@lance-gatefixer

Copy link
Copy Markdown
Contributor Author

Blocked: the Java 17 check remains failed after a Maven Central dependency-resolution error unrelated to the history-reference repair.

At verified repair head dde965ca87b48d96b1923a5fd4f7550c415dc1c4, Build and Test with Java 17 fails in spotless:check because Maven reports com.google.googlejavaformat:google-java-format:1.22.0 absent from Central. Tests were skipped; the failure is not a formatting violation or test failure.

I inspected the live check, annotations, and complete job log. Java sources, Maven configuration, and the Java workflow are identical to current main at e448432c2d7b7d23e68b42a2d7c0350b21f13a14, whose Java 17 job passed. On this repair head, ./mvnw --batch-mode spotless:check passed under Java 17, downloading the same formatter POM and JAR successfully and checking all 253 Java files. Central also returned HTTP 200 for the JAR. No files changed, no merge was needed, and no push was made.

Rerun the failed Java 17 job on this same head. If dependency resolution fails again, retry with a fresh Maven dependency cache and investigate Central delivery before changing the formatter or workflow.

Separately, the maintainer’s long-cleanup publication and mixed-version protection requests remain outstanding. The retention-rule proposal still needs an ownership-safe rejection contract: select ownership-checked tombstones with defined reader behavior, short per-reference coordination with interruption/recovery rules, or another mechanism that preserves a later successful publication during rollback.

@lance-gatekeeper lance-gatekeeper Bot removed the K-decision Latest Gatekeeper review requires a maintainer decision. label Nov 18, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-decision Latest Gatekeeper review requires a maintainer decision. label Nov 18, 2026
@lance-gatefixer

Copy link
Copy Markdown
Contributor Author

Blocked: the outstanding reference-publication and mixed-version review requests require an ownership-safe rejection contract for the replacement retention protocol.

At verified repair head 4d85d7b6c35adaae627d76f8e474e0d2412128ad, cleanup still holds the table-wide lock throughout its run, leaving the maintainer’s long-cleanup publication and mixed-version protection requests unsatisfied. The replacement side-file specification in #10924 remains open and unmerged; it supersedes the earlier manifest-config proposal, whose per-branch commits the maintainer ruled out.

I refreshed the Issue, all review conversations, the replacement specification and its discussion, and the current publication/deletion paths. The specification still requires writing a reference, checking retention, then removing the reference on rejection. Publisher A can write an expired reference, publisher B can replace it with an allowed reference and succeed, and A’s delayed removal can then delete B’s successful update. Current deletion accepts only a path; checking ownership immediately before deletion leaves the same race. The newer pin-lifetime and cleanup-serialization rules do not settle this reference-removal contract. No code changed, tests were not rerun, and no push was made.

To continue, specify ownership-checked tombstones with defined reader behavior, or permit short per-reference coordination with interruption and recovery rules; an alternative ownership-safe rejection mechanism would also unblock implementation. Either approach can preserve the side-file direction while allowing reference publication during a long cleanup.

Preserve incremental cleanup batch allocation while draining listing errors before releasing history exclusion. Cover error drainage at maximum delete concurrency on memory and local stores.
Incorporate schema-dependent commit fencing across cleaned-up history while preserving history-reference exclusion.
@lance-gatekeeper lance-gatekeeper Bot removed the K-decision Latest Gatekeeper review requires a maintainer decision. label Nov 18, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Gate recommendation: maintainer decision required.

Cleanup still holds the table-wide history lock throughout its run, so the long-cleanup publication and mixed-version protection concerns remain. #10924 is the open, unmerged replacement specification; its side files avoid the per-branch manifest commits the maintainer ruled out.

The author's latest analysis leaves one contract to choose before implementing that replacement: rejecting a publication must preserve any later successful update to the same reference. Ownership-checked tombstones need defined reader behavior; short per-reference coordination needs interruption and recovery rules. Select one of these or another ownership-safe rejection mechanism.

The current lock requires every participant to implement the protocol. An interrupted owner requires stopping cleanup and reference mutations before manually removing <table-root>/_refs/history.lock. The separate expire_versions API from #9438 still needs external coordination; its exposure is unchanged from the base.

@lance-gatekeeper lance-gatekeeper Bot added the K-decision Latest Gatekeeper review requires a maintainer decision. label Nov 18, 2026
@lance-gatefixer

Copy link
Copy Markdown
Contributor Author

Blocked: the outstanding publication and mixed-version review requests require a maintainer decision on safe rejection of references in the replacement retention protocol.

At verified repair head 7d7b5c2272a7d59d50f2a54eaf24368963bf5c61, cleanup still holds the table-wide lock throughout its run. The maintainer's long-cleanup publication and mixed-version protection requests remain unsatisfied. The replacement side-file specification in #10924 remains open and unmerged; the maintainer has ruled out per-branch manifest commits.

I refreshed the Issue, complete review conversations, replacement specification and discussion, and current cleanup/publication/deletion paths. The specification still requires writing a reference, checking retention, then removing the reference on rejection. Publisher A can write an expired tag, publisher B can replace it with an allowed tag and succeed, and A's delayed removal can delete B's successful update. Deletion accepts only a path; reading ownership before deleting leaves the same race. Cleanup serialization and restore-pin lifetimes do not settle this reference-removal contract. The previously acknowledged error-drainage fix remains intact. No code changed, tests were not rerun, and no push was made; review threads remain unresolved.

To continue, specify ownership-checked tombstones with reader semantics, or permit short per-reference coordination with interruption and recovery rules. Another ownership-safe rejection mechanism would also unblock implementation while preserving publication during long cleanups.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-docs Documentation bug Something isn't working K-decision Latest Gatekeeper review requires a maintainer decision.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cleanup_old_versions deletes a version tagged while it runs

2 participants