Repository navigation
fix(cleanup): guard history references against concurrent deletion - #10504
lance-gatefixer[bot] wants to merge 20 commits into
Conversation
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
Preserve bulk cleanup batching while draining partial batches and sibling sweeps before releasing history exclusion. Cover listing errors in both single-file and bulk delete paths.
|
Blocked: the outstanding publication and mixed-version review requests depend on an ownership-safe rollback contract for the proposed retention-rule replacement. The verified repair head is I refreshed the Issue and complete review conversations and checked the current locking and deletion paths. Deletion still takes only a path: after publisher A writes a rejected reference, publisher B can successfully replace it before A’s rollback deletes B’s update. Checking ownership immediately before deletion leaves the same race. No code was changed or pushed this Turn. The previously addressed cleanup-error drainage remains intact, and all review threads remain unresolved. To continue, specify whether rejected publications may use ownership-checked tombstones with defined reader behavior, or short per-reference coordination with defined interruption and recovery behavior; alternatively, supply another ownership-safe removal mechanism. Either choice can avoid holding the table-wide lock throughout cleanup. |
|
Blocked: the outstanding reference-publication and mixed-version requests require a decision on ownership-safe rejection in the proposed retention protocol. At verified repair head I refreshed the Issue, all Reviews and threads, and the current publication, deletion, and cleanup paths. Deletion still accepts only a path: publisher A can write an expired reference, publisher B can replace it with an allowed reference and succeed, and A’s rejection can then delete B’s update. Reading ownership immediately before deleting leaves the same race. Cleanup-error drainage and its prior acknowledged disposition remain intact. No code changed, no tests were rerun, and no push or thread-state change was made this Turn. To continue, select ownership-checked tombstones with defined reader behavior, or short per-reference coordination with defined interruption and recovery behavior; alternatively, supply another ownership-safe rejection mechanism. Either can allow publication during long cleanup without retaining the current table-wide exclusion. |
|
Blocked: the outstanding publication and mixed-version review requests depend on an ownership-safe rejection contract for the proposed retention protocol. At verified repair head I refreshed the Issue and all review conversations and inspected the current cleanup, publication, and deletion paths. Deletion still accepts only a path: publisher A can publish an expired reference, publisher B can replace it with an allowed reference and succeed, and A’s rollback can then delete B’s update. Reading ownership immediately before deletion leaves the same race. No code changed, no tests were rerun, and no push was made. To continue, select ownership-checked tombstones with defined reader behavior, or short per-reference coordination with defined interruption and recovery behavior; alternatively, supply another ownership-safe rejection mechanism. Either can allow publication during long cleanup without table-wide exclusion. |
|
Blocked: the Java 17 check remains failed after a Maven Central dependency-resolution error unrelated to the history-reference repair. At verified repair head I inspected the live check, annotations, and complete job log. Java sources, Maven configuration, and the Java workflow are identical to current Rerun the failed Java 17 job on this same head. If dependency resolution fails again, retry with a fresh Maven dependency cache and investigate Central delivery before changing the formatter or workflow. Separately, the maintainer’s long-cleanup publication and mixed-version protection requests remain outstanding. The retention-rule proposal still needs an ownership-safe rejection contract: select ownership-checked tombstones with defined reader behavior, short per-reference coordination with interruption/recovery rules, or another mechanism that preserves a later successful publication during rollback. |
|
Blocked: the outstanding reference-publication and mixed-version review requests require an ownership-safe rejection contract for the replacement retention protocol. At verified repair head I refreshed the Issue, all review conversations, the replacement specification and its discussion, and the current publication/deletion paths. The specification still requires writing a reference, checking retention, then removing the reference on rejection. Publisher A can write an expired reference, publisher B can replace it with an allowed reference and succeed, and A’s delayed removal can then delete B’s successful update. Current deletion accepts only a path; checking ownership immediately before deletion leaves the same race. The newer pin-lifetime and cleanup-serialization rules do not settle this reference-removal contract. No code changed, tests were not rerun, and no push was made. To continue, specify ownership-checked tombstones with defined reader behavior, or permit short per-reference coordination with interruption and recovery rules; an alternative ownership-safe rejection mechanism would also unblock implementation. Either approach can preserve the side-file direction while allowing reference publication during a long cleanup. |
Preserve incremental cleanup batch allocation while draining listing errors before releasing history exclusion. Cover error drainage at maximum delete concurrency on memory and local stores.
Incorporate schema-dependent commit fencing across cleaned-up history while preserving history-reference exclusion.
There was a problem hiding this comment.
🟡 Gate recommendation: maintainer decision required.
Cleanup still holds the table-wide history lock throughout its run, so the long-cleanup publication and mixed-version protection concerns remain. #10924 is the open, unmerged replacement specification; its side files avoid the per-branch manifest commits the maintainer ruled out.
The author's latest analysis leaves one contract to choose before implementing that replacement: rejecting a publication must preserve any later successful update to the same reference. Ownership-checked tombstones need defined reader behavior; short per-reference coordination needs interruption and recovery rules. Select one of these or another ownership-safe rejection mechanism.
The current lock requires every participant to implement the protocol. An interrupted owner requires stopping cleanup and reference mutations before manually removing <table-root>/_refs/history.lock. The separate expire_versions API from #9438 still needs external coordination; its exposure is unchanged from the base.
|
Blocked: the outstanding publication and mixed-version review requests require a maintainer decision on safe rejection of references in the replacement retention protocol. At verified repair head I refreshed the Issue, complete review conversations, replacement specification and discussion, and current cleanup/publication/deletion paths. The specification still requires writing a reference, checking retention, then removing the reference on rejection. Publisher A can write an expired tag, publisher B can replace it with an allowed tag and succeed, and A's delayed removal can delete B's successful update. Deletion accepts only a path; reading ownership before deleting leaves the same race. Cleanup serialization and restore-pin lifetimes do not settle this reference-removal contract. The previously acknowledged error-drainage fix remains intact. No code changed, tests were not rerun, and no push was made; review threads remain unresolved. To continue, specify ownership-checked tombstones with reader semantics, or permit short per-reference coordination with interruption and recovery rules. Another ownership-safe rejection mechanism would also unblock implementation while preserving publication during long cleanups. |
Cleanup snapshots tags, branch lineage, and manifests before deleting files. A tag, branch, or restore published after that snapshot can therefore succeed while cleanup removes the version or data it references.
This lock complements the retention interval needed by writes and indexing jobs by protecting publication of references to versions already eligible for cleanup.
Coordinate cleanup and history reference publication with a storage lock at the table root, shared across branches and held throughout cascading cleanup. Tag creation, tag updates, metadata rewrites, branch creation, and attached or detached restore acquire the same guard. Combined branch creation holds it across reservation and cloning; standalone reservation and completion each guard their own phase. Overlapping operations return a reference conflict and can be retried. Remote lock acquisition and release use conditional writes bound to a unique owner; local stores use atomic create and delete. On a listing error, cleanup finishes the deletion sweep and waits for every sibling lineage before returning the first error and releasing the lock.
The regression pauses cleanup before its first deletion and attempts each affected reference operation, including standalone branch reservation and completion, against a multi-fragment source, on memory and local stores and on the default and a child branch. Additional tests cover cancellation during lock I/O, interrupted protected operations, nested-branch contention, and delayed release from an earlier owner. Ten error regressions cover partial listing batches at default and maximum delete concurrency, bulk requests, standalone cascades, and failing sibling lineages on both stores, including backend deletes that survive cancellation of their waiting futures. Restore regressions also verify that deleting a branch during normal or ambiguous publication returns an error and releases history exclusion.
The existing fragment-reuse garbage collection tests now live in a child module, preserving their module paths and assertions while keeping
cleanup.rsbelow the 400 KB CI limit.Validation passed after merging
mainataef737d36:cargo fmt --allandcargo fmt --all --checkcargo clippy --all --tests --benches --locked -- -D warningscargo test -p lance --lib --locked --features tokio/test-util -- dataset::cleanup:: dataset::refs:: dataset::tests::dataset_versioning io::commit:: dataset::blob::managed::tests dataset::tests::dataset_transactions::: 1,256 tests passed; two discovery utilities were ignored. Coverage includes the upstream large-delete-concurrency and cleaned-history schema-conflict regressions, maximum-concurrency error drainage on memory and local stores, history-reference races, restore, commit conflict resolution, and transactions.python3 ci/check_module_size.py origin/main HEADandpython3 ci/check_module_size.py 4d85d7b6c35adaae627d76f8e474e0d2412128ad HEAD: passed with advisory size warnings.python3 ci/check_advisory_ignores.py: passed.Full
cargo-denycould not run locally because it is not installed; dependency validation remains for CI on the pushed head.The lock deliberately has no expiry: a paused cleanup must retain its exclusion when it resumes. Cancelling a protected operation also retains the lock because submitted storage requests may still complete. After cancellation, a process crash, or failed release, an operator must stop cleanup and history reference mutations before removing the active lock marker. All participating clients must implement this protocol, and remote stores must support reliable conditional updates. These constraints and recovery steps are documented in the tags and branches guide.
Fixes #10499
Change walkthrough
flowchart LR A[Acquire history lock] --> B[Grow candidate batch] B --> C[Drain deletes] C --> D{Stream exhausted?} D -->|No| B D -->|Yes| E[Finish cleanup and unlock]Listing errors are recorded while remaining batches and sibling lineages finish; allocation grows with actual candidates even at maximum concurrency.
Read: History exclusion → Incremental batches and error drainage → Combined regression cases.
Snapshot: 7d7b5c2