Repository navigation
feat: expose a public toggle for disable_transaction_file - #11232
Draft
vaibhavdabas16 wants to merge 1 commit into
Draft
vaibhavdabas16 wants to merge 1 commit into
vaibhavdabas16 wants to merge 1 commit into
Conversation
Add `WriteParams::disable_transaction_file` and `CommitBuilder::with_disable_transaction_file` so a commit can store its transaction only inline in the manifest and skip the redundant file under `_transactions/`. The flag applies per commit: later commits that leave it unset write the transaction file again and drop the writer feature flag. Expose the same option as `disable_transaction_file` on Python's `write_dataset`. Closes lance-format#7486
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #7486
Summary
disable_transaction_filewas plumbed end to end but could only be turned on from tests. This adds a public switch:WriteParams::disable_transaction_file(Rust), forwarded byInsertBuilderCommitBuilder::with_disable_transaction_file(bool)(Rust)disable_transaction_file=onlance.write_dataset(Python), as a thin wrapper overWriteParamsWhen set, the commit stores its transaction only in the manifest's inline
transaction_section, skips the_transactions/file, and setsFLAG_DISABLE_TRANSACTION_FILE. The setting is per commit, as the flag is not inherited: a later commit that leaves it unset writes the file again and drops the flag. The default staysfalse; turning it on by default is #7487.Java bindings are not included and can follow in a separate PR with the same parameter name.
Tests
test_disable_transaction_file_applies_to_its_commit_onlychecks that no transaction file is written, the flag is set, a freshly opened dataset reads the inline transaction, and a following append writes the file and clears the flag.test_disable_transaction_filecovers the same behavior throughwrite_dataset.Local verification
cargo fmt --all, ruff, ruff-format, and typos pass.cargo clippy. The locallancetest build ran out of memory on my machine, so this is a draft until CI passes.