Repository navigation
Conversation
Contributor
Author
|
Decisions I made on my own, and questions for review: Decisions
Open questions
|
This was referenced Oct 30, 2026
wjones127
force-pushed
the
will/oss-2599-conflict-parity-index-rebase
branch
from
October 30, 2026 05:58
f770809 to
649ba81
Compare
Contributor
Author
|
Rebased onto current #8646 head (8a5f8fd). Substantive changes:
|
wjones127
force-pushed
the
will/oss-2599-conflict-parity-index-rebase
branch
from
October 30, 2026 16:51
649ba81 to
c22c5f1
Compare
Contributor
Author
|
Rebased onto the updated #10474, which closed |
wjones127
force-pushed
the
will/oss-2599-conflict-parity-index-rebase
branch
from
November 4, 2026 01:00
c22c5f1 to
5a149d7
Compare
wjones127
force-pushed
the
will/oss-2599-conflict-parity-index-rebase
branch
from
November 12, 2026 23:14
5a149d7 to
97d6eb5
Compare
New index coverage gives up fragments whose indexed data changed since it read, instead of conflicting; a move recorded in the fragment reuse history is followed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
wjones127
force-pushed
the
will/oss-2599-conflict-parity-index-rebase
branch
from
November 19, 2026 01:58
97d6eb5 to
5ec7b7a
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Conflict parity 3/5 (OSS-2599). An index segment staged before a concurrent commit rewrote the data it covers used to conflict, because the
Part::IndexedFieldDatastand-in failed the check. Now the check lets it through and the coverage is repaired against the latest manifest, as legacyCreateIndexdoes. No format change.transaction::coverage_repair(layer 3).stale_coveragereturns the covered fragments a segment must give up, read off the manifests since the transaction read:repair_index_coverageruns fromfinishfor action sets and fromfinish_create_index. It applies the result withCompositeOperation::withdraw_index_coverage(forAddIndexSegmentandAdjustIndexCoverage) or by clearing bits in a legacyCreateIndexbitmap. It covers new segments and widenings alike.Part::IndexedFieldData, along withrequire_indexed_field_dataand its lookup inWindow.AddIndexSegment,AdjustIndexCoverageand legacyCreateIndexfootprints no longer claim the covered data.Did(Footprint::compacted_fragments). Whether a fragment reuse index recorded the move is not in the transaction, so the resolver checks it: if a set adds coverage over a compacted fragment and the latest reuse index records no move out of it, the commit is a retryable conflict. That matches legacyCreateIndexover an eager compaction.needs_committed_history,record_history), instead of only the footprints.load_current_lineagealso loads the lineage and untagged reuse history for action sets that add index segments or adjust coverage.Evidence
coverage_repairunit tests, with no dataset:test_a_column_rewritten_in_place(keyed, carried, unrelated, nested child under a rewritten parent, parent over a rewritten child, sibling),test_an_overlay_on_a_live_fragment_keeps_coverage,test_a_removed_fragment_keeps_coverage,test_rows_moved_somewhere_rewritten_since_lose_coverage,test_a_segment_that_cannot_follow_moves_loses_moved_coverage,test_a_folded_overlay,test_a_history_that_cannot_be_walked_loses_coverage,test_a_reencode_loses_coverage.test_withdrawing_coverage_rewrites_only_that_segmentpins the rewrite of the action set.test_a_widening_lands_over_a_compactionandtest_a_compaction_states_the_fragments_it_movedare new.test_a_widening_over_a_change_to_what_it_coversis removed, since widenings no longer conflict.action_create_index(vi) andaction_append_and_index(vj), in rows and columns, with no new*gaps.dataset_transactions.rsthat query the index after the landing commit:test_an_index_segment_rebases_over_a_legacy_column_rewrite: a real BTree lands over a legacyRewriteColumnsupdate with fragment 0 pruned, and indexed and unindexed scans agree.test_an_index_segment_rebases_over_a_deferred_compaction: the segment lands overcompact_files(defer_index_remap)and queries agree.test_an_index_segment_over_an_unrecorded_compaction_retries: an eager compaction still conflicts.test_an_index_segment_staged_before_a_rewrite_does_not_cover_itandtest_a_child_index_staged_before_a_packed_parent_rewrite_does_not_cover_it: flipped from "conflicts" to "lands with the fragment pruned".conflict_resolvertests:test_a_legacy_index_build_prunes_a_column_an_action_set_rewrote(flipped from conflict to prune, matchingtest_create_index_rebase_prunes_updated_field_coverage), andtest_a_legacy_index_build_lands_over_an_action_compaction. The index-over-stable-partition tests now run for both a legacyCreateIndexand an action-set segment.test_index_withdraws_a_folded_overlayinconcurrent_reindex.rsgains action-set cases.Merge Risk
Door: two-way. This changes in-memory footprints, the conflict resolver and the catch-up path. The wire format is unchanged.
Blast Radius: concurrency for action sets with
AddIndexSegmentorAdjustIndexCoverage, and legacyCreateIndexagainst action sets (it now loads and keeps the manifests since its read when an action set committed underneath it). Legacy-vs-legacy pairs are untouched.🤖 Generated with Claude Code