Skip to content

feat(transaction): repair new index coverage after the conflict check - #10477

Draft
wjones127 wants to merge 1 commit into
will/oss-2598-conflict-parity-schema-evolutionfrom
will/oss-2599-conflict-parity-index-rebase
Draft

wjones127 wants to merge 1 commit into
will/oss-2598-conflict-parity-schema-evolutionfrom
will/oss-2599-conflict-parity-index-rebase

Conversation

@wjones127

@wjones127 wjones127 commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Conflict parity 3/5 (OSS-2599). An index segment staged before a concurrent commit rewrote the data it covers used to conflict, because the Part::IndexedFieldData stand-in failed the check. Now the check lets it through and the coverage is repaired against the latest manifest, as legacy CreateIndex does. No format change.

  • New transaction::coverage_repair (layer 3). stale_coverage returns the covered fragments a segment must give up, read off the manifests since the transaction read:
    • A live fragment loses coverage when the data files backing a keyed or carried field (or a field containing or nested in one) differ from when it first appears in the window, in it or in a fragment its rows were moved into.
    • A removed fragment keeps its coverage, since a plain delete leaves nothing to describe and a recorded move is followed through the fragment reuse history. It loses it if the rows moved somewhere rewritten since, if a compaction folded in an overlay the segment never saw, or if the segment cannot follow moves (NGram).
    • Where the reuse history cannot be walked, coverage goes.
  • repair_index_coverage runs from finish for action sets and from finish_create_index. It applies the result with CompositeOperation::withdraw_index_coverage (for AddIndexSegment and AdjustIndexCoverage) or by clearing bits in a legacy CreateIndex bitmap. It covers new segments and widenings alike.
  • Removes stand-in 1, Part::IndexedFieldData, along with require_indexed_field_data and its lookup in Window. AddIndexSegment, AdjustIndexCoverage and legacy CreateIndex footprints no longer claim the covered data.
  • A compaction now records the fragments it moved in Did (Footprint::compacted_fragments). Whether a fragment reuse index recorded the move is not in the transaction, so the resolver checks it: if a set adds coverage over a compacted fragment and the latest reuse index records no move out of it, the commit is a retryable conflict. That matches legacy CreateIndex over an eager compaction.
  • Catch-up keeps every manifest since the read, for a transaction that adds coverage over committed fragments (needs_committed_history, record_history), instead of only the footprints. load_current_lineage also loads the lineage and untagged reuse history for action sets that add index segments or adjust coverage.

Evidence

  • coverage_repair unit tests, with no dataset: test_a_column_rewritten_in_place (keyed, carried, unrelated, nested child under a rewritten parent, parent over a rewritten child, sibling), test_an_overlay_on_a_live_fragment_keeps_coverage, test_a_removed_fragment_keeps_coverage, test_rows_moved_somewhere_rewritten_since_lose_coverage, test_a_segment_that_cannot_follow_moves_loses_moved_coverage, test_a_folded_overlay, test_a_history_that_cannot_be_walked_loses_coverage, test_a_reencode_loses_coverage.
  • test_withdrawing_coverage_rewrites_only_that_segment pins the rewrite of the action set.
  • Dataset-free footprint tests: the segment-vs-column-rewrite and segment-vs-reencode cases flip from Conflicts to Lands. test_a_widening_lands_over_a_compaction and test_a_compaction_states_the_fragments_it_moved are new. test_a_widening_over_a_change_to_what_it_covers is removed, since widenings no longer conflict.
  • Parity table: new scenarios action_create_index (vi) and action_append_and_index (vj), in rows and columns, with no new * gaps.
  • Integration tests in dataset_transactions.rs that query the index after the landing commit:
    • test_an_index_segment_rebases_over_a_legacy_column_rewrite: a real BTree lands over a legacy RewriteColumns update with fragment 0 pruned, and indexed and unindexed scans agree.
    • test_an_index_segment_rebases_over_a_deferred_compaction: the segment lands over compact_files(defer_index_remap) and queries agree.
    • test_an_index_segment_over_an_unrecorded_compaction_retries: an eager compaction still conflicts.
    • test_an_index_segment_staged_before_a_rewrite_does_not_cover_it and test_a_child_index_staged_before_a_packed_parent_rewrite_does_not_cover_it: flipped from "conflicts" to "lands with the fragment pruned".
  • conflict_resolver tests: test_a_legacy_index_build_prunes_a_column_an_action_set_rewrote (flipped from conflict to prune, matching test_create_index_rebase_prunes_updated_field_coverage), and test_a_legacy_index_build_lands_over_an_action_compaction. The index-over-stable-partition tests now run for both a legacy CreateIndex and an action-set segment.
  • test_index_withdraws_a_folded_overlay in concurrent_reindex.rs gains action-set cases.

Merge Risk

Door: two-way. This changes in-memory footprints, the conflict resolver and the catch-up path. The wire format is unchanged.
Blast Radius: concurrency for action sets with AddIndexSegment or AdjustIndexCoverage, and legacy CreateIndex against action sets (it now loads and keeps the manifests since its read when an action set committed underneath it). Legacy-vs-legacy pairs are untouched.

🤖 Generated with Claude Code

@github-actions github-actions Bot added the enhancement New feature or request label Oct 30, 2026
@wjones127

Copy link
Copy Markdown
Contributor Author

Decisions I made on my own, and questions for review:

Decisions

  • rebase(&mut self, &[Overlap]) has no current: &Manifest. Neither handler needs it. Whether a compaction is recorded by the fragment reuse index can't be read from a Manifest either way, since indices aren't on it. So the resolver settles CompactedUnderIndex (can_rebase_past) before calling rebase. Later milestones can add the parameter when a handler needs it.
  • CompactedUnderIndex keeps the coverage as is. The segment keeps naming the old fragment ids, which is byte-identical to what legacy CreateIndex publishes, and the reader translates them through the FRI. It does not rewrite them to the new ids. The integration test checks that the loaded coverage is the compacted fragment and that queries agree.
  • Index data requirements left the generic claims. A segment's coverage is now recorded separately (IndexBuild) and produces overlaps, not Requires on FieldData. A covered fragment that is deleted outright (not moved) is not an overlap. This matches legacy ci over df = L.
  • "Moved" vs "deleted": a legacy Rewrite (from the diff), and a RemoveFragment with data_change = false.
  • Stricter than legacy, kept:
    • On tagged FRI histories, an action-set segment conflicts over a compaction. Legacy follows the lineage. Only the untagged history is loaded for action sets.
    • A compaction that folded an overlay newer than the segment conflicts. Legacy withdraws the group from coverage.
    • NGram segments conflict over any compaction. Legacy does too.
  • Harness action_append_and_index indexes committed fragment 0, not the appended fragment. An index built before the commit can't know the ids the appended rows get. "Append + index what was appended" (a Ref::Local coverage) is covered in the dataset-free tier instead.
  • New oracle exemption: action_add_column over action_append_and_index. It has the same reason as the existing add-column-over-append exemptions.

Open questions

  • Should an action compaction be able to carry its own FRI record, so that legacy CreateIndex over it can rebase? Today it always conflicts, which is legacy parity for eager compactions only.
  • Is it OK to conflict rather than withdraw on folded newer overlays? It's rare, and withdrawing would need the rewrite groups, which the action path doesn't have.

@wjones127

Copy link
Copy Markdown
Contributor Author

Rebased onto current #8646 head (8a5f8fd). Substantive changes:

@wjones127
wjones127 force-pushed the will/oss-2599-conflict-parity-index-rebase branch from 649ba81 to c22c5f1 Compare October 30, 2026 16:51
@wjones127

Copy link
Copy Markdown
Contributor Author

Rebased onto the updated #10474, which closed DROPPED_FIELD_REJECTED. Cells: vj over pj/pa/mg/ma R(*) → L. I added dropped_field_overlaps next to index_overlaps in compare, and a FieldDropped arm next to the index arms in CompositeOperation::rebase. No other changes.

@wjones127
wjones127 force-pushed the will/oss-2599-conflict-parity-index-rebase branch from c22c5f1 to 5a149d7 Compare November 4, 2026 01:00
@wjones127
wjones127 force-pushed the will/oss-2599-conflict-parity-index-rebase branch from 5a149d7 to 97d6eb5 Compare November 12, 2026 23:14
@wjones127
wjones127 added this pull request to stack #8647 November 12, 2026 23:17
New index coverage gives up fragments whose indexed data changed since it read, instead of conflicting; a move recorded in the fragment reuse history is followed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wjones127
wjones127 force-pushed the will/oss-2599-conflict-parity-index-rebase branch from 97d6eb5 to 5ec7b7a Compare November 19, 2026 01:58
@wjones127 wjones127 changed the title feat(transaction): rebase index builds over compaction and rewrites feat(transaction): repair new index coverage after the conflict check Nov 19, 2026
@wjones127
wjones127 removed this pull request from stack #8647 November 19, 2026 02:01
@wjones127
wjones127 added this pull request to stack #11242 November 19, 2026 02:02

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant