Skip to content

perf(transaction): bound long-job conflict catch-up by what it touches - #11282

Draft
wjones127 wants to merge 1 commit into
will/oss-2601-conflict-parity-compactionfrom
claude/conflict-perf-e31bdc
Draft

wjones127 wants to merge 1 commit into
will/oss-2601-conflict-parity-compactionfrom
claude/conflict-perf-e31bdc

Conversation

@wjones127

Copy link
Copy Markdown
Contributor

Summary

A long-running job (compaction, index build) that read version R and commits after V versions landed pays more on this stack than on main. It reloads up to 2V manifests when it is an action set or catches up on action sets, and it holds every intermediate manifest when it adds index coverage, which runs out of memory on big tables. This PR adds two harnesses that measure that cost. It will then fix the regressions one commit at a time, measured against both harnesses.

So far, one commit:

  • conflict_resolver::tests::catchup: a deterministic test. A job staged at R commits through CommitBuilder over V=40 seeded concurrent commits (named, half action sets, all action sets). The job runs from a cold-cache session over an IOTracker-wrapped shared-memory:// store. The test bounds manifest loads and reads, deletion and index reads, manifests held for coverage repair, and window checks. It also covers the retry after a lost race, injected through ProxyObjectStore. Bounds are calibrated on a named job over named commits, which takes the same path as main. 19 of 29 cases fail today. The test adds a cfg(test) probe in conflict_resolver for the history size and window checks.
  • test_a_retry_repairs_coverage_as_one_attempt_would: on a retry, coverage repair loses the manifests from earlier attempts. An overlay that landed in the first window on a fragment removed in the retry's window is judged differently than if one attempt had caught up on both. Fails today.
  • benches/conflict_catchup.rs: the same scenario through the public API on local files, sweeping V × N × mix × job. It reports wall time, peak heap (via a tracking allocator), GETs and bytes by kind, retry time, and time per phase. The phase times come from new debug tracing spans on load_and_sort_new_transactions, record_committed_history, check_txns and finish_with_tagged_rewrite. Fixtures are built once and reused.

Plan

One commit each, with before/after numbers from both harnesses:

  1. Fold manifests into the rebase as the catch-up streams them, in version order. Use ordered try_buffered instead of buffer_unordered, and take a named commit's footprint from the consecutive pair. This removes record_committed_history's second pass of up to 2V loads and bounds memory at about parallelism × M.
  2. Replace the per-version manifest history with a fragment trace. Keep the first and last Fragment seen for each fragment that the new coverage names or the window creates. Carry the trace across retry attempts, which fixes the retry repair gap. The repair becomes a lookup per covered fragment, not a walk over V manifests.
  3. Check a pending named op's window once per attempt. Gather all committed action-set footprints, build its own footprint once, and stop Window::live from cloning fragments.
  4. Share committed footprints instead of cloning them. Pass footprints by reference to the check. Hold IndexWrite.adds as a shared bitmap rather than a HashSet<u64> copied into both did and forbids. Drop parts from committed footprints.
  5. Build an inverted sources map once in UntaggedReuse::moved_to.
  6. settle_rows: read deletion files in parallel and look fragments up by id.
  7. Only if measurements show it matters: classify_failed_apply's O(N) rebuild per failed attempt, and keeping intermediate catch-up manifests out of the metadata cache.

Fragment ordering: Manifest::fragment_by_id binary-searches by id, and normalize_fragments sorts by id on every manifest build. The harness also asserts the order after each compaction it commits.

Evidence

Baseline numbers from both harnesses, stack vs main, are posted as PR comments. "Main" is 840f6eb60: main plus the constructors refactor only, needed for the bench to compile.

Merge Risk

Door: two-way. Tests, a bench, cfg(test) probes, and debug tracing spans so far. No format change.
Blast Radius: concurrency and perf once the fixes land.

🤖 Generated with Claude Code

…d holds

Adds a deterministic harness (conflict_resolver::tests::catchup) that
commits a job staged at R over V seeded concurrent commits through
CommitBuilder, from a cold-cache session over an IOTracker-wrapped store,
and bounds manifest loads/reads, deletion and index reads, manifests held
for coverage repair, and window checks, also for the retry after a lost
race. Bounds are calibrated on a named job over named commits. Action-set
jobs, index-coverage jobs and named jobs over action sets fail them today.

Also adds a test that a retry repairs index coverage as one attempt would,
which fails today, and the conflict_catchup criterion bench with tracing
spans on the catch-up phases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wjones127

Copy link
Copy Markdown
Contributor Author

Baseline: before any fix

Deterministic harness (conflict_resolver::tests::catchup)

V=40 commits over N=1000 fragments, cold cache. Bounds are calibrated on a named job over named commits, which takes the same path as main: manifest loads ≤ V+1, manifest reads ≤ 4(V+1)+2, deletion reads ≤ 2 per merged fragment, index reads ≤ 8, manifests held ≤ 1, window checks ≤ 1. 19 of 29 cases fail:

job over manifest loads manifest reads manifests held window checks fails on
any named job named 40 162 0 0 passes
named append/delete half / action 41 165 0 17-18 / 40 window checks (item 3)
legacy CreateIndex half / action 80 283 40 17 / 40 loads, history (items 1-3)
action append/delete/compaction named / half 80 / 70 283 / 253 0 1 loads (item 2)
action append/delete/compaction action 41 166 0 1 passes
AddIndexSegment / AdjustIndexCoverage any 80 284 40 1 loads, history (items 1-2)
retry after a lost race, 5 jobs half 3-4 12-14 ≤ 1 1 passes

test_a_retry_repairs_coverage_as_one_attempt_would fails as predicted: the retry keeps coverage of fragment 0 ([0, 1]), while one attempt over both commits withdraws it ([1]).

Bench (conflict_catchup, release-with-debug, local disk, no metadata cache)

No metadata cache, as on a table whose manifests outgrow the 1 GiB cache: a decoded manifest is about 0.8 KB per fragment. With the default cache, V=100 x N=10k fits in it, which hides both the extra reads and the history (every job then peaks at about 860 MB). "main" is 840f6eb60: main plus the constructors refactor only, run on the same fixtures. "history" and "finish" are wall time in record_committed_history and finish_with_tagged_rewrite.

V=1000, N=10k (median of 3)

job mix main s stack s main heap MB stack heap MB main manifest GB stack manifest GB stack history s stack finish s
legacy_append legacy 6.9 6.1 50.0 50.3 0.73 0.73 0.0 0.00
legacy_delete legacy 7.1 6.2 50.0 50.3 0.74 0.74 0.0 0.00
legacy_create_index legacy 7.0 6.1 50.1 50.3 0.73 0.73 0.0 0.01
action_append legacy – 12.4 – 50.3 – 1.47 6.3 0.00
action_delete legacy – 12.7 – 50.3 – 1.47 6.4 0.00
action_compaction legacy – 12.6 – 50.3 – 1.47 6.3 0.00
action_add_index_segment legacy – 13.7 – 7913.7 – 1.47 5.8 1.60
action_adjust_index_coverage legacy – 13.5 – 7913.7 – 1.47 5.7 1.57
legacy_append half – 6.2 – 50.1 – 0.74 0.0 0.00
legacy_delete half – 6.2 – 50.1 – 0.74 0.0 0.00
legacy_create_index half – 13.5 – 7911.4 – 1.46 5.7 1.57
action_append half – 10.9 – 50.1 – 1.31 4.7 0.00
action_delete half – 10.9 – 50.1 – 1.31 4.7 0.00
action_compaction half – 10.9 – 50.1 – 1.31 4.7 0.00
action_add_index_segment half – 13.6 – 7912.2 – 1.47 5.7 1.60
action_adjust_index_coverage half – 13.5 – 7912.2 – 1.47 5.7 1.57
legacy_append action – 6.3 – 49.9 – 0.74 0.0 0.00
legacy_delete action – 6.2 – 49.9 – 0.74 0.0 0.00
legacy_create_index action – 14.0 – 7911.8 – 1.46 5.8 1.74
action_append action – 6.5 – 49.9 – 0.74 0.0 0.00
action_delete action – 6.4 – 49.9 – 0.74 0.0 0.00
action_compaction action – 6.5 – 49.9 – 0.74 0.0 0.00
action_add_index_segment action – 13.9 – 7912.0 – 1.47 5.8 1.75
action_adjust_index_coverage action – 14.0 – 7911.9 – 1.47 5.8 1.74

V=100, N=100k (median of 3)

job mix main s stack s main heap MB stack heap MB main manifest GB stack manifest GB stack history s stack finish s
legacy_append legacy 8.5 7.4 409.0 409.0 0.79 0.79 0.0 0.00
legacy_delete legacy 8.7 7.1 409.0 409.0 0.80 0.80 0.0 0.00
legacy_create_index legacy 8.9 7.3 409.0 409.0 0.79 0.79 0.0 0.11
action_append legacy – 13.6 – 409.0 – 1.57 6.6 0.03
action_delete legacy – 13.5 – 409.0 – 1.57 6.6 0.03
action_compaction legacy – 13.4 – 409.0 – 1.57 6.6 0.03
action_add_index_segment legacy – 16.3 – 7602.8 – 1.57 7.3 1.94
action_adjust_index_coverage legacy – 15.4 – 7602.0 – 1.57 6.8 1.71
legacy_append half – 6.9 – 409.0 – 0.80 0.0 0.02
legacy_delete half – 6.7 – 409.0 – 0.80 0.0 0.02
legacy_create_index half – 15.1 – 7601.1 – 1.56 6.6 1.83
action_append half – 11.9 – 409.0 – 1.41 5.0 0.03
action_delete half – 11.9 – 409.0 – 1.41 5.0 0.03
action_compaction half – 12.1 – 409.0 – 1.41 5.1 0.03
action_add_index_segment half – 15.3 – 7602.6 – 1.57 6.8 1.91
action_adjust_index_coverage half – 15.3 – 7601.9 – 1.57 6.6 1.74
legacy_append action – 7.1 – 409.0 – 0.80 0.0 0.02
legacy_delete action – 6.8 – 409.0 – 0.80 0.0 0.02
legacy_create_index action – 15.4 – 7601.1 – 1.56 6.7 1.80
action_append action – 6.9 – 409.0 – 0.80 0.1 0.03
action_delete action – 7.0 – 409.0 – 0.80 0.1 0.03
action_compaction action – 6.9 – 409.0 – 0.80 0.1 0.03
action_add_index_segment action – 15.4 – 7602.6 – 1.57 6.6 1.87
action_adjust_index_coverage action – 15.5 – 7601.9 – 1.57 6.7 1.73

V=1000, N=300k, named commits only: main 272 s, 1.44 GB peak heap, 22 GB of manifest reads for a legacy append. The stack is the same for a named job (280 s). I stopped the run there. The catch-up costs 0.6–0.9 µs per version × fragment at every size measured (V=100 x N=10k: 0.75, V=1000 x N=10k: 0.69, V=100 x N=100k: 0.85, V=1000 x N=300k: 0.91, all on main), so the small sizes predict this one. The history costs about 0.8 KB per version × fragment (7.9 GB at V=1000 x N=10k, 7.6 GB at V=100 x N=100k). An index-coverage job at V=1000 x N=300k would hold about 230 GB, so it was not run.

Reading

  • Item 2: an action job, or a legacy CreateIndex over action sets, reloads up to V manifests in record_committed_history. That doubles manifest bytes and wall time (12.4 s vs 6.1 s at V=1000 x N=10k).
  • Item 1: index-coverage jobs hold every manifest, so peak heap grows with V x N (7.9 GB vs 50 MB). The repair walk over that history adds 1.6 s in finish.
  • Item 3: a named job over action sets checks the window once per committed action set (40 checks at V=40). Its time is small here, under 0.1 s, but it grows with V.
  • Retries are already bounded, at 3-4 manifest loads.
  • Action deletes read 2 deletion files per merged fragment, against 1 for legacy. That is bounded, so it is not a regression of concern.

@wjones127
wjones127 force-pushed the claude/conflict-perf-e31bdc branch from 7d7b7f5 to 6c89b27 Compare November 20, 2026 02:52

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant