Repository navigation
Conversation
…d holds Adds a deterministic harness (conflict_resolver::tests::catchup) that commits a job staged at R over V seeded concurrent commits through CommitBuilder, from a cold-cache session over an IOTracker-wrapped store, and bounds manifest loads/reads, deletion and index reads, manifests held for coverage repair, and window checks, also for the retry after a lost race. Bounds are calibrated on a named job over named commits. Action-set jobs, index-coverage jobs and named jobs over action sets fail them today. Also adds a test that a retry repairs index coverage as one attempt would, which fails today, and the conflict_catchup criterion bench with tracing spans on the catch-up phases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Baseline: before any fixDeterministic harness (
|
| job | over | manifest loads | manifest reads | manifests held | window checks | fails on |
|---|---|---|---|---|---|---|
| any named job | named | 40 | 162 | 0 | 0 | passes |
| named append/delete | half / action | 41 | 165 | 0 | 17-18 / 40 | window checks (item 3) |
| legacy CreateIndex | half / action | 80 | 283 | 40 | 17 / 40 | loads, history (items 1-3) |
| action append/delete/compaction | named / half | 80 / 70 | 283 / 253 | 0 | 1 | loads (item 2) |
| action append/delete/compaction | action | 41 | 166 | 0 | 1 | passes |
| AddIndexSegment / AdjustIndexCoverage | any | 80 | 284 | 40 | 1 | loads, history (items 1-2) |
| retry after a lost race, 5 jobs | half | 3-4 | 12-14 | ≤ 1 | 1 | passes |
test_a_retry_repairs_coverage_as_one_attempt_would fails as predicted: the retry keeps coverage of fragment 0 ([0, 1]), while one attempt over both commits withdraws it ([1]).
Bench (conflict_catchup, release-with-debug, local disk, no metadata cache)
No metadata cache, as on a table whose manifests outgrow the 1 GiB cache: a decoded manifest is about 0.8 KB per fragment. With the default cache, V=100 x N=10k fits in it, which hides both the extra reads and the history (every job then peaks at about 860 MB). "main" is 840f6eb60: main plus the constructors refactor only, run on the same fixtures. "history" and "finish" are wall time in record_committed_history and finish_with_tagged_rewrite.
V=1000, N=10k (median of 3)
| job | mix | main s | stack s | main heap MB | stack heap MB | main manifest GB | stack manifest GB | stack history s | stack finish s |
|---|---|---|---|---|---|---|---|---|---|
| legacy_append | legacy | 6.9 | 6.1 | 50.0 | 50.3 | 0.73 | 0.73 | 0.0 | 0.00 |
| legacy_delete | legacy | 7.1 | 6.2 | 50.0 | 50.3 | 0.74 | 0.74 | 0.0 | 0.00 |
| legacy_create_index | legacy | 7.0 | 6.1 | 50.1 | 50.3 | 0.73 | 0.73 | 0.0 | 0.01 |
| action_append | legacy | – | 12.4 | – | 50.3 | – | 1.47 | 6.3 | 0.00 |
| action_delete | legacy | – | 12.7 | – | 50.3 | – | 1.47 | 6.4 | 0.00 |
| action_compaction | legacy | – | 12.6 | – | 50.3 | – | 1.47 | 6.3 | 0.00 |
| action_add_index_segment | legacy | – | 13.7 | – | 7913.7 | – | 1.47 | 5.8 | 1.60 |
| action_adjust_index_coverage | legacy | – | 13.5 | – | 7913.7 | – | 1.47 | 5.7 | 1.57 |
| legacy_append | half | – | 6.2 | – | 50.1 | – | 0.74 | 0.0 | 0.00 |
| legacy_delete | half | – | 6.2 | – | 50.1 | – | 0.74 | 0.0 | 0.00 |
| legacy_create_index | half | – | 13.5 | – | 7911.4 | – | 1.46 | 5.7 | 1.57 |
| action_append | half | – | 10.9 | – | 50.1 | – | 1.31 | 4.7 | 0.00 |
| action_delete | half | – | 10.9 | – | 50.1 | – | 1.31 | 4.7 | 0.00 |
| action_compaction | half | – | 10.9 | – | 50.1 | – | 1.31 | 4.7 | 0.00 |
| action_add_index_segment | half | – | 13.6 | – | 7912.2 | – | 1.47 | 5.7 | 1.60 |
| action_adjust_index_coverage | half | – | 13.5 | – | 7912.2 | – | 1.47 | 5.7 | 1.57 |
| legacy_append | action | – | 6.3 | – | 49.9 | – | 0.74 | 0.0 | 0.00 |
| legacy_delete | action | – | 6.2 | – | 49.9 | – | 0.74 | 0.0 | 0.00 |
| legacy_create_index | action | – | 14.0 | – | 7911.8 | – | 1.46 | 5.8 | 1.74 |
| action_append | action | – | 6.5 | – | 49.9 | – | 0.74 | 0.0 | 0.00 |
| action_delete | action | – | 6.4 | – | 49.9 | – | 0.74 | 0.0 | 0.00 |
| action_compaction | action | – | 6.5 | – | 49.9 | – | 0.74 | 0.0 | 0.00 |
| action_add_index_segment | action | – | 13.9 | – | 7912.0 | – | 1.47 | 5.8 | 1.75 |
| action_adjust_index_coverage | action | – | 14.0 | – | 7911.9 | – | 1.47 | 5.8 | 1.74 |
V=100, N=100k (median of 3)
| job | mix | main s | stack s | main heap MB | stack heap MB | main manifest GB | stack manifest GB | stack history s | stack finish s |
|---|---|---|---|---|---|---|---|---|---|
| legacy_append | legacy | 8.5 | 7.4 | 409.0 | 409.0 | 0.79 | 0.79 | 0.0 | 0.00 |
| legacy_delete | legacy | 8.7 | 7.1 | 409.0 | 409.0 | 0.80 | 0.80 | 0.0 | 0.00 |
| legacy_create_index | legacy | 8.9 | 7.3 | 409.0 | 409.0 | 0.79 | 0.79 | 0.0 | 0.11 |
| action_append | legacy | – | 13.6 | – | 409.0 | – | 1.57 | 6.6 | 0.03 |
| action_delete | legacy | – | 13.5 | – | 409.0 | – | 1.57 | 6.6 | 0.03 |
| action_compaction | legacy | – | 13.4 | – | 409.0 | – | 1.57 | 6.6 | 0.03 |
| action_add_index_segment | legacy | – | 16.3 | – | 7602.8 | – | 1.57 | 7.3 | 1.94 |
| action_adjust_index_coverage | legacy | – | 15.4 | – | 7602.0 | – | 1.57 | 6.8 | 1.71 |
| legacy_append | half | – | 6.9 | – | 409.0 | – | 0.80 | 0.0 | 0.02 |
| legacy_delete | half | – | 6.7 | – | 409.0 | – | 0.80 | 0.0 | 0.02 |
| legacy_create_index | half | – | 15.1 | – | 7601.1 | – | 1.56 | 6.6 | 1.83 |
| action_append | half | – | 11.9 | – | 409.0 | – | 1.41 | 5.0 | 0.03 |
| action_delete | half | – | 11.9 | – | 409.0 | – | 1.41 | 5.0 | 0.03 |
| action_compaction | half | – | 12.1 | – | 409.0 | – | 1.41 | 5.1 | 0.03 |
| action_add_index_segment | half | – | 15.3 | – | 7602.6 | – | 1.57 | 6.8 | 1.91 |
| action_adjust_index_coverage | half | – | 15.3 | – | 7601.9 | – | 1.57 | 6.6 | 1.74 |
| legacy_append | action | – | 7.1 | – | 409.0 | – | 0.80 | 0.0 | 0.02 |
| legacy_delete | action | – | 6.8 | – | 409.0 | – | 0.80 | 0.0 | 0.02 |
| legacy_create_index | action | – | 15.4 | – | 7601.1 | – | 1.56 | 6.7 | 1.80 |
| action_append | action | – | 6.9 | – | 409.0 | – | 0.80 | 0.1 | 0.03 |
| action_delete | action | – | 7.0 | – | 409.0 | – | 0.80 | 0.1 | 0.03 |
| action_compaction | action | – | 6.9 | – | 409.0 | – | 0.80 | 0.1 | 0.03 |
| action_add_index_segment | action | – | 15.4 | – | 7602.6 | – | 1.57 | 6.6 | 1.87 |
| action_adjust_index_coverage | action | – | 15.5 | – | 7601.9 | – | 1.57 | 6.7 | 1.73 |
V=1000, N=300k, named commits only: main 272 s, 1.44 GB peak heap, 22 GB of manifest reads for a legacy append. The stack is the same for a named job (280 s). I stopped the run there. The catch-up costs 0.6–0.9 µs per version × fragment at every size measured (V=100 x N=10k: 0.75, V=1000 x N=10k: 0.69, V=100 x N=100k: 0.85, V=1000 x N=300k: 0.91, all on main), so the small sizes predict this one. The history costs about 0.8 KB per version × fragment (7.9 GB at V=1000 x N=10k, 7.6 GB at V=100 x N=100k). An index-coverage job at V=1000 x N=300k would hold about 230 GB, so it was not run.
Reading
- Item 2: an action job, or a legacy CreateIndex over action sets, reloads up to V manifests in
record_committed_history. That doubles manifest bytes and wall time (12.4 s vs 6.1 s at V=1000 x N=10k). - Item 1: index-coverage jobs hold every manifest, so peak heap grows with V x N (7.9 GB vs 50 MB). The repair walk over that history adds 1.6 s in
finish. - Item 3: a named job over action sets checks the window once per committed action set (40 checks at V=40). Its time is small here, under 0.1 s, but it grows with V.
- Retries are already bounded, at 3-4 manifest loads.
- Action deletes read 2 deletion files per merged fragment, against 1 for legacy. That is bounded, so it is not a regression of concern.
7d7b7f5 to
6c89b27
Compare
Summary
A long-running job (compaction, index build) that read version R and commits after V versions landed pays more on this stack than on main. It reloads up to 2V manifests when it is an action set or catches up on action sets, and it holds every intermediate manifest when it adds index coverage, which runs out of memory on big tables. This PR adds two harnesses that measure that cost. It will then fix the regressions one commit at a time, measured against both harnesses.
So far, one commit:
conflict_resolver::tests::catchup: a deterministic test. A job staged at R commits throughCommitBuilderover V=40 seeded concurrent commits (named, half action sets, all action sets). The job runs from a cold-cache session over anIOTracker-wrappedshared-memory://store. The test bounds manifest loads and reads, deletion and index reads, manifests held for coverage repair, and window checks. It also covers the retry after a lost race, injected throughProxyObjectStore. Bounds are calibrated on a named job over named commits, which takes the same path as main. 19 of 29 cases fail today. The test adds acfg(test)probe inconflict_resolverfor the history size and window checks.test_a_retry_repairs_coverage_as_one_attempt_would: on a retry, coverage repair loses the manifests from earlier attempts. An overlay that landed in the first window on a fragment removed in the retry's window is judged differently than if one attempt had caught up on both. Fails today.benches/conflict_catchup.rs: the same scenario through the public API on local files, sweeping V × N × mix × job. It reports wall time, peak heap (via a tracking allocator), GETs and bytes by kind, retry time, and time per phase. The phase times come from newdebugtracing spans onload_and_sort_new_transactions,record_committed_history,check_txnsandfinish_with_tagged_rewrite. Fixtures are built once and reused.Plan
One commit each, with before/after numbers from both harnesses:
try_bufferedinstead ofbuffer_unordered, and take a named commit's footprint from the consecutive pair. This removesrecord_committed_history's second pass of up to 2V loads and bounds memory at about parallelism × M.Fragmentseen for each fragment that the new coverage names or the window creates. Carry the trace across retry attempts, which fixes the retry repair gap. The repair becomes a lookup per covered fragment, not a walk over V manifests.Window::livefrom cloning fragments.IndexWrite.addsas a shared bitmap rather than aHashSet<u64>copied into bothdidandforbids. Droppartsfrom committed footprints.UntaggedReuse::moved_to.settle_rows: read deletion files in parallel and look fragments up by id.classify_failed_apply's O(N) rebuild per failed attempt, and keeping intermediate catch-up manifests out of the metadata cache.Fragment ordering:
Manifest::fragment_by_idbinary-searches by id, andnormalize_fragmentssorts by id on every manifest build. The harness also asserts the order after each compaction it commits.Evidence
Baseline numbers from both harnesses, stack vs main, are posted as PR comments. "Main" is
840f6eb60: main plus the constructors refactor only, needed for the bench to compile.Merge Risk
Door: two-way. Tests, a bench,
cfg(test)probes, anddebugtracing spans so far. No format change.Blast Radius:
concurrencyandperfonce the fixes land.🤖 Generated with Claude Code