Repository navigation
rpc: serve GetObject max_version, and at_checkpoint as a visibility bound - #28434
Merged
Merged
Conversation
nickvikeras
force-pushed
the
nickv/object-versions-rpc
branch
2 times, most recently
from
November 3, 2026 00:46
1752a76 to
dd84d96
Compare
nickvikeras
force-pushed
the
nickv/object-max-version
branch
from
November 3, 2026 00:46
bee1392 to
6af2ce0
Compare
nickvikeras
force-pushed
the
nickv/object-versions-rpc
branch
from
November 3, 2026 04:34
dd84d96 to
dd3d532
Compare
nickvikeras
force-pushed
the
nickv/object-max-version
branch
from
November 3, 2026 04:34
6af2ce0 to
b5f96e8
Compare
nickvikeras
force-pushed
the
nickv/object-versions-rpc
branch
from
November 3, 2026 16:30
dd3d532 to
45e4f3f
Compare
nickvikeras
force-pushed
the
nickv/object-max-version
branch
from
November 3, 2026 16:30
b5f96e8 to
bf03c51
Compare
nickvikeras
force-pushed
the
nickv/object-versions-rpc
branch
from
November 3, 2026 16:40
45e4f3f to
a52f497
Compare
nickvikeras
force-pushed
the
nickv/object-max-version
branch
from
November 3, 2026 16:40
bf03c51 to
7f0e185
Compare
nickvikeras
force-pushed
the
nickv/object-versions-rpc
branch
from
November 4, 2026 15:26
a52f497 to
4bdc9ae
Compare
nickvikeras
force-pushed
the
nickv/object-max-version
branch
from
November 4, 2026 15:27
7f0e185 to
6f8051b
Compare
nickvikeras
force-pushed
the
nickv/object-versions-rpc
branch
from
November 4, 2026 17:00
4bdc9ae to
34ec19e
Compare
nickvikeras
force-pushed
the
nickv/object-max-version
branch
from
November 4, 2026 17:00
6f8051b to
3c56b6d
Compare
nickvikeras
force-pushed
the
nickv/object-versions-rpc
branch
from
November 4, 2026 21:30
34ec19e to
5183c9d
Compare
nickvikeras
force-pushed
the
nickv/object-max-version
branch
from
November 4, 2026 21:30
3c56b6d to
ff00b53
Compare
nickvikeras
force-pushed
the
nickv/object-versions-rpc
branch
from
November 5, 2026 00:05
5183c9d to
1d2504d
Compare
nickvikeras
force-pushed
the
nickv/object-max-version
branch
from
November 6, 2026 19:42
ff00b53 to
1027630
Compare
nickvikeras
force-pushed
the
nickv/object-versions-rpc
branch
2 times, most recently
from
November 6, 2026 21:34
2090827 to
d533742
Compare
nickvikeras
force-pushed
the
nickv/object-max-version
branch
from
November 6, 2026 21:34
1027630 to
d13a7d2
Compare
nickvikeras
force-pushed
the
nickv/object-versions-rpc
branch
from
November 6, 2026 23:56
d533742 to
8c6b8b8
Compare
nickvikeras
force-pushed
the
nickv/object-max-version
branch
from
November 6, 2026 23:56
d13a7d2 to
cc08144
Compare
nickvikeras
force-pushed
the
nickv/object-versions-rpc
branch
from
November 7, 2026 15:56
8c6b8b8 to
b4634f1
Compare
nickvikeras
force-pushed
the
nickv/object-max-version
branch
from
November 7, 2026 15:56
cc08144 to
859b594
Compare
2 of 8 tasks
An object's newest version at or below a version: None when there is none, or when the object was deleted or wrapped at that version. Every implementor serves it: - RestReadStore and RpcStoreReadStore read it through the execution cache's find_object_lt_or_eq_version, which stops at deletion and wrap markers. - RpcStoreReader runs a bounded reverse scan of the rpc-store objects CF, which also stops at tombstones. - ForkStore resolves through its overlay and local store, then the remote pinned at the fork checkpoint: it answers with the object's version at the fork checkpoint when that is within the bound, and otherwise asks the remote for the newest version at or below the bound as of the fork checkpoint. Cached local versions count only from the object's earliest version pinned at the fork checkpoint upward. - Simulacrum and the transactional test runner's persisted store record no deletions or wraps, so they answer only for a live object at or below the bound and fail otherwise, as their bounded child-object reads do.
…ound GetObjectRequest.max_version reads an object's newest version at or below it: NOT_FOUND when there is none, or when the object was deleted or wrapped at that version. version and max_version are mutually exclusive (INVALID_ARGUMENT on max_version, nested at requests[i]). at_checkpoint now combines with either as a visibility bound: the selected version is NOT_FOUND unless it was written at or before the checkpoint, with no fallback to an older version. Version R is visible at checkpoint C iff R is at or below the object's version at the end of C, deletion and wrap rows included, from the index at_checkpoint already reads; an object with no version by C has none visible. The at_checkpoint limits (OUT_OF_RANGE above the indexed tip and below the pruned floor, UNAVAILABLE while history backfills or on nodes without object version history) apply to the bound too. A max_version read without at_checkpoint is also UNAVAILABLE on nodes without object version history, since it reads versions later transactions replaced. validate_get_object_requests takes the GetObjectRequests themselves. ObjectSelector's Version and MaxVersion variants carry the optional bound, and ObjectSelector::not_found builds the NOT_FOUND error, which can now name a max_version. The fullnode resolves max_version through RpcStateReader::get_object_lt_or_eq_version, checks the at_checkpoint upper bound before any read and the lower bound after, and checks visibility with StateReader::is_object_version_visible_at_checkpoint. kv-rpc serves max_version behind the existing enable-object-at-checkpoint flag (Unimplemented when off), which now also folds the alpha kvstore_object_tombstones pipeline into the served height and requires it backfilled from genesis. The two object pipelines are deployed and backfilled together, so one flag covers both. A single read looks up the bound's version alongside the object read. A batch adds each bound's (object, checkpoint) to the stage 1 index scans, reads bounded exact versions in the stage 2 objects multi-gets, and runs one gated objects reverse scan per distinct (object, max_version), at most MAX_CONCURRENT_UNVERSIONED_SCANS at a time. Items whose at_checkpoint is above the served height read nothing. Tests: version + at_checkpoint is no longer rejected, so the kv_rpc_tests rejection checks move to version + max_version, and test_get_object_at_checkpoint_disabled_by_default becomes test_get_object_at_checkpoint_and_max_version_disabled_by_default. The fullnode lifecycle test, now get_object_lifecycle, also covers max_version and both bounds. The kv-rpc test server seeds its watermarks through the shared seed_watermarks helper, which takes the pipeline list.
…visible at the checkpoint With max_version and at_checkpoint K, both servers now return the newest version at or below max_version among the versions written in checkpoints at or before K: the newest version at or below min(max_version, v_K), where v_K is the object's version at the end of K from the index at_checkpoint already reads (object_version_by_checkpoint on the fullnode, object_versions_by_checkpoint on kv-rpc). NOT_FOUND if the object had no version by K or the selected version is a deletion or wrap. Previously the server selected the newest version at or below max_version among every version it held and failed with NOT_FOUND if that version was written after K, so the answer for a fixed K depended on data written after K and on how far the server had indexed. kv-rpc also bounds a max_version read without at_checkpoint by its served height, reading it as though at_checkpoint were the served height, the way its bare GetPackage is bounded. kvstore_objects is a concurrent pipeline, so a version written above the served height can be in the table before its deletion or wrap tombstone is, and an unbounded read could answer with it; an answer addressed by version looks immutable, so clients cache it. This costs one object_versions_by_checkpoint scan per read. Unchanged: version with at_checkpoint, at_checkpoint alone, max_version alone on the fullnode, the checkpoint limit errors, and kv-rpc's flag (enable-object-at-checkpoint gates every max_version read). Fullnode: the MaxVersion path reads v_K through StateReader::get_object_version_at_checkpoint (which get_object_at_checkpoint and the version visibility check also use) and scans down from min(max_version, v_K). Stores without rpc indexes (simulacrum, the transactional test runner) keep refusing any at_checkpoint read with UNAVAILABLE before reading. kv-rpc: at_checkpoint and max_version reads share one rule (bounded_read), with at_checkpoint alone read as max_version u64::MAX. Given the index row (V, status) at the bounding checkpoint and max_version M: no row is NOT_FOUND; M < V scans the objects table down from M; otherwise a deletion or wrap at V is NOT_FOUND with no further read, and a live V is read like an at_checkpoint item (an objects multi-get, or straight from the index for a mask of only object_id and version). A single read looks up V, then reads. A batch starts the exact-version multi-gets alongside the stage 1 index scans, since they do not depend on them, and after stage 1 reads the live versions no exact-version read covers in multi-gets, alongside one reverse scan per distinct (object, scan start). Tests: the kv-rpc model test, kv_rpc_tests::test_get_object_max_version_and_checkpoint_bounds and the fullnode get_object_lifecycle model the new rule, and the no-fallback cases now expect the older version visible at the checkpoint. test_max_version_without_checkpoint_is_bounded_by_the_served_height checks that a version written above kv-rpc's served height never answers an unbounded max_version read. kv_rpc_tests' flag-off test keeps only the at_checkpoint read (test_get_object_at_checkpoint_disabled_by_default); the kv-rpc unit flag test covers the other shapes.
1 of 8 tasks
nickvikeras
force-pushed
the
nickv/object-versions-rpc
branch
from
November 10, 2026 18:18
b4634f1 to
a6e9581
Compare
nickvikeras
force-pushed
the
nickv/object-max-version
branch
from
November 10, 2026 18:19
859b594 to
3eac85c
Compare
Base automatically changed from
nickv/object-versions-rpc
to
nickv/objects-v2
November 10, 2026 18:19
1 of 8 tasks
Contributor
Author
|
Superseded by #28631; this was closed as merged into a stack branch by mistake. Nothing reached main. |
nickvikeras
added a commit
that referenced
this pull request
Nov 10, 2026
## Description kv-rpc's object history reads (`at_checkpoint`, `max_version`, `ListObjectVersions`, the running epoch's `system_state`) find an object's state at a checkpoint with a reverse scan of `object_versions_by_checkpoint` limited to one row. BigTable answers that scan by reading every row of its range that is not compacted yet, so near recently written data a single lookup read 25k rows (a Cetus pool) to 1.1M rows (the Clock) and took 90–1,600 ms. This PR re-keys the index so the same lookup is a forward scan that reads only the row it returns. ## Mini design doc **Design.** The row key of `object_versions_by_checkpoint` becomes `object_id ‖ be64(u64::MAX − checkpoint)`, so an object's rows sort newest checkpoint first. The cells (`v` version, `s` status) and the pipeline's logic do not change. `get_object_version_at_checkpoint(id, C)` reads forward from `(id, C)` to `(id, 0)` with `rows_limit` 1; the first row is the object's state at `C`. Both ends of the range are keys of the same object, so the read never reaches a neighbouring object's rows. **Decisions.** - *In place, not a `_v2` table.* The table and the `kvstore_object_versions_by_checkpoint` pipeline keep their names. Nothing in production reads the table: its only readers are the object history features in the stack above (#28401, #28434, #28435, #28607), which are off unless `enable-object-at-checkpoint` is set, and that flag is an alpha gate. A versioned table would add a schema switch for a table with no reader to switch. - *`objects` stays as it is.* Latest and exact-version reads keep the `objects` table (`object_id ‖ be64(version)`), the `kvstore_object_tombstones` pipeline and the tombstone-aware reads. An `objects_v2` table keyed newest-first, which makes latest, `max_version` and descending `ListObjectVersions` reads forward scans too, is in #28612. That PR is optional: it costs a ~4 TB backfill for every operator, and nothing in this PR or the stack depends on it. **Surprises.** Measured on mainnet's index during its backfill (`kv_object_latency_probe.py`, BigTable request stats): a reverse scan with `rows_limit` 1 read every row of its range in uncompacted data, and about one block (hundreds to ~3,000 rows) in compacted data, while a forward scan with `rows_limit` 1 read one row everywhere. At the backfill frontier the deployed reverse lookup read 25,500 rows / 87–100 ms for a Cetus pool, 39,100 rows / 89 ms for `0x5`, and 1,140,000 rows / 1,584 ms for the Clock; a forward read over the same ranges read 1 row in 1.2–1.5 ms. Total history length does not matter; the number of the object's rows in uncompacted data below `C` does. At the frontier one hot object's `max_version` reads stopped at about 24 req/s, because each read scanned 30–40k rows on one tablet. **Impact and rollout.** - Rows written with the old key do not decode as the new key, and the two layouts must not mix in one table. Once every writer of `kvstore_object_versions_by_checkpoint` runs an image with this change, or is stopped: drop and recreate `object_versions_by_checkpoint`, delete the `kvstore_object_versions_by_checkpoint` row of `watermark_alt`, and backfill the pipeline from genesis. kv-rpc instances with `enable-object-at-checkpoint` must not serve from the table until the backfill completes (as today). - No change for anyone who does not run the alpha pipeline. **Test coverage.** - Codec: `keys_round_trip` and `keys_sort_checkpoints_newest_first` cover checkpoints 0, 1, 255, 256, 2^32, `u64::MAX − 1` and `u64::MAX`, and that the keys of checkpoints 0 and `u64::MAX` end in `0xff…` and `0x00…`. - Read shape: `object_version_at_checkpoint_is_one_forward_row` (mock BigTable) checks every checkpoint from 0 to 12 and at `u64::MAX − 1` / `u64::MAX` against a model, with neighbouring objects' rows at checkpoints 0, 1 and `u64::MAX`, and that each lookup is one forward ReadRows with `rows_limit` 1 over `[(id, C), (id, 0)]`. The mock now records each call's row ranges, `rows_limit` and direction. - Emulator: `object_version_at_checkpoint_stays_within_its_object` checks that a missing object between two neighbours resolves to nothing at any checkpoint, and the neighbours to their newest row at or below it. - End to end: `test_object_versions_by_checkpoint` drives an object through creation, mutation, wrap, unwrap and deletion and checks every checkpoint, now also `u64::MAX`. - Not covered: BigTable's actual cost of the forward read; that comes from the probe numbers above, not from a test. ## Test plan New tests: `tables::object_versions_by_checkpoint::tests::{keys_round_trip, keys_sort_checkpoints_newest_first}`, `bigtable::client::tests::object_version_at_checkpoint_is_one_forward_row`, `bigtable_client_emulator_tests::object_version_at_checkpoint_stays_within_its_object`. Updated: `e2e::test_object_versions_by_checkpoint` (adds checkpoint `u64::MAX`), and the handler test, which decodes keys with the new `decode_key`. --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] gRPC: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [x] Indexing Framework: The alpha `kvstore_object_versions_by_checkpoint` pipeline writes `object_versions_by_checkpoint` with a new row key, `object_id || (u64::MAX - checkpoint)`. Operators who run this pipeline must, once every writer runs the new version or is stopped, drop and recreate the `object_versions_by_checkpoint` table, delete the pipeline's `watermark_alt` row, and backfill it from genesis.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
GraphQL's
rootVersionreads take an object's newest version at or below a version, and its checkpoint-scoped exact-version reads hide versions written after the viewed checkpoint. Both come from Postgresobj_versionstoday. This PR serves the newGetObjectRequest.max_version(MystenLabs/sui-apis#42, MystenLabs/sui-rust-sdk#329) from the fullnode and kv-rpc. It also servesat_checkpointcombined withversionormax_versionas a visibility bound. Withversion, the answer matches Postgres:NOT_FOUNDif that version was written after the checkpoint. Withmax_version, the server picks the newest version at or belowmin(max_version, v_K), wherev_Kis the object's version at the end of checkpoint K, so the answer doesn't depend on anything written after K or on how far the server has indexed. Postgres differs in one case: when the newest version at or belowmax_versionwas written after K, Postgres returns null and this RPC returns the older version visible at K. GraphQL'srootVersionwith a viewed checkpoint will need to account for that when it moves onto this RPC.Stacked on #28401.
max_versionreadsfind_object_lt_or_eq_versionthrough a newRpcStateReader::get_object_lt_or_eq_version. A version is visible at checkpoint C when it is at or below the object's version at the end of C in theobject_version_by_checkpointindex. Simulacrum and the transactional test runner's store record no deletions or wraps. Theirmax_versionreturns the newest live version at or below the bound, orINTERNALwhen there is none.max_versionis served behind the same gate asat_checkpoint:enable-object-at-checkpointwithobject-schema: v2([optional][kvstore] objects_v2: newest-first object versions behind object-schema #28612),UNIMPLEMENTEDotherwise. Amax_versionread stops at the deletion and wrap tombstones thatkvstore_objects_v2writes alongside every live version, and is one single-row forward read ofobjects_v2frommax_version. That pipeline already bounds the served height underv2, somax_versionadds no pipeline to it.max_versionwithoutat_checkpointis bounded by the served height, read asat_checkpoint= served height:kvstore_objects_v2is a concurrent pipeline, so above its watermark a version can be in the table before the tombstone that removed it, and a version-addressed answer looks immutable, so clients would cache it. kv-rpc's bareGetPackageis bounded the same way.max_versionread whosemax_versionis at or abovev_Kis answered from the index row (NOT_FOUNDfor a deletion or wrap, otherwise versionv_Kthrough the batchedobjects_v2reads), and reads down frommax_versiononly when it is belowv_K.max_versionresolves against the remote pinned at the fork checkpoint, so it never returns a version created on the live chain after the fork, and cached local versions count only from the object's version at the fork checkpoint upward.Test plan
max_version, single and batch, with and without contents. Unit tests also cover the gate (flag off, and flag on underobject-schema: v1) and the served height, includingtest_max_version_without_checkpoint_is_bounded_by_the_served_height.kv_rpc_tests::test_get_object_max_version_and_checkpoint_boundsruns the same reads against the BigTable emulator over the create, mutate, wrap, unwrap and delete lifecycle. The object skips a version mid-lifecycle, and amax_versioninside the gap returns the older version. It also covers theversion+max_versionrejection.get_object_lifecycleadds every-versionmax_versionreads and bounded reads, with the same version gap.GetObject. The not-yet-indexed and pruned-floor tests cover bounded reads.NOT_FOUNDboth unversioned and at its removal version.Release notes
Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.
For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.
LedgerService.GetObjectandBatchGetObjectsacceptmax_version, which returns the object's newest version at or below it, orNOT_FOUNDif the object was deleted or wrapped at that version. A request that sets bothversionandmax_versionfails withINVALID_ARGUMENT.at_checkpointcan now be combined withversion(the read fails withNOT_FOUNDif that version was written after the checkpoint) or withmax_version(only versions written at or before the checkpoint are considered). The archival kv-rpc service servesmax_versiononly withenable-object-at-checkpoint: trueandobject-schema: v2, as it doesat_checkpoint, and bounds it by the served height.