Repository navigation
rpc: serve ListObjectVersions from kv-rpc; fullnodes answer UNIMPLEMENTED - #28435
Merged
Merged
Conversation
This was referenced Oct 23, 2026
nickvikeras
force-pushed
the
nickv/object-max-version
branch
from
November 3, 2026 00:46
bee1392 to
6af2ce0
Compare
nickvikeras
force-pushed
the
nickv/list-object-versions
branch
from
November 3, 2026 00:46
f7252f4 to
7da4ab0
Compare
nickvikeras
force-pushed
the
nickv/object-max-version
branch
from
November 3, 2026 04:34
6af2ce0 to
b5f96e8
Compare
nickvikeras
force-pushed
the
nickv/list-object-versions
branch
from
November 3, 2026 04:34
7da4ab0 to
e29519b
Compare
nickvikeras
force-pushed
the
nickv/object-max-version
branch
from
November 3, 2026 16:30
b5f96e8 to
bf03c51
Compare
nickvikeras
force-pushed
the
nickv/list-object-versions
branch
from
November 3, 2026 16:30
e29519b to
f355266
Compare
nickvikeras
force-pushed
the
nickv/object-max-version
branch
from
November 3, 2026 16:40
bf03c51 to
7f0e185
Compare
nickvikeras
force-pushed
the
nickv/list-object-versions
branch
from
November 3, 2026 16:40
f355266 to
2896907
Compare
nickvikeras
force-pushed
the
nickv/object-max-version
branch
from
November 4, 2026 15:27
7f0e185 to
6f8051b
Compare
nickvikeras
force-pushed
the
nickv/list-object-versions
branch
from
November 4, 2026 15:27
2896907 to
9577f0f
Compare
nickvikeras
force-pushed
the
nickv/object-max-version
branch
from
November 4, 2026 17:00
6f8051b to
3c56b6d
Compare
nickvikeras
force-pushed
the
nickv/list-object-versions
branch
from
November 4, 2026 17:00
9577f0f to
d258996
Compare
nickvikeras
force-pushed
the
nickv/object-max-version
branch
from
November 4, 2026 21:30
3c56b6d to
ff00b53
Compare
nickvikeras
force-pushed
the
nickv/list-object-versions
branch
2 times, most recently
from
November 6, 2026 19:42
fe6e172 to
6c8ef94
Compare
nickvikeras
force-pushed
the
nickv/object-max-version
branch
from
November 6, 2026 19:42
ff00b53 to
1027630
Compare
nickvikeras
added a commit
that referenced
this pull request
Nov 6, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-fork/src/tests/store_execution.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kvstore/src/bigtable/client/mod.rs # crates/sui-kvstore/src/bigtable/mock_server.rs
nickvikeras
force-pushed
the
nickv/object-max-version
branch
from
November 6, 2026 21:34
1027630 to
d13a7d2
Compare
nickvikeras
force-pushed
the
nickv/list-object-versions
branch
from
November 6, 2026 21:35
6c8ef94 to
ea58372
Compare
nickvikeras
added a commit
that referenced
this pull request
Nov 6, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-fork/src/tests/store_execution.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kvstore/src/bigtable/client/mod.rs # crates/sui-kvstore/src/bigtable/mock_server.rs
nickvikeras
added a commit
that referenced
this pull request
Nov 6, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-fork/src/tests/store_execution.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kvstore/src/bigtable/client/mod.rs # crates/sui-kvstore/src/bigtable/mock_server.rs
nickvikeras
added a commit
that referenced
this pull request
Nov 6, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-fork/src/tests/store_execution.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kvstore/src/bigtable/client/mod.rs # crates/sui-kvstore/src/bigtable/mock_server.rs
nickvikeras
force-pushed
the
nickv/object-max-version
branch
from
November 6, 2026 23:56
d13a7d2 to
cc08144
Compare
nickvikeras
force-pushed
the
nickv/list-object-versions
branch
from
November 6, 2026 23:56
ea58372 to
2943e57
Compare
nickvikeras
added a commit
that referenced
this pull request
Nov 7, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-fork/src/tests/store_execution.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-rpc-api/src/reader.rs
nickvikeras
added a commit
that referenced
this pull request
Nov 7, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-fork/src/tests/store_execution.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-rpc-api/src/reader.rs
nickvikeras
added a commit
that referenced
this pull request
Nov 7, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-fork/src/tests/store_execution.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-rpc-api/src/reader.rs
nickvikeras
added a commit
that referenced
this pull request
Nov 7, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-fork/src/tests/store_execution.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-rpc-api/src/reader.rs
nickvikeras
added a commit
that referenced
this pull request
Nov 7, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-fork/src/tests/store_execution.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-rpc-api/src/reader.rs
nickvikeras
force-pushed
the
nickv/object-max-version
branch
from
November 7, 2026 15:56
cc08144 to
859b594
Compare
nickvikeras
force-pushed
the
nickv/list-object-versions
branch
from
November 7, 2026 15:56
2943e57 to
9d5a22f
Compare
2 of 8 tasks
nickvikeras
added a commit
that referenced
this pull request
Nov 7, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-fork/src/tests/store_execution.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kv-rpc/src/lib.rs # crates/sui-kvstore/src/bigtable/client/mod.rs # crates/sui-kvstore/src/bigtable/mock_server.rs # crates/sui-rpc-api/src/reader.rs
…NTED ListObjectVersions lists one object's live versions, ascending or descending, inside the intersection of [start_version, end_version), the (after, before) cursors, and versions at or below v_C, the object's version at the end of the bounding checkpoint, deletion and wrap rows included (GetObject's visibility rule). The bounding checkpoint is at_checkpoint, or the served height when it is unset: kvstore_objects is a concurrent pipeline, so rows above its watermark can land out of checkpoint order, and a listing past the served height could list a version while an older one is still being written, which a client paging on from the newer cursor would never see. No v_C means an empty page. sui-rpc-api: ObjectVersionsQuery parses a request for any backend: object_id, read_mask (default object_id,version,digest), options.limit (default 50, clamped to 1..=1000), ordering, and the version range and cursors folded into one half-open range, failing with INVALID_ARGUMENT (FieldInvalid on options.after/options.before) for a malformed cursor or one naming another object. A cursor is a BCS-encoded ObjectVersionCursor, an enum whose one variant is (object_id, version), so a later listing can add a variant and each backend accepts the other's cursors. The fullnode prunes superseded object versions, so its handler answers UNIMPLEMENTED, pointing at archival kv-rpc. kv-rpc: the handler checks enable-object-at-checkpoint, the flag that also gates GetObject's at_checkpoint and max_version reads (UNIMPLEMENTED when off, for every request, since every listing reads object_versions_by_checkpoint for its bound), and that at_checkpoint is at or below the served height (OUT_OF_RANGE / ABOVE_LATEST_AVAILABLE above it), answers an empty version range without a read, reads v_C, and reads one page with get_live_object_versions over the folded inclusive range, which skips tombstones server-side, so a short page means the range is exhausted. Both reads go through the request's gated client; the new BigTableClient::get_live_object_versions wrapper runs under one objects-stage permit. AtCheckpointReads is shared with the new module, and the kv-rpc test server, mask and history helpers move from get_object's tests to v2::test_utils. Tests: kv-rpc unit tests over MockBigtableServer check every bound alone and combined against the history in both orders, at_checkpoint before creation and through a wrap, backward-then-forward paging, two cursors, the served-height bound with a newer version written above it, rejections, and the flag. kv_rpc_tests drives ObjectLifecycle through the same bounds with response cursors, and checks rejections and the flag off by default. The fullnode e2e test checks UNIMPLEMENTED.
1 of 8 tasks
nickvikeras
force-pushed
the
nickv/object-max-version
branch
from
November 10, 2026 18:19
859b594 to
3eac85c
Compare
nickvikeras
force-pushed
the
nickv/list-object-versions
branch
from
November 10, 2026 18:19
9d5a22f to
160c62f
Compare
Base automatically changed from
nickv/object-max-version
to
nickv/objects-v2
November 10, 2026 18:19
1 of 8 tasks
Contributor
Author
|
Superseded by #28632; this was closed as merged into a stack branch by mistake. Nothing reached main. |
nickvikeras
added a commit
that referenced
this pull request
Nov 10, 2026
## Description kv-rpc's object history reads (`at_checkpoint`, `max_version`, `ListObjectVersions`, the running epoch's `system_state`) find an object's state at a checkpoint with a reverse scan of `object_versions_by_checkpoint` limited to one row. BigTable answers that scan by reading every row of its range that is not compacted yet, so near recently written data a single lookup read 25k rows (a Cetus pool) to 1.1M rows (the Clock) and took 90–1,600 ms. This PR re-keys the index so the same lookup is a forward scan that reads only the row it returns. ## Mini design doc **Design.** The row key of `object_versions_by_checkpoint` becomes `object_id ‖ be64(u64::MAX − checkpoint)`, so an object's rows sort newest checkpoint first. The cells (`v` version, `s` status) and the pipeline's logic do not change. `get_object_version_at_checkpoint(id, C)` reads forward from `(id, C)` to `(id, 0)` with `rows_limit` 1; the first row is the object's state at `C`. Both ends of the range are keys of the same object, so the read never reaches a neighbouring object's rows. **Decisions.** - *In place, not a `_v2` table.* The table and the `kvstore_object_versions_by_checkpoint` pipeline keep their names. Nothing in production reads the table: its only readers are the object history features in the stack above (#28401, #28434, #28435, #28607), which are off unless `enable-object-at-checkpoint` is set, and that flag is an alpha gate. A versioned table would add a schema switch for a table with no reader to switch. - *`objects` stays as it is.* Latest and exact-version reads keep the `objects` table (`object_id ‖ be64(version)`), the `kvstore_object_tombstones` pipeline and the tombstone-aware reads. An `objects_v2` table keyed newest-first, which makes latest, `max_version` and descending `ListObjectVersions` reads forward scans too, is in #28612. That PR is optional: it costs a ~4 TB backfill for every operator, and nothing in this PR or the stack depends on it. **Surprises.** Measured on mainnet's index during its backfill (`kv_object_latency_probe.py`, BigTable request stats): a reverse scan with `rows_limit` 1 read every row of its range in uncompacted data, and about one block (hundreds to ~3,000 rows) in compacted data, while a forward scan with `rows_limit` 1 read one row everywhere. At the backfill frontier the deployed reverse lookup read 25,500 rows / 87–100 ms for a Cetus pool, 39,100 rows / 89 ms for `0x5`, and 1,140,000 rows / 1,584 ms for the Clock; a forward read over the same ranges read 1 row in 1.2–1.5 ms. Total history length does not matter; the number of the object's rows in uncompacted data below `C` does. At the frontier one hot object's `max_version` reads stopped at about 24 req/s, because each read scanned 30–40k rows on one tablet. **Impact and rollout.** - Rows written with the old key do not decode as the new key, and the two layouts must not mix in one table. Once every writer of `kvstore_object_versions_by_checkpoint` runs an image with this change, or is stopped: drop and recreate `object_versions_by_checkpoint`, delete the `kvstore_object_versions_by_checkpoint` row of `watermark_alt`, and backfill the pipeline from genesis. kv-rpc instances with `enable-object-at-checkpoint` must not serve from the table until the backfill completes (as today). - No change for anyone who does not run the alpha pipeline. **Test coverage.** - Codec: `keys_round_trip` and `keys_sort_checkpoints_newest_first` cover checkpoints 0, 1, 255, 256, 2^32, `u64::MAX − 1` and `u64::MAX`, and that the keys of checkpoints 0 and `u64::MAX` end in `0xff…` and `0x00…`. - Read shape: `object_version_at_checkpoint_is_one_forward_row` (mock BigTable) checks every checkpoint from 0 to 12 and at `u64::MAX − 1` / `u64::MAX` against a model, with neighbouring objects' rows at checkpoints 0, 1 and `u64::MAX`, and that each lookup is one forward ReadRows with `rows_limit` 1 over `[(id, C), (id, 0)]`. The mock now records each call's row ranges, `rows_limit` and direction. - Emulator: `object_version_at_checkpoint_stays_within_its_object` checks that a missing object between two neighbours resolves to nothing at any checkpoint, and the neighbours to their newest row at or below it. - End to end: `test_object_versions_by_checkpoint` drives an object through creation, mutation, wrap, unwrap and deletion and checks every checkpoint, now also `u64::MAX`. - Not covered: BigTable's actual cost of the forward read; that comes from the probe numbers above, not from a test. ## Test plan New tests: `tables::object_versions_by_checkpoint::tests::{keys_round_trip, keys_sort_checkpoints_newest_first}`, `bigtable::client::tests::object_version_at_checkpoint_is_one_forward_row`, `bigtable_client_emulator_tests::object_version_at_checkpoint_stays_within_its_object`. Updated: `e2e::test_object_versions_by_checkpoint` (adds checkpoint `u64::MAX`), and the handler test, which decodes keys with the new `decode_key`. --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] gRPC: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [x] Indexing Framework: The alpha `kvstore_object_versions_by_checkpoint` pipeline writes `object_versions_by_checkpoint` with a new row key, `object_id || (u64::MAX - checkpoint)`. Operators who run this pipeline must, once every writer runs the new version or is stopped, drop and recreate the `object_versions_by_checkpoint` table, delete the pipeline's `watermark_alt` row, and backfill it from genesis.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
GraphQL's
objectVersionsconnections page through one object's live versions in both directions, bounded by the checkpoint being viewed. They read Postgresobj_versionstoday. This PR serves the newLedgerService.ListObjectVersions(MystenLabs/sui-apis#43, MystenLabs/sui-rust-sdk#330) from kv-rpc. Stacked on #28434.sui-rpc-api):ObjectVersionCursoris a BCS enum with one variant,{object_id, version}, so cursors from either backend decode on the other.ObjectVersionsQueryfolds the version range and cursors into one range and validates the request for both backends.at_checkpoint, or by the served height when it is unset.objects_v2rows above the watermark can land out of checkpoint order, so an unbounded page could skip a version that was still being written. A page costs oneobject_versions_by_checkpointread plus oneget_live_object_versionsread ofobjects_v2, which drops tombstones on the BigTable side. A descending page is a forward read. An ascending page, the default order, is one reversed read over two ranges with the range's newest 2^20 versions last ([optional][kvstore] objects_v2: newest-first object versions behind object-schema #28612), so a page found below them never reads the object's uncompacted newest rows. The RPC therefore needsenable-object-at-checkpointwithobject-schema: v2and isUNIMPLEMENTEDotherwise.at_checkpointunset, a cursor above the object's version at the served height, or any cursor for an object with no version there, fails withOUT_OF_RANGE/ABOVE_LATEST_AVAILABLE(boundcursor), in either order and asafterorbefore. A cursor minted by a replica that has indexed further would otherwise make abeforepage skip versions and anafterpage look like the end of the listing.ListPackageVersionsrejects such cursors the same way. Withat_checkpointset, the listing is already pinned, so the cursor is clamped.UNIMPLEMENTED. It prunes superseded object versions, so a listing would silently miss them.Open questions:
bcsorjsonmask can make a very large response.ListOwnedObjectsalso stops at 512 KiB, but a byte cap would break the rule that a short page is the last one.object_idandversionstill reads object contents.Test plan
object-schema: v1), andtest_list_object_versions_stops_at_the_served_height: the served-height bound, and cursors above it rejected in both orders.kv_rpc_tests::test_list_object_versions_lifecyclepages the create, mutate, wrap, unwrap and delete lifecycle against the BigTable emulator.list_object_versions_is_unimplemented.get_live_object_versions(forward descending pages, the two-range ascending read) are covered in [optional][kvstore] objects_v2: newest-first object versions behind object-schema #28612's kvstore mock and emulator tests.Release notes
Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.
For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.
LedgerService.ListObjectVersionslists one object's live versions, paged by opaque cursor in either direction. It is bounded bystart_version/end_versionand byat_checkpoint, which defaults to the highest indexed checkpoint. The archival kv-rpc service serves it only withenable-object-at-checkpoint: trueandobject-schema: v2; fullnodes answerUNIMPLEMENTED.