Skip to content

rpc: serve ListObjectVersions from kv-rpc; fullnodes answer UNIMPLEMENTED - #28435

Merged
nickvikeras merged 2 commits into
nickv/objects-v2from
nickv/list-object-versions
Nov 10, 2026
Merged

nickvikeras merged 2 commits into
nickv/objects-v2from
nickv/list-object-versions

Conversation

@nickvikeras

@nickvikeras nickvikeras commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Description

GraphQL's objectVersions connections page through one object's live versions in both directions, bounded by the checkpoint being viewed. They read Postgres obj_versions today. This PR serves the new LedgerService.ListObjectVersions (MystenLabs/sui-apis#43, MystenLabs/sui-rust-sdk#330) from kv-rpc. Stacked on #28434.

  • Shared (sui-rpc-api): ObjectVersionCursor is a BCS enum with one variant, {object_id, version}, so cursors from either backend decode on the other. ObjectVersionsQuery folds the version range and cursors into one range and validates the request for both backends.
  • kv-rpc: Every listing is bounded by at_checkpoint, or by the served height when it is unset. objects_v2 rows above the watermark can land out of checkpoint order, so an unbounded page could skip a version that was still being written. A page costs one object_versions_by_checkpoint read plus one get_live_object_versions read of objects_v2, which drops tombstones on the BigTable side. A descending page is a forward read. An ascending page, the default order, is one reversed read over two ranges with the range's newest 2^20 versions last ([optional][kvstore] objects_v2: newest-first object versions behind object-schema #28612), so a page found below them never reads the object's uncompacted newest rows. The RPC therefore needs enable-object-at-checkpoint with object-schema: v2 and is UNIMPLEMENTED otherwise.
  • Cursors past the served height: with at_checkpoint unset, a cursor above the object's version at the served height, or any cursor for an object with no version there, fails with OUT_OF_RANGE/ABOVE_LATEST_AVAILABLE (bound cursor), in either order and as after or before. A cursor minted by a replica that has indexed further would otherwise make a before page skip versions and an after page look like the end of the listing. ListPackageVersions rejects such cursors the same way. With at_checkpoint set, the listing is already pinned, so the cursor is clamped.
  • Fullnode: answers UNIMPLEMENTED. It prunes superseded object versions, so a listing would silently miss them.

Open questions:

  • Pages have no byte cap. Up to 1000 versions of a large object with a bcs or json mask can make a very large response. ListOwnedObjects also stops at 512 KiB, but a byte cap would break the rule that a short page is the last one.
  • A mask of only object_id and version still reads object contents.

Test plan

  • A kv-rpc unit test checks 1,944 bound combinations against a model of an object's history with tombstones. Others cover backward-then-forward paging from cursors, rejections, the gate (flag off, and flag on under object-schema: v1), and test_list_object_versions_stops_at_the_served_height: the served-height bound, and cursors above it rejected in both orders.
  • kv_rpc_tests::test_list_object_versions_lifecycle pages the create, mutate, wrap, unwrap and delete lifecycle against the BigTable emulator.
  • Fullnode simtest list_object_versions_is_unimplemented.
  • The read shapes of get_live_object_versions (forward descending pages, the two-range ascending read) are covered in [optional][kvstore] objects_v2: newest-first object versions behind object-schema #28612's kvstore mock and emulator tests.

Release notes

Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.

For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.

  • Protocol:
  • Nodes (Validators and Full nodes):
  • gRPC: New LedgerService.ListObjectVersions lists one object's live versions, paged by opaque cursor in either direction. It is bounded by start_version/end_version and by at_checkpoint, which defaults to the highest indexed checkpoint. The archival kv-rpc service serves it only with enable-object-at-checkpoint: true and object-schema: v2; fullnodes answer UNIMPLEMENTED.
  • JSON-RPC:
  • GraphQL:
  • CLI:
  • Rust SDK:
  • Indexing Framework:

@nickvikeras
nickvikeras force-pushed the nickv/object-max-version branch from bee1392 to 6af2ce0 Compare November 3, 2026 00:46
@nickvikeras
nickvikeras force-pushed the nickv/list-object-versions branch from f7252f4 to 7da4ab0 Compare November 3, 2026 00:46
@nickvikeras
nickvikeras force-pushed the nickv/object-max-version branch from 6af2ce0 to b5f96e8 Compare November 3, 2026 04:34
@nickvikeras
nickvikeras force-pushed the nickv/list-object-versions branch from 7da4ab0 to e29519b Compare November 3, 2026 04:34
@nickvikeras
nickvikeras force-pushed the nickv/object-max-version branch from b5f96e8 to bf03c51 Compare November 3, 2026 16:30
@nickvikeras
nickvikeras force-pushed the nickv/list-object-versions branch from e29519b to f355266 Compare November 3, 2026 16:30
@nickvikeras
nickvikeras force-pushed the nickv/object-max-version branch from bf03c51 to 7f0e185 Compare November 3, 2026 16:40
@nickvikeras
nickvikeras force-pushed the nickv/list-object-versions branch from f355266 to 2896907 Compare November 3, 2026 16:40
@nickvikeras
nickvikeras force-pushed the nickv/object-max-version branch from 7f0e185 to 6f8051b Compare November 4, 2026 15:27
@nickvikeras
nickvikeras force-pushed the nickv/list-object-versions branch from 2896907 to 9577f0f Compare November 4, 2026 15:27
@nickvikeras
nickvikeras force-pushed the nickv/object-max-version branch from 6f8051b to 3c56b6d Compare November 4, 2026 17:00
@nickvikeras
nickvikeras force-pushed the nickv/list-object-versions branch from 9577f0f to d258996 Compare November 4, 2026 17:00
@nickvikeras
nickvikeras force-pushed the nickv/object-max-version branch from 3c56b6d to ff00b53 Compare November 4, 2026 21:30
@nickvikeras
nickvikeras force-pushed the nickv/list-object-versions branch 2 times, most recently from fe6e172 to 6c8ef94 Compare November 6, 2026 19:42
@nickvikeras
nickvikeras force-pushed the nickv/object-max-version branch from ff00b53 to 1027630 Compare November 6, 2026 19:42
nickvikeras added a commit that referenced this pull request Nov 6, 2026
# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	crates/sui-fork/src/tests/store_execution.rs
#	crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs
#	crates/sui-kvstore/src/bigtable/client/mod.rs
#	crates/sui-kvstore/src/bigtable/mock_server.rs
@nickvikeras
nickvikeras force-pushed the nickv/object-max-version branch from 1027630 to d13a7d2 Compare November 6, 2026 21:34
@nickvikeras
nickvikeras force-pushed the nickv/list-object-versions branch from 6c8ef94 to ea58372 Compare November 6, 2026 21:35
nickvikeras added a commit that referenced this pull request Nov 6, 2026
# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	crates/sui-fork/src/tests/store_execution.rs
#	crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs
#	crates/sui-kvstore/src/bigtable/client/mod.rs
#	crates/sui-kvstore/src/bigtable/mock_server.rs
nickvikeras added a commit that referenced this pull request Nov 6, 2026
# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	crates/sui-fork/src/tests/store_execution.rs
#	crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs
#	crates/sui-kvstore/src/bigtable/client/mod.rs
#	crates/sui-kvstore/src/bigtable/mock_server.rs
nickvikeras added a commit that referenced this pull request Nov 6, 2026
# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	crates/sui-fork/src/tests/store_execution.rs
#	crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs
#	crates/sui-kvstore/src/bigtable/client/mod.rs
#	crates/sui-kvstore/src/bigtable/mock_server.rs
@nickvikeras
nickvikeras force-pushed the nickv/object-max-version branch from d13a7d2 to cc08144 Compare November 6, 2026 23:56
@nickvikeras
nickvikeras force-pushed the nickv/list-object-versions branch from ea58372 to 2943e57 Compare November 6, 2026 23:56
nickvikeras added a commit that referenced this pull request Nov 7, 2026
# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	crates/sui-fork/src/tests/store_execution.rs
#	crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs
#	crates/sui-rpc-api/src/reader.rs
nickvikeras added a commit that referenced this pull request Nov 7, 2026
# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	crates/sui-fork/src/tests/store_execution.rs
#	crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs
#	crates/sui-rpc-api/src/reader.rs
nickvikeras added a commit that referenced this pull request Nov 7, 2026
# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	crates/sui-fork/src/tests/store_execution.rs
#	crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs
#	crates/sui-rpc-api/src/reader.rs
nickvikeras added a commit that referenced this pull request Nov 7, 2026
# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	crates/sui-fork/src/tests/store_execution.rs
#	crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs
#	crates/sui-rpc-api/src/reader.rs
nickvikeras added a commit that referenced this pull request Nov 7, 2026
# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	crates/sui-fork/src/tests/store_execution.rs
#	crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs
#	crates/sui-rpc-api/src/reader.rs
@nickvikeras
nickvikeras force-pushed the nickv/object-max-version branch from cc08144 to 859b594 Compare November 7, 2026 15:56
@nickvikeras
nickvikeras force-pushed the nickv/list-object-versions branch from 2943e57 to 9d5a22f Compare November 7, 2026 15:56
nickvikeras added a commit that referenced this pull request Nov 7, 2026
# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	crates/sui-fork/src/tests/store_execution.rs
#	crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs
#	crates/sui-kv-rpc/src/lib.rs
#	crates/sui-kvstore/src/bigtable/client/mod.rs
#	crates/sui-kvstore/src/bigtable/mock_server.rs
#	crates/sui-rpc-api/src/reader.rs
…NTED

ListObjectVersions lists one object's live versions, ascending or descending, inside the intersection of [start_version, end_version), the (after, before) cursors, and versions at or below v_C, the object's version at the end of the bounding checkpoint, deletion and wrap rows included (GetObject's visibility rule). The bounding checkpoint is at_checkpoint, or the served height when it is unset: kvstore_objects is a concurrent pipeline, so rows above its watermark can land out of checkpoint order, and a listing past the served height could list a version while an older one is still being written, which a client paging on from the newer cursor would never see. No v_C means an empty page.

sui-rpc-api: ObjectVersionsQuery parses a request for any backend: object_id, read_mask (default object_id,version,digest), options.limit (default 50, clamped to 1..=1000), ordering, and the version range and cursors folded into one half-open range, failing with INVALID_ARGUMENT (FieldInvalid on options.after/options.before) for a malformed cursor or one naming another object. A cursor is a BCS-encoded ObjectVersionCursor, an enum whose one variant is (object_id, version), so a later listing can add a variant and each backend accepts the other's cursors. The fullnode prunes superseded object versions, so its handler answers UNIMPLEMENTED, pointing at archival kv-rpc.

kv-rpc: the handler checks enable-object-at-checkpoint, the flag that also gates GetObject's at_checkpoint and max_version reads (UNIMPLEMENTED when off, for every request, since every listing reads object_versions_by_checkpoint for its bound), and that at_checkpoint is at or below the served height (OUT_OF_RANGE / ABOVE_LATEST_AVAILABLE above it), answers an empty version range without a read, reads v_C, and reads one page with get_live_object_versions over the folded inclusive range, which skips tombstones server-side, so a short page means the range is exhausted. Both reads go through the request's gated client; the new BigTableClient::get_live_object_versions wrapper runs under one objects-stage permit. AtCheckpointReads is shared with the new module, and the kv-rpc test server, mask and history helpers move from get_object's tests to v2::test_utils.

Tests: kv-rpc unit tests over MockBigtableServer check every bound alone and combined against the history in both orders, at_checkpoint before creation and through a wrap, backward-then-forward paging, two cursors, the served-height bound with a newer version written above it, rejections, and the flag. kv_rpc_tests drives ObjectLifecycle through the same bounds with response cursors, and checks rejections and the flag off by default. The fullnode e2e test checks UNIMPLEMENTED.
@nickvikeras
nickvikeras force-pushed the nickv/object-max-version branch from 859b594 to 3eac85c Compare November 10, 2026 18:19
@nickvikeras
nickvikeras force-pushed the nickv/list-object-versions branch from 9d5a22f to 160c62f Compare November 10, 2026 18:19
Base automatically changed from nickv/object-max-version to nickv/objects-v2 November 10, 2026 18:19
@nickvikeras
nickvikeras merged commit 160c62f into nickv/objects-v2 Nov 10, 2026
16 of 20 checks passed
@nickvikeras
nickvikeras deleted the nickv/list-object-versions branch November 10, 2026 18:19
@nickvikeras
nickvikeras restored the nickv/list-object-versions branch November 10, 2026 18:22
@nickvikeras

Copy link
Copy Markdown
Contributor Author

Superseded by #28632; this was closed as merged into a stack branch by mistake. Nothing reached main.

nickvikeras added a commit that referenced this pull request Nov 10, 2026
## Description

kv-rpc's object history reads (`at_checkpoint`, `max_version`,
`ListObjectVersions`, the running epoch's `system_state`) find an
object's state at a checkpoint with a reverse scan of
`object_versions_by_checkpoint` limited to one row. BigTable answers
that scan by reading every row of its range that is not compacted yet,
so near recently written data a single lookup read 25k rows (a Cetus
pool) to 1.1M rows (the Clock) and took 90–1,600 ms. This PR re-keys the
index so the same lookup is a forward scan that reads only the row it
returns.

## Mini design doc

**Design.** The row key of `object_versions_by_checkpoint` becomes
`object_id ‖ be64(u64::MAX − checkpoint)`, so an object's rows sort
newest checkpoint first. The cells (`v` version, `s` status) and the
pipeline's logic do not change. `get_object_version_at_checkpoint(id,
C)` reads forward from `(id, C)` to `(id, 0)` with `rows_limit` 1; the
first row is the object's state at `C`. Both ends of the range are keys
of the same object, so the read never reaches a neighbouring object's
rows.

**Decisions.**
- *In place, not a `_v2` table.* The table and the
`kvstore_object_versions_by_checkpoint` pipeline keep their names.
Nothing in production reads the table: its only readers are the object
history features in the stack above (#28401, #28434, #28435, #28607),
which are off unless `enable-object-at-checkpoint` is set, and that flag
is an alpha gate. A versioned table would add a schema switch for a
table with no reader to switch.
- *`objects` stays as it is.* Latest and exact-version reads keep the
`objects` table (`object_id ‖ be64(version)`), the
`kvstore_object_tombstones` pipeline and the tombstone-aware reads. An
`objects_v2` table keyed newest-first, which makes latest, `max_version`
and descending `ListObjectVersions` reads forward scans too, is in
#28612. That PR is optional: it costs a ~4 TB backfill for every
operator, and nothing in this PR or the stack depends on it.

**Surprises.** Measured on mainnet's index during its backfill
(`kv_object_latency_probe.py`, BigTable request stats): a reverse scan
with `rows_limit` 1 read every row of its range in uncompacted data, and
about one block (hundreds to ~3,000 rows) in compacted data, while a
forward scan with `rows_limit` 1 read one row everywhere. At the
backfill frontier the deployed reverse lookup read 25,500 rows / 87–100
ms for a Cetus pool, 39,100 rows / 89 ms for `0x5`, and 1,140,000 rows /
1,584 ms for the Clock; a forward read over the same ranges read 1 row
in 1.2–1.5 ms. Total history length does not matter; the number of the
object's rows in uncompacted data below `C` does. At the frontier one
hot object's `max_version` reads stopped at about 24 req/s, because each
read scanned 30–40k rows on one tablet.

**Impact and rollout.**
- Rows written with the old key do not decode as the new key, and the
two layouts must not mix in one table. Once every writer of
`kvstore_object_versions_by_checkpoint` runs an image with this change,
or is stopped: drop and recreate `object_versions_by_checkpoint`, delete
the `kvstore_object_versions_by_checkpoint` row of `watermark_alt`, and
backfill the pipeline from genesis. kv-rpc instances with
`enable-object-at-checkpoint` must not serve from the table until the
backfill completes (as today).
- No change for anyone who does not run the alpha pipeline.

**Test coverage.**
- Codec: `keys_round_trip` and `keys_sort_checkpoints_newest_first`
cover checkpoints 0, 1, 255, 256, 2^32, `u64::MAX − 1` and `u64::MAX`,
and that the keys of checkpoints 0 and `u64::MAX` end in `0xff…` and
`0x00…`.
- Read shape: `object_version_at_checkpoint_is_one_forward_row` (mock
BigTable) checks every checkpoint from 0 to 12 and at `u64::MAX − 1` /
`u64::MAX` against a model, with neighbouring objects' rows at
checkpoints 0, 1 and `u64::MAX`, and that each lookup is one forward
ReadRows with `rows_limit` 1 over `[(id, C), (id, 0)]`. The mock now
records each call's row ranges, `rows_limit` and direction.
- Emulator: `object_version_at_checkpoint_stays_within_its_object`
checks that a missing object between two neighbours resolves to nothing
at any checkpoint, and the neighbours to their newest row at or below
it.
- End to end: `test_object_versions_by_checkpoint` drives an object
through creation, mutation, wrap, unwrap and deletion and checks every
checkpoint, now also `u64::MAX`.
- Not covered: BigTable's actual cost of the forward read; that comes
from the probe numbers above, not from a test.

## Test plan

New tests:
`tables::object_versions_by_checkpoint::tests::{keys_round_trip,
keys_sort_checkpoints_newest_first}`,
`bigtable::client::tests::object_version_at_checkpoint_is_one_forward_row`,
`bigtable_client_emulator_tests::object_version_at_checkpoint_stays_within_its_object`.
Updated: `e2e::test_object_versions_by_checkpoint` (adds checkpoint
`u64::MAX`), and the handler test, which decodes keys with the new
`decode_key`.

---

## Release notes

Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.

For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.

- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] gRPC:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [x] Indexing Framework: The alpha
`kvstore_object_versions_by_checkpoint` pipeline writes
`object_versions_by_checkpoint` with a new row key, `object_id ||
(u64::MAX - checkpoint)`. Operators who run this pipeline must, once
every writer runs the new version or is stopped, drop and recreate the
`object_versions_by_checkpoint` table, delete the pipeline's
`watermark_alt` row, and backfill it from genesis.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant