Skip to content

rpc: stream ListObjectVersions from kv-rpc; fullnodes answer UNIMPLEMENTED - #28632

Draft
nickvikeras wants to merge 1 commit into
nickv/object-max-versionfrom
nickv/list-object-versions
Draft

nickvikeras wants to merge 1 commit into
nickv/object-max-versionfrom
nickv/list-object-versions

Conversation

@nickvikeras

@nickvikeras nickvikeras commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Replaces #28435, which GitHub marked merged into a stack branch by mistake during a rebase; its review history is there.

Description

GraphQL's objectVersions connections page through one object's versions in both directions, bounded by the checkpoint being viewed. They read Postgres obj_versions today. This PR serves the new streaming LedgerService.ListObjectVersions (MystenLabs/sui-apis#43, MystenLabs/sui-rust-sdk#330) from kv-rpc, so a page of up to 1,000 large objects never sits in server memory. A listing is the object's history: its live versions and the versions at which it was deleted or wrapped (removals), so a client can tell a deletion, or a gap while the object was wrapped, from versions it has not been sent. A deletion need not end the history: an object ID can be used again at a higher version, as when a dynamic field is removed and re-added under the same name. Stacked on #28631. The fullnode serves it within its retention window in #28645, stacked on this one.

Mini design doc

Design. The RPC streams one item per frame with its cursor, a live version as object or a deletion or wrap as removal (the fields removal_read_mask selects: version,kind by default, and transaction on request, which is then always set), like ListCheckpoints/ListTransactions/ListEvents, but without a filter, so there is no scan limit and no Watermark. The final frame carries a QueryEnd. sui-rpc-api's ledger_history::object_versions module holds everything both backends must agree on:

  • ObjectVersionsQuery parses the request.
  • bounding_checkpoint picks at_checkpoint, checked against the tip, or the tip itself.
  • plan takes the object's version at that checkpoint and returns the inclusive range to read plus the end reason. It applies the ledger lists' cursor-past-the-tip rule: descending cursors above the tip are rejected, ascending ones are served from this server's view.
  • frames turns ObjectVersionItems (Live { object, version } or Removal { version, kind, transaction }) into frames: ITEM_LIMIT on the limit-th item, live or removed, otherwise a final frame with the plan's reason, or an error, which the fullnode uses at its retention floor.

kv-rpc reads the range as one streaming objects range scan limited to the page's limit, live rows and tombstones alike, listing each tombstone as a removal, like the unfiltered ListCheckpoints scan. Rows are decoded as render_ahead pulls them, at most render-ahead items ahead of the client, so the page never sits in server memory.

Decisions.

  • QueryEnd reasons follow the ledger lists' rules (ResolvedCheckpointRange::from_request, apply_cursor_bounds, starting_cursor_end):
    • CURSOR_BOUND only when the far cursor in listing order binds. That is before at or below the window's end ascending, or after at or above its start descending; a tie goes to the cursor, and after one below start_version is not a tie.
    • LEDGER_TIP when an ascending listing without at_checkpoint reaches the object's newest version before end_version. This holds even from an after cursor at that version, which is how a client follows new versions. An empty range (end_version == start_version) never ends at the tip.
    • CHECKPOINT_BOUND otherwise. end_version exactly one above the newest version counts as CHECKPOINT_BOUND, like end_checkpoint at the tip. As in the ledger lists, an end_version below start_version is INVALID_ARGUMENT.
  • One streaming scan per listing, held open under its permit (open_gated_stream) while the client reads, as the other List APIs' scans are. Permits are per request (limited_client), so a slow client only holds its own request's permit, and the method timeout bounds how long the scan stays open.
  • Rendering of json/contents goes through render_ahead (ledger-history.list-object-versions.render-ahead).
  • Removals count toward the limit and every bound, like live versions (the proto's rule). plan needs no change for them: the object's version at the bounding checkpoint was already its newest version live or not.
  • A mask of only object_id/version reads live rows' keys only. object_versions_stream without contents reads with an Interleave that empties the live-object column's values and keeps the tombstone columns, so removals still carry their kind and transaction; the mock BigTable gains Interleave support for it.
  • Tombstone transactions are read from column t, which sui-kvstore: record the removing transaction in object tombstones #28653 (on main) writes: ObjectRow::Tombstone now carries ObjectTombstone { status, transaction: Option<TransactionDigest> }. removal_read_mask (default version,kind, parsed in the shared ObjectVersionsQuery) selects each removal's fields; frames renders only those. With transaction selected, every removal carries it, and a tombstone without the cell is INTERNAL, because enable-object-at-checkpoint asserts the tombstones were backfilled by a build that writes it; without transaction, such a tombstone lists like any other. The same rows are read either way: the digest sits in the tombstone row the scan already reads. A cell of any length other than 32 bytes is a decode error. sui-kvstore: record the removing transaction in object tombstones #28653's encode_tombstone(status, &transaction) and col::TRANSACTION are used as is; its handler test now expects the decoded ObjectTombstone.
  • ObjectVersionItem::Live holds the rendered Object unboxed with #[allow(clippy::large_enum_variant)]: almost every item is live and moved once into its frame, so boxing would add an allocation per item.
  • The cursor-past-the-tip rule lives in the shared plan and matches the ledger lists (reject_descending_cursors_past_tip, ascending_after_past_tip_end), so the fullnode applies it against its indexed tip. Without at_checkpoint, a cursor is past the tip when it names a version above the object's version there, or any version of an object with none. Descending, it is OUT_OF_RANGE/ABOVE_LATEST_AVAILABLE (bound cursor), since listing from this view would start below versions the server lacks. Ascending, the listing is served from this view: in a window reaching past the tip, an after past the tip lists nothing and ends with LEDGER_TIP (the client resumes from the cursor it holds), or with CURSOR_BOUND against a before at the version after it; a before past the tip leaves the tip as the end.
  • Request limits come from config: ledger-history.list-object-versions, default 50, max 1,000, and the timeout. Validation rejects max-limit-items = 0, which no limit can be clamped into.

Surprises. The keys-only read needs each tombstone's status and transaction, which live in its cell values, so stripping every value (as GetObject's keys-only read does) would lose them; hence the Interleave.

Impact and rollout. Nothing has shipped. kv-rpc still needs enable-object-at-checkpoint; without it every request is UNIMPLEMENTED / NOT_SERVED_HERE (cause disabled_by_config), because each listing reads object_versions_by_checkpoint for its bound. That flag already requires kvstore_object_tombstones backfilled from genesis, and now from a build that writes each removal's transaction (#28653 or later), so removals and their transactions are complete wherever the method is served. Deployments that ran the tombstone pipeline before #28653 must rerun its backfill before enabling the flag; kv-rpc then has no transaction floor: removal transactions start at genesis, so only the object floor applies. Until #28645 lands, the fullnode answers UNIMPLEMENTED / NOT_SERVED_HERE (cause no_object_history) through a streaming stub.

Test coverage.

  • The 1,944-combination model test checks listed items, live versions and removals with their kinds, with cursors at both, and that ITEM_LIMIT appears exactly when a page fills, including on a removal.
  • A table test pins each QueryEnd reason and tie, including ITEM_LIMIT on a wrap and on a deletion, LEDGER_TIP after a deletion, and a live newest version followed from its own cursor. Another lists an object whose only item is a deletion. The model test and the reason table also run over an object deleted and created again, with cursors and limits across the gap.
  • Chunk tests list pages that span several reads in both orders, with removals at and right after a read's last version, and check each read's row limit.
  • Keys-only masks are checked to empty only live objects' values on every read, with removals keeping their kind. test_list_object_versions_renders_the_removal_read_mask checks the default, version-only, kind-only and empty removal masks (removals keep their frames and cursors), a selected transaction being set, and INTERNAL on a tombstone without one; removal_read_mask paths are validated like read_mask.
  • The kv_rpc_tests lifecycle test lists create, mutate, wrap, unwrap and delete (and the wrapper's create and delete) as items with their kinds, through every bound, and checks end reasons against the BigTable emulator. With transaction in removal_read_mask, each wrap and deletion names the transaction that did it, as written by sui-kvstore: record the removing transaction in object tombstones #28653's tombstone pipeline. test_list_object_versions_continues_after_a_deletion removes and re-adds a Table<u64, u64> entry, whose dynamic field keeps its ID, and pages across the deletion in both orders and under at_checkpoint.

Not covered: memory under real large objects (rows the HTTP/2 window buffers ahead of render_ahead), and real BigTable's handling of the Interleave filter beyond the mock.

Test plan

  • kv-rpc unit tests: test_list_object_versions_ends_with_the_reason_of_its_bound, test_list_object_versions_lists_an_object_that_is_only_a_deletion, test_list_object_versions_version_only_masks_read_keys, test_list_object_versions_renders_the_removal_read_mask, plus the updated model, paging, served-height and rejection tests.
  • kvstore object_version_reads_list_tombstones (both reads) and decode_reads_tombstone_transactions.
  • kv_rpc_tests::test_list_object_versions_lifecycle against the BigTable emulator.

Release notes

Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.

For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.

  • Protocol:
  • Nodes (Validators and Full nodes):
  • gRPC: New streaming LedgerService.ListObjectVersions lists one object's history, one item per frame with an opaque cursor, in either direction: each live version as object, and each deletion or wrap as removal. It ends with a QueryEnd reason: ITEM_LIMIT, CURSOR_BOUND, CHECKPOINT_BOUND or LEDGER_TIP. Listings are bounded by start_version/end_version and by at_checkpoint, which defaults to the highest indexed checkpoint. The archival kv-rpc service serves it only with enable-object-at-checkpoint: true; without it, kv-rpc answers UNIMPLEMENTED with a NOT_SERVED_HERE error reason (cause disabled_by_config).
  • JSON-RPC:
  • GraphQL:
  • CLI:
  • Rust SDK:
  • Indexing Framework:

nickvikeras added a commit that referenced this pull request Nov 10, 2026
# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	crates/sui-fork/src/tests/store_execution.rs
#	crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs
#	crates/sui-kvstore/src/bigtable/client/mod.rs
#	crates/sui-kvstore/src/bigtable/mock_server.rs
#	crates/sui-rpc-api/src/reader.rs
@nickvikeras
nickvikeras force-pushed the nickv/object-max-version branch from 3eac85c to 64cb09e Compare November 11, 2026 01:21
@nickvikeras
nickvikeras force-pushed the nickv/list-object-versions branch from 160c62f to 0892ed7 Compare November 11, 2026 01:21
@nickvikeras
nickvikeras force-pushed the nickv/list-object-versions branch 2 times, most recently from df5ad0f to b2f45fd Compare November 11, 2026 02:28
@nickvikeras
nickvikeras force-pushed the nickv/object-max-version branch from 64cb09e to ca6de3b Compare November 11, 2026 02:33
@nickvikeras nickvikeras changed the title rpc: serve ListObjectVersions from kv-rpc; fullnodes answer UNIMPLEMENTED rpc: stream ListObjectVersions from kv-rpc; fullnodes answer UNIMPLEMENTED Nov 11, 2026
@nickvikeras
nickvikeras force-pushed the nickv/list-object-versions branch from b2f45fd to c90a27d Compare November 11, 2026 02:44
nickvikeras added a commit that referenced this pull request Nov 11, 2026
# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	crates/sui-fork/src/tests/store_execution.rs
#	crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs
#	crates/sui-kvstore/src/bigtable/client/mod.rs
#	crates/sui-rpc-api/src/reader.rs
@nickvikeras
nickvikeras force-pushed the nickv/list-object-versions branch 11 times, most recently from 6dc4790 to b24c252 Compare November 13, 2026 04:15
@nickvikeras
nickvikeras force-pushed the nickv/object-max-version branch from ca6de3b to a8c49c2 Compare November 13, 2026 04:15
@nickvikeras
nickvikeras force-pushed the nickv/list-object-versions branch 3 times, most recently from 9a6e75f to 9721a19 Compare November 13, 2026 11:11
@nickvikeras
nickvikeras force-pushed the nickv/object-max-version branch from a8c49c2 to 57d4bc0 Compare November 13, 2026 11:11
@nickvikeras
nickvikeras force-pushed the nickv/list-object-versions branch from 9721a19 to 1912f75 Compare November 14, 2026 01:04
@nickvikeras
nickvikeras force-pushed the nickv/object-max-version branch from 57d4bc0 to 03f6b9c Compare November 14, 2026 01:04
@nickvikeras
nickvikeras force-pushed the nickv/list-object-versions branch 2 times, most recently from ecb0307 to d4a6aaf Compare November 14, 2026 18:50
@nickvikeras
nickvikeras force-pushed the nickv/object-max-version branch 2 times, most recently from 28bd38f to a14241d Compare November 15, 2026 02:09
@nickvikeras
nickvikeras force-pushed the nickv/list-object-versions branch from d4a6aaf to 509c3be Compare November 15, 2026 02:09
…ENTED

ListObjectVersions streams one object's history, one item per frame with its cursor: each live version as `object`, and each version at which it was deleted or wrapped as `removal`, with the fields removal_read_mask selects (default version,kind; transaction on request, and then always set). Items come ascending or descending, inside the intersection of [start_version, end_version), the (after, before) cursors, and the versions written by the bounding checkpoint: at_checkpoint, or the served height when it is unset. kvstore_objects is a concurrent pipeline, so rows above its watermark can land out of checkpoint order; a listing past the served height could list a version while an older one is still being written. Pins sui-rust-sdk to 895350f (#330) for the streaming RPC and removal_read_mask.

sui-rpc-api's ledger_history::object_versions holds what both backends must agree on. ObjectVersionsQuery parses a request: object_id, read_mask (default object_id,version,digest), removal_read_mask (default version,kind), options.limit (caller's default and max), and ordering. A malformed cursor, one naming another object, or an end_version below start_version is INVALID_ARGUMENT. bounding_checkpoint checks at_checkpoint against the indexed tip. plan takes the object's version at the bounding checkpoint and returns the inclusive range to read and the QueryEnd reason, following the ledger lists' rules: CURSOR_BOUND when the far cursor binds, LEDGER_TIP for an ascending listing without at_checkpoint that reaches the newest version before end_version, and CHECKPOINT_BOUND otherwise. Without at_checkpoint, a cursor past the tip (above the object's version at the indexed height) follows the ledger lists: descending, it is OUT_OF_RANGE / ABOVE_LATEST_AVAILABLE (bound "cursor"); ascending, the listing is served from this server's view, and in a window reaching past the tip an after past the tip lists nothing and ends with LEDGER_TIP, or CURSOR_BOUND against a before at the version after it. frames turns ObjectVersionItems (Live { object, version } or Removal { version, kind, transaction }) into frames, rendering only the removal fields removal_read_mask selects, with ITEM_LIMIT on the limit-th item, live or removed, and otherwise a final frame with the plan's reason or an error. Removals count toward the limit and every bound like live versions; plan already took the object's newest version live or not, so it needs no change for them. Cursors are a BCS enum (ObjectVersionCursor), so either backend accepts the other's. lists_removal_transactions tells a backend whether the mask selects transaction; a backend then sets every removal's transaction or fails the listing. The fullnode answers UNIMPLEMENTED through a streaming stub.

kv-rpc needs enable-object-at-checkpoint, since every listing reads object_versions_by_checkpoint for its bound; that flag asserts kvstore_object_tombstones was backfilled from genesis by a build that writes each removal's transaction (#28653 or later), so removals and their transactions are complete and there is no transaction floor: with transaction requested, a tombstone without one is INTERNAL. It reads the range as one streaming objects range scan limited to the page's limit, live rows and tombstones alike (the new BigTableClient::object_versions_stream), listing tombstones as removals, and renders at most render-ahead items beyond what the client has received. A mask of only object_id/version reads live rows' keys only, with an Interleave filter that empties only the live-object column, so tombstones keep their status and transaction. The mock BigTable supports that Interleave.

Tombstone rows' transaction column t (written by #28653) is decoded into ObjectRow::Tombstone(ObjectTombstone { status, transaction }): a missing cell decodes as no transaction, and any length other than 32 bytes is a decode error.

Tests: kv-rpc unit tests over MockBigtableServer cover the 1,944-combination model (items with removal kinds, cursors at removals, and ITEM_LIMIT exactly on full pages), every QueryEnd reason and tie including ITEM_LIMIT on a removal and LEDGER_TIP after a deletion, an object whose only item is a deletion, keys-only masks keeping removal kinds, removal_read_mask (the default, version-only, kind-only and empty masks, transaction selected, and INTERNAL for a tombstone without one), backward-then-forward paging, the served-height bound, rejections, and the flag. kvstore tests cover both all-row reads and tombstone transaction decoding. kv_rpc_tests pages ObjectLifecycle's live versions, wrap, unwrap and deletions through the same bounds, checks end reasons, and checks that each removal names the transaction that did it when removal_read_mask selects transaction. The model and reason tests also cover an object deleted and created again, and kv_rpc_tests removes and re-adds a dynamic field, which keeps its ID, and pages across the deletion.
@nickvikeras
nickvikeras force-pushed the nickv/object-max-version branch from a14241d to 54aaf67 Compare November 15, 2026 19:59
@nickvikeras
nickvikeras force-pushed the nickv/list-object-versions branch from 509c3be to 4806862 Compare November 15, 2026 19:59

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant