Skip to content

rpc: serve GetObject max_version, and at_checkpoint as a visibility bound - #28434

Merged
nickvikeras merged 4 commits into
nickv/objects-v2from
nickv/object-max-version
Nov 10, 2026
Merged

nickvikeras merged 4 commits into
nickv/objects-v2from
nickv/object-max-version

Conversation

@nickvikeras

@nickvikeras nickvikeras commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Description

GraphQL's rootVersion reads take an object's newest version at or below a version, and its checkpoint-scoped exact-version reads hide versions written after the viewed checkpoint. Both come from Postgres obj_versions today. This PR serves the new GetObjectRequest.max_version (MystenLabs/sui-apis#42, MystenLabs/sui-rust-sdk#329) from the fullnode and kv-rpc. It also serves at_checkpoint combined with version or max_version as a visibility bound. With version, the answer matches Postgres: NOT_FOUND if that version was written after the checkpoint. With max_version, the server picks the newest version at or below min(max_version, v_K), where v_K is the object's version at the end of checkpoint K, so the answer doesn't depend on anything written after K or on how far the server has indexed. Postgres differs in one case: when the newest version at or below max_version was written after K, Postgres returns null and this RPC returns the older version visible at K. GraphQL's rootVersion with a viewed checkpoint will need to account for that when it moves onto this RPC.

Stacked on #28401.

  • Fullnode: max_version reads find_object_lt_or_eq_version through a new RpcStateReader::get_object_lt_or_eq_version. A version is visible at checkpoint C when it is at or below the object's version at the end of C in the object_version_by_checkpoint index. Simulacrum and the transactional test runner's store record no deletions or wraps. Their max_version returns the newest live version at or below the bound, or INTERNAL when there is none.
  • kv-rpc: max_version is served behind the same gate as at_checkpoint: enable-object-at-checkpoint with object-schema: v2 ([optional][kvstore] objects_v2: newest-first object versions behind object-schema #28612), UNIMPLEMENTED otherwise. A max_version read stops at the deletion and wrap tombstones that kvstore_objects_v2 writes alongside every live version, and is one single-row forward read of objects_v2 from max_version. That pipeline already bounds the served height under v2, so max_version adds no pipeline to it. max_version without at_checkpoint is bounded by the served height, read as at_checkpoint = served height: kvstore_objects_v2 is a concurrent pipeline, so above its watermark a version can be in the table before the tombstone that removed it, and a version-addressed answer looks immutable, so clients would cache it. kv-rpc's bare GetPackage is bounded the same way.
  • kv-rpc latency: in a batch, exact-version reads run alongside the version-index reads instead of after them. A bounded max_version read whose max_version is at or above v_K is answered from the index row (NOT_FOUND for a deletion or wrap, otherwise version v_K through the batched objects_v2 reads), and reads down from max_version only when it is below v_K.
  • sui-fork: max_version resolves against the remote pinned at the fork checkpoint, so it never returns a version created on the live chain after the fork, and cached local versions count only from the object's version at the fork checkpoint upward.

Test plan

  • A kv-rpc unit test checks every version against every checkpoint bound against a model: exact and max_version, single and batch, with and without contents. Unit tests also cover the gate (flag off, and flag on under object-schema: v1) and the served height, including test_max_version_without_checkpoint_is_bounded_by_the_served_height.
  • kv_rpc_tests::test_get_object_max_version_and_checkpoint_bounds runs the same reads against the BigTable emulator over the create, mutate, wrap, unwrap and delete lifecycle. The object skips a version mid-lifecycle, and a max_version inside the gap returns the older version. It also covers the version + max_version rejection.
  • The fullnode simtest get_object_lifecycle adds every-version max_version reads and bounded reads, with the same version gap.
  • sui-fork unit tests: a bounded read never serves a version created after the fork, through the store and through GetObject. The not-yet-indexed and pruned-floor tests cover bounded reads.
  • Contents-free reads of deleted or wrapped objects: kvstore unit and emulator tests, plus a kv-rpc test that a deleted object is NOT_FOUND both unversioned and at its removal version.

Release notes

Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.

For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.

  • Protocol:
  • Nodes (Validators and Full nodes):
  • gRPC: LedgerService.GetObject and BatchGetObjects accept max_version, which returns the object's newest version at or below it, or NOT_FOUND if the object was deleted or wrapped at that version. A request that sets both version and max_version fails with INVALID_ARGUMENT. at_checkpoint can now be combined with version (the read fails with NOT_FOUND if that version was written after the checkpoint) or with max_version (only versions written at or before the checkpoint are considered). The archival kv-rpc service serves max_version only with enable-object-at-checkpoint: true and object-schema: v2, as it does at_checkpoint, and bounds it by the served height.
  • JSON-RPC:
  • GraphQL:
  • CLI:
  • Rust SDK:
  • Indexing Framework:

@nickvikeras
nickvikeras force-pushed the nickv/object-versions-rpc branch 2 times, most recently from 1752a76 to dd84d96 Compare November 3, 2026 00:46
@nickvikeras
nickvikeras force-pushed the nickv/object-max-version branch from bee1392 to 6af2ce0 Compare November 3, 2026 00:46
@nickvikeras
nickvikeras force-pushed the nickv/object-versions-rpc branch from dd84d96 to dd3d532 Compare November 3, 2026 04:34
@nickvikeras
nickvikeras force-pushed the nickv/object-max-version branch from 6af2ce0 to b5f96e8 Compare November 3, 2026 04:34
@nickvikeras
nickvikeras force-pushed the nickv/object-versions-rpc branch from dd3d532 to 45e4f3f Compare November 3, 2026 16:30
@nickvikeras
nickvikeras force-pushed the nickv/object-max-version branch from b5f96e8 to bf03c51 Compare November 3, 2026 16:30
@nickvikeras
nickvikeras force-pushed the nickv/object-versions-rpc branch from 45e4f3f to a52f497 Compare November 3, 2026 16:40
@nickvikeras
nickvikeras force-pushed the nickv/object-max-version branch from bf03c51 to 7f0e185 Compare November 3, 2026 16:40
@nickvikeras
nickvikeras force-pushed the nickv/object-versions-rpc branch from a52f497 to 4bdc9ae Compare November 4, 2026 15:26
@nickvikeras
nickvikeras force-pushed the nickv/object-max-version branch from 7f0e185 to 6f8051b Compare November 4, 2026 15:27
@nickvikeras
nickvikeras force-pushed the nickv/object-versions-rpc branch from 4bdc9ae to 34ec19e Compare November 4, 2026 17:00
@nickvikeras
nickvikeras force-pushed the nickv/object-max-version branch from 6f8051b to 3c56b6d Compare November 4, 2026 17:00
@nickvikeras
nickvikeras force-pushed the nickv/object-versions-rpc branch from 34ec19e to 5183c9d Compare November 4, 2026 21:30
@nickvikeras
nickvikeras force-pushed the nickv/object-max-version branch from 3c56b6d to ff00b53 Compare November 4, 2026 21:30
@nickvikeras
nickvikeras force-pushed the nickv/object-versions-rpc branch from 5183c9d to 1d2504d Compare November 5, 2026 00:05
@nickvikeras
nickvikeras force-pushed the nickv/object-max-version branch from ff00b53 to 1027630 Compare November 6, 2026 19:42
@nickvikeras
nickvikeras force-pushed the nickv/object-versions-rpc branch 2 times, most recently from 2090827 to d533742 Compare November 6, 2026 21:34
@nickvikeras
nickvikeras force-pushed the nickv/object-max-version branch from 1027630 to d13a7d2 Compare November 6, 2026 21:34
@nickvikeras
nickvikeras force-pushed the nickv/object-versions-rpc branch from d533742 to 8c6b8b8 Compare November 6, 2026 23:56
@nickvikeras
nickvikeras force-pushed the nickv/object-max-version branch from d13a7d2 to cc08144 Compare November 6, 2026 23:56
@nickvikeras
nickvikeras force-pushed the nickv/object-versions-rpc branch from 8c6b8b8 to b4634f1 Compare November 7, 2026 15:56
@nickvikeras
nickvikeras force-pushed the nickv/object-max-version branch from cc08144 to 859b594 Compare November 7, 2026 15:56
An object's newest version at or below a version: None when there is none, or when the object was deleted or wrapped at that version. Every implementor serves it:

- RestReadStore and RpcStoreReadStore read it through the execution cache's find_object_lt_or_eq_version, which stops at deletion and wrap markers.
- RpcStoreReader runs a bounded reverse scan of the rpc-store objects CF, which also stops at tombstones.
- ForkStore resolves through its overlay and local store, then the remote pinned at the fork checkpoint: it answers with the object's version at the fork checkpoint when that is within the bound, and otherwise asks the remote for the newest version at or below the bound as of the fork checkpoint. Cached local versions count only from the object's earliest version pinned at the fork checkpoint upward.
- Simulacrum and the transactional test runner's persisted store record no deletions or wraps, so they answer only for a live object at or below the bound and fail otherwise, as their bounded child-object reads do.
…ound

GetObjectRequest.max_version reads an object's newest version at or below it: NOT_FOUND when there is none, or when the object was deleted or wrapped at that version. version and max_version are mutually exclusive (INVALID_ARGUMENT on max_version, nested at requests[i]). at_checkpoint now combines with either as a visibility bound: the selected version is NOT_FOUND unless it was written at or before the checkpoint, with no fallback to an older version. Version R is visible at checkpoint C iff R is at or below the object's version at the end of C, deletion and wrap rows included, from the index at_checkpoint already reads; an object with no version by C has none visible. The at_checkpoint limits (OUT_OF_RANGE above the indexed tip and below the pruned floor, UNAVAILABLE while history backfills or on nodes without object version history) apply to the bound too. A max_version read without at_checkpoint is also UNAVAILABLE on nodes without object version history, since it reads versions later transactions replaced.

validate_get_object_requests takes the GetObjectRequests themselves. ObjectSelector's Version and MaxVersion variants carry the optional bound, and ObjectSelector::not_found builds the NOT_FOUND error, which can now name a max_version.

The fullnode resolves max_version through RpcStateReader::get_object_lt_or_eq_version, checks the at_checkpoint upper bound before any read and the lower bound after, and checks visibility with StateReader::is_object_version_visible_at_checkpoint.

kv-rpc serves max_version behind the existing enable-object-at-checkpoint flag (Unimplemented when off), which now also folds the alpha kvstore_object_tombstones pipeline into the served height and requires it backfilled from genesis. The two object pipelines are deployed and backfilled together, so one flag covers both. A single read looks up the bound's version alongside the object read. A batch adds each bound's (object, checkpoint) to the stage 1 index scans, reads bounded exact versions in the stage 2 objects multi-gets, and runs one gated objects reverse scan per distinct (object, max_version), at most MAX_CONCURRENT_UNVERSIONED_SCANS at a time. Items whose at_checkpoint is above the served height read nothing.

Tests: version + at_checkpoint is no longer rejected, so the kv_rpc_tests rejection checks move to version + max_version, and test_get_object_at_checkpoint_disabled_by_default becomes test_get_object_at_checkpoint_and_max_version_disabled_by_default. The fullnode lifecycle test, now get_object_lifecycle, also covers max_version and both bounds. The kv-rpc test server seeds its watermarks through the shared seed_watermarks helper, which takes the pipeline list.
…visible at the checkpoint

With max_version and at_checkpoint K, both servers now return the newest version at or below max_version among the versions written in checkpoints at or before K: the newest version at or below min(max_version, v_K), where v_K is the object's version at the end of K from the index at_checkpoint already reads (object_version_by_checkpoint on the fullnode, object_versions_by_checkpoint on kv-rpc). NOT_FOUND if the object had no version by K or the selected version is a deletion or wrap. Previously the server selected the newest version at or below max_version among every version it held and failed with NOT_FOUND if that version was written after K, so the answer for a fixed K depended on data written after K and on how far the server had indexed.

kv-rpc also bounds a max_version read without at_checkpoint by its served height, reading it as though at_checkpoint were the served height, the way its bare GetPackage is bounded. kvstore_objects is a concurrent pipeline, so a version written above the served height can be in the table before its deletion or wrap tombstone is, and an unbounded read could answer with it; an answer addressed by version looks immutable, so clients cache it. This costs one object_versions_by_checkpoint scan per read.

Unchanged: version with at_checkpoint, at_checkpoint alone, max_version alone on the fullnode, the checkpoint limit errors, and kv-rpc's flag (enable-object-at-checkpoint gates every max_version read).

Fullnode: the MaxVersion path reads v_K through StateReader::get_object_version_at_checkpoint (which get_object_at_checkpoint and the version visibility check also use) and scans down from min(max_version, v_K). Stores without rpc indexes (simulacrum, the transactional test runner) keep refusing any at_checkpoint read with UNAVAILABLE before reading.

kv-rpc: at_checkpoint and max_version reads share one rule (bounded_read), with at_checkpoint alone read as max_version u64::MAX. Given the index row (V, status) at the bounding checkpoint and max_version M: no row is NOT_FOUND; M < V scans the objects table down from M; otherwise a deletion or wrap at V is NOT_FOUND with no further read, and a live V is read like an at_checkpoint item (an objects multi-get, or straight from the index for a mask of only object_id and version). A single read looks up V, then reads. A batch starts the exact-version multi-gets alongside the stage 1 index scans, since they do not depend on them, and after stage 1 reads the live versions no exact-version read covers in multi-gets, alongside one reverse scan per distinct (object, scan start).

Tests: the kv-rpc model test, kv_rpc_tests::test_get_object_max_version_and_checkpoint_bounds and the fullnode get_object_lifecycle model the new rule, and the no-fallback cases now expect the older version visible at the checkpoint. test_max_version_without_checkpoint_is_bounded_by_the_served_height checks that a version written above kv-rpc's served height never answers an unbounded max_version read. kv_rpc_tests' flag-off test keeps only the at_checkpoint read (test_get_object_at_checkpoint_disabled_by_default); the kv-rpc unit flag test covers the other shapes.
@nickvikeras
nickvikeras force-pushed the nickv/object-versions-rpc branch from b4634f1 to a6e9581 Compare November 10, 2026 18:18
@nickvikeras
nickvikeras force-pushed the nickv/object-max-version branch from 859b594 to 3eac85c Compare November 10, 2026 18:19
Base automatically changed from nickv/object-versions-rpc to nickv/objects-v2 November 10, 2026 18:19
@nickvikeras
nickvikeras merged commit 3eac85c into nickv/objects-v2 Nov 10, 2026
18 of 20 checks passed
@nickvikeras
nickvikeras deleted the nickv/object-max-version branch November 10, 2026 18:19
@nickvikeras
nickvikeras restored the nickv/object-max-version branch November 10, 2026 18:22
@nickvikeras

Copy link
Copy Markdown
Contributor Author

Superseded by #28631; this was closed as merged into a stack branch by mistake. Nothing reached main.

nickvikeras added a commit that referenced this pull request Nov 10, 2026
## Description

kv-rpc's object history reads (`at_checkpoint`, `max_version`,
`ListObjectVersions`, the running epoch's `system_state`) find an
object's state at a checkpoint with a reverse scan of
`object_versions_by_checkpoint` limited to one row. BigTable answers
that scan by reading every row of its range that is not compacted yet,
so near recently written data a single lookup read 25k rows (a Cetus
pool) to 1.1M rows (the Clock) and took 90–1,600 ms. This PR re-keys the
index so the same lookup is a forward scan that reads only the row it
returns.

## Mini design doc

**Design.** The row key of `object_versions_by_checkpoint` becomes
`object_id ‖ be64(u64::MAX − checkpoint)`, so an object's rows sort
newest checkpoint first. The cells (`v` version, `s` status) and the
pipeline's logic do not change. `get_object_version_at_checkpoint(id,
C)` reads forward from `(id, C)` to `(id, 0)` with `rows_limit` 1; the
first row is the object's state at `C`. Both ends of the range are keys
of the same object, so the read never reaches a neighbouring object's
rows.

**Decisions.**
- *In place, not a `_v2` table.* The table and the
`kvstore_object_versions_by_checkpoint` pipeline keep their names.
Nothing in production reads the table: its only readers are the object
history features in the stack above (#28401, #28434, #28435, #28607),
which are off unless `enable-object-at-checkpoint` is set, and that flag
is an alpha gate. A versioned table would add a schema switch for a
table with no reader to switch.
- *`objects` stays as it is.* Latest and exact-version reads keep the
`objects` table (`object_id ‖ be64(version)`), the
`kvstore_object_tombstones` pipeline and the tombstone-aware reads. An
`objects_v2` table keyed newest-first, which makes latest, `max_version`
and descending `ListObjectVersions` reads forward scans too, is in
#28612. That PR is optional: it costs a ~4 TB backfill for every
operator, and nothing in this PR or the stack depends on it.

**Surprises.** Measured on mainnet's index during its backfill
(`kv_object_latency_probe.py`, BigTable request stats): a reverse scan
with `rows_limit` 1 read every row of its range in uncompacted data, and
about one block (hundreds to ~3,000 rows) in compacted data, while a
forward scan with `rows_limit` 1 read one row everywhere. At the
backfill frontier the deployed reverse lookup read 25,500 rows / 87–100
ms for a Cetus pool, 39,100 rows / 89 ms for `0x5`, and 1,140,000 rows /
1,584 ms for the Clock; a forward read over the same ranges read 1 row
in 1.2–1.5 ms. Total history length does not matter; the number of the
object's rows in uncompacted data below `C` does. At the frontier one
hot object's `max_version` reads stopped at about 24 req/s, because each
read scanned 30–40k rows on one tablet.

**Impact and rollout.**
- Rows written with the old key do not decode as the new key, and the
two layouts must not mix in one table. Once every writer of
`kvstore_object_versions_by_checkpoint` runs an image with this change,
or is stopped: drop and recreate `object_versions_by_checkpoint`, delete
the `kvstore_object_versions_by_checkpoint` row of `watermark_alt`, and
backfill the pipeline from genesis. kv-rpc instances with
`enable-object-at-checkpoint` must not serve from the table until the
backfill completes (as today).
- No change for anyone who does not run the alpha pipeline.

**Test coverage.**
- Codec: `keys_round_trip` and `keys_sort_checkpoints_newest_first`
cover checkpoints 0, 1, 255, 256, 2^32, `u64::MAX − 1` and `u64::MAX`,
and that the keys of checkpoints 0 and `u64::MAX` end in `0xff…` and
`0x00…`.
- Read shape: `object_version_at_checkpoint_is_one_forward_row` (mock
BigTable) checks every checkpoint from 0 to 12 and at `u64::MAX − 1` /
`u64::MAX` against a model, with neighbouring objects' rows at
checkpoints 0, 1 and `u64::MAX`, and that each lookup is one forward
ReadRows with `rows_limit` 1 over `[(id, C), (id, 0)]`. The mock now
records each call's row ranges, `rows_limit` and direction.
- Emulator: `object_version_at_checkpoint_stays_within_its_object`
checks that a missing object between two neighbours resolves to nothing
at any checkpoint, and the neighbours to their newest row at or below
it.
- End to end: `test_object_versions_by_checkpoint` drives an object
through creation, mutation, wrap, unwrap and deletion and checks every
checkpoint, now also `u64::MAX`.
- Not covered: BigTable's actual cost of the forward read; that comes
from the probe numbers above, not from a test.

## Test plan

New tests:
`tables::object_versions_by_checkpoint::tests::{keys_round_trip,
keys_sort_checkpoints_newest_first}`,
`bigtable::client::tests::object_version_at_checkpoint_is_one_forward_row`,
`bigtable_client_emulator_tests::object_version_at_checkpoint_stays_within_its_object`.
Updated: `e2e::test_object_versions_by_checkpoint` (adds checkpoint
`u64::MAX`), and the handler test, which decodes keys with the new
`decode_key`.

---

## Release notes

Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.

For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.

- [ ] Protocol:
- [ ] Nodes (Validators and Full nodes):
- [ ] gRPC:
- [ ] JSON-RPC:
- [ ] GraphQL:
- [ ] CLI:
- [ ] Rust SDK:
- [x] Indexing Framework: The alpha
`kvstore_object_versions_by_checkpoint` pipeline writes
`object_versions_by_checkpoint` with a new row key, `object_id ||
(u64::MAX - checkpoint)`. Operators who run this pipeline must, once
every writer runs the new version or is stopped, drop and recreate the
`object_versions_by_checkpoint` table, delete the pipeline's
`watermark_alt` row, and backfill it from genesis.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant