Repository navigation
kv-rpc: serve the running epoch's system_state at the served checkpoint - #28607
Open
nickvikeras wants to merge 2 commits into
Open
nickvikeras wants to merge 2 commits into
nickvikeras wants to merge 2 commits into
Conversation
1 of 8 tasks
nickvikeras
added a commit
that referenced
this pull request
Nov 6, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kv-rpc/src/config.rs # crates/sui-kv-rpc/src/lib.rs # crates/sui-kv-rpc/src/v2/get_epoch.rs # crates/sui-kv-rpc/src/v2/mod.rs
nickvikeras
added a commit
that referenced
this pull request
Nov 6, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kv-rpc/src/config.rs # crates/sui-kv-rpc/src/lib.rs # crates/sui-kv-rpc/src/v2/get_epoch.rs # crates/sui-kv-rpc/src/v2/mod.rs
nickvikeras
force-pushed
the
nickv/object-versions-rpc
branch
from
November 6, 2026 23:56
d533742 to
8c6b8b8
Compare
nickvikeras
force-pushed
the
nickv/epoch-system-state
branch
from
November 6, 2026 23:59
fab526a to
c6f71dc
Compare
nickvikeras
added a commit
that referenced
this pull request
Nov 7, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kv-rpc/src/config.rs # crates/sui-kv-rpc/src/lib.rs # crates/sui-kv-rpc/src/v2/get_epoch.rs # crates/sui-kv-rpc/src/v2/mod.rs
nickvikeras
added a commit
that referenced
this pull request
Nov 7, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kv-rpc/src/config.rs # crates/sui-kv-rpc/src/lib.rs # crates/sui-kv-rpc/src/v2/get_epoch.rs # crates/sui-kv-rpc/src/v2/mod.rs
nickvikeras
added a commit
that referenced
this pull request
Nov 7, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kv-rpc/src/config.rs # crates/sui-kv-rpc/src/lib.rs # crates/sui-kv-rpc/src/v2/get_epoch.rs # crates/sui-kv-rpc/src/v2/mod.rs
nickvikeras
added a commit
that referenced
this pull request
Nov 7, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kv-rpc/src/config.rs # crates/sui-kv-rpc/src/lib.rs # crates/sui-kv-rpc/src/v2/get_epoch.rs # crates/sui-kv-rpc/src/v2/mod.rs
nickvikeras
added a commit
that referenced
this pull request
Nov 7, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-indexer-alt-e2e-tests/tests/get_epoch_parity_tests.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kv-rpc/src/config.rs # crates/sui-kv-rpc/src/lib.rs # crates/sui-kv-rpc/src/v2/get_epoch.rs # crates/sui-kv-rpc/src/v2/mod.rs
nickvikeras
force-pushed
the
nickv/object-versions-rpc
branch
from
November 7, 2026 15:56
8c6b8b8 to
b4634f1
Compare
nickvikeras
force-pushed
the
nickv/epoch-system-state
branch
from
November 7, 2026 15:56
c6f71dc to
5e4597e
Compare
2 of 8 tasks
nickvikeras
added a commit
that referenced
this pull request
Nov 7, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-indexer-alt-e2e-tests/tests/get_epoch_parity_tests.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kv-rpc/src/config.rs # crates/sui-kv-rpc/src/lib.rs # crates/sui-kv-rpc/src/v2/get_epoch.rs # crates/sui-kv-rpc/src/v2/mod.rs
1 of 8 tasks
nickvikeras
force-pushed
the
nickv/object-versions-rpc
branch
from
November 10, 2026 18:18
b4634f1 to
a6e9581
Compare
nickvikeras
force-pushed
the
nickv/epoch-system-state
branch
from
November 10, 2026 18:19
5e4597e to
3906797
Compare
Base automatically changed from
nickv/object-versions-rpc
to
nickv/objects-v2
November 10, 2026 18:19
nickvikeras
changed the base branch from
nickv/objects-v2
to
nickv/object-versions-rpc
November 10, 2026 18:22
nickvikeras
added a commit
that referenced
this pull request
Nov 10, 2026
## Description kv-rpc's object history reads (`at_checkpoint`, `max_version`, `ListObjectVersions`, the running epoch's `system_state`) find an object's state at a checkpoint with a reverse scan of `object_versions_by_checkpoint` limited to one row. BigTable answers that scan by reading every row of its range that is not compacted yet, so near recently written data a single lookup read 25k rows (a Cetus pool) to 1.1M rows (the Clock) and took 90–1,600 ms. This PR re-keys the index so the same lookup is a forward scan that reads only the row it returns. ## Mini design doc **Design.** The row key of `object_versions_by_checkpoint` becomes `object_id ‖ be64(u64::MAX − checkpoint)`, so an object's rows sort newest checkpoint first. The cells (`v` version, `s` status) and the pipeline's logic do not change. `get_object_version_at_checkpoint(id, C)` reads forward from `(id, C)` to `(id, 0)` with `rows_limit` 1; the first row is the object's state at `C`. Both ends of the range are keys of the same object, so the read never reaches a neighbouring object's rows. **Decisions.** - *In place, not a `_v2` table.* The table and the `kvstore_object_versions_by_checkpoint` pipeline keep their names. Nothing in production reads the table: its only readers are the object history features in the stack above (#28401, #28434, #28435, #28607), which are off unless `enable-object-at-checkpoint` is set, and that flag is an alpha gate. A versioned table would add a schema switch for a table with no reader to switch. - *`objects` stays as it is.* Latest and exact-version reads keep the `objects` table (`object_id ‖ be64(version)`), the `kvstore_object_tombstones` pipeline and the tombstone-aware reads. An `objects_v2` table keyed newest-first, which makes latest, `max_version` and descending `ListObjectVersions` reads forward scans too, is in #28612. That PR is optional: it costs a ~4 TB backfill for every operator, and nothing in this PR or the stack depends on it. **Surprises.** Measured on mainnet's index during its backfill (`kv_object_latency_probe.py`, BigTable request stats): a reverse scan with `rows_limit` 1 read every row of its range in uncompacted data, and about one block (hundreds to ~3,000 rows) in compacted data, while a forward scan with `rows_limit` 1 read one row everywhere. At the backfill frontier the deployed reverse lookup read 25,500 rows / 87–100 ms for a Cetus pool, 39,100 rows / 89 ms for `0x5`, and 1,140,000 rows / 1,584 ms for the Clock; a forward read over the same ranges read 1 row in 1.2–1.5 ms. Total history length does not matter; the number of the object's rows in uncompacted data below `C` does. At the frontier one hot object's `max_version` reads stopped at about 24 req/s, because each read scanned 30–40k rows on one tablet. **Impact and rollout.** - Rows written with the old key do not decode as the new key, and the two layouts must not mix in one table. Once every writer of `kvstore_object_versions_by_checkpoint` runs an image with this change, or is stopped: drop and recreate `object_versions_by_checkpoint`, delete the `kvstore_object_versions_by_checkpoint` row of `watermark_alt`, and backfill the pipeline from genesis. kv-rpc instances with `enable-object-at-checkpoint` must not serve from the table until the backfill completes (as today). - No change for anyone who does not run the alpha pipeline. **Test coverage.** - Codec: `keys_round_trip` and `keys_sort_checkpoints_newest_first` cover checkpoints 0, 1, 255, 256, 2^32, `u64::MAX − 1` and `u64::MAX`, and that the keys of checkpoints 0 and `u64::MAX` end in `0xff…` and `0x00…`. - Read shape: `object_version_at_checkpoint_is_one_forward_row` (mock BigTable) checks every checkpoint from 0 to 12 and at `u64::MAX − 1` / `u64::MAX` against a model, with neighbouring objects' rows at checkpoints 0, 1 and `u64::MAX`, and that each lookup is one forward ReadRows with `rows_limit` 1 over `[(id, C), (id, 0)]`. The mock now records each call's row ranges, `rows_limit` and direction. - Emulator: `object_version_at_checkpoint_stays_within_its_object` checks that a missing object between two neighbours resolves to nothing at any checkpoint, and the neighbours to their newest row at or below it. - End to end: `test_object_versions_by_checkpoint` drives an object through creation, mutation, wrap, unwrap and deletion and checks every checkpoint, now also `u64::MAX`. - Not covered: BigTable's actual cost of the forward read; that comes from the probe numbers above, not from a test. ## Test plan New tests: `tables::object_versions_by_checkpoint::tests::{keys_round_trip, keys_sort_checkpoints_newest_first}`, `bigtable::client::tests::object_version_at_checkpoint_is_one_forward_row`, `bigtable_client_emulator_tests::object_version_at_checkpoint_stays_within_its_object`. Updated: `e2e::test_object_versions_by_checkpoint` (adds checkpoint `u64::MAX`), and the handler test, which decodes keys with the new `decode_key`. --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] gRPC: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [x] Indexing Framework: The alpha `kvstore_object_versions_by_checkpoint` pipeline writes `object_versions_by_checkpoint` with a new row key, `object_id || (u64::MAX - checkpoint)`. Operators who run this pipeline must, once every writer runs the new version or is stopped, drop and recreate the `object_versions_by_checkpoint` table, delete the pipeline's `watermark_alt` row, and backfill it from genesis.
nickvikeras
added a commit
that referenced
this pull request
Nov 10, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-indexer-alt-e2e-tests/tests/get_epoch_parity_tests.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kv-rpc/src/config.rs # crates/sui-kv-rpc/src/lib.rs # crates/sui-kv-rpc/src/v2/get_epoch.rs # crates/sui-kv-rpc/src/v2/mod.rs
nickvikeras
marked this pull request as ready for review
November 11, 2026 00:59
nickvikeras
force-pushed
the
nickv/epoch-system-state
branch
from
November 11, 2026 01:21
3906797 to
3da9767
Compare
nickvikeras
force-pushed
the
nickv/object-versions-rpc
branch
from
November 11, 2026 01:21
a6e9581 to
2eae992
Compare
nickvikeras
added a commit
that referenced
this pull request
Nov 11, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-indexer-alt-e2e-tests/tests/get_epoch_parity_tests.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kv-rpc/src/config.rs # crates/sui-kv-rpc/src/lib.rs # crates/sui-kv-rpc/src/v2/get_epoch.rs # crates/sui-kv-rpc/src/v2/mod.rs
nickvikeras
added a commit
that referenced
this pull request
Nov 11, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-indexer-alt-e2e-tests/tests/get_epoch_parity_tests.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kv-rpc/src/config.rs # crates/sui-kv-rpc/src/lib.rs # crates/sui-kv-rpc/src/v2/get_epoch.rs # crates/sui-kv-rpc/src/v2/mod.rs
nickvikeras
added a commit
that referenced
this pull request
Nov 12, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-indexer-alt-e2e-tests/tests/get_epoch_parity_tests.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kv-rpc/src/config.rs # crates/sui-kv-rpc/src/lib.rs # crates/sui-kv-rpc/src/v2/get_epoch.rs # crates/sui-kv-rpc/src/v2/mod.rs
nickvikeras
added a commit
that referenced
this pull request
Nov 12, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-indexer-alt-e2e-tests/tests/get_epoch_parity_tests.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kv-rpc/src/config.rs # crates/sui-kv-rpc/src/lib.rs # crates/sui-kv-rpc/src/v2/get_epoch.rs # crates/sui-kv-rpc/src/v2/mod.rs
nickvikeras
added a commit
that referenced
this pull request
Nov 12, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-indexer-alt-e2e-tests/tests/get_epoch_parity_tests.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kv-rpc/src/config.rs # crates/sui-kv-rpc/src/lib.rs # crates/sui-kv-rpc/src/v2/get_epoch.rs # crates/sui-kv-rpc/src/v2/mod.rs
nickvikeras
added a commit
that referenced
this pull request
Nov 12, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-indexer-alt-e2e-tests/tests/get_epoch_parity_tests.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kv-rpc/src/config.rs # crates/sui-kv-rpc/src/lib.rs # crates/sui-kv-rpc/src/v2/get_epoch.rs # crates/sui-kv-rpc/src/v2/mod.rs
nickvikeras
added a commit
that referenced
this pull request
Nov 12, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-indexer-alt-e2e-tests/tests/get_epoch_parity_tests.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kv-rpc/src/config.rs # crates/sui-kv-rpc/src/lib.rs # crates/sui-kv-rpc/src/v2/get_epoch.rs # crates/sui-kv-rpc/src/v2/mod.rs
nickvikeras
added a commit
that referenced
this pull request
Nov 13, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-indexer-alt-e2e-tests/tests/get_epoch_parity_tests.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kv-rpc/src/config.rs # crates/sui-kv-rpc/src/lib.rs # crates/sui-kv-rpc/src/v2/get_epoch.rs # crates/sui-kv-rpc/src/v2/mod.rs
nickvikeras
added a commit
that referenced
this pull request
Nov 13, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-indexer-alt-e2e-tests/tests/get_epoch_parity_tests.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kv-rpc/src/config.rs # crates/sui-kv-rpc/src/lib.rs # crates/sui-kv-rpc/src/v2/get_epoch.rs # crates/sui-kv-rpc/src/v2/mod.rs
nickvikeras
force-pushed
the
nickv/object-versions-rpc
branch
from
November 13, 2026 02:51
2eae992 to
10e06f1
Compare
nickvikeras
force-pushed
the
nickv/epoch-system-state
branch
from
November 13, 2026 04:15
3da9767 to
f064ee1
Compare
nickvikeras
force-pushed
the
nickv/object-versions-rpc
branch
from
November 13, 2026 04:15
10e06f1 to
f8a29aa
Compare
nickvikeras
added a commit
that referenced
this pull request
Nov 13, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-indexer-alt-e2e-tests/tests/get_epoch_parity_tests.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kv-rpc/src/config.rs # crates/sui-kv-rpc/src/lib.rs # crates/sui-kv-rpc/src/v2/get_epoch.rs # crates/sui-kv-rpc/src/v2/mod.rs
nickvikeras
added a commit
that referenced
this pull request
Nov 13, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-indexer-alt-e2e-tests/tests/get_epoch_parity_tests.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kv-rpc/src/config.rs # crates/sui-kv-rpc/src/lib.rs # crates/sui-kv-rpc/src/v2/get_epoch.rs # crates/sui-kv-rpc/src/v2/mod.rs
nickvikeras
added a commit
that referenced
this pull request
Nov 13, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-indexer-alt-e2e-tests/tests/get_epoch_parity_tests.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kv-rpc/src/config.rs # crates/sui-kv-rpc/src/lib.rs # crates/sui-kv-rpc/src/v2/get_epoch.rs # crates/sui-kv-rpc/src/v2/mod.rs
nickvikeras
force-pushed
the
nickv/epoch-system-state
branch
from
November 13, 2026 11:11
f064ee1 to
e806290
Compare
nickvikeras
added a commit
that referenced
this pull request
Nov 13, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-indexer-alt-e2e-tests/tests/get_epoch_parity_tests.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kv-rpc/src/config.rs # crates/sui-kv-rpc/src/lib.rs # crates/sui-kv-rpc/src/v2/get_epoch.rs # crates/sui-kv-rpc/src/v2/mod.rs
nickvikeras
force-pushed
the
nickv/object-versions-rpc
branch
from
November 14, 2026 01:04
93bc0e8 to
1632e3b
Compare
nickvikeras
force-pushed
the
nickv/epoch-system-state
branch
from
November 14, 2026 06:25
e806290 to
d4c44ab
Compare
nickvikeras
force-pushed
the
nickv/object-versions-rpc
branch
from
November 14, 2026 06:25
1632e3b to
7216bc0
Compare
nickvikeras
added a commit
that referenced
this pull request
Nov 14, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-indexer-alt-e2e-tests/tests/get_epoch_parity_tests.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kv-rpc/src/bigtable_client.rs # crates/sui-kv-rpc/src/config.rs # crates/sui-kv-rpc/src/lib.rs # crates/sui-kv-rpc/src/v2/get_epoch.rs # crates/sui-kv-rpc/src/v2/mod.rs # crates/sui-rpc-api/src/grpc/v2/ledger_service/get_epoch.rs
nickvikeras
force-pushed
the
nickv/epoch-system-state
branch
from
November 14, 2026 18:50
d4c44ab to
9486389
Compare
nickvikeras
force-pushed
the
nickv/object-versions-rpc
branch
from
November 14, 2026 18:50
7216bc0 to
008a861
Compare
nickvikeras
changed the base branch from
nickv/object-versions-rpc
to
nickv/object-removed-error
November 14, 2026 18:50
nickvikeras
added a commit
that referenced
this pull request
Nov 14, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-fork/src/tests/store_execution.rs # crates/sui-indexer-alt-e2e-tests/tests/get_epoch_parity_tests.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kv-rpc/src/bigtable_client.rs # crates/sui-kv-rpc/src/v2/get_epoch.rs # crates/sui-kv-rpc/src/v2/mod.rs # crates/sui-kvstore/src/bigtable/client/mod.rs # crates/sui-kvstore/src/bigtable/mock_server.rs # crates/sui-rpc-api/src/grpc/v2/ledger_service/get_epoch.rs # crates/sui-rpc-api/src/grpc/v2/ledger_service/mod.rs # crates/sui-rpc-api/src/lib.rs # crates/sui-types/src/storage/read_store.rs
nickvikeras
force-pushed
the
nickv/object-removed-error
branch
from
November 15, 2026 02:09
2a6712c to
b61901e
Compare
nickvikeras
force-pushed
the
nickv/epoch-system-state
branch
from
November 15, 2026 02:09
9486389 to
1578347
Compare
nickvikeras
added a commit
that referenced
this pull request
Nov 15, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-fork/src/tests/store_execution.rs # crates/sui-indexer-alt-e2e-tests/tests/get_epoch_parity_tests.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kv-rpc/src/bigtable_client.rs # crates/sui-kv-rpc/src/v2/get_epoch.rs # crates/sui-kv-rpc/src/v2/mod.rs # crates/sui-kvstore/src/bigtable/client/mod.rs # crates/sui-kvstore/src/bigtable/mock_server.rs # crates/sui-rpc-api/src/grpc/v2/ledger_service/get_epoch.rs # crates/sui-rpc-api/src/grpc/v2/ledger_service/mod.rs # crates/sui-rpc-api/src/lib.rs # crates/sui-types/src/storage/read_store.rs
GraphQL's Epoch reads move from Postgres to kv-rpc, so GetEpoch serves the end-of-epoch data GraphQL reads there, from both the fullnode and kv-rpc: transaction_end_exclusive, ended_in_safe_mode, system_epoch_info and epoch_commitments come from the stored epoch rows (rpc-store EpochInfo now carries them) and are set exactly when last_checkpoint is; transaction_start_inclusive comes from the previous epoch's end. kv-rpc reads epochs through the request's limited client. The fullnode's latest epoch is the live system state's, and its end-of-epoch fields appear as soon as the rpc-store row has them. kv-rpc answers as of its served height: an epoch that ends above it has not ended yet. The rpc-store merge no longer lets a mid-epoch restore seed overwrite a known epoch-start state, and the fullnode does not serve a seeded mid-epoch state as a past epoch's system_state. Also bumps sui-rust-sdk to 8ec4d2f (MystenLabs/sui-rust-sdk#344 stacked on #345) and drops the unused CheckpointData::epoch_info.
kv-rpc's GetEpoch serves the running epoch's system_state as of its served checkpoint, as Epoch.system_state documents. It reads the 0x5 wrapper and the inner object its version keys, each as of that checkpoint, the way GetObject's at_checkpoint reads do. The inner version is read at the checkpoint, not taken from the start snapshot, because the first mutable access after a framework upgrade migrates the inner object mid-epoch. At an epoch's last checkpoint kv-rpc reads the state at the checkpoint before: the change-epoch transaction there has already turned the system state into the next epoch's. A client polling the latest epoch sees its state move forward across the boundary, never back to the start snapshot. A past epoch serves its start snapshot. Each read is a single-row forward read of the object_versions_by_checkpoint index, then a point read of objects, through the request's limited client. It needs enable-object-at-checkpoint; without it a request for the running epoch's system_state is refused with UNIMPLEMENTED / NOT_SERVED_HERE, cause disabled_by_config. When system_state is read for the running epoch, committee comes from that same state; committee alone still comes from the start snapshot. The fullnode is unchanged: it serves the live state for its live epoch.
nickvikeras
force-pushed
the
nickv/epoch-system-state
branch
from
November 15, 2026 19:59
1578347 to
ec204dc
Compare
nickvikeras
force-pushed
the
nickv/object-removed-error
branch
from
November 15, 2026 19:59
b61901e to
744b958
Compare
nickvikeras
added a commit
that referenced
this pull request
Nov 15, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-fork/src/tests/store_execution.rs # crates/sui-indexer-alt-e2e-tests/tests/get_epoch_parity_tests.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kv-rpc/src/bigtable_client.rs # crates/sui-kv-rpc/src/v2/get_epoch.rs # crates/sui-kv-rpc/src/v2/mod.rs # crates/sui-kvstore/src/bigtable/client/mod.rs # crates/sui-kvstore/src/bigtable/mock_server.rs # crates/sui-rpc-api/src/grpc/v2/ledger_service/get_epoch.rs # crates/sui-rpc-api/src/grpc/v2/ledger_service/mod.rs # crates/sui-rpc-api/src/lib.rs # crates/sui-types/src/storage/read_store.rs
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
kv-rpc's
LedgerService/GetEpochserves the running epoch'ssystem_stateas of its served checkpoint, asEpoch.system_statedocuments. Before this it returned the epoch-start snapshot. The fullnode is unchanged: its plain reads are live (decision 31 of the min-checkpoint design), so it serves the live state for the running epoch, as on main.Stacked on #28663, the top of the object stack, which provides kv-rpc's checkpoint-pinned object reads. It also carries epoch A (#28600) as its first commit, because this change builds on that code; that commit drops out once #28600 merges. The SDK pin is MystenLabs/sui-rust-sdk#344 stacked on #345 (
8ec4d2f), which this branch needs for both. Review only the last commit.Mini design doc
Design. On kv-rpc the running epoch's
system_stateis the state at its served height H. kv-rpc reads the0x5wrapper and the inner object itsversionkeys, each as of H, asGetObject'sat_checkpointdoes. Each is a single-row forward read of theobject_versions_by_checkpointindex, then a point read ofobjects, through the request's limited BigTable client. A past epoch serves its start snapshot.Decisions.
system_state_version. The first mutable access after a framework upgrade migrates the inner object mid-epoch and deletes the old one.system_stateis read for the running epoch,committeecomes from that same state and the start snapshot is not read; the committee does not change within an epoch.committeealone still comes from the start snapshot.x-sui-min-checkpointyet, so for the running epoch the two servers agree only on a stopped chain once both reach the same height; ended epochs match.Surprises. The system state's inner version can change mid-epoch, in any epoch after a framework upgrade adds a version, not only at genesis.
Impact and rollout.
system_stateneedsenable-object-at-checkpoint, which is off by default. With it off, a request for the served epoch'ssystem_state, including at the epoch's last checkpoint, is refused withUNIMPLEMENTED/NOT_SERVED_HEREand causedisabled_by_config, asGetObject'sat_checkpointreads are under the same flag; clients do not retry it on that instance. The default read mask,committeealone, and past epochs are unaffected.Test coverage. A kv-rpc unit test against a mock BigTable, and e2e tests against the BigTable emulator, including a fullnode/kv-rpc parity test around an epoch change.
Test plan
New tests:
sui-kv-rpc:served_epoch_system_state_is_read_at_the_served_checkpoint: the state at the served checkpoint across a mid-epoch V1→V2 migration, and the state at the checkpoint before at the epoch's last one. Without the flag it is refused withNOT_SERVED_HERE/disabled_by_config; a past epoch's snapshot is served with or without the flag and reads no object versions.sui-indexer-alt-e2e-tests:test_get_epoch_serves_the_running_epochs_system_state: stake added mid-epoch shows in kv-rpc'ssystem_state.test_get_epoch_running_system_state_needs_object_at_checkpoint_reads: with the flag off, the current epoch'ssystem_stateis refused withNOT_SERVED_HERE/disabled_by_config, while itscommitteeand the default read mask are still served.test_get_epoch_matches_between_fullnode_and_kv_rpc(extended): stakes mid-epoch, holds kv-rpc at an epoch's last checkpoint L and at L − 1, checks kv-rpc's latestsystem_stateandcommitteeat both (the state at L − 1, different from the start snapshot), checks that kv-rpc's latest epoch moves on to epoch 1 at epoch 1's first checkpoint, and compares the ended epoch's fields andsystem_statebetween the servers.Release notes
GetEpochreturns the running epoch'ssystem_stateas of its served checkpoint, instead of the epoch-start snapshot. At an epoch's last checkpoint it is the state at the checkpoint before. This needsenable-object-at-checkpoint: true; without it, a request for the running epoch'ssystem_statefails withUNIMPLEMENTEDand anErrorInfoof reasonNOT_SERVED_HERE, causedisabled_by_config. Fullnodes are unchanged.