Skip to content

rpc: bound checkpoint-pinned reads, and report reads outside the indexed range as OUT_OF_RANGE - #28577

Open
nickvikeras wants to merge 1 commit into
mainfrom
nickv/pinned-read-bounds
Open

nickvikeras wants to merge 1 commit into
mainfrom
nickv/pinned-read-bounds

Conversation

@nickvikeras

@nickvikeras nickvikeras commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Description

The package gRPC work (#28555, #28573, #28429, #28439, #28440, #28561) needs shared bounds for reads pinned to a checkpoint, but not the GetObject at_checkpoint API in #28630. This PR holds those shared pieces so the package PRs can merge without it; #28630 sits on top of this PR and holds only the object API. It also defines the one set of bound types (AtCheckpointUpperBound, AtCheckpointLowerBound) and refusal constructors (not_served_here / not_ready with a cause, on sui-rust-sdk c20455f's NOT_SERVED_HERE and NOT_READY reasons, and below_oldest_retained_version for a version below the pruning floor) that the PRs above use unchanged, and routes every existing fullnode "rpc indexes disabled", "index empty", "restore backfill" and "no object history" refusal through them.

Mini design doc

Design

A read pinned to checkpoint K is answered only if K lies between two bounds that StateReader computes, and is refused otherwise. The refusal is the same on the fullnode and kv-rpc:

  • Upper bound: the highest checkpoint every rpc index has committed (at_checkpoint_upper_bound). It's read before the data, since it only grows. With rpc indexes disabled it returns UNIMPLEMENTED with NOT_SERVED_HERE (cause no_rpc_indexes). Before the indexes commit a checkpoint, or while a restored node is still backfilling index history, it returns UNAVAILABLE with NOT_READY (cause index_empty or restore_backfill) instead of a bound.

  • Lower bound: max(L, RpcStateReader::get_lowest_checkpoint_with_index_history()) − 1, where L is the object store's lowest available checkpoint (at_checkpoint_lower_bound). It's read after the data, since pruning can raise it while the read runs. AtCheckpointLowerBound stores max(L, first checkpoint with index history) and applies the −1 when it checks; pruned() and pruned_floor() give the same floor to reads that are not pinned to a checkpoint.

  • Refusals:

    • K above the upper bound gets OUT_OF_RANGE with ABOVE_LATEST_AVAILABLE.
    • K below the lower bound gets OUT_OF_RANGE with BELOW_EARLIEST_AVAILABLE.

    The ledger List methods already use these errors. kv-rpc builds the upper bound from its served height with the public AtCheckpointUpperBound::new and runs the same check; checkpoint_not_yet_indexed_error stays public for reads that compare the bound themselves, as rpc: serve GetObject/BatchGetObjects at_checkpoint from the fullnode and kv-rpc #28630 does. not_served_here / not_ready and their cause enums are public for the same reason.

  • Compactor nodes: nodes that run the tidehunter objects compactor keep only each object's latest version. On them, reads that need replaced versions return UNIMPLEMENTED with NOT_SERVED_HERE (cause no_object_history) (require_object_version_history).

Users in this PR: GetPackage at_checkpoint, which checks the upper bound before looking the package up, so an unknown package above the tip gets OUT_OF_RANGE on both servers rather than NOT_FOUND. Also v2alpha GetCheckpointObjectProof, whose floor is the higher of the checkpoint and object floors, read after the reads. A proof loads the checkpoint's input objects and the object version it wrote, so it is refused on compactor nodes as well, before any other check; with rpc indexes disabled it returns UNIMPLEMENTED like at_checkpoint_upper_bound, and before the index has covered a checkpoint UNAVAILABLE / NOT_READY (index_empty). The package PRs above and #28630 use the same bounds.

Decisions

  • The lower bound is the lowest available checkpoint L minus 1, not L. L means checkpoint L's input objects are still stored, and those inputs are the state at the end of L−1. Pruning through L−1 deletes only versions replaced at or before L−1, so every version live at the end of L−1 is still stored. Using L would refuse a checkpoint the node can answer. The rpc-store pruner already keeps the index row that answers reads at L−1; this PR corrects its docs and tests to say so.
  • The upper bound is read before the data, the lower bound after. The indexes only add data above the upper bound. Pruning raises the floor before deleting data (the rpc-store commits its retractions and floor before object versions are deleted), and never lowers it. So a read accepted by a floor read after it saw no pruned data.
  • Restore backfill → UNAVAILABLE / NOT_READY, not OUT_OF_RANGE. When a prefix of the chain is pruned (L > 0), a restore loads object_version_by_checkpoint and package_versions floor rows at the restore checkpoint T. Until the backfill reaches T, a pinned read can be wrong at any checkpoint, not just at old ones. No range says what this node can answer, so it refuses everything, and a client retries another replica. When nothing is pruned (L = 0), only the live cohort is restored and the history cohort backfills from genesis with no restore rows; its reads are exact, but the indexed tip is the backfill's height, far below the restored live state, so the gate also counts any history pipeline that has committed below the live cohort's restore target.
  • One fork floor, as "first checkpoint with index history", not a pinned-read floor. A fork's indexes have no history before its fork checkpoint F. So it reports F + 1 through get_lowest_checkpoint_with_index_history (default 0), giving a pinned-read floor of F. The package listing in rpc: list the versions of every package in ListPackageVersions on the fullnode and kv-rpc #28440 uses the same value as its floor (max(L, it)), so one trait method serves both. The fork's object floor stays min(remote floor, F) because it serves pre-fork exact-version and proof reads from the remote chain. Rejected options:
    • Two methods that always differ by one.
    • Raising the object floor, which loses those reads.
    • Forwarding pinned reads below F to the remote, which adds complexity to the fork.
  • OUT_OF_RANGE with an ErrorReason instead of NOT_FOUND. NOT_FOUND can't be told apart from a package or checkpoint that doesn't exist, so a client can't tell "retry later or elsewhere" from "no such thing". This matches the ledger List methods and kv-rpc.
  • Capability refusals are UNIMPLEMENTED with NOT_SERVED_HERE, temporary ones UNAVAILABLE with NOT_READY. A node without rpc indexes, or without object version history, fails the same request the same way every time, and clients (the SDK ledger streams, authenticated events) retry UNAVAILABLE forever; UNIMPLEMENTED tells them to use a differently configured node. The cause keeps it apart from gRPC's "method not found". When both apply, no_object_history is checked first: enabling indexes alone would not fix it. One predicate, AuthorityPerpetualTablesOptions::objects_compactor_enabled, decides compaction for sui-node, the pruner and the read stores, which derive it from the node config. It is always false on RocksDB builds.

Surprises

Impact and rollout

  • No protocol change and no config gating. sui-rust-sdk moves to c20455f, which adds ErrorReason::NotServedHere (8) and ErrorReason::NotReady (9).
  • gRPC behavior change: GetPackage with at_checkpoint and GetCheckpointObjectProof return OUT_OF_RANGE with an ErrorReason where they returned NOT_FOUND. This applies on fullnodes, and above kv-rpc's served height. Clients that match on NOT_FOUND for these cases need updating; the light-client pruned-checkpoint test needed it.
  • Every fullnode refusal for missing rpc indexes ("rpc indexes are disabled": the ledger List methods, GetPackage with a non-latest version or at_checkpoint, ListPackageVersions) moves from UNAVAILABLE to UNIMPLEMENTED with NOT_SERVED_HERE (no_rpc_indexes). An empty rpc index stays UNAVAILABLE on those methods and gains NOT_READY (index_empty). kv-rpc's refusals before it has a served checkpoint are unchanged.
  • New refusals for GetPackage with at_checkpoint on full nodes: UNAVAILABLE / NOT_READY (restore_backfill) while a restore is backfilling index history, and UNIMPLEMENTED / NOT_SERVED_HERE (no_object_history) on compactor nodes (validators and num_epochs_to_retain = 0 on tidehunter builds). GetCheckpointObjectProof returns UNIMPLEMENTED / NOT_SERVED_HERE on compactor nodes (no_object_history, where it returned INTERNAL or NOT_FOUND) and with rpc indexes disabled (no_rpc_indexes, where it returned NOT_FOUND), and UNAVAILABLE / NOT_READY (index_empty) before the index has covered a checkpoint, where it returned NOT_FOUND.
  • After a restore of a pruned node, pinned reads wait for every history pipeline to reach the restore checkpoint T, not only object_version_by_checkpoint and package_versions.
  • sui-fork rejects pinned reads below its fork checkpoint instead of answering them from restore rows.

Test coverage

  • Covered:
    • serving at the last pruned checkpoint and refusing below it, and the exact upper bound (a read at the indexed tip is served, tip + 1 is refused) (fullnode simtests);
    • an unknown package above the tip gets OUT_OF_RANGE, not NOT_FOUND;
    • proofs below the floor get BELOW_EARLIEST_AVAILABLE and a proof at the floor is served, by pruning deterministically (TestCluster::prune_fullnode_checkpoints_for_testing);
    • the backfill gate's flag logic for both restore shapes (unit), and StateReader::at_checkpoint_upper_bound over a real rpc-store through an empty index, an in-progress restore, a completed restore whose backfill is behind, a caught-up backfill, and a live-only restore whose from-genesis history backfill is behind (unit);
    • the lower bound's −1 and pruned_floor at 0, 1 and above (unit);
    • the pruner keeping the anchor at the last pruned checkpoint (unit);
    • fork rejection below the fork checkpoint, and a read at it passing both bounds;
    • a proof with rpc indexes disabled gets UNIMPLEMENTED with NOT_SERVED_HERE (no_rpc_indexes).
  • Could slip past:
    • The prune-during-read race is argued from write order, not exercised by a test. The checkpoint pruner deletes data before writing its watermark, so a proof read in that window can still get NOT_FOUND.
    • The compactor refusal has no test, since tests don't run tidehunter builds. The at_checkpoint e2e tests disable fullnode pruning so they'd still pass on tidehunter.
    • The restore gate is tested on restore rows written directly, not through a real restore.

Test plan

  • Fullnode simtests:
    • GetPackage at_checkpoint is served at the last pruned checkpoint and refused below it, and served at the indexed tip but refused at tip + 1, for a known and an unknown package (get_package.rs).
    • test_pruned_checkpoint_is_below_earliest_available (get_object_inclusion_proof_tests.rs), which also gets a non-inclusion proof at the lowest available checkpoint, and the light-client test_client_pruned_checkpoint_error, which now requires OUT_OF_RANGE with BELOW_EARLIEST_AVAILABLE.
    • test_proof_without_rpc_indexes_is_not_served_here (get_object_inclusion_proof_tests.rs).
  • Unit:
    • the restore backfill gate (history_backfill_pending_until_floors_reach_restore_anchor, history_backfill_pending_until_history_reaches_live_restore);
    • upper_bound_waits_for_index_and_restore_backfill, upper_bound_waits_for_history_backfill_after_live_only_restore and lower_bound_serves_the_checkpoint_before_the_lowest_available in sui-rpc-api's reader.rs;
    • below_oldest_retained_version_names_checkpoint_and_version in sui-rpc-api's query_options.rs;
    • the pruner's retraction at the last pruned checkpoint;
    • test_checkpoint_pinned_reads_below_the_fork_checkpoint_are_rejected in sui-fork, which also reads at the fork checkpoint.

Release notes

Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.

For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.

  • Protocol:
  • Nodes (Validators and Full nodes):
  • gRPC: MovePackageService.GetPackage with at_checkpoint, and v2alpha ProofService.GetCheckpointObjectProof, now fail with OUT_OF_RANGE and an ErrorReason for a checkpoint outside the available range, where they returned NOT_FOUND: ABOVE_LATEST_AVAILABLE above the highest indexed checkpoint, and BELOW_EARLIEST_AVAILABLE below the lowest available one. Full-node refusals for running without rpc indexes ("rpc indexes are disabled") move from UNAVAILABLE to UNIMPLEMENTED with a NOT_SERVED_HERE ErrorInfo (cause = no_rpc_indexes); do not retry them against the same node. An empty rpc index, and GetPackage with at_checkpoint while a restore is still backfilling index history, return UNAVAILABLE with a NOT_READY ErrorInfo (cause = index_empty or restore_backfill). GetCheckpointObjectProof without rpc indexes moves from NOT_FOUND to UNIMPLEMENTED (no_rpc_indexes), and on an empty rpc index from NOT_FOUND to UNAVAILABLE (index_empty). Requests that need replaced object versions (GetPackage with at_checkpoint, GetCheckpointObjectProof) on nodes that run the objects compactor return UNIMPLEMENTED with NOT_SERVED_HERE (cause = no_object_history).
  • JSON-RPC:
  • GraphQL:
  • CLI:
  • Rust SDK:
  • Indexing Framework:

/// The checkpoint after the fork checkpoint. The local indexes record only the fork's own
/// checkpoints. Every pre-fork object and package version in them was loaded as of the fork
/// checkpoint, without the history before it.
fn get_lowest_checkpoint_with_index_history(&self) -> StorageResult<CheckpointSequenceNumber> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is needed because the ReadStore interface changed. It's not really used in the fork though, unless it's called from grpc APIs.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah the rpc server calls this method to check if the requested checkpoint is within the range of it's index. The ForkStore implements it by returning when it was forked which I think is correct.

@nickvikeras
nickvikeras force-pushed the nickv/pinned-read-bounds branch 2 times, most recently from 04f3962 to b042d20 Compare November 6, 2026 00:47

let response = prove_object_at_checkpoint(service, object_id, checkpoint_seq);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The proof loads checkpoint K's objects through get_checkpoint_data, and on compactor nodes the replaced versions are gone while the object floor never moves. Should the proof also call require_object_version_history, so those nodes return UNAVAILABLE instead of Internal?

.reader
.inner()
.indexes()
.ok_or_else(RpcError::not_found)?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With rpc indexes disabled, this returns NOT_FOUND, while at_checkpoint_upper_bound returns UNAVAILABLE. Should we make them consistent?

@nickvikeras
nickvikeras force-pushed the nickv/pinned-read-bounds branch from 486ca3e to c1f4b9b Compare November 14, 2026 06:25
@nickvikeras nickvikeras mentioned this pull request Nov 14, 2026
1 of 8 tasks

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants