Repository navigation
rpc: bound checkpoint-pinned reads, and report reads outside the indexed range as OUT_OF_RANGE - #28577
rpc: bound checkpoint-pinned reads, and report reads outside the indexed range as OUT_OF_RANGE#28577nickvikeras wants to merge 1 commit into
Conversation
| /// The checkpoint after the fork checkpoint. The local indexes record only the fork's own | ||
| /// checkpoints. Every pre-fork object and package version in them was loaded as of the fork | ||
| /// checkpoint, without the history before it. | ||
| fn get_lowest_checkpoint_with_index_history(&self) -> StorageResult<CheckpointSequenceNumber> { |
There was a problem hiding this comment.
This is needed because the ReadStore interface changed. It's not really used in the fork though, unless it's called from grpc APIs.
There was a problem hiding this comment.
Yeah the rpc server calls this method to check if the requested checkpoint is within the range of it's index. The ForkStore implements it by returning when it was forked which I think is correct.
04f3962 to
b042d20
Compare
|
|
||
| let response = prove_object_at_checkpoint(service, object_id, checkpoint_seq); |
There was a problem hiding this comment.
The proof loads checkpoint K's objects through get_checkpoint_data, and on compactor nodes the replaced versions are gone while the object floor never moves. Should the proof also call require_object_version_history, so those nodes return UNAVAILABLE instead of Internal?
| .reader | ||
| .inner() | ||
| .indexes() | ||
| .ok_or_else(RpcError::not_found)?; |
There was a problem hiding this comment.
With rpc indexes disabled, this returns NOT_FOUND, while at_checkpoint_upper_bound returns UNAVAILABLE. Should we make them consistent?
9ee6fd3 to
5bf1de4
Compare
df6ebad to
486ca3e
Compare
…xed range as OUT_OF_RANGE
486ca3e to
c1f4b9b
Compare
Description
The package gRPC work (#28555, #28573, #28429, #28439, #28440, #28561) needs shared bounds for reads pinned to a checkpoint, but not the
GetObjectat_checkpointAPI in #28630. This PR holds those shared pieces so the package PRs can merge without it; #28630 sits on top of this PR and holds only the object API. It also defines the one set of bound types (AtCheckpointUpperBound,AtCheckpointLowerBound) and refusal constructors (not_served_here/not_readywith acause, on sui-rust-sdkc20455f'sNOT_SERVED_HEREandNOT_READYreasons, andbelow_oldest_retained_versionfor a version below the pruning floor) that the PRs above use unchanged, and routes every existing fullnode "rpc indexes disabled", "index empty", "restore backfill" and "no object history" refusal through them.Mini design doc
Design
A read pinned to checkpoint K is answered only if K lies between two bounds that
StateReadercomputes, and is refused otherwise. The refusal is the same on the fullnode and kv-rpc:Upper bound: the highest checkpoint every rpc index has committed (
at_checkpoint_upper_bound). It's read before the data, since it only grows. With rpc indexes disabled it returnsUNIMPLEMENTEDwithNOT_SERVED_HERE(causeno_rpc_indexes). Before the indexes commit a checkpoint, or while a restored node is still backfilling index history, it returnsUNAVAILABLEwithNOT_READY(causeindex_emptyorrestore_backfill) instead of a bound.Lower bound:
max(L, RpcStateReader::get_lowest_checkpoint_with_index_history()) − 1, where L is the object store's lowest available checkpoint (at_checkpoint_lower_bound). It's read after the data, since pruning can raise it while the read runs.AtCheckpointLowerBoundstores max(L, first checkpoint with index history) and applies the −1 when it checks;pruned()andpruned_floor()give the same floor to reads that are not pinned to a checkpoint.Refusals:
OUT_OF_RANGEwithABOVE_LATEST_AVAILABLE.OUT_OF_RANGEwithBELOW_EARLIEST_AVAILABLE.The ledger
Listmethods already use these errors. kv-rpc builds the upper bound from its served height with the publicAtCheckpointUpperBound::newand runs the samecheck;checkpoint_not_yet_indexed_errorstays public for reads that compare the bound themselves, as rpc: serve GetObject/BatchGetObjects at_checkpoint from the fullnode and kv-rpc #28630 does.not_served_here/not_readyand their cause enums are public for the same reason.Compactor nodes: nodes that run the tidehunter objects compactor keep only each object's latest version. On them, reads that need replaced versions return
UNIMPLEMENTEDwithNOT_SERVED_HERE(causeno_object_history) (require_object_version_history).Users in this PR:
GetPackageat_checkpoint, which checks the upper bound before looking the package up, so an unknown package above the tip getsOUT_OF_RANGEon both servers rather thanNOT_FOUND. Also v2alphaGetCheckpointObjectProof, whose floor is the higher of the checkpoint and object floors, read after the reads. A proof loads the checkpoint's input objects and the object version it wrote, so it is refused on compactor nodes as well, before any other check; with rpc indexes disabled it returnsUNIMPLEMENTEDlikeat_checkpoint_upper_bound, and before the index has covered a checkpointUNAVAILABLE/NOT_READY(index_empty). The package PRs above and #28630 use the same bounds.Decisions
UNAVAILABLE/NOT_READY, notOUT_OF_RANGE. When a prefix of the chain is pruned (L > 0), a restore loadsobject_version_by_checkpointandpackage_versionsfloor rows at the restore checkpoint T. Until the backfill reaches T, a pinned read can be wrong at any checkpoint, not just at old ones. No range says what this node can answer, so it refuses everything, and a client retries another replica. When nothing is pruned (L = 0), only the live cohort is restored and the history cohort backfills from genesis with no restore rows; its reads are exact, but the indexed tip is the backfill's height, far below the restored live state, so the gate also counts any history pipeline that has committed below the live cohort's restore target.get_lowest_checkpoint_with_index_history(default0), giving a pinned-read floor of F. The package listing in rpc: list the versions of every package in ListPackageVersions on the fullnode and kv-rpc #28440 uses the same value as its floor (max(L, it)), so one trait method serves both. The fork's object floor staysmin(remote floor, F)because it serves pre-fork exact-version and proof reads from the remote chain. Rejected options:OUT_OF_RANGEwith anErrorReasoninstead ofNOT_FOUND.NOT_FOUNDcan't be told apart from a package or checkpoint that doesn't exist, so a client can't tell "retry later or elsewhere" from "no such thing". This matches the ledgerListmethods and kv-rpc.UNIMPLEMENTEDwithNOT_SERVED_HERE, temporary onesUNAVAILABLEwithNOT_READY. A node without rpc indexes, or without object version history, fails the same request the same way every time, and clients (the SDK ledger streams, authenticated events) retryUNAVAILABLEforever;UNIMPLEMENTEDtells them to use a differently configured node. Thecausekeeps it apart from gRPC's "method not found". When both apply,no_object_historyis checked first: enabling indexes alone would not fix it. One predicate,AuthorityPerpetualTablesOptions::objects_compactor_enabled, decides compaction for sui-node, the pruner and the read stores, which derive it from the node config. It is always false on RocksDB builds.Surprises
This PR was split out of the
GetObjectat_checkpointPR (now rpc: serve GetObject/BatchGetObjects at_checkpoint from the fullnode and kv-rpc #28630) so the package stack could merge without theGetObjectat_checkpointAPI. Besides the bounds, the package PRs needed three more pieces, which moved here:get_lowest_checkpoint_with_index_historyand its sui-fork implementation);checkpoint_not_yet_indexed_error;RpcIndexes::get_object_version_at_checkpoint, which fullnodeListSystemPackagesreads.The
GetObjecthunks and fork assertions moved to rpc: serve GetObject/BatchGetObjects at_checkpoint from the fullnode and kv-rpc #28630. The fullnodeGetPackagebound-order fix moved here from kv-rpc: bound bare-ID GetPackage, GetDatatype and GetFunction reads by the served height #28573, so the two servers agree from this PR on.GetCheckpointObjectProofchecked only the object floor, and checked it before the read. With checkpoint retention shorter than object retention, a pruned checkpoint gotNOT_FOUND. It now uses the higher of both floors, read after the reads.Impact and rollout
c20455f, which addsErrorReason::NotServedHere(8) andErrorReason::NotReady(9).GetPackagewithat_checkpointandGetCheckpointObjectProofreturnOUT_OF_RANGEwith anErrorReasonwhere they returnedNOT_FOUND. This applies on fullnodes, and above kv-rpc's served height. Clients that match onNOT_FOUNDfor these cases need updating; the light-client pruned-checkpoint test needed it.Listmethods,GetPackagewith a non-latestversionorat_checkpoint,ListPackageVersions) moves fromUNAVAILABLEtoUNIMPLEMENTEDwithNOT_SERVED_HERE(no_rpc_indexes). An empty rpc index staysUNAVAILABLEon those methods and gainsNOT_READY(index_empty). kv-rpc's refusals before it has a served checkpoint are unchanged.GetPackagewithat_checkpointon full nodes:UNAVAILABLE/NOT_READY(restore_backfill) while a restore is backfilling index history, andUNIMPLEMENTED/NOT_SERVED_HERE(no_object_history) on compactor nodes (validators andnum_epochs_to_retain = 0on tidehunter builds).GetCheckpointObjectProofreturnsUNIMPLEMENTED/NOT_SERVED_HEREon compactor nodes (no_object_history, where it returnedINTERNALorNOT_FOUND) and with rpc indexes disabled (no_rpc_indexes, where it returnedNOT_FOUND), andUNAVAILABLE/NOT_READY(index_empty) before the index has covered a checkpoint, where it returnedNOT_FOUND.object_version_by_checkpointandpackage_versions.Test coverage
OUT_OF_RANGE, notNOT_FOUND;BELOW_EARLIEST_AVAILABLEand a proof at the floor is served, by pruning deterministically (TestCluster::prune_fullnode_checkpoints_for_testing);StateReader::at_checkpoint_upper_boundover a real rpc-store through an empty index, an in-progress restore, a completed restore whose backfill is behind, a caught-up backfill, and a live-only restore whose from-genesis history backfill is behind (unit);pruned_floorat 0, 1 and above (unit);UNIMPLEMENTEDwithNOT_SERVED_HERE(no_rpc_indexes).NOT_FOUND.at_checkpointe2e tests disable fullnode pruning so they'd still pass on tidehunter.Test plan
GetPackageat_checkpointis served at the last pruned checkpoint and refused below it, and served at the indexed tip but refused at tip + 1, for a known and an unknown package (get_package.rs).test_pruned_checkpoint_is_below_earliest_available(get_object_inclusion_proof_tests.rs), which also gets a non-inclusion proof at the lowest available checkpoint, and the light-clienttest_client_pruned_checkpoint_error, which now requiresOUT_OF_RANGEwithBELOW_EARLIEST_AVAILABLE.test_proof_without_rpc_indexes_is_not_served_here(get_object_inclusion_proof_tests.rs).history_backfill_pending_until_floors_reach_restore_anchor,history_backfill_pending_until_history_reaches_live_restore);upper_bound_waits_for_index_and_restore_backfill,upper_bound_waits_for_history_backfill_after_live_only_restoreandlower_bound_serves_the_checkpoint_before_the_lowest_availablein sui-rpc-api'sreader.rs;below_oldest_retained_version_names_checkpoint_and_versionin sui-rpc-api'squery_options.rs;test_checkpoint_pinned_reads_below_the_fork_checkpoint_are_rejectedin sui-fork, which also reads at the fork checkpoint.Release notes
Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.
For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.
MovePackageService.GetPackagewithat_checkpoint, and v2alphaProofService.GetCheckpointObjectProof, now fail withOUT_OF_RANGEand anErrorReasonfor a checkpoint outside the available range, where they returnedNOT_FOUND:ABOVE_LATEST_AVAILABLEabove the highest indexed checkpoint, andBELOW_EARLIEST_AVAILABLEbelow the lowest available one. Full-node refusals for running without rpc indexes ("rpc indexes are disabled") move fromUNAVAILABLEtoUNIMPLEMENTEDwith aNOT_SERVED_HEREErrorInfo(cause=no_rpc_indexes); do not retry them against the same node. An empty rpc index, andGetPackagewithat_checkpointwhile a restore is still backfilling index history, returnUNAVAILABLEwith aNOT_READYErrorInfo(cause=index_emptyorrestore_backfill).GetCheckpointObjectProofwithout rpc indexes moves fromNOT_FOUNDtoUNIMPLEMENTED(no_rpc_indexes), and on an empty rpc index fromNOT_FOUNDtoUNAVAILABLE(index_empty). Requests that need replaced object versions (GetPackagewithat_checkpoint,GetCheckpointObjectProof) on nodes that run the objects compactor returnUNIMPLEMENTEDwithNOT_SERVED_HERE(cause=no_object_history).