Repository navigation
rpc: describe removals in GetObject and BatchGetObjects NOT_FOUND errors - #28663
Draft
nickvikeras wants to merge 1 commit into
Draft
nickvikeras wants to merge 1 commit into
nickvikeras wants to merge 1 commit into
Conversation
nickvikeras
added a commit
that referenced
this pull request
Nov 13, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-fork/src/tests/store_execution.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kvstore/src/bigtable/client/mod.rs # crates/sui-kvstore/src/bigtable/mock_server.rs # crates/sui-kvstore/src/tables/objects.rs # crates/sui-rpc-api/src/reader.rs # crates/sui-types/src/storage/read_store.rs
nickvikeras
force-pushed
the
nickv/object-removed-error
branch
from
November 13, 2026 04:15
0f97326 to
f113d02
Compare
nickvikeras
added a commit
that referenced
this pull request
Nov 13, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-fork/src/tests/store_execution.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kvstore/src/bigtable/client/mod.rs # crates/sui-kvstore/src/bigtable/mock_server.rs # crates/sui-rpc-api/src/reader.rs # crates/sui-types/src/storage/read_store.rs
nickvikeras
force-pushed
the
nickv/object-removed-error
branch
from
November 13, 2026 04:53
f113d02 to
ccb49e7
Compare
nickvikeras
force-pushed
the
nickv/list-object-versions-fullnode
branch
from
November 13, 2026 04:53
cbbf11a to
698c3ef
Compare
nickvikeras
added a commit
that referenced
this pull request
Nov 13, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-fork/src/tests/store_execution.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kvstore/src/bigtable/client/mod.rs # crates/sui-kvstore/src/bigtable/mock_server.rs # crates/sui-rpc-api/src/reader.rs # crates/sui-types/src/storage/read_store.rs
nickvikeras
force-pushed
the
nickv/object-removed-error
branch
from
November 13, 2026 05:24
ccb49e7 to
a00d058
Compare
nickvikeras
force-pushed
the
nickv/list-object-versions-fullnode
branch
from
November 13, 2026 05:24
698c3ef to
44cdb68
Compare
nickvikeras
added a commit
that referenced
this pull request
Nov 13, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-fork/src/tests/store_execution.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kvstore/src/bigtable/client/mod.rs # crates/sui-kvstore/src/bigtable/mock_server.rs # crates/sui-rpc-api/src/reader.rs # crates/sui-types/src/storage/read_store.rs
nickvikeras
force-pushed
the
nickv/list-object-versions-fullnode
branch
from
November 13, 2026 11:11
44cdb68 to
ba3452c
Compare
nickvikeras
force-pushed
the
nickv/object-removed-error
branch
from
November 13, 2026 11:11
a00d058 to
24e539e
Compare
nickvikeras
added a commit
that referenced
this pull request
Nov 13, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-fork/src/tests/store_execution.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kvstore/src/bigtable/client/mod.rs # crates/sui-kvstore/src/bigtable/mock_server.rs # crates/sui-rpc-api/src/reader.rs # crates/sui-types/src/storage/read_store.rs
nickvikeras
force-pushed
the
nickv/list-object-versions-fullnode
branch
from
November 14, 2026 01:04
ba3452c to
f34e4e1
Compare
nickvikeras
force-pushed
the
nickv/object-removed-error
branch
2 times, most recently
from
November 14, 2026 06:25
6267e23 to
5b59834
Compare
nickvikeras
force-pushed
the
nickv/list-object-versions-fullnode
branch
from
November 14, 2026 06:25
f34e4e1 to
1b966b0
Compare
nickvikeras
added a commit
that referenced
this pull request
Nov 14, 2026
# Conflicts: # Cargo.lock # Cargo.toml # crates/sui-fork/src/tests/store_execution.rs # crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs # crates/sui-kv-rpc/src/bigtable_client.rs # crates/sui-kvstore/src/bigtable/client/mod.rs # crates/sui-kvstore/src/bigtable/mock_server.rs # crates/sui-types/src/storage/read_store.rs
nickvikeras
force-pushed
the
nickv/list-object-versions-fullnode
branch
from
November 14, 2026 18:50
1b966b0 to
b530dec
Compare
nickvikeras
force-pushed
the
nickv/object-removed-error
branch
from
November 14, 2026 18:50
5b59834 to
2a6712c
Compare
1 of 8 tasks
nickvikeras
force-pushed
the
nickv/list-object-versions-fullnode
branch
from
November 15, 2026 02:09
b530dec to
b0e9aa5
Compare
nickvikeras
force-pushed
the
nickv/object-removed-error
branch
from
November 15, 2026 02:09
2a6712c to
b61901e
Compare
1 of 8 tasks
A read that resolves to a version at which the object was deleted or wrapped still answers NOT_FOUND, now with an OBJECT_REMOVED ErrorInfo naming the removal's version, kind and, when the server has it, transaction. kv-rpc reads tombstone rows of the objects table, and the version index for at_checkpoint reads; the fullnode repeats the read's selection over the execution store's tombstones and finds the transaction through the transaction bitmap. The ListObjectVersions checks in get_object_parity_tests and kv_rpc_tests list with transaction in removal_read_mask and with the default version,kind mask, so get_object_parity_tests compares both paths between the fullnode and kv-rpc, and kv_rpc_tests checks both against the lifecycle. Also pins sui-rust-sdk to #345 cfae7c5 for removal_read_mask.
nickvikeras
force-pushed
the
nickv/object-removed-error
branch
from
November 15, 2026 19:59
b61901e to
744b958
Compare
nickvikeras
force-pushed
the
nickv/list-object-versions-fullnode
branch
from
November 15, 2026 19:59
b0e9aa5 to
0ee2270
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
A
GetObjectorBatchGetObjectsread that lands on a deletion or wrap answersNOT_FOUNDexactly like a read of an object that never existed, so clients cannot tell the two apart or find out which transaction removed the object. This attaches that information as an error detail. Older clients see no difference.Mini design doc
Design. When a read resolves to a version at which the object was deleted or wrapped, both servers still answer
NOT_FOUND. They also attach agoogle.rpc.ErrorInfowith reasonOBJECT_REMOVED(ErrorReason::ObjectRemoved, MystenLabs/sui-rust-sdk#345), domainsui.io, and metadataremovedVersion,removalKind(DELETED/WRAPPED) andtransaction(base58 digest of the removing transaction, left out when the server does not have it). In a batch, the detail goes in that item'sStatus. The detail is attached by:versionread, when that version is a removal;max_versionread, when the newest version at or below the bound is a removal. Withoutat_checkpoint, the fullnode reads live (decision 31) and finds the removal with a lookup through the execution cache (see Decisions);at_checkpointread, when the object's state at that checkpoint is a removal, andversion/max_versionreads withat_checkpoint, only when the removal was written at or before the checkpoint. A removal written after it answers plainNOT_FOUND, like any other version written afterat_checkpoint.Never-existed and not-yet-indexed reads answer as before.
Nodes without object version history (decision 27). #28669, below this stack, refuses exact
versionreads (noat_checkpoint) on compactor nodes unless the version is the object's latest. With tombstone reads available here, that rule decides from the object's newest entry, live or a tombstone, which is the entry the compactor keeps. A version above it isNOT_FOUND. A deletion or wrap at exactly the requested version isNOT_FOUNDwithOBJECT_REMOVED, as on other nodes. An object with no entry at all isNOT_FOUND. Any older version is still refused withUNIMPLEMENTED/NOT_SERVED_HERE(causeno_object_history).Decisions.
NOT_FOUND, so existing clients and error handling are unchanged, and no request opts in. A response field would turn a not-found into a success with no object, which breaks every client that checks forobject.ErrorInfois the pattern the API already uses for structured errors (INSUFFICIENT_GAS_BALANCE,ABOVE_LATEST_AVAILABLE).at_checkpointnever reports a removal from after that checkpoint.NOT_FOUNDit would have been, with less detail or none. On kv-rpc, when the version index records the removal, its kind and version come from the index, and theobjectsread for its transaction is best effort: if it fails, kv-rpc logs a warning and the detail names no transaction, as for a tombstone written without one. The reverse scan for a removal's tombstone (BoundedRead::ScanForRemoval) decides whether there is a detail at all, so if it fails the answer is plainNOT_FOUND. Neither ever fails the request or the batch. So aNOT_FOUNDwithout the detail does not prove that the object never existed: a fullnode that has pruned the removal's tombstone or fails to read it, or a store that keeps no tombstones, answers plainNOT_FOUND.ObjectNotFoundError::removeddocuments this.max_versionreads withoutat_checkpointfind the removal through the execution cache. Such a read is live (decision 31), so its newest versions can still be uncommitted in the execution cache. A tombstone lookup of the store alone (object_versions_iter) would miss them and could name an older committed removal: v5 deleted and committed; v7 re-created, v9 deleted and v11 re-created, all uncommitted;max_version= 10 would select the v5 removal. The lookup is insteadObjectCacheRead::find_object_or_tombstone_lt_or_eq_version, whichfind_object_lt_or_eq_versionnow calls: the same cache-then-store selection, returning the newest entry at or below the version, live object or tombstone, wherefind_object_lt_or_eq_versionreturnsNonefor a tombstone. Its store fallback reads the newest row at or below the version, tombstones included. So the removal it finds is the one the live read resolved to (v9 in the example). The removing transaction stays best effort: it is looked up only up to the highest indexed checkpoint, so a removal above it is described without its transaction.versionormax_versionread whose live lookup finds nothing is refused withOUT_OF_RANGE/BELOW_EARLIEST_AVAILABLEon a node whose object lower bound is above 0, since the version may have been pruned. A read that resolves to a removal whose tombstone the node holds answersNOT_FOUNDwith the detail instead: the removal is looked for before the refusal is decided. An exactversionabove the object's latest version, live or removed, isNOT_FOUND(it does not exist yet). Amax_versionread withoutat_checkpointwhose object's latest version, read through the execution cache, is a removal at or belowmax_versionanswersNOT_FOUND, without the detail only if the tombstone lookup fails.INTERNAL. When a read fetches theobjectsrow thatobject_versions_by_checkpointnames as live and finds it missing or a tombstone, the read isINTERNAL, notNOT_FOUND: the tables are written so they never disagree.objectsread goes through the request's limiter, like the rest ofGetObjectandBatchGetObjects.Where each server gets kind, version and transaction, and what it costs. All extra reads happen only on the
NOT_FOUNDpath, or for a fullnodeversion/max_versionread whose live lookup finds nothing.versionobjectsrow. A tombstone row gives kind, version and the transaction (columnt), so this costs no extra read. These reads now return tombstone rows instead of dropping them: the latest scan and the keyedobjectsreads keep tombstones whole, and strip only live objects' contents for version-only masks.StoreObject::Deleted/Wrapped). A latest read looks up the object's latest entry through the execution cache and needs nothing more. An exactversionread takes the newest entry at or belowversionthroughfind_object_or_tombstone_lt_or_eq_version: the cached latest entry when it is at or belowversion, otherwise the cached versions, committed or not, then the store's latest entry, and a store scan only when that entry is aboveversion; an object with no entry at all stops at the latest-entry lookup. So an exact-version read of an ID that never existed costs the live lookup plus the latest-entry lookup. On a pruned node (object lower bound above 0), the refusal check then reads the lower bound and the latest entry again, plus the latest live object when there is no entry.at_checkpoint, andmax_versionwith or without it (kv-rpc readsmax_versionat the served height; the fullnode reads it withoutat_checkpointlive)object_versions_by_checkpointrow at the checkpoint says the object was removed at the selected version, it gives kind and version. The tombstone rows of such removals are then read for their transactions, together in one best-effortobjectsread per group of scans (answer_scans; a group holds at mostmax-seeks-per-readlookups). When the selection goes through theobjectsreverse scan instead, the scan lands on the tombstone at no extra cost. With a version-only read mask, themax_versionreverse scan reads only row keys and tombstones, not live objects' contents.max_versionread withoutat_checkpointreads no index: when its live lookup finds nothing, onefind_object_or_tombstone_lt_or_eq_versionlookup atmax_versionselects the removal, as for an exactversion. Withat_checkpoint, when the version index records no version of the object as of the checkpoint, the object did not exist yet and nothing more is read, so an ID that never existed costs one index read. Otherwise, if the live selection finds no live object, the selection runs again over the store's tombstones, with itsversion_at_checkpointindex reads, plus one or two store reads for the tombstone (one formax_versionalone). Each index read is followed by a store seek to the object's next item (stored_version_of_indexed), which maps a V1-effects removal's index record to its tombstone.max_version+at_checkpoint, removed at the checkpoint and changed again before the served heightobjectsreverse scan that live reads already use is scheduled (BoundedRead::ScanForRemoval). It counts only if it lands on the removal's tombstone, so it reads no live object's contents, and it is best effort.t, written by #28653. Absent on tombstones written before it.all_tombstoneseffects check), only when the tombstone selection picks a removal, bounded by the highest indexed checkpoint of an upper bound read at most once per request (the request's own when it loaded one): oneversion_at_checkpointread at the indexed tip, a lookup of the object's next item, a reverse scan of the object's bitmap buckets to its last transaction before that item, one digest read and one effects read. Left unset when anything it needs is pruned, when the indexes are disabled, or when the removal is not indexed yet.Other
RpcStateReaderimplementers. Two new trait methods,get_latest_object_or_tombstoneandget_object_or_tombstone_lt_or_eq_version, default toNone(no detail). Onlysui-core'sRestReadStoreandRpcStoreReadStoreimplement them, through the execution cache (get_latest_object_or_tombstoneand the newfind_object_or_tombstone_lt_or_eq_version). sui-fork, simulacrum, the transactional test runner andRpcStoreReaderkeep the default, so their reads answer plainNOT_FOUND. For sui-fork, its local rows below the fork checkpoint are a sparse cache, so a missing tombstone there cannot be told apart from absence.Surprises. For transactions with V1 effects, the fullnode's version index records a removal below its tombstone's version (see
newest_version_written_by): the index recordslamport_version, which V1 effects compute from the modified objects alone, while the tombstone sits at the version the transaction executed at. No item lies between the two, so the tombstone selection maps each index read to the version the store holds (stored_version_of_indexed): when the store holds nothing at the indexed version, the tombstone right above it is that removal. Every read kind,at_checkpointincluded, describes these removals. Live reads keep the indexed version, which a live object always has. In practice, a fullnode's retention window holds no V1 effects.Impact and rollout. Wire-compatible: same codes and messages, plus a new error detail. No config, no protocol change. The SDK pin moves to sui-rust-sdk
cfae7c5(#345) forErrorReason::ObjectRemoved.find_object_lt_or_eq_version, which transaction execution uses, behaves as before: it maps a tombstone fromfind_object_or_tombstone_lt_or_eq_versiontoNone, as it did when it found one itself, and its store fallback (AuthorityPerpetualTables::find_object_or_tombstone_lt_or_eq_version) reads the same row with the same bounds as the store read it replaces.Test coverage. The tests cover every read kind on both servers, single and batched, through wrap, unwrap and delete, including checkpoint bounds on either side of a removal and the removing transaction. Fullnode pruned-node answers and read counts are covered by unit models, and the cache-aware lookup, uncommitted removals included, by a writeback cache unit test. Not covered end to end: pruned fullnode history, uncommitted removals in the execution cache, and V1-effects removals (only the unit model covers them).
Test plan
New tests:
sui-kv-rpcget_object::tests::test_removed_objects_report_their_removal: mock BigTable. Covers latest, exact,max_versionandat_checkpointreads, alone and combined, single and in one batch with live and never-existed items, with and without contents, and tombstones with and withoutt. Also checks that removals cost latest and exact reads no extraobjectsread, and costat_checkpointreads one.sui-kv-rpcget_object::tests::test_failed_removal_row_reads_answer_not_found: when the tombstone row of a removal the index records cannot be read, the read answersNOT_FOUNDwith the index's version and kind and no transaction; whenBoundedRead::ScanForRemovalfails, plainNOT_FOUND. Single and batched, and the batch's other items are answered.sui-kv-rpcget_object::tests::test_at_checkpoint_live_version_with_tombstone_row_is_internal: the version index names a version live whoseobjectsrow is a tombstone:INTERNAL, single and batched.sui-rpc-apiget_object::tests::max_version_at_checkpoint_selects_removals_written_by_the_checkpoint: the fullnode selection over tombstones, for a lifecycle removed by transactions with V2 effects and one removed by transactions with V1 effects, whose index records each removal below its tombstone.sui-rpc-apiget_object::tests::reads_resolving_to_removals_describe_them:read_objectover a modeled history: every read kind that resolves to a removal describes it, including those whose live lookup finds nothing (exactversion,versionbelow the checkpoint's version withat_checkpoint,max_versionwith and withoutat_checkpoint), with one transaction lookup. Withoutat_checkpoint, an object deleted at 5, re-created at 7, deleted at 9 and re-created at 11 names the removal at 9 formax_version= 10.sui-rpc-apiget_object::tests::reads_of_objects_that_never_existed_look_for_no_removal:at_checkpointreads of an ID that never existed cost one index read and no tombstone or transaction lookup.sui-rpc-apiget_object::tests::missing_versions_resolving_to_the_latest_removal_are_not_foundandremoval_upper_bound_is_loaded_once_per_request.sui-coreexecution_cache::writeback_cache::writeback_cache_tests::test_object_or_tombstone_lt_or_eq:find_object_or_tombstone_lt_or_eq_versionreturns the newest entry at or below each bound, live or a tombstone, with entries uncommitted, committed, and read back from the store alone after the cache is reset, under every cache-eviction point. With a removal at 5 committed, and the object re-created at 7, removed at 9 and re-created at 11 still uncommitted, a bound of 10 finds the removal at 9. An object whose latest entry is its deletion returns that tombstone (OBJECT_DIGEST_DELETED) for every bound at or above it.find_object_lt_or_eq_versionagrees with it, with a tombstone read as no object.sui-kv-rpcget_object::tests::test_version_only_masks_skip_object_contents(extended): a version-onlymax_versionread below the version atat_checkpoint, single and batched, scansobjectswithout object contents.sui-indexer-alt-e2e-teststest_get_object_reports_removals: kv-rpc lifecycle. It checks reason, kind, version and the removing transaction, which every detail must name now that the tombstone writer records it (sui-kvstore: record the removing transaction in object tombstones #28653).sui-indexer-alt-e2e-testsget_object_parity_tests::test_object_reads_match_between_fullnode_and_kv_rpc: the same object reads against a fullnode and kv-rpc answer the same, removal details included. Amax_versionread withoutat_checkpointis checked against the chain on each server instead, since the fullnode reads it live and kv-rpc at its served height. ItsListObjectVersionscases (full listing, eachat_checkpoint, cross-server paging, both orders) run withtransactioninremoval_read_mask, where every removal carries its transaction, and with the default mask, where none does.sui-indexer-alt-e2e-testskv_rpc_tests::test_list_object_versions_lifecycle(extended): lists each lifecycle object withremoval_read_maskversion,kind,transaction, checking each removal's kind and removing transaction, and with the default mask, checking kinds and no transaction.sui-e2e-testsrpc::v2::ledger_service::get_object::get_object_lifecycle(extended): fullnode lifecycle, single and batched. It checks reason, kind, version and the transaction found through the bitmap, including for amax_versionread withoutat_checkpointat each removal's version.Release notes
GetObjectandBatchGetObjectsattach anOBJECT_REMOVEDErrorInfotoNOT_FOUNDanswers for reads that resolve to a deletion or wrap, with the removal's version, kind and, when known, transaction. On nodes that run the objects compactor, an exactversionequal to the object's deletion or wrap now answersNOT_FOUNDwith that detail, and anyversionof an object with no entry at all answersNOT_FOUND, where rpc: refuse with UNIMPLEMENTED / BELOW_EARLIEST_AVAILABLE where nodes are not set up or have pruned #28669 refuses both withUNIMPLEMENTED/NOT_SERVED_HERE.