Skip to content

rpc: describe removals in GetObject and BatchGetObjects NOT_FOUND errors - #28663

Draft
nickvikeras wants to merge 1 commit into
nickv/list-object-versions-fullnodefrom
nickv/object-removed-error
Draft

nickvikeras wants to merge 1 commit into
nickv/list-object-versions-fullnodefrom
nickv/object-removed-error

Conversation

@nickvikeras

@nickvikeras nickvikeras commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Description

A GetObject or BatchGetObjects read that lands on a deletion or wrap answers NOT_FOUND exactly like a read of an object that never existed, so clients cannot tell the two apart or find out which transaction removed the object. This attaches that information as an error detail. Older clients see no difference.

Mini design doc

Design. When a read resolves to a version at which the object was deleted or wrapped, both servers still answer NOT_FOUND. They also attach a google.rpc.ErrorInfo with reason OBJECT_REMOVED (ErrorReason::ObjectRemoved, MystenLabs/sui-rust-sdk#345), domain sui.io, and metadata removedVersion, removalKind (DELETED/WRAPPED) and transaction (base58 digest of the removing transaction, left out when the server does not have it). In a batch, the detail goes in that item's Status. The detail is attached by:

  • a latest read, when the object's current state is a removal;
  • an exact version read, when that version is a removal;
  • a max_version read, when the newest version at or below the bound is a removal. Without at_checkpoint, the fullnode reads live (decision 31) and finds the removal with a lookup through the execution cache (see Decisions);
  • an at_checkpoint read, when the object's state at that checkpoint is a removal, and version/max_version reads with at_checkpoint, only when the removal was written at or before the checkpoint. A removal written after it answers plain NOT_FOUND, like any other version written after at_checkpoint.

Never-existed and not-yet-indexed reads answer as before.

Nodes without object version history (decision 27). #28669, below this stack, refuses exact version reads (no at_checkpoint) on compactor nodes unless the version is the object's latest. With tombstone reads available here, that rule decides from the object's newest entry, live or a tombstone, which is the entry the compactor keeps. A version above it is NOT_FOUND. A deletion or wrap at exactly the requested version is NOT_FOUND with OBJECT_REMOVED, as on other nodes. An object with no entry at all is NOT_FOUND. Any older version is still refused with UNIMPLEMENTED / NOT_SERVED_HERE (cause no_object_history).

Decisions.

  • Error detail, not a request flag or a new response field. The answer stays NOT_FOUND, so existing clients and error handling are unchanged, and no request opts in. A response field would turn a not-found into a success with no object, which breaks every client that checks for object. ErrorInfo is the pattern the API already uses for structured errors (INSUFFICIENT_GAS_BALANCE, ABOVE_LATEST_AVAILABLE).
  • The detail describes the version the read resolved to, not the object's later history. A read with at_checkpoint never reports a removal from after that checkpoint.
  • Best effort, never an error. If the lookup behind the detail fails, or what it needs is pruned, the answer is the NOT_FOUND it would have been, with less detail or none. On kv-rpc, when the version index records the removal, its kind and version come from the index, and the objects read for its transaction is best effort: if it fails, kv-rpc logs a warning and the detail names no transaction, as for a tombstone written without one. The reverse scan for a removal's tombstone (BoundedRead::ScanForRemoval) decides whether there is a detail at all, so if it fails the answer is plain NOT_FOUND. Neither ever fails the request or the batch. So a NOT_FOUND without the detail does not prove that the object never existed: a fullnode that has pruned the removal's tombstone or fails to read it, or a store that keeps no tombstones, answers plain NOT_FOUND. ObjectNotFoundError::removed documents this.
  • Fullnode max_version reads without at_checkpoint find the removal through the execution cache. Such a read is live (decision 31), so its newest versions can still be uncommitted in the execution cache. A tombstone lookup of the store alone (object_versions_iter) would miss them and could name an older committed removal: v5 deleted and committed; v7 re-created, v9 deleted and v11 re-created, all uncommitted; max_version = 10 would select the v5 removal. The lookup is instead ObjectCacheRead::find_object_or_tombstone_lt_or_eq_version, which find_object_lt_or_eq_version now calls: the same cache-then-store selection, returning the newest entry at or below the version, live object or tombstone, where find_object_lt_or_eq_version returns None for a tombstone. Its store fallback reads the newest row at or below the version, tombstones included. So the removal it finds is the one the live read resolved to (v9 in the example). The removing transaction stays best effort: it is looked up only up to the highest indexed checkpoint, so a removal above it is described without its transaction.
  • Removed objects on pruned fullnodes are answers, not refusals. A version or max_version read whose live lookup finds nothing is refused with OUT_OF_RANGE / BELOW_EARLIEST_AVAILABLE on a node whose object lower bound is above 0, since the version may have been pruned. A read that resolves to a removal whose tombstone the node holds answers NOT_FOUND with the detail instead: the removal is looked for before the refusal is decided. An exact version above the object's latest version, live or removed, is NOT_FOUND (it does not exist yet). A max_version read without at_checkpoint whose object's latest version, read through the execution cache, is a removal at or below max_version answers NOT_FOUND, without the detail only if the tombstone lookup fails.
  • kv-rpc index and storage disagreements are INTERNAL. When a read fetches the objects row that object_versions_by_checkpoint names as live and finds it missing or a tombstone, the read is INTERNAL, not NOT_FOUND: the tables are written so they never disagree.
  • kv-rpc reads stay on the request's gated client. Every new or changed objects read goes through the request's limiter, like the rest of GetObject and BatchGetObjects.

Where each server gets kind, version and transaction, and what it costs. All extra reads happen only on the NOT_FOUND path, or for a fullnode version/max_version read whose live lookup finds nothing.

Read kv-rpc Fullnode
latest, exact version The read already lands on the objects row. A tombstone row gives kind, version and the transaction (column t), so this costs no extra read. These reads now return tombstone rows instead of dropping them: the latest scan and the keyed objects reads keep tombstones whole, and strip only live objects' contents for version-only masks. After the live read finds nothing, the same selection runs again over the execution store's tombstones (StoreObject::Deleted/Wrapped). A latest read looks up the object's latest entry through the execution cache and needs nothing more. An exact version read takes the newest entry at or below version through find_object_or_tombstone_lt_or_eq_version: the cached latest entry when it is at or below version, otherwise the cached versions, committed or not, then the store's latest entry, and a store scan only when that entry is above version; an object with no entry at all stops at the latest-entry lookup. So an exact-version read of an ID that never existed costs the live lookup plus the latest-entry lookup. On a pruned node (object lower bound above 0), the refusal check then reads the lower bound and the latest entry again, plus the latest live object when there is no entry.
at_checkpoint, and max_version with or without it (kv-rpc reads max_version at the served height; the fullnode reads it without at_checkpoint live) When the object_versions_by_checkpoint row at the checkpoint says the object was removed at the selected version, it gives kind and version. The tombstone rows of such removals are then read for their transactions, together in one best-effort objects read per group of scans (answer_scans; a group holds at most max-seeks-per-read lookups). When the selection goes through the objects reverse scan instead, the scan lands on the tombstone at no extra cost. With a version-only read mask, the max_version reverse scan reads only row keys and tombstones, not live objects' contents. The live selection runs first. A max_version read without at_checkpoint reads no index: when its live lookup finds nothing, one find_object_or_tombstone_lt_or_eq_version lookup at max_version selects the removal, as for an exact version. With at_checkpoint, when the version index records no version of the object as of the checkpoint, the object did not exist yet and nothing more is read, so an ID that never existed costs one index read. Otherwise, if the live selection finds no live object, the selection runs again over the store's tombstones, with its version_at_checkpoint index reads, plus one or two store reads for the tombstone (one for max_version alone). Each index read is followed by a store seek to the object's next item (stored_version_of_indexed), which maps a V1-effects removal's index record to its tombstone.
max_version + at_checkpoint, removed at the checkpoint and changed again before the served height The version at the served height (v_H) decides this case too: it shows the object changed again after the removal at the checkpoint's version, so the objects reverse scan that live reads already use is scheduled (BoundedRead::ScanForRemoval). It counts only if it lands on the removal's tombstone, so it reads no live object's contents, and it is best effort. Same as above.
transaction Column t, written by #28653. Absent on tombstones written before it. #28645's removal-transaction lookup (the AffectedObject bitmap plus the all_tombstones effects check), only when the tombstone selection picks a removal, bounded by the highest indexed checkpoint of an upper bound read at most once per request (the request's own when it loaded one): one version_at_checkpoint read at the indexed tip, a lookup of the object's next item, a reverse scan of the object's bitmap buckets to its last transaction before that item, one digest read and one effects read. Left unset when anything it needs is pruned, when the indexes are disabled, or when the removal is not indexed yet.

Other RpcStateReader implementers. Two new trait methods, get_latest_object_or_tombstone and get_object_or_tombstone_lt_or_eq_version, default to None (no detail). Only sui-core's RestReadStore and RpcStoreReadStore implement them, through the execution cache (get_latest_object_or_tombstone and the new find_object_or_tombstone_lt_or_eq_version). sui-fork, simulacrum, the transactional test runner and RpcStoreReader keep the default, so their reads answer plain NOT_FOUND. For sui-fork, its local rows below the fork checkpoint are a sparse cache, so a missing tombstone there cannot be told apart from absence.

Surprises. For transactions with V1 effects, the fullnode's version index records a removal below its tombstone's version (see newest_version_written_by): the index records lamport_version, which V1 effects compute from the modified objects alone, while the tombstone sits at the version the transaction executed at. No item lies between the two, so the tombstone selection maps each index read to the version the store holds (stored_version_of_indexed): when the store holds nothing at the indexed version, the tombstone right above it is that removal. Every read kind, at_checkpoint included, describes these removals. Live reads keep the indexed version, which a live object always has. In practice, a fullnode's retention window holds no V1 effects.

Impact and rollout. Wire-compatible: same codes and messages, plus a new error detail. No config, no protocol change. The SDK pin moves to sui-rust-sdk cfae7c5 (#345) for ErrorReason::ObjectRemoved. find_object_lt_or_eq_version, which transaction execution uses, behaves as before: it maps a tombstone from find_object_or_tombstone_lt_or_eq_version to None, as it did when it found one itself, and its store fallback (AuthorityPerpetualTables::find_object_or_tombstone_lt_or_eq_version) reads the same row with the same bounds as the store read it replaces.

Test coverage. The tests cover every read kind on both servers, single and batched, through wrap, unwrap and delete, including checkpoint bounds on either side of a removal and the removing transaction. Fullnode pruned-node answers and read counts are covered by unit models, and the cache-aware lookup, uncommitted removals included, by a writeback cache unit test. Not covered end to end: pruned fullnode history, uncommitted removals in the execution cache, and V1-effects removals (only the unit model covers them).

Test plan

New tests:

  • sui-kv-rpc get_object::tests::test_removed_objects_report_their_removal: mock BigTable. Covers latest, exact, max_version and at_checkpoint reads, alone and combined, single and in one batch with live and never-existed items, with and without contents, and tombstones with and without t. Also checks that removals cost latest and exact reads no extra objects read, and cost at_checkpoint reads one.
  • sui-kv-rpc get_object::tests::test_failed_removal_row_reads_answer_not_found: when the tombstone row of a removal the index records cannot be read, the read answers NOT_FOUND with the index's version and kind and no transaction; when BoundedRead::ScanForRemoval fails, plain NOT_FOUND. Single and batched, and the batch's other items are answered.
  • sui-kv-rpc get_object::tests::test_at_checkpoint_live_version_with_tombstone_row_is_internal: the version index names a version live whose objects row is a tombstone: INTERNAL, single and batched.
  • sui-rpc-api get_object::tests::max_version_at_checkpoint_selects_removals_written_by_the_checkpoint: the fullnode selection over tombstones, for a lifecycle removed by transactions with V2 effects and one removed by transactions with V1 effects, whose index records each removal below its tombstone.
  • sui-rpc-api get_object::tests::reads_resolving_to_removals_describe_them: read_object over a modeled history: every read kind that resolves to a removal describes it, including those whose live lookup finds nothing (exact version, version below the checkpoint's version with at_checkpoint, max_version with and without at_checkpoint), with one transaction lookup. Without at_checkpoint, an object deleted at 5, re-created at 7, deleted at 9 and re-created at 11 names the removal at 9 for max_version = 10.
  • sui-rpc-api get_object::tests::reads_of_objects_that_never_existed_look_for_no_removal: at_checkpoint reads of an ID that never existed cost one index read and no tombstone or transaction lookup.
  • sui-rpc-api get_object::tests::missing_versions_resolving_to_the_latest_removal_are_not_found and removal_upper_bound_is_loaded_once_per_request.
  • sui-core execution_cache::writeback_cache::writeback_cache_tests::test_object_or_tombstone_lt_or_eq: find_object_or_tombstone_lt_or_eq_version returns the newest entry at or below each bound, live or a tombstone, with entries uncommitted, committed, and read back from the store alone after the cache is reset, under every cache-eviction point. With a removal at 5 committed, and the object re-created at 7, removed at 9 and re-created at 11 still uncommitted, a bound of 10 finds the removal at 9. An object whose latest entry is its deletion returns that tombstone (OBJECT_DIGEST_DELETED) for every bound at or above it. find_object_lt_or_eq_version agrees with it, with a tombstone read as no object.
  • sui-kv-rpc get_object::tests::test_version_only_masks_skip_object_contents (extended): a version-only max_version read below the version at at_checkpoint, single and batched, scans objects without object contents.
  • sui-indexer-alt-e2e-tests test_get_object_reports_removals: kv-rpc lifecycle. It checks reason, kind, version and the removing transaction, which every detail must name now that the tombstone writer records it (sui-kvstore: record the removing transaction in object tombstones #28653).
  • sui-indexer-alt-e2e-tests get_object_parity_tests::test_object_reads_match_between_fullnode_and_kv_rpc: the same object reads against a fullnode and kv-rpc answer the same, removal details included. A max_version read without at_checkpoint is checked against the chain on each server instead, since the fullnode reads it live and kv-rpc at its served height. Its ListObjectVersions cases (full listing, each at_checkpoint, cross-server paging, both orders) run with transaction in removal_read_mask, where every removal carries its transaction, and with the default mask, where none does.
  • sui-indexer-alt-e2e-tests kv_rpc_tests::test_list_object_versions_lifecycle (extended): lists each lifecycle object with removal_read_mask version,kind,transaction, checking each removal's kind and removing transaction, and with the default mask, checking kinds and no transaction.
  • sui-e2e-tests rpc::v2::ledger_service::get_object::get_object_lifecycle (extended): fullnode lifecycle, single and batched. It checks reason, kind, version and the transaction found through the bitmap, including for a max_version read without at_checkpoint at each removal's version.

Release notes

  • Protocol:
  • Nodes (Validators and Full nodes):
  • gRPC: GetObject and BatchGetObjects attach an OBJECT_REMOVED ErrorInfo to NOT_FOUND answers for reads that resolve to a deletion or wrap, with the removal's version, kind and, when known, transaction. On nodes that run the objects compactor, an exact version equal to the object's deletion or wrap now answers NOT_FOUND with that detail, and any version of an object with no entry at all answers NOT_FOUND, where rpc: refuse with UNIMPLEMENTED / BELOW_EARLIEST_AVAILABLE where nodes are not set up or have pruned #28669 refuses both with UNIMPLEMENTED / NOT_SERVED_HERE.
  • JSON-RPC:
  • GraphQL:
  • CLI:
  • Rust SDK:
  • Indexing Framework:

nickvikeras added a commit that referenced this pull request Nov 13, 2026
# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	crates/sui-fork/src/tests/store_execution.rs
#	crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs
#	crates/sui-kvstore/src/bigtable/client/mod.rs
#	crates/sui-kvstore/src/bigtable/mock_server.rs
#	crates/sui-kvstore/src/tables/objects.rs
#	crates/sui-rpc-api/src/reader.rs
#	crates/sui-types/src/storage/read_store.rs
@nickvikeras
nickvikeras force-pushed the nickv/object-removed-error branch from 0f97326 to f113d02 Compare November 13, 2026 04:15
nickvikeras added a commit that referenced this pull request Nov 13, 2026
# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	crates/sui-fork/src/tests/store_execution.rs
#	crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs
#	crates/sui-kvstore/src/bigtable/client/mod.rs
#	crates/sui-kvstore/src/bigtable/mock_server.rs
#	crates/sui-rpc-api/src/reader.rs
#	crates/sui-types/src/storage/read_store.rs
@nickvikeras
nickvikeras force-pushed the nickv/object-removed-error branch from f113d02 to ccb49e7 Compare November 13, 2026 04:53
@nickvikeras
nickvikeras force-pushed the nickv/list-object-versions-fullnode branch from cbbf11a to 698c3ef Compare November 13, 2026 04:53
nickvikeras added a commit that referenced this pull request Nov 13, 2026
# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	crates/sui-fork/src/tests/store_execution.rs
#	crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs
#	crates/sui-kvstore/src/bigtable/client/mod.rs
#	crates/sui-kvstore/src/bigtable/mock_server.rs
#	crates/sui-rpc-api/src/reader.rs
#	crates/sui-types/src/storage/read_store.rs
@nickvikeras
nickvikeras force-pushed the nickv/object-removed-error branch from ccb49e7 to a00d058 Compare November 13, 2026 05:24
@nickvikeras
nickvikeras force-pushed the nickv/list-object-versions-fullnode branch from 698c3ef to 44cdb68 Compare November 13, 2026 05:24
nickvikeras added a commit that referenced this pull request Nov 13, 2026
# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	crates/sui-fork/src/tests/store_execution.rs
#	crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs
#	crates/sui-kvstore/src/bigtable/client/mod.rs
#	crates/sui-kvstore/src/bigtable/mock_server.rs
#	crates/sui-rpc-api/src/reader.rs
#	crates/sui-types/src/storage/read_store.rs
@nickvikeras
nickvikeras force-pushed the nickv/list-object-versions-fullnode branch from 44cdb68 to ba3452c Compare November 13, 2026 11:11
@nickvikeras
nickvikeras force-pushed the nickv/object-removed-error branch from a00d058 to 24e539e Compare November 13, 2026 11:11
nickvikeras added a commit that referenced this pull request Nov 13, 2026
# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	crates/sui-fork/src/tests/store_execution.rs
#	crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs
#	crates/sui-kvstore/src/bigtable/client/mod.rs
#	crates/sui-kvstore/src/bigtable/mock_server.rs
#	crates/sui-rpc-api/src/reader.rs
#	crates/sui-types/src/storage/read_store.rs
@nickvikeras
nickvikeras force-pushed the nickv/list-object-versions-fullnode branch from ba3452c to f34e4e1 Compare November 14, 2026 01:04
@nickvikeras
nickvikeras force-pushed the nickv/object-removed-error branch 2 times, most recently from 6267e23 to 5b59834 Compare November 14, 2026 06:25
@nickvikeras
nickvikeras force-pushed the nickv/list-object-versions-fullnode branch from f34e4e1 to 1b966b0 Compare November 14, 2026 06:25
nickvikeras added a commit that referenced this pull request Nov 14, 2026
# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	crates/sui-fork/src/tests/store_execution.rs
#	crates/sui-indexer-alt-e2e-tests/tests/kv_rpc_tests.rs
#	crates/sui-kv-rpc/src/bigtable_client.rs
#	crates/sui-kvstore/src/bigtable/client/mod.rs
#	crates/sui-kvstore/src/bigtable/mock_server.rs
#	crates/sui-types/src/storage/read_store.rs
@nickvikeras
nickvikeras force-pushed the nickv/list-object-versions-fullnode branch from 1b966b0 to b530dec Compare November 14, 2026 18:50
@nickvikeras
nickvikeras force-pushed the nickv/object-removed-error branch from 5b59834 to 2a6712c Compare November 14, 2026 18:50
@nickvikeras
nickvikeras force-pushed the nickv/list-object-versions-fullnode branch from b530dec to b0e9aa5 Compare November 15, 2026 02:09
@nickvikeras
nickvikeras force-pushed the nickv/object-removed-error branch from 2a6712c to b61901e Compare November 15, 2026 02:09
@nickvikeras nickvikeras mentioned this pull request Nov 15, 2026
1 of 8 tasks
A read that resolves to a version at which the object was deleted or wrapped
still answers NOT_FOUND, now with an OBJECT_REMOVED ErrorInfo naming the
removal's version, kind and, when the server has it, transaction. kv-rpc reads
tombstone rows of the objects table, and the version index for at_checkpoint
reads; the fullnode repeats the read's selection over the execution store's
tombstones and finds the transaction through the transaction bitmap.

The ListObjectVersions checks in get_object_parity_tests and kv_rpc_tests list
with transaction in removal_read_mask and with the default version,kind mask,
so get_object_parity_tests compares both paths between the fullnode and
kv-rpc, and kv_rpc_tests checks both against the lifecycle. Also pins
sui-rust-sdk to #345 cfae7c5 for removal_read_mask.
@nickvikeras
nickvikeras force-pushed the nickv/object-removed-error branch from b61901e to 744b958 Compare November 15, 2026 19:59
@nickvikeras
nickvikeras force-pushed the nickv/list-object-versions-fullnode branch from b0e9aa5 to 0ee2270 Compare November 15, 2026 19:59

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant