Skip to content

refactor(io): route direct object requests through the Lance facade - #11223

Open
Xuanwo wants to merge 5 commits into
mainfrom
xuanwo/lance-storage-3-operations
Open

Xuanwo wants to merge 5 commits into
mainfrom
xuanwo/lance-storage-3-operations

Conversation

@Xuanwo

@Xuanwo Xuanwo commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Depends on #11221.

Move direct GET, ranged GET, HEAD, PUT, LIST, rename and conditional-copy calls behind the Lance handle. Dataset, MemWAL, table and directory-namespace code use the facade. A caller can still distinguish failure to open a response from failure to consume its body.

Expose strict create-only copy separately from the existing staged-file promotion helper that accepts an existing destination with the same size. Namespace marker and manifest fallbacks keep their own conflict handling and cleanup order.

Storage stack (3/6) · #11220 → #11221 → #11223 → #11225 → #11226 → #11227

@github-actions github-actions Bot added A-encoding Encoding, IO, file reader/writer A-namespace Namespace impls labels Nov 18, 2026
Xuanwo added a commit that referenced this pull request Nov 18, 2026
…name (#11220)

First of six draft changes to the Lance storage API, based on c377b70.

Local conditional writes stage beside the destination using a
fixed-length basename. Construct the sibling from the parsed parent to
preserve encoded path components. The existing conditional rename,
conflict result, and cleanup behavior are preserved.

The local regression writes twice under a non-ASCII parent using a
241-byte basename and checks that the original bytes survive the
conflict. The two put-if-absent cases and workspace Clippy also passed
with this patch applied to main at 134c2bf. Main already preserves
parent encoding for namespace marker staging.

**Storage stack (1/6)** · #11220 → #11221 → #11223 → #11225 → #11226 →
#11227

The six commits preserve the reviewed tree based on `c377b70b6`. Layer 1
also passed its targeted tests and workspace Clippy on main at
`134c2bf95`. Integrating the later layers with newer main changes to S3
markers, cleanup and shallow clones remains separate work.
Xuanwo added a commit that referenced this pull request Nov 18, 2026
Depends on #11220.

A wrapper installed by the registry or on an existing handle must see
local file reads, writes and copies, including external blob reads. Keep
native directory maintenance available on built-in file stores. Track
the ordinary and external views once at assembly so later wrappers
retain their existing relationship to the tracker.

Verified 113 lance-io object store tests, including local,
file-object-store and Linux io_uring with registry and late wrappers.
Workspace Clippy passed for tests and benches.

**Storage stack (2/6)** · #11220 → #11221 → #11223 → #11225 → #11226 →
#11227
Base automatically changed from xuanwo/lance-storage-2-wrappers to main November 18, 2026 17:42
@Xuanwo
Xuanwo marked this pull request as ready for review November 18, 2026 17:59

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

The migrated paths preserve backend preconditions, streaming behavior, and the separation between request errors and body errors. Strict create-only copy remains separate from staged promotion's same-size retry policy.

Please mark this PR with the breaking-change label: lance_io::object_store::ObjectStore::copy_if_not_exists now has strict create-only semantics and returns object_store::Result<()>, so callers relying on same-size retries or WriteResult must use copy_if_not_exists_or_same_size.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 18, 2026
@codecov

codecov Bot commented Nov 18, 2026

Copy link
Copy Markdown

❌ 1 Tests Failed:

Tests completed Failed Passed Skipped
21174 1 21173 0
View the full list of 1 ❄️ flaky test(s)
lance-io::scheduler::tests::test_object_store_selects_scheduler

Flake rate in main: 100.00% (Passed 0 times, Failed 1 times)

Stack Traces | 0.123s run time
thread 'scheduler::tests::test_object_store_selects_scheduler' (124476) panicked at .../lance-io/src/scheduler.rs:3227:9:
assertion failed: uring_store.prefers_lite_scheduler()
stack backtrace:
   0: __rustc::rust_begin_unwind
             at .../std/src/panicking.rs:679:5
   1: core::panicking::panic_fmt
             at .../core/src/panicking.rs:80:14
   2: core::panicking::panic
             at .../core/src/panicking.rs:150:5
   3: {async_block#0}
             at ./src/scheduler.rs:3227:9
   4: poll<&mut dyn core::future::future::Future<Output=()>>
             at .../src/future/future.rs:133:9
   5: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:830:70
   6: with_budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../task/coop/mod.rs:167:5
   7: budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../task/coop/mod.rs:133:5
   8: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:830:25
   9: <tokio::runtime::scheduler::current_thread::Context>::enter::<core::task::poll::Poll<()>, <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}::{closure#0}>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:486:19
  10: <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on::<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:829:44
  11: <tokio::runtime::scheduler::current_thread::CoreGuard>::enter::<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:906:68
  12: <tokio::runtime::context::scoped::Scoped<tokio::runtime::scheduler::Context>>::set::<<tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>)>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../runtime/context/scoped.rs:40:9
  13: tokio::runtime::context::set_scheduler::<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}>::{closure#0}
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../src/runtime/context.rs:186:38
  14: try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>), tokio::runtime::scheduler::current_thread::{impl#9}::enter::{closure_env#0}<tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure_env#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>, core::option::Option<()>>>, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>)>
             at .../src/thread/local.rs:463:12
  15: <std::thread::local::LocalKey<tokio::runtime::context::Context>>::with::<tokio::runtime::context::set_scheduler<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}>::{closure#0}, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>)>
             at .../src/thread/local.rs:427:20
  16: tokio::runtime::context::set_scheduler::<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../src/runtime/context.rs:186:17
  17: <tokio::runtime::scheduler::current_thread::CoreGuard>::enter::<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:906:27
  18: <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on::<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:819:24
  19: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:216:33
  20: tokio::runtime::context::runtime::enter_runtime::<<tokio::runtime::scheduler::current_thread::CurrentThread>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, ()>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../runtime/context/runtime.rs:65:16
  21: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:204:9
  22: <tokio::runtime::runtime::Runtime>::block_on_inner::<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../src/runtime/runtime.rs:379:52
  23: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../src/runtime/runtime.rs:348:18
  24: test_object_store_selects_scheduler
             at ./src/scheduler.rs:3249:49
  25: lance_io::scheduler::tests::test_object_store_selects_scheduler::{closure#0}
             at ./src/scheduler.rs:3204:51
  26: <lance_io::scheduler::tests::test_object_store_selects_scheduler::{closure#0} as core::ops::function::FnOnce<()>>::call_once
             at .../src/ops/function.rs:250:5
  27: <fn() -> core::result::Result<(), alloc::string::String> as core::ops::function::FnOnce<()>>::call_once
             at .../src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-encoding Encoding, IO, file reader/writer A-namespace Namespace impls K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants