Skip to content

fix(io): route wrapped local file IO through the selected store - #11221

Merged
Xuanwo merged 3 commits into
mainfrom
xuanwo/lance-storage-2-wrappers
Nov 18, 2026
Merged

Xuanwo merged 3 commits into
mainfrom
xuanwo/lance-storage-2-wrappers

Conversation

@Xuanwo

@Xuanwo Xuanwo commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Depends on #11220.

A wrapper installed by the registry or on an existing handle must see local file reads, writes and copies, including external blob reads. Keep native directory maintenance available on built-in file stores. Track the ordinary and external views once at assembly so later wrappers retain their existing relationship to the tracker.

Verified 113 lance-io object store tests, including local, file-object-store and Linux io_uring with registry and late wrappers. Workspace Clippy passed for tests and benches.

Storage stack (2/6) · #11220 → #11221 → #11223 → #11225 → #11226 → #11227

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

Using the existing object-store path for wrapped handles addresses the local I/O interception problem. Keeping direct access as a capability of unwrapped built-in file stores is a reasonable boundary for this focused fix.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 18, 2026
Xuanwo added a commit that referenced this pull request Nov 18, 2026
…name (#11220)

First of six draft changes to the Lance storage API, based on c377b70.

Local conditional writes stage beside the destination using a
fixed-length basename. Construct the sibling from the parsed parent to
preserve encoded path components. The existing conditional rename,
conflict result, and cleanup behavior are preserved.

The local regression writes twice under a non-ASCII parent using a
241-byte basename and checks that the original bytes survive the
conflict. The two put-if-absent cases and workspace Clippy also passed
with this patch applied to main at 134c2bf. Main already preserves
parent encoding for namespace marker staging.

**Storage stack (1/6)** · #11220 → #11221 → #11223 → #11225 → #11226 →
#11227

The six commits preserve the reviewed tree based on `c377b70b6`. Layer 1
also passed its targeted tests and workspace Clippy on main at
`134c2bf95`. Integrating the later layers with newer main changes to S3
markers, cleanup and shallow clones remains separate work.
Base automatically changed from xuanwo/lance-storage-1-staging to main November 18, 2026 17:36
@Xuanwo
Xuanwo merged commit a863c71 into main Nov 18, 2026
27 of 33 checks passed
@Xuanwo
Xuanwo deleted the xuanwo/lance-storage-2-wrappers branch November 18, 2026 17:42
@codecov

codecov Bot commented Nov 18, 2026

Copy link
Copy Markdown

❌ 1 Tests Failed:

Tests completed Failed Passed Skipped
21170 1 21169 0
View the top 1 failed test(s) by shortest run time
lance-io::scheduler::tests::test_object_store_selects_scheduler
Stack Traces | 0.122s run time
thread 'scheduler::tests::test_object_store_selects_scheduler' (123803) panicked at .../lance-io/src/scheduler.rs:3227:9:
assertion failed: uring_store.prefers_lite_scheduler()
stack backtrace:
   0: __rustc::rust_begin_unwind
             at .../std/src/panicking.rs:679:5
   1: core::panicking::panic_fmt
             at .../core/src/panicking.rs:80:14
   2: core::panicking::panic
             at .../core/src/panicking.rs:150:5
   3: {async_block#0}
             at ./src/scheduler.rs:3227:9
   4: poll<&mut dyn core::future::future::Future<Output=()>>
             at .../src/future/future.rs:133:9
   5: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:830:70
   6: with_budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../task/coop/mod.rs:167:5
   7: budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../task/coop/mod.rs:133:5
   8: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:830:25
   9: <tokio::runtime::scheduler::current_thread::Context>::enter::<core::task::poll::Poll<()>, <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}::{closure#0}>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:486:19
  10: <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on::<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:829:44
  11: <tokio::runtime::scheduler::current_thread::CoreGuard>::enter::<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:906:68
  12: <tokio::runtime::context::scoped::Scoped<tokio::runtime::scheduler::Context>>::set::<<tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>)>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../runtime/context/scoped.rs:40:9
  13: tokio::runtime::context::set_scheduler::<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}>::{closure#0}
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../src/runtime/context.rs:186:38
  14: try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>), tokio::runtime::scheduler::current_thread::{impl#9}::enter::{closure_env#0}<tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure_env#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>, core::option::Option<()>>>, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>)>
             at .../src/thread/local.rs:463:12
  15: <std::thread::local::LocalKey<tokio::runtime::context::Context>>::with::<tokio::runtime::context::set_scheduler<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}>::{closure#0}, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>)>
             at .../src/thread/local.rs:427:20
  16: tokio::runtime::context::set_scheduler::<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../src/runtime/context.rs:186:17
  17: <tokio::runtime::scheduler::current_thread::CoreGuard>::enter::<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:906:27
  18: <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on::<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:819:24
  19: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:216:33
  20: tokio::runtime::context::runtime::enter_runtime::<<tokio::runtime::scheduler::current_thread::CurrentThread>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, ()>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../runtime/context/runtime.rs:65:16
  21: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:204:9
  22: <tokio::runtime::runtime::Runtime>::block_on_inner::<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../src/runtime/runtime.rs:379:52
  23: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../src/runtime/runtime.rs:348:18
  24: test_object_store_selects_scheduler
             at ./src/scheduler.rs:3249:49
  25: lance_io::scheduler::tests::test_object_store_selects_scheduler::{closure#0}
             at ./src/scheduler.rs:3204:51
  26: <lance_io::scheduler::tests::test_object_store_selects_scheduler::{closure#0} as core::ops::function::FnOnce<()>>::call_once
             at .../src/ops/function.rs:250:5
  27: <fn() -> core::result::Result<(), alloc::string::String> as core::ops::function::FnOnce<()>>::call_once
             at .../src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

Xuanwo added a commit that referenced this pull request Nov 18, 2026
…#11234)

Rust CI is red on main in `lance-io`:
`test_object_store_selects_scheduler` fails on Linux, macOS and Windows
at `assertion failed: uring_store.prefers_lite_scheduler()`.

#11221 made direct local I/O a provider declaration, so the lite
scheduler preference now follows what the store will actually do. The
test built its uring store with the legacy `ObjectStore::new`, which
never declares that mapping, and it asserted the preference on platforms
where the `file+uring` provider is not registered.

Build the store with `ObjectStore::from_uri`, the path real callers use,
and keep the uring assertions where the scheme exists (Linux). The
memory-store preference and the explicit `Some(true)` override still run
on every platform; no runtime behavior changes.

Validated on Linux with the `ci` profile: `lance-io` library tests (700
passed, including the previously failing test) and workspace Clippy.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-encoding Encoding, IO, file reader/writer bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants