Skip to content

fix: cap IVF partition search at target_partitions only when the caller sets it - #11233

Open
dshepelev15 wants to merge 2 commits into
lance-format:mainfrom
dshepelev15:fix/target-partitions-honor-cpu-threads
Open

dshepelev15 wants to merge 2 commits into
lance-format:mainfrom
dshepelev15:fix/target-partitions-honor-cpu-threads

Conversation

@dshepelev15

@dshepelev15 dshepelev15 commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

ANNIvfSubIndexExec limits partition search, including loading each probed partition from object storage, to min(LANCE_CPU_THREADS, target_partitions). The target_partitions cap came in with #7087, so a caller can bound a query by lowering target_partitions. #7087 assumed the default target_partitions (the detected core count) is never below the CPU pool. That stops being true once LANCE_CPU_THREADS is set above the core count, which is how callers oversubscribe for object-storage-bound work. In an 8-CPU container (cgroup quota) an IVF search then loads at most 8 probed partitions at a time, whatever LANCE_CPU_THREADS says.

effective_query_parallelism now applies the target_partitions cap only when the session value differs from DataFusion's default, get_available_parallelism(). The pure decision is in capped_cpu_pool_size.

  • A caller-set target_partitions still caps (fix: cap exec-node parallelism to DataFusion target_partitions #7087).
  • Without LANCE_CPU_THREADS nothing changes: the pool is cores minus LANCE_IO_CORE_RESERVATION, already below the default.
  • Other exec nodes and the session config are untouched. Partition counts and the memory pool of every other plan stay as they are.
  • One edge case: a caller that sets target_partitions to exactly the core count is treated as the default.

Measurements

1B-row table on S3, IVF_PQ (31,623 partitions, PQ 512x8, ~15 MB per partition), k=5000, nprobes=178, query_parallelism=-1, LANCE_CPU_THREADS=64, process limited to 8 CPUs with systemd-run -p CPUQuota=800%, warm IVF centroids:

per search
before 38.8 s, 44.9 s
after 10.3 s, 14.2 s

User and sys CPU per search stay about the same. Before the fix, a chrome trace showed at most 8 load_partition spans in flight (64 without the CPU limit), with unchanged get_range latency. With all 64 cores free, LANCE_CPU_THREADS=8 reproduces the slow case (43 s).

Test

test_capped_cpu_pool_size (rstest) covers the default target with an oversubscribed pool and with the default pool, an explicit lower target, an explicit higher target, and an empty pool. The existing test_effective_query_parallelism_* tests pass unchanged.

cargo fmt --all and cargo clippy --all --tests --benches -- -D warnings pass.

Exec nodes cap their concurrency at DataFusion's target_partitions (lance-format#7087).
When the caller does not set it, the session took DataFusion's default, the
detected core count, so an oversubscribed LANCE_CPU_THREADS stopped mattering
for those nodes. In an 8-CPU container an IVF search loaded at most 8 probed
partitions at once from object storage, whatever LANCE_CPU_THREADS said.

The default is now max(available_parallelism, LANCE_CPU_THREADS). Without
LANCE_CPU_THREADS nothing changes (the CPU pool is cores minus the I/O
reservation), and an explicit target_partition still caps the session.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions github-actions Bot added the bug Something isn't working label Nov 18, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 18, 2026
@codecov

codecov Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

❌ 2 Tests Failed:

Tests completed Failed Passed Skipped
21195 2 21193 0
View the top 1 failed test(s) by shortest run time
lance-io::object_store::providers::aws::tests::test_injected_aws_creds_option_is_used
Stack Traces | 5.13s run time
thread 'object_store::providers::aws::tests::test_injected_aws_creds_option_is_used' (122399) panicked at .../object_store/providers/aws.rs:977:14:
called `Result::unwrap()` on an `Err` value: IO { source: Generic { store: "S3", source: ResolveRegion { bucket: "not-a-bucket", source: reqwest::Error { kind: Request, url: "https://not-a-bucket.s3.amazonaws.com/", source: hyper_util::client::legacy::Error(Connect, TimedOut) } } }, location: Location { file: ".../object_store/providers/aws.rs", line: 372, column: 13 }, backtrace: MaybeBacktrace(Some(Backtrace [{ fn: "<std::backtrace::Backtrace as snafu::GenerateImplicitData>::generate" }, { fn: "<core::option::Option<std::backtrace::Backtrace> as snafu::GenerateImplicitData>::generate" }, { fn: "<lance_core::error::backtrace_support::MaybeBacktrace as snafu::GenerateImplicitData>::generate", file: ".../lance-core/src/error.rs", line: 23 }, { fn: "<lance_core::error::backtrace_support::MaybeBacktrace as snafu::GenerateImplicitData>::generate_with_source", file: "........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/snafu-0.9.2/src/lib.rs", line: 1340 }, { fn: "<lance_core::error::IOSnafu as snafu::IntoError<lance_core::error::Error>>::into_error", file: ".../lance-core/src/error.rs", line: 158 }, { fn: "<lance_core::error::Error>::io_source", file: ".../lance-core/src/error.rs", line: 619 }, { fn: "<lance_core::error::Error as core::convert::From<object_store::Error>>::from", file: ".../lance-core/src/error.rs", line: 950 }, { fn: "<core::result::Result<core::option::Option<alloc::string::String>, lance_core::error::Error> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible, object_store::Error>>>::from_residual", file: ".../core/src/result.rs", line: 2192 }, { fn: "{async_fn#0}", file: "..../object_store/providers/aws.rs", line: 372 }, { fn: "{async_fn#0}", file: "..../object_store/providers/aws.rs", line: 108 }, { fn: "{async_block#0}", file: "..../object_store/providers/aws.rs", line: 296 }, { fn: "poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<lance_io::object_store::ObjectStore, lance_core::error::Error>> + core::marker::Send), alloc::alloc::Global>>", file: ".../src/future/future.rs", line: 133 }, { fn: "{async_fn#0}", file: "......./src/object_store/providers.rs", line: 294 }, { fn: "{async_fn#0}", file: "......./src/object_store/providers.rs", line: 478 }, { fn: "{async_fn#0}", file: "./src/object_store.rs", line: 873 }, { fn: "{async_fn#0}", file: "./src/object_store.rs", line: 800 }, { fn: "{async_block#0}", file: "..../object_store/providers/aws.rs", line: 976 }, { fn: "poll<&mut dyn core::future::future::Future<Output=()>>", file: ".../src/future/future.rs", line: 133 }, { fn: "{closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>", file: "........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs", line: 830 }, { fn: "with_budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>", file: "........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../task/coop/mod.rs", line: 167 }, { fn: "budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>", file: "........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../task/coop/mod.rs", line: 133 }, { fn: "{closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>", file: "........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs", line: 830 }, { fn: "<tokio::runtime::scheduler::current_thread::Context>::enter::<core::task::poll::Poll<()>, <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}::{closure#0}>", file: "........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs", line: 486 }, { fn: "<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on::<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}", file: "........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs", line: 829 }, { fn: "<tokio::runtime::scheduler::current_thread::CoreGuard>::enter::<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}", file: "........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs", line: 906 }, { fn: "<tokio::runtime::context::scoped::Scoped<tokio::runtime::scheduler::Context>>::set::<<tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>)>", file: "........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../runtime/context/scoped.rs", line: 40 }, { fn: "tokio::runtime::context::set_scheduler::<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}>::{closure#0}", file: "........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../src/runtime/context.rs", line: 186 }, { fn: "try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>), tokio::runtime::scheduler::current_thread::{impl#9}::enter::{closure_env#0}<tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure_env#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>, core::option::Option<()>>>, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>)>", file: ".../src/thread/local.rs", line: 463 }, { fn: "<std::thread::local::LocalKey<tokio::runtime::context::Context>>::with::<tokio::runtime::context::set_scheduler<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}>::{closure#0}, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>)>", file: ".../src/thread/local.rs", line: 427 }, { fn: "tokio::runtime::context::set_scheduler::<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}>", file: "........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../src/runtime/context.rs", line: 186 }, { fn: "<tokio::runtime::scheduler::current_thread::CoreGuard>::enter::<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>", file: "........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs", line: 906 }, { fn: "<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on::<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>", file: "........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs", line: 819 }, { fn: "{closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>", file: "........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs", line: 216 }, { fn: "tokio::runtime::context::runtime::enter_runtime::<<tokio::runtime::scheduler::current_thread::CurrentThread>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, ()>", file: "........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../runtime/context/runtime.rs", line: 65 }, { fn: "block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>", file: "........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs", line: 204 }, { fn: "<tokio::runtime::runtime::Runtime>::block_on_inner::<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>", file: "........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../src/runtime/runtime.rs", line: 379 }, { fn: "block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>", file: "........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../src/runtime/runtime.rs", line: 348 }, { fn: "test_injected_aws_creds_option_is_used", file: "..../object_store/providers/aws.rs", line: 988 }, { fn: "lance_io::object_store::providers::aws::tests::test_injected_aws_creds_option_is_used::{closure#0}", file: "..../object_store/providers/aws.rs", line: 963 }, { fn: "<lance_io::object_store::providers::aws::tests::test_injected_aws_creds_option_is_used::{closure#0} as core::ops::function::FnOnce<()>>::call_once", file: ".../src/ops/function.rs", line: 250 }, { fn: "<fn() -> core::result::Result<(), alloc::string::String> as core::ops::function::FnOnce<()>>::call_once", file: ".../src/ops/function.rs", line: 250 }, { fn: "test::__rust_begin_short_backtrace::<core::result::Result<(), alloc::string::String>, fn() -> core::result::Result<(), alloc::string::String>>", file: ".../test/src/lib.rs", line: 724 }, { fn: "test::run_test_in_process::{closure#0}", file: ".../test/src/lib.rs", line: 747 }, { fn: "<core::panic::unwind_safe::AssertUnwindSafe<test::run_test_in_process::{closure#0}> as core::ops::function::FnOnce<()>>::call_once", file: ".../src/panic/unwind_safe.rs", line: 275 }, { fn: "std::panicking::catch_unwind::do_call::<core::panic::unwind_safe::AssertUnwindSafe<test::run_test_in_process::{closure#0}>, core::result::Result<(), alloc::string::String>>", file: ".../std/src/panicking.rs", line: 576 }, { fn: "std::panicking::catch_unwind::<core::result::Result<(), alloc::string::String>, core::panic::unwind_safe::AssertUnwindSafe<test::run_test_in_process::{closure#0}>>", file: ".../std/src/panicking.rs", line: 544 }, { fn: "std::panic::catch_unwind::<core::panic::unwind_safe::AssertUnwindSafe<test::run_test_in_process::{closure#0}>, core::result::Result<(), alloc::string::String>>", file: ".../std/src/panic.rs", line: 359 }, { fn: "test::run_test_in_process", file: ".../test/src/lib.rs", line: 747 }, { fn: "test::run_test::{closure#0}", file: ".../test/src/lib.rs", line: 668 }, { fn: "test::run_test::{closure#1}", file: ".../test/src/lib.rs", line: 698 }, { fn: "std::sys::backtrace::__rust_begin_short_backtrace::<test::run_test::{closure#1}, ()>", file: ".../src/sys/backtrace.rs", line: 166 }, { fn: "std::thread::lifecycle::spawn_unchecked::<test::run_test::{closure#1}, ()>::{closure#1}::{closure#0}", file: ".../src/thread/lifecycle.rs", line: 70 }, { fn: "<core::panic::unwind_safe::AssertUnwindSafe<std::thread::lifecycle::spawn_unchecked<test::run_test::{closure#1}, ()>::{closure#1}::{closure#0}> as core::ops::function::FnOnce<()>>::call_once", file: ".../src/panic/unwind_safe.rs", line: 275 }, { fn: "std::panicking::catch_unwind::do_call::<core::panic::unwind_safe::AssertUnwindSafe<std::thread::lifecycle::spawn_unchecked<test::run_test::{closure#1}, ()>::{closure#1}::{closure#0}>, ()>", file: ".../std/src/panicking.rs", line: 576 }, { fn: "std::panicking::catch_unwind::<(), core::panic::unwind_safe::AssertUnwindSafe<std::thread::lifecycle::spawn_unchecked<test::run_test::{closure#1}, ()>::{closure#1}::{closure#0}>>", file: ".../std/src/panicking.rs", line: 544 }, { fn: "std::panic::catch_unwind::<core::panic::unwind_safe::AssertUnwindSafe<std::thread::lifecycle::spawn_unchecked<test::run_test::{closure#1}, ()>::{closure#1}::{closure#0}>, ()>", file: ".../std/src/panic.rs", line: 359 }, { fn: "std::thread::lifecycle::spawn_unchecked::<test::run_test::{closure#1}, ()>::{closure#1}", file: ".../src/thread/lifecycle.rs", line: 68 }, { fn: "<std::thread::lifecycle::spawn_unchecked<test::run_test::{closure#1}, ()>::{closure#1} as core::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}", file: ".../src/ops/function.rs", line: 250 }, { fn: "<alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output = ()> + core::marker::Send> as core::ops::function::FnOnce<()>>::call_once", file: ".../alloc/src/boxed.rs", line: 2319 }, { fn: "<std::sys::thread::unix::Thread>::new::thread_start", file: ".../sys/thread/unix.rs", line: 123 }])) }
stack backtrace:
   0: __rustc::rust_begin_unwind
             at .../std/src/panicking.rs:679:5
   1: core::panicking::panic_fmt
             at .../core/src/panicking.rs:80:14
   2: core::result::unwrap_failed
             at .../core/src/result.rs:1870:5
   3: unwrap<(alloc::sync::Arc<lance_io::object_store::ObjectStore, alloc::alloc::Global>, object_store::path::Path), lance_core::error::Error>
             at .../core/src/result.rs:1231:23
   4: {async_block#0}
             at ..../object_store/providers/aws.rs:977:14
   5: poll<&mut dyn core::future::future::Future<Output=()>>
             at .../src/future/future.rs:133:9
   6: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at ........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:830:70
   7: with_budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
             at ........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../task/coop/mod.rs:167:5
   8: budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
             at ........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../task/coop/mod.rs:133:5
   9: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at ........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:830:25
  10: <tokio::runtime::scheduler::current_thread::Context>::enter::<core::task::poll::Poll<()>, <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}::{closure#0}>
             at ........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:486:19
  11: <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on::<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}
             at ........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:829:44
  12: <tokio::runtime::scheduler::current_thread::CoreGuard>::enter::<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}
             at ........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:906:68
  13: <tokio::runtime::context::scoped::Scoped<tokio::runtime::scheduler::Context>>::set::<<tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>)>
             at ........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../runtime/context/scoped.rs:40:9
  14: tokio::runtime::context::set_scheduler::<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}>::{closure#0}
             at ........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../src/runtime/context.rs:186:38
  15: try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>), tokio::runtime::scheduler::current_thread::{impl#9}::enter::{closure_env#0}<tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure_env#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>, core::option::Option<()>>>, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>)>
             at .../src/thread/local.rs:463:12
  16: <std::thread::local::LocalKey<tokio::runtime::context::Context>>::with::<tokio::runtime::context::set_scheduler<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}>::{closure#0}, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>)>
             at .../src/thread/local.rs:427:20
  17: tokio::runtime::context::set_scheduler::<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}>
             at ........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../src/runtime/context.rs:186:17
  18: <tokio::runtime::scheduler::current_thread::CoreGuard>::enter::<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>
             at ........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:906:27
  19: <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on::<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>
             at ........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:819:24
  20: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at ........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:216:33
  21: tokio::runtime::context::runtime::enter_runtime::<<tokio::runtime::scheduler::current_thread::CurrentThread>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, ()>
             at ........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../runtime/context/runtime.rs:65:16
  22: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at ........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:204:9
  23: <tokio::runtime::runtime::Runtime>::block_on_inner::<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>
             at ........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../src/runtime/runtime.rs:379:52
  24: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at ........................................................................................................./home/runner/.cargo........................................................................................................./registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../src/runtime/runtime.rs:348:18
  25: test_injected_aws_creds_option_is_used
             at ..../object_store/providers/aws.rs:988:62
  26: lance_io::object_store::providers::aws::tests::test_injected_aws_creds_option_is_used::{closure#0}
             at ..../object_store/providers/aws.rs:963:54
  27: <lance_io::object_store::providers::aws::tests::test_injected_aws_creds_option_is_used::{closure#0} as core::ops::function::FnOnce<()>>::call_once
             at .../src/ops/function.rs:250:5
  28: <fn() -> core::result::Result<(), alloc::string::String> as core::ops::function::FnOnce<()>>::call_once
             at .../src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
View the full list of 1 ❄️ flaky test(s)
lance-io::scheduler::tests::test_object_store_selects_scheduler

Flake rate in main: 100.00% (Passed 0 times, Failed 2 times)

Stack Traces | 0.171s run time
thread 'scheduler::tests::test_object_store_selects_scheduler' (123881) panicked at .../lance-io/src/scheduler.rs:3227:9:
assertion failed: uring_store.prefers_lite_scheduler()
stack backtrace:
   0: __rustc::rust_begin_unwind
             at .../std/src/panicking.rs:679:5
   1: core::panicking::panic_fmt
             at .../core/src/panicking.rs:80:14
   2: core::panicking::panic
             at .../core/src/panicking.rs:150:5
   3: {async_block#0}
             at ./src/scheduler.rs:3227:9
   4: poll<&mut dyn core::future::future::Future<Output=()>>
             at .../src/future/future.rs:133:9
   5: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:830:70
   6: with_budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../task/coop/mod.rs:167:5
   7: budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../task/coop/mod.rs:133:5
   8: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:830:25
   9: <tokio::runtime::scheduler::current_thread::Context>::enter::<core::task::poll::Poll<()>, <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}::{closure#0}>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:486:19
  10: <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on::<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:829:44
  11: <tokio::runtime::scheduler::current_thread::CoreGuard>::enter::<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:906:68
  12: <tokio::runtime::context::scoped::Scoped<tokio::runtime::scheduler::Context>>::set::<<tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>)>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../runtime/context/scoped.rs:40:9
  13: tokio::runtime::context::set_scheduler::<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}>::{closure#0}
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../src/runtime/context.rs:186:38
  14: try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>), tokio::runtime::scheduler::current_thread::{impl#9}::enter::{closure_env#0}<tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure_env#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>, core::option::Option<()>>>, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>)>
             at .../src/thread/local.rs:463:12
  15: <std::thread::local::LocalKey<tokio::runtime::context::Context>>::with::<tokio::runtime::context::set_scheduler<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}>::{closure#0}, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>)>
             at .../src/thread/local.rs:427:20
  16: tokio::runtime::context::set_scheduler::<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>::{closure#0}>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../src/runtime/context.rs:186:17
  17: <tokio::runtime::scheduler::current_thread::CoreGuard>::enter::<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, core::option::Option<()>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:906:27
  18: <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on::<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:819:24
  19: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:216:33
  20: tokio::runtime::context::runtime::enter_runtime::<<tokio::runtime::scheduler::current_thread::CurrentThread>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, ()>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../runtime/context/runtime.rs:65:16
  21: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../scheduler/current_thread/mod.rs:204:9
  22: <tokio::runtime::runtime::Runtime>::block_on_inner::<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../src/runtime/runtime.rs:379:52
  23: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at .................................................../home/runner/.cargo.................................................../registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.54..../src/runtime/runtime.rs:348:18
  24: test_object_store_selects_scheduler
             at ./src/scheduler.rs:3249:49
  25: lance_io::scheduler::tests::test_object_store_selects_scheduler::{closure#0}
             at ./src/scheduler.rs:3204:51
  26: <lance_io::scheduler::tests::test_object_store_selects_scheduler::{closure#0} as core::ops::function::FnOnce<()>>::call_once
             at .../src/ops/function.rs:250:5
  27: <fn() -> core::result::Result<(), alloc::string::String> as core::ops::function::FnOnce<()>>::call_once
             at .../src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

Narrow the fix to the IVF node instead of raising every session's default
target_partitions, which also scaled partition counts and the memory pool of
all plans. The target_partitions cap from lance-format#7087 now applies only when the
session value differs from DataFusion's default (the core count), so an
oversubscribed LANCE_CPU_THREADS again sets how many probed partitions load
at once, while a caller-set target_partitions still caps it.
@dshepelev15 dshepelev15 changed the title fix: let LANCE_CPU_THREADS raise the default target_partitions fix: cap IVF partition search at target_partitions only when the caller sets it Nov 18, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 18, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Gate recommendation: approve with a non-blocking risk.

The IVF-local fix restores oversubscription while keeping the existing session and memory-pool defaults. The documented ambiguity between an implicit default and an explicit core-count limit remains an operational caveat; a positive query_parallelism bounds partition-search concurrency within each IVF segment.

/// oversubscribe partition search: each probed partition is loaded from object
/// storage, so an 8-CPU container would otherwise load at most 8 at a time.
fn capped_cpu_pool_size(cpu_pool_size: usize, target_partitions: usize, cores: usize) -> usize {
if target_partitions == cores {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An explicit session target equal to the core count is also bypassed here. With LANCE_CPU_THREADS=64, 8 detected cores, target_partitions=8, and query_parallelism=-1, this selects partition-search concurrency 64 despite the session target of 8. Deployments relying on that target can set query_parallelism=8 to retain the previous concurrency bound within each IVF segment.

The bounded reproduction against the current helper evaluates:

assert_eq!(capped_cpu_pool_size(64, 8, 8), 64);

The helper receives the same inputs whether the session target was implicit or explicitly set. This is the edge case already documented in the PR description.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Nov 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant