Repository navigation
fix: cap IVF partition search at target_partitions only when the caller sets it - #11233
dshepelev15 wants to merge 2 commits into
Conversation
Exec nodes cap their concurrency at DataFusion's target_partitions (lance-format#7087). When the caller does not set it, the session took DataFusion's default, the detected core count, so an oversubscribed LANCE_CPU_THREADS stopped mattering for those nodes. In an 8-CPU container an IVF search loaded at most 8 probed partitions at once from object storage, whatever LANCE_CPU_THREADS said. The default is now max(available_parallelism, LANCE_CPU_THREADS). Without LANCE_CPU_THREADS nothing changes (the CPU pool is cores minus the I/O reservation), and an explicit target_partition still caps the session.
❌ 2 Tests Failed:
View the top 1 failed test(s) by shortest run time
View the full list of 1 ❄️ flaky test(s)
To view more test analytics, go to the Test Analytics Dashboard |
Narrow the fix to the IVF node instead of raising every session's default target_partitions, which also scaled partition counts and the memory pool of all plans. The target_partitions cap from lance-format#7087 now applies only when the session value differs from DataFusion's default (the core count), so an oversubscribed LANCE_CPU_THREADS again sets how many probed partitions load at once, while a caller-set target_partitions still caps it.
There was a problem hiding this comment.
The IVF-local fix restores oversubscription while keeping the existing session and memory-pool defaults. The documented ambiguity between an implicit default and an explicit core-count limit remains an operational caveat; a positive query_parallelism bounds partition-search concurrency within each IVF segment.
| /// oversubscribe partition search: each probed partition is loaded from object | ||
| /// storage, so an 8-CPU container would otherwise load at most 8 at a time. | ||
| fn capped_cpu_pool_size(cpu_pool_size: usize, target_partitions: usize, cores: usize) -> usize { | ||
| if target_partitions == cores { |
There was a problem hiding this comment.
An explicit session target equal to the core count is also bypassed here. With LANCE_CPU_THREADS=64, 8 detected cores, target_partitions=8, and query_parallelism=-1, this selects partition-search concurrency 64 despite the session target of 8. Deployments relying on that target can set query_parallelism=8 to retain the previous concurrency bound within each IVF segment.
The bounded reproduction against the current helper evaluates:
assert_eq!(capped_cpu_pool_size(64, 8, 8), 64);The helper receives the same inputs whether the session target was implicit or explicitly set. This is the edge case already documented in the PR description.
Summary
ANNIvfSubIndexExeclimits partition search, including loading each probed partition from object storage, tomin(LANCE_CPU_THREADS, target_partitions). Thetarget_partitionscap came in with #7087, so a caller can bound a query by loweringtarget_partitions. #7087 assumed the defaulttarget_partitions(the detected core count) is never below the CPU pool. That stops being true onceLANCE_CPU_THREADSis set above the core count, which is how callers oversubscribe for object-storage-bound work. In an 8-CPU container (cgroup quota) an IVF search then loads at most 8 probed partitions at a time, whateverLANCE_CPU_THREADSsays.effective_query_parallelismnow applies thetarget_partitionscap only when the session value differs from DataFusion's default,get_available_parallelism(). The pure decision is incapped_cpu_pool_size.target_partitionsstill caps (fix: cap exec-node parallelism to DataFusion target_partitions #7087).LANCE_CPU_THREADSnothing changes: the pool is cores minusLANCE_IO_CORE_RESERVATION, already below the default.target_partitionsto exactly the core count is treated as the default.Measurements
1B-row table on S3, IVF_PQ (31,623 partitions, PQ 512x8, ~15 MB per partition), k=5000, nprobes=178,
query_parallelism=-1,LANCE_CPU_THREADS=64, process limited to 8 CPUs withsystemd-run -p CPUQuota=800%, warm IVF centroids:User and sys CPU per search stay about the same. Before the fix, a chrome trace showed at most 8
load_partitionspans in flight (64 without the CPU limit), with unchangedget_rangelatency. With all 64 cores free,LANCE_CPU_THREADS=8reproduces the slow case (43 s).Test
test_capped_cpu_pool_size(rstest) covers the default target with an oversubscribed pool and with the default pool, an explicit lower target, an explicit higher target, and an empty pool. The existingtest_effective_query_parallelism_*tests pass unchanged.cargo fmt --allandcargo clippy --all --tests --benches -- -D warningspass.