Repository navigation
[SOL][Fix] Accumulate K/V pooling in FP32 across autotune configurations - #595
Open
lishunyang12 wants to merge 1 commit into
Open
lishunyang12 wants to merge 1 commit into
lishunyang12 wants to merge 1 commit into
Conversation
This was referenced Nov 18, 2026
Author
|
The standalone public-source reproduction now completed on the same SM100 engineering GPU and pinned CUDA environment:
bash reproduce_pool_fp32.sh --workdir /tmp/empty-pool-checkWrapper and environment instructions, raw old/fixed logs and source/environment manifests. |
Collaborator
|
Thanks @lishunyang12! cc @hp-l33 to review and validate on GPU. Note the vendored Sol-H3 copy ( |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SOL's native TMA and pointer preprocessing passed BF16 values directly to
tl.sum. In the tested Triton runtime, the reduction combines and returns BF16 values, so autotuning between 4 and 8 warps changes the reduction tree and pooled K/V values. Those changes can also alter routing thresholds and exact/approximate attention decisions for identical inputs. The T8193 regression reproduces differences in full blocks as well as the partial tail.Cast K/V reduction inputs to FP32 in both preprocessing implementations, then retain the existing BF16 output storage. All autotune configurations and public interfaces are preserved. The new test forces every production configuration instead of allowing the autotuner to hide a differing result. It compares KC and VC against Float64 pooling rounded to BF16, repeats each configuration, checks output bytes across configurations/repeats, and verifies input immutability and pointer-padding sentinels. Cases cover T63/64/65/129/257/8193, full/partial blocks, multiple batches/heads, cancellation-prone dyadic inputs and the seed71 BF16 regression input.
Validation:
py_compile,git diff --check, Ruff E/W/F/UP on all changed files, and E/W/F/I/N/UP on the new test passed. The production files have no new strict Ruff diagnostics; their existing import-format and uppercase-kernel-argument diagnostics are unchanged.Environment: pinned
vllm/vllm-openai:cu134-nightly-1648fefab91204f3b15bf8062217316f07f2c4bb, Torch2.16.0.dev20261009+cu134, CUDA13.4. GPU command:PYTHONPATH=techniques/sparse_backends python3 -m pytest -q techniques/sparse_backends/tests/test_preprocess_precision.py.Redacted raw validation logs, source patches and reproducible scripts; see
results/fp32-validation,results/fp32-matrix/comparison.json, andCORRECTNESS_BLOCKER.txtfor the correction to our earlier attention-kernel attribution. Numerical integration tests do not establish end-to-end generated-video SSIM/PSNR/LPIPS.