Skip to content

[SOL][Fix] Accumulate K/V pooling in FP32 across autotune configurations - #595

Open
lishunyang12 wants to merge 1 commit into
NVlabs:sol-enginefrom
lishunyang12:fix/sol-fp32-pooling-20261118
Open

lishunyang12 wants to merge 1 commit into
NVlabs:sol-enginefrom
lishunyang12:fix/sol-fp32-pooling-20261118

Conversation

@lishunyang12

Copy link
Copy Markdown

SOL's native TMA and pointer preprocessing passed BF16 values directly to tl.sum. In the tested Triton runtime, the reduction combines and returns BF16 values, so autotuning between 4 and 8 warps changes the reduction tree and pooled K/V values. Those changes can also alter routing thresholds and exact/approximate attention decisions for identical inputs. The T8193 regression reproduces differences in full blocks as well as the partial tail.

Cast K/V reduction inputs to FP32 in both preprocessing implementations, then retain the existing BF16 output storage. All autotune configurations and public interfaces are preserved. The new test forces every production configuration instead of allowing the autotuner to hide a differing result. It compares KC and VC against Float64 pooling rounded to BF16, repeats each configuration, checks output bytes across configurations/repeats, and verifies input immutability and pointer-padding sentinels. Cases cover T63/64/65/129/257/8193, full/partial blocks, multiple batches/heads, cancellation-prone dyadic inputs and the seed71 BF16 regression input.

Validation:

  • SM100 engineering GPU (cluster label B100; driver reports NVIDIA Graphics Device, CC10.0,148SMs,183359MiB; not a production B200 validation): all 12 new pooling tests passed. A separate native sweep over 4 sequence lengths and all 8 configurations produced zero KC/K-as-VC mismatches against Float64-to-BF16 references.
  • Integration validation of the same shared fix on Support arbitrary exact Q/KV blocks in fused Sol-Attn #593 and [SOL][Perf] Reduce SM100 rescale and full-tile mask overhead #594: all 73 attention checks passed; 29 independent-process cases had identical input, prepared KC/VC/threshold, and output hashes.
  • Local: py_compile, git diff --check, Ruff E/W/F/UP on all changed files, and E/W/F/I/N/UP on the new test passed. The production files have no new strict Ruff diagnostics; their existing import-format and uppercase-kernel-argument diagnostics are unchanged.

Environment: pinned vllm/vllm-openai:cu134-nightly-1648fefab91204f3b15bf8062217316f07f2c4bb, Torch 2.16.0.dev20261009+cu134, CUDA13.4. GPU command: PYTHONPATH=techniques/sparse_backends python3 -m pytest -q techniques/sparse_backends/tests/test_preprocess_precision.py.

Redacted raw validation logs, source patches and reproducible scripts; see results/fp32-validation, results/fp32-matrix/comparison.json, and CORRECTNESS_BLOCKER.txt for the correction to our earlier attention-kernel attribution. Numerical integration tests do not establish end-to-end generated-video SSIM/PSNR/LPIPS.

@lishunyang12

Copy link
Copy Markdown
Author

The standalone public-source reproduction now completed on the same SM100 engineering GPU and pinned CUDA environment:

  • Unmodified cf67a3a: 12/12 numerical failures against the Float64-to-BF16 references.
  • This PR 0f0d973: 12/12 passed; no collection/runtime errors or hardware skips on either run.
  • The wrapper downloads both SHA-pinned public archives and checks that the entire archive diff contains only the four casts plus the exact regression test. No weights are required.
bash reproduce_pool_fp32.sh --workdir /tmp/empty-pool-check

Wrapper and environment instructions, raw old/fixed logs and source/environment manifests.

@lawrence-cj

Copy link
Copy Markdown
Collaborator

Thanks @lishunyang12! cc @hp-l33 to review and validate on GPU. Note the vendored Sol-H3 copy (models/minimax_h3/Sol-H3/h3_runtime/third_party/sol_attn/) has the same BF16 tl.sum and needs the same fix.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants