Skip to content

Support arbitrary exact Q/KV blocks in fused Sol-Attn - #593

Open
lishunyang12 wants to merge 4 commits into
NVlabs:sol-enginefrom
lishunyang12:feat/sol-exact-block-mask
Open

lishunyang12 wants to merge 4 commits into
NVlabs:sol-enginefrom
lishunyang12:feat/sol-exact-block-mask

Conversation

@lishunyang12

@lishunyang12 lishunyang12 commented Nov 18, 2026 •

Copy link
Copy Markdown

MiniMax-H3 Ref2VA can interleave text/audio segments with reference and target video. The existing contiguous KV sink cannot protect those disjoint segments on both attention axes without a separate integration-side dense/residual path.

Add an optional exact_block_mask Boolean vector (one entry per 64-token block, shared across batch/heads). Marked Q rows and KV columns use the exact route inside the existing fused routing loop. Omitted masks preserve the current API; the legacy KV sink can still be combined with the mask. Omni can forward its model-owned metadata directly through its custom op.

The argument is threaded through CuTe SM89, SM90 (including split KV), SM100/SM103, SM120, and both Triton implementations. CuTe caches distinguish mask presence while mask values remain runtime inputs. Input shape/device/dtype/contiguity validation and independent token-level hybrid oracle tests are included.

Earlier validation at PR head 1c522ed3e1e5d43f24258f6cc27bb7b7e189d508, after merging current sol-engine and resolving the dispatch conflict:

  • Full sparse-backend suite on GH200 / SM90: 47 passed, 17 skipped.
  • Full sparse-backend suite on GB200 / SM100: 45 passed, 19 skipped.
  • Full sparse-backend suite on RTX 6000D / SM120: 45 passed, 19 skipped; runtime confirms cute_sm120 and imports this exact PR source. SM120 skips are 11 SM80-only cases, 6 MPS-only cases and 2 SM90-only KV-split cases.
  • This includes independent hybrid-oracle protection of disjoint Q rows/KV columns, mutable/malformed masks, nonzero byte offsets, and 65,537-token SM90 KV splits 2/4. Native cases execute CuTe; TMA and pointer Triton paths are exercised separately.
  • SM80, SM89 and MPS hardware remains untested. Metal rejects non-None masks explicitly, with a hardware-independent regression test.
  • The release-pinned backport 088897e0c6a9cb970f00b43ab2177c3be7336b36 on f4035e0a separately passes 35 exact-block cases on GH200 and 33 cases on each of GB200 / RTX 6000D (2 SM90-only cases skipped). The original SM90/SM100/SM120 forward measurements use this pin; the SM103 follow-up below uses d9ef632.

Commands from a checkout with techniques/sparse_backends installed:

python3 -m pytest -q techniques/sparse_backends/tests

Environment: vllm/vllm-openai:cu134-nightly-1648fefab91204f3b15bf8062217316f07f2c4bb, Python 3.12.3, PyTorch 2.16.0.dev20261009+cu134, Triton 3.8.0+git3f6e4113, CuTe DSL 4.8.0; pytest 9.2.0. Related integration: vllm-project/vllm-omni#10513.

Redacted logs, source manifest and reproducible commands are published alongside the related Omni benchmark evidence.

The downstream H3 first-forward comparison now covers all three architectures; see the full report. SM100 results include an explicit power/clock limitation. The measured dependency and current upstream PR provenance remain separate.

SM100 power controls and SM103 follow-up

The SM103 dispatch fix is pushed: upstream PR head 63b4e65e9db1dfea45cc7d394902145c2f520821, release-pinned backport d9ef6321230cff13398cfc86aa107f97f093e925, Omni head ca5cc9fdeddf654d2ebaf08d116d46c15327a27e. SM103 was absent from both the CuTe architecture map and SM100 launch branch. It now automatically uses the existing upstream Blackwell kernel family; no provider/native flag or approximation setting is added.

Same complete first-step H3 forward, BF16/eager/TP1, official request/checkpoint, 3 warmup pairs + 9 alternating measured pairs per process. Speedup is Dense P50 / SOL P50, so values below 1 mean SOL is slower.

Hardware / operating condition Dense P50 (s) SOL P50 (s) Dense / SOL
2× GB200, default 7.331 7.042 1.041×
2× GB200, fixed 600 MHz, process 1 11.190 13.634 0.821×
2× GB200, fixed 600 MHz, process 2 11.204 13.635 0.822×
2× GB200, fixed 750 MHz 9.000 10.942 0.822×
2× GB300, CuTe default, process 1 3.880 4.328 0.896×
2× GB300, CuTe default, process 2 3.877 4.321 0.897×
2× GB300, CuTe fixed 1200 MHz 4.949 6.723 0.736×
1× B300A engineering device, CuTe default 15.647 17.477 0.895×

SM100: explicit site-delegated nvidia-smi clock management worked despite Slurm's disabled frequency control. Only allocated GPUs were locked and all clock locks were restored. At 600/750 MHz, measured telemetry polls report the requested clocks with no software power-cap or thermal-throttle flags. The 1200 W default limit was already the firmware maximum and was not raised. Matched-clock SOL is about 22% slower, reproduced across two 600 MHz processes and a separate 750 MHz process. Default-node gains are power/clock dependent, not a clock-normalized kernel speedup. Underclocking is a diagnostic control, not a production speed optimization. Inputs and each implementation's outputs are byte-identical across all four processes.

At 750 MHz, a separate post-measurement profile shows the 50 Dense attention kernels total 6.493 / 6.494 s on ranks 0/1 versus SOL 8.372 / 8.343 s; major GEMM totals are nearly unchanged. The gap is in upstream attention execution. These kernel activity sums are diagnostic, not wall-latency samples.

SM103: automatic CuTe dispatch reduces SOL latency from 20.681 → 17.477 s on B300A (15.5%) and 5.052 → 4.321 s on GB300 (14.5%) compared with the old Triton route. It still loses to Dense. The two GB300 default CuTe processes have about 0.23% CV and byte-identical per-implementation outputs. GB300 default Dense also hits its 1400 W limit (sampled median ~1.54 GHz versus SOL 2.07 GHz); the 1200 MHz row is a separate controlled comparison, not pooled with default-clock results. At 1200 MHz, Dense is 4.949 s versus SOL 6.723 s (35.8% slower), with only 1200 MHz in measured polls and no power/thermal flags. Its outputs are byte-identical to the default-clock runs and its clock locks were restored.

B300A is the cluster's SM103/GB112 engineering device: driver name NVIDIA Graphics Device, ~139 GiB VRAM and a 600 W maximum cap. It is not a full production B300 result. It uses one GPU and equal CPU offload for both paths; GB200/GB300 use Ulysses2 without offload. B300A default runs contain power/thermal flags and one missing telemetry window after an NVML query timeout; all nine latency samples remain recorded. Raw/resumed telemetry and per-window coverage preserve these limitations. SM100 600 MHz polling gaps stay below 0.66 s; the 750 MHz diagnostic has gaps up to 3.58 s. Telemetry is sampled, not continuous.

Validation: new release/API and exact-block tests on B300A: 40 passed, 2 SM90-only skips. Current upstream full suite on GB300: 53 passed, 19 hardware-dependent skips. Exact Omni head plus new pin on GB300: 629 passed, 3 skipped, including selector/options precedence, Wan no-recompile, H3, real CUDA kernels and Ulysses2; SP4 is untested. Local pre-commit passed. No exact-mask coverage was weakened. Full-forward tensor comparisons are finite and stable; they do not certify decoded-video quality or SSIM/PSNR/LPIPS.

Full report · Reproducible scripts and exact commands · Raw sample CSV · Timing and telemetry · Test logs · Source provenance.

AI assistance: OpenAI Codex assisted with implementation, testing and evidence preparation.

Signed-off-by: lishunyang12 <lishunyang12@163.com>
Signed-off-by: lishunyang12 <lishunyang12@163.com>
Signed-off-by: lishunyang12 <lishunyang12@163.com>
Signed-off-by: lishunyang12 <lishunyang12@163.com>
@lishunyang12

Copy link
Copy Markdown
Author

Completed a same-input follow-up to explain the SM120/SM100 speedup difference.

These are single-attention-operator measurements, separate from the full 50-layer DiT forward results already posted. Real H3 Ref2VA step-0 Q/K/V were captured from a dense reference trajectory on GB200 and replayed unchanged on both GPUs: BF16, [1, 82747, 28, 128], tau=1, identical text/audio protection. Input hashes are checked before every replay. The fixed image and source pins are recorded in the manifest.

Five warmups + 15 measured calls per variant, rotating/reversing order; CUDA-event P50 includes SOL preprocessing. Routing diagnostics and Nsight collection run separately.

GPU / actual clock Layer Exact block fraction Dense ms Native SOL ms Dense / SOL
GB200 / 750 MHz 0 44.58% 129.206 192.752 0.670x
GB200 / 750 MHz 12 37.91% 129.807 169.910 0.764x
GB200 / 750 MHz 25 38.09% 129.775 171.213 0.758x
GB200 / 750 MHz 49 36.50% 129.363 165.403 0.782x
RTX 6000D / 1492 MHz 0 44.58% 705.215 332.897 2.118x
RTX 6000D / 1492 MHz 25 38.09% 705.208 286.846 2.458x

Layer 0 independent-process repeats are 129.217 / 192.653 ms on GB200 and 705.214 / 332.924 ms on RTX 6000D. All measured per-process CVs are below 0.17%. Clock requests were 750 and 1500 MHz; RTX hardware rounded to the observed 1492 MHz. Covered polls have no power-cap or thermal flags. Power limits were unchanged, clocks restored, and allocations released. Per-window sampling coverage is explicit in the clock report.

The experiments narrow the explanation:

  • Routing density is effectively identical across architectures. Layer 0 differs by only 2 of 46,811,772 decisions; layer 25 matches exactly. All required Q rows/KV columns remain protected, and tracing preserves the uninstrumented outputs bitwise.
  • Selecting the SM120 warp-MMA recipe on SM100 regresses further. It compiles for GB200 and passes 12 independent hybrid-oracle checks, but takes 309.341 ms in layer 0 versus native SOL's 192.752 ms, with practically identical routing. The repeat is 309.332 ms. This experimental replacement was rejected; production dispatch is unchanged.
  • SM100 Dense uses its matrix pipeline more effectively. At the same measured 750 MHz, Nsight reports TC-pipeline active cycles of 84.02% Dense vs 51.55% SOL, achieved active-warp occupancy of 24.97% vs 18.28%, and barrier-stall ratios of 0.161 vs 0.995. SOL uses 168 registers/thread and about 86.27 KB dynamic shared memory/CTA, limiting residency to two CTAs. Its HBM bandwidth utilization is only 0.18%. These support improving SM100 scheduling/resource use; they do not yet identify a specific source instruction as the root cause. Dense also has substantial long-scoreboard stalls, so that stall metric alone is not evidence of the cause.
  • Odd-pair padding is small: layer 0's extra masked peer blocks are only 0.448% of its selected-block count.

The next upstream experiment should retain the block64 routing/protection contract while improving the SM100 tcgen05 pipeline or tile/CTA layout. Changing the provider flag or relaxing quality thresholds is not supported by this result. No SM120 hardware-counter profile or decoded-video SSIM/PSNR/LPIPS claim is made in this follow-up.

Full report · Reproducible commands · CSV · Routing comparison · Nsight reports

@lishunyang12

Copy link
Copy Markdown
Author

SM100 follow-up: isolated upstream kernel changes now show a repeatable 8.3–8.9% reduction in SOL latency on frozen H3 Ref2VA attention inputs. SOL still trails cuDNN Dense at this operating point.

GB200, actual 750 MHz matched clock, unchanged 1200 W power limit; BF16 [1, 82747, 28, 128], same input hashes, tau=1, diag threshold and exact-block mask. CUDA-event P50 in ms, 5 warmups + 15 alternating/rotating samples per variant:

Input Dense Upstream SOL Combined candidate SOL latency reduction
Layer 0 129.382 192.892 175.767 8.88%
Layer 25 129.919 171.257 156.981 8.34%
Layer 0, fresh-process repeat 129.365 192.803 175.719 8.86%

The candidate combines three changes: skip identity accumulator rescaling with a warp-uniform ballot; widen rescale transfers from 16 to 32 columns; bypass zero-mask work for fully valid exact tiles. Partial/odd pairs retain masking and all existing synchronization joins remain. Separate ablations are included.

60 correctness checks passed, including 30 independent hybrid-oracle comparisons; all candidate-vs-upstream checks are bitwise equal. Real H3 outputs also match bitwise before and after timing. Maximum timing CV is 0.177%. Covered telemetry polls report 750 MHz with no power-cap/thermal flags; uncovered timing windows are explicitly recorded. Clocks restored; allocation released.

This reproduces removable overhead outside Omni and supports a workload-specific weakness in the upstream SM100 implementation. It does not establish an algorithm-wide flaw or correctness bug. The candidate still trails Dense by 35.85% / 20.83% on these layers. These are single-attention measurements, not full-forward or video-generation results; no new decoded SSIM/PSNR/LPIPS is claimed.

The candidate remains experimental: production kernels, provider dispatch and the Omni dependency pin are unchanged. SM103 and wider-workload validation remain before production enablement.

@lishunyang12

Copy link
Copy Markdown
Author

The native SM100/SM103 provider optimization is now submitted upstream as #594, head 150114b, stacked on Sana#593. The kernel implementation is fe6be47; the follow-up strengthens output checking to compare bytes, including signed zero. Thresholds, exact protection and precision are unchanged.

Captured H3 attention latency is about 8–9.4% lower across the measured SM100/SM103 layers. A targeted SM100 layer-25 repeat measured 177.386 → 162.502 ms, an 8.39% latency reduction, with 750 MHz and no power/thermal flags in overlapping polls.

The actual 50-layer H3 Ref2VA single forward, 2× SM100, TP1/Ulysses2, BF16 eager also improves:

Condition Dense Original SOL Patched SOL SOL latency reduction
450 MHz, stable in every observed measured window 21.799 s 25.115 s 23.894 s 4.86% / 1.051×
750 MHz requested, power-limited 14.727 s 16.503 s 15.764 s 4.48% / 1.047×

The 450 MHz comparison uses 3 full warmup rounds and 9 paired measured rounds; original/patched CVs are 0.0248%/0.0180%, with no observed power-cap or thermal flags. Both completed conditions pass byte comparisons for saved first/last outputs on both ranks. The input hashes also match across clock conditions. SOL still trails Dense; this is an incremental provider improvement.

Validation: 94 tests passed with 19 hardware-specific skips on each GPU; an additional byte-level follow-up passed all 8 synthetic cases and 4 captured H3 layers in eager/graph mode, plus 12 stress cases per GPU. Fresh-compile resource inspection reports 168 registers/thread and zero static local/stack allocation for both variants on both architectures. Nsight runtime-counter collection stalled on the unchanged SM103 baseline; no runtime occupancy/spill claim is made.

Hardware is cluster-labelled B100 / B300A engineering, reported as NVIDIA Graphics Device (CC10.0 / CC10.3), not production GB200/GB300. The incomplete 600 MHz attempt hit the framework request timeout and is excluded; its logs are retained. No decoded-video SSIM/PSNR/LPIPS claim is made from this single-forward test.

Reproducible scripts, full-forward comparison, raw samples, logs and telemetry. The new PR remains draft pending its upstream dependency and maintainer approval of CI.

@lawrence-cj

Copy link
Copy Markdown
Collaborator

Thanks @lishunyang12! cc @hp-l33 for kernel review and GPU validation.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants