Repository navigation
Support arbitrary exact Q/KV blocks in fused Sol-Attn - #593
lishunyang12 wants to merge 4 commits into
Conversation
Signed-off-by: lishunyang12 <lishunyang12@163.com>
Signed-off-by: lishunyang12 <lishunyang12@163.com>
Signed-off-by: lishunyang12 <lishunyang12@163.com>
Signed-off-by: lishunyang12 <lishunyang12@163.com>
|
Completed a same-input follow-up to explain the SM120/SM100 speedup difference. These are single-attention-operator measurements, separate from the full 50-layer DiT forward results already posted. Real H3 Ref2VA step-0 Q/K/V were captured from a dense reference trajectory on GB200 and replayed unchanged on both GPUs: BF16, Five warmups + 15 measured calls per variant, rotating/reversing order; CUDA-event P50 includes SOL preprocessing. Routing diagnostics and Nsight collection run separately.
Layer 0 independent-process repeats are 129.217 / 192.653 ms on GB200 and 705.214 / 332.924 ms on RTX 6000D. All measured per-process CVs are below 0.17%. Clock requests were 750 and 1500 MHz; RTX hardware rounded to the observed 1492 MHz. Covered polls have no power-cap or thermal flags. Power limits were unchanged, clocks restored, and allocations released. Per-window sampling coverage is explicit in the clock report. The experiments narrow the explanation:
The next upstream experiment should retain the block64 routing/protection contract while improving the SM100 Full report · Reproducible commands · CSV · Routing comparison · Nsight reports |
|
SM100 follow-up: isolated upstream kernel changes now show a repeatable 8.3–8.9% reduction in SOL latency on frozen H3 Ref2VA attention inputs. SOL still trails cuDNN Dense at this operating point. GB200, actual 750 MHz matched clock, unchanged 1200 W power limit; BF16
The candidate combines three changes: skip identity accumulator rescaling with a warp-uniform ballot; widen rescale transfers from 16 to 32 columns; bypass zero-mask work for fully valid exact tiles. Partial/odd pairs retain masking and all existing synchronization joins remain. Separate ablations are included. 60 correctness checks passed, including 30 independent hybrid-oracle comparisons; all candidate-vs-upstream checks are bitwise equal. Real H3 outputs also match bitwise before and after timing. Maximum timing CV is 0.177%. Covered telemetry polls report 750 MHz with no power-cap/thermal flags; uncovered timing windows are explicitly recorded. Clocks restored; allocation released. This reproduces removable overhead outside Omni and supports a workload-specific weakness in the upstream SM100 implementation. It does not establish an algorithm-wide flaw or correctness bug. The candidate still trails Dense by 35.85% / 20.83% on these layers. These are single-attention measurements, not full-forward or video-generation results; no new decoded SSIM/PSNR/LPIPS is claimed. The candidate remains experimental: production kernels, provider dispatch and the Omni dependency pin are unchanged. SM103 and wider-workload validation remain before production enablement.
|
|
The native SM100/SM103 provider optimization is now submitted upstream as #594, head Captured H3 attention latency is about 8–9.4% lower across the measured SM100/SM103 layers. A targeted SM100 layer-25 repeat measured 177.386 → 162.502 ms, an 8.39% latency reduction, with 750 MHz and no power/thermal flags in overlapping polls. The actual 50-layer H3 Ref2VA single forward, 2× SM100, TP1/Ulysses2, BF16 eager also improves:
The 450 MHz comparison uses 3 full warmup rounds and 9 paired measured rounds; original/patched CVs are 0.0248%/0.0180%, with no observed power-cap or thermal flags. Both completed conditions pass byte comparisons for saved first/last outputs on both ranks. The input hashes also match across clock conditions. SOL still trails Dense; this is an incremental provider improvement. Validation: 94 tests passed with 19 hardware-specific skips on each GPU; an additional byte-level follow-up passed all 8 synthetic cases and 4 captured H3 layers in eager/graph mode, plus 12 stress cases per GPU. Fresh-compile resource inspection reports 168 registers/thread and zero static local/stack allocation for both variants on both architectures. Nsight runtime-counter collection stalled on the unchanged SM103 baseline; no runtime occupancy/spill claim is made. Hardware is cluster-labelled B100 / B300A engineering, reported as NVIDIA Graphics Device (CC10.0 / CC10.3), not production GB200/GB300. The incomplete 600 MHz attempt hit the framework request timeout and is excluded; its logs are retained. No decoded-video SSIM/PSNR/LPIPS claim is made from this single-forward test. Reproducible scripts, full-forward comparison, raw samples, logs and telemetry. The new PR remains draft pending its upstream dependency and maintainer approval of CI. |
|
Thanks @lishunyang12! cc @hp-l33 for kernel review and GPU validation. |
MiniMax-H3 Ref2VA can interleave text/audio segments with reference and target video. The existing contiguous KV sink cannot protect those disjoint segments on both attention axes without a separate integration-side dense/residual path.
Add an optional
exact_block_maskBoolean vector (one entry per 64-token block, shared across batch/heads). Marked Q rows and KV columns use the exact route inside the existing fused routing loop. Omitted masks preserve the current API; the legacy KV sink can still be combined with the mask. Omni can forward its model-owned metadata directly through its custom op.The argument is threaded through CuTe SM89, SM90 (including split KV), SM100/SM103, SM120, and both Triton implementations. CuTe caches distinguish mask presence while mask values remain runtime inputs. Input shape/device/dtype/contiguity validation and independent token-level hybrid oracle tests are included.
Earlier validation at PR head
1c522ed3e1e5d43f24258f6cc27bb7b7e189d508, after merging currentsol-engineand resolving the dispatch conflict:cute_sm120and imports this exact PR source. SM120 skips are 11 SM80-only cases, 6 MPS-only cases and 2 SM90-only KV-split cases.Nonemasks explicitly, with a hardware-independent regression test.088897e0c6a9cb970f00b43ab2177c3be7336b36onf4035e0aseparately passes 35 exact-block cases on GH200 and 33 cases on each of GB200 / RTX 6000D (2 SM90-only cases skipped). The original SM90/SM100/SM120 forward measurements use this pin; the SM103 follow-up below uses d9ef632.Commands from a checkout with
techniques/sparse_backendsinstalled:Environment:
vllm/vllm-openai:cu134-nightly-1648fefab91204f3b15bf8062217316f07f2c4bb, Python 3.12.3, PyTorch 2.16.0.dev20261009+cu134, Triton 3.8.0+git3f6e4113, CuTe DSL 4.8.0; pytest 9.2.0. Related integration: vllm-project/vllm-omni#10513.Redacted logs, source manifest and reproducible commands are published alongside the related Omni benchmark evidence.
The downstream H3 first-forward comparison now covers all three architectures; see the full report. SM100 results include an explicit power/clock limitation. The measured dependency and current upstream PR provenance remain separate.
SM100 power controls and SM103 follow-up
The SM103 dispatch fix is pushed: upstream PR head
63b4e65e9db1dfea45cc7d394902145c2f520821, release-pinned backportd9ef6321230cff13398cfc86aa107f97f093e925, Omni headca5cc9fdeddf654d2ebaf08d116d46c15327a27e. SM103 was absent from both the CuTe architecture map and SM100 launch branch. It now automatically uses the existing upstream Blackwell kernel family; no provider/native flag or approximation setting is added.Same complete first-step H3 forward, BF16/eager/TP1, official request/checkpoint, 3 warmup pairs + 9 alternating measured pairs per process. Speedup is Dense P50 / SOL P50, so values below 1 mean SOL is slower.
SM100: explicit site-delegated
nvidia-smiclock management worked despite Slurm's disabled frequency control. Only allocated GPUs were locked and all clock locks were restored. At 600/750 MHz, measured telemetry polls report the requested clocks with no software power-cap or thermal-throttle flags. The 1200 W default limit was already the firmware maximum and was not raised. Matched-clock SOL is about 22% slower, reproduced across two 600 MHz processes and a separate 750 MHz process. Default-node gains are power/clock dependent, not a clock-normalized kernel speedup. Underclocking is a diagnostic control, not a production speed optimization. Inputs and each implementation's outputs are byte-identical across all four processes.At 750 MHz, a separate post-measurement profile shows the 50 Dense attention kernels total 6.493 / 6.494 s on ranks 0/1 versus SOL 8.372 / 8.343 s; major GEMM totals are nearly unchanged. The gap is in upstream attention execution. These kernel activity sums are diagnostic, not wall-latency samples.
SM103: automatic CuTe dispatch reduces SOL latency from 20.681 → 17.477 s on B300A (15.5%) and 5.052 → 4.321 s on GB300 (14.5%) compared with the old Triton route. It still loses to Dense. The two GB300 default CuTe processes have about 0.23% CV and byte-identical per-implementation outputs. GB300 default Dense also hits its 1400 W limit (sampled median ~1.54 GHz versus SOL 2.07 GHz); the 1200 MHz row is a separate controlled comparison, not pooled with default-clock results. At 1200 MHz, Dense is 4.949 s versus SOL 6.723 s (35.8% slower), with only 1200 MHz in measured polls and no power/thermal flags. Its outputs are byte-identical to the default-clock runs and its clock locks were restored.
B300A is the cluster's SM103/GB112 engineering device: driver name
NVIDIA Graphics Device, ~139 GiB VRAM and a 600 W maximum cap. It is not a full production B300 result. It uses one GPU and equal CPU offload for both paths; GB200/GB300 use Ulysses2 without offload. B300A default runs contain power/thermal flags and one missing telemetry window after an NVML query timeout; all nine latency samples remain recorded. Raw/resumed telemetry and per-window coverage preserve these limitations. SM100 600 MHz polling gaps stay below 0.66 s; the 750 MHz diagnostic has gaps up to 3.58 s. Telemetry is sampled, not continuous.Validation: new release/API and exact-block tests on B300A: 40 passed, 2 SM90-only skips. Current upstream full suite on GB300: 53 passed, 19 hardware-dependent skips. Exact Omni head plus new pin on GB300: 629 passed, 3 skipped, including selector/options precedence, Wan no-recompile, H3, real CUDA kernels and Ulysses2; SP4 is untested. Local pre-commit passed. No exact-mask coverage was weakened. Full-forward tensor comparisons are finite and stable; they do not certify decoded-video quality or SSIM/PSNR/LPIPS.
Full report · Reproducible scripts and exact commands · Raw sample CSV · Timing and telemetry · Test logs · Source provenance.
AI assistance: OpenAI Codex assisted with implementation, testing and evidence preparation.