Skip to content
Swapnil-GandhiPublic

About

Pacing Branch Parallelism in LLM Serving, NeurIPS'26

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

3 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Pacing Branch Parallelism in LLM Serving

PACE is a per-step branch-admission controller for intra-request (branch) parallelism in LLM serving. Built on the Multiverse Engine (an SGLang fork that provides the Map → Process → Reduce branch substrate), it decides at every decode step which forked branches advance — admitting an extra branch only when its predicted latency cost (its branch externality) fits the batch's current slack budget. The payoff: higher goodput than eager admission, while protecting the latency SLO of every co-batched request. For more information, please refer to our NeurIPS 2026 paper: [📄 Pacing Branch Parallelism in LLM Serving ].

📰 News

  • September 2026: 🎉 PACE has been accepted to NeurIPS 2026!

🧠 How it works

Each decode step, PACE evaluates a calibrated step-latency predictor T(S) = a + b·n_tokens + c·L_context, then greedily admits the highest utility-per-cost branch while predicted latency stays within the slack budget T_0 + ρ·B_t. The rest are deferred and re-admitted on a later step.

  • Branch externality — extra branches inflate the shared decode step that every co-batched request pays; PACE admits one only when it fits the slack budget.
  • The safe width is dynamic — it shifts with batch composition, context lengths, and accumulated slack, so PACE re-decides per step instead of using a fixed cap.
  • Regulation is cheap — branches share the request's prefix KV, so changing width needs no memory reclamation; deferral is pure batch-membership control (ScheduleBatch.pace_split).
  • Policies are pluggable — subclass AdmissionPolicy, register it, and select with --pace-mode <name> (built-ins: off / cap / eager / pace).

🚀 Quick start

conda create -n pace python=3.11 && conda activate pace
git clone https://github.com/Multiverse4FM/Multiverse-Engine && cd Multiverse-Engine
bash install.sh
# torch is pinned to 2.6.0 (cu124); if install.sh pulls newer transitive deps, pin these:
pip install "transformers==4.51.1" "torchao==0.9.0" "compressed-tensors==0.9.4"

PACE is opt-in and a strict no-op unless enabled (--pace-enable false reproduces the stock engine):

python -m sglang.launch_server \
  --model-path Multiverse4FM/Multiverse-32B --tp 8 --dtype bfloat16 \
  --disable-overlap-schedule \
  --pace-enable --pace-mode pace --pace-tpot-slo-ms 25 \
  --pace-coeffs-path pace_coeffs.json
Flag Default Meaning
--pace-enable false master switch
--pace-mode pace off / cap / eager / pace
--pace-rho 0.8 fraction of slack budget to spend
--pace-tpot-slo-ms 50 TPOT SLO target (drives deadlines)
--pace-coeffs-path null offline-calibrated (a, b, c) json

Calibrate the predictor on your hardware first — scripts/pace_calibrate.py fits (a, b, c) from a sglang.bench_one_batch sweep — and pass the result via --pace-coeffs-path. On 8×H200 (TP=8, bf16) this fits T(S) to ~3.5% MAPE, with < 1 ms/step planner overhead.

🧪 Testing

python -m unittest discover -s test/srt/pace -p "test_*.py"

GPU-free unit tests cover the predictor, greedy planner, slack budget, branch grouping, and deadline model; heavy GPU correctness gates live in test_pace_e2e.py (run with PACE_RUN_E2E=1).

📚 Citation

@inproceedings{gandhi2026,
      title={Pacing Branch Parallelism in {LLM} Serving},
      author={Gandhi, Swapnil and Hari, Siva Kumar Sastry and Dally, William J. and Kozyrakis, Christos},
      booktitle={The Fortieth Annual Conference on Neural Information Processing Systems},
      year={2026},
      url={https://arxiv.org/abs/2605.06914},
}

🙏 Acknowledgments

PACE is built on the Multiverse Engine, which provides the Map → Process → Reduce branch execution substrate, and on SGLang. We thank their authors.

About

Pacing Branch Parallelism in LLM Serving, NeurIPS'26

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages