- The paper introduces Tri-serve, a software-based DVFS controller that reduces energy-per-token by 22.1% in multimodal inference serving for Qwen2.5-Omni-7B, by optimizing clock frequencies for stalled, compute-bound, and memory-bound phases.
- Tri-serve utilizes a stall-aware mode that adjustments for idle states and optimized the frequency scaling based on arithmetic intensity (A.I.), these changes improved energy efficiency by up to 20% and low throughput degrading.
- The full Tri-serve system greatly reduces the effective temperature increase which leads securing more stable computing conditions.
Motivation and problem statement
Multimodal LLM (MLLM) serving pipelines, exemplified by vLLM-Omni disaggregating Qwen2.5-Omni-7B into Thinker, Talker, and Code2Wav/Vocoder stages across GPUs, introduce power-management pathologies that unimodal LLM DVFS controllers do not address. The paper's central claim is that the GPU's on-board power management unit (PMU), which autonomously governs auto-boost frequency decisions, is structurally mismatched to multimodal inference workloads. The authors support this with three characterization studies on NVIDIA RTX A6000 Ada GPUs profiling vLLM-Omni via Nsight Systems.
Three characterized inefficiencies
Inter-stage dependency stalls waste idle power. Because each stage holds its own weights and KV cache (exceeding a single 48 GB device), stages map to separate GPUs connected by Python producer/consumer queues. Downstream stages block on sem_wait() while upstream stages compute. In concurrent online serving, stalls account for roughly 16% of total GPU time across stages and 34% for the Vocoder; stall durations are typically at least 20 ms for Thinker/Talker and above 10 seconds for Vocoder. Critically, during these stalls the SM clock remains pinned near 2500 MHz and package power stays above 60 W—far above the P8 idle state of 210 MHz—so active-idle power exceeds static power.
Auto-boost frequency is anti-correlated with arithmetic intensity (A.I.). Compute-bound prefill runs at ~1000–1200 MHz while memory-bound decode reaches the auto-boost ceiling—the opposite of an energy-optimal allocation. To root-cause this, the authors introduce a frequency-locked roofline microbenchmark sweeping A.I. from 1 to 200 FLOPs/B against locked SM clocks of 450–3000 MHz. Two findings emerge: the achievable frequency ceiling under auto-boost varies with A.I. because the PMU reallocates power between SMs and memory (frequency dips near the roofline ridge where both domains are stressed); and at low A.I., identical throughput is achievable at the lowest clock, meaning auto-boost wastes energy during decode.
Thermal throttling degrades compute-heavy phases. During Thinker/Talker prefill, frequency decays monotonically from ~1300 MHz to ~1000 MHz, a ~20% performance loss over the phase. A 240-second thermal soak benchmark across the A.I. grid shows that memory-bound kernels lose almost no frequency at steady-state temperature, whereas compute-bound kernels lose ~400–800 MHz. Higher A.I. implies higher dissipated power, higher steady-state temperature, less thermal headroom, and larger PMU-imposed clock cuts—a chain that compounds the A.I.-anti-correlation effect. The authors state this phenomenon is universal across multiple GPU models, though they demonstrate it primarily on one platform.
Tri-serve design
Tri-serve is a software-based DVFS controller with three components sharing a single NVML actuation primitive (nvmlDeviceSetGpuLockedClocks):
- Stall-aware idle scaling:
sem_wait() calls trigger locking SM/memory clocks to 210/810 MHz (P8 levels), restoring auto-boost on exit.
- A.I.-aware scaling: Phase boundaries are detected via the vLLM v1 EngineCore scheduler (
num_scheduled_tokens plans) anchored by torch.cuda.Event markers. Per-kernel A.I. values are obtained with NCU and aggregated into a duration-weighted phase A.I. The controller solves a constrained optimization selecting the lowest-power frequency whose modeled throughput Θ(A.I.,f) stays within a tolerance ϵ of maximum throughput. Throughput follows a modified roofline form with an A.I.-saturation term converging to the true roofline asymptote; power combines frequency-dependent idle and dynamic terms capped by TDP, with a utilization factor capturing the memory/compute balance peak. Because frequency levels are discrete and monotonic in power and performance, binary search replaces sequential quadratic programming for low-overhead solving.
- Thermal-aware pace-and-race scaling: For high-A.I. prefill, the controller paces at ~1800 MHz (the highest sustainable frequency across all A.I.) for 10% of prefill duration, conserving thermal headroom, then races under auto-boost for the remaining 90%.
A limitation worth noting: the pace/race split (10%/90%) and the pace frequency are empirically selected rather than derived from a thermal model, so their transferability across GPU SKUs and ambient conditions is not established.
Evaluation results
The ablation on Qwen2.5-Omni-7B with a 3× RTX 6000 Ada pipeline shows each component contributing:
| Configuration |
Energy/tok (J) |
Δ vs. baseline |
Throughput Δ |
Peak temp |
| Auto-boost baseline |
4.12 |
— |
— |
74 °C |
| Stall-aware only |
3.58 |
−13.1% |
−0.1% |
72 °C |
| A.I.-aware only |
3.85 |
−6.5% |
−1.9% |
68 °C |
| Thermal-aware only |
4.05 |
−1.7% |
+1.4% |
65 °C |
| Full Tri-serve |
3.21 |
−22.1% |
+0.8% |
64 °C |
Stall-aware scaling is the dominant single contributor, which follows directly from the coarse-grained, long-duration nature of dependency stalls. Against baselines—including fixed medium frequency (1500 MHz) and throttLL'eM adapted to Thinker/Talker phases—Tri-serve achieves 20.8–23.7% energy-per-token reduction across offline and online (λ = 0.5, 1.0 RPS) scenarios and both 2-GPU and 3-GPU configurations, while nearly matching auto-boost TTFT and TPOT tails. Fixed-frequency capping yields 5.3–16.6% savings but degrades TTFT substantially (e.g., 220 ms vs. 185 ms mean TTFT in the 3-GPU offline case); throttLL'eM, designed for unimodal LLMs, achieves only 4.4–16.0%. An implication of these comparisons is that phase-awareness—not merely lower average frequency—is what preserves latency SLOs while reclaiming energy.
The evaluation is confined to one GPU model, one MLLM family, and moderate arrival rates; energy savings under saturation, where inter-stage stalls compress, are not characterized in the final results, though the mechanism suggests Part 1's contribution would shrink.
Prior energy-aware LLM serving work (POLCA, throttLL'eM, μ-Serve, PowerInfer) targets unimodal workloads as black boxes or optimizes single phases reactively. Tri-serve's distinction is joint control over three workload-specific signals—queue-blocking events, per-phase arithmetic intensity, and thermal headroom—specialized to disaggregated multimodal pipelines. The roofline-based modeling follows Williams et al., and the pace-then-race pattern adapts prior pacing controllers to multimodal prefill.
Conclusion
Tri-serve demonstrates that hardware PMU-managed auto-boosting leaves roughly 20% of multimodal inference energy unreclaimed, attributable to identifiable and software-correctable causes: dependency-stall idle power, A.I.-misaligned frequency selection, and reactive thermal throttling. The controller delivers 20.8–23.7% energy-per-token savings with negligible latency or throughput cost on Qwen2.5-Omni-7B. Open questions include generalization of the empirically tuned thermal policy across GPU architectures, behavior under high-load saturation where stalls vanish, and whether driver-level rather than NVML-level actuation could reduce switching overheads for finer-grained control.