An Efficient vLLM-Based Inference Pipeline for Unified Audio Understanding and Generation

Abstract: While Large Multimodal Models excel in comprehension, high-throughput inference engines lack native support for multimodal generation. This is severe in Speech LLMs, where generating multi-layered audio tokens via decoupled AR+NAR or synchronous Multi-Token Prediction (MTP) with delay-pattern interleaving conflicts with standard single-stream loops. We present a vLLM-based inference pipeline for unified speech understanding and generation. We extend autoregressive decoding to natively execute delay-pattern de-interleaving and coordinated multi-stream sampling, integrating an on-GPU acoustic decoder for end-to-end waveform synthesis. Crucially, we overcome the shared intuition that Classifier-Free Guidance (CFG) halves throughput. By co-scheduling paired conditional and unconditional requests within a continuous batch, our CFG implementation sustains 80% of non-CFG throughput, absorbing dual-request and logit merging overheads. We open-source our framework.

• The paper presents an efficient vLLM-based inference pipeline achieving around 100× speedup for unified audio understanding and generation tasks.
• The paper introduces architectural modifications including multi-stream token sampling, on-GPU acoustic decoding, and paired CFG that merges conditional and unconditional passes into a single operation.
• The paper demonstrates maintained output fidelity and numerical integrity with minimal CFG overhead, enabling real-time, multimodal speech applications.

An Efficient vLLM-Based Inference Pipeline for Unified Audio Understanding and Generation

Introduction

This paper presents a high-throughput inference framework for Speech LLMs (SpeechLMs) that unifies both audio understanding and generation tasks within a single, efficient vLLM-based serving architecture (2607.02119). The core problem addressed is the incompatibility of current text-centric serving engines—such as vLLM, originally optimized for unimodal, text-only generation—with the requirements of SpeechLMs. SpeechLMs generate and consume dense, multi-layered discrete acoustic tokens, a process that fundamentally diverges from text-only autoregressive generation, especially in the context of decoder-side tasks such as synchronous multi-token prediction and Classifier-Free Guidance (CFG).

The proposed framework introduces several architectural modifications and algorithmic contributions, including native multi-stream audio token sampling, on-GPU acoustic decoding, and a CFG implementation that maintains high throughput despite its conventionally prohibitive overhead. Detailed evaluations on state-of-the-art SpeechLMs illustrate substantial improvements in both efficiency and maintainance of output fidelity.

Unified Model Architecture for Speech Understanding and Generation

The framework targets SpeechLMs that follow the delay-pattern coding schema, where residual vector quantization (RVQ) compresses each audio frame into $S$ discrete codebook tokens per time step. The pipeline is model-agnostic, validated on architectures such as Bagpiper, OpusLM, and OpusLM-Dialogue.

During inference, an autoregressive transformer processes both textual and acoustic tokens within a unified vocabulary. For audio, the model projects hidden states via multiple stream-specific codebook heads, each corresponding to a different RVQ level, with temporal alignment governed by delay-pattern interleaving. This necessitates de-interleaving and batched multi-stream sampling to recover the generation order for audio reconstruction, followed by on-the-fly waveform synthesis through an integrated, GPU-resident acoustic decoder.

Figure 1: SpeechLM architecture: hidden states are projected into $S$ parallel codebooks for delayed, multi-stream token generation, supporting both audio and text modalities in a unified transformer.

This design enables seamless transitions between text and audio phases within a single autoregressive loop while maintaining high-fidelity, discrete audio representations. The unified vocabulary and de-interleaving logic facilitate straightforward handling of mixed-modality outputs, a requirement for real-time conversational and interactive applications.

High-Throughput Inference Engine Enhancements

Extension of vLLM Serving Engine

The inherent single-stream constraint of vLLM and similar text LLM serving engines—originally predicated on homogeneous, one-token-per-step scheduling—precludes naïve deployment of delay-patterned, multi-token audio generation. To address this, the framework introduces a primary-auxiliary decomposition:

• The engine schedules a main "primary" token stream compatible with the standard one-token-per-step paradigm.
• Auxiliary streams are sampled internally at each step and buffered per request.
• At generation completion, the aggregate codebook matrix is de-interleaved and synthesized to audio by the integrated acoustic decoder.

This approach preserves compatibility with vLLM’s advanced batching and memory management mechanisms (e.g., PagedAttention) without invasive changes to low-level schedulers. Phase management (between text and audio) is implemented via dynamic vocabulary masking and run-time inference from model outputs, ensuring robust operation under continuous dynamic batching.

Integrated on-GPU Acoustic Decoding

The framework directly incorporates the neural vocoder (acoustic decoder) into the serving pipeline, performing all waveform synthesis on the GPU. This eliminates dependencies on separate external decode services, reduces inter-process latency, and allows end-to-end, single-process audio delivery.

Figure 2: Unified inference framework: continuous batching and primary-auxiliary token management with on-GPU decoding for mixed-modality streaming; CFG implemented via paired co-scheduling and logit merging.

Classifier-Free Guidance via Paired Co-Scheduling

A core technical advancement is the efficient implementation of Classifier-Free Guidance (CFG). Conventional CFG requires two forward passes (conditional and unconditional) per token generation step, which would naïvely halve throughput due to doubled compute and cache pressure. The framework instead co-schedules these as paired, atomic requests sharing the same batch, executing a single joint forward pass and merging logits before sampling.

This enables:

• Full hardware utilization through maximal cache reuse and shared batch compute
• Strict synchronization of the autoregressive history between conditional and unconditional branches
• Selective activation of CFG only during the audio phase, amortizing overhead

Empirically, this implementation yields up to 80% of the throughput of non-CFG inference, directly challenging prior assumptions about the prohibitive cost of CFG for speech generation.

Experimental Results

Throughput and Hardware Utilization

The vLLM-enhanced pipeline achieves a $\sim 100\times$ speedup in token generation throughput compared to a sequential PyTorch baseline across all evaluated SpeechLMs (Bagpiper, OpusLM, OpusLM-Dialogue). Model FLOPs Utilization (MFU) increases from <0.3% in the baseline to up to 9.95% (Bagpiper/OpusLM) given efficient GPU saturation.

Notably, CFG overhead is tightly controlled: with CFG, Bagpiper maintains over 80% throughput and even higher MFU (12.8%)—an artifact of saturating otherwise-underused compute resources.

Numerical Correctness and Output Fidelity

Under float32 precision, the new pipeline yields identical output tokens and minimal logit RMSE compared to the sequential baseline, confirming numerical integrity even under continuous batching and multi-stream synchronization. With bf16+FlashAttention-3, expected divergence arises from floating-point non-associativity, but this is shown to have no practically adverse effect on end-to-end ASR or TTS performance: understanding and generation metrics are statistically equivalent across all configurations.

Formal Evaluation

• Bagpiper (vLLM) achieves 5694 tok/s decode speed and a 9.95% MFU.
• On TTS/ASR quality, MMAU-mini accuracy, LibriSpeech WER, and UTMOS scores remain effectively unchanged.
• CFG-enabled throughput is 3960 tok/s, with only modest latency increments relative to non-CFG serving.

Practical and Theoretical Implications

The described architecture provides a modular, high-performance foundation for deploying unified SpeechLMs capable of simultaneous audio and text interaction, with immediate applicability to real-time dialogue systems, ASR/TTS services, and complex multimodal reasoning agents.

The primary theoretical implication is the demonstrated viability of natively supporting delay-pattern, multi-stream sequence models—and high-overhead guidance techniques like CFG—inside text-optimized, dynamic batching engines with minimal architectural perturbation. This challenges a widely held assumption within the community about throughput and efficiency trade-offs for advanced multimodal generation.

Looking forward, these architectural abstractions may be extended and generalized to more complex multimodal settings, further integrating image/video token generation or supporting compositional, non-autoregressive modalities with similarly optimized high-throughput schedulers.

Conclusion

This work delivers a high-throughput, numerically robust inference framework for SpeechLMs, marking a decisive step toward harmonizing the deployment of large-scale, unified audio-understanding and generation models within mainstream LLM serving pipelines. By resolving multi-codebook generation and CFG scheduling bottlenecks within the vLLM ecosystem, the system enables practical, real-time, multimodal speech agents without compromising performance or output fidelity. As multimodal LMs continue to expand in both scale and scope, these engineering advancements provide a critical base for next-generation, low-latency, interactive voice-based applications.