PathRelax: Parallel-Path Relaxed Speculative Jacobi Decoding for Accelerating Auto-Regressive Text-to-Image Generation

Abstract: The growing need for high-resolution image generation in autoregressive text-to-image models has resulted in extended token sequences, significantly increasing computational costs and inference times. However, existing state-of-the-art methods for accelerating autoregressive text-to-image models rely on chain-structured draft token sequences, leading to inefficient draft token search and limited acceptance lengths. To address this, we propose parallel-path cross-relaxed speculative Jacobi decoding (\textbf{PathSpec}), a novel framework that enhances efficiency through a multi-sequence draft tree structure. Our parallel-path speculative Jacobi decoding (\textbf{PathExplore}) expands the token search space, achieving a higher speedup ratio without sacrificing image quality. Additionally, we introduce cross-path relaxed verification (\textbf{PathRelax}) that exploits semantic similarities across sequences to further boost token acceptance rates. Evaluated on the Parti-Prompts, MSCOCO2017, and T2ICompBench datasets, our method achieves a speedup ratio of 4.14 $\times$, 3.95$\times$, and 4.18$\times$, respectively. Remarkably, PathExplore, without any relaxed sampling, outperforms relaxed sampling methods in the speedup ratio, such as GSD and LANTERN. Moreover, PathRelax's relaxation mechanism can be seamlessly integrated with other relaxation techniques, enabling further acceleration and providing an efficient solution for real-time text-to-image generation. Our code is available at https://github.com/Haodong-Lei-Ray/PathSpec.

• The paper introduces PathExplore and PathRelax methods that enable parallel tree-structured speculative decoding, achieving nearly 4× speedup in auto-regressive text-to-image generation.
• It leverages semantic similarities among image tokens via cross-path relaxed verification, aggregating probabilities to boost token acceptance without additional model overhead.
• The approach scales efficiently, maintaining key perceptual metrics such as CLIP-score and FID, which supports its practical use in real-time, high-resolution image synthesis.

PathRelax: Parallel-Path Relaxed Speculative Jacobi Decoding for Accelerating Auto-Regressive Text-to-Image Generation

Introduction

Autoregressive (AR) text-to-image models have become pivotal for synthesizing high-resolution, photorealistic images from textual prompts. Despite high fidelity, these models frequently suffer from excessive inference latency due to the token-by-token generation paradigm required for large-scale image sequences: thousands of image tokens per generation, a significant bottleneck for real-time applications. Prior speculative decoding frameworks, such as SJD and GSD, aim to decrease wall-clock latency by proposing and subsequently verifying multiple draft tokens per forward pass. However, those methods are fundamentally constrained by the chain-structured draft sequence design—rejecting an early token invalidates all subsequent candidates, limiting the effective token acceptance length and acceleration.

This paper introduces PathRelax (“Parallel-Path Cross-Relaxed Speculative Jacobi Decoding”), which extends speculative decoding to tree-structured multi-path drafting, and further introduces a relaxation mechanism, PathRelax, that leverages semantic similarities between candidate sequences for higher token acceptance rates. The two chief contributions—PathExplore and PathRelax—not only generalize speculative Jacobi decoding to a lossless multi-branch regime, but also propose cross-path relaxation techniques to exploit ambiguity in image token distributions, supporting joint probability aggregation and acceptance sharing among sibling branches. Collectively, these innovations yield state-of-the-art acceleration for AR text-to-image generation, with minimal impact on perceptual quality.

Parallel-Path Speculative Jacobi Decoding: PathExplore

The core innovation of PathExplore is to depart from the strictly chained proposal scheme, instead constructing a candidate draft tree in which multiple speculative paths are expanded in parallel. At each Jacobi decoding iteration, candidate nodes are sampled conditionally on prefix sequences. The tree is populated by random initializations or autoregressive sampling. All candidate paths are jointly verified via parallel forward passes, with accepted nodes from multiple branches being eligible for selection. Crucially, rejecting one path does not lead to cascading invalidation, in contrast to chain-structured speculative decoding.

PathExplore enables token acceptance at a given position if any candidate in the corresponding sibling cluster is accepted, formalized by an acceptance probability $\alpha_i = 1 - \prod_{k=1}^K (1 - \alpha_{i,k})$, which directly upper bounds the acceptance length achievable by chain-based methods. The method is architected for compatibility with the underlying Transformer’s KV cache to minimize additional computational or memory cost; empirical results indicate that draft tree depth and breadth can be substantially increased without significant overhead when appropriately tuned.

Figure 2: Memory usage and one-forward time variations for PathExplore on Lumina-GPT-7B, as a function of width and tree size. Overhead remains modest with practical settings.

Cross-Path Relaxed Verification: PathRelax

While PathExplore already boosts acceptance rates via parallel-path drafting, the authors recognize further opportunity in the probabilistic ambiguity of image token distributions. Distinct from text, where next-token prediction is sharply peaked, image token prediction exhibits considerable uncertainty; many candidates may have similar probabilities (see Figure 3).

Figure 4: Probability distribution for text token vs. image token generation. Image token probability is notably ambiguous, supporting the motivation for cross-path relaxation.

To leverage this, PathRelax introduces cross-path probability aggregation: for each sibling group (child nodes sharing the same parent prefix), the target probability for verification is computed as an affine combination of self-probability and the mean probability among siblings (modulated by a relaxation hyperparameter $\lambda$). This not only encourages cooperative acceptance among semantically similar drafts, but also dense exploration of the local image manifold—a significant improvement over uncoordinated, independent path verification. Acceptance proceeds by a relaxed rejection sampling rule using the aggregated probability.

The capacity of PathRelax to combine semantic proximity among candidates allows for a significant boost in acceptance ratio without auxiliary draft models, codebook heuristics, or additional training overhead.

Empirical Results

The PathSpec framework, incorporating both PathExplore and PathRelax, achieves strong empirical results on benchmark datasets (MSCOCO2017, T2ICompBench, Parti-Prompts) and models (Lumina-GPT-7B, Emu3). The following speedup ratios (SR) and average acceptance length ($\tau$) are reported:

• MSCOCO2017: 3.95× SR (τ = 5.55, Lumina-GPT-7B)
• T2ICompBench: 4.18× SR (τ = 5.62)
• Parti-Prompts: 4.14× SR (τ = 5.64)

PathSpec clearly surpasses previous state-of-the-art (e.g., SJD, GSD, LANTERN++) both in effective speedup and token acceptance length. Even without relaxation ($\lambda = 0$), PathExplore outperforms all chain-structured baselines in throughput, supporting its theoretical motivation.

Importantly, these speedups are realized without significant compromise in generation quality; metrics such as CLIP-score, FID, and IS remain comparable to non-accelerated baselines, with perceptual demerits appearing only at aggressive relaxation settings (large $\lambda$). In ablation studies, PathRelax is complementary to prior relaxation methods (see Figure 5).

Figure 1: Acceleration versus tree depth, width and relaxation. Speedup ratio (SR) increases with tree scale and relaxation, saturating as verification overheads emerge.

Figure 6: Example generations (“an eagle”) for different relaxation settings. Visual quality is preserved up to moderate relaxation strength ($\lambda=0.01$).

Theoretical and Practical Implications

At a theoretical level, PathExplore is proven to be a lossless acceleration method when relaxation is disabled (i.e., $\lambda = 0$), meaning samples are unbiased from the target distribution. The cross-path relaxation operates as a controlled bias-variance tradeoff mechanism; empirical findings indicate that modest values of $\lambda$ yield acceleration benefits with minimal perceptual degradation—a valuable practical knob for system designers.

As autoregressive models grow in scale and target higher resolutions (increasing the length of image token sequences), inference bottlenecks threaten applicability to real-time or interactive scenarios. PathSpec demonstrates that tree-structured, parallel speculative decoding is a critical innovation to keep AR models viable in future domains, particularly in multimodal and compositional generation tasks.

Furthermore, PathRelax’s reliance on semantic token similarity, rather than codebook-specific heuristics or auxiliary models, posits an extensible strategy for other multimodal AR applications, such as video, 3D, or speech token generation.

Future Directions

Key areas for further investigation include fine-tuning the relaxation schema for task-adaptive tradeoffs, integrating PathRelax into diffusion-AR hybrid models, and exploring dynamic path pruning and adaptive hyperparameter schedules. The theoretical guarantees for lossless decoding in PathExplore suggest potential for generalization to broader classes of generative models.

Conclusion

PathRelax systematically advances speculative decoding for autoregressive text-to-image generation by introducing parallel path drafting and cross-relaxed verification, yielding significant latency reductions (≈4× speedup) at minimal quality cost. It combines the best attributes of self-speculative and relaxed sampling strategies, demonstrating robust acceleration without compromising distributional fidelity. The methods are readily adaptable to existing AR architectures and represent important progress toward scalable, real-time, high-resolution multimodal synthesis.

---

Reference:

"PathRelax: Parallel-Path Relaxed Speculative Jacobi Decoding for Accelerating Auto-Regressive Text-to-Image Generation" (2606.10492)