Soft Parallel Decoding (SPD)

• Soft Parallel Decoding (SPD) is a family of decoding strategies that uses approximate linearity and automorphism groups to generate multiple candidate outputs concurrently.
• It reuses computations to boost throughput, enabling the simultaneous generation of outputs in autoregressive LMs, diffusion models, and error-correction codes.
• Empirical benchmarks show SPD achieves significant speedups (e.g., >2.44×) with minimal loss in quality, making it valuable for real-time applications.

Soft Parallel Decoding (SPD) refers to a family of decoding strategies across disparate computational paradigms—autoregressive LLMs, diffusion LLMs, and algebraic error-correction codes—that enable the simultaneous or near-simultaneous production of multiple candidate outputs during generation or inference. SPD fundamentally exploits structural properties (such as the approximate linearity of token embeddings in deep nets or the automorphism groups in code constructions) to maximize throughput and minimize redundancy, while preserving or improving output quality and accuracy. Distinct instantiations of SPD include Superposed Decoding in LLMs (Shen et al., 2024), hybrid embedding decoding in diffusion LLMs (Chen et al., 9 Apr 2026), and Polar Orbit Decoding (POD) in block code decoders (Li et al., 16 Jan 2026).

1. Conceptual Foundations and Principles

Soft Parallel Decoding is characterized by its ability to produce $k$ or more candidate outputs at the cost of a single or modestly more expensive inference/evaluation pass, rather than the conventional approach of running the model or decoder $k$ times. This is achieved by:

• Exploiting approximate linearity or superposability within embedding or codeword space.
• Sharing computation across parallel candidate hypotheses while avoiding irrevocable “hard” decisions at intermediate steps.
• Incorporating mechanisms for score reconciliation, uncertainty propagation, or decoding diversity, often through hybrid distributions, interpolation, or group-theoretic symmetries.

The term "soft" refers to the preservation of uncertainty or the avoidance of immediate hard commitments to a single hypothesis at each incremental step, thus allowing later correction, resampling, or refinement.

2. SPD in Autoregressive LLMs: Superposed Decoding

Superposed Decoding (Shen et al., 2024) is a concrete realization of SPD for autoregressive transformers, enabling the generation of $k$ distinct drafts in a single inference pass. The workflow is:

• At each time-step, form a superposed (probability-weighted) embedding $$\tilde x_{t-1} = \sum_{i=1}^k \gamma_i z(x_{t-1}^i)$$, with $\gamma_i$ proportional to the current likelihood of the $i$-th partial draft.
• Perform a single model forward call to produce a shared token distribution $p_\theta(\cdot|\tilde x_{1:t-1})$.
• Expand each of the $k$ current drafts by combining them with the top-$k$ predicted tokens, resulting in $k^2$ new candidates.
• Score candidates by a geometric interpolation of the model’s distribution and a cached $k$0-gram model proposal.
• Select and renormalize the top $k$1 survivors for the next step.

This approach reuses key/value caches and largely avoids overhead due to multiple forward evaluations, yielding a theoretical and measured speedup of at least $k$2 for $k$3.

Implementation Pseudocode (abridged)

$\gamma_i$8

Notably, this method is a wrapper around standard decoding loops and is compatible with pre-trained transformer decoders without retraining.

3. SPD in Diffusion LLMs: Hybrid Soft-State Decoding

In the context of masked diffusion LLMs, SPD (Chen et al., 9 Apr 2026) is designed to counteract error accumulation due to aggressive “hard” mask-to-token transitions. Instead, at each iteration:

• Each decoding position maintains a hybrid embedding representing a probability-weighted interpolation between the [MASK] embedding and the predicted token embedding.
• For a position $k$4, after a model step with prediction $k$5 and confidence $k$6, the hybrid embedding is:

$k$7

with subsequent norm renormalization to stabilize the scale.

• Model uncertainty propagates through this soft state, enabling revision and error correction in future denoising steps.
• Promotion from masked to soft (hybrid) token state occurs per position based on adjustable thresholds.

Integration with On-Policy Uniform Training (OPUT)—where models learn to recover from both masked and self-predicted-noisy inputs—is essential for SPD to function reliably, as it exposes models during training to their own errors and soft states.

Empirically, SPD nearly triples throughput (TPF) with negligible accuracy degradation, as demonstrated on GSM8K and MBPP tasks.

4. SPD in Coding Theory: Polar Orbit Decoding

Polar Orbit Decoding (POD) (Li et al., 16 Jan 2026) applies SPD to binary linear block codes under polar transformations. Here, SPD leverages automorphism groups of codes to produce $k$8 decoding candidates (branches):

• For each automorphism $k$9 in the code’s group $k$0, generate a permuted LLR input and decode using the same dynamic-frozen constraints.
• Each branch thus traverses a different permutation (orbit) of the code’s bit channels, delivering diversity and mitigating the effect of early errors.
• Outputs from all $k$1 branches are combined via metric-based or parity-check-based selection.
• Using a Base and Strong Generating Set (BSGS) representation via the Schreier-Sims algorithm, automorphism orbits can be enumerated systematically and efficiently.

POD yields a continuum of speed–performance trade-offs: for instance, $k$2 parallel SCL($k$3) branches can reach the effective list size $k$4 at the latency of SCL($k$5), rather than SCL($k$6).

5. Complexity, Hardware, and Empirical Characterization

A cross-domain summary of computational and empirical characteristics:

Domain	# Outputs/Pass	Theoretic Speedup	Quality
Autoregressive LM	$k$7	$k$8 at $k$9	PPL $$\tilde x_{t-1} = \sum_{i=1}^k \gamma_i z(x_{t-1}^i)$$0, P@3 $$\tilde x_{t-1} = \sum_{i=1}^k \gamma_i z(x_{t-1}^i)$$1
Diffusion LM	$$\tilde x_{t-1} = \sum_{i=1}^k \gamma_i z(x_{t-1}^i)$$2 block	$$\tilde x_{t-1} = \sum_{i=1}^k \gamma_i z(x_{t-1}^i)$$3–$$\tilde x_{t-1} = \sum_{i=1}^k \gamma_i z(x_{t-1}^i)$$4 TPF	$$\tilde x_{t-1} = \sum_{i=1}^k \gamma_i z(x_{t-1}^i)$$5100% of base acc
BLBC (POD)	$$\tilde x_{t-1} = \sum_{i=1}^k \gamma_i z(x_{t-1}^i)$$6	$$\tilde x_{t-1} = \sum_{i=1}^k \gamma_i z(x_{t-1}^i)$$7 over SCL	Near-ML at $$\tilde x_{t-1} = \sum_{i=1}^k \gamma_i z(x_{t-1}^i)$$8 lower latency

In each case, SPD reduces wall-clock time and memory overhead (by avoiding duplicative runs or storing enlarged KV caches). For SPD/POD, the hardware area/latency trade-off is controlled via the choice of $$\tilde x_{t-1} = \sum_{i=1}^k \gamma_i z(x_{t-1}^i)$$9 (number of orbits) and $\gamma_i$0 (list size).

Empirical benchmarks:

• Superposed Decoding achieves best-of-3 perplexity improvements of 5% (Llama-2-7B) and is preferred by human evaluators in $\gamma_i$1 of trials (Shen et al., 2024).
• DMax (OPUT+SPD) raises TPF to 5.48 with $\gamma_i$2 accuracy penalty on GSM8K (Chen et al., 9 Apr 2026).
• POD matches SCL$\gamma_i$3 performance at $\gamma_i$4 lower latency on eBCH(64,16) (Li et al., 16 Jan 2026).

6. Limitations and Extensions

SPD methods are subject to several limitations intrinsic to their specific instantiations:

• Linearity Approximation: In embedding-based SPD, true linear superposition of semantics is only approximate. Quality may degrade for longer generations (Shen et al., 2024).
• External Resource Overhead: N-gram filtering in language applications requires precomputed $\gamma_i$5-gram models with nontrivial memory footprints.
• Semantic Diversity: Single shared distributions at each step curtail diversity among outputs.
• Saturation: Hardware or algorithmic benefit saturates as $\gamma_i$6 or $\gamma_i$7 increases, especially if resources are not truly parallel.

Proposed extensions include superposed-decoding resets, orthogonal projections to enhance per-draft signal, hybridization with speculative or multi-token prediction (Medusa, ProphetNet), and dynamic tuning of interpolation parameters (Shen et al., 2024).

7. Applications and Broader Impact

SPD has demonstrable impact in:

• Efficient Generation: Real-time applications demanding multiple suggestions—autocomplete, dialog systems, code and text completion—benefit from SPD’s multiplicity at reduced latency (Shen et al., 2024).
• Large-Scale Diffusion Generation: SPD enables aggressive block-wise promotion and uncertainty-aware revision in diffusion LMs, alleviating error cascades and enabling high-throughput decoding (Chen et al., 9 Apr 2026).
• Low-Latency Decoding in Communications: SPD via POD enables hardware designers to approach ML decoding performance for complex block codes at practical latency and cost (Li et al., 16 Jan 2026).

Across these domains, SPD unifies parallel generation strategies founded on “soft” hypothesis management, marking an overview of probabilistic, algebraic, and neural techniques.