Avoidance Decoding Strategies

• Avoidance decoding is a strategy that adjusts decoding probabilities in real time to penalize undesirable continuations and enhance output diversity.
• It finds applications in multi-branch story generation, code completion, error correction, and quantum decoding to boost performance and reliability.
• Techniques include similarity penalties, pivot classification, and constrained token suppression, each designed to target specific failure modes.

Avoidance decoding denotes a family of decoding-time strategies in which inference is steered away from undesirable continuations rather than merely sampled from an unconstrained model or solved as a purely unconstrained search problem. In current usage, the term appears explicitly as a decoding strategy for diverse multi-branch story generation that modifies token logits by penalizing similarity to previously generated outputs, and it appears in closely related forms that avoid only utility-critical draft tokens, deny the generation of vulnerability labels, avoid errors confined to stopping sets, or reduce dependency-induced waiting in windowed decoding (Park et al., 2 Sep 2025, Ziashahabi et al., 1 Nov 2025, Storhaug et al., 2023) 0702050.

1. Scope and recurring objective

Across the cited literatures, what is being avoided differs by application, but the operational pattern is stable: a decoder is given an explicit notion of an undesirable continuation, and decoding is modified so that those continuations are penalized, rejected, blocked, or bypassed. In story generation, the negatives are previously generated branches; in Pivot-Aware Speculative Decoding, the negatives are pivot tokens; in smart-contract completion, the negatives are vulnerability labels; in binary-erasure-channel decoding, the target is errors confined to stopping sets; and in speculative window decoding for fault-tolerant quantum programs, the objective is dependency-delay avoidance or blocking-avoidance via speculation (Park et al., 2 Sep 2025, Ziashahabi et al., 1 Nov 2025, Storhaug et al., 2023) 0702050.

Setting	What is avoided	Mechanism
Multi-branch story generation	Similarity to previously generated outputs	Token-logit penalty
Speculative LLM decoding	Pivot tokens	Selective rejection override
Smart-contract code completion	Vulnerability labels	Logit set to $-\infty$
Linear block codes on the BEC	Errors confined to stopping sets	Iterative message passing plus permutation decoding
Fault-tolerant quantum decoding	Dependency-induced delay	Speculative prediction of boundary dependency bits

A plausible implication is that “avoidance decoding” is not a single algorithmic family in the narrow sense. Rather, it is a recurrent design principle: define a harmful subset of continuations and alter decoding so that the harmful subset is traversed less often, later, or not at all.

2. Similarity penalties and multi-branch story generation

The paper “Avoidance Decoding for Diverse Multi-Branch Story Generation” introduces Avoidance Decoding as a decoding-time method that modifies token logits by penalizing similarity to previously generated stories, which are treated as negative samples (Park et al., 2 Sep 2025). The method is motivated by the observation that standard decoding methods mostly work at the token level and often fail to produce meaningfully different stories at the level of concept, narrative trajectory, and overall plot structure.

The probabilistic motivation begins by defining the probability of avoiding all negatives $N=\{n_1,\dots,n_N\}$:

$$P_s(-N \mid x_{1:t}) = \prod_{i=1}^N \left(1 - P_s(n_i \mid x_{1:t}, -n_{1:i-1})\right).$$

Using the approximation

$$P_s(n_i \mid x_{1:t}, -n_{1:i-1}) \approx P_s(n_i \mid x_{1:t}),$$

the paper writes

$$P(x_t \mid x_{1:t-1}, -N) = P_\theta(x_t \mid x_{1:t-1}) \prod_{i=1}^N \left(1 - P_s(n_i \mid x_{1:t})\right).$$

After taking logs and using $\log(1-p)\approx -\alpha_i p$, the modified token score becomes

$$\tilde{l}_t = l_t - \sum_{i=1}^N \alpha_i P_s(n_i \mid x_{1:t}),$$

and the paper then replaces the sum with a max:

$$\tilde{l}_t = l_t - \max_i \alpha_i P_s(n_i \mid x_{1:t}).$$

The final token is chosen greedily:

$x_t^\* = \arg\max_v \tilde{l}_t(v).$

The penalty itself is hybrid. The Concept-level Similarity Penalty measures similarity between the last hidden state of each candidate next token and the hidden states of all tokens in the negative samples, and it is prioritized in early stages to diversify initial story concepts. The Narrative-level Similarity Penalty compares the current generated sentence with each negative sample using Sentence-BERT embeddings, and it is increasingly emphasized later to ensure natural yet diverse plot development. The balance is adaptive:

$$\gamma = \epsilon + (1-\epsilon)\cdot \mathrm{sigmoid}(t - T_0).$$

When $N=\{n_1,\dots,n_N\}$0 is small, $N=\{n_1,\dots,n_N\}$1, so CSP dominates; as $N=\{n_1,\dots,n_N\}$2 increases, NSP gets more weight.

The paper evaluates the method by generating 15 branches of about 200 tokens from a single prompt on ReedsyPrompts and WritingPrompts, with Mistral 7B, Llama 3B, Llama 8B, and Qwen 7B. It reports up to 2.6 times higher output diversity and an average reduction in repetition of 30% compared to strong baselines, while effectively mitigating text degeneration. It also reports that dormant neuron ratio decreases under Avoidance Decoding, whereas most baselines show increasing dormant neuron ratios. The authors interpret this as evidence that the method activates a broader range of neurons and leverages the model’s intrinsic creativity.

3. Utility-preserving avoidance in speculative decoding

Pivot-Aware Speculative Decoding reframes speculative decoding as an avoidance decoding problem: instead of trying to preserve the exact sampling distribution of the target model, it tries to avoid only those draft tokens that would reduce task utility (Ziashahabi et al., 1 Nov 2025). Standard speculative decoding is distribution-faithful, whereas PAD changes the optimization target from distribution matching to expected utility.

The paper defines utility as a binary task-performance function and defines expected utility as $N=\{n_1,\dots,n_N\}$3. A candidate token $N=\{n_1,\dots,n_N\}$4 is pivot if its inclusion would lower the expected utility of the final completed output if the rest of the sequence were finished by the target model. PAD’s avoidance principle is therefore specific: do not reject all draft tokens that differ from the target distribution—reject only those predicted to be utility-critical.

Direct evaluation of the pivot condition is intractable, so the paper uses an offline Monte Carlo labeling pipeline. Only tokens that standard speculative decoding would reject are considered as labeling candidates. For each candidate, the method estimates the utility of the base prefix and the utility of the candidate-extended prefix using target-model rollouts, then applies an LLM-as-judge sanity check that can only convert non-pivot to pivot. The classifier uses the target-model hidden state at layer $N=\{n_1,\dots,n_N\}$5, the target-model probability of the candidate token, and the entropy of the target distribution. It is trained as a small MLP with weighted cross-entropy, an 80/20 train-test split, and checkpoint selection by lowest validation loss.

At inference, PAD sits on top of standard speculative decoding. If SD would accept a token, PAD accepts it. If SD would reject a token, PAD queries the pivot classifier; if the classifier predicts non-pivot with score below threshold $N=\{n_1,\dots,n_N\}$6, PAD overrides SD and accepts the token anyway. The paper adds a guardrail that rejects any token whose target-model probability is below $N=\{n_1,\dots,n_N\}$7, regardless of classifier score.

The empirical trade-off is explicit. The paper reports ROC AUC $N=\{n_1,\dots,n_N\}$8 for the pivot classifier, and it reports up to $N=\{n_1,\dots,n_N\}$9 speedup with comparable utility. For example, at $$P_s(-N \mid x_{1:t}) = \prod_{i=1}^N \left(1 - P_s(n_i \mid x_{1:t}, -n_{1:i-1})\right).$$0, GSM8K reaches 93 $$P_s(-N \mid x_{1:t}) = \prod_{i=1}^N \left(1 - P_s(n_i \mid x_{1:t}, -n_{1:i-1})\right).$$1 1.2 accuracy, $$P_s(-N \mid x_{1:t}) = \prod_{i=1}^N \left(1 - P_s(n_i \mid x_{1:t}, -n_{1:i-1})\right).$$2, and speedup $$P_s(-N \mid x_{1:t}) = \prod_{i=1}^N \left(1 - P_s(n_i \mid x_{1:t}, -n_{1:i-1})\right).$$3; AIME24 reaches 57 $$P_s(-N \mid x_{1:t}) = \prod_{i=1}^N \left(1 - P_s(n_i \mid x_{1:t}, -n_{1:i-1})\right).$$4 7.8 accuracy, $$P_s(-N \mid x_{1:t}) = \prod_{i=1}^N \left(1 - P_s(n_i \mid x_{1:t}, -n_{1:i-1})\right).$$5, and speedup $$P_s(-N \mid x_{1:t}) = \prod_{i=1}^N \left(1 - P_s(n_i \mid x_{1:t}, -n_{1:i-1})\right).$$6; MBPP reaches 64.7 $$P_s(-N \mid x_{1:t}) = \prod_{i=1}^N \left(1 - P_s(n_i \mid x_{1:t}, -n_{1:i-1})\right).$$7 1.7 accuracy, $$P_s(-N \mid x_{1:t}) = \prod_{i=1}^N \left(1 - P_s(n_i \mid x_{1:t}, -n_{1:i-1})\right).$$8, and speedup $$P_s(-N \mid x_{1:t}) = \prod_{i=1}^N \left(1 - P_s(n_i \mid x_{1:t}, -n_{1:i-1})\right).$$9. The method therefore rejects only tokens believed to be utility-critical, rather than insisting on exact probabilistic fidelity.

4. Constraint-based avoidance in code generation

In smart-contract code completion, avoidance is implemented through vulnerability-constrained decoding (Storhaug et al., 2023). The method uses a small dataset of labeled vulnerable lines of code to fine-tune a LLM so that vulnerability labels such as <IOU> or <TD> are inserted after indentation and before the vulnerable statement. The model is therefore trained to surface vulnerability information as part of generation.

The decoding rule is simple. Given a set of vulnerability labels $$P_s(n_i \mid x_{1:t}, -n_{1:i-1}) \approx P_s(n_i \mid x_{1:t}),$$0, the decoder manipulates the probability so that

$$P_s(n_i \mid x_{1:t}, -n_{1:i-1}) \approx P_s(n_i \mid x_{1:t}),$$1

In practice, the logits for all vulnerability-label tokens are set to $$P_s(n_i \mid x_{1:t}, -n_{1:i-1}) \approx P_s(n_i \mid x_{1:t}),$$2 before the softmax, so the decoder never emits a vulnerability label token. Because the model was trained to place a vulnerability label at the start of a vulnerable line, blocking those labels changes the generation path: if the highest-probability continuation would lead through a vulnerable pattern, the label token that would annotate that pattern is suppressed, and the decoder chooses the next-best continuation instead.

The experimental pipeline uses GPT-J-6B. The paper first fine-tunes the model on 186,397 unique Ethereum smart contracts after deduplication from 2,217,692 verified contracts and reports validation accuracy 0.917, perplexity 1.510, and average BLEU 0.557. It then evaluates vulnerability generation by taking 176 vulnerable smart contracts, splitting each contract above the vulnerable line, and asking the model to complete it. With the initial fine-tuned model, 123 of 173 evaluated completions were vulnerable, or about 71%. After an additional vulnerability-tuning stage on 941 vulnerable smart contracts, the model correctly labeled 58 vulnerable cases, and constrained decoding made 39 of those 58 secure, giving an avoidance rate of about 67% on correctly labeled vulnerable cases. The post-vulnerability-tuning BLEU is 0.544, a decrease of about 2.33%.

The method does not directly detect arbitrary vulnerabilities at inference time. Instead, it avoids generating code paths that the model itself has learned to mark as vulnerable using the special label tokens.

5. Coding-theoretic antecedents: stopping sets, permutation decoding, and automorphisms

A much earlier coding-theoretic use of the same underlying idea appears in “Permutation Decoding and the Stopping Redundancy Hierarchy of Linear Block Codes” [0702050]. The paper investigates the stopping redundancy hierarchy of linear block codes and its connection to permutation decoding techniques. An element in the ordered list of stopping redundancy values represents the smallest number of possibly linearly dependent rows in any parity-check matrix of a code that avoids stopping sets of a given size.

The key connection is that redundant parity-check equations can be shown to have a similar effect on decoding performance as permuting the coordinates of the received codeword according to a selected set of automorphisms of the code. Based on this finding, the paper develops new decoding strategies for data transmission over the binary erasure channel that combine iterative message passing and permutation decoding in order to avoid errors confined to stopping sets.

The paper also introduces the notion of $$P_s(n_i \mid x_{1:t}, -n_{1:i-1}) \approx P_s(n_i \mid x_{1:t}),$$3-SAD sets, containing the smallest number of automorphisms of a code with the property that they move any set of not more than $$P_s(n_i \mid x_{1:t}, -n_{1:i-1}) \approx P_s(n_i \mid x_{1:t}),$$4 erasures into positions that do not correspond to stopping sets within a judiciously chosen parity-check matrix. This is not “avoidance decoding” in the contemporary LLM sense, but it is a direct instance of decoding organized around the avoidance of harmful structural configurations.

6. Dependency-delay avoidance in fault-tolerant quantum decoding

In fault-tolerant quantum programs, the relevant avoidance target is not a token or a stopping set but dependency-induced delay (Viszlai et al., 2024). The paper “Predictive Window Decoding for Fault-Tolerant Quantum Programs” is closely related in spirit to “avoidance decoding” only in the broad sense that it tries to avoid waiting or blocking in the decoding pipeline. It introduces SWIPER, a speculative window decoding scheme inspired by branch prediction in classical processors.

The core distinction is between decoder latency and decoder throughput or reaction time. Prior parallel window decoding methods solve the throughput or backlog problem, but they do not eliminate waiting between dependent windows. If the time to decode one window is $$P_s(n_i \mid x_{1:t}, -n_{1:i-1}) \approx P_s(n_i \mid x_{1:t}),$$5, then the reaction time for a dependent window becomes at least $$P_s(n_i \mid x_{1:t}, -n_{1:i-1}) \approx P_s(n_i \mid x_{1:t}),$$6 under prior parallel-window methods. SWIPER inserts a lightweight predictor that tries to infer the boundary dependency bits between adjacent windows early, so a downstream window can begin decoding before the upstream full decode is complete.

The predictor is explicitly built in three steps. The 1-step predictor checks only single edges crossing the boundary and examines $$P_s(n_i \mid x_{1:t}, -n_{1:i-1}) \approx P_s(n_i \mid x_{1:t}),$$7 candidate boundary edges. The 2-step predictor extends to edges up to distance 2 from the boundary and resolves candidates in a way analogous to peeling. The 3-step predictor adds a precomputed table of weight-2 boundary-crossing error-chain pairs. The paper claims the final 3-step predictor runs in $$P_s(n_i \mid x_{1:t}, -n_{1:i-1}) \approx P_s(n_i \mid x_{1:t}),$$8 with respect to code distance, and it reports a constant 60 ns runtime for $$P_s(n_i \mid x_{1:t}, -n_{1:i-1}) \approx P_s(n_i \mid x_{1:t}),$$9 through $$P(x_t \mid x_{1:t-1}, -N) = P_\theta(x_t \mid x_{1:t-1}) \prod_{i=1}^N \left(1 - P_s(n_i \mid x_{1:t})\right).$$0.

SWIPER’s speculative result is temporary and gets checked by the full decoder. If the speculation is wrong, the affected window is restarted. The paper reports predictor accuracy above 70% for the 1-step predictor and above 90% for the 3-step predictor across simulated code distances, up to about 50% reaction-time reduction compared to prior parallel window decoding, and roughly 40% average reduction in fault-tolerant program runtime relative to prior parallel window decoders. The paper does not define or use the phrase “avoidance decoding” as a formal term; the relevant sense is dependency-delay avoidance or blocking-avoidance via speculation.

7. Conceptual unification, limits, and common misconceptions

The cited works do not use the phrase uniformly. Avoidance Decoding in multi-branch story generation is a named decoding strategy; PAD is described as an avoidance decoding problem because it avoids only pivot tokens; vulnerability-constrained decoding blocks learned vulnerability labels; the coding-theoretic work focuses on avoiding stopping sets; and SWIPER is explicitly said not to be a separate avoidance-decoding framework (Park et al., 2 Sep 2025, Ziashahabi et al., 1 Nov 2025, Storhaug et al., 2023) 0702050.

Several misconceptions are therefore worth excluding. First, avoidance decoding is not necessarily distribution-preserving: PAD explicitly argues that exact distribution matching is too strict, and it preserves expected utility rather than the target model’s sampling distribution. Second, avoidance decoding is not necessarily a new model or training regime: the smart-contract method is implemented by blocking logits at inference time, and the story-generation method is a decoding-time method with no extra training. Third, avoidance decoding does not necessarily eliminate the underlying dependency or failure mode: SWIPER does not eliminate dependencies altogether, and the vulnerability-constrained method does not perform semantic security analysis at inference time. Fourth, aggressive avoidance often introduces a trade-off. The PAD paper states that higher $$P(x_t \mid x_{1:t-1}, -N) = P_\theta(x_t \mid x_{1:t-1}) \prod_{i=1}^N \left(1 - P_s(n_i \mid x_{1:t})\right).$$1 increases acceptance and speed but can reduce accuracy; the story-generation paper reports that larger $$P(x_t \mid x_{1:t-1}, -N) = P_\theta(x_t \mid x_{1:t-1}) \prod_{i=1}^N \left(1 - P_s(n_i \mid x_{1:t})\right).$$2 increases diversity but also degeneration; and SWIPER increases the number of concurrent classical decoders by about 31% relative to prior parallel window decoding.

This suggests that the most stable encyclopedia-level definition is functional rather than domain-specific: avoidance decoding is a decoding strategy in which the decoder is modified so that a specified harmful subset of continuations is less likely to be chosen, whether the harmful subset is defined by similarity, utility loss, vulnerability annotations, stopping sets, or dependency stalls.