Asymmetric Unlikelihood Optimization

• Asymmetric unlikelihood optimization is a training paradigm that integrates likelihood with task-specific unlikelihood losses to suppress undesired outputs.
• Methodologies include weighting penalties by readability scores, using discriminator guidance, and applying popularity-aware penalties for tailored generation.
• Empirical results show improved readability, attribute control, and tail item exposure, though often balanced against trade-offs in fluency or overall utility.

Asymmetric unlikelihood optimization denotes a family of training strategies in which standard likelihood-based sequence learning is augmented with an unlikelihood term that suppresses particular undesired tokens or sequences more strongly than others. Across recent work, the asymmetry is realized in different ways: by weighting penalties with word-level readability scores and masking hallucinated entities in medical text simplification (Flores et al., 2023), by using a discriminator to define desired and undesired next-token candidates in controllable generation (Zhang et al., 2022), and by penalizing popular head-item tokens only when training on long-tail items in generative recommendation (Yin et al., 16 May 2026). The common pattern is directional optimization: some outputs are explicitly pushed down, while others are either left unaffected or reinforced by a separate likelihood term.

1. Conceptual scope

The basic unlikelihood form is the familiar $-\log(1-p)$ penalty on negative targets, but the cited systems depart from symmetric treatment of errors. In the medical simplification setting, the total loss is

$$\mathcal{L} = \mathcal{L}_{\text{NLL}} + \lambda_R \,\mathit{UL}_R + \lambda_C \,\mathit{UL}_C,$$

where $\mathit{UL}_R$ is readability-oriented and $\mathit{UL}_C$ is factual-consistency-oriented, with fixed hyperparameters $\lambda_R = 7.5 \times 10^{-4}$ and $\lambda_C = 2.5 \times 10^{-4}$ (Flores et al., 2023). In DisCup, the objective adds a likelihood term over desirable top-$k$ candidates and an unlikelihood term over undesirable top-$k$ candidates, with no separate coefficient between them; only the prompt parameters are updated, while the CLM and discriminator remain frozen (Zhang et al., 2022). In Ghost, the recommendation loss is

$$\mathcal{L}_{\text{All}} = \mathcal{L}_{\text{NLL}} + \alpha \cdot \mathcal{L}_{\text{AUO}},$$

where $\mathcal{L}_{\text{AUO}}$ is applied only to tail-item training examples and uses head-item SIDs as negatives (Yin et al., 16 May 2026).

Taken together, these formulations suggest that “asymmetric unlikelihood optimization” is not a single canonical algorithm. Rather, it is a design pattern in which the negative target set, the penalty strength, and the population to which the penalty is applied are all task-specific and intentionally non-uniform.

2. Objective design and mathematical structure

In medical text simplification, the readability unlikelihood term penalizes only the most probable token at each generation step, and scales the penalty by a precomputed Flesch-Kincaid score: $$\mathcal{L} = \mathcal{L}_{\text{NLL}} + \lambda_R \,\mathit{UL}_R + \lambda_C \,\mathit{UL}_C,$$0 Here $$\mathcal{L} = \mathcal{L}_{\text{NLL}} + \lambda_R \,\mathit{UL}_R + \lambda_C \,\mathit{UL}_C,$$1 only for the argmax token at step $$\mathcal{L} = \mathcal{L}_{\text{NLL}} + \lambda_R \,\mathit{UL}_R + \lambda_C \,\mathit{UL}_C,$$2, so complex words with high $$\mathcal{L} = \mathcal{L}_{\text{NLL}} + \lambda_R \,\mathit{UL}_R + \lambda_C \,\mathit{UL}_C,$$3 incur larger penalties, while simpler words receive weaker penalties (Flores et al., 2023). The factual-consistency term has the analogous form

$$\mathcal{L} = \mathcal{L}_{\text{NLL}} + \lambda_R \,\mathit{UL}_R + \lambda_C \,\mathit{UL}_C,$$4

where $$\mathcal{L} = \mathcal{L}_{\text{NLL}} + \lambda_R \,\mathit{UL}_R + \lambda_C \,\mathit{UL}_C,$$5 marks tokens in a hallucinated-entity set $$\mathcal{L} = \mathcal{L}_{\text{NLL}} + \lambda_R \,\mathit{UL}_R + \lambda_C \,\mathit{UL}_C,$$6 extracted from greedy model outputs and filtered by spaCy en_core_web_lg NER (Flores et al., 2023).

In DisCup, asymmetry enters through a discriminator-guided partition of the frozen CLM’s top-$$\mathcal{L} = \mathcal{L}_{\text{NLL}} + \lambda_R \,\mathit{UL}_R + \lambda_C \,\mathit{UL}_C,$$7 candidate set $$\mathcal{L} = \mathcal{L}_{\text{NLL}} + \lambda_R \,\mathit{UL}_R + \lambda_C \,\mathit{UL}_C,$$8. For a target attribute $$\mathcal{L} = \mathcal{L}_{\text{NLL}} + \lambda_R \,\mathit{UL}_R + \lambda_C \,\mathit{UL}_C,$$9, desirable candidates are weighted by

$\mathit{UL}_R$0

and undesirable candidates by

$\mathit{UL}_R$1

where $\mathit{UL}_R$2 (Zhang et al., 2022). The two loss components are

$\mathit{UL}_R$3

$\mathit{UL}_R$4

Desired tokens are pulled up by likelihood; undesired tokens are pushed down by unlikelihood (Zhang et al., 2022).

In Ghost, the negative set is not token-local but item-structured. For a tail item, the AUO term is

$\mathit{UL}_R$5

where $\mathit{UL}_R$6 is a collection of SIDs from undesired head items associated with that tail item (Yin et al., 16 May 2026). The construction is explicitly popularity-aware: the negatives come from the head set, not the tail.

3. Forms of asymmetry

Three distinct forms recur in the literature.

Token-wise asymmetry: In medical simplification, the penalty weight is proportional to $\mathit{UL}_R$7, so “complex top-1 words” are discouraged more than simple ones. The same system also exhibits “top-1 asymmetry,” because only the single highest-probability token at each step contributes to the unlikelihood loss (Flores et al., 2023).

Content-wise asymmetry: The factual-consistency term in medical simplification applies only to tokens in the hallucinated entity set $\mathit{UL}_R$8; supported or neutral entities are unaffected by this term (Flores et al., 2023). A closely related pattern appears in Ghost, where head-item SIDs are treated as undesired only in the context of tail-item supervision (Yin et al., 16 May 2026).

Different treatment of positive vs negative attribution: DisCup explicitly separates desirable and undesirable candidates. Desired tokens receive a likelihood objective, undesirable ones receive an unlikelihood objective, and the roles of tokens depend on the chosen target attribute. For positive sentiment control, low $\mathit{UL}_R$9 candidates are undesirable; for negative sentiment control, the target attribute changes and so does the undesirable set (Zhang et al., 2022).

Population asymmetry: Ghost applies AUO only for tail-item training examples. Head items are trained only with likelihood, whereas tail items are trained with likelihood plus an unlikelihood term that suppresses competing head items. The paper characterizes this as “asymmetric token-level unlikelihood between the tail items and their head counterparts” (Yin et al., 16 May 2026).

These designs also differ in how they define negative targets. Medical simplification extracts them dynamically from model-preferred sequences and then filters them by NER (Flores et al., 2023). DisCup derives them from discriminator judgments over top-$\mathit{UL}_C$0 CLM candidates (Zhang et al., 2022). Ghost constructs them by a two-stage selection over semantically similar head items and SID prefix structure (Yin et al., 16 May 2026).

4. Representative implementations

The major instantiations occupy different application regimes, but all preserve the core directional logic.

Setting	Negative target construction	Additional mechanism
Medical text simplification	Complex top-1 words; hallucinated entities	Reranked beam search
Controllable text generation	Discriminator-defined undesirable top-$\mathit{UL}_C$1 candidates	Prompt tuning with frozen GPT-2 large
Generative recommendation	Popular head-item SIDs for tail examples	Skeleton-Founded Tokenization

In medical text simplification, a BART-XSum encoder-decoder is fine-tuned on Cochrane, MedEasi, and Radiology, with unlikelihood active throughout training and implemented in HuggingFace Transformers. The decoding stage introduces another asymmetric component: every $\mathit{UL}_C$2 steps, beams are reranked by a composite readability-plus-consistency score, and beams containing entities absent from the source are assigned score $\mathit{UL}_C$3 and eliminated (Flores et al., 2023). The reranking score uses normalized Flesch-Kincaid Grade Level $\mathit{UL}_C$4, normalized BERTScore $\mathit{UL}_C$5, and the squared harmonic mean

$\mathit{UL}_C$6

In DisCup, the base generator is GPT-2 large with frozen parameters, and the control mechanism is a small number of continuous prompts, typically around 10 virtual tokens. A GPT-2 small discriminator is trained first on labeled attribute data and then frozen. At each training position, the CLM supplies top-$\mathit{UL}_C$7 candidates, the discriminator scores each candidate, and the prompt parameters are updated so that high-scoring candidates are more likely and low-scoring candidates less likely (Zhang et al., 2022). The method therefore moves discriminator guidance from decoding time into training time.

In Ghost, asymmetry is coupled to a structural change in item tokenization. Skeleton-Founded Tokenization uses head-item SIDs as a skeleton and lets each tail item inherit the first $\mathit{UL}_C$8 tokens from its closest head item before appending $\mathit{UL}_C$9 additional tokens of its own (Yin et al., 16 May 2026). The stated purpose is to collapse many unpredictable branching points into a single branching point at step $\lambda_R = 7.5 \times 10^{-4}$0, making AUO’s gradient corrections more effective.

5. Empirical behavior and trade-offs

The empirical record supports the view that asymmetric unlikelihood terms can improve the target property, but typically under a measurable trade-off.

In medical simplification, the Cochrane ablation isolates the roles of the two unlikelihood components. Full UL achieves FK $\lambda_R = 7.5 \times 10^{-4}$1, BERTScore $\lambda_R = 7.5 \times 10^{-4}$2, GPT-Eval hallucination count $\lambda_R = 7.5 \times 10^{-4}$3, and SARI $\lambda_R = 7.5 \times 10^{-4}$4. Using $\lambda_R = 7.5 \times 10^{-4}$5 alone yields FK $\lambda_R = 7.5 \times 10^{-4}$6, GPT-Eval $\lambda_R = 7.5 \times 10^{-4}$7 hallucinations, and SARI $\lambda_R = 7.5 \times 10^{-4}$8. Using $\lambda_R = 7.5 \times 10^{-4}$9 alone yields FK $\lambda_C = 2.5 \times 10^{-4}$0, GPT-Eval $\lambda_C = 2.5 \times 10^{-4}$1 hallucinations, and SARI $\lambda_C = 2.5 \times 10^{-4}$2 (Flores et al., 2023). The paper also reports that UL+Decoder improves FK on Cochrane by up to $\lambda_C = 2.5 \times 10^{-4}$3 points over the next-best model, and that on 30 Cochrane examples human judges preferred UL outputs over a BART-XSum baseline in 43% of cases, versus 3% for prior SOTA NAPSS (Flores et al., 2023).

In DisCup, explicit comparison with and without unlikelihood shows that the unlikelihood term contributes materially to control performance. For positive target steering from neutral prompts, correctness rises from $\lambda_C = 2.5 \times 10^{-4}$4 without unlikelihood to 94.98\% with unlikelihood, while PPL changes from $\lambda_C = 2.5 \times 10^{-4}$5 to $\lambda_C = 2.5 \times 10^{-4}$6 and coverage from $\lambda_C = 2.5 \times 10^{-4}$7 to $\lambda_C = 2.5 \times 10^{-4}$8. For negative target steering from neutral prompts, correctness rises from $\lambda_C = 2.5 \times 10^{-4}$9 to 68.76\%, with PPL changing from $k$0 to $k$1. In toxicity avoidance, average toxicity probability changes from $k$2 to 0.064 (Zhang et al., 2022). The paper further reports inference times to generate 20 tokens of $k$3 s for GPT-2 large, $k$4 s for DisCup, $k$5 s for DExperts, and $k$6 s for PPLM (Zhang et al., 2022).

In Ghost, AUO is evaluated jointly with SKT against standard generative recommenders and debiasing baselines. Across three datasets, the reported average gains versus standard GRs are +$k$7 Tail HR and +$k$8 Tail NDCG. MGU is reduced by about $k$9 on average, while overall HR and NDCG drop by about $k$0 and $k$1 relative to LC-Rec (Yin et al., 16 May 2026). The paper also reports that recommendation lists from standard GRs are more than $k$2 head items, whereas Ghost changes exposure from approximately $k$3k head versus about $k$4 tail items across users to about $k$5k head versus about $k$6k tail items (Yin et al., 16 May 2026).

A common empirical theme is that the asymmetric term improves the targeted property—readability, attribute control, or tail exposure—but must be balanced against fluency, factuality, or overall utility.

6. Theoretical interpretation, limitations, and extensions

The most explicit theoretical diagnosis appears in Ghost. Under token-level MLE, the paper derives “gradient starvation”: for a tail-specific token that almost never appears as ground truth, the expected update is approximately non-positive, so the token embedding is pushed away from the user preference direction (Yin et al., 16 May 2026). It then derives “head token dominance at branching point” and “bias amplification via undifferentiated tokenization,” arguing that standard SID tokenization turns local head–tail preference gaps into geometric suppression over multiple branching points. AUO is presented as adding a “Rescue Force” for tail tokens by pushing down undesired head tokens (Yin et al., 16 May 2026).

The medical simplification paper provides a different theoretical caution: readability is not optimized directly at the sentence level. Instead, the system uses word-level FK scores as proxies, and “there’s no direct gradient from a document-level FK metric” (Flores et al., 2023). This is a useful corrective to the misconception that asymmetric unlikelihood necessarily optimizes the final evaluation metric itself. In that system, the method optimizes an indirect proxy during training and a reranking criterion during decoding.

DisCup addresses another misconception: the technique is not restricted to full-model fine-tuning. Its asymmetric unlikelihood objective updates only prompt parameters, leaving both the generator and discriminator frozen (Zhang et al., 2022). This suggests that asymmetric unlikelihood can be coupled either to model adaptation or to lightweight control modules.

The limitations are correspondingly domain-specific. In medical simplification, hallucinations persist even with $k$7 and decoder heuristics, data inconsistencies make hallucination difficult to define, and inter-annotator agreement on readability is low (Flores et al., 2023). In DisCup, larger candidate sets improve control but hurt PPL, overly long prompts degrade performance, and the method depends heavily on the base CLM quality (Zhang et al., 2022). In Ghost, there is an inevitable utility-versus-fairness trade-off, AUO is sensitive to $k$8 and the undesired collection size, the undesired set is static rather than dynamically updated, and the scope is limited to supervised fine-tuning rather than reinforcement-learning-based GRs (Yin et al., 16 May 2026).

The proposed extensions also remain consistent across domains. The medical work suggests more domain-specific readability criteria, additional medical subdomains, and other languages (Flores et al., 2023). DisCup points to broader controllable generation settings in which a discriminator can define desired and undesired candidates (Zhang et al., 2022). Ghost suggests that popularity-aware, asymmetric unlikelihood may generalize to other frequency-biased generative tasks (Yin et al., 16 May 2026). Taken together, these studies support a broad methodological claim: asymmetric unlikelihood optimization is best understood as a configurable way of shaping generation by specifying which outputs should be actively suppressed, under what context, and with what asymmetry.