---
title: Asymmetric Unlikelihood Optimization
url: https://www.emergentmind.com/topics/asymmetric-unlikelihood-optimization
type: topic
---

# Asymmetric Unlikelihood Optimization

Asymmetric unlikelihood optimization denotes a family of training strategies in which standard likelihood-based sequence learning is augmented with an unlikelihood term that suppresses particular undesired tokens or sequences more strongly than others. Across recent work, the asymmetry is realized in different ways: by weighting penalties with word-level readability scores and masking hallucinated entities in medical text simplification [2310.11191], by using a discriminator to define desired and undesired next-token candidates in controllable generation [2210.09551], and by penalizing popular head-item tokens only when training on long-tail items in generative recommendation [2605.16825]. The common pattern is directional optimization: some outputs are explicitly pushed down, while others are either left unaffected or reinforced by a separate likelihood term.

## 1. Conceptual scope

The basic unlikelihood form is the familiar $-\log(1-p)$ penalty on negative targets, but the cited systems depart from symmetric treatment of errors. In the medical simplification setting, the total loss is
\[
\mathcal{L} = \mathcal{L}_{\text{NLL}} + \lambda_R \,\mathit{UL}_R + \lambda_C \,\mathit{UL}_C,
\]
where $\mathit{UL}_R$ is readability-oriented and $\mathit{UL}_C$ is factual-consistency-oriented, with fixed hyperparameters $\lambda_R = 7.5 \times 10^{-4}$ and $\lambda_C = 2.5 \times 10^{-4}$ [2310.11191]. In DisCup, the objective adds a likelihood term over desirable top-$k$ candidates and an unlikelihood term over undesirable top-$k$ candidates, with no separate coefficient between them; only the prompt parameters are updated, while the CLM and discriminator remain frozen [2210.09551]. In Ghost, the recommendation loss is
\[
\mathcal{L}_{\text{All}} = \mathcal{L}_{\text{NLL}} + \alpha \cdot \mathcal{L}_{\text{AUO}},
\]
where $\mathcal{L}_{\text{AUO}}$ is applied only to tail-item training examples and uses head-item SIDs as negatives [2605.16825].

Taken together, these formulations suggest that “asymmetric unlikelihood optimization” is not a single canonical algorithm. Rather, it is a design pattern in which the negative target set, the penalty strength, and the population to which the penalty is applied are all task-specific and intentionally non-uniform.

## 2. Objective design and mathematical structure

In medical text simplification, the readability unlikelihood term penalizes only the most probable token at each generation step, and scales the penalty by a precomputed Flesch-Kincaid score:
\[
\mathit{UL}_R = - \sum_{t=1}^{|\hat{y}|} \sum_{v=1}^{|\mathcal{V}|} \mathbbm{1}_{v,t}\, FK_v \, \log\bigl(1 - p(v \mid \hat{y}_{<t})\bigr).
\]
Here $\mathbbm{1}_{v,t}=1$ only for the argmax token at step $t$, so complex words with high $FK_v$ incur larger penalties, while simpler words receive weaker penalties [2310.11191]. The factual-consistency term has the analogous form
\[
\mathit{UL}_C = - \sum_{t=1}^{|\hat{y}|} \sum_{v=1}^{|\mathcal{V}|} \mathbbm{1}_{v,t}\,\mathbbm{1}_{v,e} \log\bigl(1 - p(v \mid \hat{y}_{<t})\bigr),
\]
where $\mathbbm{1}_{v,e}$ marks tokens in a hallucinated-entity set $e$ extracted from greedy model outputs and filtered by spaCy `en_core_web_lg` NER [2310.11191].

In DisCup, asymmetry enters through a discriminator-guided partition of the frozen CLM’s top-$k$ candidate set $\mathcal{C}$. For a target attribute $a$, desirable candidates are weighted by
\[
\boldsymbol{s}[c] = \operatorname{softmax}(\mathbf{d}[c] / \alpha),
\]
and undesirable candidates by
\[
\boldsymbol{s}'[c] = \operatorname{softmax}(\mathbf{d}'[c] / \alpha), \qquad \mathbf{d}'[c]=1-\mathbf{d}[c],
\]
where $\mathbf{d}[c]=p_d(a\mid X_{1:t}^c)$ [2210.09551]. The two loss components are
\[
\mathcal{L}_{like}(x_t) = -\sum_{c \in \mathcal{C}} \boldsymbol{s}[c] \log p_{\theta}(c \mid x_{<t}, P_k),
\]
\[
\mathcal{L}_{unlike}(x_t) = -\sum_{c \in \mathcal{C}} \boldsymbol{s}'[c] \log\bigl(1-p_{\theta}(c \mid x_{<t}, P_k)\bigr).
\]
Desired tokens are pulled up by likelihood; undesired tokens are pushed down by unlikelihood [2210.09551].

In Ghost, the negative set is not token-local but item-structured. For a tail item, the AUO term is
\[
\mathcal{L}_{\text{AUO}} = -\sum_{\Omega \in \bar{\mathbf{\Omega}}} \sum_i \log\Big(1 - \mathcal{P}_\theta(c^{(i)} \mid h_u, c^{<i})\Big),
\]
where $\bar{\mathbf{\Omega}}$ is a collection of SIDs from undesired head items associated with that tail item [2605.16825]. The construction is explicitly popularity-aware: the negatives come from the head set, not the tail.

## 3. Forms of asymmetry

Three distinct forms recur in the literature.

**Token-wise asymmetry**: In medical simplification, the penalty weight is proportional to $FK_v$, so “complex top-1 words” are discouraged more than simple ones. The same system also exhibits “top-1 asymmetry,” because only the single highest-probability token at each step contributes to the unlikelihood loss [2310.11191].

**Content-wise asymmetry**: The factual-consistency term in medical simplification applies only to tokens in the hallucinated entity set $e$; supported or neutral entities are unaffected by this term [2310.11191]. A closely related pattern appears in Ghost, where head-item SIDs are treated as undesired only in the context of tail-item supervision [2605.16825].

**Different treatment of positive vs negative attribution**: DisCup explicitly separates desirable and undesirable candidates. Desired tokens receive a likelihood objective, undesirable ones receive an unlikelihood objective, and the roles of tokens depend on the chosen target attribute. For positive sentiment control, low $p_d(\text{positive}\mid\cdot)$ candidates are undesirable; for negative sentiment control, the target attribute changes and so does the undesirable set [2210.09551].

**Population asymmetry**: Ghost applies AUO only for tail-item training examples. Head items are trained only with likelihood, whereas tail items are trained with likelihood plus an unlikelihood term that suppresses competing head items. The paper characterizes this as “asymmetric token-level unlikelihood between the tail items and their head counterparts” [2605.16825].

These designs also differ in how they define negative targets. Medical simplification extracts them dynamically from model-preferred sequences and then filters them by NER [2310.11191]. DisCup derives them from discriminator judgments over top-$k$ CLM candidates [2210.09551]. Ghost constructs them by a two-stage selection over semantically similar head items and SID prefix structure [2605.16825].

## 4. Representative implementations

The major instantiations occupy different application regimes, but all preserve the core directional logic.

| Setting | Negative target construction | Additional mechanism |
|---|---|---|
| Medical text simplification | Complex top-1 words; hallucinated entities | Reranked beam search |
| Controllable text generation | Discriminator-defined undesirable top-$k$ candidates | Prompt tuning with frozen GPT-2 large |
| Generative recommendation | Popular head-item SIDs for tail examples | Skeleton-Founded Tokenization |

In medical text simplification, a BART-XSum encoder-decoder is fine-tuned on Cochrane, MedEasi, and Radiology, with unlikelihood active throughout training and implemented in HuggingFace Transformers. The decoding stage introduces another asymmetric component: every $k$ steps, beams are reranked by a composite readability-plus-consistency score, and beams containing entities absent from the source are assigned score $0$ and eliminated [2310.11191]. The reranking score uses normalized Flesch-Kincaid Grade Level $r_F(s)$, normalized BERTScore $r_B(s)$, and the squared harmonic mean
\[
r(s) = \left(\frac{2 r_F(s) r_B(s)}{r_F(s) + r_B(s)} \right)^2.
\]

In DisCup, the base generator is GPT-2 large with frozen parameters, and the control mechanism is a small number of continuous prompts, typically around 10 virtual tokens. A GPT-2 small discriminator is trained first on labeled attribute data and then frozen. At each training position, the CLM supplies top-$k$ candidates, the discriminator scores each candidate, and the prompt parameters are updated so that high-scoring candidates are more likely and low-scoring candidates less likely [2210.09551]. The method therefore moves discriminator guidance from decoding time into training time.

In Ghost, asymmetry is coupled to a structural change in item tokenization. Skeleton-Founded Tokenization uses head-item SIDs as a skeleton and lets each tail item inherit the first $L^h$ tokens from its closest head item before appending $L^t$ additional tokens of its own [2605.16825]. The stated purpose is to collapse many unpredictable branching points into a single branching point at step $L^h+1$, making AUO’s gradient corrections more effective.

## 5. Empirical behavior and trade-offs

The empirical record supports the view that asymmetric unlikelihood terms can improve the target property, but typically under a measurable trade-off.

In medical simplification, the Cochrane ablation isolates the roles of the two unlikelihood components. Full UL achieves FK $8.00$, BERTScore $0.862$, GPT-Eval hallucination count $27/50$, and SARI $42.07$. Using $\mathit{UL}_R$ alone yields FK $8.74$, GPT-Eval $41/50$ hallucinations, and SARI $41.37$. Using $\mathit{UL}_C$ alone yields FK $11.86$, GPT-Eval $16/50$ hallucinations, and SARI $35.69$ [2310.11191]. The paper also reports that UL+Decoder improves FK on Cochrane by up to $2.43$ points over the next-best model, and that on 30 Cochrane examples human judges preferred UL outputs over a BART-XSum baseline in **43%** of cases, versus **3%** for prior SOTA NAPSS [2310.11191].

In DisCup, explicit comparison with and without unlikelihood shows that the unlikelihood term contributes materially to control performance. For positive target steering from neutral prompts, correctness rises from $91.58\%$ without unlikelihood to **94.98\%** with unlikelihood, while PPL changes from $44.20$ to $48.71$ and coverage from $3.94\%$ to $3.24\%$. For negative target steering from neutral prompts, correctness rises from $60.80\%$ to **68.76\%**, with PPL changing from $36.72$ to $45.60$. In toxicity avoidance, average toxicity probability changes from $0.066$ to **0.064** [2210.09551]. The paper further reports inference times to generate 20 tokens of $0.78$ s for GPT-2 large, $0.94$ s for DisCup, $2.54$ s for DExperts, and $37.39$ s for PPLM [2210.09551].

In Ghost, AUO is evaluated jointly with SKT against standard generative recommenders and debiasing baselines. Across three datasets, the reported average gains versus standard GRs are +$63.91\%$ Tail HR and +$70.66\%$ Tail NDCG. MGU is reduced by about $55.76\%$ on average, while overall HR and NDCG drop by about $7.46\%$ and $6.34\%$ relative to LC-Rec [2605.16825]. The paper also reports that recommendation lists from standard GRs are more than $97\%$ head items, whereas Ghost changes exposure from approximately $24$k head versus about $560$ tail items across users to about $15$k head versus about $9.4$k tail items [2605.16825].

A common empirical theme is that the asymmetric term improves the targeted property—readability, attribute control, or tail exposure—but must be balanced against fluency, factuality, or overall utility.

## 6. Theoretical interpretation, limitations, and extensions

The most explicit theoretical diagnosis appears in Ghost. Under token-level MLE, the paper derives “gradient starvation”: for a tail-specific token that almost never appears as ground truth, the expected update is approximately non-positive, so the token embedding is pushed away from the user preference direction [2605.16825]. It then derives “head token dominance at branching point” and “bias amplification via undifferentiated tokenization,” arguing that standard SID tokenization turns local head–tail preference gaps into geometric suppression over multiple branching points. AUO is presented as adding a “Rescue Force” for tail tokens by pushing down undesired head tokens [2605.16825].

The medical simplification paper provides a different theoretical caution: readability is not optimized directly at the sentence level. Instead, the system uses word-level FK scores as proxies, and “there’s no direct gradient from a document-level FK metric” [2310.11191]. This is a useful corrective to the misconception that asymmetric unlikelihood necessarily optimizes the final evaluation metric itself. In that system, the method optimizes an indirect proxy during training and a reranking criterion during decoding.

DisCup addresses another misconception: the technique is not restricted to full-model fine-tuning. Its asymmetric unlikelihood objective updates only prompt parameters, leaving both the generator and discriminator frozen [2210.09551]. This suggests that asymmetric unlikelihood can be coupled either to model adaptation or to lightweight control modules.

The limitations are correspondingly domain-specific. In medical simplification, hallucinations persist even with $\mathit{UL}_C$ and decoder heuristics, data inconsistencies make hallucination difficult to define, and inter-annotator agreement on readability is low [2310.11191]. In DisCup, larger candidate sets improve control but hurt PPL, overly long prompts degrade performance, and the method depends heavily on the base CLM quality [2210.09551]. In Ghost, there is an inevitable utility-versus-fairness trade-off, AUO is sensitive to $\alpha$ and the undesired collection size, the undesired set is static rather than dynamically updated, and the scope is limited to supervised fine-tuning rather than reinforcement-learning-based GRs [2605.16825].

The proposed extensions also remain consistent across domains. The medical work suggests more domain-specific readability criteria, additional medical subdomains, and other languages [2310.11191]. DisCup points to broader controllable generation settings in which a discriminator can define desired and undesired candidates [2210.09551]. Ghost suggests that popularity-aware, asymmetric unlikelihood may generalize to other frequency-biased generative tasks [2605.16825]. Taken together, these studies support a broad methodological claim: asymmetric unlikelihood optimization is best understood as a configurable way of shaping generation by specifying which outputs should be actively suppressed, under what context, and with what asymmetry.

Source: https://www.emergentmind.com/topics/asymmetric-unlikelihood-optimization