---
title: Fidelity-Enriched Contrastive Search (FECS)
url: https://www.emergentmind.com/topics/fidelity-enriched-contrastive-search-fecs
type: topic
---

# Fidelity-Enriched Contrastive Search (FECS)

Fidelity-Enriched Contrastive Search (FECS) is a decoding algorithm for neural language models designed to address the persistent trade-off between semantic faithfulness to source information and output diversity in natural language generation. FECS extends the Contrastive Search framework by incorporating a source-aware faithfulness reward to encourage generated content that remains consistent with the input source, reducing hallucinations while maintaining lexical and semantic diversity. This method is particularly targeted at tasks such as abstractive summarization and knowledge-grounded dialogue, where model-generated hallucinations undermine factual accuracy. FECS optimizes token selection at each decoding step according to a combined objective involving model confidence, repetition penalty, and source similarity, resulting in consistent improvements in factual alignment without sacrificing output diversity [2310.14981].

## 1. Hallucination in Neural Text Generation and the Faithfulness–Diversity Trade-Off

Large pretrained language models frequently generate fluent textual outputs that are not grounded in, or may directly contradict, the provided source. In applications such as abstractive summarization or knowledge-grounded dialogue, such hallucinations degrade factual consistency. Conventional decoding strategies present a trade-off:

- Deterministic methods (e.g., greedy, beam search) maximize likelihood but often result in repetitiveness or generic phrasing, leading to poor diversity.
- Stochastic sampling methods (top-$k$, nucleus sampling) improve diversity but permit off-topic or unsupported content, increasing hallucination risk.

This tension arises because enforcing high model-confidence in output selection often diminishes diversity, while seeking diversity can reduce semantic alignment with the source. FECS addresses this trade-off by supplementing a diversity-preserving base (Contrastive Search) with an explicit faithfulness reward, biasing token selection towards semantic similarity with the source context [2310.14981].

## 2. Algorithmic Formulation of FECS

FECS augments the Contrastive Search mechanism by introducing a third term to the scoring function that measures source faithfulness. At each generation step $t$, let the input prefix be $x_{0:c+t} = [x_0,\dots,x_c, x_{c+1},\dots,x_{c+t-1}]$, where tokens decompose as:

- $[x_0,\dots,x_{s-1}]$: prompt tokens
- $[x_s,\dots,x_{c-1}]$: source tokens (to which generation must be faithful)
- $[x_c,\dots,x_{c+t-1}]$: tokens generated so far

For $\alpha, \beta \ge 0$ and $k$ denoting the candidate pool size, at each step, select the next token as:

\[
x_{c+t} = \arg\max_{v \in V_k}
  (1-\alpha-\beta)\,\log p_{\text{LM}}(v \mid x_{0:c+t-1})
  - \alpha\,\underset{c \le j \le c+t-1}{\max}\; \text{sim}(h_v, h_{x_j})
  + \beta\,\underset{s \le j \le c-1}{\max}\; \text{sim}(h_v, h_{x_j})
\]

where:

- $V_k$: top-$k$ candidates by model probability,
- $h_v$, $h_{x_j}$: final-layer hidden-state embeddings for candidate $v$ and token $x_j$,
- $\text{sim}(u,v)$: cosine similarity,
- $(1-\alpha-\beta)$: weight for model confidence,
- $\alpha$: weight for repetition (degeneration) penalty,
- $\beta$: weight for source faithfulness reward.

The weights form a convex combination if $\alpha+\beta\le1$. The additional $\beta$-weighted term incentivizes the model to generate tokens closer, in embedding space, to any source token, thereby mitigating hallucinations [2310.14981].

**Pseudocode summary:**
```
Input: LM, prefix x[0:c]=[prompt, source], max_len T, (k, α, β)
Output: generated continuation x[c : c+T-1]
generated = []
for t in 1..T:
    p(v | x[0:c-1] ++ generated)
    V_k = top-k tokens by p(·)
    for each v ∈ V_k:
        score_conf = log p(v)
        score_deg = max_{u in generated} cos(h_v, h_u)
        score_fth = max_{s in source_tokens} cos(h_v, h_s)
        FECS_score[v] = (1 - α - β)*score_conf
                        - α*score_deg
                        + β*score_fth
    v* = argmax_v FECS_score[v]
    Append v* to generated
return generated
```

## 3. Experimental Methodology

Experiments evaluate FECS on two tasks prone to hallucination:

- **Abstractive Summarization:** Using the CNN-DailyMail dataset.
- **Knowledge-Grounded Dialogue:** Using Wizard of Wikipedia ("WoW").

Language models of varying sizes were used: OPT (1.3B, 2.7B, 6.7B parameters) for summarization and GPT-Neo (1.3B, 2.7B), GPT-J (6B) for dialogue. Each prompt incorporated two few-shot examples without further finetuning. Baseline decoders included greedy, beam (beam=4), nucleus sampling (p=0.95), and Contrastive Search (k=4, $\alpha=0.6$) [2310.14981].

Standard evaluation metrics encompassed:

- **Quality:** ROUGE-1/2/L, BERTScore (summarization); BLEU-4, ROUGE-L, BERTScore (dialogue).
- **Faithfulness:** FEQA (summarization), Q2 (dialogue).
- **Diversity:** $1 - \text{Rep-n}(x)$, with $\text{Rep-n}(x) = 1 - \frac{|\text{unique n-grams}|}{|\text{total n-grams}|}$.

## 4. Quantitative and Qualitative Results

FECS delivers consistent improvements in faithfulness scores (FEQA/Q2) while maintaining or minimally impacting diversity and standard quality metrics. On CNN-DailyMail, FEQA increases by 21.8% (1.3B), 19.2% (2.7B), and 27.6% (6.7B), with only marginal decreases in diversity rate (5.0%, 0.2%, 1.1%). For WoW, Q2 improves by 26.2%, 63.9%, and 63.6% (corresponding to model size), with diversity reduction of −35%, −11.2%, and −3.3% respectively [2310.14981].

| Task/Dataset | Faithfulness Gain (FEQA/Q2) | Diversity Change |
|--------------|-----------------------------|------------------|
| CNN-DM 1.3B  | +21.8%                      | −5.0%            |
| CNN-DM 6.7B  | +27.6%                      | −1.1%            |
| WoW 2.7B     | +63.9%                      | −11.2%           |

Qualitative analysis highlights that FECS can recover factual content missed by other decoders. For instance, in summarization, FECS-generated summaries included all salient information from the source, whereas Contrastive Search omitted details and introduced hallucinated entities. FECS ensures low n-gram repetition across scales, on par with Contrastive Search and significantly below greedy/beam [2310.14981].

## 5. Hyperparameterization and Ablation

Typical FECS operation employs $(k, \alpha, \beta) = (4, 0.3, 0.3)$ without additional tuning, while Contrastive Search uses $(k, \alpha) = (4, 0.6)$. Comparison reveals that lowering $\alpha$ alone in Contrastive Search does not produce comparable faithfulness gains; the explicit inclusion of $\beta$ (faithfulness reward) is essential for FECS's improvements. Decoding overhead for FECS is modestly higher than that for greedy or beam but remains similar to Contrastive Search [2310.14981].

## 6. Limitations and Prospects

FECS presumes the source segment is reliable; if the source contains errors or contradictions, the faithfulness term may amplify these inaccuracies. Standard faithfulness metrics (e.g., FEQA, Q2) quantify factual alignment but do not assess implicit or nuanced semantic consistency. Potential directions include modeling source uncertainty, extending FECS to other text generation regimes (e.g., machine translation, data-to-text), and jointly or adaptively tuning hyperparameters $(\alpha, \beta)$ based on validation data or per-example characteristics [2310.14981]. A plausible implication is that FECS’s explicit source anchoring may prove broadly beneficial for tasks requiring grounded generation, provided appropriate handling of source reliability is incorporated.

Source: https://www.emergentmind.com/topics/fidelity-enriched-contrastive-search-fecs