Sequence Likelihood Calibration (SLiC)

• Sequence Likelihood Calibration (SLiC) is a post hoc method that recalibrates conditional language models by reordering candidate sequences for quality alignment.
• It employs a margin-based ranking loss to directly correct misalignments from maximum likelihood estimation, bypassing heuristic decoding adjustments.
• The SLiC-HF variant integrates human feedback to further refine output quality, achieving notable gains in generation performance and efficiency.

Sequence Likelihood Calibration (SLiC) is a post hoc sequence-level fine-tuning paradigm that explicitly calibrates the output probabilities of conditional LLMs such that higher-quality outputs receive greater likelihood. The core principle is to correct the misalignment between model likelihoods and actual generation quality often observed after conventional maximum likelihood estimation (MLE). SLiC and its human-feedback-augmented variant SLiC-HF offer strong empirical gains in conditional generation tasks, simplify quality alignment with human preferences, and remove dependence on ad hoc decoding heuristics common in standard workflows (Zhao et al., 2022, Zhao et al., 2023).

1. Motivation and Conceptual Background

Conditional LLMs, such as encoder–decoder Transformers, are typically trained under the MLE objective:

$$L_{\mathrm{MLE}}(\theta) = -\sum_{(x, y) \in D} \sum_{t=1}^{|y|} \log p_\theta(y_t|y_{<t}, x)$$

where supervision is restricted to a single reference $y$ per input $x$. This regime lacks explicit supervision for ranking multiple plausible candidates. Empirically, this manifests in phenomena such as the beam-size curse—where wider beam search degrades output quality—and the necessity for workarounds like length normalization, n-gram blocking, and expert-tuned decoding hyperparameters. These issues arise because the model’s sequence-level likelihoods do not accurately track output quality (Zhao et al., 2022).

Sequence Likelihood Calibration (SLiC) introduces a dedicated post-finetuning stage to address this misranking. By reordering candidate sequences according to a similarity or preference metric and further optimizing the model so that higher-quality (or human-preferred) outputs have greater sequence likelihood, SLiC directly aligns probabilistic scoring with “true” output quality, obviating downstream heuristics (Zhao et al., 2022, Zhao et al., 2023).

2. SLiC Mathematical Formulation and Training Objectives

After initial MLE finetuning, SLiC uses a margin-based, sequence-level ranking loss. Given a pair of sequences $(y^+, y^-)$ for a common input $x$, the calibration loss is

$$L_{\mathrm{cal}}(\theta; x, y^+, y^-) = \max(0, \delta - \log p_\theta(y^+|x) + \log p_\theta(y^-|x))$$

where $\delta > 0$ is a margin parameter. The optimizer minimizes the combined objective:

$$L_{\mathrm{SLiC}}(\theta) = \mathbb{E}_{(x, y^+, y^-) \sim \mathcal{D}_{\mathrm{rank}}}[ L_{\mathrm{cal}}(\theta; x, y^+, y^-) ] + \lambda L_{\mathrm{reg}}(\theta; x, y_{\mathrm{ref}})$$

with $L_{\mathrm{reg}}$ typically being cross-entropy or token-level KL divergence against the initial finetuned model, ensuring the calibrated model does not drift excessively from the original task distribution. In practical variants, $\mathcal{D}_{\mathrm{rank}}$ is constructed by sampling multiple candidates per $(x, y_{\mathrm{ref}})$ and ranking them by a latent similarity metric (Zhao et al., 2022). Regularization weight $\lambda$ is selected to maintain scale between ranking and reference-preservation terms.

3. SLiC-HF: Calibration with Human Feedback

Sequence Likelihood Calibration with Human Feedback (SLiC-HF) extends SLiC by using explicit human preference data to determine which outputs should be assigned higher likelihood (Zhao et al., 2023). Human feedback arrives as ranking triples $(x, y^+, y^-)$, where $y^+$ is judged superior to $y^-$ for input $x$. The SLiC-HF loss is

$$L_{\mathrm{SLiC-HF}}(\theta) = \mathbb{E}_{(x, y^+, y^-) \sim \mathcal{D}_{\mathrm{HF}}} [ \max(0, \delta - \log p_\theta(y^+|x) + \log p_\theta(y^-|x)) ] + \lambda \mathbb{E}_{(x, y_{\mathrm{ref}}) \sim \mathcal{D}_{\mathrm{SFT}}} [ -\log p_\theta(y_{\mathrm{ref}} | x) ]$$

This provides a competitive alternative to Reinforcement Learning from Human Feedback (RLHF), particularly the Proximal Policy Optimization (PPO) variant, by eliminating the need for auxiliary value/reward networks, roll-out decoding, and extensive hyperparameter tuning. SLiC-HF enables purely off-policy use of existing human feedback—i.e., it decouples calibration from the particular distribution of the current model’s outputs (Zhao et al., 2023).

The gradient update is:

$$\nabla_\theta L = \mathbb{E}_{(x, y^+, y^-)}[ -\mathbf{1}_{\Delta < \delta}(\nabla_\theta \log p_\theta(y^+|x) - \nabla_\theta \log p_\theta(y^-|x)) ] + \lambda \mathbb{E}_{(x, y_{\mathrm{ref}})} [ -\nabla_\theta \log p_\theta(y_{\mathrm{ref}}|x) ]$$

with $$\Delta = \log p_\theta(y^+|x) - \log p_\theta(y^-|x)$$.

4. Algorithmic Workflow and Implementation Aspects

SLiC and SLiC-HF follow a common multi-stage workflow:

1. Pretrain or load an MLE-finetuned sequence model $p_\theta^{(0)}$ on supervised data.
2. For each $(x, y_{\mathrm{ref}})$, decode $m$ candidates from $p_\theta^{(0)}$.
3. Rank candidate sequences by a similarity metric (e.g., model latent similarity, possibly supplemented or replaced by human preference using a ranking model).
4. For each $x$, form ranked pairs $(y^+, y^-)$ and apply the margin ranking loss.
5. Optionally, supplement with cross-entropy or KL regularization on $(x, y_{\mathrm{ref}})$.
6. Optimize over minibatches until convergence.

Key hyperparameters include margin $\delta$ (default $\sim$1.0 for SLiC-HF or 3.0 for SLiC), number of sampled candidates $m$ ($\sim8$ for SLiC-HF, $5$–$10$ for SLiC), learning rate, and regularization coefficient $\lambda$ (Zhao et al., 2022, Zhao et al., 2023). Candidate generation and ranking can be performed entirely offline, parallelizing throughput. No value network or live roll-out is required, in contrast to PPO-RLHF, yielding an order-of-magnitude reduction in GPU memory footprint ($\sim p$ vs $\sim 4p$ for PPO) and per-step compute. SLiC-HF further supports caching encoder states for additional speedup (Zhao et al., 2023).

5. Empirical Performance and Evaluation

SLiC consistently increases automatic and human-measured generation quality, as well as matching or surpassing strong SOTA baselines across diverse language generation tasks:

• On PEGASUS_LARGE, the pairwise rank loss plus KL regularization yields a +4.3% relative ROUGE geometric mean over four summarization settings (Zhao et al., 2022).
• Calibration removes the beam-size curse: after SLiC, performance monotonically improves with increasing beam size up to at least 20, without need for length normalization or n-gram blocking. Repetition rates align with those of gold references.
• Under inference-FLOP constraints, a smaller calibrated model can outperform a larger uncalibrated model if allowed a wider beam.
• SLiC-HF on the Reddit TL;DR summarization benchmark achieves a 96.10% win-rate (model summaries preferred over reference) on T5-XXL, compared to 62.34% for SFT alone (Zhao et al., 2023).
• Human evaluation (side-by-side and best-of-$n$) shows SLiC-HF is judged "best" in 73% of comparisons and exceeds RLHF-PPO models in pairwise preference, average quality, and factuality metrics.

Abridged empirical results:

Model	Reference Win-rate	Human "Best" (%)	Quality (/5)	Factuality (%)
SFT (no HF)	44.96%	–	–	–
SFT on positive only	51.65%	–	–	–
SLiC-HF-direct (no ranking model)	82.92%	–	–	–
SLiC-HF-sample-rank	86.21%	73%	3.82	96.6%

SLiC and SLiC-HF demonstrate no diminishing returns with model size scaling and maintain performance improvements for increasing capacities (Zhao et al., 2022, Zhao et al., 2023).

6. Analysis: Theoretical Insights, Limitations, and Practical Considerations

The sequence-level margin-based (hinge) loss utilized by SLiC and SLiC-HF functions as a convex surrogate, leading to substantially lower gradient variance than REINFORCE-type RL objectives; however, no formal RL convergence guarantee for the hinge calibration is established. SLiC’s regularization is not strictly necessary to realize most performance gains but aids in preserving model behavior (Zhao et al., 2022, Zhao et al., 2023).

Limitations and open theoretical questions include:

• SLiC-HF currently applies to summarization; extension to richer domains such as dialogue or code generation is yet to be validated.
• It consumes pairwise or listwise preferences; handling richer feedback (e.g., continuous scores) is a plausible future research direction.
• Calibration is an offline process, requiring candidate decoding for all training samples; the cost is amortized and does not affect inference.
• Off-policy, non-iid data scenarios remain an open problem for guarantees.

A plausible implication is that SLiC’s machinery may be adapted with minimal modification for other alignment objectives, or even combined with token-level calibration approaches (e.g., context-weighted likelihoods (Lin et al., 2024)) to further refine model ranking behavior.

7. Relation to Broader Methodological Landscape

SLiC and SLiC-HF can be situated in the broader context of policy alignment and calibration methods in conditional generation:

• In contrast to RLHF-PPO, SLiC’s margin-based loss removes dependence on auxiliary networks and is operationally equivalent to offline policy optimization with strong practical advantages in memory, compute, and tuning.
• Unlike maximum-likelihood fine-tuning, SLiC directly optimizes the probability order among plausible outputs.
• Similar paradigms leveraging candidate re-ranking, contrastive learning, and human preference pairings exist, but SLiC's simplicity and empirical robustness distinguish it from more complex actor-critic or sampled-reward frameworks (Zhao et al., 2022, Zhao et al., 2023).
• Contextualized Sequence Likelihood (CSL) (Lin et al., 2024) introduces per-token weighting for sequence confidence via attention, representing an orthogonal calibration axis that could, in principle, be integrated with SLiC’s sequence-level margin objectives.

In summary, Sequence Likelihood Calibration and its feedback-augmented variants constitute efficient, interpretable, and empirically validated solutions for aligning sequence probability scores with output quality—substantially improving both automated and human-perceived model performance, while significantly reducing the operational complexity of fine-tuning pipelines (Zhao et al., 2022, Zhao et al., 2023).