--- title: Sequence Likelihood Calibration (SLiC) url: https://www.emergentmind.com/topics/sequence-likelihood-calibration-slic type: topic --- # Sequence Likelihood Calibration (SLiC) Sequence Likelihood Calibration (SLiC) is a post hoc sequence-level fine-tuning paradigm that explicitly calibrates the output probabilities of conditional language models such that higher-quality outputs receive greater likelihood. The core principle is to correct the misalignment between model likelihoods and actual generation quality often observed after conventional maximum likelihood estimation (MLE). SLiC and its human-feedback-augmented variant SLiC-HF offer strong empirical gains in conditional generation tasks, simplify quality alignment with human preferences, and remove dependence on ad hoc decoding heuristics common in standard workflows [2210.00045][2305.10425]. ## 1. Motivation and Conceptual Background Conditional language models, such as encoder–decoder Transformers, are typically trained under the MLE objective: $$L_{\mathrm{MLE}}(\theta) = -\sum_{(x, y) \in D} \sum_{t=1}^{|y|} \log p_\theta(y_t|y_{ 0$ is a margin parameter. The optimizer minimizes the combined objective: $$ L_{\mathrm{SLiC}}(\theta) = \mathbb{E}_{(x, y^+, y^-) \sim \mathcal{D}_{\mathrm{rank}}}[ L_{\mathrm{cal}}(\theta; x, y^+, y^-) ] + \lambda L_{\mathrm{reg}}(\theta; x, y_{\mathrm{ref}}) $$ with $L_{\mathrm{reg}}$ typically being cross-entropy or token-level KL divergence against the initial finetuned model, ensuring the calibrated model does not drift excessively from the original task distribution. In practical variants, $\mathcal{D}_{\mathrm{rank}}$ is constructed by sampling multiple candidates per $(x, y_{\mathrm{ref}})$ and ranking them by a latent similarity metric [2210.00045]. Regularization weight $\lambda$ is selected to maintain scale between ranking and reference-preservation terms. ## 3. SLiC-HF: Calibration with Human Feedback Sequence Likelihood Calibration with Human Feedback (SLiC-HF) extends SLiC by using explicit human preference data to determine which outputs should be assigned higher likelihood [2305.10425]. Human feedback arrives as ranking triples $(x, y^+, y^-)$, where $y^+$ is judged superior to $y^-$ for input $x$. The SLiC-HF loss is $$ L_{\mathrm{SLiC-HF}}(\theta) = \mathbb{E}_{(x, y^+, y^-) \sim \mathcal{D}_{\mathrm{HF}}} [ \max(0, \delta - \log p_\theta(y^+|x) + \log p_\theta(y^-|x)) ] + \lambda \mathbb{E}_{(x, y_{\mathrm{ref}}) \sim \mathcal{D}_{\mathrm{SFT}}} [ -\log p_\theta(y_{\mathrm{ref}} | x) ] $$ This provides a competitive alternative to Reinforcement Learning from Human Feedback (RLHF), particularly the Proximal Policy Optimization (PPO) variant, by eliminating the need for auxiliary value/reward networks, roll-out decoding, and extensive hyperparameter tuning. SLiC-HF enables purely off-policy use of existing human feedback—i.e., it decouples calibration from the particular distribution of the current model’s outputs [2305.10425]. The gradient update is: $$ \nabla_\theta L = \mathbb{E}_{(x, y^+, y^-)}[ -\mathbf{1}_{\Delta < \delta}(\nabla_\theta \log p_\theta(y^+|x) - \nabla_\theta \log p_\theta(y^-|x)) ] + \lambda \mathbb{E}_{(x, y_{\mathrm{ref}})} [ -\nabla_\theta \log p_\theta(y_{\mathrm{ref}}|x) ] $$ with $\Delta = \log p_\theta(y^+|x) - \log p_\theta(y^-|x)$. ## 4. Algorithmic Workflow and Implementation Aspects SLiC and SLiC-HF follow a common multi-stage workflow: 1. Pretrain or load an MLE-finetuned sequence model $p_\theta^{(0)}$ on supervised data. 2. For each $(x, y_{\mathrm{ref}})$, decode $m$ candidates from $p_\theta^{(0)}$. 3. Rank candidate sequences by a similarity metric (e.g., model latent similarity, possibly supplemented or replaced by human preference using a ranking model). 4. For each $x$, form ranked pairs $(y^+, y^-)$ and apply the margin ranking loss. 5. Optionally, supplement with cross-entropy or KL regularization on $(x, y_{\mathrm{ref}})$. 6. Optimize over minibatches until convergence. Key hyperparameters include margin $\delta$ (default $\sim$1.0 for SLiC-HF or 3.0 for SLiC), number of sampled candidates $m$ ($\sim8$ for SLiC-HF, $5$–$10$ for SLiC), learning rate, and regularization coefficient $\lambda$ [2210.00045][2305.10425]. Candidate generation and ranking can be performed entirely offline, parallelizing throughput. No value network or live roll-out is required, in contrast to PPO-RLHF, yielding an order-of-magnitude reduction in GPU memory footprint ($\sim p$ vs $\sim 4p$ for PPO) and per-step compute. SLiC-HF further supports caching encoder states for additional speedup [2305.10425]. ## 5. Empirical Performance and Evaluation SLiC consistently increases automatic and human-measured generation quality, as well as matching or surpassing strong SOTA baselines across diverse language generation tasks: - On PEGASUS_LARGE, the pairwise rank loss plus KL regularization yields a +4.3% relative ROUGE geometric mean over four summarization settings [2210.00045]. - Calibration removes the beam-size curse: after SLiC, performance monotonically improves with increasing beam size up to at least 20, without need for length normalization or n-gram blocking. Repetition rates align with those of gold references. - Under inference-FLOP constraints, a smaller calibrated model can outperform a larger uncalibrated model if allowed a wider beam. - SLiC-HF on the Reddit TL;DR summarization benchmark achieves a 96.10% win-rate (model summaries preferred over reference) on T5-XXL, compared to 62.34% for SFT alone [2305.10425]. - Human evaluation (side-by-side and best-of-$n$) shows SLiC-HF is judged "best" in 73% of comparisons and exceeds RLHF-PPO models in pairwise preference, average quality, and factuality metrics. Abridged empirical results: | Model | Reference Win-rate | Human "Best" (%) | Quality (/5) | Factuality (%) | |-----------------------------------|-------------------|------------------|--------------|---------------| | SFT (no HF) | 44.96% | – | – | – | | SFT on positive only | 51.65% | – | – | – | | SLiC-HF-direct (no ranking model) | 82.92% | – | – | – | | SLiC-HF-sample-rank | 86.21% | 73% | 3.82 | 96.6% | SLiC and SLiC-HF demonstrate no diminishing returns with model size scaling and maintain performance improvements for increasing capacities [2210.00045][2305.10425]. ## 6. Analysis: Theoretical Insights, Limitations, and Practical Considerations The sequence-level margin-based (hinge) loss utilized by SLiC and SLiC-HF functions as a convex surrogate, leading to substantially lower gradient variance than REINFORCE-type RL objectives; however, no formal RL convergence guarantee for the hinge calibration is established. SLiC’s regularization is not strictly necessary to realize most performance gains but aids in preserving model behavior [2210.00045][2305.10425]. Limitations and open theoretical questions include: - SLiC-HF currently applies to summarization; extension to richer domains such as dialogue or code generation is yet to be validated. - It consumes pairwise or listwise preferences; handling richer feedback (e.g., continuous scores) is a plausible future research direction. - Calibration is an offline process, requiring candidate decoding for all training samples; the cost is amortized and does not affect inference. - Off-policy, non-iid data scenarios remain an open problem for guarantees. A plausible implication is that SLiC’s machinery may be adapted with minimal modification for other alignment objectives, or even combined with token-level calibration approaches (e.g., context-weighted likelihoods [2406.01806]) to further refine model ranking behavior. ## 7. Relation to Broader Methodological Landscape SLiC and SLiC-HF can be situated in the broader context of policy alignment and calibration methods in conditional generation: - In contrast to RLHF-PPO, SLiC’s margin-based loss removes dependence on auxiliary networks and is operationally equivalent to offline policy optimization with strong practical advantages in memory, compute, and tuning. - Unlike maximum-likelihood fine-tuning, SLiC directly optimizes the probability order among plausible outputs. - Similar paradigms leveraging candidate re-ranking, contrastive learning, and human preference pairings exist, but SLiC's simplicity and empirical robustness distinguish it from more complex actor-critic or sampled-reward frameworks [2210.00045][2305.10425]. - Contextualized Sequence Likelihood (CSL) [2406.01806] introduces per-token weighting for sequence confidence via attention, representing an orthogonal calibration axis that could, in principle, be integrated with SLiC’s sequence-level margin objectives. In summary, Sequence Likelihood Calibration and its feedback-augmented variants constitute efficient, interpretable, and empirically validated solutions for aligning sequence probability scores with output quality—substantially improving both automated and human-perceived model performance, while significantly reducing the operational complexity of fine-tuning pipelines [2210.00045][2305.10425]. Source: https://www.emergentmind.com/topics/sequence-likelihood-calibration-slic