---
title: Negative Certainty Supervision Loss
url: https://www.emergentmind.com/topics/negative-certainty-supervision-loss
type: topic
---

# Negative Certainty Supervision Loss

Searching arXiv for the cited paper and related context.
arxiv_search(query="2411.14797 Continual SFT Matches Multimodal RLHF with Negative Supervision", max_results=5, sort_by="relevance")
Negative supervised finetuning (nSFT) is a continual alignment objective for vision-language models (VLMs) in which the supervision signal consists of two standard supervised-finetuning terms: one on the chosen response and one on a constructed corrective dialogue derived from a rejected response. In “Continual SFT Matches Multimodal RLHF with Negative Supervision,” nSFT is presented as a way to disentangle the negative supervision embedded in multimodal RLHF, especially the logit of rejected responses, and to recover that signal within a simple cross-entropy training regime rather than with PPO- or DPO-style preference optimization [2411.14797]. The method is framed as a memory-efficient alternative to multimodal RLHF, where 2 (e.g., DPO) or 4 (e.g., PPO) large VLMs are strictly required, while nSFT requires only a single model in memory and one simple SFT loss [2411.14797].

## 1. Formal definition and notation

The method is defined on a multimodal input $x=(I,q)$, where $I$ is an image and $q$ denotes question tokens. The target chosen response is $y_c=(y_{c,1},\dots,y_{c,L})$, and the rejected response is $y_r=(y_{r,1},\dots,y_{r,L'})$, generated by the current model, for example via temperature sampling. The underlying policy is the vision-language model $\pi_\theta$ to be finetuned [2411.14797].

The standard supervised-finetuning token-wise log-loss on a target sequence $y$ is

$$
L_{\mathrm{sft}}(y;\theta)=-\sum_{i=1}^{|y|}\log \pi_\theta(y_i\mid y_{<i},x).
$$

A construction function $G(\cdot)$, implemented as a large LLM, e.g. GPT-4, takes the rejected reply $y_r$ together with the chosen response $y_c$ and a vision-error codebook $Q$, and produces a corrective conversation $G(y_r;y_c,Q)$ of length $L''$ [2411.14797].

The nSFT loss is

$$
L_{\mathrm{nSFT}}(\theta)=L_{\mathrm{sft}}(y_c;\theta)+L_{\mathrm{sft}}(G(y_r;y_c,Q);\theta).
$$

In the paper’s formulation, the “negative-response logits” are extracted by including $G(y_r)$ in the cross-entropy training data rather than by introducing a separate preference-classification objective [2411.14797].

## 2. Complete objective and optimization procedure

In a batched setting with preference dataset $D=\{(x,y_c)\}$, the training loop first samples a rejected response $y_r$ from the current policy $\pi_\theta$ for each input $x$, then constructs the corrective dialogue $G(y_r;y_c,Q)$. The overall objective is

$$
\min_\theta \mathbb{E}_{(x,y_c)\sim D}\left[L_{\mathrm{sft}}(y_c;\theta)+L_{\mathrm{sft}}(G(y_r;y_c,Q);\theta)\right].
$$

No auxiliary KL term is required, unlike PPO or DPO, although a per-token KL-penalty may optionally be added to stabilize training; this possibility is examined in ablations [2411.14797].

The step-by-step algorithm is specified as follows. For each minibatch $B$ of size $N$, one samples $\{x_j=(I_j,q_j),y_{c,j}\}_{j=1}^N$ from the alignment dataset $D$. For each $j$, a rejected reply $y_{r,j}$ is generated by sampling $\pi_\theta(\cdot\mid x_j)$, for example with top-k+temperature. The LLM $G$ is then prompted with $(y_{r,j},y_{c,j},Q)$ to identify instance- and image-level hallucinations and to output a corrective conversation $z_j=G(y_{r,j};y_{c,j},Q)$. Two target sequences are thus formed per example, $y_{c,j}$ and $z_j$, and the losses $L_{\mathrm{sft}}(y_{c,j};\theta)$ and $L_{\mathrm{sft}}(z_j;\theta)$ are computed. The batch loss is

$$
L_{\mathrm{batch}}=\frac{1}{N}\sum_{j=1}^N\left[L_{\mathrm{sft}}(y_{c,j})+L_{\mathrm{sft}}(z_j)\right].
$$

The model is then updated by backpropagating $\partial L_{\mathrm{batch}}/\partial \theta$ and taking an SGD/Adam step [2411.14797].

The significance of this formulation is that the alignment stage remains an SFT-style optimization problem. A plausible implication is that implementation complexity is reduced relative to RLHF pipelines that rely on reference models, value models, or rollout machinery, although the paper’s direct claims concern memory and runtime rather than software complexity.

## 3. Relation to DPO and the role of negative supervision

The paper motivates nSFT by arguing that the inherent value of multimodal RLHF lies in its negative supervision, specifically the logit of the rejected responses [2411.14797]. This argument is made explicit through a comparison with Direct Preference Optimization (DPO), whose binary log-loss is written as

$$
L_{\mathrm{DPO}}=-\mathbb{E}_{(x,y_c,y_r)}\log \sigma\left(\beta\cdot[\log \pi_\theta(y_c\mid x)-\log \pi_\theta(y_r\mid x)]\right).
$$

Ignoring reference-model terms, the logit inside $\sigma(\cdot)$ can be expressed as

$$
p'_{\mathrm{DPO}}=\sum_i\left[\log \pi_\theta(y_{c,i}\mid \dots)-\log \pi_\theta(y_{r,i}\mid \dots)\right]
=-(L_{\mathrm{sft}}(y_c)-L_{\mathrm{sft}}(y_r)).
$$

By differentiating $L_{\mathrm{DPO}}$ with respect to $\theta$, the paper reports

$$
\frac{\partial L_{\mathrm{DPO}}}{\partial \theta}\propto \frac{\partial L_{\mathrm{sft}}(y_c)}{\partial \theta}-\frac{\partial L_{\mathrm{sft}}(y_r)}{\partial \theta}.
$$

This is interpreted as a weighted combination of a positive and a negative SFT gradient, whereas naïve continual SFT uses only $\partial L_{\mathrm{sft}}(y_c)/\partial \theta$ and thus loses all negative supervision [2411.14797].

A second theoretical argument is a gradient-bias analysis. From the partial derivatives of $L_{\mathrm{DPO}}$ with respect to the two logits $t_1=\pi(y_c)$ and $t_2=\pi(y_r)$, and again ignoring the reference model, the paper states that

$$
\left|\frac{\partial L}{\partial t_1}\right|:\left|\frac{\partial L}{\partial t_2}\right|=\frac{t_2}{t_1}<1
$$

under standard preference pairs $(t_2<t_1)$. The reported interpretation is that DPO training emphasizes decreasing the negative logit faster than boosting the positive one, and that nSFT recovers both sides of this signal in an explicit SFT form [2411.14797].

This comparison directly addresses a common assumption stated in the paper: conventional wisdom holds the superiority of multimodal RLHF over continual SFT during the preference alignment stage. The paper’s position is narrower than a blanket rejection of RLHF; it argues specifically that the critical missing component in continual SFT is the negative supervision term, not that all RLHF machinery is intrinsically unnecessary in every setting.

## 4. Constructive negative supervision and corrective dialogue

A distinctive element of nSFT is that the negative term is not the raw rejected sequence itself. Instead, the rejected response is transformed by the construction function $G(\cdot)$ into a corrective conversation conditioned on the rejected response, the chosen response, and the vision-error codebook $Q$ [2411.14797].

The training procedure specifies that $G$ identifies instance- and image-level hallucinations. The resulting output $z_j=G(y_{r,j};y_{c,j},Q)$ becomes a second supervised target paired with the same multimodal input. In the paper’s summary, this is described as extracting and converting hallucinations into corrective dialogue, thereby recasting the rejected-response gradient into a standard cross-entropy term [2411.14797].

The vision-error codebook $Q$ is not an incidental component. In ablations on a 5 K LLaVA-150 K subset, full nSFT yields MMB $65.2$, SQA $68.4$, MME $1550$, and POPE $87.4$. Removing $Q$ degrades performance to MMB $64.4$, SQA $67.6$, MME $1505$, and POPE $86.0$. Removing $y_c$ while retaining the rest yields MMB $64.9$, SQA $68.2$, MME $1523$, and POPE $86.4$ [2411.14797].

The paper’s conclusion from these ablations is explicit: the $Q$-guided negative dialogue is crucial, and including $y_c$ is helpful on very short dialogues (OCRVQA). This suggests that the effectiveness of the negative term depends not only on exposing the model to a non-preferred response pattern, but on structuring that pattern as an error-aware corrective sequence rather than as a bare penalty target.

## 5. Experimental regime and quantitative comparisons

The experimental setup uses three alignment datasets, each with 10 K examples: OCRVQA, described as object-centric book covers with very short answers; TextCaps, described as OCR + captioning with medium length; and an LLaVA-150 K subset, described as GPT-4 captions with long length $\sim 100$ tokens [2411.14797]. The base vision-language model is LLaVA-1.5-7B with a Vicuna-7B backbone, with ablations on LLaVA-1.5-13B and LLaVA-NeXT-13B. Hyperparameters for nSFT, continual SFT, and DPO are Deepspeed ZeRO-3, batch size $=128$, learning rate $=2\times 10^{-6}$ with cosine scheduler, weight decay $=0$, and training steps $\approx 10$ K per dataset [2411.14797].

Evaluation covers nine tasks. Traditional VQA comprises SQA, GQA, and TextVQA ($\mathrm{VQA}^T$). Multimodal comprehension uses MMVet, MME, and MMBench. Hallucination uses POPE, CHAIR, and MMHal, where MMHal is rated $0$–$6$ by GPT. In-domain evaluation reports “IF score” and “accuracy” by GPT-4 on held-out $500$ per dataset, together with $\mathrm{ACC}_{10}^b$ and $\mathrm{ACC}_{10}^w$ [2411.14797].

On the main comparison in which training uses 10 K examples from each source and “total” is the sum of per-category improvements, the reported results are: Baseline, $+0.0/+0.0/+0.0$, total $0.0$; GT-DPO, $+0.7/+1.7/+0.6$, total $+3.0$; SeVa (DPO), $+0.6/+2.7/+8.7$, total $+12.0$; SIMA (DPO), $+1.4/+2.5/+5.1$, total $+9.0$; Cont. SFT, $-0.7/+2.2/-1.6$, total $0.0$; and nSFT (ours), $+2.0/+4.0/+8.9$, total $+14.9$ [2411.14797]. The category ordering in the paper is $\mathrm{VQA}^T+\mathrm{SQA}+\mathrm{GQA}$, MM comprehension, and Hallucination.

The paper also reports a direct comparison with PPO and iterative DPO, labeled CSR $\times 3$, on the 10 K LLaVA-150 K subset. Baseline obtains MMB $64.3$, SQA $66.8$, MME $1515$, and POPE $85.9$. PPO yields $64.7/67.8/1508/86.2$. CSR 1–3 gives MMB $64.3$–$64.4$, SQA $68.3$–$68.4$, MME $1501$–$1523$, and POPE $86.8$–$87.2$. nSFT yields MMB $65.2$, SQA $68.4$, MME $1550$, and POPE $87.4$ [2411.14797].

For in-domain evaluation on held-out 500 examples per dataset, the reported values are as follows. Baseline: IF score $8.52/6.41/7.79$, accuracy $8.37/5.88/7.01$, $\mathrm{ACC}_{10}^b$ $9.80/9.32/9.25$, $\mathrm{ACC}_{10}^w$ $0.00/0.40/2.02$. SeVa: IF score $8.69/6.71/7.99$, accuracy $8.51/6.07/7.13$, $\mathrm{ACC}_{10}^b$ $10.0/9.40/9.20$, $\mathrm{ACC}_{10}^w$ $0.56/1.72/2.32$. GT-DPO: IF score $8.70/6.65/7.95$, accuracy $8.41/6.22/7.12$, $\mathrm{ACC}_{10}^b$ $10.0/9.30/9.38$, $\mathrm{ACC}_{10}^w$ $0.26/1.13/2.68$. nSFT: IF score $8.71/6.78/8.09$, accuracy $8.56/6.12/7.27$, $\mathrm{ACC}_{10}^b$ $10.0/9.50/9.62$, $\mathrm{ACC}_{10}^w$ $0.37/0.70/2.30$ [2411.14797].

These quantitative results are used in the paper to support the claim that continual SFT can match multimodal RLHF once negative supervision is incorporated, and in several settings exceed DPO-, PPO-, and CSR-style baselines.

## 6. Efficiency claims, KL ablations, and interpretive scope

The efficiency claim is precise. On a single run on a 32 GB GPU, DPO, which uses two models, policy plus reference, or PPO, which uses policy, value, and rollouts, requires 2–4× more GPUs and 1.5–2× longer wall-clock time than nSFT, which uses a single model [2411.14797]. This is the paper’s main systems-level argument for preferring the nSFT formulation when the aim is to recover negative supervision without the overhead of full RLHF.

The KL ablation is reported on 5 K TextCaps. Continual SFT without KL gives MMB $63.6$, $\mathrm{VQA}^T$ $56.6$, and SQA $67.3$; with KL it gives $64.2$, $57.8$, and $67.7$. nSFT without KL gives $64.8$, $58.1$, and $68.1$; with KL it gives $65.2$, $58.4$, and $68.4$. DPO (SeVa) with KL gives MMB $65.0$, $\mathrm{VQA}^T$ $58.1$, and SQA $68.0$ [2411.14797]. The paper’s conclusion is that a mild KL-constraint further boosts both SFT and nSFT.

Two interpretive points follow directly from the reported evidence. First, the results are not presented as a claim that ordinary continual SFT is sufficient; in the main comparison, Cont. SFT has total $0.0$, with negative changes in both $\mathrm{VQA}^T+\mathrm{SQA}+\mathrm{GQA}$ and Hallucination [2411.14797]. Second, the paper does not claim that the rejected sequence alone is enough; the ablations indicate that the $Q$-guided corrective dialogue is crucial. A plausible implication is that the central design choice is not simply “train on bad outputs,” but “convert model-generated failure cases into structured negative supervision.”

In summary form, the paper states that nSFT recasts the “negative-supervision” half of RLHF, namely the rejected-response gradient, into a standard cross-entropy term by prompting an LLM to extract and convert hallucinations into corrective dialogue. Within the reported experiments, this achieves parity with—and often exceeds—the performance of heavyweight multimodal RLHF methods such as DPO, PPO, and CSR, while requiring only a single model in memory and one simple SFT loss [2411.14797].

Source: https://www.emergentmind.com/topics/negative-certainty-supervision-loss