Negative Certainty Supervision Loss

• Negative Certainty Supervision Loss is a training paradigm for vision-language models that repurposes rejected responses into corrective dialogue.
• It leverages a standard cross-entropy loss on both chosen responses and LLM-generated corrective conversations, simplifying complex RLHF pipelines.
• Experimental results on OCRVQA, TextCaps, and LLaVA-150K show that nSFT achieves competitive performance with reduced computational overhead.

Searching arXiv for the cited paper and related context. arxiv_search(query="(Zhu et al., 2024) Continual SFT Matches Multimodal RLHF with Negative Supervision", max_results=5, sort_by="relevance") Negative supervised finetuning (nSFT) is a continual alignment objective for vision-LLMs (VLMs) in which the supervision signal consists of two standard supervised-finetuning terms: one on the chosen response and one on a constructed corrective dialogue derived from a rejected response. In “Continual SFT Matches Multimodal RLHF with Negative Supervision,” nSFT is presented as a way to disentangle the negative supervision embedded in multimodal RLHF, especially the logit of rejected responses, and to recover that signal within a simple cross-entropy training regime rather than with PPO- or DPO-style preference optimization (Zhu et al., 2024). The method is framed as a memory-efficient alternative to multimodal RLHF, where 2 (e.g., DPO) or 4 (e.g., PPO) large VLMs are strictly required, while nSFT requires only a single model in memory and one simple SFT loss (Zhu et al., 2024).

1. Formal definition and notation

The method is defined on a multimodal input $x=(I,q)$, where $I$ is an image and $q$ denotes question tokens. The target chosen response is $y_c=(y_{c,1},\dots,y_{c,L})$, and the rejected response is $y_r=(y_{r,1},\dots,y_{r,L'})$, generated by the current model, for example via temperature sampling. The underlying policy is the vision-LLM $\pi_\theta$ to be finetuned (Zhu et al., 2024).

The standard supervised-finetuning token-wise log-loss on a target sequence $y$ is

$$L_{\mathrm{sft}}(y;\theta)=-\sum_{i=1}^{|y|}\log \pi_\theta(y_i\mid y_{<i},x).$$

A construction function $G(\cdot)$, implemented as a large LLM, e.g. GPT-4, takes the rejected reply $y_r$ together with the chosen response $I$0 and a vision-error codebook $I$1, and produces a corrective conversation $I$2 of length $I$3 (Zhu et al., 2024).

The nSFT loss is

$I$4

In the paper’s formulation, the “negative-response logits” are extracted by including $I$5 in the cross-entropy training data rather than by introducing a separate preference-classification objective (Zhu et al., 2024).

2. Complete objective and optimization procedure

In a batched setting with preference dataset $I$6, the training loop first samples a rejected response $I$7 from the current policy $I$8 for each input $I$9, then constructs the corrective dialogue $q$0. The overall objective is

$q$1

No auxiliary KL term is required, unlike PPO or DPO, although a per-token KL-penalty may optionally be added to stabilize training; this possibility is examined in ablations (Zhu et al., 2024).

The step-by-step algorithm is specified as follows. For each minibatch $q$2 of size $q$3, one samples $q$4 from the alignment dataset $q$5. For each $q$6, a rejected reply $q$7 is generated by sampling $q$8, for example with top-k+temperature. The LLM $q$9 is then prompted with $y_c=(y_{c,1},\dots,y_{c,L})$0 to identify instance- and image-level hallucinations and to output a corrective conversation $y_c=(y_{c,1},\dots,y_{c,L})$1. Two target sequences are thus formed per example, $y_c=(y_{c,1},\dots,y_{c,L})$2 and $y_c=(y_{c,1},\dots,y_{c,L})$3, and the losses $y_c=(y_{c,1},\dots,y_{c,L})$4 and $y_c=(y_{c,1},\dots,y_{c,L})$5 are computed. The batch loss is

$y_c=(y_{c,1},\dots,y_{c,L})$6

The model is then updated by backpropagating $y_c=(y_{c,1},\dots,y_{c,L})$7 and taking an SGD/Adam step (Zhu et al., 2024).

The significance of this formulation is that the alignment stage remains an SFT-style optimization problem. A plausible implication is that implementation complexity is reduced relative to RLHF pipelines that rely on reference models, value models, or rollout machinery, although the paper’s direct claims concern memory and runtime rather than software complexity.

3. Relation to DPO and the role of negative supervision

The paper motivates nSFT by arguing that the inherent value of multimodal RLHF lies in its negative supervision, specifically the logit of the rejected responses (Zhu et al., 2024). This argument is made explicit through a comparison with Direct Preference Optimization (DPO), whose binary log-loss is written as

$y_c=(y_{c,1},\dots,y_{c,L})$8

Ignoring reference-model terms, the logit inside $y_c=(y_{c,1},\dots,y_{c,L})$9 can be expressed as

$y_r=(y_{r,1},\dots,y_{r,L'})$0

By differentiating $y_r=(y_{r,1},\dots,y_{r,L'})$1 with respect to $y_r=(y_{r,1},\dots,y_{r,L'})$2, the paper reports

$y_r=(y_{r,1},\dots,y_{r,L'})$3

This is interpreted as a weighted combination of a positive and a negative SFT gradient, whereas naïve continual SFT uses only $y_r=(y_{r,1},\dots,y_{r,L'})$4 and thus loses all negative supervision (Zhu et al., 2024).

A second theoretical argument is a gradient-bias analysis. From the partial derivatives of $y_r=(y_{r,1},\dots,y_{r,L'})$5 with respect to the two logits $y_r=(y_{r,1},\dots,y_{r,L'})$6 and $y_r=(y_{r,1},\dots,y_{r,L'})$7, and again ignoring the reference model, the paper states that

$y_r=(y_{r,1},\dots,y_{r,L'})$8

under standard preference pairs $y_r=(y_{r,1},\dots,y_{r,L'})$9. The reported interpretation is that DPO training emphasizes decreasing the negative logit faster than boosting the positive one, and that nSFT recovers both sides of this signal in an explicit SFT form (Zhu et al., 2024).

This comparison directly addresses a common assumption stated in the paper: conventional wisdom holds the superiority of multimodal RLHF over continual SFT during the preference alignment stage. The paper’s position is narrower than a blanket rejection of RLHF; it argues specifically that the critical missing component in continual SFT is the negative supervision term, not that all RLHF machinery is intrinsically unnecessary in every setting.

4. Constructive negative supervision and corrective dialogue

A distinctive element of nSFT is that the negative term is not the raw rejected sequence itself. Instead, the rejected response is transformed by the construction function $\pi_\theta$0 into a corrective conversation conditioned on the rejected response, the chosen response, and the vision-error codebook $\pi_\theta$1 (Zhu et al., 2024).

The training procedure specifies that $\pi_\theta$2 identifies instance- and image-level hallucinations. The resulting output $\pi_\theta$3 becomes a second supervised target paired with the same multimodal input. In the paper’s summary, this is described as extracting and converting hallucinations into corrective dialogue, thereby recasting the rejected-response gradient into a standard cross-entropy term (Zhu et al., 2024).

The vision-error codebook $\pi_\theta$4 is not an incidental component. In ablations on a 5 K LLaVA-150 K subset, full nSFT yields MMB $\pi_\theta$5, SQA $\pi_\theta$6, MME $\pi_\theta$7, and POPE $\pi_\theta$8. Removing $\pi_\theta$9 degrades performance to MMB $y$0, SQA $y$1, MME $y$2, and POPE $y$3. Removing $y$4 while retaining the rest yields MMB $y$5, SQA $y$6, MME $y$7, and POPE $y$8 (Zhu et al., 2024).

The paper’s conclusion from these ablations is explicit: the $y$9-guided negative dialogue is crucial, and including $$L_{\mathrm{sft}}(y;\theta)=-\sum_{i=1}^{|y|}\log \pi_\theta(y_i\mid y_{<i},x).$$0 is helpful on very short dialogues (OCRVQA). This suggests that the effectiveness of the negative term depends not only on exposing the model to a non-preferred response pattern, but on structuring that pattern as an error-aware corrective sequence rather than as a bare penalty target.

5. Experimental regime and quantitative comparisons

The experimental setup uses three alignment datasets, each with 10 K examples: OCRVQA, described as object-centric book covers with very short answers; TextCaps, described as OCR + captioning with medium length; and an LLaVA-150 K subset, described as GPT-4 captions with long length $$L_{\mathrm{sft}}(y;\theta)=-\sum_{i=1}^{|y|}\log \pi_\theta(y_i\mid y_{<i},x).$$1 tokens (Zhu et al., 2024). The base vision-LLM is LLaVA-1.5-7B with a Vicuna-7B backbone, with ablations on LLaVA-1.5-13B and LLaVA-NeXT-13B. Hyperparameters for nSFT, continual SFT, and DPO are Deepspeed ZeRO-3, batch size $$L_{\mathrm{sft}}(y;\theta)=-\sum_{i=1}^{|y|}\log \pi_\theta(y_i\mid y_{<i},x).$$2, learning rate $$L_{\mathrm{sft}}(y;\theta)=-\sum_{i=1}^{|y|}\log \pi_\theta(y_i\mid y_{<i},x).$$3 with cosine scheduler, weight decay $$L_{\mathrm{sft}}(y;\theta)=-\sum_{i=1}^{|y|}\log \pi_\theta(y_i\mid y_{<i},x).$$4, and training steps $$L_{\mathrm{sft}}(y;\theta)=-\sum_{i=1}^{|y|}\log \pi_\theta(y_i\mid y_{<i},x).$$5 K per dataset (Zhu et al., 2024).

Evaluation covers nine tasks. Traditional VQA comprises SQA, GQA, and TextVQA ($$L_{\mathrm{sft}}(y;\theta)=-\sum_{i=1}^{|y|}\log \pi_\theta(y_i\mid y_{<i},x).$$6). Multimodal comprehension uses MMVet, MME, and MMBench. Hallucination uses POPE, CHAIR, and MMHal, where MMHal is rated $$L_{\mathrm{sft}}(y;\theta)=-\sum_{i=1}^{|y|}\log \pi_\theta(y_i\mid y_{<i},x).$$7–$$L_{\mathrm{sft}}(y;\theta)=-\sum_{i=1}^{|y|}\log \pi_\theta(y_i\mid y_{<i},x).$$8 by GPT. In-domain evaluation reports “IF score” and “accuracy” by GPT-4 on held-out $$L_{\mathrm{sft}}(y;\theta)=-\sum_{i=1}^{|y|}\log \pi_\theta(y_i\mid y_{<i},x).$$9 per dataset, together with $G(\cdot)$0 and $G(\cdot)$1 (Zhu et al., 2024).

On the main comparison in which training uses 10 K examples from each source and “total” is the sum of per-category improvements, the reported results are: Baseline, $G(\cdot)$2, total $G(\cdot)$3; GT-DPO, $G(\cdot)$4, total $G(\cdot)$5; SeVa (DPO), $G(\cdot)$6, total $G(\cdot)$7; SIMA (DPO), $G(\cdot)$8, total $G(\cdot)$9; Cont. SFT, $y_r$0, total $y_r$1; and nSFT (ours), $y_r$2, total $y_r$3 (Zhu et al., 2024). The category ordering in the paper is $y_r$4, MM comprehension, and Hallucination.

The paper also reports a direct comparison with PPO and iterative DPO, labeled CSR $y_r$5, on the 10 K LLaVA-150 K subset. Baseline obtains MMB $y_r$6, SQA $y_r$7, MME $y_r$8, and POPE $y_r$9. PPO yields $I$00. CSR 1–3 gives MMB $I$01–$I$02, SQA $I$03–$I$04, MME $I$05–$I$06, and POPE $I$07–$I$08. nSFT yields MMB $I$09, SQA $I$10, MME $I$11, and POPE $I$12 (Zhu et al., 2024).

For in-domain evaluation on held-out 500 examples per dataset, the reported values are as follows. Baseline: IF score $I$13, accuracy $I$14, $I$15 $I$16, $I$17 $I$18. SeVa: IF score $I$19, accuracy $I$20, $I$21 $I$22, $I$23 $I$24. GT-DPO: IF score $I$25, accuracy $I$26, $I$27 $I$28, $I$29 $I$30. nSFT: IF score $I$31, accuracy $I$32, $I$33 $I$34, $I$35 $I$36 (Zhu et al., 2024).

These quantitative results are used in the paper to support the claim that continual SFT can match multimodal RLHF once negative supervision is incorporated, and in several settings exceed DPO-, PPO-, and CSR-style baselines.

6. Efficiency claims, KL ablations, and interpretive scope

The efficiency claim is precise. On a single run on a 32 GB GPU, DPO, which uses two models, policy plus reference, or PPO, which uses policy, value, and rollouts, requires 2–4× more GPUs and 1.5–2× longer wall-clock time than nSFT, which uses a single model (Zhu et al., 2024). This is the paper’s main systems-level argument for preferring the nSFT formulation when the aim is to recover negative supervision without the overhead of full RLHF.

The KL ablation is reported on 5 K TextCaps. Continual SFT without KL gives MMB $I$37, $I$38 $I$39, and SQA $I$40; with KL it gives $I$41, $I$42, and $I$43. nSFT without KL gives $I$44, $I$45, and $I$46; with KL it gives $I$47, $I$48, and $I$49. DPO (SeVa) with KL gives MMB $I$50, $I$51 $I$52, and SQA $I$53 (Zhu et al., 2024). The paper’s conclusion is that a mild KL-constraint further boosts both SFT and nSFT.

Two interpretive points follow directly from the reported evidence. First, the results are not presented as a claim that ordinary continual SFT is sufficient; in the main comparison, Cont. SFT has total $I$54, with negative changes in both $I$55 and Hallucination (Zhu et al., 2024). Second, the paper does not claim that the rejected sequence alone is enough; the ablations indicate that the $I$56-guided corrective dialogue is crucial. A plausible implication is that the central design choice is not simply “train on bad outputs,” but “convert model-generated failure cases into structured negative supervision.”

In summary form, the paper states that nSFT recasts the “negative-supervision” half of RLHF, namely the rejected-response gradient, into a standard cross-entropy term by prompting an LLM to extract and convert hallucinations into corrective dialogue. Within the reported experiments, this achieves parity with—and often exceeds—the performance of heavyweight multimodal RLHF methods such as DPO, PPO, and CSR, while requiring only a single model in memory and one simple SFT loss (Zhu et al., 2024).