Context-DPO: Context-Aware Preference Loss

• The paper introduces Context-DPO, a method that incorporates explicit external context into preference optimization to produce more context-faithful responses.
• It employs a specialized loss function comparing context-faithful and stubborn completions, validated across benchmarks like ConFiQA and NaturalQuestions.
• Empirical results show significant gains in context adherence with improvements up to +210%, illustrating robust integration of context in LLM output ranking.

Context-aware Preference Loss (C-DPO), formally introduced as Context-DPO, is an extension of Direct Preference Optimization (DPO) established to align LLMs explicitly with external context. Unlike standard preference optimization—which optimizes for criteria such as style or toxicity using human- or reward-model-annotated preference pairs—Context-DPO incorporates context into both the model input and the preference signal, directly encouraging context-faithful completions. This mechanism leverages a specialized loss function and is evaluated with systematic, conflict-rich benchmarks to ensure robust improvements in context adherence without degrading base generative capabilities (Bi et al., 2024).

1. Formal Definition and Objective

Let $\pi_\theta(y|x)$ denote a trainable LLM policy and $\pi_{\mathrm{ref}}(y|x)$ a frozen reference policy (typically the original pretrained model). Context-DPO’s training employs a preference dataset

$\mathcal{D} = \{(x, y_w, y_l)\}$

where $x$ is the concatenation of context $C$ and question $Q$; $y_w$ is the “winning” (context-faithful) response; $y_l$ is the “losing” (“stubborn”, context-ignoring or contradicting) response.

The Context-DPO loss, which closely mirrors standard DPO but uses explicitly context-contrasting preference pairs, is:

$$\mathcal{L}_{cf} = -\mathbb{E}_{(x, y_w, y_l)\sim \mathcal{D}} \left[ \log \sigma\left( \beta \cdot \left( \log \frac{\pi_\theta(y_w|x)}{\pi_{\mathrm{ref}}(y_w|x)} - \log \frac{\pi_\theta(y_l|x)}{\pi_{\mathrm{ref}}(y_l|x)} \right) \right) \right]$$

where $\sigma$ is the logistic sigmoid and $\pi_{\mathrm{ref}}(y|x)$0 is a temperature parameter controlling margin. Optimizing $\pi_{\mathrm{ref}}(y|x)$1 increases the relative likelihood of context-faithful outputs over stubborn ones, forcing $\pi_{\mathrm{ref}}(y|x)$2 to encode the effect of the explicit context in ranking possible completions.

2. Distinction from Standard DPO

Standard DPO, as introduced by Rafailov et al. (2024), operates over preference pairs that usually capture human judgments regarding generic qualities such as helpfulness or factuality, without ensuring that external context is a decisive part of the preference. In contrast, Context-DPO “plugs in” a structured input $\pi_{\mathrm{ref}}(y|x)$3—including not only instructions or prompts but also outside knowledge (e.g., retrieved passages)—and constructs preference tuples where the difference between $\pi_{\mathrm{ref}}(y|x)$4 and $\pi_{\mathrm{ref}}(y|x)$5 is strictly context faithfulness. No extra penalties or regularizers are added; context-awareness arises purely by introducing context into the input and structuring preferences to contrast obedience versus defiance to that context (Bi et al., 2024).

3. Counterfactual Benchmark Design: ConFiQA

Context-DPO’s efficacy depends on a high-fidelity evaluation of context utilization. To this end, the ConFiQA benchmark is introduced, comprising three task types:

• QA (single-hop)
• MR (multi-hop reasoning)
• MC (multi-conflict)

The pipeline constructs counterfactuals as follows:

• Sample facts $\pi_{\mathrm{ref}}(y|x)$6 from Wikidata to create original and counterfactual fact paths.
• Generate context $\pi_{\mathrm{ref}}(y|x)$7 (faithful) and $\pi_{\mathrm{ref}}(y|x)$8 (counterfactual, by entity substitution).
• Prompt GPT-4 to produce a question $\pi_{\mathrm{ref}}(y|x)$9, $\mathcal{D} = \{(x, y_w, y_l)\}$0, and $\mathcal{D} = \{(x, y_w, y_l)\}$1.
• Faithful response $\mathcal{D} = \{(x, y_w, y_l)\}$2 summarizes $\mathcal{D} = \{(x, y_w, y_l)\}$3; stubborn response $\mathcal{D} = \{(x, y_w, y_l)\}$4 summarizes the original factual path.

Because $\mathcal{D} = \{(x, y_w, y_l)\}$5 intentionally conflicts with the model’s likely parametric memory, these constructed preference pairs yield definitive preference signals for context adherence (Bi et al., 2024).

4. Training Configuration and Implementation

Context-DPO has been instantiated on open-source LLMs including Llama2-7B-chat, Llama3-8B-instruct, Mistral-7B-instruct, and Qwen2-7B-instruct. The reference policy $\mathcal{D} = \{(x, y_w, y_l)\}$6 is frozen to the base model, while $\mathcal{D} = \{(x, y_w, y_l)\}$7 is finetuned using AdamW with batch size $\mathcal{D} = \{(x, y_w, y_l)\}$8, gradient accumulation of $\mathcal{D} = \{(x, y_w, y_l)\}$9, and model-specific tuning of $x$0. Training is performed on datasets covering:

• Retrieval following (ConFiQA, NaturalQuestions)
• Instruction following (in-context editing on MQuAKE)

Baseline comparisons include prompt-based Attribution (Attr) and “O&I” methods, as well as supervised fine-tuning (SFT) on $x$1 pairs.

5. Empirical Results: Performance and Generalization

Experimental results on ConFiQA demonstrate that Context-DPO yields substantial relative gains in context faithfulness—quantified as $x$2, the proportion of completions correctly grounded in the provided counterfactual context. Notable improvements include:

Model	$x$3 (Baseline)	$x$4 (C-DPO)	Relative Gain
Llama2-7B-chat	61.5%	92.3%	+50%
Llama3-8B-instruct	35.8%	69.7%	+94%
Mistral-7B-instruct	39.3%	78.6%	+100%
Qwen2-7B-instruct	24.0%	74.3%	+210%

On NaturalQuestions, where context conflicts are weaker, overall accuracy surpasses 93%. Instruction-following on MQuAKE records consistent improvements (up to 20 percentage points). TruthfulQA scores remain nearly unchanged (within $x$5), establishing that enhancing context faithfulness via C-DPO does not degrade “unconditional” factuality (Bi et al., 2024).

6. Mechanistic Interpretability of Context Utilization

Token-level attribution analyses reveal that, post-training, the average logit for context-specific “key tokens” (those encoding the counterfactual knowledge, e.g., “South America” vs. “Europe”) increases by $x$6–$x$7 points, with the token probability density shifting toward higher values for faithful tokens. Rank histograms indicate more context-faithful tokens populating top-k predictions. This redistribution occurs throughout the model’s token distribution, rather than as a post-hoc heuristic in the final layer, implying that learned context sensitivity is realized deep within the model’s preference structure (Bi et al., 2024).

7. Significance, Scope, and Position in Context-Aware Alignment

Context-DPO is the first alignment method explicitly targeting context-faithfulness in LLMs by constructing training objectives and data such that context determines preference rewards. Unlike other alignment approaches (e.g., semantic regularization in Sem-DPO (Mohamed et al., 27 Jul 2025)), Context-DPO’s innovation is in treating context as foundational to both model input and optimization signal, without ad-hoc penalty terms or semantic similarity constraints. Its straightforward integration with standard DPO frameworks, robust quantitative gains across architectures and benchmarks, and mechanistically interpretable behavior mark it as a canonical approach for advancing context-following behavior in LLMs. This approach sets a precedent for subsequent context-centric alignment and evaluation methodologies in retrieval-augmented and editing-intensive NLP systems.