Relevance-aware Multi-context Contrastive Decoding

• RMCD is a decoding method that modifies next-token predictions by contrasting model priors with multiple context-conditioned outputs to mitigate noisy or conflicting evidence.
• The approach employs logit manipulation using positive and negative context streams to recalibrate probabilities and enhance factual consistency.
• Empirical results show RMCD improves performance in open-domain QA, dialogue, and visual question answering by robustly integrating retrieval and parametric knowledge.

Relevance-aware Multi-context Contrastive Decoding (RMCD) denotes a class of training-free, inference-time decoding procedures in which a model’s next-token distribution is modified by contrasting multiple context-conditioned predictions and weighting those predictions by relevance, confidence, or both. In the literature, RMCD addresses a common failure mode of retrieval-augmented generation: large models often rely excessively on parametric knowledge, insufficiently integrate external context, or become distracted by noisy, stale, or conflicting evidence. RMCD-style methods have been studied in open-domain question answering, multi-document retrieval-augmented generation, spoken dialogue systems, retrieval-augmented visual question answering, and multimodal hallucination mitigation (Zhao et al., 2024, Kim et al., 14 Jan 2026, Li et al., 1 Jul 2026, Lee et al., 15 Jun 2026, Liang et al., 2024).

1. Conceptual emergence and problem setting

The immediate motivation for RMCD is the observation that LLMs and large vision-LLMs draw on two distinct knowledge sources: parametric knowledge stored in model parameters and non-parametric knowledge supplied by prompts, retrieved passages, dialogue history, or retrieved external documents. In open-domain question answering, standard open-book prompting often leaves the model to decide implicitly how much to trust context versus its own prior knowledge, which can yield factual inconsistencies or contextually unfaithful generations when the retrieved material is noisy, outdated, or adversarial (Zhao et al., 2024).

A closely related line of work studies noisy-context robustness in retrieval-augmented generation. In that setting, retrieved top-1 context may either contain the gold answer or fail to contain it, and classic context-aware contrastive decoding methods can become distracted by misleading context, especially when the model already knows the answer from parametric memory. This led to adaptive weighting schemes that estimate whether context actually reduces uncertainty at the current decoding step (Kim et al., 2024).

A second expansion of the problem concerns intra-context conflict rather than parametric-context conflict. In multi-document retrieval, the retrieved bundle may contain one correct source together with misinformation, temporal documents containing stale facts, and topically related noise. Here the central question is not merely whether to trust retrieval over memory, but which retrieved document should drive the next token. Dual-Confidence Contrastive Decoding formalizes this source-selection problem explicitly (Li et al., 1 Jul 2026).

The same structural issue appears in multimodal settings. In spoken dialogue, models may exhibit latent context awareness yet fail to adhere to relevant dialogue history during decoding, because strong parametric priors dominate the output distribution. In retrieval-augmented visual question answering, prior decoding strategies were found to be sub-optimal because they did not sufficiently leverage multiple relevant contexts or suppress irrelevant ones. The phrase “Relevance-aware Multi-context Contrastive Decoding” appears as the formal method name in retrieval-augmented visual question answering, while a 2024 text QA method already has the same core structure when framed as a relevance-aware multi-context procedure (Lee et al., 15 Jun 2026, Kim et al., 14 Jan 2026, Zhao et al., 2024). This suggests an umbrella concept rather than a single fixed algorithm.

2. Core decoding formulations

RMCD-style decoding is characterized by logit manipulation at inference time. A canonical text QA form uses three distributions at each decoding step $t$: a base no-context distribution with logits $z_t$, a relevant-context distribution with logits $z_t^+$, and an irrelevant-context distribution with logits $z_t^-$. The resulting token is sampled from

$$y_t \sim \mathrm{softmax}\Big(z_t + \alpha \,(z_t^+ - z_t^-)\Big),$$

with the equivalent probabilistic form

$$p_\theta(y_t \mid x, y_{<t}, c^+, c^-) \propto p_\theta(y_t \mid x, y_{<t}) \cdot \bigg( \frac{p_\theta(y_t \mid c^+, x, y_{<t})} {p_\theta(y_t \mid c^-, x, y_{<t})} \bigg)^\alpha.$$

In this formulation, tokens supported by relevant context and disfavored by irrelevant context are boosted, while tokens favored by irrelevant context are down-weighted. The method is purely inference-time and requires no additional training (Zhao et al., 2024).

A multi-context variant for retrieval-augmented visual question answering replaces the single positive-negative difference with a weighted sum over separately decoded contexts. If $q(y_i\mid c_j)$ denotes the context-specific logits for context $c_j$, RMCD defines

$$p(y_i) = \mathrm{Softmax}\Big(\sum_{j=1}^{|\mathcal{C}|} \alpha_j q(y_i\mid c_j)\Big),$$

where the $\alpha_j$ are signed context weights derived from retrieval relevance. Highly relevant contexts receive large positive weights, while low-relevance contexts can receive negative weights, producing an explicit reflect/deflect mechanism over multiple retrieved contexts (Kim et al., 14 Jan 2026).

In multi-document text RAG, Dual-Confidence Contrastive Decoding uses the full retrieved bundle as an anchor and then contrasts the most confident and least confident document-conditioned streams:

$z_t$0

with $z_t$1 and $z_t$2 selected by dual-confidence scores and $z_t$3 given by their confidence margin. This retains a full-context prior while tilting decoding toward the currently most useful source and away from the currently least useful one (Li et al., 1 Jul 2026).

Across these formulations, the invariant structure is a relevance-sensitive contrastive correction of the base distribution. RMCD therefore differs from ordinary greedy decoding not by changing the LLM itself, but by redefining which context-conditioned evidence is allowed to influence the next-token choice and with what sign.

3. Relevance estimation and context construction

The defining feature of RMCD is not merely multi-context decoding, but the use of explicit relevance signals to decide which contextual streams should be amplified, attenuated, or negated.

Setting	Context units	Relevance signal
Open-domain QA	$z_t$4, $z_t$5	retrieval rank, adversarial negatives, “most distant” passage
Noisy-context RAG	single retrieved context	entropy reduction $z_t$6
Multi-document RAG	per-document streams	document-level confidence $z_t$7, token-level confidence $z_t$8
Parallel context-of-experts RAG	per-document experts	fused retrieval and reranker score $z_t$9
Spoken dialogue	dialogue rounds	attention-derived key rounds
Retrieval-augmented VQA	retrieved contexts $z_t^+$0	retriever scores $z_t^+$1, relative scores $z_t^+$2, signed weights $z_t^+$3

In the original text QA construction, relevant context $z_t^+$4 is taken from the top retrieved passage from BM25 or Contriever, or from gold context in NQ and TriviaQA. Irrelevant context $z_t^+$5 is selected from several strategies: a random passage, a fixed adversarial paragraph, a permuted adversarial paragraph, or the passage whose embedding is most distant from the relevant passage in the retriever’s embedding space. The “most distant” strategy performed best, indicating that semantically distant negatives provide a stronger contrastive signal (Zhao et al., 2024).

Adaptive weighting can itself serve as an online relevance estimator. In Adaptive Contrastive Decoding, the context influence coefficient is

$z_t^+$6

where $z_t^+$7 is the entropy of the closed-book distribution and $z_t^+$8 is the entropy of the context-conditioned distribution. If context lowers entropy, $z_t^+$9 increases; if context raises entropy, it decreases. In Known-noisy versus Unknown-gold scenarios, this functions as a relevance or reliability proxy without any explicit document classifier (Kim et al., 2024).

In multi-document RAG, relevance is made more explicit. Dual-Confidence Contrastive Decoding computes a document-level sufficiency score $z_t^-$0 from a yes/no support probe and a token-level confidence $z_t^-$1 from top-$z_t^-$2 Dirichlet-entropy calculations, then combines them as $z_t^-$3. Parallel Context-of-Experts Decoding instead derives a fixed per-document relevance score $z_t^-$4 by harmonic-mean fusion of retrieval and reranking scores, and injects $z_t^-$5 into the document expert logits (Li et al., 1 Jul 2026, Corallo et al., 13 Jan 2026).

Spoken dialogue introduces a different relevance mechanism. Rather than retrieved passages, the context units are historical dialogue rounds. Relevance is estimated from internal cross-attention from the current user query to past rounds, with token-level scores aggregated to turns and then to rounds. The best configuration uses last_4 layers, mean aggregation, $z_t^-$6, and $z_t^-$7, after which contrastive decoding is performed between the full context and a version with the key rounds masked out (Lee et al., 15 Jun 2026).

Retrieval-augmented visual question answering operationalizes relevance directly from retriever scores. RMCD computes relative context scores

$z_t^-$8

and maps them to signed weights

$z_t^-$9

This makes highly relevant contexts positive contributors and low-relevance contexts negative contributors. It also defines a thresholded constraint context set $$y_t \sim \mathrm{softmax}\Big(z_t + \alpha \,(z_t^+ - z_t^-)\Big),$$0 for building an ensembled plausibility distribution before final decoding (Kim et al., 14 Jan 2026).

4. Empirical findings across domains

In open-domain text QA, RMCD-style decoding improves Exact Match in settings where retrieval and parametric memory conflict. On Natural Questions with LLaMA-2-70B, the reported scores are Reg-Op $$y_t \sim \mathrm{softmax}\Big(z_t + \alpha \,(z_t^+ - z_t^-)\Big),$$1, CAD $$y_t \sim \mathrm{softmax}\Big(z_t + \alpha \,(z_t^+ - z_t^-)\Big),$$2, Ours-F $$y_t \sim \mathrm{softmax}\Big(z_t + \alpha \,(z_t^+ - z_t^-)\Big),$$3, and Ours-D $$y_t \sim \mathrm{softmax}\Big(z_t + \alpha \,(z_t^+ - z_t^-)\Big),$$4. Negative-context selection also matters: for LLaMA-2-70B, Random $$y_t \sim \mathrm{softmax}\Big(z_t + \alpha \,(z_t^+ - z_t^-)\Big),$$5 gives NQ $$y_t \sim \mathrm{softmax}\Big(z_t + \alpha \,(z_t^+ - z_t^-)\Big),$$6, TQA $$y_t \sim \mathrm{softmax}\Big(z_t + \alpha \,(z_t^+ - z_t^-)\Big),$$7, PopQA $$y_t \sim \mathrm{softmax}\Big(z_t + \alpha \,(z_t^+ - z_t^-)\Big),$$8; Fixed adversarial $$y_t \sim \mathrm{softmax}\Big(z_t + \alpha \,(z_t^+ - z_t^-)\Big),$$9; Fixed adversarial + word permutation $$p_\theta(y_t \mid x, y_{<t}, c^+, c^-) \propto p_\theta(y_t \mid x, y_{<t}) \cdot \bigg( \frac{p_\theta(y_t \mid c^+, x, y_{<t})} {p_\theta(y_t \mid c^-, x, y_{<t})} \bigg)^\alpha.$$0; and “Most distant” $$p_\theta(y_t \mid x, y_{<t}, c^+, c^-) \propto p_\theta(y_t \mid x, y_{<t}) \cdot \bigg( \frac{p_\theta(y_t \mid c^+, x, y_{<t})} {p_\theta(y_t \mid c^-, x, y_{<t})} \bigg)^\alpha.$$1, the best setting. On the knowledge-conflict benchmark NQ-SUB, RMCD-style decoding yields the best EM for all reported models, including LLaMA-2-70B at $$p_\theta(y_t \mid x, y_{<t}, c^+, c^-) \propto p_\theta(y_t \mid x, y_{<t}) \cdot \bigg( \frac{p_\theta(y_t \mid c^+, x, y_{<t})} {p_\theta(y_t \mid c^-, x, y_{<t})} \bigg)^\alpha.$$2 versus $$p_\theta(y_t \mid x, y_{<t}, c^+, c^-) \propto p_\theta(y_t \mid x, y_{<t}) \cdot \bigg( \frac{p_\theta(y_t \mid c^+, x, y_{<t})} {p_\theta(y_t \mid c^-, x, y_{<t})} \bigg)^\alpha.$$3 for regular open-book decoding and $$p_\theta(y_t \mid x, y_{<t}, c^+, c^-) \propto p_\theta(y_t \mid x, y_{<t}) \cdot \bigg( \frac{p_\theta(y_t \mid c^+, x, y_{<t})} {p_\theta(y_t \mid c^-, x, y_{<t})} \bigg)^\alpha.$$4 for CAD (Zhao et al., 2024).

In noisy-context retrieval-augmented generation, adaptive weighting improves robustness. On TriviaQA with LLAMA2-7B, ACD obtains SubsetNoisy EM $$p_\theta(y_t \mid x, y_{<t}, c^+, c^-) \propto p_\theta(y_t \mid x, y_{<t}) \cdot \bigg( \frac{p_\theta(y_t \mid c^+, x, y_{<t})} {p_\theta(y_t \mid c^-, x, y_{<t})} \bigg)^\alpha.$$5, compared with CAD $$p_\theta(y_t \mid x, y_{<t}, c^+, c^-) \propto p_\theta(y_t \mid x, y_{<t}) \cdot \bigg( \frac{p_\theta(y_t \mid c^+, x, y_{<t})} {p_\theta(y_t \mid c^-, x, y_{<t})} \bigg)^\alpha.$$6, MICD$$p_\theta(y_t \mid x, y_{<t}, c^+, c^-) \propto p_\theta(y_t \mid x, y_{<t}) \cdot \bigg( \frac{p_\theta(y_t \mid c^+, x, y_{<t})} {p_\theta(y_t \mid c^-, x, y_{<t})} \bigg)^\alpha.$$7 $$p_\theta(y_t \mid x, y_{<t}, c^+, c^-) \propto p_\theta(y_t \mid x, y_{<t}) \cdot \bigg( \frac{p_\theta(y_t \mid c^+, x, y_{<t})} {p_\theta(y_t \mid c^-, x, y_{<t})} \bigg)^\alpha.$$8, and MICD$$p_\theta(y_t \mid x, y_{<t}, c^+, c^-) \propto p_\theta(y_t \mid x, y_{<t}) \cdot \bigg( \frac{p_\theta(y_t \mid c^+, x, y_{<t})} {p_\theta(y_t \mid c^-, x, y_{<t})} \bigg)^\alpha.$$9 $q(y_i\mid c_j)$0. On SubsetGold, ACD reaches $q(y_i\mid c_j)$1, compared with MICD$q(y_i\mid c_j)$2 at $q(y_i\mid c_j)$3. The same work reports that ACD’s entropy-based $q(y_i\mid c_j)$4 has higher AUROC than MICD$q(y_i\mid c_j)$5 for predicting context noisiness; for LLAMA2-7B on TriviaQA, first-token AUROC is $q(y_i\mid c_j)$6 for ACD versus $q(y_i\mid c_j)$7 for MICD$q(y_i\mid c_j)$8 (Kim et al., 2024).

When the retrieved bundle itself is internally conflicting, document-aware RMCD variants become most salient. On DRQA, DCCD consistently exceeds full-context and prior contrastive baselines. For Qwen3.5-2B, DRQA @5 improves from full $q(y_i\mid c_j)$9 to DCCD $c_j$0, and DRQA @10 improves from full $c_j$1 to DCCD $c_j$2. Similar gains hold for Qwen3.5-9B and Phi-3-medium, with the largest margins on DRQA rather than on standard multi-document QA benchmarks (Li et al., 1 Jul 2026).

In spoken dialogue, relevance-aware two-context contrastive decoding substantially raises Average Pass Ratio on Audio MultiChallenge. MiMo-Audio improves from $c_j$3 to $c_j$4, Qwen3-Omni from $c_j$5 to $c_j$6, and Kimi-Audio from $c_j$7 to $c_j$8. The same study reports that Whole-History CAD degrades MiMo-Audio to $c_j$9 APR, whereas attention-based key-context CAD reaches $$p(y_i) = \mathrm{Softmax}\Big(\sum_{j=1}^{|\mathcal{C}|} \alpha_j q(y_i\mid c_j)\Big),$$0 APR under fixed $$p(y_i) = \mathrm{Softmax}\Big(\sum_{j=1}^{|\mathcal{C}|} \alpha_j q(y_i\mid c_j)\Big),$$1 and $$p(y_i) = \mathrm{Softmax}\Big(\sum_{j=1}^{|\mathcal{C}|} \alpha_j q(y_i\mid c_j)\Big),$$2, indicating that relevance-aware context selection is crucial (Lee et al., 15 Jun 2026).

In retrieval-augmented visual question answering, RMCD consistently outperforms alternative decoding strategies across LVLMs. With InternVL-2.5 on InfoSeek, total accuracy is $$p(y_i) = \mathrm{Softmax}\Big(\sum_{j=1}^{|\mathcal{C}|} \alpha_j q(y_i\mid c_j)\Big),$$3 for Unconditional, $$p(y_i) = \mathrm{Softmax}\Big(\sum_{j=1}^{|\mathcal{C}|} \alpha_j q(y_i\mid c_j)\Big),$$4 for RAG, $$p(y_i) = \mathrm{Softmax}\Big(\sum_{j=1}^{|\mathcal{C}|} \alpha_j q(y_i\mid c_j)\Big),$$5 for SCD, $$p(y_i) = \mathrm{Softmax}\Big(\sum_{j=1}^{|\mathcal{C}|} \alpha_j q(y_i\mid c_j)\Big),$$6 for Concat, and $$p(y_i) = \mathrm{Softmax}\Big(\sum_{j=1}^{|\mathcal{C}|} \alpha_j q(y_i\mid c_j)\Big),$$7 for RMCD. On Encyclopedic-VQA with the same model, RMCD reaches $$p(y_i) = \mathrm{Softmax}\Big(\sum_{j=1}^{|\mathcal{C}|} \alpha_j q(y_i\mid c_j)\Big),$$8 for one-hop / full, exceeding RAG $$p(y_i) = \mathrm{Softmax}\Big(\sum_{j=1}^{|\mathcal{C}|} \alpha_j q(y_i\mid c_j)\Big),$$9, SCD $\alpha_j$0, and Concat $\alpha_j$1. On OK-VQA, FLMR improves from $\alpha_j$2 to $\alpha_j$3 when RMCD replaces the original decoding, and the method is reported to remain the best across the weakest to the strongest retrieval results (Kim et al., 14 Jan 2026).

5. Relation to adjacent decoding paradigms

RMCD inherits its general decoding logic from contrastive decoding, where generation is guided by the difference between a stronger “expert” distribution and a weaker “amateur” distribution, subject to a plausibility constraint. In foundational contrastive decoding, the contrastive score is the expert-amateur log-probability difference, and the plausibility constraint excludes tokens whose expert probability falls below an $\alpha_j$4-scaled threshold relative to the expert’s top token (Li et al., 2022).

Context-Aware Decoding transposes this idea from model contrast to context contrast by comparing the same model with and without context. The original open-domain QA analysis argues that CAD tends to privilege context and implicitly treat it as the gold source when there is conflict, whereas RMCD-style formulations instead use a base parametric prior plus positive and negative contexts, or multiple context streams, to balance parametric and contextual evidence more explicitly (Zhao et al., 2024).

Adaptive Contrastive Decoding retains the two-stream structure but makes the interpolation coefficient entropy-dependent. Relative to RMCD, ACD is best understood as a single-context adaptive precursor: it does not explicitly compute per-document relevance scores, yet it already treats uncertainty reduction as a token-wise proxy for context informativeness (Kim et al., 2024).

Parallel Context-of-Experts Decoding extends the setting from “context versus no context” to “document experts versus prior.” Each retrieved document becomes an isolated expert with its own KV cache, and decoding uses the retrieval-aware contrastive score

$\alpha_j$5

followed by max aggregation over experts. This is structurally close to RMCD, but it emphasizes parallel document experts and decode-time evidence aggregation rather than a single weighted sum of all context-conditioned logits (Corallo et al., 13 Jan 2026).

Dual-Confidence Contrastive Decoding addresses a complementary axis: source-aware selection inside a conflicting retrieved bundle. Its positive and negative streams are document-conditioned rather than “with context” versus “without context,” and its contrast strength is the confidence margin between the most and least useful documents. Spoken-dialogue CAD adds yet another axis by defining context not as retrieved documents but as attention-derived key rounds from dialogue history, turning context adherence into a two-view relevance-aware contrastive decoding problem (Li et al., 1 Jul 2026, Lee et al., 15 Jun 2026).

Taken together, these methods indicate that RMCD is best viewed as the point where three themes intersect: source separation, relevance-aware weighting, and contrastive token scoring. This suggests a broader typology in which the contrastive axis may be model scale, context presence, document identity, temporal reliability, or modality.

6. Limitations and directions of development

The most persistent limitation is computational cost. The three-way QA formulation requires one no-context, one relevant-context, and one irrelevant-context forward pass per token and is therefore roughly $\alpha_j$6 slower than normal greedy decoding. ACD roughly doubles inference cost, while DCCD and related per-document methods require $\alpha_j$7 streams per token. Pced mitigates prefill bottlenecks through precomputed document KV caches, but this shifts cost to storage; the reported figure is $\alpha_j$8GB for $\alpha_j$9 passages with Llama-3.1-8B (Zhao et al., 2024, Kim et al., 2024, Li et al., 1 Jul 2026, Corallo et al., 13 Jan 2026).

A second limitation is dependence on retrieval or relevance estimation quality. If the relevant context is absent, misranked, or assigned a low score, relevance-aware decoding can underweight the very source that should dominate. The text QA work explicitly reports sensitivity to retrieval quality; the adaptive RAG work notes smaller benefits under poor retrieval; Pced states that it cannot recover evidence that retrieval never found; and visual RMCD assumes that retriever scores correlate reasonably with true relevance (Zhao et al., 2024, Kim et al., 2024, Corallo et al., 13 Jan 2026, Kim et al., 14 Jan 2026).

A third limitation concerns task scope and evaluation coverage. Several studies are restricted to short-form question answering with greedy decoding, even when the mathematical construction could in principle be integrated with beam search, top-$z_t$00, or nucleus sampling. Spoken-dialogue RMCD depends on accurate attention-based key-round extraction, and DCCD can still be confidently wrong if an incorrect document receives both high document-level confidence and high token-level confidence (Zhao et al., 2024, Lee et al., 15 Jun 2026, Li et al., 1 Jul 2026).

The main future directions are consistent across the literature. These include other generative tasks such as summarization and hallucination mitigation, learned relevance predictors instead of heuristic negative selection, multiple positive and negative contexts, group-level or soft weighting over documents rather than hard min/max selection, and more efficient implementations through joint encoding, shared prefix computation, or approximate relevance estimation. This suggests a transition from simple context-versus-prior contrast toward fully source-aware, confidence-gated, and modality-aware decoding systems (Zhao et al., 2024, Kim et al., 2024, Li et al., 1 Jul 2026, Corallo et al., 13 Jan 2026, Kim et al., 14 Jan 2026).