---
title: 'LayerCD: Mitigating Hallucinations in MLLMs'
url: https://www.emergentmind.com/topics/layer-contrastive-decoding-layercd
type: topic
---

# LayerCD: Mitigating Hallucinations in MLLMs

Layer Contrastive Decoding (LayerCD) is a decoding-time method for mitigating hallucinations in Multimodal Large Language Models (MLLMs) by contrasting the output distributions generated from visual features of different levels, specifically those from the shallow and deep layers of the vision encoder, respectively. In the formulation introduced in "Mitigating Hallucination in Multimodal LLMs with Layer Contrastive Decoding" [2509.25177], the method is motivated by the observation that shallow visual features are much more likely than deep visual features to cause an MLLM to hallucinate, because they only capture biased, low-level information that is insufficient for high-level reasoning, whereas deep visual features provide more abstract, semantically aligned representations. LayerCD is inference-only, model-agnostic, requires no parameter retraining or LLM modification, and is designed to filter out hallucinations that are supported by shallow features alone but not by deep, more accurate ones [2509.25177].

## 1. Conceptual basis

Hallucinations in MLLMs are instances where outputs, though linguistically plausible, do not faithfully reflect the content of the input image. The LayerCD paper characterizes these failures as including inaccuracies in objects, attributes, and relations. Its central claim is that the depth of the visual representation materially affects hallucination behavior: shallow visual features extracted from early layers of a vision encoder mostly encode low-level information such as edges, local color, and textures, but lack the semantic richness needed for high-level image understanding, while deep visual features from the final layers provide more abstract, semantically aligned representations [2509.25177].

The paper further argues that when MLLMs condition their generation more on shallow features, hallucinations are more likely, since these features are easily dominated by dataset biases and fail to ground the generated tokens in concrete visual content. Figure 1, as summarized in the paper, empirically demonstrates that using only shallow features leads to markedly higher hallucination error rates. This establishes the core rationale for LayerCD: if one contrasts the output distributions produced under shallow and deep visual conditioning, then tokens favored by shallow features but not supported by deep features can be downweighted during generation [2509.25177].

A common source of ambiguity is terminological. In the broader literature, “layer contrastive decoding” can refer to several related procedures that contrast predictions across internal layers. In the specific sense of LayerCD, however, the contrast is not between two language-model layers of the same text-only model; it is between shallow and deep visual features supplied to an MLLM. That distinction is central to the method’s multimodal grounding objective.

## 2. Decoding formulation

LayerCD operates at each generation step by running the MLLM twice: once conditioned on shallow features $\bm{z}_s$ from an early encoder layer and once conditioned on deep features $\bm{z}_d$ from the final layer. Let $\bm{x}$ denote the textual prompt, $\bm{y}_{<t}$ the previously generated tokens, $f(\bm{x}, \bm{z})$ the logits from the LLM head given visual features $\bm{z}$, and $\alpha$ the contrastive amplification hyperparameter. The LayerCD contrastive next-token distribution is defined as

\[
p(\bm{y}|\bm{x}, \bm{z}_d, \bm{z}_s) = \sigma \big[ (1+\alpha)\, f(\bm{x}, \bm{z}_d) - \alpha\, f(\bm{x}, \bm{z}_s) \big]
\]

where $\sigma$ denotes the softmax [2509.25177].

This formulation has a direct interpretation. When $\alpha = 0$, the procedure reduces to normal decoding using deep features only. As $\alpha$ increases, the difference between the two distributions is exaggerated. Tokens favored by shallow features but less supported by deep features are downweighted, which is intended to filter hallucinations. The method therefore treats the shallow-feature-conditioned trajectory as a source of negative evidence and the deep-feature-conditioned trajectory as the grounding-preserving path [2509.25177].

The paper presents LayerCD as a simple approach, but the simplicity is specific: the contrast is computed at token level, yet the intervention is defined by the depth of the visual representation. This makes the mechanism distinct from approaches that degrade the input image or remove visual tokens entirely. Compared to Visual Contrastive Decoding (VCD), LayerCD needs only one vision encoder forward, further reducing compute, while still using two LLM-conditioned passes for $\bm{z}_s$ and $\bm{z}_d$ [2509.25177].

## 3. Adaptive plausibility constraint and decoding procedure

Because contrastive subtraction can undesirably penalize universally likely tokens, the method introduces an Adaptive Plausibility Constraint (APC). The paper gives function words such as “a” and “the” as examples of tokens that are likely irrespective of visual grounding and should not be aggressively suppressed. To address this, LayerCD restricts the effective candidate set to tokens that remain plausible under the deep-feature-conditioned distribution [2509.25177].

The constraint set is defined as

\[
\mathcal{V}_{\text{head}(\bm{y}_{<t})} = \left\{ y_t \in \mathcal{V} :
    p(y_t | \bm{x}, \bm{z}_d, \bm{y}_{<t}) \geq
    \beta \max_{w} p(w | \bm{x}, \bm{z}_d, \bm{y}_{<t}) \right\}
\]

where $\mathcal{V}$ is the vocabulary and $\beta \in [0,1]$ is a confidence threshold [2509.25177].

Operationally, the method proceeds in four steps at each decoding step: extract $\bm{z}_s$ from the first encoder layer and $\bm{z}_d$ from the last layer; compute LLM logits for both; form the contrastive distribution using the equation above; apply plausibility filtering through APC; and then sample or select the output token using the chosen decoding strategy [2509.25177]. The paper states that LayerCD is orthogonal to other decoding methods and can be combined with greedy decoding, beam, top-k, nucleus sampling, and related strategies.

The ablation results summarized in the paper make the role of APC explicit. Disabling the Adaptive Plausibility Constraint significantly degrades LayerCD’s effectiveness, with accuracy dropping from 85.67% with APC to 62.21% without it. The same ablations report that intermediate $\beta$ performs best, whereas a cutoff that is too low can let in spurious tokens and a cutoff that is too high can overly reduce diversity [2509.25177].

## 4. Empirical evaluation

LayerCD was evaluated on two hallucination benchmarks: POPE and MME. POPE evaluates object hallucination via yes/no questions about object presence on MSCOCO, A-OKVQA, and GQA images, and reports accuracy, precision, recall, and F1. MME includes subsets for existence, count, position, and color, thereby covering object and attribute hallucinations. The models used were LLaVA-v1.5-7B, Cambrian-8B, and MoLmo-7B-D, and the principal baselines were regular decoding and VCD [2509.25177].

| Benchmark or setting | Reported finding | Source |
|---|---|---|
| POPE, LLaVA1.5, random | Accuracy 85.77 vs 82.33 for VCD vs 83.21 for Regular; F1 83.93 vs 79.36 vs 81.20 | [2509.25177] |
| POPE across random, popular, adversarial | LayerCD consistently yields higher accuracy, precision, recall, and F1 than both VCD and regular decoding | [2509.25177] |
| MME | LayerCD outperforms both VCD and regular decoding on most hallucination subsets | [2509.25177] |
| MME exception | For Molmo on count/color, LayerCD underperforms VCD, attributed to a high bias toward “No” answers | [2509.25177] |

Across POPE’s random, popular, and adversarial sampling settings, LayerCD consistently yields higher accuracy, precision, recall, and F1 than both VCD and regular decoding. Similar trends are reported for Cambrian and Molmo models, and across all sampling types. On MME, LayerCD outperforms both VCD and regular decoding on most hallucination subsets, though the paper notes an exception for Molmo on count and color, where LayerCD underperforms VCD, attributed to a high bias toward “No” answers [2509.25177].

The paper also reports several ablation findings. Increasing the contrastive amplification $\alpha$ generally improves performance up to a point, with the best result at $\alpha = 1.0$ in POPE random for LLaVA1.5. It further states that when LayerCD is combined with standard decoding strategies such as greedy, beam search, top-k/p, and temperature sampling, it consistently boosts performance, indicating orthogonality to the base decoding policy [2509.25177].

## 5. Relation to adjacent research

LayerCD takes inspiration from contrastive decoding in text-only LLMs and extends the idea to MLLMs by contrasting different levels of visual features rather than input perturbations [2509.25177]. A relevant precursor is "The Benefits of Bad Advice: Autocontrastive Decoding across Model Layers" [2305.01628], which uses the contrast between higher and lower layers inside a single language model during inference. That work frames the final layer as the “expert” and an intermediate layer as the “amateur,” and uses the lower-layer prediction to highlight which candidates are best avoided. The resemblance is structural, but the contrastive axis differs: ACD is intra-language-model, whereas LayerCD is based on visual feature depth.

Subsequent work diversified the source of the contrastive signal. "VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decoding" [2411.15839] shifts the focus from shallow-versus-deep feature contrast to uncertainty-guided visual layer selection and fusion. "YARD: Y-Architecture Register Decoding for Efficient Hallucination Mitigation in Large Vision-Language Models" [2605.31429] constructs the degraded branch internally by branching at a middle decoder layer and replacing patch-level visual tokens with register tokens, thereby avoiding a full second forward pass. "Watermarking for Factuality: Guiding Vision-Language Models Toward Truth via Tri-layer Contrastive Decoding" [2510.14304] introduces mature, amateur, and watermark-selected visually grounded layers into a tri-layer contrastive scheme.

In text-only generation, the family of layer-aware decoding methods expanded further. "Active Layer-Contrastive Decoding Reduces Hallucination in Large Language Model Generation" [2505.23657] casts the decision of when to apply contrasting layers as a sequential decision-making problem and learns a policy via reinforcement learning. "LayerCake: Token-Aware Contrastive Decoding within Large Language Model Layers" [2507.04404] uses token-aware, layer-localized attention suppression to induce controlled factual degradation and derive contrastive signals. "Multilingual Contrastive Decoding via Language-Agnostic Layers Skipping" [2407.10795] addresses a language mismatch problem in non-English tasks by skipping a span of bottom, language-agnostic layers.

Taken together, these works indicate that “layer contrastive decoding” has become a family of methods rather than a single algorithmic template. Within that family, LayerCD is specifically the multimodal method that contrasts shallow and deep visual features to suppress hallucinations at inference time [2509.25177].

## 6. Limitations, applications, and significance

The principal limitation reported for LayerCD is computational overhead. Because it requires two LLM forward passes per generation step for $\bm{z}_s$ and $\bm{z}_d$, it doubles inference cost, and the paper states that this may hinder deployment for very large models [2509.25177]. The method’s failure cases are also explicit: in some attribute-focused tasks, such as color and count on MME for Molmo, LayerCD can exacerbate already poor bias in the underlying model.

At the same time, the paper emphasizes that LayerCD is inference-only and architecture-agnostic, requiring no parameter retraining or LLM modification. It is presented as a plug-in hallucination suppression module for any MLLM supporting feature extraction at multiple encoder depths, and as particularly suitable for high-stakes image question answering, captioning, and general VQA where hallucination undermines trustworthiness [2509.25177].

A plausible implication is that LayerCD helped sharpen a broader research question: whether hallucination in multimodal generation should be addressed by perturbing inputs, by selecting or fusing internal visual layers, or by constructing degraded branches inside the decoder. Later proposals such as VaLiD and YARD suggest that visual grounding errors can be attacked from multiple internal loci, but LayerCD’s specific contribution lies in demonstrating that contrast between shallow and deep vision encoder features is itself a strong and general-purpose signal for hallucination mitigation [2411.15839].

Source: https://www.emergentmind.com/topics/layer-contrastive-decoding-layercd