SAVE: Sparse Autoencoder-Driven Visual Enhancement

• SAVE is a training-free framework that leverages a sparse autoencoder to steer latent visual features and mitigate object hallucination in MLLMs.
• It identifies a key visual understanding feature through binary object-presence probing, thereby reducing reliance on misleading language priors.
• SAVE achieves significant performance gains on benchmarks, including a 10% point improvement in CHAIR_S and consistent improvements on POPE and MMHal-Bench.

SAVE, short for Sparse Autoencoder-Driven Visual Information Enhancement, is a training-free framework for mitigating object hallucination in Multimodal LLMs (MLLMs). It addresses hallucination as a consequence of language priors and visual information loss by identifying Sparse Autoencoder (SAE) latent directions that are most indicative of grounded visual processing and then steering the model along those directions during inference. In the reported evaluations, SAVE outperforms state-of-the-art training-free methods on standard benchmarks, including a 10\%p improvement in CHAIR_S together with consistent gains on POPE and MMHal-Bench (Park et al., 8 Dec 2025).

1. Problem setting and central idea

Object hallucination in MLLMs refers to the generation of object mentions that are not supported by the image. SAVE is motivated by two causes stated explicitly: visual information loss, where intermediate representations may under-attend or suppress crucial image features, and language priors, where the LLM’s statistical tendencies dominate when visual evidence is weak (Park et al., 8 Dec 2025).

Within this framing, hallucination is treated as a systematic failure mode rather than a random error. SAVE is designed to diagnose which internal latent directions encode genuine visual understanding and which correlate with hallucination, and then to selectively boost the former. The framework is therefore not based on additional supervised fine-tuning of the base MLLM, but on inference-time intervention in its internal hidden states (Park et al., 8 Dec 2025).

The method has three core stages. First, an SAE is trained on hidden representations from chosen transformer layers. Second, a binary object-presence question-answering probe is used to identify the SAE latent feature most associated with correct visual grounding, termed the visual understanding feature. Third, the model is steered along that feature’s decoder direction during inference, with the goal of reinforcing grounded visual understanding and suppressing hallucinated object generation (Park et al., 8 Dec 2025).

2. Sparse autoencoder formulation

SAVE builds on a two-layer sparse autoencoder inserted into chosen transformer layers of an MLLM. Given a hidden representation

$x \in \mathbb{R}^{T\times d_n},$

where $T$ is the number of tokens and $d_n$ the model hidden size, the SAE computes

$$a(x) = \mathrm{TopK}[ \mathrm{ReLU}( W_{\mathrm{enc}} \cdot(x - b_{\mathrm{pre}}) + b_{\mathrm{enc}} ) ]$$

and

$$\mathrm{SAE}(x) = W_{\mathrm{dec}} \cdot a(x) + b_{\mathrm{dec}}.$$

Here $a(x)\in\mathbb{R}^{T\times d_M}$ is a sparse activation in which only the top $k$ entries per token survive, with $d_M\gg d_n$, and each row of $W_{\mathrm{dec}}$ is interpreted as a latent feature direction (Park et al., 8 Dec 2025).

The training objective is

$$L_{\mathrm{SAE}}(x) = \| x - \mathrm{SAE}(x)\|_2^2 + \lambda \| a(x)\|_0.$$

The reconstruction term enforces fidelity to the original hidden representation, while the sparsity term encourages only a few latent features to activate per token. The hyperparameter $T$0 balances reconstruction fidelity against monosemantic sparsity (Park et al., 8 Dec 2025).

The reported SAE training setup uses $T$1 M image-caption pairs such as ShareGPT4V, optimized with Adam, with density scheduling of $T$2 from $T$3 value over initial steps. For any activation $T$4, the vector $T$5 constitutes a sparse code whose nonzero indices identify which latent features are active. The stated hyperparameters include Top-k in SAE encoder: $T$6, and SAE sparsity $T$7: typically increased linearly to 5 over early steps (Park et al., 8 Dec 2025).

3. Identification of visual understanding features

To isolate SAE features that reflect genuine visual grounding, SAVE uses a balanced set of 10 K yes/no queries, comprising 5 K positive cases in which the object is present and 5 K negative cases in which the object is absent. The model is asked a binary question of the form “Is there a … in this image?”, and the response is labeled as correct or hallucinated (Park et al., 8 Dec 2025).

Let $T$8 and $T$9 denote the sets of SAE activations $d_n$0 associated with correct and hallucinated answers. For each feature index $d_n$1, SAVE defines activation frequencies

$d_n$2

and

$d_n$3

The separation score is then

$d_n$4

The feature with the largest positive $d_n$5 is defined as the visual understanding feature, whereas reversing the difference identifies a top hallucination feature (Park et al., 8 Dec 2025).

Empirically, the top visual feature achieves AUROC$d_n$6 for separating correct from hallucinated activations. This suggests that a single SAE latent direction can carry a strong signal about whether the model’s answer is grounded in the image rather than driven by hallucination-related internal dynamics (Park et al., 8 Dec 2025).

4. Steering mechanism and inference procedure

Once the most indicative visual feature $d_n$7 has been selected, SAVE performs latent-space activation steering by modifying the hidden representation along that feature’s decoder direction:

$d_n$8

Here $d_n$9 is the steering strength. The intended effect is to boost the internal activation along the identified visual-understanding direction and thereby reinforce the model’s visual grounding (Park et al., 8 Dec 2025).

The intervention is applied during inference and is explicitly described as simple. Steering is applied at a single transformer layer, with layer 24 in LLaVA-1.6 given as an example. Only query (input) tokens are modified, while generated tokens are left unchanged. The steering strength is tuned per layer, with typical values 3–15 and common choices $$a(x) = \mathrm{TopK}[ \mathrm{ReLU}( W_{\mathrm{enc}} \cdot(x - b_{\mathrm{pre}}) + b_{\mathrm{enc}} ) ]$$0 (Park et al., 8 Dec 2025).

Layer selection is part of the method’s practical behavior. SAEs are trained at layers 8, 12, 16, 20, 24, and the results indicate that steering in early (8, 12) or late (24) layers is most effective, while mid-layers (16, 20) are less so. The optimal steering strength increases with network depth, reported as $$a(x) = \mathrm{TopK}[ \mathrm{ReLU}( W_{\mathrm{enc}} \cdot(x - b_{\mathrm{pre}}) + b_{\mathrm{enc}} ) ]$$1 at layer 8 and $$a(x) = \mathrm{TopK}[ \mathrm{ReLU}( W_{\mathrm{enc}} \cdot(x - b_{\mathrm{pre}}) + b_{\mathrm{enc}} ) ]$$2 at layer 24 (Park et al., 8 Dec 2025).

5. Benchmarks and quantitative results

SAVE is evaluated on three benchmark families. CHAIR_S and CHAIR_I measure the proportion of captions or object mentions that hallucinate, where lower is better. POPE evaluates binary presence questions on MSCOCO and reports F1 and accuracy across random, popular, and adversarial splits. MMHal-Bench contains 96 image-question pairs spanning eight categories, scored by GPT-4, and reports average score $$a(x) = \mathrm{TopK}[ \mathrm{ReLU}( W_{\mathrm{enc}} \cdot(x - b_{\mathrm{pre}}) + b_{\mathrm{enc}} ) ]$$3 and hallucination rate $$a(x) = \mathrm{TopK}[ \mathrm{ReLU}( W_{\mathrm{enc}} \cdot(x - b_{\mathrm{pre}}) + b_{\mathrm{enc}} ) ]$$4 (Park et al., 8 Dec 2025).

On LLaVA-1.6 (7B), the reported gains are as follows (Park et al., 8 Dec 2025):

Benchmark	Baseline	SAVE
CHAIR_S	31.2%	21.4%
CHAIR_I	7.9%	5.4%
MMHal-Bench score	2.88	3.12
MMHal-Bench HalRate	0.41	0.36

For POPE, the paper reports that average F1/Acc improve by $$a(x) = \mathrm{TopK}[ \mathrm{ReLU}( W_{\mathrm{enc}} \cdot(x - b_{\mathrm{pre}}) + b_{\mathrm{enc}} ) ]$$5–1 pts on LLaVA-1.6 (7B). The comparisons are made against state-of-the-art, training-free methods including VCD, DeCO, Devils, VTI, VISTA (Park et al., 8 Dec 2025).

The method is also reported to generalize across architectures. Similar consistent gains appear on LLaVA-NeXT (8B) and Qwen2-VL (7B). The paper describes this as evidence of both robustness and generalizability across multiple models and layers (Park et al., 8 Dec 2025).

6. Mechanistic analysis, ablations, and stated limitations

SAVE is accompanied by analyses intended to explain why the steering intervention reduces hallucination. At the token level, under the same prefix such as “In the distance, a…”, the vanilla model’s penultimate layer can sharply increase the probability of a hallucinated token such as “boat”; SAVE prevents that spike and instead favors a grounded alternative such as “mountain” (Park et al., 8 Dec 2025).

Attention analysis shows that SAVE decreases text-token attention and increases image-token attention when measuring average cross-attention from the final generated token back to query versus image tokens. The reported interpretation is that steering along visual understanding features reduces reliance on language priors and increases grounding in the visual input (Park et al., 8 Dec 2025).

The ablations further constrain the claim space. Steering toward the top-1 visual feature outperforms steering toward hallucination features, randomly chosen features, and combinations of the top-3 or top-5 features. The paper also reports balanced yes–no transition counts ($$a(x) = \mathrm{TopK}[ \mathrm{ReLU}( W_{\mathrm{enc}} \cdot(x - b_{\mathrm{pre}}) + b_{\mathrm{enc}} ) ]$$6 “yes $$a(x) = \mathrm{TopK}[ \mathrm{ReLU}( W_{\mathrm{enc}} \cdot(x - b_{\mathrm{pre}}) + b_{\mathrm{enc}} ) ]$$7 no” vs. $$a(x) = \mathrm{TopK}[ \mathrm{ReLU}( W_{\mathrm{enc}} \cdot(x - b_{\mathrm{pre}}) + b_{\mathrm{enc}} ) ]$$8 “no $$a(x) = \mathrm{TopK}[ \mathrm{ReLU}( W_{\mathrm{enc}} \cdot(x - b_{\mathrm{pre}}) + b_{\mathrm{enc}} ) ]$$9 yes”), which is presented as evidence that SAVE does not merely bias the model toward a fixed answer. Qualitative examples on CHAIR and adversarial POPE/MMHal-Bench are described as showing visually accurate outputs relative to hallucinated baselines (Park et al., 8 Dec 2025).

The released implementation includes /save/sae_train.py for SAE training, /save/probe.py for binary object-presence probing and activation collection, /save/steer.py for inference-time steering wrappers for LLaVA and Qwen2-VL, /configs/ for YAML hyperparameter files, and /experiments/ for reproducing CHAIR, POPE, and MMHal-Bench evaluations (Park et al., 8 Dec 2025).

The paper also states potential limitations and future directions. The listed limitations are reliance on a single-feature steering, which may miss multi-faceted visual concepts, and the need to train separate SAEs per layer. Proposed future directions include extending feature identification to richer probes such as counting and relations, exploring multi-direction steering strategies, and integrating SAVE into end-to-end fine-tuning for additional gains (Park et al., 8 Dec 2025).