VisionWeaver: Context-Aware Visual Fusion
- VisionWeaver is a dynamic, multi-expert visual fusion module that reduces hallucination by leveraging specialized encoders with distinct inductive biases.
- It uses a context-aware routing mechanism to selectively aggregate visual features from experts like CLIP, ConvNext, EVA, SAM, and others based on scene context.
- Empirical results on benchmarks such as VHBench-10 and POPE demonstrate its capability to improve detection, segmentation, and localization performance in LVLMs.
Searching arXiv for the VisionWeaver paper and closely related LVLM hallucination work. VisionWeaver is a context-aware multi-encoder visual fusion module for large vision-LLMs (LVLMs) that targets hallucination at the level of visual representation rather than only through data curation or decoding-time correction. It is introduced as a plug-and-play, encoder-level intervention built on a LLaVA-style architecture, motivated by the claim that object hallucination in LVLMs is often rooted in encoder-specific perceptual deficits rather than in language modeling alone. The central thesis is that different visual encoders carry different inductive biases because of their distinct pretraining objectives and datasets, and that these biases produce distinct hallucination profiles; VisionWeaver therefore uses a routing mechanism to aggregate specialized visual experts dynamically rather than relying on a single encoder or naive multi-encoder fusion (Wang et al., 17 Sep 2025).
1. Problem formulation and conceptual scope
VisionWeaver studies hallucination in a broad visual sense. In this setting, hallucination is not restricted to mentioning a nonexistent object. It also includes unsupported attributes, text, counts, positions, and interactions. The paper frames this as a vision-side reliability problem: if the LLM receives incomplete or biased visual evidence, later-stage corrections can only partially repair the failure. This makes VisionWeaver a complementary alternative to mitigation strategies centered on supervision changes or decoding heuristics (Wang et al., 17 Sep 2025).
A recurring misconception in LVLM evaluation is that hallucination is adequately captured by coarse object-presence benchmarks. VisionWeaver rejects that view. Its argument is that many errors are more precisely failures of detection, segmentation, localization, or classification. This reframing matters because it shifts the design question from “how should the model avoid saying the wrong thing?” to “what visual evidence should the model have encoded in the first place?” A plausible implication is that hallucination reduction should be treated partly as an encoder composition problem rather than solely as a language alignment problem.
The paper’s empirical examples illustrate this broader notion. A model may hallucinate a brand name, confuse the number of entities, misread text, or infer an incorrect spatial relation even when it does not invent a completely absent object. VisionWeaver is designed specifically for such heterogeneous failure modes.
2. VHBench-10 and the diagnosis of encoder bias
To test the hypothesis that encoder inductive bias predicts hallucination behavior, the paper introduces VHBench-10, a vision-centric fine-grained hallucination benchmark with approximately 10,000 samples. It is built from 2,000 images selected from LLaVA-ReCap-118K. For each image, the factual caption is paired with hallucinated captions generated by GPT-4o-mini, each injecting exactly one hallucination type, yielding ternaries . Evaluation is based on perplexity: the model is considered wrong when it prefers the hallucinated caption, i.e. when (Wang et al., 17 Sep 2025).
VHBench-10 organizes hallucinations into ten categories grouped by four broader visual competencies.
| Competency | Hallucination categories |
|---|---|
| Detection | Category, Counting, Occlusion |
| Segmentation-related | Text, Shape |
| Localization | Absolute positioning, Relative positioning |
| Classification | Color, Action, Relative interaction |
This taxonomy is central to the paper’s diagnosis. Existing benchmarks such as POPE are treated as useful for coarse object hallucination, but insufficient for isolating visual sub-capabilities such as OCR, counting, boundary sensitivity, or relation perception. VHBench-10 is therefore not merely an evaluation set; it is the paper’s instrument for showing that encoders fail differently.
The appendix table reports distinctive error profiles for individual encoders. CLIP is relatively good for global perception such as object presence; DINOv2 is better on fine-grained details such as color and action; Vary is stronger on text-related perception; and SAM is helpful for occlusion- and boundary-sensitive phenomena. VisionWeaver achieves the lowest error rate in all ten categories. Examples against CLIP include Category $1.49$ vs. $2.04$, Color $0.31$ vs. $4.39$, Shape $0.84$ vs. $4.30$, Action 0 vs. 1, Counting 2 vs. 3, Text 4 vs. 5, Absolute position 6 vs. 7, Relative position 8 vs. 9, and Relative interaction 0 vs. 1. This supports the paper’s main claim that hallucination is not monolithic and that encoder specialization is diagnostically meaningful.
3. Architecture and routing mechanism
VisionWeaver is a multi-expert visual encoding framework. The visual experts used in the paper are CLIP-ViT-L-336px, ConvNext, EVA-02, SAM, DINOv2, and Vary. CLIP serves a dual role: it provides the base visual representation and supplies the global signal for routing. All experts are aligned to image resolution 2, output token count 3, and feature dimension 4 via linear adapters (Wang et al., 17 Sep 2025).
The architecture defines expert feature extraction as
5
where 6 is the input image, 7 is the 8-th expert, and 9 is that expert’s representation. CLIP produces a global CLS token and patch tokens,
0
and the routing network maps the CLS token to expert scores,
1
Routing weights are then obtained by softmax,
2
The paper writes the aggregation step as
3
followed by fusion with CLIP patch tokens,
4
The intended meaning is a router-weighted combination of specialist features added residually to the CLIP patch sequence. The resulting visual tokens are then passed through a two-layer MLP projector into the LLM embedding space.
This design is explicitly contrasted with simple feature addition and concatenation. The paper argues that naive fusion is suboptimal because different experts are not equally relevant for all images or visual subtasks. Concatenation is described as especially harmful because it creates a high-dimensional feature space that is difficult to project into the LLM embedding space. The routing network is therefore the essential component: it turns multi-expert fusion into a context-dependent selection mechanism rather than a uniform merge.
The article’s architectural emphasis is thus not on replacing CLIP, but on augmenting it. CLIP patch tokens remain the base representation, while specialized experts inject complementary evidence conditioned on the scene-level CLS signal.
4. Training pipeline and implementation regime
Experiments are built on LLaVA-1.5. The base visual encoder is CLIP-ViT-L-336px, the projector is a two-layer MLP, and the LLM backbones tested are Llama3.2-Instruct-3B and Qwen2.5-Instruct-3B. The experts are pretrained and explicitly not trained from scratch (Wang et al., 17 Sep 2025).
Training proceeds in two stages. In pre-training, the dataset is LLaVA-Pretrain, the optimizer is AdamW, batch size is 5, learning rate is 6, epochs equal 7, and only all projectors are trainable. In supervised fine-tuning, the dataset is LLaVA-Finetune, the optimizer is AdamW, batch size is 8, learning rate is 9, epochs equal $1.49$0, and all parameters are trainable. Training uses 8 Nvidia A100 GPUs, with reported runtimes of 8 hours for pretraining and 16 hours for fine-tuning.
The paper does not specify an auxiliary hallucination loss or a dedicated routing regularizer. It states only that the training pipeline follows LLaVA-style pretraining and supervised fine-tuning. Likewise, although the text says that the system obtains the top-$1.49$1 experts and corresponding importance scores, no precise top-$1.49$2 selection equation or implementation detail is given beyond the soft routing equations.
Appendix experiments explore parameter-efficiency variants in which the vision part is frozen and the projector plus LLM are tuned either fully or with LoRA. On POPE, the reported F1 scores are 85.9 for LLaVA-1.5-Llama3.2-3B without vision, 86.8 for the same baseline with vision, 87.1 for VisionWeaver without vision using LoRA, 87.8 for VisionWeaver without vision, and 88.8 for VisionWeaver with vision. The persistence of gains in constrained tuning settings is used as evidence that the benefit is not tied solely to full end-to-end finetuning.
5. Empirical performance and ablation structure
VisionWeaver is evaluated on hallucination-specific benchmarks and on broader LVLM benchmarks. On POPE and AutoHallusion, the method improves over both CLIP-only and naive multi-encoder baselines. For Llama3.2-3B, CLIP only yields POPE F1 $1.49$3, AutoHallusion Overall $1.49$4, and average $1.49$5; naive multi-encoder yields $1.49$6, $1.49$7, and $1.49$8; multi-encoder plus VisionWeaver yields $1.49$9, $2.04$0, and $2.04$1. For Qwen2.5-3B, the corresponding values are $2.04$2, $2.04$3, and $2.04$4 for CLIP only; $2.04$5, $2.04$6, and $2.04$7 for naive multi-encoder; and $2.04$8, $2.04$9, and $0.31$0 for VisionWeaver (Wang et al., 17 Sep 2025).
These gains do not come at the expense of general multimodal capability. On Llama3.2, CLIP achieves MME 1382.15, MMStar 37.54, MMB 67.41, OCRB 31.41, and MathVista 27.14; VisionWeaver reaches 1392.45, 39.86, 69.76, 35.61, and 29.63. On Qwen2.5, CLIP reports 1444.26, 40.94, 64.09, 29.47, and 31.76, while VisionWeaver reports 1465.92, 43.65, 69.24, 36.48, and 35.81. The paper uses this to argue that encoder-side hallucination reduction need not degrade overall LVLM performance.
Ablation results are particularly important for interpreting the method. With Llama3.2-3B and all six experts, additive fusion yields POPE Accuracy 89.0, POPE F1 88.2, AutoHallusion Accuracy 47.6, and average 67.9. Concatenation yields 88.7, 87.9, 42.6, and 65.3. VisionWeaver yields 89.5, 88.8, 48.2, and 68.5. The contrast with concatenation is especially strong, supporting the claim that simple feature pooling is not enough.
The paper also shows that more experts are not always better. Using all six encoders with additive fusion gives average 67.9, which is slightly worse than using only four specific encoders—CLIP, ConvNext, EVA, and SAM—with additive fusion, average 68.4. This is a direct argument against the misconception that multi-expert robustness can be obtained by indiscriminately adding more visual backbones. VisionWeaver’s dynamic routing is presented as the mechanism that exploits complementarity without forcing all experts to contribute equally on every image.
In practical terms, the method changes prefill cost more than decode cost. Base LLaVA-1.5-Llama3.2-3B reports Prefill 50.99 ms and Inference 1273.20 ms, whereas VisionWeaver reports Prefill 99.46 ms and Inference 1201.52 ms. The paper interprets the added overhead as negligible in practice because experts are relatively small compared with the LLM, visual tokens sent to the LLM remain unchanged, image-side processing occurs once in prefill, and the visual tokens are then KV-cached.
6. Qualitative behavior, limitations, and broader significance
The qualitative case studies are aligned with the fine-grained benchmark taxonomy. In one example, the baseline hallucinated the clock brand as OMEGA instead of ROLEX and additionally invented a courthouse, a British flag, and a historic European village setting. With VisionWeaver, the description became grounded in a black post clock, the correct ROLEX brand, time approximately 10:09, a Lincoln St sign, an American flag, and a realistic downtown streetscape. In another example, the baseline described a single skier performing a triple backflip, whereas VisionWeaver identified three skiers, with two airborne and one on the ground. These examples reflect improvements in text recognition, counting, attribute grounding, and action understanding (Wang et al., 17 Sep 2025).
The paper nonetheless states several limitations. VHBench-10 does not fully cover realistic hallucination space: about 20% of hard-to-categorize samples do not fit neatly into its taxonomy. The hallucinated captions are generated with GPT-4, which may introduce some error. Experiments are confined to relatively small 3B-scale models because of resource limits. In addition, the paper does not provide a detailed analysis of the router’s internal decision logic, does not specify an auxiliary routing loss, and does not fully specify the exact top-$0.31$1 implementation mentioned in the text.
The broader significance of VisionWeaver lies in its argument about where hallucination originates. Rather than treating hallucination primarily as a decoding pathology, the paper relocates part of the problem to visual evidence formation. Its empirical result is that encoder pretraining objectives induce stable strengths and weaknesses, and that adaptive routing among specialized encoders can reduce hallucination while improving general multimodal benchmarks. This suggests a shift in LVLM design priorities: reliable multimodal systems may require not one universal visual encoder, but a controlled composition of experts whose contributions are conditioned on scene context.
In that sense, VisionWeaver is best understood not merely as a fusion module, but as a vision-centric reliability framework. Its benchmark, routing architecture, and ablation results together support a specific thesis: hallucination mitigation is partly a visual representation problem, and context-aware multi-expert grounding is a workable response to that problem.