Synergos-VQA: A Multi-Evidence KBVQA Framework

• The paper introduces a synergistic framework that concurrently generates and fuses holistic, structural, and causal evidence to overcome uni-dimensional reasoning in KBVQA.
• It employs an evidence-generation engine with region localization, prototype-driven object detection, and counterfactual probing, integrated via a T5-large decision module.
• Empirical results on OK-VQA, A-OKVQA, and ScienceQA demonstrate state-of-the-art performance, validating the benefits of diversified, plug-and-play multimodal reasoning.

Searching arXiv for the named topic and closely related work to ground the article. Synergos-VQA is a synergistic reasoning framework for Knowledge-Based Visual Question Answering (KBVQA) that was introduced to address the claim that contemporary Multimodal LLM pipelines are bottlenecked by reliance on “uni-dimensional evidence” (Wang et al., 23 Jul 2025). Rather than forcing reasoning through a single caption or context paragraph, the framework concurrently generates and fuses three complementary evidence streams at inference time: Holistic Evidence for scene-level perception, Structural Evidence from a prototype-driven module for object-centric grounding, and Causal Evidence from a counterfactual probe for robustness checking. These streams are integrated by a Synergistic Decision Module built on Fusion-in-Decoder T5-large, yielding state-of-the-art results on OK-VQA, A-OKVQA, and the image subset of ScienceQA while remaining plug-and-play with open-source MLLMs (Wang et al., 23 Jul 2025).

1. Conceptual basis and problem setting

Synergos-VQA is defined within KBVQA rather than standard VQA. In standard VQA, many questions can be answered from pixels alone, such as colors, counts, or object categories. KBVQA introduces questions whose answers require external or commonsense knowledge that is not directly visible in the image. The motivating example given for this distinction is a violin image with the question “What type of music is this instrument commonly used for?”, where recognizing the instrument is insufficient without world knowledge about its use (Wang et al., 23 Jul 2025).

The framework is explicitly motivated by a critique of the prevailing “LLM-as-an-engine” paradigm. In that regime, an MLLM or captioning pipeline produces a single descriptive context, and a downstream LLM answers from that context. The Synergos-VQA formulation argues that this is effectively uni-dimensional evidence: it emphasizes descriptive surface details but often lacks an explicit structural backbone and an explicit causal check on whether the answer truly depends on the right evidence (Wang et al., 23 Jul 2025). The paper characterizes this failure mode with the metaphor of “seeing only the trees, but not the forest,” and proposes a corresponding three-part remedy: scene-level description, object-level structure, and causal robustness.

This framing places Synergos-VQA in a broader line of work that treats robust VQA as more than answer classification. Earlier universal-VQA analyses had already shown that architectures often fail to generalize across natural and synthetic domains (Shrestha et al., 2019), but Synergos-VQA narrows the problem to knowledge-grounded multimodal reasoning and responds with evidence diversification rather than a single unified encoder.

2. System organization and inference pipeline

At the systems level, Synergos-VQA has two main layers: an evidence-generation engine and a Synergistic Decision Module. The evidence-generation engine runs at inference time and contains three tracks: Holistic Scene Analysis, Structural Backbone Reasoning, and Causal Robustness Probing. The decision layer is a FiD-style T5-large model that fuses the resulting evidence streams with visual features and the original question to generate the final answer (Wang et al., 23 Jul 2025).

The operational pipeline is sequential but multi-track. Given an image $I$ and question $Q$, the framework first localizes question-relevant regions and captions them to produce Holistic Evidence. It then detects objects, maps them to a prototype library, and constructs a Prototype Chain-of-Thought together with prototypical training cases to produce Structural Evidence. On top of that structural trace, it runs a Causal Reasoning Probe that performs a textual counterfactual analysis to generate Causal Evidence. These evidence sources are serialized into multiple passages and passed to the FiD decoder, which cross-attends over all encoded passages and emits the answer (Wang et al., 23 Jul 2025).

This design is explicitly described as an inference-time framework: the multi-track reasoning is not jointly learned end-to-end with the evidence generators. Only the T5-based decision module is fine-tuned on KBVQA data, while the detection stack, prototype library, and MLLM evidence generator remain fixed. That separation is central to the framework’s plug-and-play claim.

Component	Primary mechanism	Output
Holistic Evidence	Grad-CAM region localization + MLLM captioning	$E_{\text{desc}}$
Structural Evidence	DETR + prototype retrieval + MLLM label selection	$E_{\text{prot}}$, $E_{\text{case}}$
Causal Evidence	Counterfactual prompting over Proto-CoT	$E_{\text{causal}}$

3. The three evidence streams

Holistic Evidence is the “forest” view. The framework first applies a pre-trained vision-LLM with Image-Text Matching, such as BLIP-2, and uses Grad-CAM over the ITM scores to localize the Top-$Z$ question-relevant regions. It then crops each region and uses Qwen2.5-VL-7B to produce a fine-grained caption conditioned on the question. Caption generation is formalized as

$$P(c_k \mid r_k, Q) = \prod_{t=1}^{L} P(w_t \mid w_{<t}, r_k, Q; \theta_{\text{cap}})$$

for region $r_k$ and caption $c_k$, and the full Holistic Evidence is $Q$0 (Wang et al., 23 Jul 2025). In the reported implementation, $Q$1 is used as a trade-off point.

Structural Evidence is the “trees” view. Offline, the framework uses DETR object features extracted from COCO and clusters them with K-Means into a prototype library by minimizing

$Q$2

where $Q$3 is the centroid of cluster $Q$4 (Wang et al., 23 Jul 2025). The chosen library size is $Q$5. Online, DETR detects object proposals in the input image, each proposal is matched to nearest prototypes by cosine similarity, and Qwen2.5-VL-7B chooses the most accurate label from the retrieved candidate set using the full image, the cropped object, and the question as context. The selected labels are arranged into a Prototype Chain-of-Thought $Q$6, while an inverted index over prototype sets retrieves the Top-$Q$7 most similar training examples as prototypical cases $Q$8, with $Q$9 (Wang et al., 23 Jul 2025).

Causal Evidence is the robustness track. The framework motivates it with Pearl’s causal language: standard MLLMs may exploit spurious correlations along a backdoor path rather than the intended causal path from relevant evidence to answer. The Causal Reasoning Probe therefore takes the Prototype CoT and prompts the MLLM to identify which prototype or object is causally necessary, explain the factual chain, and then generate a counterfactual by removing or changing that prototype (Wang et al., 23 Jul 2025). The output $E_{\text{desc}}$0 is a textual explanation that contrasts factual reasoning with its counterfactual variant. Importantly, this is a prompt-based intervention rather than a formally identified structural causal model.

A common misconception is that Synergos-VQA is merely caption ensembling. The evidence design contradicts that interpretation: the structural track is prototype-driven and retrieval-based, and the causal track explicitly performs counterfactual prompting over the Proto-CoT rather than producing another descriptive caption (Wang et al., 23 Jul 2025).

4. Synergistic Decision Module, training, and implementation

The Synergistic Decision Module is a Fusion-in-Decoder model built on T5-large. Its encoder receives multiple passages, each composed of the question $E_{\text{desc}}$1, projected visual evidence $E_{\text{desc}}$2, and one evidence stream: holistic, structural plus cases, or causal. Visual evidence comes from DETR features projected into T5-large’s hidden size with a two-layer MLP. The shared encoder processes each passage independently, and the decoder performs cross-attention over the concatenated encoder outputs, effectively learning

$E_{\text{desc}}$3

to generate the answer sequence (Wang et al., 23 Jul 2025).

Training is limited to the T5 module. The optimization objective is standard sequence-to-sequence cross-entropy,

$E_{\text{desc}}$4

with AdamW, learning rate $E_{\text{desc}}$5, weight decay $E_{\text{desc}}$6, a linear scheduler with $E_{\text{desc}}$7 warmup steps and $E_{\text{desc}}$8 total steps, batch size $E_{\text{desc}}$9 per GPU with gradient accumulation $E_{\text{prot}}$0, and seed $E_{\text{prot}}$1 (Wang et al., 23 Jul 2025). The evidence generator itself is not fine-tuned: DETR remains fixed, the prototype library is built once offline, and Qwen2.5-VL-7B is used through prompting for region captioning, prototype disambiguation, and causal probing.

Implementation details emphasize practicality. The system uses PyTorch 2.1.0, CUDA 12.1, Python 3.10, and NVIDIA A100 GPUs. Evidence generation with Qwen2.5-VL-7B uses beam size $E_{\text{prot}}$2, temperature $E_{\text{prot}}$3, top-$E_{\text{prot}}$4 $E_{\text{prot}}$5, and max new tokens $E_{\text{prot}}$6. Training the T5 module takes about three days on $E_{\text{prot}}$7A100 GPUs. Reported inference cost is about $E_{\text{prot}}$8 seconds per OK-VQA sample on a single GPU, or about $E_{\text{prot}}$9 seconds with three GPUs when the three evidence tracks are parallelized (Wang et al., 23 Jul 2025). This is presented as faster than QACap plus Claude 3.5 while remaining fully local.

5. Empirical performance and ablation evidence

Synergos-VQA is evaluated on OK-VQA, A-OKVQA, and the image subset of ScienceQA, and the reported results establish new state of the art on all three benchmarks (Wang et al., 23 Jul 2025).

Benchmark	Representative comparison	Synergos-VQA
OK-VQA test	QACap + Claude 3.5: 68.2	69.6
A-OKVQA Avg	ReAuSE: 74.8	77.5
ScienceQA image subset	EfficientLLaVA: 83.1	89.6

On OK-VQA test, Synergos-VQA reaches 69.6 accuracy, surpassing QACap + Claude 3.5 at 68.2, PaLM-E-562B at 66.1, and Qwen2.5-VL-7B at 60.4 (Wang et al., 23 Jul 2025). On A-OKVQA, it records 84.8 on multiple choice, 70.2 on direct answer, and 77.5 average, improving over ReAuSE’s 74.8 and QACap’s 71.5 (Wang et al., 23 Jul 2025). On the ScienceQA image subset, it reaches 89.6, ahead of EfficientLLaVA at 83.1 and ReflectiVA at 75.4 (Wang et al., 23 Jul 2025).

The ablation study on OK-VQA validation isolates the contribution of each evidence stream. Full Synergos-VQA scores 69.8. Removing Causal Evidence drops performance to 65.8, removing Structural Evidence drops it to 67.1, and replacing the evidence design with a “caption-only” Qwen+FiD baseline yields 63.4 (Wang et al., 23 Jul 2025). These numbers support the paper’s central claim that evidence diversity, rather than caption richness alone, is responsible for the gain. The paper also reports that results averaged over multiple seeds are stable, citing OK-VQA at $E_{\text{case}}$0.

Qualitative analysis further distinguishes the evidence streams. In skiing-style examples, the structural track supplies object-centric terms such as “parallel skis,” the holistic track describes posture and slope, and the causal probe tests whether changing the salient structural elements would alter the answer (Wang et al., 23 Jul 2025). This suggests that the framework’s gains come from complementary inductive biases rather than simple redundancy.

6. Relation to the wider literature, limitations, and extensions

Synergos-VQA belongs to a broader movement toward multi-source and multi-step VQA reasoning, but its particular combination of holistic, structural, and causal evidence is distinctive. Earlier multimodal fusion work such as Reciprocal Attention Fusion jointly attended to object-level and grid-level features for VQA, emphasizing complementary granularity in visual evidence (Farazi et al., 2018). Co-VQA shifted the problem into a conversation-based internal decomposition using sub-question sequences and an adaptive chain visual reasoning model (Wang et al., 2022). VLR reformulated VQA as lattice-based retrieval over scene graphs, prioritizing explicit visual grounding and interpretable search paths (Reich et al., 2022). Synergos-VQA differs from all three by combining inference-time multi-evidence generation with a learned FiD fusion module specialized for KBVQA (Wang et al., 23 Jul 2025).

Recent multimodal orchestration research points in similar directions. VideoMultiAgents uses independent modality-specific agents and a report-style organizer for zero-shot video QA (Kugo et al., 25 Apr 2025). “Team of One” coordinates multiple prompting pathways and an LLM integrator for complex video QA (Xie et al., 18 Jul 2025). V3Fusion selects and fuses heterogeneous VLMs using focal diversity, visual CKA, and uncertainty-based rectification (Tekin et al., 13 Mar 2026). A plausible implication is that Synergos-VQA could be combined with routing or multi-agent selection mechanisms when no single MLLM backbone is optimal across tasks and domains, an issue also highlighted by VQA360’s model-selection study (Sinha et al., 2024).

The framework also has explicit limitations. First, it depends on detection and attention quality: if DETR or Grad-CAM mislocalizes the relevant evidence, all three evidence streams can become self-consistent but wrong. The paper’s dog-and-frisbee failure case illustrates such a cascade, where “tail” is favored over the correct “mouth” because the structural track begins from a flawed grounding hypothesis (Wang et al., 23 Jul 2025). Second, prototype coverage is bounded by a COCO-derived library, which may be suboptimal for specialized domains. Third, the Causal Reasoning Probe remains heuristic: it performs textual counterfactual analysis but does not provide formal causal guarantees. Fourth, inference is heavier than single-pass VQA because the system performs several MLLM calls per sample.

These constraints clarify what Synergos-VQA is and is not. It is not a formally causal VQA model in the strict identifiability sense, and it is not an end-to-end jointly trained architecture. It is instead an inference-time synergistic reasoning framework whose central claim is methodological: superior evidence design can outperform sheer model scale on KBVQA (Wang et al., 23 Jul 2025). In that respect, it serves both as a concrete state-of-the-art system and as a design template for future multimodal reasoning systems that need to combine scene description, symbolic structure, and robustness-oriented verification within a single answering pipeline.