Veagle: Dynamic Multimodal Visual-Language Bridge

• Veagle is a multimodal model that dynamically projects instruction-aware visual features into a frozen language model, enhancing precise image-question alignment.
• The architecture integrates a frozen ViT-L/14, a Q-Former with an MLP projector, and a Mistral 7B LLM to fuse visual and textual data without modifying LLM internals.
• Empirical evaluations indicate a 5–6% improvement on reasoning-focused VQA benchmarks, while showing limitations on OCR-heavy tasks.

Veagle is a multimodal LLM designed to improve visual question answering (VQA) and image understanding, especially in text-rich and real-world scenarios. Introduced in "Veagle: Advancements in Multimodal Representation Learning" (Chawla et al., 2024), it builds on recent advances in the VLM/MLLM ecosystem—BLIP-2, LLaVA, BLIVA, mPLUG-Owl, and InstructBLIP—and introduces a dynamic mechanism that projects instruction-aware visual features directly into a frozen LLM via soft prompts. The model’s stated target is to improve interpretation of embedded text in images and nuanced image-question alignment by making the visual-to-language projection dynamic and instruction-aware, rather than relying on fixed linear projections or modifying the internal layers of the LLM (Chawla et al., 2024).

1. Definition and problem setting

Veagle is presented as a response to limitations in existing VLMs and MLLMs. The paper identifies a recurring weakness in models that rely on fixed linear projections from image features to language token space or on cross-attention bridges that are not instruction-aware. In the paper’s formulation, these approaches often struggle with fine-grained grounding, especially for text embedded in images such as OCR, charts, and documents; robust VQA in real-world scenarios requiring precise detail recognition and compositional reasoning; and efficient alignment between frozen LLMs and visual encoders that capture low- and high-level vision but still require instruction-aware selection of relevant details (Chawla et al., 2024).

Within that framing, Veagle’s novelty is not a new end-to-end multimodal pretraining paradigm, but a particular bridge from visual representations into a frozen language backbone. The model is described as using a dynamic mechanism to project encoded visual information directly into the LLM, with the resulting visual tokens depending on both the image and the instruction. This differentiates Veagle from static projection approaches that feed the same style of visual tokenization to the LLM regardless of the query and from architectures such as Flamingo that insert cross-attention blocks into the LLM stack (Chawla et al., 2024).

A central implication of this design is that the model aims to improve image-question alignment without modifying the internal architecture of the LLM. The paper explicitly positions this as a way to preserve pretrained capabilities while still increasing the model’s ability to select and condition on the most relevant visual details.

2. Architectural organization and dynamic projection

Veagle combines three principal modules: a visual encoder, a visual abstractor, and a frozen language backbone. The architectural instantiation reported in the paper is highly specific and combines components drawn from several earlier multimodal systems (Chawla et al., 2024).

Component	Instantiation and role
Visual encoder	mPLUG-Owl ViT-L/14, frozen, produces patch embeddings
Visual abstractor	InstructBLIP Q-Former plus an MLP projector
Language backbone	Mistral 7B, frozen, receives visual soft prompts concatenated with text tokens

The visual encoder is ViT-L/14 from mPLUG-Owl, trained with image-caption data including LAION-400M, COYO-700M, Conceptual Captions, and MSCOCO. In Veagle’s training, this visual encoder remains frozen in both pretraining and fine-tuning. The language backbone is Mistral 7B, also kept frozen; the paper notes that its grouped-query attention and sliding-window attention offer efficient inference, but Veagle does not modify the LLM’s internal layers (Chawla et al., 2024).

The bridging mechanism is the core of the model. Veagle adopts a Q-Former module from InstructBLIP to extract instruction-aware visual features by querying the frozen image encoder outputs, and augments this with a multi-layer perceptron projector to overcome the limited capacity of a single linear projection. The outputs of the Q-Former and the MLP projector are concatenated to form visual soft prompt tokens, which are then concatenated with the text token embeddings at the LLM input. No cross-attention blocks are inserted into the LLM; the LLM consumes visual tokens as a prefix to the textual instruction (Chawla et al., 2024).

The paper gives the mechanism in explicit notation. Let $V \in \mathbb{R}^{T_v \times d_v}$ be the frozen image encoder outputs and let $L \in \mathbb{R}^{T_l \times d_l}$ be the text token embeddings. The MLP projection is written as

$P(V) = W_p V + b_p,$

where $W_p \in \mathbb{R}^{d_l \times d_v}$ and $b_p \in \mathbb{R}^{d_l}$, with the note that this may be multi-layer, i.e., an MLP with nonlinearity. The Q-Former is described conceptually with a simplified cross-attention step:

$$Q = Q_q W_Q,\quad K = V W_K,\quad V_{\text{att}} = V W_V,$$

$$\mathrm{Attention}(Q, K, V_{\text{att}}) = \mathrm{softmax}(Q K^T / \sqrt{d}) V_{\text{att}},$$

producing $H_q \in \mathbb{R}^{N_q \times d_l}$ after suitable projections or MLP heads to the LLM embedding dimension. The injected visual prompt is then

$T_v = \mathrm{concat}(H_q, P(V)),$

and the effective language-model input becomes

$L' = \mathrm{concat}(T_v, L).$

In this formalization, “dynamic” means that the produced tokens $L \in \mathbb{R}^{T_l \times d_l}$0 depend on the specific image and instruction or question. The paper’s claim is that this instruction-aware adaptation is what distinguishes Veagle from a static visual-token mapping (Chawla et al., 2024).

3. Training scheme and data curation

Veagle is trained in two stages, with the visual encoder and the LLM frozen throughout. Stage 1 aligns projection layers with caption data while freezing the visual encoder, Q-Former, and LLM. Stage 2 fine-tunes the visual abstractor and projector on VQA-style datasets while the LLM and visual encoder remain frozen. The stated objective in both stages is language modeling rather than contrastive alignment (Chawla et al., 2024).

The loss is written as next-token negative log-likelihood:

$L \in \mathbb{R}^{T_l \times d_l}$1

where $L \in \mathbb{R}^{T_l \times d_l}$2 are target tokens and the context includes both text and image features via $L \in \mathbb{R}^{T_l \times d_l}$3. The paper explicitly states that no contrastive or alignment losses are reported and that training focuses on the language modeling loss while freezing large components to preserve pretrained knowledge (Chawla et al., 2024).

The fine-tuning corpus reported for Veagle includes COCO, TextCaps, VQAv2, OK-VQA, AOK-VQA, GQA, OCR-VQA, TextVQA, VizWiz, plus in-house curated data. The paper also reports two data-processing interventions intended to better match the generative language-modeling objective. For datasets with single-word answers, responses were expanded to detailed answers using GPT-4 and Mixtral, and redundant questions were mitigated by generating varied formulations to improve generalization. Tokenization follows standard Mistral-compatible LLM tokenization for text; images are processed to ViT-L/14 resolution, although the details are noted as not explicitly enumerated in the paper (Chawla et al., 2024).

The reported training compute is 8× NVIDIA A100 for training and 1× NVIDIA A6000 for inference. The Appendix gives the optimization details as follows: pretraining for 3 epochs with AdamW, learning rate $L \in \mathbb{R}^{T_l \times d_l}$4, batch size 8, and weight decay 0.05; fine-tuning for 2 epochs with AdamW, learning rate $L \in \mathbb{R}^{T_l \times d_l}$5, batch size 10, and weight decay 0.05. The paper also notes that techniques such as LoRA, adapters, quantization, and KV-caching are not reported (Chawla et al., 2024).

4. Empirical evaluation and observed performance

Veagle is evaluated on benchmark datasets emphasizing visual question answering and image understanding. The paper reports accuracy via an automatic judging setup in which GPT-4 labels model outputs as correct or incorrect. It explicitly notes that this is unconventional relative to standard VQA metrics such as exact match or consensus scoring, but states that the method is applied consistently across baselines (Chawla et al., 2024).

The abstract reports an improvement of 5–6% in performance, with Veagle outperforming existing models by a notable margin. In the detailed results, several benchmark-specific comparisons are reported. On OK-VQA, Veagle scores 49.3, compared with BLIVA 43.4, InstructBLIP 30.8, mPLUG-Owl 34.1, and LLaVA 46.2. On ScienceQA, Veagle scores 58.1, compared with BLIVA 16.1, InstructBLIP 40.2, mPLUG-Owl 51.8, and LLaVA 56.5. On GQA, Veagle scores 44.2, compared with BLIVA 28.6, InstructBLIP 40.8, mPLUG-Owl 33.9, and LLaVA 43.9. On COCO captioning, Veagle scores 57.9, compared with BLIVA 56.4, InstructBLIP 51.2, mPLUG-Owl 55.6, and LLaVA 62.7 (Chawla et al., 2024).

The reported picture is not uniformly favorable. On OCR-VQA and TextVQA, the paper states that Veagle trails top baselines; the examples provided are LLaVA 67.2 versus Veagle 48.3 on OCR-VQA, and LLaVA 37.2 versus Veagle 22.5 on TextVQA. On ChartQA, Veagle scores 13.4 versus BLIVA 13.2, and the paper remarks that performance is overall low across models. This makes the empirical profile more specific than a general claim of broad dominance: Veagle appears stronger on reasoning-oriented VQA such as ScienceQA and on several general VQA settings, while remaining weaker on OCR-heavy tasks (Chawla et al., 2024).

The paper also reports an in-house test dataset never seen during training, on which Veagle scores 76.4 versus BLIVA 63.1, InstructBLIP 59.3, mPLUG-Owl 68.6, and LLaVA 66.5. Statistical significance or error bars are not provided. Qualitative examples are said to demonstrate intricate scene understanding and reasoning, multi-turn visual conversations conditioned on the question, and creative generation from images, including poems, ads, and songs (Chawla et al., 2024).

5. Relation to adjacent multimodal architectures

Veagle is best understood as a bridge architecture in the lineage connecting BLIP-2, InstructBLIP, LLaVA, Flamingo, BLIVA, and larger multimodal pretraining systems such as PaLI and Kosmos. The paper’s comparisons are explicit about what is reused and what is altered (Chawla et al., 2024).

Relative to BLIP-2, Veagle adopts the Q-Former bridging strategy between a frozen image encoder and a frozen LLM, but adds an MLP projector and concatenates the outputs to form soft prompts. Relative to LLaVA, which uses a simple projection to map CLIP or ViT features into LLM space and trains with visual instruction tuning, Veagle’s distinguishing feature is that the visual tokens are adapted to the question through the Q-Former. Relative to Flamingo, which inserts cross-attention layers that directly condition text on image features within the LLM stack, Veagle avoids modifying LLM internals and instead injects visual prompts at the input. Relative to BLIVA, the paper presents Veagle as extending the bridge with a strengthened visual abstractor from mPLUG-Owl, an instruction-aware Q-Former, and an MLP projector (Chawla et al., 2024).

This placement clarifies both the system’s ambition and its constraints. Veagle does not attempt large-scale end-to-end multimodal retraining. Instead, it emphasizes efficient bridging with a frozen LLM and dynamic soft prompting. A plausible implication is that the model’s improvements are tied less to scale than to the specificity of how visual content is translated into the language-model input space. The paper itself formulates the contribution in those terms: instruction-aware, dynamically generated visual soft prompts injected directly into a frozen LLM.

6. Limitations, reproducibility, and nomenclature

The paper identifies several limitations directly. It reports no formal ablations varying the number of visual tokens, projector capacity, injection placement, or alternative encoders and gating strategies. It also notes that evaluation via GPT-4-as-judge may introduce biases, and that standard VQA metrics and human evaluation would strengthen the claims. A further limitation is architectural: because the LLM and visual encoder remain frozen, ceiling performance on specialized domains may be capped. The empirical weakness on OCR-heavy tasks such as OCR-VQA and TextVQA is presented as evidence that OCR robustness and fine-grained text reading remain open problems for the model (Chawla et al., 2024).

Reproducibility is comparatively strong. The paper states that code and models are openly available, with a GitHub repository at https://github.com/superagi/Veagle and a Hugging Face release at https://huggingface.co/SuperAGI/Veagle. Training parameters, hardware, and major datasets are reported, although licenses and instructions are noted as not explicitly stated in the paper itself (Chawla et al., 2024).

The term “Veagle” also requires disambiguation. Several unrelated papers use the acronym EAGLE for different systems and datasets: "Seeing Before Agreeing: Aligning Multi-Agent Consensus with Visual Evidence" (Wang et al., 29 May 2026), "EAGLE: Large-scale Vehicle Detection Dataset in Real-World Scenarios using Aerial Imagery" (Azimi et al., 2020), "EAGLE: Egocentric AGgregated Language-video Engine" (Bi et al., 2024), and "EAGLE: Enhanced Visual Grounding Minimizes Hallucinations in Instructional Multimodal Models" (Villa et al., 6 Jan 2025). In the supplied summaries for those papers, “Veagle” is described as not appearing in the text and, depending on the case, as likely an informal or mistaken name. In strict bibliographic usage, Veagle therefore denotes the multimodal representation-learning model introduced in 2024, rather than those unrelated EAGLE acronyms.

Taken together, these features place Veagle in a specific niche within multimodal research: a frozen-backbone MLLM that seeks better image-question alignment through instruction-aware visual abstraction and direct soft-prompt injection. Its reported strengths are reasoning-oriented VQA, several open-source benchmarks, and generalization on an in-house test suite; its reported weaknesses include OCR-heavy tasks, the absence of formal ablations, and reliance on GPT-4-based judging rather than standard community metrics (Chawla et al., 2024).