Llama 3.2 Vision-8B-2 Cross-Attention Model

• Llama 3.2 Vision-8B-2 is defined as an adapter-based extension of Llama 3.1 8B, employing cross-attention layers to integrate visual data from a ViT-H/14 encoder.
• It leverages a structured fusion method by inserting cross-attention layers after every fourth self-attention block, ensuring robust multimodal integration even with long image token sequences.
• The model shows lower adversarial accuracy drops on VQA tests and introduces visual token trimming strategies to alleviate KV cache bottlenecks and improve efficiency.

Llama 3.2 Vision-8B-2 is Meta’s open-weight multimodal foundation model in the Llama 3.2 family, described in published evaluations as an adapter-based extension of a pretrained Llama 3.1 8B LLM rather than a natively end-to-end multimodal transformer. In the reported characterization, it combines a ViT-H/14 image encoder, a cross-attention image adapter, and a temporal video adapter, with the image pathway serving as the focus of the published experiments discussed here. Relative to projection-based vision-LLMs such as LLaVA, its defining architectural feature is the insertion of cross-attention layers after every fourth self-attention layer of the core LLM, yielding a more structured multimodal fusion mechanism (Fox et al., 19 Dec 2025).

1. Architectural definition and model identity

In the published description, Llama 3.2 Vision-8B-2 is organized around three components: an image encoder, an image adapter, and a video adapter. The image encoder is a ViT-H/14 with about 850M parameters; the image adapter injects visual information into the LLM through cross-attention; and the video adapter is a temporal adapter for video reasoning, although the reported robustness experiments concentrate on images rather than video (Fox et al., 19 Dec 2025).

Component	Reported form	Role
Language backbone	pretrained Llama 3.1 8B	Core decoder-only LLM
Image encoder	ViT-H/14, about 850M parameters	Extracts visual features
Image adapter	cross-attention adapter	Injects image information
Video adapter	temporal adapter	Supports video reasoning

A notable point in the literature is that the “Vision-8B-2” designation is treated as the model identifier for the 8B-parameter Llama 3.2 Vision variant used as an open multimodal foundation model. The available paper does not report a bespoke architectural modification uniquely associated with the “-2” suffix beyond that identifier and the adapter-based multimodal stack (Fox et al., 19 Dec 2025).

This placement matters because it distinguishes Llama 3.2 Vision-8B-2 from two adjacent design families often conflated in practice. One is projection-based multimodal composition, exemplified by LLaVA-style systems in which visual features are linearly projected into the LLM embedding space. The other is derivative models built on the broader Llama 3.2 family but with different vision encoders and fusion bridges. In this taxonomy, Llama 3.2 Vision-8B-2 is specifically a cross-attention-based multimodal extension of Llama 3.1 8B, not a simple ViT-MLP-LLM stack (Fox et al., 19 Dec 2025).

2. Vision fusion mechanism and computational profile

The model’s vision integration mechanism is defined by cross-attention rather than direct feature insertion. Unlike LLaVA, which uses a simple linear projection from visual features into the LLM embedding space, Llama 3.2 Vision uses cross-attention layers inserted after every fourth self-attention layer in the LLM (Fox et al., 19 Dec 2025). In a standard cross-attention block, text tokens provide queries and image tokens provide keys and values, which can be written as

$$A = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d}}\right).$$

Here, the crucial systems implication is not only fusion quality but cache geometry. Research on efficient inference for LLaMA-3.2-Vision identifies the image-token key-value cache in cross-attention layers as a major compute and memory bottleneck, often exceeding the cache size of text tokens in self-attention layers (Lee et al., 1 Apr 2025). The paper reports visual token lengths of 1,601 tokens at about 384×384 and up to 6,404 tokens at 720p / 1080p, and emphasizes that the image KV cache is persistent across decoding steps.

A simplified image-side cache scaling law is given as

$$\text{KV}_{image} \propto m \cdot d \cdot (\text{\# cross-attention layers}) \cdot 2,$$

with the factor of 2 arising from storage of both keys and values (Lee et al., 1 Apr 2025). The appendix-level FLOPs estimates likewise show explicit dependence on the image feature length $n_k$:

$\text{FLOPs}_{self} = 4nd^{2} + 2n^{2}d + 2ndm$

$$\text{FLOPs}_{cross} = 2nd^2 + 2n_kd^2 + 2nn_kd + 2ndm.$$

These expressions formalize an important point about the Llama 3.2 Vision design: cross-attention makes multimodal fusion more structured, but it does not remove the cost of long visual sequences. A plausible implication is that the model’s architectural advantage over projection-only systems is coupled to a distinctive inference-time memory burden concentrated in the visual KV cache.

3. Pretraining and alignment pipeline

The reported training pipeline for Llama 3.2 Vision is substantially larger than that of LLaVA-1.5-13B. The summary provided in the robustness study states that Llama 3.2 Vision underwent initial multimodal pretraining on approximately 6 billion image-text pairs, followed by an adapter annealing or refinement stage on an additional 500 million higher-quality images (Fox et al., 19 Dec 2025).

A second defining design choice is preservation of the language backbone during adapter training. The paper states that the LLM parameters are kept frozen while the image encoder is updated, with the stated purpose of preserving text-only capabilities (Fox et al., 19 Dec 2025). After pretraining, the model undergoes multiple alignment stages, including supervised fine-tuning, rejection sampling, direct preference optimization, and additional safety mitigations.

This training recipe is often discussed comparatively. In the same study, LLaVA-1.5-13B is described as having been trained on 595K image-text pairs from CC3M and 158K GPT-4-generated multimodal instruction samples (Fox et al., 19 Dec 2025). The contrast does not by itself prove a causal explanation for downstream behavior, but it frames the main hypothesis advanced in the literature: larger-scale multimodal pretraining and a more structured alignment pipeline may contribute to the relative robustness properties observed for Llama 3.2 Vision.

The available source also notes an important limitation of public documentation: detailed pretraining hyperparameters such as learning rate, batch size, and optimizer settings are not reported in that paper for Llama 3.2 Vision (Fox et al., 19 Dec 2025). As a result, architectural and scale-level descriptions are more precise in the current literature than full reproduction details.

4. Adversarial robustness on VQA

A dedicated evaluation of open foundation models tests Llama 3.2 Vision-8B-2 under white-box untargeted PGD attacks applied only to the visual modality on a random subset of 500 examples from the VQA v2 validation split (Fox et al., 19 Dec 2025). The threat model keeps the prompt fixed, perturbs only the image, and imposes an $L_\infty$ constraint. The attack increases the model’s generation loss $L(\theta, x, q)$ with the standard projected gradient sign update

$$x_{t+1} = \Pi_{\mathcal{B}_\epsilon(x)}\left(x_t + \alpha \,\mathrm{sign}\left(\nabla_{x_t} L(\theta, x_t, q)\right)\right).$$

The perturbation budgets evaluated are

$$\epsilon \in \left\{ \frac{2}{255}, \frac{4}{255}, \frac{8}{255}, \frac{16}{255}, \frac{128}{255}, \frac{255}{255} \right\},$$

with step sizes and iteration counts scaled to attack strength; the paper gives examples ranging from $\alpha = 0.00196$ with 5 iterations for $$\text{KV}_{image} \propto m \cdot d \cdot (\text{\# cross-attention layers}) \cdot 2,$$0 to $$\text{KV}_{image} \propto m \cdot d \cdot (\text{\# cross-attention layers}) \cdot 2,$$1 with 30 iterations for $$\text{KV}_{image} \propto m \cdot d \cdot (\text{\# cross-attention layers}) \cdot 2,$$2 (Fox et al., 19 Dec 2025).

The evaluation metric is the standard VQA accuracy metric, and robustness is summarized as

$$\text{KV}_{image} \propto m \cdot d \cdot (\text{\# cross-attention layers}) \cdot 2,$$3

For Llama 3.2 Vision-8B-2 on the 500-sample subset, the reported clean accuracy is 42.8%. The corresponding attacked accuracies are as follows (Fox et al., 19 Dec 2025).

$$\text{KV}_{image} \propto m \cdot d \cdot (\text{\# cross-attention layers}) \cdot 2,$$4	Accuracy (%)	Drop
clean	42.8	0.0
$$\text{KV}_{image} \propto m \cdot d \cdot (\text{\# cross-attention layers}) \cdot 2,$$5	36.2	6.6
$$\text{KV}_{image} \propto m \cdot d \cdot (\text{\# cross-attention layers}) \cdot 2,$$6	32.4	10.4
$$\text{KV}_{image} \propto m \cdot d \cdot (\text{\# cross-attention layers}) \cdot 2,$$7	33.0	9.8
$$\text{KV}_{image} \propto m \cdot d \cdot (\text{\# cross-attention layers}) \cdot 2,$$8	36.2	6.6
$$\text{KV}_{image} \propto m \cdot d \cdot (\text{\# cross-attention layers}) \cdot 2,$$9	37.4	5.4
$n_k$0	31.4	10.2

The same study reports a much higher clean baseline for LLaVA-1.5-13B, 87.4%, but also a much larger degradation under stronger perturbations, reaching a 36.0-point drop at $n_k$1 (Fox et al., 19 Dec 2025). The core comparative conclusion is therefore not that Llama 3.2 Vision-8B-2 is more accurate on clean VQA in this setup—it is not—but that it exhibits a smaller attack-induced accuracy drop, particularly at larger perturbation levels.

This finding directly addresses a recurring misconception in multimodal benchmarking: higher clean benchmark accuracy does not necessarily imply greater adversarial robustness. The authors explicitly treat robustness and raw task performance as distinct properties, and the Llama 3.2 Vision-8B-2 results are presented as an example of that dissociation (Fox et al., 19 Dec 2025).

5. Visual-token trimming and inference efficiency

Research on efficient LLaMA-3.2-Vision inference argues that cross-attention maps are structured and sparse: within a layer, attention concentrates on a small set of image-token indices, and across layers, salient positions are remarkably stable (Lee et al., 1 Apr 2025). On that basis, the paper introduces a training-free trimming method that prunes redundant visual features after the first cross-attention block and reuses the reduced visual state for subsequent cross-attention layers.

The per-head cumulative importance score for image token $n_k$2 is defined as

$n_k$3

where $n_k$4 is the attention weight from image token $n_k$5 to query token $n_k$6 in head $n_k$7 (Lee et al., 1 Apr 2025). For each head, the retained set is selected by top-$n_k$8:

$n_k$9

and the full retained feature set is the union across heads:

$\text{FLOPs}_{self} = 4nd^{2} + 2n^{2}d + 2ndm$0

The reported implementation trims visual tokens after the first cross-attention layer, updates the KV cache with the reduced key and value tensors, and then performs later cross-attention using only the compressed cache (Lee et al., 1 Apr 2025). No retraining, fine-tuning, or extra supervision is required.

The experiments in that paper are reported for Llama-3.2-Vision-Instruct 11B and 90B rather than the 8B-2 checkpoint specifically, but they are directly relevant to the family’s cross-attention design. At around 50% visual-feature retention, the paper reports benchmark parity and lower latency; for example, on an A100 80GB GPU, first-token latency for Llama-3.2-11B-Vision-Instruct decreases from 95.1 ms to 91.2 ms at batch 1 and from 3940.0 ms to 3165.5 ms at batch 32 when feature utilization is reduced to 50.9% (Lee et al., 1 Apr 2025). The gains grow with batch size, which is consistent with the paper’s claim that KV-cache savings compound across concurrent requests.

Because these latency measurements are not reported for Vision-8B-2 itself, they should not be read as direct 8B-2 benchmarks. However, the underlying bottleneck analysis is architectural rather than checkpoint-specific. This suggests that Llama 3.2 Vision-8B-2 inherits the same class of efficiency trade-offs characteristic of cross-attention-based Llama 3.2 Vision models.

6. Applications, derivatives, and model-family context

The published literature places Llama 3.2 Vision-8B-2 within a broader ecosystem of domain adaptation and derivative model development. One application paper explores parameter-efficient LoRA fine-tuning of the multimodal LLaMA 3.2 model for ECG image interpretation using ECGInstruct, a large-scale instruction dataset with 1 Million samples synthesized from MIMIC-IV ECG and PTB-XL (M et al., 30 Jan 2025). In that setting, the fine-tuning procedure updates only a small set of parameters while specifically ignoring the lm_head and embed_tokens layers, and the abstract reports performance comparable to or exceeding traditional CNN-based methods across over 70 conditions from the PTB-XL dataset. The available source does not provide full paper text, so this evidence should be treated as an application-level indication of the LLaMA 3.2 multimodal stack’s adaptability rather than a checkpoint-specific characterization of Vision-8B-2.

A separate line of work, Breeze2, illustrates how the Llama 3.2 model family serves as a substrate for downstream multimodal systems focused on Traditional Chinese (Research et al., 23 Jan 2025). Breeze2 Vision-8B-2 is built on Llama 3.1 8B and adopts a ViT-MLP-LLM architecture with an InternViT-300M-448px vision encoder and an MLP projector, alongside prompt-based function calling and 128k long-context support. Its multimodal fusion mechanism is therefore distinct from the cross-attention adapter reported for Meta’s Llama 3.2 Vision-8B-2. This distinction is important because model names that share the “Vision-8B-2” pattern need not share the same internal vision-text coupling mechanism.

The broader significance of these adaptations is methodological rather than merely nominal. They show that the Llama 3.2 family is being used both as a base for specialized visual reasoning tasks and as an infrastructural foundation for multilingual, tool-augmented derivatives. At the same time, they caution against collapsing all Llama-derived vision systems into a single architecture class.

7. Interpretation and open technical questions

Across the available papers, three technical themes define Llama 3.2 Vision-8B-2. First, it is an adapter-based multimodal extension of Llama 3.1 8B, with visual information integrated through periodically inserted cross-attention layers rather than a projection-only bridge (Fox et al., 19 Dec 2025). Second, this design appears to produce a distinct robustness profile: lower clean VQA accuracy than LLaVA-1.5-13B in the cited setup, but a smaller accuracy drop under strong white-box PGD perturbations (Fox et al., 19 Dec 2025). Third, the same cross-attention architecture creates an inference bottleneck in the visual KV cache, motivating trimming methods that exploit sparsity in cross-attention maps to reduce memory and latency without retraining (Lee et al., 1 Apr 2025).

Several open questions remain. The robustness paper hypothesizes that relative robustness may stem from a combination of cross-attention fusion, much larger multimodal training scale, a richer post-training alignment pipeline, and the ViT-H/14 encoder, but it does not establish causal attribution (Fox et al., 19 Dec 2025). The efficiency paper shows that attention-guided visual token pruning is markedly better than random or fixed-pattern selection at 50% compression, but the strongest latency numbers are reported for larger family members rather than the 8B-2 checkpoint (Lee et al., 1 Apr 2025). In addition, detailed pretraining hyperparameters for the base model are not reported in the cited sources (Fox et al., 19 Dec 2025).

Taken together, the published evidence supports a precise characterization of Llama 3.2 Vision-8B-2 as a cross-attention-based, open-weight multimodal foundation model whose main research significance lies in the interaction between structured vision-language fusion, large-scale adapter-centric training, nontrivial adversarial behavior, and a visual KV-cache bottleneck that has become a target for inference-time optimization.