LLaVA-1.5-13B: Enhanced Multimodal Model

• LLaVA-1.5-13B is a 13B-parameter variant that integrates a CLIP-ViT-L-336px vision encoder with a two-layer MLP connector to fuse visual and textual data.
• It employs an extensive training recipe using 558K pretraining samples and 665K instruction-tuning samples, achieving robust performance on diverse academic and instruction-following benchmarks.
• Efficiency methods like Spec-LLaVA and E-AdaPrune enable notable speedups and resource reductions while preserving output quality.

LLaVA-1.5-13B is the 13B-parameter variant of the LLaVA-1.5 family introduced in “Improved Baselines with Visual Instruction Tuning,” where it is defined by three principal architectural and training changes relative to earlier LLaVA baselines: the use of CLIP-ViT-L-336px, a two-layer MLP vision-language connector, and an expanded visual instruction tuning mixture with academic-task-oriented VQA/OCR/region-level datasets and explicit response formatting prompts. In the original report, the final 13B checkpoint uses 558K pretraining samples and 665K instruction-tuning samples, relies only on publicly available data, and completes full training in about 1 day on a single 8-A100 node (Liu et al., 2023).

1. Definition and lineage

LLaVA-1.5-13B is the final 13B checkpoint of the improved LLaVA baseline. The originating paper positions it as a strengthened version of LLaVA rather than a departure from the basic LLaVA recipe: a pretrained vision encoder, a lightweight connector, and a LLM jointly optimized by visual instruction tuning. Its defining modifications are a higher-resolution visual backbone, an MLP projector in place of the original linear projector, and an expanded instruction-tuning corpus with output-format prompts tailored to heterogeneous tasks such as short-answer VQA, multiple-choice reasoning, OCR, and region grounding (Liu et al., 2023).

Subsequent papers describe the same backbone in closely aligned terms. TG-LLaVA characterizes the baseline as CLIP ViT-L/14-336px plus an MLP connector/projector and Vicuna-13B; LLaVA-c describes the 13B configuration as Vicuna-13B, CLIP-L/14@336, and an MLP projection layer; Spec-LLaVA treats LLaVA-1.5-13B as a standard LLaVA-1.5 target VLM with a CLIP ViT-L/14 vision encoder and a LLaMA-based language decoder of 13B parameters (Yan et al., 2024, Liu et al., 10 Jun 2025, Huo et al., 15 Sep 2025). This convergence of descriptions suggests that LLaVA-1.5-13B became a stable reference configuration for later work on inference acceleration, token reduction, preference alignment, and task-specific fine-tuning.

2. Core architecture and training recipe

At the architectural level, LLaVA-1.5-13B combines CLIP-ViT-L-336px as the vision encoder, a 2-layer MLP vision-language connector, and Vicuna v1.5 13B as the language backbone. The original LLaVA connector was a single linear layer, whereas LLaVA-1.5 replaces it with a two-layer MLP. The paper does not numerically specify the internal CLIP feature dimensionality, the MLP hidden width, the connector output dimension, or the activation function; it specifies only the component identities, the image resolution of $336 \times 336$, and the use of a fully connected connector architecture (Liu et al., 2023).

Training follows the standard two-stage LLaVA procedure. The pretraining stage uses 558K samples with batch size 256, learning rate $1 \times 10^{-3}$, cosine decay, warmup ratio 0.03, weight decay 0, 1 epoch, AdamW, and DeepSpeed stage 2. The visual instruction tuning stage uses 665K samples with batch size 128, learning rate $2 \times 10^{-5}$, cosine decay, warmup ratio 0.03, weight decay 0, 1 epoch, AdamW, and DeepSpeed stage 3. The reported compute footprint is about 6 hours of pretraining plus 20 hours of visual instruction tuning on 8 × A100s (Liu et al., 2023).

The instruction-tuning mixture is explicitly enumerated as LLaVA 158K, ShareGPT 40K, VQAv2 83K, GQA 72K, OKVQA 9K, OCRVQA 80K, A-OKVQA 66K, TextCaps 22K, RefCOCO 48K, and VG 86K, for a total of 665K samples. Response-format prompts are central to the design: VQAv2-style tasks use “Answer the question using a single word or phrase.”; A-OKVQA uses “Answer with the option's letter from the given choices directly.”; TextCaps uses “Provide a one-sentence caption for the provided image.”; and RefCOCO/VG alternate between region description and bounding-box prompting. Preprocessing choices reported in the paper include merging QA pairs from the same training image into a single conversation, truncating rather than splitting ShareGPT conversations longer than 2048 tokens, augmenting A-OKVQA by the number of answer choices, sampling 80K OCRVQA conversations, subsampling 10 annotations for over-annotated Visual Genome images, and sampling each batch from a single modality to obtain a reported 25% training speedup with no effect on final outcome (Liu et al., 2023).

3. Baseline capability profile

The original benchmark profile of LLaVA-1.5-13B spans both academic-task-oriented evaluations and instruction-following multimodal benchmarks. On the academic set, the reported scores are 80.0 on VQAv2, 63.3 on GQA, 53.6 on VizWiz, 71.6 on ScienceQA-IMG, and 61.3 on TextVQA. On the instruction-following side, the reported results are 87.1 on POPE-rand F1, 86.2 on POPE-pop F1, 84.5 on POPE-adv F1, 1531.3 on MME, 67.7 on MMBench-en, 63.6 on MMBench-cn, 61.6 on SEED-Bench all, 68.2 on SEED-Bench img, 42.7 on SEED-Bench vid, 72.5 on LLaVA-Bench-in-the-Wild, and 36.1 on MM-Vet (Liu et al., 2023).

Benchmark group	Metric or benchmark	LLaVA-1.5-13B
Academic	VQAv2	80.0
Academic	GQA	63.3
Academic	VizWiz	53.6
Academic	ScienceQA-IMG	71.6
Academic	TextVQA	61.3
Instruction-following	POPE-rand F1	87.1
Instruction-following	POPE-pop F1	86.2
Instruction-following	POPE-adv F1	84.5
Instruction-following	MME	1531.3
Instruction-following	MMBench-en	67.7
Instruction-following	MMBench-cn	63.6
Instruction-following	SEED-Bench all	61.6
Instruction-following	SEED-Bench img	68.2
Instruction-following	SEED-Bench vid	42.7
Instruction-following	LLaVA-Bench-in-the-Wild	72.5
Instruction-following	MM-Vet	36.1

The accompanying ablation sequence clarifies how these results were assembled. The paper reports cumulative gains from adding VQAv2, response formatting prompts, the MLP vision-language connector, OKVQA/OCR, region-level VQA, scaling resolution from 224 to 336, adding GQA, adding ShareGPT, and finally scaling the LLM from 7B to 13B. In this sequence, the move from LLaVA-1.5-7B to LLaVA-1.5-13B raises GQA from 62.0 to 63.3, MME from 1510.7 to 1531.3, and MM-Vet from 31.1 to 36.1, indicating that the 13B scaling is especially consequential for open-ended visual conversation and complex multimodal dialogue (Liu et al., 2023).

4. Efficiency-oriented reinterpretations

A substantial line of later work treats LLaVA-1.5-13B not as an endpoint, but as a systems baseline to be accelerated or compressed. Spec-LLaVA instantiates a lossless acceleration framework by pairing LLaVA-1.5-13B as the target model with lightweight draft VLMs, LLaVA-68M and LLaVA-160M, that share the same CLIP ViT-L/14 encoder so that image encoding is not repeated. The method uses dynamic tree-based speculative decoding with confidence-controlled expansion and pruning, followed by conservative exact-match verification by the target model. For LLaVA-1.5-13B specifically, the reported acceptance length is 2.1 with the 68M draft and 3.0 with the 160M draft, corresponding to 2.12× and 2.95× speedups, respectively, while preserving outputs exactly under greedy decoding. The broader abstract reports up to 3.28× speedup across the 7B and 13B settings, but the 13B table-specific peak is 2.95× (Huo et al., 15 Sep 2025).

E-AdaPrune addresses efficiency at the visual-token level. It is presented as a training-free, plug-and-play, model-agnostic, and text-agnostic budget-selection mechanism that computes an image-dependent token budget from the singular value spectrum of the visual feature matrix and then feeds that budget into an existing pruning method. For LLaVA-1.5-13B, the paper integrates E-AdaPrune with FastV, with pruning after layer 2. At $\tau = 99.8\%$, the reported average token budget is 290, and normalized average performance improves from 99.2% with static FastV to 99.4% with FastV+E; at $\tau = 99.0\%$, the average budget is 156, and normalized average performance improves from 97.5% to 97.8%. The paper further reports that using randomized SVD limits the added latency to 8 ms per image (He et al., 6 Mar 2026).

PruMerge performs a more aggressive post-encoder token reduction. In its LLaVA-1.5-13B insertion, 576 visual tokens produced from a $336 \times 336$ image are adaptively reduced after the CLIP ViT-L encoder and before the projector and Vicuna-13B. It uses CLS-to-patch attention sparsity with IQR thresholding to retain salient tokens and then supplements them by key-similarity clustering and weighted merging. The headline configuration compresses the visual tokens to roughly 32 tokens average, about 5.5% of the original token count. In FP16, the reported resource changes are FLOPs 18.2T $\rightarrow$ 1.80T, prefill time 170.5 ms $\rightarrow$ 29.5 ms, total memory 41.6 GB $\rightarrow$ 26.6 GB, and activation storage 7.30 GB $\rightarrow$ 0.44 GB, accompanied by task-dependent accuracy reductions; the PruMerge+ variant keeps a larger token budget and recovers more accuracy (Shang et al., 2024).

Method	Mechanism on LLaVA-1.5-13B	Reported effect
Spec-LLaVA	Dynamic tree-based speculative decoding with 68M/160M draft VLMs	2.12× or 2.95× speedup; exact greedy output preservation
E-AdaPrune	Spectral-energy adaptive visual token budgeting with FastV	99.2% $1 \times 10^{-3}$0 99.4% at average 290 tokens; 97.5% $1 \times 10^{-3}$1 97.8% at average 156 tokens; 8 ms/image overhead
PruMerge	Post-encoder adaptive token selection plus merging	Roughly 32 tokens average; strong FLOP, memory, and prefill reductions with some accuracy drop

5. Architectural extensions and domain-specialized derivatives

Several later models retain LLaVA-1.5-13B as the base backbone but alter what information reaches the LLM. TG-LLaVA adds text-conditioned guidance directly to the visual branch through TG-FOM, a global text-guided feature optimization mask, and TG-DP, a detail perceiver that extracts instruction-relevant information from high-resolution local patches. The base system remains CLIP ViT-L/14-336px, an MLP connector, and Vicuna-13B. The paper reports that the added parameters are roughly 0.18B relative to LLaVA-1.5 and that training time increases by about 10%. On the ten reported benchmarks, the Vicuna-13B setting improves from 64.0 to 65.3 on MMBench, 34.3 to 35.9 on MMStar, 37.0 to 39.6 on MMMU, 33.7 to 34.4 on OCRBench, 24.5 to 25.9 on HallusionBench, 66.1 to 67.8 on LLaVABench, 72.6 to 72.7 on ScienceQA, and 1781 to 1858 on MME, with small regressions of -0.1 on MathVista and AI2D. The paper summarizes this as about 1.0% average improvement across the ten datasets (Yan et al., 2024).

Math-LLaVA is explicitly data-centric rather than architectural. It uses LLaVA-1.5-13B as the base model, keeps the standard pretrained ViT, linear projector, and Vicuna-v1.5-13B decoder arrangement, and fine-tunes on MathV360K, a dataset consisting of 40K selected real image-QA pairs from 24 source datasets and 320K synthesized QA pairs. The paper states that the projection linear layer is trainable and the LLM is trainable, but reports no added architectural block. On MathVista minitest, Math-LLaVA reaches 46.6%, compared with 27.6 for LLaVA-1.5-13B, a gain of 19.0 points; on MMMU, it reaches 38.3 versus 36.4, a gain of 1.9 (Shi et al., 2024).

MM-PhyQA supplies a further example of domain-specific adaptation. It fine-tunes LLaVA-1.5 13B on a high-school-level multimodal physics benchmark and introduces Multi-Image Chain-of-Thought (MI-CoT) prompting, in which two solved multimodal exemplars and the target question are paired with three vertically stacked images. The paper does not modify the LLaVA-1.5 transformer internals; the reported changes are LoRA adaptation, multi-image prompt construction, and an autoencoder-based resizing step for stacked images. The best reported configuration is LLaVA-1.5 13B + MI-CoT + LoRA rank 128, with Accuracy = 71.65% on the test set, alongside Rouge1 0.677, Rouge2 0.582, and RougeL 0.650 (Anand et al., 2024).

6. Preference alignment, continual learning, and robustness

Preference alignment work has also taken LLaVA-1.5-13B as a base checkpoint. VaPR uses LLaVA-1.5-Instruct-13B as the starting model and applies LoRA plus Direct Preference Optimization on a 30K synthetic preference dataset whose rejected responses are minimally edited hard negatives designed to preserve style and approximate length while changing task-relevant semantics. The paper reports LoRA rank $1 \times 10^{-3}$2, LoRA alpha 256, DPO $1 \times 10^{-3}$3, max sequence length 2048, learning rate $1 \times 10^{-3}$4, effective batch size 32, 5 epochs, and 2 A100 GPUs. Relative to the base LLaVA-1.5-13B, the VaPR-aligned model improves LLaVA$1 \times 10^{-3}$5 from 72.3 to 80.5, ConTextual from 18.6 to 21.2, MM-VET from 36.7 to 37.3, SEED$1 \times 10^{-3}$6 from 68.2 to 68.7, CV-Bench from 62.5 to 64.6, MathVista from 30.7 to 32.3, MMStar from 33.8 to 35.6, POPE from 86.0 to 86.3, and NaturalBench from 14.9 to 18.2, while MMMU changes from 36.1 to 35.8. The paper also states that VaPR reduces the tendency to answer “Yes” in binary questions and that this effect is most pronounced for LLaVA-VaPR-13B (Wadhawan et al., 2 Oct 2025).

Continual-learning and sparse-adaptation studies are more limited for the 13B configuration. LLaVA-c explicitly states that it does not report dedicated experiments for LLaVA-1.5-13B / LLaVA-c-13B. What it provides for the 13B variant is architectural inheritance: Vicuna-13B, CLIP-L/14@336, an MLP projection layer, unchanged stage-1 alignment, and stage-2 visual instruction tuning on LLaVA-665k. Its two proposed additions, spectral-aware consolidation and unsupervised inquiry regularization, are therefore architecturally applicable to LLaVA-1.5-13B, but the paper does not supply 13B-specific benchmarks or ablations (Liu et al., 10 Jun 2025). LLaVA-MoLE is similar in this respect: it discusses the LLaVA-1.5 architecture pattern but reports experiments only with Vicuna-7B-v1.5, not a 13B backbone. A common misconception is therefore that all LLaVA-1.5 adaptation papers provide 13B-specific evidence; several do not, and the distinction between architectural applicability and reported 13B results is material (Chen et al., 2024).

Adversarial evaluation supplies a complementary picture. A dedicated robustness study tests LLaVA-1.5-13B under untargeted white-box PGD attacks on the visual input using a 500-sample subset of VQA v2 validation. In that study, the model is specified as CLIP ViT-L/14 plus a trainable projection layer and Vicuna-13B, loaded as llava-hf/llava-1.5-13b-hf in float16 on an NVIDIA A100 80GB. The reported clean accuracy is 87.4%, and adversarial accuracies are 80.4 at $1 \times 10^{-3}$7, 80.6 at $1 \times 10^{-3}$8, 79.0 at $1 \times 10^{-3}$9, 76.8 at $2 \times 10^{-5}$0, 67.4 at $2 \times 10^{-5}$1, and 51.4 at $2 \times 10^{-5}$2. The paper’s central conclusion is that LLaVA-1.5-13B retains higher absolute accuracy than the compared Llama 3.2 Vision-8B-2 at every perturbation level, but suffers a larger accuracy drop at moderate-to-high perturbation budgets, showing that strong clean-task performance does not directly imply strong adversarial robustness (Fox et al., 19 Dec 2025).

Taken together, these later studies position LLaVA-1.5-13B less as a closed model artifact than as a reusable multimodal reference point. Some lines of work leave its outputs unchanged while reducing latency; others alter the visual token budget, the visual feature pathway, the preference objective, or the task-specific supervision. A plausible implication is that the enduring value of LLaVA-1.5-13B lies in the combination of a simple base architecture, a fully specified training recipe, and sufficiently strong baseline performance to make it a stable substrate for systems, alignment, and robustness research (Liu et al., 2023).