---
title: 'LLaVA-1.5-13B: Enhanced Multimodal Model'
url: https://www.emergentmind.com/topics/llava-1-5-13b
type: topic
---

# LLaVA-1.5-13B: Enhanced Multimodal Model

LLaVA-1.5-13B is the 13B-parameter variant of the LLaVA-1.5 family introduced in “Improved Baselines with Visual Instruction Tuning,” where it is defined by three principal architectural and training changes relative to earlier LLaVA baselines: the use of **CLIP-ViT-L-336px**, a **two-layer MLP** vision-language connector, and an expanded **visual instruction tuning** mixture with **academic-task-oriented VQA/OCR/region-level datasets** and explicit **response formatting prompts**. In the original report, the final 13B checkpoint uses **558K pretraining samples** and **665K instruction-tuning samples**, relies only on **publicly available data**, and completes full training in about **1 day on a single 8-A100 node** [2310.03744].

## 1. Definition and lineage

LLaVA-1.5-13B is the final **13B** checkpoint of the improved LLaVA baseline. The originating paper positions it as a strengthened version of LLaVA rather than a departure from the basic LLaVA recipe: a pretrained vision encoder, a lightweight connector, and a large language model jointly optimized by visual instruction tuning. Its defining modifications are a higher-resolution visual backbone, an MLP projector in place of the original linear projector, and an expanded instruction-tuning corpus with output-format prompts tailored to heterogeneous tasks such as short-answer VQA, multiple-choice reasoning, OCR, and region grounding [2310.03744].

Subsequent papers describe the same backbone in closely aligned terms. TG-LLaVA characterizes the baseline as **CLIP ViT-L/14-336px** plus an **MLP connector/projector** and **Vicuna-13B**; LLaVA-c describes the 13B configuration as **Vicuna-13B**, **CLIP-L/14@336**, and an **MLP projection layer**; Spec-LLaVA treats LLaVA-1.5-13B as a standard LLaVA-1.5 target VLM with a **CLIP ViT-L/14 vision encoder** and a **LLaMA-based language decoder of 13B parameters** [2409.09564] [2506.08666] [2509.11961]. This convergence of descriptions suggests that LLaVA-1.5-13B became a stable reference configuration for later work on inference acceleration, token reduction, preference alignment, and task-specific fine-tuning.

## 2. Core architecture and training recipe

At the architectural level, LLaVA-1.5-13B combines **CLIP-ViT-L-336px** as the vision encoder, a **2-layer MLP vision-language connector**, and **Vicuna v1.5 13B** as the language backbone. The original LLaVA connector was a single linear layer, whereas LLaVA-1.5 replaces it with a two-layer MLP. The paper does not numerically specify the internal CLIP feature dimensionality, the MLP hidden width, the connector output dimension, or the activation function; it specifies only the component identities, the image resolution of **\(336 \times 336\)**, and the use of a fully connected connector architecture [2310.03744].

Training follows the standard two-stage LLaVA procedure. The pretraining stage uses **558K** samples with **batch size 256**, **learning rate \(1 \times 10^{-3}\)**, **cosine decay**, **warmup ratio 0.03**, **weight decay 0**, **1 epoch**, **AdamW**, and **DeepSpeed stage 2**. The visual instruction tuning stage uses **665K** samples with **batch size 128**, **learning rate \(2 \times 10^{-5}\)**, **cosine decay**, **warmup ratio 0.03**, **weight decay 0**, **1 epoch**, **AdamW**, and **DeepSpeed stage 3**. The reported compute footprint is about **6 hours of pretraining** plus **20 hours of visual instruction tuning** on **8 × A100s** [2310.03744].

The instruction-tuning mixture is explicitly enumerated as **LLaVA 158K**, **ShareGPT 40K**, **VQAv2 83K**, **GQA 72K**, **OKVQA 9K**, **OCRVQA 80K**, **A-OKVQA 66K**, **TextCaps 22K**, **RefCOCO 48K**, and **VG 86K**, for a total of **665K** samples. Response-format prompts are central to the design: VQAv2-style tasks use **“Answer the question using a single word or phrase.”**; A-OKVQA uses **“Answer with the option's letter from the given choices directly.”**; TextCaps uses **“Provide a one-sentence caption for the provided image.”**; and RefCOCO/VG alternate between region description and bounding-box prompting. Preprocessing choices reported in the paper include merging QA pairs from the same training image into a single conversation, truncating rather than splitting ShareGPT conversations longer than **2048** tokens, augmenting A-OKVQA by the number of answer choices, sampling **80K** OCRVQA conversations, subsampling **10** annotations for over-annotated Visual Genome images, and sampling each batch from a single modality to obtain a reported **25%** training speedup with no effect on final outcome [2310.03744].

## 3. Baseline capability profile

The original benchmark profile of LLaVA-1.5-13B spans both academic-task-oriented evaluations and instruction-following multimodal benchmarks. On the academic set, the reported scores are **80.0** on **VQAv2**, **63.3** on **GQA**, **53.6** on **VizWiz**, **71.6** on **ScienceQA-IMG**, and **61.3** on **TextVQA**. On the instruction-following side, the reported results are **87.1** on **POPE-rand F1**, **86.2** on **POPE-pop F1**, **84.5** on **POPE-adv F1**, **1531.3** on **MME**, **67.7** on **MMBench-en**, **63.6** on **MMBench-cn**, **61.6** on **SEED-Bench all**, **68.2** on **SEED-Bench img**, **42.7** on **SEED-Bench vid**, **72.5** on **LLaVA-Bench-in-the-Wild**, and **36.1** on **MM-Vet** [2310.03744].

| Benchmark group | Metric or benchmark | LLaVA-1.5-13B |
|---|---|---:|
| Academic | VQAv2 | 80.0 |
| Academic | GQA | 63.3 |
| Academic | VizWiz | 53.6 |
| Academic | ScienceQA-IMG | 71.6 |
| Academic | TextVQA | 61.3 |
| Instruction-following | POPE-rand F1 | 87.1 |
| Instruction-following | POPE-pop F1 | 86.2 |
| Instruction-following | POPE-adv F1 | 84.5 |
| Instruction-following | MME | 1531.3 |
| Instruction-following | MMBench-en | 67.7 |
| Instruction-following | MMBench-cn | 63.6 |
| Instruction-following | SEED-Bench all | 61.6 |
| Instruction-following | SEED-Bench img | 68.2 |
| Instruction-following | SEED-Bench vid | 42.7 |
| Instruction-following | LLaVA-Bench-in-the-Wild | 72.5 |
| Instruction-following | MM-Vet | 36.1 |

The accompanying ablation sequence clarifies how these results were assembled. The paper reports cumulative gains from adding **VQAv2**, response formatting prompts, the **MLP vision-language connector**, **OKVQA/OCR**, **region-level VQA**, scaling resolution from **224** to **336**, adding **GQA**, adding **ShareGPT**, and finally scaling the language model from **7B** to **13B**. In this sequence, the move from **LLaVA-1.5-7B** to **LLaVA-1.5-13B** raises **GQA** from **62.0** to **63.3**, **MME** from **1510.7** to **1531.3**, and **MM-Vet** from **31.1** to **36.1**, indicating that the 13B scaling is especially consequential for open-ended visual conversation and complex multimodal dialogue [2310.03744].

## 4. Efficiency-oriented reinterpretations

A substantial line of later work treats LLaVA-1.5-13B not as an endpoint, but as a systems baseline to be accelerated or compressed. Spec-LLaVA instantiates a **lossless acceleration framework** by pairing LLaVA-1.5-13B as the **target model** with lightweight draft VLMs, **LLaVA-68M** and **LLaVA-160M**, that share the same **CLIP ViT-L/14** encoder so that image encoding is not repeated. The method uses **dynamic tree-based speculative decoding** with confidence-controlled expansion and pruning, followed by conservative exact-match verification by the target model. For LLaVA-1.5-13B specifically, the reported acceptance length is **2.1** with the **68M draft** and **3.0** with the **160M draft**, corresponding to **2.12×** and **2.95×** speedups, respectively, while preserving outputs exactly under greedy decoding. The broader abstract reports up to **3.28×** speedup across the **7B** and **13B** settings, but the 13B table-specific peak is **2.95×** [2509.11961].

E-AdaPrune addresses efficiency at the visual-token level. It is presented as a **training-free**, **plug-and-play**, **model-agnostic**, and **text-agnostic** budget-selection mechanism that computes an image-dependent token budget from the singular value spectrum of the visual feature matrix and then feeds that budget into an existing pruning method. For LLaVA-1.5-13B, the paper integrates E-AdaPrune with **FastV**, with pruning **after layer 2**. At **\(\tau = 99.8\%\)**, the reported average token budget is **290**, and normalized average performance improves from **99.2%** with static FastV to **99.4%** with **FastV+E**; at **\(\tau = 99.0\%\)**, the average budget is **156**, and normalized average performance improves from **97.5%** to **97.8%**. The paper further reports that using randomized SVD limits the added latency to **8 ms per image** [2603.05950].

PruMerge performs a more aggressive post-encoder token reduction. In its LLaVA-1.5-13B insertion, **576 visual tokens** produced from a **\(336 \times 336\)** image are adaptively reduced after the **CLIP ViT-L** encoder and before the projector and **Vicuna-13B**. It uses **CLS-to-patch attention sparsity** with **IQR thresholding** to retain salient tokens and then supplements them by **key-similarity clustering and weighted merging**. The headline configuration compresses the visual tokens to roughly **32 tokens average**, about **5.5%** of the original token count. In FP16, the reported resource changes are **FLOPs 18.2T \(\rightarrow\) 1.80T**, **prefill time 170.5 ms \(\rightarrow\) 29.5 ms**, **total memory 41.6 GB \(\rightarrow\) 26.6 GB**, and **activation storage 7.30 GB \(\rightarrow\) 0.44 GB**, accompanied by task-dependent accuracy reductions; the **PruMerge+** variant keeps a larger token budget and recovers more accuracy [2403.15388].

| Method | Mechanism on LLaVA-1.5-13B | Reported effect |
|---|---|---|
| Spec-LLaVA | Dynamic tree-based speculative decoding with 68M/160M draft VLMs | 2.12× or 2.95× speedup; exact greedy output preservation |
| E-AdaPrune | Spectral-energy adaptive visual token budgeting with FastV | 99.2% \(\rightarrow\) 99.4% at average 290 tokens; 97.5% \(\rightarrow\) 97.8% at average 156 tokens; 8 ms/image overhead |
| PruMerge | Post-encoder adaptive token selection plus merging | Roughly 32 tokens average; strong FLOP, memory, and prefill reductions with some accuracy drop |

## 5. Architectural extensions and domain-specialized derivatives

Several later models retain LLaVA-1.5-13B as the base backbone but alter what information reaches the language model. TG-LLaVA adds text-conditioned guidance directly to the visual branch through **TG-FOM**, a global text-guided feature optimization mask, and **TG-DP**, a detail perceiver that extracts instruction-relevant information from high-resolution local patches. The base system remains **CLIP ViT-L/14-336px**, an **MLP connector**, and **Vicuna-13B**. The paper reports that the added parameters are roughly **0.18B** relative to LLaVA-1.5 and that training time increases by about **10%**. On the ten reported benchmarks, the **Vicuna-13B** setting improves from **64.0** to **65.3** on **MMBench**, **34.3** to **35.9** on **MMStar**, **37.0** to **39.6** on **MMMU**, **33.7** to **34.4** on **OCRBench**, **24.5** to **25.9** on **HallusionBench**, **66.1** to **67.8** on **LLaVABench**, **72.6** to **72.7** on **ScienceQA**, and **1781** to **1858** on **MME**, with small regressions of **-0.1** on **MathVista** and **AI2D**. The paper summarizes this as about **1.0% average improvement across the ten datasets** [2409.09564].

Math-LLaVA is explicitly data-centric rather than architectural. It uses **LLaVA-1.5-13B** as the base model, keeps the standard **pretrained ViT**, **linear projector**, and **Vicuna-v1.5-13B** decoder arrangement, and fine-tunes on **MathV360K**, a dataset consisting of **40K** selected real image-QA pairs from **24** source datasets and **320K** synthesized QA pairs. The paper states that **the projection linear layer is trainable** and **the language model is trainable**, but reports **no added architectural block**. On **MathVista minitest**, Math-LLaVA reaches **46.6%**, compared with **27.6** for **LLaVA-1.5-13B**, a gain of **19.0 points**; on **MMMU**, it reaches **38.3** versus **36.4**, a gain of **1.9** [2406.17294].

MM-PhyQA supplies a further example of domain-specific adaptation. It fine-tunes **LLaVA-1.5 13B** on a high-school-level multimodal physics benchmark and introduces **Multi-Image Chain-of-Thought (MI-CoT)** prompting, in which two solved multimodal exemplars and the target question are paired with three vertically stacked images. The paper does not modify the LLaVA-1.5 transformer internals; the reported changes are **LoRA** adaptation, multi-image prompt construction, and an autoencoder-based resizing step for stacked images. The best reported configuration is **LLaVA-1.5 13B + MI-CoT + LoRA rank 128**, with **Accuracy = 71.65%** on the test set, alongside **Rouge1 0.677**, **Rouge2 0.582**, and **RougeL 0.650** [2404.08704].

## 6. Preference alignment, continual learning, and robustness

Preference alignment work has also taken LLaVA-1.5-13B as a base checkpoint. VaPR uses **LLaVA-1.5-Instruct-13B** as the starting model and applies **LoRA** plus **Direct Preference Optimization** on a **30K** synthetic preference dataset whose rejected responses are minimally edited hard negatives designed to preserve style and approximate length while changing task-relevant semantics. The paper reports **LoRA rank \(128\)**, **LoRA alpha 256**, **DPO \(\beta = 0.1\)**, **max sequence length 2048**, **learning rate \(1 \times 10^{-6}\)**, **effective batch size 32**, **5 epochs**, and **2 A100 GPUs**. Relative to the base LLaVA-1.5-13B, the VaPR-aligned model improves **LLaVA\(^W\)** from **72.3** to **80.5**, **ConTextual** from **18.6** to **21.2**, **MM-VET** from **36.7** to **37.3**, **SEED\(^I\)** from **68.2** to **68.7**, **CV-Bench** from **62.5** to **64.6**, **MathVista** from **30.7** to **32.3**, **MMStar** from **33.8** to **35.6**, **POPE** from **86.0** to **86.3**, and **NaturalBench** from **14.9** to **18.2**, while **MMMU** changes from **36.1** to **35.8**. The paper also states that VaPR reduces the tendency to answer **“Yes”** in binary questions and that this effect is most pronounced for **LLaVA-VaPR-13B** [2510.01700].

Continual-learning and sparse-adaptation studies are more limited for the 13B configuration. LLaVA-c explicitly states that it does **not** report dedicated experiments for **LLaVA-1.5-13B / LLaVA-c-13B**. What it provides for the 13B variant is architectural inheritance: **Vicuna-13B**, **CLIP-L/14@336**, an **MLP projection layer**, unchanged stage-1 alignment, and stage-2 visual instruction tuning on **LLaVA-665k**. Its two proposed additions, **spectral-aware consolidation** and **unsupervised inquiry regularization**, are therefore architecturally applicable to LLaVA-1.5-13B, but the paper does not supply 13B-specific benchmarks or ablations [2506.08666]. LLaVA-MoLE is similar in this respect: it discusses the LLaVA-1.5 architecture pattern but reports experiments only with **Vicuna-7B-v1.5**, not a 13B backbone. A common misconception is therefore that all LLaVA-1.5 adaptation papers provide 13B-specific evidence; several do not, and the distinction between architectural applicability and reported 13B results is material [2401.16160].

Adversarial evaluation supplies a complementary picture. A dedicated robustness study tests **LLaVA-1.5-13B** under untargeted white-box **PGD** attacks on the visual input using a **500-sample subset of VQA v2 validation**. In that study, the model is specified as **CLIP ViT-L/14** plus a **trainable projection layer** and **Vicuna-13B**, loaded as **`llava-hf/llava-1.5-13b-hf`** in **float16** on an **NVIDIA A100 80GB**. The reported clean accuracy is **87.4%**, and adversarial accuracies are **80.4** at **\(2/255\)**, **80.6** at **\(4/255\)**, **79.0** at **\(8/255\)**, **76.8** at **\(16/255\)**, **67.4** at **\(128/255\)**, and **51.4** at **\(255/255\)**. The paper’s central conclusion is that LLaVA-1.5-13B retains higher absolute accuracy than the compared **Llama 3.2 Vision-8B-2** at every perturbation level, but suffers a larger accuracy drop at moderate-to-high perturbation budgets, showing that strong clean-task performance does not directly imply strong adversarial robustness [2512.17902].

Taken together, these later studies position LLaVA-1.5-13B less as a closed model artifact than as a reusable multimodal reference point. Some lines of work leave its outputs unchanged while reducing latency; others alter the visual token budget, the visual feature pathway, the preference objective, or the task-specific supervision. A plausible implication is that the enduring value of LLaVA-1.5-13B lies in the combination of a simple base architecture, a fully specified training recipe, and sufficiently strong baseline performance to make it a stable substrate for systems, alignment, and robustness research [2310.03744].

Source: https://www.emergentmind.com/topics/llava-1-5-13b