MetaVL: In-Context Vision-Language Learning

• MetaVL is a vision–language model that transfers in-context learning from a meta-trained language model to a multimodal setting by aligning visual features with token embeddings.
• It employs a two-stage training process: first, meta-training on a vast set of NLP tasks, and then adapting a visual encoder using a projection layer on COCO captions.
• Empirical results on VQA, OK-VQA, and GQA show that MetaVL achieves competitive few-shot performance with significantly smaller models compared to larger, conventional multimodal systems.

Searching arXiv for the primary paper and closely related works cited in the provided data. {"query":"arXiv (Monajatipoor et al., 2023) MetaVL Transferring In-Context Learning Ability From LLMs to Vision-LLMs"} {"query":"MetaICL arXiv Min 2022 Meta-trained In-Context Learning"} {"query":"site:arxiv.org MetaICL Learning to Learn In Context arXiv"} {"query":"MAGMA foundation model multimodal arXiv Eichenberg 2021"} MetaVL is a vision–LLM designed to exhibit strong in-context learning by transferring in-context learning capabilities from a meta-trained LLM into a multimodal setting. In the formulation introduced in "MetaVL: Transferring In-Context Learning Ability From LLMs to Vision-LLMs" (Monajatipoor et al., 2023), the central claim is that a LLM explicitly meta-trained to solve new tasks from a few textual demonstrations can retain that capability after a visual encoder is attached and aligned to its token space. The resulting system targets few-shot visual question answering without gradient updates at test time and is positioned as an alternative to multimodal in-context learning systems that rely on very large language backbones or massive multimodal corpora.

1. Problem setting and central hypothesis

MetaVL addresses a specific gap between LLMs and most pre-trained vision–LLMs. In language, in-context learning refers to the setting in which the model is given a support set of $k$ labeled examples,

$(x_1, y_1), (x_2, y_2), \dots, (x_k, y_k),$

followed by a query input $x_{k+1}$, and must predict $y_{k+1}$ without parameter updates. LLMs such as GPT-3 showed that this behavior can emerge at scale, while MetaICL showed that it can also be trained explicitly through meta-learning across many NLP tasks (Min et al., 2021).

The motivating observation for MetaVL is that most pre-trained vision–language systems do not natively support this mode of adaptation. Standard visual question answering models are optimized for fixed supervised objectives and usually require fine-tuning on new tasks or output formats. Promptable multimodal systems such as Frozen, MAGMA, and Flamingo can perform few-shot multimodal inference, but the provided description emphasizes that these systems typically depend on large frozen language backbones and/or extensive multimodal corpora containing in-context-style examples (Tsimpoukelli et al., 2021); (Alayrac et al., 2022).

MetaVL is built around a direct cross-modal transfer hypothesis: if a LLM is meta-trained specifically for in-context learning on many NLP tasks, then a vision–LLM created by attaching a visual encoder to that frozen model can inherit its in-context learning ability. The paper describes this design as essentially "MetaICL + vision bridge" (Monajatipoor et al., 2023). A plausible implication is that explicit meta-training can substitute, at least partly, for the extreme scale otherwise associated with multimodal in-context learning.

2. Architectural design

MetaVL consists of three components: a meta-trained language encoder–decoder, a visual encoder, and a visual prefix that projects image features into the LLM embedding space (Monajatipoor et al., 2023).

The language backbone is GPT-2 Medium, with approximately 375M parameters. In the description used for MetaVL, it is a Transformer-based encoder–decoder trained with Prefix Language Modeling. The language encoder maps text to token embeddings $t_1, t_2, \dots, t_m$, and the decoder autoregressively predicts text tokens. This backbone is first meta-trained on NLP tasks and then frozen during the vision–language stage to preserve its in-context learning ability.

The visual encoder is CLIP-RN50x16. Its output is taken before the pooling layer as a feature grid of shape $n \times D_v$, where $n = 144$ spatial positions and $D_v = 3072$. Each spatial feature is treated as a visual token. Formally, the encoder is a function $V_e(x)$ that maps an image $x$ to $(x_1, y_1), (x_2, y_2), \dots, (x_k, y_k),$0 features.

The visual prefix is an MLP implemented here as a single linear layer of size $(x_1, y_1), (x_2, y_2), \dots, (x_k, y_k),$1, with the target dimensionality $(x_1, y_1), (x_2, y_2), \dots, (x_k, y_k),$2 matching the GPT-2 embedding space. If the CLIP features are $(x_1, y_1), (x_2, y_2), \dots, (x_k, y_k),$3, the projection produces $(x_1, y_1), (x_2, y_2), \dots, (x_k, y_k),$4, which are then interpreted as pseudo-language tokens. Given an image and text, MetaVL forms a multimodal sequence

$(x_1, y_1), (x_2, y_2), \dots, (x_k, y_k),$5

The decoder is trained on the text portion of this sequence with the objective

$(x_1, y_1), (x_2, y_2), \dots, (x_k, y_k),$6

During vision–language training, gradients update only the visual encoder $(x_1, y_1), (x_2, y_2), \dots, (x_k, y_k),$7 and visual prefix $(x_1, y_1), (x_2, y_2), \dots, (x_k, y_k),$8, while the LLM remains frozen. The architecture is therefore intentionally simple: a frozen meta-trained LLM, a trainable CLIP image encoder, and a simple projection layer, without additional architectural tricks beyond the PrefixLM-based alignment objective (Monajatipoor et al., 2023).

3. Two-stage training procedure

MetaVL is trained in two stages. The first stage is language-only meta-training, inherited from MetaICL. The second stage transfers the resulting capability into a vision–LLM by training only the visual side (Monajatipoor et al., 2023).

In the first stage, the model is meta-trained over many NLP tasks. At each iteration, a task is sampled from a large pool of datasets, and $(x_1, y_1), (x_2, y_2), \dots, (x_k, y_k),$9 examples are drawn:

$x_{k+1}$0

The first $x_{k+1}$1 examples serve as demonstrations, and the model is trained to maximize

$x_{k+1}$2

This is episodic meta-learning: the model is not updated on the query instance itself, but instead learns to infer task structure from the support examples. MetaVL adopts the MetaICL "HR→LR with instructions (all)" setup over 142 NLP datasets, where datasets with at least 10,000 training examples serve as meta-training tasks and smaller datasets are used as target tasks for evaluation in the MetaICL framework (Min et al., 2021).

The language meta-training implementation uses GPT-2 Medium, a learning rate of $x_{k+1}$3, batch size 8, and 80,000 steps on 8 NVIDIA RTX A6000 GPUs, with a reported training time of approximately 6 hours.

In the second stage, the meta-trained language encoder–decoder is completely frozen. The visual encoder and visual prefix are then trained on COCO captions using the language-modeling objective over text tokens conditioned on image-derived visual tokens. The rationale given in the paper is that freezing the LLM preserves its meta-learned in-context learning knowledge, while the visual side learns to represent images in a form the LLM can already exploit (Monajatipoor et al., 2023).

The vision–language stage uses the COCO training split, CLIP-RN50x16 as the visual encoder, the $x_{k+1}$4 visual prefix, learning rate $x_{k+1}$5 for the prefix, learning rate $x_{k+1}$6 for the visual encoder, batch size 32, and 8 epochs on 4 A6000 GPUs, with a reported training time of approximately 48 hours.

4. In-context prompting and evaluation protocol

At inference time, MetaVL casts vision–language tasks into the same in-context format used during language meta-training. For a task such as visual question answering, each example is an image, question, answer triple. A prompt is constructed from a task induction sentence, $x_{k+1}$7 support examples taken from the training set, and a query example consisting of an image and question without the answer (Monajatipoor et al., 2023).

The support examples and query are represented multimodally: each support image is converted into visual tokens, and each support question and answer is tokenized as text; the query contributes visual tokens and question text only. The model then generates the answer as a completion. The conditional form is given as

$x_{k+1}$8

where $x_{k+1}$9 is the support set of multimodal examples.

The evaluation focuses on three visual question answering datasets: VQA, OK-VQA, and GQA. The setup uses 0-, 1-, 2-, and 3-shot in-context learning, with random sampling of demonstrations from the training set for each query. Because of computational constraints, evaluation is performed on 5,000 instances from each validation or test split (Monajatipoor et al., 2023).

Automatic evaluation uses accuracy with exact string match or nearest candidate answer via Sentence-BERT cosine similarity. The candidate set is defined as the distinct answers in the training set, reported as approximately 3,000 to 4,200 answers. The paper also reports human evaluation on 2,000 examples to account for semantically correct generations that do not exactly match a canonical candidate string.

Two baseline families are used. FrozenA is a re-implementation of Frozen with GPT-2 Medium, and FrozenB uses GPT-J 6B, approximately 6B parameters, as a much larger frozen language backbone (Tsimpoukelli et al., 2021). The experiments also include adapter variants, labeled Frozen w/adap and MetaVL w/adap, in the style associated in the description with MAGMA.

5. Empirical findings

The central empirical result is that in-context learning ability can be transferred from language-only meta-training into the vision–language setting. On 3-shot automatic evaluation over 5,000 test samples, FrozenA achieves 18.63 on VQA, 3.30 on OK-VQA, and 13.86 on GQA; FrozenB achieves 34.07, 11.97, and 25.76; MetaVL achieves 33.12, 9.60, and 31.96, respectively (Monajatipoor et al., 2023). Relative to FrozenA at the same 375M language-model scale, this corresponds to gains of 14.5 points on VQA, roughly a threefold gain on OK-VQA, and 18.1 points on GQA. Relative to FrozenB, MetaVL is close on VQA, competitive on OK-VQA, and substantially better on GQA, despite using a model described as about 20 times smaller.

Human evaluation changes the picture further. In 3-shot evaluation, MetaVL and FrozenB are both reported at 35.09 on VQA, 19.22 on OK-VQA, and 38.29 on GQA, while FrozenA remains much lower at 16.68, 6.41, and 19.96. The interpretation given in the paper is that semantic correctness was partly obscured by strict string matching in automatic evaluation, and that MetaVL in effect matches the 7B baseline once answer phrasing variability is judged by humans (Monajatipoor et al., 2023).

A major piece of evidence for genuine in-context learning is shot scaling. In the human evaluation without adapters, MetaVL improves from 25.49 to 26.61 to 30.24 to 35.09 on VQA as the number of shots increases from 0 to 3; from 6.70 to 9.97 to 14.72 to 19.22 on OK-VQA; and from 30.01 to 32.09 to 35.08 to 38.29 on GQA. The monotonic or near-monotonic gains with additional demonstrations are treated as a behavioral signature of in-context learning rather than mere promptable inference.

The adapter ablation is also diagnostically important. Frozen w/adap improves zero-shot performance, plausibly because adapter tuning improves image–text alignment. By contrast, MetaVL w/adap hurts few-shot performance. The best few-shot results are obtained with a fully frozen meta-trained LLM, which supports the argument that the LLM should remain frozen in order to preserve the meta-learned in-context learning capability (Monajatipoor et al., 2023).

A further ablation reduces the COCO vision–language training data to 50%. Under this change, automatic evaluation drops from 33.12 to 30.32 on VQA, from 9.60 to 7.56 on OK-VQA, and from 31.96 to 27.77 on GQA. The paper interprets this as evidence that MetaVL maintains in-context learning behavior under reduced vision–language data, although with an expected reduction in absolute performance.

The qualitative analysis shows cases in which support examples steer the expected output space or wording, such as choosing "left" versus "right" consistently with the in-context demonstrations, and cases in which the examples implicitly specify the expected answer type, such as color, object identity, or yes/no format. Failure cases include miscounting and errors in positional interpretation. This suggests that the transferred capability is not unrestricted reasoning, but rather multimodal adaptation within the representational limits of the frozen language backbone and visual alignment module.

6. Position in the literature, limitations, and broader usage of the term

MetaVL is explicitly positioned relative to MetaICL, Frozen, MAGMA, and Flamingo. MetaICL provides the text-only meta-learning procedure on which MetaVL depends (Min et al., 2021). Frozen supplies the basic architectural motif of projecting image features into a frozen LLM’s token space (Tsimpoukelli et al., 2021). The difference is that Frozen does not start from a LLM explicitly meta-trained for in-context learning, whereas MetaVL does. Flamingo and related systems are described as examples of multimodal in-context learning at much larger scales (Alayrac et al., 2022). The paper’s broader argument is that explicit meta-training for in-context learning can, to a substantial extent, compensate for model size in the vision–language domain (Monajatipoor et al., 2023).

The reported limitations are narrow but consequential. The evaluation covers only visual question answering datasets—VQA, OK-VQA, and GQA—so generalization to other vision–language tasks such as captioning, retrieval, referring expressions, or grounding is left unresolved. The experiments also use only a moderate-sized LLM, GPT-2 Medium, so the behavior of the method under larger backbones is not established. The description also notes implied limitations: dependence on CLIP for visual representation quality, reliance on a simple prompt pattern such as "Please answer the question.", and lack of explicit study of demonstration order sensitivity or heavy domain shift.

A recurrent source of confusion is terminological. MetaVL, in the strict sense, refers to the 2023 model that transfers in-context learning from a meta-trained LLM to a vision–LLM (Monajatipoor et al., 2023). Later work uses "MetaVL-style" more loosely. For example, MetaVQA is described as "sometimes referred to in the broader context as a MetaVL-style benchmark" for embodied scene understanding in driving, but it is a benchmark and fine-tuning framework for evaluating and improving vision-LLMs with Set-of-Mark prompting and closed-loop simulation, not the MetaVL architecture itself (Wang et al., 15 Jan 2025). Similarly, work on metamorphic testing of vision-language-action robots treats "MetaVL-like systems" as generalist embodied controllers and develops trajectory-based robustness tests for such systems, again extending the term beyond the original 2023 model (Valle et al., 26 Feb 2026).

In that narrower and historically primary sense, MetaVL denotes a specific claim about transfer: in-context learning is not exclusively a property of language-only prompting or of massive multimodal scale, but can be induced in a compact vision–LLM by freezing a meta-trained LLM and teaching a visual frontend to speak in its embedding space.