---
title: 'MetaVL: In-Context Vision-Language Learning'
url: https://www.emergentmind.com/topics/metavl
type: topic
---

# MetaVL: In-Context Vision-Language Learning

Searching arXiv for the primary paper and closely related works cited in the provided data.
{"query":"arXiv 2306.01311 MetaVL Transferring In-Context Learning Ability From Language Models to Vision-Language Models"}
{"query":"MetaICL arXiv Min 2022 Meta-trained In-Context Learning"}
{"query":"site:arxiv.org MetaICL Learning to Learn In Context arXiv"}
{"query":"MAGMA foundation model multimodal arXiv Eichenberg 2021"}
MetaVL is a vision–language model designed to exhibit strong in-context learning by transferring in-context learning capabilities from a meta-trained language model into a multimodal setting. In the formulation introduced in "MetaVL: Transferring In-Context Learning Ability From Language Models to Vision-Language Models" [2306.01311], the central claim is that a language model explicitly meta-trained to solve new tasks from a few textual demonstrations can retain that capability after a visual encoder is attached and aligned to its token space. The resulting system targets few-shot visual question answering without gradient updates at test time and is positioned as an alternative to multimodal in-context learning systems that rely on very large language backbones or massive multimodal corpora.

## 1. Problem setting and central hypothesis

MetaVL addresses a specific gap between large language models and most pre-trained vision–language models. In language, in-context learning refers to the setting in which the model is given a support set of $k$ labeled examples,
$$
(x_1, y_1), (x_2, y_2), \dots, (x_k, y_k),
$$
followed by a query input $x_{k+1}$, and must predict $y_{k+1}$ without parameter updates. Large language models such as GPT-3 showed that this behavior can emerge at scale, while MetaICL showed that it can also be trained explicitly through meta-learning across many NLP tasks [2110.15943].

The motivating observation for MetaVL is that most pre-trained vision–language systems do not natively support this mode of adaptation. Standard visual question answering models are optimized for fixed supervised objectives and usually require fine-tuning on new tasks or output formats. Promptable multimodal systems such as Frozen, MAGMA, and Flamingo can perform few-shot multimodal inference, but the provided description emphasizes that these systems typically depend on large frozen language backbones and/or extensive multimodal corpora containing in-context-style examples [2106.13884]; [2204.14198].

MetaVL is built around a direct cross-modal transfer hypothesis: if a language model is meta-trained specifically for in-context learning on many NLP tasks, then a vision–language model created by attaching a visual encoder to that frozen model can inherit its in-context learning ability. The paper describes this design as essentially "MetaICL + vision bridge" [2306.01311]. A plausible implication is that explicit meta-training can substitute, at least partly, for the extreme scale otherwise associated with multimodal in-context learning.

## 2. Architectural design

MetaVL consists of three components: a meta-trained language encoder–decoder, a visual encoder, and a visual prefix that projects image features into the language model embedding space [2306.01311].

The language backbone is GPT-2 Medium, with approximately 375M parameters. In the description used for MetaVL, it is a Transformer-based encoder–decoder trained with Prefix Language Modeling. The language encoder maps text to token embeddings $t_1, t_2, \dots, t_m$, and the decoder autoregressively predicts text tokens. This backbone is first meta-trained on NLP tasks and then frozen during the vision–language stage to preserve its in-context learning ability.

The visual encoder is CLIP-RN50x16. Its output is taken before the pooling layer as a feature grid of shape $n \times D_v$, where $n = 144$ spatial positions and $D_v = 3072$. Each spatial feature is treated as a visual token. Formally, the encoder is a function $V_e(x)$ that maps an image $x$ to $n \times D_v$ features.

The visual prefix is an MLP implemented here as a single linear layer of size $3072 \times 768$, with the target dimensionality $D_t = 768$ matching the GPT-2 embedding space. If the CLIP features are $v_i \in \mathbb{R}^{3072}$, the projection produces $\tilde{v}_i \in \mathbb{R}^{768}$, which are then interpreted as pseudo-language tokens. Given an image and text, MetaVL forms a multimodal sequence
$$
U = (\tilde{v}_1, \tilde{v}_2, \dots, \tilde{v}_n, t_1, t_2, \dots, t_m).
$$

The decoder is trained on the text portion of this sequence with the objective
$$
\mathcal{L}(U) = \sum_{i=1}^{m} \log P\big(t_i \mid \tilde{v}_1, \dots, \tilde{v}_n, t_1, \dots, t_{i-1}; \theta\big).
$$
During vision–language training, gradients update only the visual encoder $V_e$ and visual prefix $V_p$, while the language model remains frozen. The architecture is therefore intentionally simple: a frozen meta-trained language model, a trainable CLIP image encoder, and a simple projection layer, without additional architectural tricks beyond the PrefixLM-based alignment objective [2306.01311].

## 3. Two-stage training procedure

MetaVL is trained in two stages. The first stage is language-only meta-training, inherited from MetaICL. The second stage transfers the resulting capability into a vision–language model by training only the visual side [2306.01311].

In the first stage, the model is meta-trained over many NLP tasks. At each iteration, a task is sampled from a large pool of datasets, and $k+1$ examples are drawn:
$$
(x_1, y_1), \dots, (x_k, y_k), (x_{k+1}, y_{k+1}).
$$
The first $k$ examples serve as demonstrations, and the model is trained to maximize
$$
P(y_{k+1} \mid x_1, y_1, \dots, x_k, y_k, x_{k+1}).
$$
This is episodic meta-learning: the model is not updated on the query instance itself, but instead learns to infer task structure from the support examples. MetaVL adopts the MetaICL "HR→LR with instructions (all)" setup over 142 NLP datasets, where datasets with at least 10,000 training examples serve as meta-training tasks and smaller datasets are used as target tasks for evaluation in the MetaICL framework [2110.15943].

The language meta-training implementation uses GPT-2 Medium, a learning rate of $1 \times 10^{-5}$, batch size 8, and 80,000 steps on 8 NVIDIA RTX A6000 GPUs, with a reported training time of approximately 6 hours.

In the second stage, the meta-trained language encoder–decoder is completely frozen. The visual encoder and visual prefix are then trained on COCO captions using the language-modeling objective over text tokens conditioned on image-derived visual tokens. The rationale given in the paper is that freezing the language model preserves its meta-learned in-context learning knowledge, while the visual side learns to represent images in a form the language model can already exploit [2306.01311].

The vision–language stage uses the COCO training split, CLIP-RN50x16 as the visual encoder, the $3072 \rightarrow 768$ visual prefix, learning rate $5 \times 10^{-5}$ for the prefix, learning rate $2 \times 10^{-6}$ for the visual encoder, batch size 32, and 8 epochs on 4 A6000 GPUs, with a reported training time of approximately 48 hours.

## 4. In-context prompting and evaluation protocol

At inference time, MetaVL casts vision–language tasks into the same in-context format used during language meta-training. For a task such as visual question answering, each example is an image, question, answer triple. A prompt is constructed from a task induction sentence, $k$ support examples taken from the training set, and a query example consisting of an image and question without the answer [2306.01311].

The support examples and query are represented multimodally: each support image is converted into visual tokens, and each support question and answer is tokenized as text; the query contributes visual tokens and question text only. The model then generates the answer as a completion. The conditional form is given as
$$
p_\theta\Big(y^{(\text{query})} \;\big|\; D^{(\text{support})}, x^{(\text{query})}, v^{(\text{query})}\Big),
$$
where $D^{(\text{support})}$ is the support set of multimodal examples.

The evaluation focuses on three visual question answering datasets: VQA, OK-VQA, and GQA. The setup uses 0-, 1-, 2-, and 3-shot in-context learning, with random sampling of demonstrations from the training set for each query. Because of computational constraints, evaluation is performed on 5,000 instances from each validation or test split [2306.01311].

Automatic evaluation uses accuracy with exact string match or nearest candidate answer via Sentence-BERT cosine similarity. The candidate set is defined as the distinct answers in the training set, reported as approximately 3,000 to 4,200 answers. The paper also reports human evaluation on 2,000 examples to account for semantically correct generations that do not exactly match a canonical candidate string.

Two baseline families are used. FrozenA is a re-implementation of Frozen with GPT-2 Medium, and FrozenB uses GPT-J 6B, approximately 6B parameters, as a much larger frozen language backbone [2106.13884]. The experiments also include adapter variants, labeled Frozen w/adap and MetaVL w/adap, in the style associated in the description with MAGMA.

## 5. Empirical findings

The central empirical result is that in-context learning ability can be transferred from language-only meta-training into the vision–language setting. On 3-shot automatic evaluation over 5,000 test samples, FrozenA achieves 18.63 on VQA, 3.30 on OK-VQA, and 13.86 on GQA; FrozenB achieves 34.07, 11.97, and 25.76; MetaVL achieves 33.12, 9.60, and 31.96, respectively [2306.01311]. Relative to FrozenA at the same 375M language-model scale, this corresponds to gains of 14.5 points on VQA, roughly a threefold gain on OK-VQA, and 18.1 points on GQA. Relative to FrozenB, MetaVL is close on VQA, competitive on OK-VQA, and substantially better on GQA, despite using a model described as about 20 times smaller.

Human evaluation changes the picture further. In 3-shot evaluation, MetaVL and FrozenB are both reported at 35.09 on VQA, 19.22 on OK-VQA, and 38.29 on GQA, while FrozenA remains much lower at 16.68, 6.41, and 19.96. The interpretation given in the paper is that semantic correctness was partly obscured by strict string matching in automatic evaluation, and that MetaVL in effect matches the 7B baseline once answer phrasing variability is judged by humans [2306.01311].

A major piece of evidence for genuine in-context learning is shot scaling. In the human evaluation without adapters, MetaVL improves from 25.49 to 26.61 to 30.24 to 35.09 on VQA as the number of shots increases from 0 to 3; from 6.70 to 9.97 to 14.72 to 19.22 on OK-VQA; and from 30.01 to 32.09 to 35.08 to 38.29 on GQA. The monotonic or near-monotonic gains with additional demonstrations are treated as a behavioral signature of in-context learning rather than mere promptable inference.

The adapter ablation is also diagnostically important. Frozen w/adap improves zero-shot performance, plausibly because adapter tuning improves image–text alignment. By contrast, MetaVL w/adap hurts few-shot performance. The best few-shot results are obtained with a fully frozen meta-trained language model, which supports the argument that the language model should remain frozen in order to preserve the meta-learned in-context learning capability [2306.01311].

A further ablation reduces the COCO vision–language training data to 50%. Under this change, automatic evaluation drops from 33.12 to 30.32 on VQA, from 9.60 to 7.56 on OK-VQA, and from 31.96 to 27.77 on GQA. The paper interprets this as evidence that MetaVL maintains in-context learning behavior under reduced vision–language data, although with an expected reduction in absolute performance.

The qualitative analysis shows cases in which support examples steer the expected output space or wording, such as choosing "left" versus "right" consistently with the in-context demonstrations, and cases in which the examples implicitly specify the expected answer type, such as color, object identity, or yes/no format. Failure cases include miscounting and errors in positional interpretation. This suggests that the transferred capability is not unrestricted reasoning, but rather multimodal adaptation within the representational limits of the frozen language backbone and visual alignment module.

## 6. Position in the literature, limitations, and broader usage of the term

MetaVL is explicitly positioned relative to MetaICL, Frozen, MAGMA, and Flamingo. MetaICL provides the text-only meta-learning procedure on which MetaVL depends [2110.15943]. Frozen supplies the basic architectural motif of projecting image features into a frozen language model’s token space [2106.13884]. The difference is that Frozen does not start from a language model explicitly meta-trained for in-context learning, whereas MetaVL does. Flamingo and related systems are described as examples of multimodal in-context learning at much larger scales [2204.14198]. The paper’s broader argument is that explicit meta-training for in-context learning can, to a substantial extent, compensate for model size in the vision–language domain [2306.01311].

The reported limitations are narrow but consequential. The evaluation covers only visual question answering datasets—VQA, OK-VQA, and GQA—so generalization to other vision–language tasks such as captioning, retrieval, referring expressions, or grounding is left unresolved. The experiments also use only a moderate-sized language model, GPT-2 Medium, so the behavior of the method under larger backbones is not established. The description also notes implied limitations: dependence on CLIP for visual representation quality, reliance on a simple prompt pattern such as "Please answer the question.", and lack of explicit study of demonstration order sensitivity or heavy domain shift.

A recurrent source of confusion is terminological. MetaVL, in the strict sense, refers to the 2023 model that transfers in-context learning from a meta-trained language model to a vision–language model [2306.01311]. Later work uses "MetaVL-style" more loosely. For example, MetaVQA is described as "sometimes referred to in the broader context as a MetaVL-style benchmark" for embodied scene understanding in driving, but it is a benchmark and fine-tuning framework for evaluating and improving vision-language models with Set-of-Mark prompting and closed-loop simulation, not the MetaVL architecture itself [2501.09167]. Similarly, work on metamorphic testing of vision-language-action robots treats "MetaVL-like systems" as generalist embodied controllers and develops trajectory-based robustness tests for such systems, again extending the term beyond the original 2023 model [2602.22579].

In that narrower and historically primary sense, MetaVL denotes a specific claim about transfer: in-context learning is not exclusively a property of language-only prompting or of massive multimodal scale, but can be induced in a compact vision–language model by freezing a meta-trained language model and teaching a visual frontend to speak in its embedding space.

Source: https://www.emergentmind.com/topics/metavl