---
title: Kosmos-2 Multimodal Language Model
url: https://www.emergentmind.com/topics/kosmos-2
type: topic
---

# Kosmos-2 Multimodal Language Model

Kosmos-2 is a Multimodal Large Language Model (MLLM) that introduces explicit grounding into causal multimodal generation by representing referring expressions as Markdown-style links between text spans and bounding boxes, thereby enabling the model to perceive object descriptions and ground text to the visual world [2306.14824]. The model is evaluated across multimodal grounding, multimodal referring, perception-language tasks, and language understanding and generation, with grounding treated as part of the token stream rather than as a separate detection problem [2306.14824]. A later system, KOSMOS-2.5, applies related generative principles to machine reading of text-intensive images; this suggests a research trajectory in which the Kosmos line expands from object-level grounding toward multimodal literacy and document understanding [2309.11419].

## 1. Conceptual scope and problem formulation

Kosmos-2 is defined by the claim that grounded visual reference can be absorbed into the same generative interface used for language modeling. In its core formulation, a referring expression is represented as a Markdown-like construct, written conceptually as `"[ text-span ]( bounding-box )"`, and realized in the model vocabulary through special tags that delimit phrases and regions [2306.14824]. This makes grounding a sequence modeling problem: the model predicts both lexical tokens and location tokens by autoregressive next-token prediction.

The immediate consequence is that grounding is not implemented by a separate detection head or a bounding-box regression branch. Instead, image tokens, text tokens, and location tokens are concatenated into one causal stream and decoded by the same Transformer [2306.14824]. This is a substantive departure from pipelines that isolate perception, grounding, and language generation into separate modules.

Within the paper’s own framing, Kosmos-2 extends the capabilities usually associated with MLLMs—perceiving general modalities, following instructions, and performing in-context learning—by integrating grounding into downstream applications [2306.14824]. The discussion further situates this design in “Embodiment AI,” arguing that a “grounding → language → action” pipeline is a concrete step toward agents that can both speak and point [2306.14824]. A common misunderstanding is therefore to read Kosmos-2 as merely a captioning system with added region tags; the model is presented instead as a unified causal architecture in which reference resolution is internal to generation.

## 2. Architecture and sequence interface

Kosmos-2 builds on Kosmos-1’s two-stage design. Its main architectural components are summarized below [2306.14824].

| Component | Specification |
|---|---|
| Vision encoder | 24 Transformer layers, hidden size 1024, FFN size 4096 |
| Resampler | Reduces raw patch embeddings from a ViT over 224×224 images, patch = 14×14, to $M = 64$ fixed image tokens |
| Multimodal causal decoder | “Magneto” Transformer, 24 layers, hidden size $d = 2048$, 32 attention heads, FFN size 8192 |

The unified input sequence is formed by simple concatenation of image tokens and text tokens, after which the entire sequence is processed by the same causal Transformer [2306.14824]. This design removes the need for specialized grounding branches. All outputs—ordinary words, phrase delimiters, box markers, and location tokens—are generated in a single autoregressive regime.

The decoder uses standard multi-head self-attention. In the notation given in the paper, with hidden state $h^{(\ell-1)}$ at layer $\ell-1$,
$$
Q = h^{(\ell-1)}W^Q,\quad K = h^{(\ell-1)}W^K,\quad V = h^{(\ell-1)}W^V,
$$
$$
\text{head}_i = \text{softmax}(QK^\top/\sqrt{d_k})V,
$$
followed by residual connection, projection, and a standard feed-forward sublayer [2306.14824]. The significance of this formulation is not the novelty of the attention rule itself, but the fact that grounding is embedded in the same causal mechanism used for linguistic continuation.

## 3. Grounding representation, data curation, and training objective

The grounding representation is central to Kosmos-2. Every grounded phrase is encoded in a “hyperlink” format using text-span delimiters and box delimiters:
`<p> text-span </p> <box> <loc_1> <loc_2> </box>` [2306.14824]. Spatial coordinates are discretized rather than regressed continuously. Given an image of width $W$ and height $H$, both axes are split into $P$ bins, with $P = 32$:
$$
\text{bin}_x = \lfloor x \cdot P / W \rfloor,\qquad
\text{bin}_y = \lfloor y \cdot P / H \rfloor,
$$
$$
\text{token\_id} = \text{bin}_y \cdot P + \text{bin}_x.
$$
The vocabulary is augmented with $P^2$ location tokens, `<loc_0>` through `<loc_{P^2-1}>`. A bounding box with top-left corner $(x_1,y_1)$ and bottom-right corner $(x_2,y_2)$ becomes
`<box> <loc_{t_1}> <loc_{t_2}> </box>`, and multiple boxes are interleaved with `<delim>` [2306.14824].

Training uses the GrIT dataset, curated at scale from LAION-2B and COYO-700M. The paper reports approximately 90 million images, approximately 115 million text spans, and approximately 137 million bounding boxes [2306.14824]. The curation pipeline first extracts noun chunks with spaCy, detects candidate boxes via GLIP, filters by confidence \(> 0.65\) and NMS, expands chunks to full referring expressions via dependency-tree traversal, assigns the chunk’s box to the longer expression, and drops nested spans [2306.14824].

GrIT is then mixed with Kosmos-1 corpora. Training batches sample monolingual text, ungrounded image-caption pairs, interleaved multimodal data, and grounded image-text pairs, for a total of approximately 25 billion tokens over 60,000 steps on 256 V100s [2306.14824]. The objective is pure causal cross-entropy over the joint vocabulary:
$$
L = - \sum_{t=1}^{T} \log P(v_t \mid v_{<t}, I).
$$
No explicit bounding-box regression loss or contrastive grounding loss is added; the grounding capability is learned by treating region coordinates as discrete tokens [2306.14824].

Kosmos-2 also includes an instruction-tuning stage of 10,000 steps that combines LLaVA-Instruct, Unnatural-Instructions, FLANv2, and custom grounded templates such as “What is `<p>it</p><box>…</box>`? It is {expression}.” [2306.14824]. This matters because grounding is not only pre-trained as a perceptual alignment problem, but also adapted into instruction-following behavior.

## 4. Inference modes and grounded capabilities

Kosmos-2 supports several grounded inference modes through prompt design. For referring-expression comprehension, the prompt is
```text
<s> <image>…</image> <grounding> <p> referring-expression </p>
```
and the model autoregressively emits `<box>…</box>` [2306.14824]. The first box is decoded, mapped from tokens back to coordinates, and counted as correct if the Intersection over Union exceeds 0.5 [2306.14824].

Phrase grounding follows a closely related protocol, but each phrase is supplied in turn together with its left context as a disambiguator under the ANY-BOX protocol [2306.14824]. The model produces candidate boxes, from which top-$k$ boxes are collected and evaluated with recall at ranks 1, 5, and 10 [2306.14824]. The operational distinction is that phrase grounding requires contextualized localization of spans embedded in larger text, whereas referring comprehension targets a single referring expression directly.

Kosmos-2 also performs referring-expression generation. Given a known box, the prompt takes the form
```text
<s> <image>…</image> <p> It </p> <box>…</box> is
```
and the model completes the referring expression in natural language [2306.14824]. This task is evaluated with METEOR and CIDEr, both in zero-shot and in few-shot settings with $k = 2$ or $4$ demonstrations [2306.14824]. Because the architecture is purely causal, few-shot improvement is obtained by concatenating demonstration pairs in the prompt rather than by gradient-based adaptation.

The paper emphasizes multimodal in-context learning as an additional capability. Demonstration pairs of box-to-expression or expression-to-box mappings can be prepended to the prompt, enabling refinement without parameter updates [2306.14824]. This positions grounding not as a fixed output head, but as a skill that can be modulated by prompt composition.

## 5. Empirical performance and comparative profile

The evaluation profile spans phrase grounding, referring comprehension, referring generation, vision-language tasks, and language benchmarks. The reported results are summarized below [2306.14824].

| Task | Benchmark / metric | Reported result |
|---|---|---|
| Phrase grounding | Flickr30k Entities, zero-shot R@1 | 77.8 (val), 78.7 (test); VisualBert 70.4/71.3; GLIP $\approx 87$ |
| Referring comprehension | RefCOCO/A/B, RefCOCO+, RefCOCOg, zero-shot accuracy | Range 45–62%; RefCOCOg 61.7% |
| Referring generation | RefCOCOg, METEOR / CIDEr | Zero-shot 12.2 / 60.3; few-shot $k=4$ gives 14.1 / 62.3 |
| Vision-language | Flickr30k captioning CIDEr; VQAv2 accuracy | 66.7 vs. Kosmos-1 65.2; 45.6 vs. 46.7 |
| Pure language | BoolQ example | 56.4 → 62.0 |

On Flickr30k Entities, Kosmos-2 achieves zero-shot phrase grounding R@1 of 77.8 on validation and 78.7 on test, exceeding VisualBert by roughly 7 points while remaining below fully finetuned detectors such as GLIP at approximately 87 R@1 [2306.14824]. The significance of this result lies in the absence of proposal mechanisms or object queries: the model produces localized references directly from autoregressive decoding.

For referring-expression comprehension on RefCOCO/A/B, RefCOCO+, and RefCOCOg, zero-shot accuracy ranges from 45% to 62%, with RefCOCOg at 61.7% [2306.14824]. The paper characterizes this as the first out-of-the-box grounding ability and explicitly notes that performance trails specialized finetuned models [2306.14824]. This is an important qualification: Kosmos-2 demonstrates generalist grounding without claiming state-of-the-art fully supervised localization.

In referring-expression generation on RefCOCOg, zero-shot performance reaches METEOR \(= 12.2\) and CIDEr \(= 60.3\), while few-shot prompting with \(k = 4\) raises these to METEOR \(= 14.1\) and CIDEr \(= 62.3\) [2306.14824]. The paper states that this outperforms a finetuned speaker-listener-reinforcer model on CIDEr without requiring a grounding-specific module [2306.14824].

Beyond grounding, Kosmos-2 reports Flickr30k captioning CIDEr \(= 66.7\), compared with 65.2 for Kosmos-1, and VQAv2 accuracy \(= 45.6\), compared with 46.7 [2306.14824]. On pure language tasks, the paper states that Kosmos-2 matches Kosmos-1’s performance overall, while giving examples of gains such as BoolQ rising from 56.4 to 62.0 [2306.14824]. The empirical profile is therefore mixed but coherent: explicit grounding adds a new capability while largely preserving broader multimodal and language behavior.

## 6. Extensions, limitations, and the transition to KOSMOS-2.5

The limitations identified for Kosmos-2 are specific and technically consequential. The discussion points to improving zero-shot comprehension on shorter, more spatial expressions such as those in RefCOCO and RefCOCO+, extending the tokenized grounding mechanism to panoptic segmentation through mask tokens, and integrating affordance or action grounding so that verbs as well as noun phrases can be linked to regions [2306.14824]. These limitations indicate that the original formulation is strongest on box-based grounded reference and less developed for denser or action-centric visual semantics.

KOSMOS-2.5 broadens the scope from grounded object-language alignment to document-centric multimodal literacy. It is presented as “a multimodal literate model for machine reading of text-intensive images,” pre-trained on “a large corpus of 357.4 million document pages spanning diverse domains,” and designed for two transcription tasks: generating spatially-aware text blocks with coordinates and producing structured text output in markdown format [2309.11419]. The model uses a shared decoder-only autoregressive Transformer architecture with task-specific prompts, and the abstract states that it demonstrates literate capabilities comparable to GPT-4o on OCREval and MarkdownEval [2309.11419].

The technical summary of KOSMOS-2.5 further describes its relation to KOSMOS-2 in terms of “a shift from encoder-only (or encoder–decoder) to a pure decoder-only generative interface,” the “addition of Perceiver Resampler in lieu of direct cross-attentions,” and unified treatment of layout- and markup-based outputs under one vocabulary and one decoder [2309.11419]. This suggests continuity at the level of sequence generation and prompt-conditioned multimodal decoding, even though the task domain moves from grounded natural-image reference to text-intensive image transcription.

There is also an important documentary nuance in the KOSMOS-2.5 materials. The abstract introduces a document understanding generalist named KOSMOS-2.5-CHAT and reports performance across nine text-rich visual question answering benchmarks, but the technical summary states that the paper “does not describe a ‘KOSMOS-2.5-CHAT’ or instruction-tuning stage,” and that for all reported evaluations the pre-trained model is used with the appropriate prompt at inference time, with no downstream fine-tuning [2309.11419]. Objectively stated, this is not a contradiction in the existence of the claim, but it is a gap in methodological description.

Taken together, the Kosmos line illustrates a specific research program: encode perception, reference, and structured output in a shared autoregressive token space. In Kosmos-2, the decisive move is to turn bounding boxes into language-like tokens; in KOSMOS-2.5, the analogous move is to turn layout and markdown structure into generative targets for text-intensive images [2306.14824] [2309.11419]. A plausible implication is that future systems in this line will continue to enlarge the class of structured visual outputs that can be handled by a single causal multimodal decoder.

Source: https://www.emergentmind.com/topics/kosmos-2