Kosmos-2 Multimodal Language Model

• Kosmos-2 is a multimodal large language model that grounds text by representing referring expressions as Markdown-style links between text spans and bounding boxes.
• It integrates visual and text tokens into a single autoregressive sequence, eliminating separate detection heads and enabling unified generation.
• The model demonstrates robust performance on tasks like phrase grounding, referring comprehension, and generation, paving the way for multimodal literacy.

Kosmos-2 is a Multimodal LLM (MLLM) that introduces explicit grounding into causal multimodal generation by representing referring expressions as Markdown-style links between text spans and bounding boxes, thereby enabling the model to perceive object descriptions and ground text to the visual world (Peng et al., 2023). The model is evaluated across multimodal grounding, multimodal referring, perception-language tasks, and language understanding and generation, with grounding treated as part of the token stream rather than as a separate detection problem (Peng et al., 2023). A later system, KOSMOS-2.5, applies related generative principles to machine reading of text-intensive images; this suggests a research trajectory in which the Kosmos line expands from object-level grounding toward multimodal literacy and document understanding (Lv et al., 2023).

1. Conceptual scope and problem formulation

Kosmos-2 is defined by the claim that grounded visual reference can be absorbed into the same generative interface used for language modeling. In its core formulation, a referring expression is represented as a Markdown-like construct, written conceptually as "[ text-span ]( bounding-box )", and realized in the model vocabulary through special tags that delimit phrases and regions (Peng et al., 2023). This makes grounding a sequence modeling problem: the model predicts both lexical tokens and location tokens by autoregressive next-token prediction.

The immediate consequence is that grounding is not implemented by a separate detection head or a bounding-box regression branch. Instead, image tokens, text tokens, and location tokens are concatenated into one causal stream and decoded by the same Transformer (Peng et al., 2023). This is a substantive departure from pipelines that isolate perception, grounding, and language generation into separate modules.

Within the paper’s own framing, Kosmos-2 extends the capabilities usually associated with MLLMs—perceiving general modalities, following instructions, and performing in-context learning—by integrating grounding into downstream applications (Peng et al., 2023). The discussion further situates this design in “Embodiment AI,” arguing that a “grounding → language → action” pipeline is a concrete step toward agents that can both speak and point (Peng et al., 2023). A common misunderstanding is therefore to read Kosmos-2 as merely a captioning system with added region tags; the model is presented instead as a unified causal architecture in which reference resolution is internal to generation.

2. Architecture and sequence interface

Kosmos-2 builds on Kosmos-1’s two-stage design. Its main architectural components are summarized below (Peng et al., 2023).

Component	Specification
Vision encoder	24 Transformer layers, hidden size 1024, FFN size 4096
Resampler	Reduces raw patch embeddings from a ViT over 224×224 images, patch = 14×14, to $M = 64$ fixed image tokens
Multimodal causal decoder	“Magneto” Transformer, 24 layers, hidden size $d = 2048$, 32 attention heads, FFN size 8192

The unified input sequence is formed by simple concatenation of image tokens and text tokens, after which the entire sequence is processed by the same causal Transformer (Peng et al., 2023). This design removes the need for specialized grounding branches. All outputs—ordinary words, phrase delimiters, box markers, and location tokens—are generated in a single autoregressive regime.

The decoder uses standard multi-head self-attention. In the notation given in the paper, with hidden state $h^{(\ell-1)}$ at layer $\ell-1$,

$$Q = h^{(\ell-1)}W^Q,\quad K = h^{(\ell-1)}W^K,\quad V = h^{(\ell-1)}W^V,$$

$$\text{head}_i = \text{softmax}(QK^\top/\sqrt{d_k})V,$$

followed by residual connection, projection, and a standard feed-forward sublayer (Peng et al., 2023). The significance of this formulation is not the novelty of the attention rule itself, but the fact that grounding is embedded in the same causal mechanism used for linguistic continuation.

3. Grounding representation, data curation, and training objective

The grounding representation is central to Kosmos-2. Every grounded phrase is encoded in a “hyperlink” format using text-span delimiters and box delimiters: <p> text-span </p> <box> <loc_1> <loc_2> </box> (Peng et al., 2023). Spatial coordinates are discretized rather than regressed continuously. Given an image of width $W$ and height $H$, both axes are split into $P$ bins, with $P = 32$:

$d = 2048$0

$d = 2048$1

The vocabulary is augmented with $d = 2048$2 location tokens, <loc_0> through <loc_{P^2-1}>. A bounding box with top-left corner $d = 2048$3 and bottom-right corner $d = 2048$4 becomes <box> <loc_{t_1}> <loc_{t_2}> </box>, and multiple boxes are interleaved with <delim> (Peng et al., 2023).

Training uses the GrIT dataset, curated at scale from LAION-2B and COYO-700M. The paper reports approximately 90 million images, approximately 115 million text spans, and approximately 137 million bounding boxes (Peng et al., 2023). The curation pipeline first extracts noun chunks with spaCy, detects candidate boxes via GLIP, filters by confidence $d = 2048$5 and NMS, expands chunks to full referring expressions via dependency-tree traversal, assigns the chunk’s box to the longer expression, and drops nested spans (Peng et al., 2023).

GrIT is then mixed with Kosmos-1 corpora. Training batches sample monolingual text, ungrounded image-caption pairs, interleaved multimodal data, and grounded image-text pairs, for a total of approximately 25 billion tokens over 60,000 steps on 256 V100s (Peng et al., 2023). The objective is pure causal cross-entropy over the joint vocabulary:

$d = 2048$6

No explicit bounding-box regression loss or contrastive grounding loss is added; the grounding capability is learned by treating region coordinates as discrete tokens (Peng et al., 2023).

Kosmos-2 also includes an instruction-tuning stage of 10,000 steps that combines LLaVA-Instruct, Unnatural-Instructions, FLANv2, and custom grounded templates such as “What is <p>it</p><box>…</box>? It is {expression}.” (Peng et al., 2023). This matters because grounding is not only pre-trained as a perceptual alignment problem, but also adapted into instruction-following behavior.

4. Inference modes and grounded capabilities

Kosmos-2 supports several grounded inference modes through prompt design. For referring-expression comprehension, the prompt is $h^{(\ell-1)}$9 and the model autoregressively emits <box>…</box> (Peng et al., 2023). The first box is decoded, mapped from tokens back to coordinates, and counted as correct if the Intersection over Union exceeds 0.5 (Peng et al., 2023).

Phrase grounding follows a closely related protocol, but each phrase is supplied in turn together with its left context as a disambiguator under the ANY-BOX protocol (Peng et al., 2023). The model produces candidate boxes, from which top-$d = 2048$7 boxes are collected and evaluated with recall at ranks 1, 5, and 10 (Peng et al., 2023). The operational distinction is that phrase grounding requires contextualized localization of spans embedded in larger text, whereas referring comprehension targets a single referring expression directly.

Kosmos-2 also performs referring-expression generation. Given a known box, the prompt takes the form $\ell-1$0 and the model completes the referring expression in natural language (Peng et al., 2023). This task is evaluated with METEOR and CIDEr, both in zero-shot and in few-shot settings with $d = 2048$8 or $d = 2048$9 demonstrations (Peng et al., 2023). Because the architecture is purely causal, few-shot improvement is obtained by concatenating demonstration pairs in the prompt rather than by gradient-based adaptation.

The paper emphasizes multimodal in-context learning as an additional capability. Demonstration pairs of box-to-expression or expression-to-box mappings can be prepended to the prompt, enabling refinement without parameter updates (Peng et al., 2023). This positions grounding not as a fixed output head, but as a skill that can be modulated by prompt composition.

5. Empirical performance and comparative profile

The evaluation profile spans phrase grounding, referring comprehension, referring generation, vision-language tasks, and language benchmarks. The reported results are summarized below (Peng et al., 2023).

Task	Benchmark / metric	Reported result
Phrase grounding	Flickr30k Entities, zero-shot R@1	77.8 (val), 78.7 (test); VisualBert 70.4/71.3; GLIP $h^{(\ell-1)}$0
Referring comprehension	RefCOCO/A/B, RefCOCO+, RefCOCOg, zero-shot accuracy	Range 45–62%; RefCOCOg 61.7%
Referring generation	RefCOCOg, METEOR / CIDEr	Zero-shot 12.2 / 60.3; few-shot $h^{(\ell-1)}$1 gives 14.1 / 62.3
Vision-language	Flickr30k captioning CIDEr; VQAv2 accuracy	66.7 vs. Kosmos-1 65.2; 45.6 vs. 46.7
Pure language	BoolQ example	56.4 → 62.0

On Flickr30k Entities, Kosmos-2 achieves zero-shot phrase grounding R@1 of 77.8 on validation and 78.7 on test, exceeding VisualBert by roughly 7 points while remaining below fully finetuned detectors such as GLIP at approximately 87 R@1 (Peng et al., 2023). The significance of this result lies in the absence of proposal mechanisms or object queries: the model produces localized references directly from autoregressive decoding.

For referring-expression comprehension on RefCOCO/A/B, RefCOCO+, and RefCOCOg, zero-shot accuracy ranges from 45% to 62%, with RefCOCOg at 61.7% (Peng et al., 2023). The paper characterizes this as the first out-of-the-box grounding ability and explicitly notes that performance trails specialized finetuned models (Peng et al., 2023). This is an important qualification: Kosmos-2 demonstrates generalist grounding without claiming state-of-the-art fully supervised localization.

In referring-expression generation on RefCOCOg, zero-shot performance reaches METEOR $h^{(\ell-1)}$2 and CIDEr $h^{(\ell-1)}$3, while few-shot prompting with $h^{(\ell-1)}$4 raises these to METEOR $h^{(\ell-1)}$5 and CIDEr $h^{(\ell-1)}$6 (Peng et al., 2023). The paper states that this outperforms a finetuned speaker-listener-reinforcer model on CIDEr without requiring a grounding-specific module (Peng et al., 2023).

Beyond grounding, Kosmos-2 reports Flickr30k captioning CIDEr $h^{(\ell-1)}$7, compared with 65.2 for Kosmos-1, and VQAv2 accuracy $h^{(\ell-1)}$8, compared with 46.7 (Peng et al., 2023). On pure language tasks, the paper states that Kosmos-2 matches Kosmos-1’s performance overall, while giving examples of gains such as BoolQ rising from 56.4 to 62.0 (Peng et al., 2023). The empirical profile is therefore mixed but coherent: explicit grounding adds a new capability while largely preserving broader multimodal and language behavior.

6. Extensions, limitations, and the transition to KOSMOS-2.5

The limitations identified for Kosmos-2 are specific and technically consequential. The discussion points to improving zero-shot comprehension on shorter, more spatial expressions such as those in RefCOCO and RefCOCO+, extending the tokenized grounding mechanism to panoptic segmentation through mask tokens, and integrating affordance or action grounding so that verbs as well as noun phrases can be linked to regions (Peng et al., 2023). These limitations indicate that the original formulation is strongest on box-based grounded reference and less developed for denser or action-centric visual semantics.

KOSMOS-2.5 broadens the scope from grounded object-language alignment to document-centric multimodal literacy. It is presented as “a multimodal literate model for machine reading of text-intensive images,” pre-trained on “a large corpus of 357.4 million document pages spanning diverse domains,” and designed for two transcription tasks: generating spatially-aware text blocks with coordinates and producing structured text output in markdown format (Lv et al., 2023). The model uses a shared decoder-only autoregressive Transformer architecture with task-specific prompts, and the abstract states that it demonstrates literate capabilities comparable to GPT-4o on OCREval and MarkdownEval (Lv et al., 2023).

The technical summary of KOSMOS-2.5 further describes its relation to KOSMOS-2 in terms of “a shift from encoder-only (or encoder–decoder) to a pure decoder-only generative interface,” the “addition of Perceiver Resampler in lieu of direct cross-attentions,” and unified treatment of layout- and markup-based outputs under one vocabulary and one decoder (Lv et al., 2023). This suggests continuity at the level of sequence generation and prompt-conditioned multimodal decoding, even though the task domain moves from grounded natural-image reference to text-intensive image transcription.

There is also an important documentary nuance in the KOSMOS-2.5 materials. The abstract introduces a document understanding generalist named KOSMOS-2.5-CHAT and reports performance across nine text-rich visual question answering benchmarks, but the technical summary states that the paper “does not describe a ‘KOSMOS-2.5-CHAT’ or instruction-tuning stage,” and that for all reported evaluations the pre-trained model is used with the appropriate prompt at inference time, with no downstream fine-tuning (Lv et al., 2023). Objectively stated, this is not a contradiction in the existence of the claim, but it is a gap in methodological description.

Taken together, the Kosmos line illustrates a specific research program: encode perception, reference, and structured output in a shared autoregressive token space. In Kosmos-2, the decisive move is to turn bounding boxes into language-like tokens; in KOSMOS-2.5, the analogous move is to turn layout and markdown structure into generative targets for text-intensive images (Peng et al., 2023, Lv et al., 2023). A plausible implication is that future systems in this line will continue to enlarge the class of structured visual outputs that can be handled by a single causal multimodal decoder.