Otter: Multimodal Instruction Model

• Otter is a multimodal model that integrates interleaved image-text conditioning with instruction tuning for effective few-shot generalization.
• It combines a frozen CLIP ViT-L/14 vision encoder and LLaMA-7B language backbone with trainable cross-attention layers to enhance image-grounded response generation.
• The model leverages the expansive MIMIC-IT dataset with 2.8M instruction-response pairs to boost interactive perception, reasoning, and multimodal in-context learning.

Searching arXiv for the Otter multimodal model and its companion MIMIC-IT dataset. Otter is a multimodal foundation model that introduces instruction tuning into the visual-language in-context learning paradigm associated with Flamingo and OpenFlamingo. It preserves interleaved multimodal conditioning, few-shot generalization, and cross-modal context exploitation, while explicitly training the model to follow user instructions through the MultI-Modal In-Context Instruction Tuning dataset, MIMIC-IT. In its 2023 formulation, Otter is based on OpenFlamingo, uses a frozen CLIP ViT-L/14 vision encoder and a frozen LLaMA-7B language backbone, fine-tunes cross-modal fusion components and language embeddings, and is engineered for comparatively accessible training and deployment through Huggingface Transformers (Li et al., 2023).

1. Historical placement and research objective

Otter emerged from a specific limitation in early open multimodal assistants: upstream interleaved pretraining provided strong multimodal conditioning, but did not by itself produce robust instruction-following behavior. The model therefore targets a gap between caption-style multimodal pretraining and downstream interaction, bringing instruction tuning to an interleaved image-text architecture rather than to a language-only or image-question-answer setting (Li et al., 2023).

Its immediate architectural antecedent is OpenFlamingo, itself an open implementation of DeepMind’s Flamingo-style design. Otter retains the interleaved conditioning structure that enables multimodal few-shot generalization, but changes the fine-tuning target from generic caption-like continuation to instruction-oriented, image-grounded response generation. This design explicitly equips the model to parse directives such as “Describe,” “Count,” “Explain,” and “Compare,” while still conditioning on preceding multimodal exemplars (Li et al., 2023).

Within the broader 2023 multimodal instruction-tuning landscape, Otter was also associated with a substantially larger and more diverse instruction corpus than earlier instruction datasets. MIMIC-IT comprises 2.8 million multimodal instruction-response pairs, with 2.2 million unique instructions derived from images and videos, and was introduced to support perception, reasoning, planning, and multimodal in-context learning in interactive settings (Li et al., 2023).

2. MIMIC-IT and the interleaved instruction format

MIMIC-IT is the data substrate that operationalizes Otter’s training objective. Each training instance contains a queried image-instruction-answer triplet together with context: a series of image-instruction-answer triplets that are contextually correlated with the queried triplet. The intent is to emulate the adjacency correlations of MMC4-style interleaved multimodal data, so that the model learns not only to answer a query but also to exploit related preceding chunks as in-context exemplars (Li et al., 2023).

The dataset draws image-instruction-answer triplets from VQAv2, GQA, LLaVA instruction-following pairs, and PVSG frames with instruction-following annotations inspired by LLaVA. In the broader MIMIC-IT release, the corpus spans general scenes, surveillance “spot-the-difference,” dense-captioned events, story albums, TV show clips, indoor RGB-D scans, and egocentric views, and is multilingual: English plus seven translations, namely zh, ja, es, de, fr, ko, and ar (Li et al., 2023).

The interleaved representation is central. One description of the training format is:

<context> [image] User:<instruction> GPT:[answers] <answer>.[endofchunk]

where [image], [answer], and [endofchunk] are unique tokens. The [answer] token marks the start of the answer region, and tokens after [answer] are masked during training and become the prediction target. [endofchunk] terminates a chunk so that multiple context chunks can be concatenated before the query (Li et al., 2023).

A related description in the MIMIC-IT release uses an interleaved conversational template of the form:

<image>Human:{instruction} Assistant:<answer>{response}<endofchunk>

with multiple in-context examples followed by the query example. Both descriptions emphasize the same structural principle: multimodal exemplars are serialized as role-marked chunks, and the query is appended after those chunks so that the model can condition on them autoregressively (Li et al., 2023).

MIMIC-IT itself was produced through the Syphus annotation pipeline, which combined system messages, visual annotations, in-context examples, GPT-based generation, post-processing, de-duplication, and translation. Safety filtering removed content with unfair opportunities, stereotyping, over/under-representation, explicit content, disinformation, or unreliable information (Li et al., 2023).

3. Architecture and multimodal conditioning mechanism

Otter inherits the OpenFlamingo architecture. Its vision encoder is CLIP ViT-L/14, frozen during Otter fine-tuning; its language backbone is LLaMA-7B, also frozen. The trainable multimodal components are the perceiver resampler, the cross-attention layers inserted into the LLM, and the language input/output embeddings. The total number of trainable parameters is approximately 1.3B (Li et al., 2023).

Images are injected as visual features rather than as raw pixel inputs to the LLM. An image $I$ is encoded by the frozen CLIP encoder, producing a sequence of image features, after which the perceiver resampler converts them into a fixed-size set of visual tokens:

$V = \mathrm{Resampler}(\mathrm{Encoder}(I)).$

The LLM then consumes an interleaved sequence in which text tokens can cross-attend to $V$. In-context learning arises because the model attends over the concatenation of earlier context chunks and the later query chunk (Li et al., 2023).

The fusion mechanism is standard cross-attention. A language-layer cross-attention block is described with the usual formulation

$$\mathrm{Attn}(Q, K, V) = \mathrm{softmax}(QK^\top / \sqrt{d})V,$$

where $Q$ are query projections from text hidden states and $K, V$ are projections of the visual token set, enabling multimodal interaction without modifying the frozen pretrained backbones themselves (Li et al., 2023).

This architecture is significant because it couples two otherwise competing aims. First, it preserves pretrained vision-language grounding and language modeling capacity by freezing CLIP ViT-L/14 and LLaMA-7B. Second, it adapts the cross-modal interface and lexical boundary conditions through the resampler, cross-attention layers, and input/output embeddings, which is precisely where instruction-following behavior and new special tokens must be accommodated. This suggests that Otter’s main contribution is not a new backbone, but a new fine-tuning target and a new data/interface regime.

4. Objective, optimization, and inference conventions

Otter is trained with an autoregressive next-token objective over the answer span, conditioned on interleaved multimodal context. If $y = (y_1,\dots,y_T)$ denotes the target answer tokens, $C$ the interleaved context, and $V_{1:K}$ the visual tokens for the images included across the context and the queried image, the conditional likelihood is

$$p(y \mid C, V_{1:K}) = \prod_{t=1}^{T} p(y_t \mid y_{<t}, C, V_{1:K}),$$

and the loss is standard autoregressive cross-entropy over the masked answer region:

$V = \mathrm{Resampler}(\mathrm{Encoder}(I)).$0

The training procedure “mask[s] all tokens after the [answer] token during training and set[s] them as the prediction objectives” (Li et al., 2023).

Optimization uses AdamW with initial learning rate $V = \mathrm{Resampler}(\mathrm{Encoder}(I)).$1, cosine annealing, gradient clipping at 1.0, batch size 4, and 6 epochs. Training uses bf16 mixed precision via Huggingface Accelerate, with support for Fully Sharded Data Parallel and DeepSpeed. A principal engineering contribution is that the implementation is optimized to train on 4$V = \mathrm{Resampler}(\mathrm{Encoder}(I)).$2 RTX-3090 GPUs rather than requiring at least 1$V = \mathrm{Resampler}(\mathrm{Encoder}(I)).$3 A100 GPU (Li et al., 2023).

The model is integrated into Huggingface Transformers and released through model hubs including luodian/openflamingo-9b-hf and luodian/otter-9b-hf. A conversion script is provided for OpenFlamingo-9B checkpoints, and inference follows the same interleaved convention as training: concatenate few-shot exemplars as context chunks, each marked with [image], User: and GPT: role labels, an [answer] separator, and [endofchunk] tokens, then append the query chunk and generate after [answer] (Li et al., 2023).

The prompting guidance is correspondingly narrow and structural rather than stylistic. Few-shot exemplars should remain discrete chunks; role labels should remain exactly User: and GPT:; instructions should be short and directive; and context length should be managed because excessive or overly long chunks can truncate conditioning. These recommendations follow directly from the training format rather than from a separate decoding heuristic (Li et al., 2023).

5. Empirical behavior and reported capabilities

The original Otter paper emphasized demonstrations rather than exhaustive quantitative benchmarking. In qualitative comparisons, Otter produced more detailed and precise image descriptions than OpenFlamingo, showed stronger situation understanding and commonsense reasoning, and preserved multimodal in-context learning in tasks such as meme explanation, safety-hazard description, and VQA-style counting (Li et al., 2023).

The broader MIMIC-IT evaluation reported quantitative benchmark results. On MMAGIBench, Otter with a LLaMA-7B language decoder achieved 65.5 average accuracy, compared with 51.1 for OpenFlamingo, 62.7 for LLaVA, 51.0 for MiniGPT-4, and 50.4 for InstructBLIP. The same table reports Coarse 68.9, Fine-grained 47.3, Attribute 66.3, Relation 61.8, and Future Prediction 83.3 for Otter (Li et al., 2023).

In video understanding, Otter surpassed VideoChatGPT by 6.8% accuracy on MSVD zero-shot question answering and by 1.8% on MSVD captioning, with similar margins on MSRVTT. In few-shot COCO Caption evaluation, it outperformed OpenFlamingo in all few-shot settings, while zero-shot gains were described as marginal. Human evaluation through Multi-Modality Arena placed Otter at the highest Elo rating among the compared VLMs, indicating stronger usefulness and alignment with user intentions (Li et al., 2023).

These results locate Otter in a specific research niche. It is neither merely a generic captioner nor only a benchmark-tuned VQA model. Rather, it is a conversational VLM designed around multimodal in-context instruction tuning, so its reported strengths cluster around interactive perception, reasoning, and style/task adaptation under interleaved prompting.

6. Limitations, availability, and terminological ambiguity

Otter inherits several limitations from its backbone and data sources. The paper explicitly notes language hallucination: the model may produce language unrelated to the image because it inherits hallucination tendencies from its LLaMA backbone. The authors suggest that negative examples could mitigate this, but the released model can still overgeneralize or fabricate details. Alignment quality also depends on the source datasets used to construct MIMIC-IT, including VQAv2, GQA, LLaVA, and PVSG, so biases in those sources can propagate. Additional modalities such as 3D or richer video modeling remain open problems, and OCR-heavy or domain-specific visual text may remain difficult without targeted data (Li et al., 2023).

Reproducibility is comparatively strong by the standards of early open multimodal assistants. Code and documentation are released at https://github.com/Luodian/Otter, the MIMIC-IT dataset and annotation pipeline are released, and the model is available in Huggingface-compatible form. The instruction-response data are released under CC BY-NC-SA, while visual data licenses vary by source (Li et al., 2023, Li et al., 2023).

The name “Otter” is, however, terminologically ambiguous in later arXiv literature. It was subsequently reused for unrelated systems, including a vision-language-action robot policy with text-aware visual feature extraction (Huang et al., 5 Mar 2025), an open-set multi-label tagging framework (Ouyang et al., 1 Oct 2025), a zero-shot label distribution adaptation method based on optimal transport (Shin et al., 2024), a universal multilingual named entity recognition model (Golde et al., 9 Jan 2026), a toxicity-evading jailbreak red-teaming system (Wang et al., 19 Jun 2026), a computationally efficient weather forecasting model family (Diaconu et al., 24 Jun 2026), and a software test-generation system for issue-driven TDD (Ahmed et al., 7 Feb 2025). In the multimodal instruction-tuning literature, however, “Otter” conventionally denotes the 2023 OpenFlamingo-based instruction-tuned model and its MIMIC-IT training regime (Li et al., 2023).