---
title: 'JavisInst-Omni: Multi-Modal Instruction Dataset'
url: https://www.emergentmind.com/topics/javisinst-omni
type: topic
---

# JavisInst-Omni: Multi-Modal Instruction Dataset

JavisInst-Omni is a large-scale, high-quality, multi-modal instruction dataset, designed to advance the capabilities of unified multimodal large language models (MLLMs) for temporally coherent joint audio-video (JAV) understanding and generation tasks. Developed and curated to support the JavisGPT framework, it provides over 200,000 GPT-4o-synthesized audio-video-text dialogues covering diverse and fine-grained synchrony-aware comprehension, generation, and multi-turn conversational scenarios. Its architecture and methodology directly inform state-of-the-art approaches in audio-video multi-modal machine learning and benchmarking [2512.22905].

## 1. Dataset Composition and Scope

JavisInst-Omni comprises more than 200,000 multimodal dialogue trajectories, resulting in approximately 600,000 single-turn samples encompassing unimodal (audio-, video-, or image-only), bimodal (audio-text, video-text), and joint audio-video-text instruction types. The dataset includes:

- Audio + Text QA: 55,000 samples
- Video + Text QA: 60,000 samples
- Image + Text QA: 20,000 samples
- Audio-Video + Text QA: 95,000 samples
- Audio-Video captioning: 20,000 samples
- Text-to-Audio-Video generation: 150,000 samples
- JavisInst-Und (synchrony-aware AV-QA): 110,000 samples
- JavisInst-Gen (generation & multi-turn): 90,000 samples

Instruction types span comprehension tasks (entity/relation/global-level AV-QA, single-turn and joint), conditional and proactive AV generation (text→AV, V→A, A→V, etc.), captioning, and rich multi-turn dialogues, including composite QA or understand-then-generate sessions. This broad coverage ensures robust evaluation and training for multimodal generative and reasoning models.

## 2. Data Construction, Curation, and Verification

Underlying audio-video pairs are sourced from benchmark data such as TAVGBench (1.5 million captioned AV clips) for pretraining/fine-tuning, with QA samples drawn from VideoLLaMA2 (95,000 AV-QA), LLaVA-Video-178K (60,000 video QA), and related datasets. Editing pairs leverage InstV2V with additional synthetic audio via FoleyCrafter. All clips maintain inherent audio-video synchronization; AV-extend samples use 1–2 s overlaps for temporal alignments.

Dialogues, instructions, and QA are synthesized by GPT-4o using a comprehensive set of 3,000+ templates spanning ten synchrony-aware QA and eleven AV generation types. Approximately 20% of text samples are paraphrased with GPT-4o-mini for linguistic diversity. Human verification is performed on at least 95% of the data, with annotators checking for logical correctness, modality matching, and explicit synchrony. Erroneous/ambiguous items are either eliminated or rewritten. This rigorous curation ensures both scale and fidelity in synchrony-aware, semantically diverse multi-modal data.

## 3. Annotation Schema and Task Typology

Each sample adopts a structured, schema-driven JSON format. Key fields include input modalities (audio/video URLs, first frames, timestamps), instruction text, answer or generation target, category/type tag, optional choices, and natural language explanations. Two schemas predominate:

- **JavisInst-Und (synchrony-aware AV-QA):**  
  Contains fields for video/audio URLs, queried time intervals, explicit reasoning category (e.g., "relation-temporal"), multi-choice options, ground truth answer, and rationale.
- **JavisInst-Gen (generation/multi-turn):**  
  Carries prior dialogue, textual instructions for AV creation or editing, and target AV clip references.

Categories are explicitly labeled for synchrony demands: entity alignment, relation (spatial/temporal/causal), global atmosphere, emotion, theme, etc. Generation instructions employ both formal and colloquial phrasing, with conditional AV manipulations (e.g., video→audio, audio→video extensions) and multi-turn sequences.

## 4. Quantitative Characteristics and Diversity Measures

The dataset features balanced sampling across comprehension and generation sub-tasks. For JavisInst-Und, ten categories each hold approximately 11,000 examples. JavisInst-Gen encompasses eleven instruction types (six conditional, three multi-turn, two register):

| Und Category         | Samples  |
|----------------------|----------|
| Existence            | 11,000   |
| Alignment            | 11,000   |
| Grounding            | 11,000   |
| Counting             | 11,000   |
| Spatial relation     | 11,000   |
| Temporal relation    | 11,000   |
| Causal relation      | 11,000   |
| Emotion              | 11,000   |
| Atmosphere           | 11,000   |
| Theme                | 11,000   |
| **Total**            | 110,000  |

Entropy of instruction type is measured as  
$$H = -\sum_{i=1}^C p_i \log p_i$$  
with $p_i \approx 1/C$ for near-uniform split, giving $H \approx \log 10 \approx 2.3$ nats ($\approx 3.3$ bits) for JavisInst-Und. This suggests substantial scenario diversity. Generation types in JavisInst-Gen exhibit similar uniformity, supporting robust generalization and evaluation.

## 5. Benchmarking Role and Empirical Impact

JavisInst-Omni underpins the three-stage JavisGPT training regime:

1. MM-PreTrain: audio/text and audio–text/vision alignment (600 K audio-text, 1.5 M captions).
2. AV-FineTune: synchronized AV comprehension and generation (360 K captions, 360 K triplets).
3. MM-InstTune: instruction-tuning on JavisInst-Omni (≈600 K instructions).

Ablation studies demonstrate critical dependence on JavisInst-Omni:  
- Omitting JavisInst-Und reduces AV comprehension accuracy by ~1–2 points.
- Omitting JavisInst-Gen lowers JavisScore synchrony from 0.157 to ~0.135.
- Joint comprehension + generation (with JavisInst-Omni) yields a +2.3 FVD improvement and +0.018 JavisScore versus separate training streams.

A plausible implication is that the fine-grained synchrony-aware and multi-turn design of JavisInst-Omni directly translates into enhanced MLLM performance for temporally coherent, multi-modal understanding and generation [2512.22905].

## 6. Significance and Usage Outlook

JavisInst-Omni represents the first GPT-4o–curated instruction corpus at scale for cross-modal, synchrony-grounded audio-video reasoning and generation. Its balanced scenario type coverage, rigorous curation, and explicit annotation of temporal, spatial, and causal alignment positions it as a standard for both supervised model development and fair evaluation in audio-video MLLM research. Typical uses span benchmark evaluation, instruction-tuning for emergent LLMs, and as a foundation for further dataset expansion targeting even more complex, temporally entangled multi-modal reasoning workflows.

Source: https://www.emergentmind.com/topics/javisinst-omni