--- title: 'JavisInst-Omni: Multimodal Instruction Dataset' url: https://www.emergentmind.com/topics/javisinst-omni-instruction-dataset type: topic --- # JavisInst-Omni: Multimodal Instruction Dataset JavisInst-Omni is a family of instruction-tuning datasets designed to enable highly capable, multimodal large language models (MLLMs) for instruction-following tasks involving complex interactions among vision, language, action, and, in certain variants, synchronized audio-video streams. Datasets under the JavisInst-Omni name span grounded embodied learning scenarios (e.g., open-world Minecraft agents) as well as synchronous sounding-video comprehension and generation, supporting unified benchmarking and instruction tuning for next-generation multimodal AI systems [2407.00114][2512.22905]. ## 1. Dataset Family: Scope and Modal Coverage JavisInst-Omni refers to two core datasets introduced independently to support large-scale vision-language-action (VLA) models and multimodal LLMs for sounding-video (JAV) comprehension and generation, respectively. - **OmniJARVIS JavisInst-Omni** [2407.00114]: A ∼1T-token-scale, fully annotated, multimodal corpus for training open-world instruction-following agents in Minecraft. It consists of 600,000 trunked Minecraft interactions with language, vision, and action segments, designed for unified tokenization and end-to-end training of VLA transformers. - **JavisGPT JavisInst-Omni** [2512.22905]: A ≈200,000-example dataset curated for instruction tuning unified audio-video-text LLMs. It supports both comprehension (QA, captioning) and generation (text→AV, AV-conditioned) in temporally aligned, sounding-video settings, with careful dialogue curation and role labeling. This dual-use of the "JavisInst-Omni" name reflects converging efforts to construct large, high-quality, instruction-centric datasets for autonomous agents in both grounded (embodied) and audio-visual media environments. ## 2. Data Collection Methodologies and Construction Pipeline The construction processes are tailored to their respective domains: **For OmniJARVIS JavisInst-Omni [2407.00114]:** - **Source Trajectories**: Draws primarily from OpenAI’s VPT Minecraft gameplay dataset and JARVIS-1 agent interactions, extracting 600,000 trunks of 128 frames each. - **Sampling Protocol**: Segments are sampled every 32 frames, enforcing regular re-planning intervals and granular behavior segmentation. - **Task Suite**: - *Atomic tasks*: Four low-level skill types (e.g., chop trees, dig dirt). - *Programmatic tasks*: Thirty medium-horizon crafting tasks, grouped by difficulty (wooden, food, stone, iron, diamond). - *Open-ended creative*: Freeform, unconstrained goals (e.g., “build an oak boat and sail”). - *Embodied QA (EQA)*: Agent-centric question answering, requiring memory, reasoning, and planning. - **Instruction and Reasoning Generation**: Instructions are synthesized by prompting GPT-3.5 based on gameplay meta-events; agent reasoning (chain-of-thought) and memory segments are generated with LLM summarization. **For JavisGPT JavisInst-Omni [2512.22905]:** - **Modalities and Aggregate Sources**: - Audio-only QA (55K) from AudioSet, AudioCaps, VGGSound, and others. - Video-only QA (60K) from LLaVA-Video-178K. - Image-only QA (20K) from LLaVA-OneVision. - AV Comprehension and Generation: 95K AV-QA from VideoLLaMA2, 20K AV captions from TAVGBench, 150K text→AV from TAVGBench, with 200K total dialogues. - **Synthetic Generation and Curation**: - JavisInst-Und (110K): Comprehension dialogues, created via GPT-4o using prompt templates specific to subcategory and difficulty. - JavisInst-Gen (90K): Generation-oriented dialogues, created with 3,000 prompt templates and further paraphrased for diversity using GPT-4o-mini. - Human-in-the-loop: Manual spot-checking (≈10%) for compliance and correctness, with re-prompting as needed. ## 3. Data Representation, Tokenization, and Dialogue Structuring Both datasets emphasize unified tokenization and rich multimodal context encoding. **OmniJARVIS JavisInst-Omni [2407.00114]:** - **Segment Structure and Format**: - $D^{(\text{inst})} \rightarrow D^{(\text{mem})} \rightarrow D^{(\text{obs})} \rightarrow D^{(\text{tht})} \rightarrow D^{(\text{bhv})} \rightarrow D^{(\text{obs})} \ldots$ - *Instruction* ($D^{(\text{inst})}$): Natural language summary of the goal. - *Memory* ($D^{(\text{mem})}$): LLM-summarized subtask history. - *Observation* ($D^{(\text{obs})}$): 128×128 RGB frame, tokenized by LLaVA-7B’s vision encoder. - *Chain-of-thought* ($D^{(\text{tht})}$): LLM-produced rationale, conditioned on the current context. - *Behavior* ($D^{(\text{bhv})}$): 128-frame trajectory, compacted into 5 discrete behavior tokens via Finite Scalar Quantization (FSQ). - **Unified Tokenizer**: A combined vocabulary merging BPE language tokens (~50K), special visual prefix tokens, and 35 behavior tokens enables a single transformer to model text, vision, and action in an autoregressive sequence. - **Packing Example**: ``` [INST] "..."; [MEM] "..."; ; [THOUGHT] "..."; , , , , ``` At inference, the agent emits chains-of-thought and behavior tokens in alternation every N=32 frames. **JavisGPT JavisInst-Omni [2512.22905]:** - **Dialogue Roles and Structure**: - *System preamble* (optional context) - *User* ("mquered:") queries or issues instructions. - *Assistant* ("mansgreen:") responds with answers and, in generation, AV placeholders ("<|av_start|>...[video+audio]...<|av_end|>"). - **Instruction Taxonomy**: - *Comprehension*: Entity, Relation, and Global-level queries (existence, alignment, grounding, counting; spatial, temporal, causal; emotion, atmosphere, theme). - *Generation*: Formal/colloquial instruction→AV, conditioning by modality (V2A, A2V, I2AV, AV-Extend, AV-Edit), and multi-turn scenarios (Proactive, Rethink, Und2Gen). - **Annotation Schema**: Each dialogue is enriched with modality context, instruction type, difficulty, ground-truth answers, and explanation fields. Provenance and human-verification flags are included. ## 4. Tokenization Algorithms and Multimodal Integration Both datasets advance unified tokenization and effective merging of multimodal information streams. **OmniJARVIS JavisInst-Omni [2407.00114]:** - **Behavior Tokenization via FSQ**: - Trajectory latent $z_e = \text{Encoder}(o_{1:128}) \in \mathbb{R}^d$ is quantized by nearest neighbor to 5 FSQ codebooks; index sets have sizes [8, 8, 8, 6, 5] yielding 35 unique behavior tokens. - Objective: $$ \underset{\phi, \theta}{\arg\min}\; \mathbb{E}_{\tau \sim D} \left[-\sum_{t=1}^{128} \log \pi_\theta(a_t \mid o_{1:t}, \{s_1...s_5\}) \right] $$ - Enables compact, semantically meaningful trajectories that facilitate long-range planning with efficient autoregressive modeling. - **Vocabulary Fusion**: Language, vision, and behavior tokens are merged in a single autoregressive stream, allowing the transformer to interleave text instructions, visual context, reasoning, and discrete actions for end-to-end instruction-following. **JavisGPT JavisInst-Omni [2512.22905]:** - **Tokenization and Training Objective**: - All instruction dialogues (text, AV, multi-turn) are tokenized for transformer-based modeling. - The loss incorporates next-token prediction, alignment between LLM queries and DiT condition embeddings, and diffusion denoising for AV generation: $$ \mathcal{L} = \mathcal{L}_{\text{ntp}} + \mathcal{L}_{\text{align}} + \mathcal{L}_{\text{diff}} $$ - **Synchrony-Aware Data**: Emphasizes explicit temporal alignment of video and audio in both comprehension and generation, enforced at the data and model architecture level. ## 5. Dataset Statistics and Quality Assurance Rigorous annotation and curation underpin both datasets. | Dataset | Scale | Modalities | Dialogue Types | Key Quality Metrics | |------------------------------------|-----------------------|---------------------|------------------------------------|--------------------------------------| | OmniJARVIS JavisInst-Omni | 600K interactions | Text, vision, action| Instruction-following, EQA | ~1T tokens, ≥0.9 atomic reward, 59% programmatic success | | JavisGPT JavisInst-Omni | ≃200K dialogues | Audio, video, text | QA (JavisInst-Und), Gen (JavisInst-Gen) | 98% format compliance, ≥95% QA accuracy, 0.55 comprehension, 0.45 generation | - **OmniJARVIS JavisInst-Omni**: - ~600,000 interaction segments, ~900M tokens for instruction-following, 300,000 QA dialogues, for a total of ~1T tokens. - Benchmarks: Atomic tasks (reward >0.9), programmatic (59% agent success vs. baselines), creative (FSD 886.25 vs. 929–975), EQA (GPT-4 Elo 8.40). - **JavisGPT JavisInst-Omni**: - N_total ≃ 200,000; p_und = 0.55, p_gen = 0.45. - Spot-check accuracy: ≥98% format compliance, ≥95% QA answer correctness. Inconsistent or incorrect dialogues (<2%) are dropped or regenerated. ## 6. Example Instances and Dialogue Illustrations Representative samples demonstrate the multimodal, instruction-centric character: **OmniJARVIS JavisInst-Omni Example [2407.00114]:** - Packed token sequence for a crafting task: ``` [INST] Give you nothing in the inventory, obtain a diamond pickaxe. [MEM] Gathered logs, crafted wooden tools, mined stone. // Encoded 128x128 image [THOUGHT] I have 3 iron ingots but no sticks—craft sticks from planks before making an iron pickaxe. , , , , ``` The agent alternately emits chain-of-thought and behavior tokens every 32 frames using the full context history. **JavisGPT JavisInst-Omni Example [2512.22905]:** - *Comprehension (entity-alignment)*: - Context: Video shows a lion; audio is a roaring sound. - User: “Does the lion in the video make the roaring sound in the audio?” - Assistant: “Yes. The audio roar aligns with the visual of the lion roaring.” - *Generation (multi-turn, proactive)*: - User: “Create a video where a car drives by with an orange balanced on its roof.” - Assistant: “Sure—here’s a clip of a red sedan passing by with a bright orange perched on top, you can hear the soft hum of the engine and the gentle thump as the fruit wobbles.” `<|av_start|>[8s 240×240 video + 16 kHz stereo audio]<|av_end|>` ## 7. Usage Scenarios, Benchmarks, and Adoption JavisInst-Omni datasets are designed as comprehensive resources for benchmarking and developing open-ended, instruction-following multimodal agents. - **OmniJARVIS JavisInst-Omni**: - Trains and evaluates VLA transformers in open-world settings; supports atomic, programmatic, and creative task spectra. - Evaluation: Task-specific metrics (reward, success rate, FSD for creativity, EQA Elo). - **JavisGPT JavisInst-Omni**: - Enables MLLMs for audio/video QA, captioning, AV generation, and multi-turn instruction-following. - Benchmarks supported: AVQA, MU-AVQA, AVSD, JavisBench-mini (generation), and chatbot scenarios. - Reported gains: AVQA accuracy gains (91.5%→93.8%), FVD improvement (327.8→317.5), and enhanced AV synchrony (0.153→0.157). *This suggests* that JavisInst-Omni datasets play a key role in the scaling and unification of multimodal LLMs for diverse, instruction-centric robotic and media domains, offering systematic evaluation and robust compositional supervision [2407.00114][2512.22905]. Source: https://www.emergentmind.com/topics/javisinst-omni-instruction-dataset