3Q Minimal Mix for Spatial Video-Language Models

• 3Q Minimal Mix is a data-efficient protocol that uses Metric Measurement, Perspective-Dependent Reasoning, and Temporal Tracking to develop transferable spatial intelligence.
• It leverages procedurally generated 3D scenes and 25K QA examples to fine-tune models, ensuring precise supervision with minimal redundancy.
• Empirical results show that this minimal mix significantly boosts macro accuracy on real-world spatial benchmarks compared to broader nine-category approaches.

The “3Q Minimal Mix” refers to a highly data-efficient, simulation-driven training protocol for spatial video-LLMs, introduced in the context of sim-to-real transfer for spatial video question answering. Its central result, as established in (Brown et al., 6 Nov 2025), is that merely three targeted question categories—Metric Measurement (absolute distance estimation), Perspective-Dependent Reasoning (relative direction/orientation), and Temporal Tracking (appearance order)—suffice to supervise the acquisition of transferable spatial intelligence in large video LLMs. Despite the much broader space of possible spatial questions enabled by simulation, this minimal set consistently outperforms comprehensive, nine-category coverage for real-world spatial reasoning benchmarks at comparable or lower data volume.

1. Formal Definition of the 3Q Minimal Mix

The 3Q Minimal Mix comprises three complementary question categories, each probing a distinct core axis of spatiotemporal cognition:

• Metric Measurement (Absolute Distance Estimation): Open-ended questions requiring the model to report the real-valued Euclidean distance between two objects. Example: “What is the direct distance, in meters, between the chair and the table?”
• Perspective-Dependent Reasoning (Relative Direction Determination): Multiple-choice questions involving egocentric frame transforms. Example: “If I stand by the sink and face the refrigerator, is the microwave to my front-left, front-right, back-left, or back-right?”
• Temporal Tracking (Appearance Order): Multiple-choice questions about the temporal order in which specified objects appear across frames in a video. Example: “Which appears first: the red mug, the green bottle, the blue plate, or the yellow apple?”

Each category is instantiated by programmatic template population in 3D simulation environments where ground-truth spatial and temporal information is available.

2. Methods for 3Q Data Generation and Instruction-Tuning

To instantiate the 3Q Minimal Mix, a large number of synthetic videos with associated QA-pairs are algorithmically generated:

• Environments: Procedurally constructed 3D indoor scenes are created via AI2-THOR and ProcTHOR, and populated with 30–50 objects per scene (from Objaverse). Each environment features 3–8 rooms, and videos are generated by shortest-path coverage and panoramic scans.
• Trajectory Capture: Each video contains per-frame segmentation masks, comprehensive 3D object trajectories, and object class lists, with durations ranging from 12 seconds to 3 minutes at 10 FPS.
• QA Generation:
  • Salient visible objects are filtered.
  • For each scene, for each desired axis—absolute distance, egocentric direction, and appearance order—templates are filled with sampled object permutations.
  • Distractor choices in multiple-choice questions are inserted automatically by evaluating semantic and geometric similarity.
  • Quality control ensures questions are answerable given the trajectories and room layouts.
• Scaling: The final training corpus for 3Q Minimal Mix is typically 25,000 synthetic QA examples, evenly split across the three categories.

3. Empirical Validation: Ablation Studies and Benchmark Results

Ablations performed in (Brown et al., 6 Nov 2025) systematically contrast one-category finetuning against multiaxial mixes:

• Isolation Trials: Finetuning LLaVA-Video-7B on only one category at a time (e.g., 5K examples in absolute distance or appearance order) delivers the greatest on-task boost for those axes (e.g., Δmacro ≈ +16.9% for distance, +28.9% for order—open-ended/multiple-choice respectively).
• Mix Trials: The 3Q Minimal Mix (using all three categories, balancing each) is compared against a nine-category “VSI-Baseline Mix” matching the real-world test set question distribution.
• Scaling: Macro accuracy on the real-world VSI-Bench increases monotonically with the size of the 3Q-mix, outperforming the baseline at every scale (e.g., 44.4% with 25K 3Q vs. ~42.8% for baseline, nearly matching Gemini-1.5 Pro at 45.4%).

Additional evaluation on distribution-debiased splits (VSI-Bench-Debiased), general video QA (VideoMME), embodied spatial tasks (OpenEQA), and egocentric schema (EgoSchema) demonstrates robust generalization. For instance, fine-tuning with 25K 3Q examples raises LLaVA-Video-7B’s macro accuracy on original VSI-Bench from 36.0% to 44.4% (+8.4 pts), and on VSI-Bench-Debiased from 30.7% to 38.4% (+7.7 pts), while leaving performance on domain-general video QA tasks unchanged.

4. Rationale: Complementarity, Sufficiency, and Data Efficiency

The 3Q Minimal Mix provides strict sufficiency by covering the minimal basis of transferable spatial reasoning:

• Metric questions (absolute scale) force grounding to real-world distances, which is notably difficult to learn from internet data due to lack of ground truth metrics.
• Perspective-dependent reasoning (coordinate frame transformation) ensures the model builds and applies egocentric representations, critical for navigation and embodied inference.
• Temporal tracking mandates memory over trajectories, requiring the model to handle the inherently temporal nature of video data.

By focusing on precisely one hard exemplar per axis, every training iteration contributes to the core spatial competency, yielding data efficiency: for a fixed number of examples, less information is “wasted” on redundant or easy subclasses, and learned skills transfer to other, harder variants.

This design also minimizes the risk of distribution mismatch: in full-coverage mixes, less-representative or overly generic categories (e.g., object counting, generic size estimation) may dilute learning and even harm real-world transfer.

5. Implementation Guidelines for Practitioners

A practical “recipe” for instantiating the 3Q Minimal Mix for video LLM finetuning is as follows (as established in (Brown et al., 6 Nov 2025)):

1. Generate a procedurally rich set of 3D scene videos using a modern simulation engine (AI2-THOR, Habitat, etc.), ensuring diverse spatial layouts and object coverage.
2. Extract full 3D object positions and per-frame trajectories from the simulator.
3. Sample balanced 3 × N question-answer pairs:
   • N Absolute Distance Estimation (open-ended, real-valued)
   • N Relative Direction (medium-difficulty egocentric, multiple-choice)
   • N Appearance Order (temporal, multiple-choice)
4. Finetune the target video LLM on these 3Q examples (e.g., 25K total), using a learning rate ≈ 1e-6 for a few epochs, matching pre-training statistics as appropriate.
5. Evaluate zero-shot on real-world spatial reasoning benchmarks; empirical expectation is a 5–10 point macro accuracy gain.

6. Theoretical and Practical Implications

The sufficiency of the 3Q Minimal Mix can be interpreted as supervision for three “degrees of freedom”: real-world metric scale, coordinate frame orientation, and temporal object memory. The empirical evidence that no additional question classes significantly improve real-world transfer implies a covering of all principal axes along which spatial generalization occurs. Counting, size, and other categories are either subsumed or add no independent supervision for the hard cases.

This finding establishes a clear and reproducible standard for data-efficient instruction tuning of spatial video-LLMs in simulation. A plausible implication is that any sim-to-real framework requiring nontrivial spatial generalization should interrogate its coverage of these three axes before scaling up breadth.

7. Comparative Performance and Generalization Scope

The 3Q Minimal Mix unlocks superior sim-to-real transfer compared to comprehensive or ad hoc mixes both in macro accuracy and sample efficiency. For models trained with 25K 3Q examples, performance matches or exceeds much larger models or those trained on proprietary data. On non-spatial general video understanding tasks, no regression is observed, indicating that specialization on 3Q does not degrade generalization.

Summary metrics (from (Brown et al., 6 Nov 2025)):

• LLaVA-Video-7B (macro accuracy, VSI-Bench): 36.0% (init) → 44.4% (3Q, 25K)
• Largest category-wise boost: Absolute Distance +20.0 pts, Appearance Order +26.4 pts
• Transfer to out-of-domain benchmarks (OpenEQA, MMRealWorld): up to +8.6 pts
• Comparable or superior to Gemini-1.5 Pro (45.4%) at a fraction of the data volume

This supports the 3Q Minimal Mix as a mathematically and empirically grounded data curriculum for scalable spatial video reasoning.