--- title: 'Ultimate3D: 3D Spatial Benchmark' url: https://www.emergentmind.com/topics/ultimate3d type: topic --- # Ultimate3D: 3D Spatial Benchmark Ultimate3D is a large-scale dataset and benchmarking platform designed to advance the spatial reasoning ability of multimodal large language models (MLLMs) with a focus on fine-grained camera-object relations. The dataset comprises 240,000 visual question-answering (VQA) pairs, produced using a systematic synthetic generation pipeline that combines 3D asset rendering with diffusion-based photorealistic image synthesis and automated language prompt generation. Each image is furnished with comprehensive, ground-truth annotations for object orientation, camera viewpoint, and camera shot type, enabling robust assessment and targeted training of spatially-aware MLLMs. Empirical results indicate that fine-tuning MLLMs on Ultimate3D yields a mean accuracy improvement of 33.4% on previously challenging camera-object relation tasks [2507.08513]. ## 1. Dataset Generation Pipeline Ultimate3D’s data generation pipeline starts with a curated collection of textured 3D meshes sourced from ObjaverseXL and ShapeNet, each labeled with an ImageNet synset category. To uniformly sample the space of camera-object relations, a parameter configuration $\beta$ is defined as follows: - **Object azimuth $\varphi$**: $[0, 2\pi)$, discretized into 8 bins of width $\frac{\pi}{4}$. - **Camera elevation $\theta$**: $[-\frac{\pi}{2},+\frac{\pi}{2}]$, discretized into 3 bins of width $\frac{\pi}{3}$. - **Camera-object distance $D$**: $[0,\infty)$, measured in Blender units relative to the object's bounding-box size, discretized at $D = 1.25$ (close-up), $D = 3.0$ (long-shot), else medium-shot. For each asset $A$ at centroid $X_0$, a classical pinhole camera model is instantiated, with extrinsic parameters $$ \beta = \{\varphi, \theta, D\},\quad R = R_{\rm elev}(\theta)R_{\rm az}(\varphi),\quad t = -R X_0 + \begin{bmatrix}0\\0\\D\end{bmatrix}. $$ where $$ R_{\rm az}(\varphi) = \begin{bmatrix} \cos\varphi & -\sin\varphi & 0 \\ \sin\varphi & \cos\varphi & 0 \\ 0 & 0 & 1 \end{bmatrix},\quad R_{\rm elev}(\theta) = \begin{bmatrix} 1 & 0 & 0 \\ 0 & \cos\theta & -\sin\theta \\ 0 & \sin\theta & \cos\theta \end{bmatrix}. $$ A 3D point $X$ transforms to image plane coordinates $x$ using a fixed intrinsic matrix $K$ (35 mm focal length) via $ \tilde x = K\,[\,R|t\,]\,\tilde X $, where $x = (\tilde x_1/\tilde x_3, \tilde x_2/\tilde x_3)$. The generation steps are: 1. **Sampling**: For each asset, uniformly sample all $8\times3\times3=72$ $\beta$ configurations. 2. **Rendering**: Produce 3D visual priors per configuration with Blender (RGB pass, depth, segmentation mask); compute Canny edges. 3. **Image Prompting**: Use GPT-4o to generate a diverse textual prompt $T_{\rm img}$ from each asset's synset. 4. **Diffusion-based Synthesis**: Condition an SDXL diffusion backbone via multiple ControlNets (depth, edge), combining guidance at each timestep, and synthesize a photorealistic image $I_{\rm syn}$ through iterative denoising: $$ z_{t-1} = \mathcal G_\theta(z_t,\,t,\,T_{\rm img},\,\sum_k w_k\,\mathcal C^k(I^k_\beta)). $$ 5. **VQA Generation**: Re-prompt GPT-4o with $\beta$ and a few-shot template to construct up to three QA instances—orientation (8-way), viewpoint (3-way), and shot type (3-way). ## 2. Dataset Composition and Schema Ultimate3D comprises: - **Assets**: 1,196 meshes covering 100 ImageNet synsets. - **Synthetic images**: Approximately 85,000 rendered and diffusion-refined images (1,180 assets $\times$ 72 settings). - **Real human images**: 18,000 instances from the MEBOW monocular orientation dataset, included for human-category representation. - **VQA pairs**: 1–3 QA annotations per synthetic image, totaling 240,000 VQA items; each MEBOW image has one orientation QA, adding 18,000 entries. Each VQA is annotated as follows: - $\varphi$ (azimuth): floating-point radian value, discretized into 8 labels. - $\theta$ (elevation): numerical, binned into three ("bottom", "horizontal", "top"). - $D$ (distance): object-to-image height ratio, labeled as "close-up", "medium-shot", "long-shot". **Example VQA Instance** - **Image**: Photorealistic sedan on a beach - **Annotation**: $\varphi=\frac{\pi}{2}$ (faces right), $\theta=0.2$ rad (horizontal), $D=1.8$ (medium) - **Q1**: Which direction is the car facing? (a) Front (b) Right (c) Back (d) Left → (b) Right - **Q2**: Is the viewpoint from above or at eye-level? (1) Top (2) Horizontal (3) Bottom → (2) - **Q3**: What kind of shot is this? (i) Close-up (ii) Medium-shot (iii) Long-shot → (ii) ## 3. Benchmarking and Empirical Evaluation Ultimate3D provides two standardized test sets: 1. **Synthetic**: 1,200 images $\times$ 3 QA = 3,600 items, verified by human reviewers. 2. **Real**: 2,443 Pascal3D+ images (10 classes) plus 800 MEBOW human images, totaling 3,243 QA items. Evaluation is by classification accuracy (model answer vs. ground truth) on three subtasks: - Orientation (8-way) - Viewpoint (3-way) - Shot type (3-way) **Comparative Baseline and Improved Results:** | Model | Orientation | Viewpoint | Shot Acc. | Mean Gain (over GPT-4o) | |--------------------------------|-------------|-----------|-----------|-------------------------| | LLaVA-1.6-13B (pre-finetune) | 16% | 31% | 42% | ~0% | | LLaVA-1.6-13B (post-Ultimate3D)| 72% | 72% | 95% | +33.4 % | | SOTA Commercial (GPT-4o, etc.) | N/A | N/A | N/A | Baseline | A similar mean gain of 19.2% is observed on the MMVP external benchmark after Ultimate3D fine-tuning. **Training Regimen:** - Fine-tune vision-language connector MLP and LLM decoder; freeze vision encoder. - 1 epoch over 240,000 VQA pairs; batch size ~64/GPU; total 12 hours on 4×H100 GPUs. - Hyperparameters as in LLaVA-1.6 (LR $1 \times 10^{-5}$, weight decay 0.01). **Ablation Results:** - Removing depth or all ControlNet priors drops generation success rates (93% with depth+edge vs. 55% for prior pipelines). - SDXL backbone achieves +5 CLIP-I fidelity, –32 FID, +4 CLIP-T over SD-v1.5. ## 4. Integration and Use in Multimodal Training Transfer of Ultimate3D to existing multimodal pipelines is direct: 1. Merge a portion (e.g., 50%) of Ultimate3D with image-based instruction corpora (e.g., LLaVA-Instruct-665K). A 50/50 split maintains VQAv2 performance and raises camera-relation accuracy from 29.7% to 74.7%. 2. Fine-tune connector+decoder modules for 1–2 epochs; vision encoders remain fixed, ensuring compute efficiency. 3. Optionally, include 18,000 human-photo VQAs for human orientation recognition tasks. Ultimate3D thus enables models to acquire spatial reasoning skills otherwise unattainable with conventional image datasets. ## 5. Limitations and Future Research Directions While Ultimate3D represents a substantial advance, several limitations remain: - All current QAs employ categorical bins; fine-grained numerical regression of camera-object angles is not yet viable at scale. - Multi-object scenes with structured occlusion and open-vocabulary object detection are not supported. - Dataset generation and evaluation are restricted to static scenes; extension to dynamic (video or synchronized multiview) contexts is a prospective avenue. *This suggests* that while foundational for spatial reasoning, further work is required to address regression targets, real-world complexity, and temporal dynamics. ## 6. Contribution and Impact Ultimate3D addresses a critical gap in spatially grounded VQA by systematically providing high-fidelity camera-object annotation at scale—enabling statistically robust MLLM evaluation and instruction tuning. Fine-tuning pre-trained open-source MLLMs on Ultimate3D achieves accuracy gains exceeding 33% on key spatial understanding metrics, surpassing contemporary commercial models on specialized tasks. *A plausible implication is* that such unbiased, synthetic 3D-aware datasets are essential for advancing multimodal agents suited for physically grounded interaction, robotics, and augmented reality [2507.08513]. Source: https://www.emergentmind.com/topics/ultimate3d