Ultimate3D: 3D Spatial Benchmark

• Ultimate3D is a comprehensive dataset and benchmarking platform designed to advance spatial reasoning in multimodal models with fine-grained camera-object annotations.
• It employs a systematic synthetic generation pipeline combining 3D asset rendering, diffusion-based photorealistic synthesis, and automated language prompt generation.
• Empirical results show a 33.4% accuracy improvement on challenging camera-object relation tasks after fine-tuning multimodal large language models.

Ultimate3D is a large-scale dataset and benchmarking platform designed to advance the spatial reasoning ability of multimodal LLMs (MLLMs) with a focus on fine-grained camera-object relations. The dataset comprises 240,000 visual question-answering (VQA) pairs, produced using a systematic synthetic generation pipeline that combines 3D asset rendering with diffusion-based photorealistic image synthesis and automated language prompt generation. Each image is furnished with comprehensive, ground-truth annotations for object orientation, camera viewpoint, and camera shot type, enabling robust assessment and targeted training of spatially-aware MLLMs. Empirical results indicate that fine-tuning MLLMs on Ultimate3D yields a mean accuracy improvement of 33.4% on previously challenging camera-object relation tasks (He et al., 11 Jul 2025).

1. Dataset Generation Pipeline

Ultimate3D’s data generation pipeline starts with a curated collection of textured 3D meshes sourced from ObjaverseXL and ShapeNet, each labeled with an ImageNet synset category. To uniformly sample the space of camera-object relations, a parameter configuration $\beta$ is defined as follows:

• Object azimuth $\varphi$: $[0, 2\pi)$, discretized into 8 bins of width $\frac{\pi}{4}$.
• Camera elevation $\theta$: $[-\frac{\pi}{2},+\frac{\pi}{2}]$, discretized into 3 bins of width $\frac{\pi}{3}$.
• Camera-object distance $D$: $[0,\infty)$, measured in Blender units relative to the object's bounding-box size, discretized at $D = 1.25$ (close-up), $\varphi$0 (long-shot), else medium-shot.

For each asset $\varphi$1 at centroid $\varphi$2, a classical pinhole camera model is instantiated, with extrinsic parameters

$\varphi$3

where

$\varphi$4

A 3D point $\varphi$5 transforms to image plane coordinates $\varphi$6 using a fixed intrinsic matrix $\varphi$7 (35 mm focal length) via $\varphi$8, where $\varphi$9.

The generation steps are:

1. Sampling: For each asset, uniformly sample all $[0, 2\pi)$0 $[0, 2\pi)$1 configurations.
2. Rendering: Produce 3D visual priors per configuration with Blender (RGB pass, depth, segmentation mask); compute Canny edges.
3. Image Prompting: Use GPT-4o to generate a diverse textual prompt $[0, 2\pi)$2 from each asset's synset.
4. Diffusion-based Synthesis: Condition an SDXL diffusion backbone via multiple ControlNets (depth, edge), combining guidance at each timestep, and synthesize a photorealistic image $[0, 2\pi)$3 through iterative denoising:

$[0, 2\pi)$4

1. VQA Generation: Re-prompt GPT-4o with $[0, 2\pi)$5 and a few-shot template to construct up to three QA instances—orientation (8-way), viewpoint (3-way), and shot type (3-way).

2. Dataset Composition and Schema

Ultimate3D comprises:

• Assets: 1,196 meshes covering 100 ImageNet synsets.
• Synthetic images: Approximately 85,000 rendered and diffusion-refined images (1,180 assets $[0, 2\pi)$6 72 settings).
• Real human images: 18,000 instances from the MEBOW monocular orientation dataset, included for human-category representation.
• VQA pairs: 1–3 QA annotations per synthetic image, totaling 240,000 VQA items; each MEBOW image has one orientation QA, adding 18,000 entries.

Each VQA is annotated as follows:

• $[0, 2\pi)$7 (azimuth): floating-point radian value, discretized into 8 labels.
• $[0, 2\pi)$8 (elevation): numerical, binned into three ("bottom", "horizontal", "top").
• $[0, 2\pi)$9 (distance): object-to-image height ratio, labeled as "close-up", "medium-shot", "long-shot".

Example VQA Instance

• Image: Photorealistic sedan on a beach
• Annotation: $\frac{\pi}{4}$0 (faces right), $\frac{\pi}{4}$1 rad (horizontal), $\frac{\pi}{4}$2 (medium)
• Q1: Which direction is the car facing? (a) Front (b) Right (c) Back (d) Left → (b) Right
• Q2: Is the viewpoint from above or at eye-level? (1) Top (2) Horizontal (3) Bottom → (2)
• Q3: What kind of shot is this? (i) Close-up (ii) Medium-shot (iii) Long-shot → (ii)

3. Benchmarking and Empirical Evaluation

Ultimate3D provides two standardized test sets:

1. Synthetic: 1,200 images $\frac{\pi}{4}$3 3 QA = 3,600 items, verified by human reviewers.
2. Real: 2,443 Pascal3D+ images (10 classes) plus 800 MEBOW human images, totaling 3,243 QA items.

Evaluation is by classification accuracy (model answer vs. ground truth) on three subtasks:

• Orientation (8-way)
• Viewpoint (3-way)
• Shot type (3-way)

Comparative Baseline and Improved Results:

Model	Orientation	Viewpoint	Shot Acc.	Mean Gain (over GPT-4o)
LLaVA-1.6-13B (pre-finetune)	16%	31%	42%	~0%
LLaVA-1.6-13B (post-Ultimate3D)	72%	72%	95%	+33.4 %
SOTA Commercial (GPT-4o, etc.)	N/A	N/A	N/A	Baseline

A similar mean gain of 19.2% is observed on the MMVP external benchmark after Ultimate3D fine-tuning.

Training Regimen:

• Fine-tune vision-language connector MLP and LLM decoder; freeze vision encoder.
• 1 epoch over 240,000 VQA pairs; batch size ~64/GPU; total 12 hours on 4×H100 GPUs.
• Hyperparameters as in LLaVA-1.6 (LR $\frac{\pi}{4}$4, weight decay 0.01).

Ablation Results:

• Removing depth or all ControlNet priors drops generation success rates (93% with depth+edge vs. 55% for prior pipelines).
• SDXL backbone achieves +5 CLIP-I fidelity, –32 FID, +4 CLIP-T over SD-v1.5.

4. Integration and Use in Multimodal Training

Transfer of Ultimate3D to existing multimodal pipelines is direct:

1. Merge a portion (e.g., 50%) of Ultimate3D with image-based instruction corpora (e.g., LLaVA-Instruct-665K). A 50/50 split maintains VQAv2 performance and raises camera-relation accuracy from 29.7% to 74.7%.
2. Fine-tune connector+decoder modules for 1–2 epochs; vision encoders remain fixed, ensuring compute efficiency.
3. Optionally, include 18,000 human-photo VQAs for human orientation recognition tasks.

Ultimate3D thus enables models to acquire spatial reasoning skills otherwise unattainable with conventional image datasets.

5. Limitations and Future Research Directions

While Ultimate3D represents a substantial advance, several limitations remain:

• All current QAs employ categorical bins; fine-grained numerical regression of camera-object angles is not yet viable at scale.
• Multi-object scenes with structured occlusion and open-vocabulary object detection are not supported.
• Dataset generation and evaluation are restricted to static scenes; extension to dynamic (video or synchronized multiview) contexts is a prospective avenue.

This suggests that while foundational for spatial reasoning, further work is required to address regression targets, real-world complexity, and temporal dynamics.

6. Contribution and Impact

Ultimate3D addresses a critical gap in spatially grounded VQA by systematically providing high-fidelity camera-object annotation at scale—enabling statistically robust MLLM evaluation and instruction tuning. Fine-tuning pre-trained open-source MLLMs on Ultimate3D achieves accuracy gains exceeding 33% on key spatial understanding metrics, surpassing contemporary commercial models on specialized tasks. A plausible implication is that such unbiased, synthetic 3D-aware datasets are essential for advancing multimodal agents suited for physically grounded interaction, robotics, and augmented reality (He et al., 11 Jul 2025).