ShapeLLM-Omni: Native 3D Multimodal LLM

• The paper introduces a novel autoregressive framework that natively integrates 3D token support via a discrete VQVAE for unified 3D generation, comprehension, and editing.
• ShapeLLM-Omni unifies text, image, and 3D modalities by merging their tokens into a single sequence for next-token prediction over a shared transformer backbone.
• Empirical results on multiple benchmarks and 3D tasks demonstrate effective multimodal integration while highlighting areas for further scaling and improved editing quality.

ShapeLLM-Omni most commonly denotes the native multimodal LLM introduced in "ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understanding" (Ye et al., 2 Jun 2025). In that formulation, the system extends a native multimodal LLM beyond text and images to include 3D generation and understanding in a single autoregressive framework, coupling a 3D vector-quantized variational autoencoder with instruction-based training of Qwen-2.5-VL-Instruct-7B on the 3D-Alpaca dataset. The resulting model supports text-to-3D, image-to-3D, 3D-to-caption, and 3D-editing within one shared transformer. A source of terminological ambiguity is that the label also appears in a separate parametric-shape context as an omni-modal extension of the BodyShapeGPT paradigm for SMPL-X, FLAME, MANO, and a synthetic quadruped model (Árbol et al., 2024). The published 2025 system and the parametric-shape extension are related by the ambition to unify multimodal shape reasoning, but they differ in representation, objectives, and output space.

1. Definition, scope, and research lineage

The 2025 ShapeLLM-Omni paper positions the model as a native 3D LLM capable of understanding and generating 3D assets and text in any sequence (Ye et al., 2 Jun 2025). Its stated goal is to extend a native multimodal LLM, described as “ChatGPT-4o style,” beyond text and images to include 3D generation and understanding in a single autoregressive framework. The paper identifies four main novelties: native 3D token support for both input and output; early fusion of discrete text and 3D tokens in one shared transformer backbone; introduction of a 3D-vector-quantized VAE to compress voxel grids into a compact discrete token stream; and construction of the 3D-Alpaca dataset with 2.56 M dialogues and 3.46 B tokens.

This work sits adjacent to, but is distinct from, the earlier "ShapeLLM: Universal 3D Object Understanding for Embodied Interaction" (Qi et al., 2024). That earlier system is a 3D multimodal LLM built around point clouds and a ReCon++ encoder for embodied interaction, rather than a native 3D token generator. The distinction matters: ShapeLLM (Qi et al., 2024) is organized around 3D object understanding and language-unified 3D interaction tasks, whereas ShapeLLM-Omni (Ye et al., 2 Jun 2025) treats 3D as part of a unified autoregressive token stream for both understanding and generation.

A second neighboring line is "BodyShapeGPT: SMPL Body Shape Manipulation with LLMs" (Árbol et al., 2024), whose detailed extension sketches an omni-modal system that generalizes from a single human-body shape model to an arbitrary library of PCA-based shape spaces, using natural language, images, sketches, and measurements. That usage of “ShapeLLM-Omni” refers to direct regression of shape coefficients $\beta \in \mathbb{R}^K$ for parametric models such as SMPL-X, FLAME, and MANO, rather than discrete-token generation of generic 3D assets. This suggests that the term has been used for two different unification programs: one centered on native 3D token modeling, and one centered on omni-modal control of registered parametric shape spaces.

2. Native 3D representation through VQVAE tokenization

The core representational component of ShapeLLM-Omni is a 3D VQVAE that maps 3D objects into a discrete latent space for efficient and accurate shape representation and reconstruction (Ye et al., 2 Jun 2025). The input representation is a $64^3$ occupancy voxel grid. A 3D U-Net encoder produces a $16^3$ latent grid, yielding $16^3 = 4096$ latent vectors. Through channel-concatenation, every 4 spatial neighbors are grouped into $1024$ vectors of 32 channels each. The codebook has size $K = 8192$, with learned embeddings $\{e_j \in \mathbb{R}^{32}\}_{j=1\ldots8192}$.

The encoder output is written as $z_e(x)\in\mathbb{R}^{1024\times32}$. Quantization uses nearest-neighbor lookup:

$k(i) = \arg\min_{j} ||\,z_e(x)_i - e_j\,||_2$

and

$z_q(x)_i = e_{k(i)}.$

The decoder reconstructs $64^3$0 voxels from the quantized $64^3$1 grid. The training objective follows the VQ-VAE loss:

$64^3$2

where $64^3$3, $64^3$4 is the stop-gradient operator, and the reported typical defaults are $64^3$5 and $64^3$6.

The resulting discrete space assigns each 3D shape a sequence of $64^3$7 tokens in $64^3$8, rather than the raw $64^3$9 voxel entries. During autoregressive generation, the model predicts the next token index $16^3$0 by standard softmax. At inference, the system decodes tokens to codewords, then to the latent grid, then through the decoder to voxels, and finally to a mesh (Ye et al., 2 Jun 2025). This suggests that the paper’s central architectural move is not merely multimodal conditioning, but a discretization strategy that makes 3D compatible with ordinary next-token prediction.

3. Shared transformer backbone and multimodal autoregression

ShapeLLM-Omni uses Qwen-2.5-VL-Instruct-7B as its backbone, with the image encoder frozen (Ye et al., 2 Jun 2025). The model adds an 8192-entry 3D VQVAE codebook for discrete 3D tokens, and merges text token IDs, image features, and 3D token IDs into one sequence by early fusion. The output is next-token prediction over the combined vocabulary, with the training objective

$16^3$1

The paper’s novelty claims emphasize native 3D token support for both input and output, including text $16^3$2 3D, image $16^3$3 3D, 3D editing, and 3D captioning (Ye et al., 2 Jun 2025). Because the model is trained autoregressively on a shared sequence, 3D tokens are not treated as a side channel or a late-stage decoder target. This differs materially from earlier multimodal 3D-language systems that rely on a separate 3D encoder and inject projected features into an LLM. In ShapeLLM (Qi et al., 2024), for example, ReCon++ produces local and global 3D embeddings, which are projected into the LLaMA embedding dimension and inserted as visual prompt tokens ahead of text. ShapeLLM-Omni instead operates on discrete 3D tokens within the same token-prediction framework.

This architectural choice changes the model’s scope. ShapeLLM (Qi et al., 2024) is optimized for universal 3D object understanding with point clouds and languages, and shows particular gains on embodied interaction. ShapeLLM-Omni (Ye et al., 2 Jun 2025) is organized around the stronger claim that 3D generation, comprehension, and editing can be handled natively inside a single multimodal autoregressive model. A plausible implication is that the latter framework is better aligned with open-ended 3D content generation, while the former remains closely tied to point-cloud reasoning and embodied perception.

4. 3D-Alpaca dataset and training pipeline

The 3D-Alpaca dataset is the principal instruction corpus used to train ShapeLLM-Omni (Ye et al., 2 Jun 2025). It is constructed from approximately 712 k unique 3D assets from Trellis and internal sources, and it covers four task subsets: Text-to-3D, Image-to-3D, 3D-to-Caption, and 3D-Editing. The paper states that instruction templates number 25 per task, and that dialogue assembly uses a random template plus discrete token encoding to produce 2.56 M dialogue samples.

Task	Sample Count	Token Count
Text-to-3D	712 k	0.77 B
Image-to-3D	712 k	1.01 B
3D-to-Caption	712 k	0.77 B
3D-Editing	420 k	0.91 B
3D-Alpaca All	2.56 M	3.46 B

The editing subset is built through a staged pipeline: category selection over 100 top classes; ChatGPT-4o taxonomy and prompt generation yielding 371 feasible prompts; 70 k image-level edits via ChatGPT-4o and an image-editing model; and mesh reconstruction of edits via Trellis (Ye et al., 2 Jun 2025). In the paper’s terminology, the overall dataset encompasses generation, comprehension, and editing tasks, and is presented as a resource for future research and training.

Training proceeds in two layers. First, the VQVAE is pretrained in two stages, each on 48 $16^3$4 H100 with batch 25 and learning rate decayed from $16^3$5 to $16^3$6 (Ye et al., 2 Jun 2025). Second, the Qwen-2.5-VL-Instruct-7B backbone is fine-tuned on 3D-Alpaca plus UltraChat for 15 epochs on 48 $16^3$7 H100, with per-GPU batch size 2, gradient accumulation 2, AdamW, and learning rate decayed from $16^3$8 to $16^3$9. The appendix also reports a 60 k-iteration variant with global batch 192, learning rate $16^3 = 4096$0, 400-step warmup, and cosine decay. These details indicate that the model is trained as a large-scale multimodal instruction follower rather than as a narrowly specialized 3D decoder.

5. Empirical results

On English reasoning benchmarks, ShapeLLM-Omni reports 63.9 on MMLU, 78.6 on PIQA, 55.1 on GSM8K, and 41.0 on SIQA (Ye et al., 2 Jun 2025). Within the comparison table, Qwen2.5-VL scores higher on MMLU and PIQA, while ShapeLLM-Omni scores higher on GSM8K. This indicates that adding 3D capability does not eliminate general language reasoning performance, although the trade-off is benchmark-dependent.

For 3D generation, the paper evaluates both text-to-3D and image-to-3D using CLIP, FD, and KD$16^3 = 4096$1 (Ye et al., 2 Jun 2025). ShapeLLM-Omni reports text-side metrics of 26.7 CLIP, 25.9 FD, and 0.25 KD, and image-side metrics of 84.5 CLIP, 12.2 FD, and 0.09 KD. In the same table, Trellis attains the strongest reported scores, with 30.8 text CLIP, 18.3 text FD, 0.19 text KD, 85.0 image CLIP, 8.31 image FD, and 0.07 image KD. The paper therefore does not present ShapeLLM-Omni as the best standalone generator on those metrics; rather, its significance lies in integrating generation and understanding within one native multimodal LLM.

For 3D-to-caption on Objaverse, ShapeLLM-Omni reports 18.51 on B-1, 21.37 on R-L, 19.89 on METEOR, 48.34 on SBERT, and 49.72 on SimCSE (Ye et al., 2 Jun 2025). In that table, the model outperforms 3D-LLM and PointLLM-13B on B-1, R-L, and METEOR, while PointLLM-13B remains higher on SBERT and SimCSE. The codebook-size ablation shows the best reconstruction quality at vocabulary size 8192, with Chamfer distance 0.0094 and Hausdorff distance 0.0525, compared with 4096 and 16384 (Ye et al., 2 Jun 2025). This supports the chosen discretization level as a practical compromise in the reported setup.

The qualitative examples emphasize image-to-3D meshes preserving geometry and texture, text-to-3D outputs aligned with prompts such as “red sports car w/ spoiler” and “wooden stool with spiral legs,” and identity-preserving 3D edits such as “open door,” “add wings,” and “rotate cannon” (Ye et al., 2 Jun 2025). These examples are consistent with the paper’s claim to cover generation, comprehension, and editing within one model.

6. Relation to adjacent systems and representational alternatives

ShapeLLM-Omni should be distinguished from the earlier ShapeLLM system for 3D object understanding (Qi et al., 2024). ShapeLLM uses ReCon++, a point-cloud encoder that extends ReCon with selective multi-view image distillation and scaled pretraining on Objaverse, ShapeNet, ABO, and 3D-FUTURE. Its instruction-following data are built from Objaverse-LVIS and GAPartNet, yielding a final dataset of approximately 75 K samples. On the 3D MM-Vet benchmark, ShapeLLM-13B reports 53.1 total accuracy versus 46.6 for PointLLM-13B, with the largest gain in Embodied Interaction: 68.4 versus 50.9, a +17.5 increase (Qi et al., 2024). That paper demonstrates strong 3D understanding and embodied planning, but its 3D input remains point-cloud-centric and encoder-mediated.

A separate representational alternative appears in the BodyShapeGPT line (Árbol et al., 2024). There, the omni-modal extension begins with an 8 B-parameter LLaMA-3 transformer, inserts LoRA adapters of rank $16^3 = 4096$2 on every self-attention and feed-forward block, and adds a regression head $16^3 = 4096$3 that predicts shape coefficients $16^3 = 4096$4. To support multimodal inputs, it augments the tokenizer and embedder with an image-embedding module, a sketch encoder, and a “measurement” token embedding table. It covers SMPL-X body with $16^3 = 4096$5, FLAME face with $16^3 = 4096$6, MANO hand with $16^3 = 4096$7, and one synthetic quadruped animal model with $16^3 = 4096$8. The system uses a two-phase output protocol in which the LLM first generates a short textual justification, stopped by $16^3 = 4096$9, and then emits $1024$0 comma-separated placeholder tokens that are detokenized into a float string for $1024$1 (Árbol et al., 2024).

These systems therefore occupy different representational regimes. ShapeLLM-Omni (Ye et al., 2 Jun 2025) operates on discrete 3D tokens produced by a VQVAE over occupancy voxels. ShapeLLM (Qi et al., 2024) operates on point-cloud embeddings projected into an LLM. The BodyShapeGPT extension (Árbol et al., 2024) operates on PCA coefficients of registered parametric models and reconstructs meshes by

$1024$2

A common misconception is to collapse these into a single method family. In fact, the overlap is primarily conceptual: all seek a unified language interface for shape reasoning, but they differ sharply in whether the target space is generic 3D assets, point-cloud semantics, or low-dimensional parametric shape coefficients.

7. Limitations and open directions

The limitations reported for ShapeLLM-Omni are explicit (Ye et al., 2 Jun 2025). The 3D-editing data are limited, with 70 k pairs, and the paper states that editing quality remains below ChatGPT-4o level. The model size is constrained to 7 B parameters, which is described as a performance trade-off. The paper lists several future directions: scaling to larger LLMs greater than 25 B and more editing pairs, incorporating other 3D modalities such as point clouds and neural fields, joint training with image diffusion for end-to-end mesh and textured output, and enhanced interactive 3D dialogue and multimodal planning.

These limitations help clarify what ShapeLLM-Omni does and does not claim. It is not presented as the strongest specialized generator in metric terms, nor as a complete solution for scene-level 3D reasoning or robotic execution. Earlier ShapeLLM work explicitly identifies open problems such as the persistence of a part-level data desert, the focus on single-object 3D shapes, and the need for integration with robotic systems such as VoxPoser and VIMA (Qi et al., 2024). The BodyShapeGPT extension likewise remains tied to registered shape spaces and neutral-pose mesh assembly (Árbol et al., 2024). Taken together, these papers suggest a broader research trajectory in which “shape language” systems are converging on multimodal interfaces, but have not yet converged on a single representation that simultaneously optimizes generative fidelity, semantic grounding, editability, and embodied usability.