3D-Alpaca: Multimodal 3D Dialogue Dataset

• 3D-Alpaca is a continuous training dataset that interleaves text, continuous image features, and discrete 3D tokens for unified autoregressive multimodal LLM training.
• It supports tasks such as text-to-3D, image-to-3D, 3D captioning, and editing by encoding 3D assets into fixed-length 1024-token sequences.
• The dataset fills a crucial gap for dialogue-based 3D LLM fine-tuning, demonstrating scalable reconstruction using a tailored 3D vector-quantized representation with an 8192-entry codebook.

3D-Alpaca is a large-scale continuous training dataset introduced in "ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understanding" (Ye et al., 2 Jun 2025). It was constructed to address the scarcity of dialogue data in the 3D LLM domain and to support supervised fine-tuning of a native 3D LLM that can understand, generate, and edit 3D content in a unified autoregressive framework. In this setting, "native 3D" denotes direct consumption and emission of 3D content represented as discrete 3D tokens rather than text surrogates such as OBJ files serialized as text. The dataset interleaves text, images represented as continuous features, and 3D discrete tokens within single sequences so that a multimodal LLM can perform next-token prediction across modalities (Ye et al., 2 Jun 2025).

1. Purpose and conceptual framing

3D-Alpaca was created to fill a notable gap in supervised fine-tuning data for 3D-native LLMs. The underlying motivation is that instruction datasets for text and image are numerous, whereas dialogue data within the 3D LLM domain remains relatively scarce. This scarcity limits training of models that can understand and generate 3D content in a unified way. The dataset therefore provides a corpus spanning generation, comprehension, and editing, with the explicit goal of enabling consistent next-token prediction across 3D and text, and optionally image, sequences (Ye et al., 2 Jun 2025).

A defining characteristic of the dataset is its "continuous training dataset" design. The authors emphasize a continuous, interleaved token training regime in which text, images, and 3D tokens can appear in any order within a single sequence. Images are not discretized into language-style tokens; instead, they are represented by continuous visual features from the frozen Qwen-2.5-VL encoder. By contrast, 3D content is discretized and inserted into the LLM vocabulary as a new modality vocabulary. This permits unified early-fusion modeling and joint instruction-following without switching task-specific heads or pipelines (Ye et al., 2 Jun 2025).

The dataset is therefore not simply a repository of 3D assets. It is a dialogue-oriented multimodal training resource organized around instruction templates and tokenized 3D representations. A common misconception is to treat it as equivalent to prior 3D asset corpora such as ShapeNet or Objaverse-style collections. Those corpora provide 3D assets and annotations, but 3D-Alpaca is designed specifically for autoregressive multimodal instruction tuning with fixed-length discrete 3D tokens (Chang et al., 2015).

2. Data composition, modalities, and scale

3D-Alpaca is organized around 3D meshes that are voxelized to a $64^3$ grid and then compressed into discrete tokens for LLM training. Raw assets come as 3D meshes; the LLM consumes and produces 3D through $64^3$ voxels, which are reconstructed back to meshes using Trellis's rectified-flow voxel-to-mesh decoders (Xiang et al., 2024). Each 3D asset is also rendered into images for captioning, conditioning, and editing workflows. For understanding and caption generation, four orthogonal views are rendered: front, back, left, and right. For image-to-3D and editing, the asset is rendered from a frontal view with a random offset for inputs (Ye et al., 2 Jun 2025).

The base 3D pool consists of approximately 712k high-quality 3D assets selected from Trellis and an internal collection. Editing is organized over 100 major categories based on Objaverse-XL distribution, with 371 curated editing prompts. The editing pipeline samples 200 assets per prompt; after filtering erroneous cases, it yields 70k valid edited image pairs and corresponding reconstructed 3D pairs. Dialogue templates are then applied to expand these data into instruction instances (Ye et al., 2 Jun 2025).

The paper reports the following subset sizes and token counts for 3D-Alpaca:

Subset	Items	Tokens
Text-to-3D	712k	0.77B
Image-to-3D	712k	1.01B
3D-to-Caption	712k	0.77B
3D-Edit	420k	0.91B
Total 3D-Alpaca	2.56M	3.46B

Each 3D object is represented by a fixed-length 1024-token sequence. Text is represented using standard subword tokens. Images are encoded into continuous visual features via the frozen Qwen-2.5-VL encoder (Bai et al., 19 Feb 2025). Train/validation/test splits are not specified. Versioning, total storage size, and dataset structure on disk are also not specified (Ye et al., 2 Jun 2025).

3. Native 3D tokenization and representation

The 3D tokenization pipeline is based on a 3D vector-quantized variational autoencoder. The representation begins with a $64^3$ voxel grid, chosen to capture shape skeletons efficiently. These voxels are compressed to a $16^3$ latent grid via a 3D U-Net VAE from Trellis (Xiang et al., 2024). Since $16^3 = 4096$ latent tokens with 8 channels, the pipeline concatenates every four neighboring tokens along the channel dimension to obtain 1024 tokens with 32 channels. Final quantization uses a codebook of size $K = 8192$, producing a 1024-token discrete sequence for each 3D object (Ye et al., 2 Jun 2025).

This gives a compression from $64^3 = 262{,}144$ voxel elements to 1,024 discrete tokens, described as an effective reduction of $256\times$ relative to the original voxel grid length. The 8192-entry 3D codebook is added as a new modality vocabulary, allowing Qwen-2.5-VL-7B-Instruct to treat 3D tokens as next-token prediction targets interleaved with text tokens, while images remain continuous features (Ye et al., 2 Jun 2025).

The paper reports an ablation of codebook size using mesh reconstruction from decoded voxels on 1000 meshes. Reconstruction quality is evaluated using Chamfer Distance and Hausdorff Distance:

• $K = 4096$: $\mathrm{CD} = 0.0102$; $64^3$0
• $64^3$1: $64^3$2; $64^3$3
• $64^3$4: $64^3$5; $64^3$6

The authors conclude that $64^3$7 balances quality and efficiency (Ye et al., 2 Jun 2025).

For reference, the reported evaluation metrics are

$64^3$8

and

$64^3$9

The exact VQVAE loss terms and hyperparameters are not explicitly listed in the paper, although a standard VQVAE objective is reproduced there for reference. Whether a straight-through estimator, specific commitment weight $64^3$0, or additional loss terms are used is not specified (Ye et al., 2 Jun 2025).

4. Construction and annotation workflow

Caption generation is performed by rendering four orthogonal views of each asset and feeding them to Qwen-2.5-VL-Instruct to produce descriptive captions. These captions serve both as prompts for text-to-3D and as ground truth for 3D-to-text tasks (Bai et al., 19 Feb 2025). The appendix states that images are resized to $64^3$1 with a white background to match Qwen-VL's image tokenization behavior (Ye et al., 2 Jun 2025).

The editing annotation pipeline consists of six steps. First, 100 frequent categories are selected based on Objaverse-XL distribution. Second, ChatGPT-4o is used to classify assets into fine-grained subcategories via frontal renders, and 311k assets belonging to the 100 categories are retained. Third, approximately 20 candidate editing prompts per category are generated with ChatGPT-4o, then manually reviewed, resulting in 371 unique prompts. Fourth, 200 assets per prompt are sampled to create a compact, high-quality edited dataset. Fifth, for each asset, the frontal render and chosen editing prompt are given to ChatGPT-4o to produce an edited image; erroneous cases are filtered, leaving 70k valid edited samples. Sixth, the edited images are reconstructed to 3D via Trellis, yielding before/after 3D pairs (OpenAI et al., 2024).

Dialogue templates are central to the dataset's organization. The paper specifies 25 templates per task. For 3D-edit, each edit pair is expanded into 6 dialogues by random template sampling; for other task instances, one template is assigned. The total is reported as approximately 2.5M 3D dialogues (Ye et al., 2 Jun 2025).

Quality control is described in limited but concrete terms. Manual prompt filtering is applied, and erroneous edited image cases are removed. Other potential preprocessing and normalization procedures, including deduplication, coordinate-frame normalization, watertightness checks, UV handling, and texturing policies, are not detailed. Mesh cleanup, storage formats, and exact file types are likewise unspecified (Ye et al., 2 Jun 2025). This suggests that the dataset description is primarily oriented toward modeling workflow rather than archival curation details.

5. Enabled tasks and training usage

The dataset supports four principal task families. For text-to-3D generation, prompts are derived from model-generated captions or user-provided text; the paper gives the template, "Please generate a 3D mesh based on the prompt I provided: <prompt>". For image-to-3D generation, the template example is, "Create a 3D asset using the following image: <image>". For 3D-to-caption comprehension, the example is, "<mesh>. Caption this 3D model in detail." For 3D editing, curated editing instructions are applied to before/after 3D pairs formed through the editing pipeline (Ye et al., 2 Jun 2025).

The instruction-tuned model uses Qwen-2.5-VL-Instruct-7B as backbone, with the visual encoder frozen to preserve image understanding (Bai et al., 19 Feb 2025). The main text reports a learning rate decaying from $64^3$2 to $64^3$3, per-GPU batch size 2, gradient accumulation 2, and training for 15 epochs on 48 $64^3$4 H100 GPUs. The appendix additionally reports 60k iterations, full-parameter fine-tuning, AdamW with learning rate $64^3$5, warmup 400 steps with cosine scheduling, a global batch size of 192, and approximately 5 days of training (Ye et al., 2 Jun 2025).

Sampling settings for generation are specified as top-$64^3$6 equal to the 3D vocabulary size, namely 8192, with top-$64^3$7 and temperature $64^3$8. For practical use, the procedure is to generate 1024 3D tokens, decode them to $64^3$9 voxels, and reconstruct meshes via the Trellis rectified-flow decoder; texture can optionally be added through the texture transformer described in the appendix (Xiang et al., 2024).

The paper also notes that UltraChat is included during training, with 1.47M items and 2.16B tokens, to preserve general conversational capability. However, explicit mixing ratios or curriculum details are not described (Ye et al., 2 Jun 2025).

6. Relation to prior datasets, evaluation, and limitations

The paper contrasts 3D-Alpaca with ShapeNet, Objaverse/Objaverse-XL, ABO, OmniObject3D, OpenShape, and CAP3D. The stated distinction is that those resources primarily provide object collections, category labels, or caption annotations, but not large-scale, 3D-native instruction dialogues that interleave text, image, and 3D in unified sequences. Nor do they provide a fixed-length discrete 3D token representation integrated into an LLM vocabulary for autoregressive generation and understanding (Chang et al., 2015). On this basis, 3D-Alpaca is characterized by four reported properties: scale, multimodal interleaving, instruction richness, and 3D-native tokenization (Ye et al., 2 Jun 2025).

For text- and image-conditioned generation quality on Toys4K test items, the paper reports use of CLIP score, Frechet Distance, and Kernel Distance computed from Inception-V3 features, with Kernel Distance scaled by $16^3$0 as reported. Other metrics such as EMD, IoU, F-score, and PSNR are not reported in the paper (Ye et al., 2 Jun 2025). For 3D understanding and captioning, the practical usage notes mention BLEU-1, ROUGE-L, METEOR, Sentence-BERT, and SimCSE as evaluation measures (Ye et al., 2 Jun 2025).

Several limitations and caveats are explicitly stated. Only 70k edited pairs are available prior to dialogue expansion, which the authors describe as "far too few to achieve ChatGPT-4o-level results in 3D editing" (Ye et al., 2 Jun 2025). The instruction-tuned model is 7B, and limited compute constrained performance. Biases may arise because captions, category classification, and editing prompts depend on Qwen-2.5-VL-Instruct and ChatGPT-4o outputs (Bai et al., 19 Feb 2025); rendering choices such as white background and $16^3$1 resolution may also bias learning toward clean, studio-like inputs (Ye et al., 2 Jun 2025).

Availability and licensing remain underspecified. The project page is hosted at the ShapeLLM-Omni repository, but the paper does not provide a direct download link for 3D-Alpaca or for its raw assets and tokens (Ye et al., 2 Jun 2025). Explicit license terms for 3D-Alpaca are not stated, nor are the licenses of the reused assets. The paper recommends research use for building and evaluating 3D-native multimodal LLMs and advises careful verification of original source licenses and model terms before redistribution or downstream use (Ye et al., 2 Jun 2025). A plausible implication is that legal and reproducibility considerations remain partly dependent on future repository releases or direct communication with the authors.