ELSA3D: Elastic Semantic Anchoring for Unified 3D Understanding and Generation

Abstract: Unified 3D foundation models aspire to generate 3D assets and reason about them in language within a single backbone, but their text-3D interaction remains largely implicit. Existing methods concatenate text and 3D tokens into a flat sequence and rely on self-attention, collapsing coarse structural cues and fine geometric details into one undifferentiated representation. We introduce ELSA3D, a unified 3D model that addresses this with elastic semantic anchoring, structuring language and geometric reasoning jointly along matched abstraction scales. ELSA3D represents geometry with a scale-aware octree tokenizer and introduces Anchor Tokens, sparse cross-modal units that select semantic cues, route them to the most relevant 3D scale, retrieve scale-specific geometric evidence, and write the fused signal back into the unified representation, keeping interaction sparse yet precise. A lightweight per-block router makes both computation and reasoning elastic, choosing which text tokens instantiate anchors at which geometric scale so that cross-modal capacity concentrates where alignment is most needed. ELSA3D achieves state-of-the-art performance across image-to-3D generation, text-to-3D generation, and 3D captioning, outperforming the strongest unified baseline while roughly halving FLOPs and inference latency relative to the non-elastic version of the same model.

• The paper demonstrates elastic semantic anchoring with scale-aware octree tokenization and dynamic cross-modal routing for unified 3D understanding and generation.
• It achieves significant improvements in quality and efficiency, reducing FLOPs and inference latency by about 50% compared to non-elastic baselines.
• The method enhances cross-modal alignment by fusing semantic and geometric tokens at multiple scales, ensuring precise semantic-geometric binding.

ELSA3D: Elastic Semantic Anchoring for Unified 3D Understanding and Generation

Motivation and Problem Context

Unified 3D foundation models strive for a single architecture that supports both 3D understanding (captioning, reasoning) and generative (text-to-3D, image-to-3D) tasks. Prior art typically concatenates language and geometric (often tokenized) representations into flat sequences, leaving cross-modal alignment to emerge from undifferentiated self-attention. This approach impedes precise semantic-geometric binding across the global-part-local hierarchy fundamental to 3D shapes, and precludes efficient, adaptive allocation of computational resources.

Overview of ELSA3D Architecture

ELSA3D introduces a suite of architectural advances to overcome these deficiencies. Central is Elastic Semantic Anchoring, comprising three main components:

1. Scale-aware Octree VQ-VAE Tokenization: Geometry is discretized into a multiscale octree, yielding explicit token streams at each geometric level. This produces a Morton/Z-ordered (Z-order curve) serialization, preserving spatial locality and level-of-detail context.
2. Semantic Trace Decomposition: Language input is structured into parallel token streams representing global, structural, and appearance cues, mirroring object-level, part-level, and local detail.
3. Anchor Tokens with Elastic Cross-modal Routing: Selected semantic tokens dynamically instantiate anchor tokens, which are routed (via a learnable router) to the most relevant geometric scale for cross-attention. The resulting scale-specific semantic-geometric fusion is then written back into the unified representation, with both computational and cross-modal bandwidth adaptively controlled at each transformer block.

   Figure 1: ELSA3D overview depicting router structure, block-level gating/width, and anchor routing for semantic-geometric grounding.

Scale-aware 3D Tokenization

The VQ-VAE-based tokenizer discretizes a $128^3$ voxel grid with octree subdivision, using scale-specific codebooks for content quantization. Each token is annotated with both position (Morton code) and level-of-detail tags. Structural bits encode subdivision topology, permitting reversible hierarchical decoding.

Figure 2: Illustration of ELSA3D octree tokenization, discretizing a 3D shape into multiscale, Morton-ordered bits preserving both structure and content.

Explicit scale tags enable the subsequent router module to perform scale-aware grounding and reasoning, a property unattainable with flat, unstructured tokenization.

Elastic Semantic Anchoring and Routing

At each transformer layer, a lightweight router predicts, for each block:

• Whether to execute or skip the block (block gating).
• MLP width (adaptive capacity).
• For each semantic token, whether and where to instantiate an anchor (anchor routing).

Selected anchor tokens use cross-attention to query scale-specific geometric tokens. The fused anchor representation is written back to the persistent unified token sequence via cross-attention, with a learned injection gate controlling influence magnitude.

Routing is fully differentiable (with straight-through estimators for discrete decisions), and auxiliary losses regulate block/width utilization, anchor sparsity, and scale diversity.

Experimental Results

ELSA3D achieves strong numerical improvements over state-of-the-art unified and specialized 3D baselines:

• Image-to-3D Generation: +2.74 CLIP, -2.03 FD, -0.54 MMD over strongest unified baseline.
• Text-to-3D Generation: +1.35 CLIP, -4.05 FD, +0.14 Q-Align.
• 3D Captioning: +3.56 Sentence-BERT.

Crucially, these quality improvements coincide with a significant efficiency gain—overall FLOPs and inference latency are reduced by approximately 50% relative to the non-elastic counterpart.

Qualitative evaluations confirm ELSA3D’s ability to bind fine-grained linguistic desiderata to the appropriate geometric level, producing 3D assets more consistent with both global intent and appearance constraints.

Figure 3: Comparison of ELSA3D outputs against prior methods for image-to-3D and text-to-3D tasks, demonstrating superior fidelity to shape and texture details.

Figure 4: Image-to-3D comparisons highlighting ELSA3D's ability to preserve thin structures, part layouts, and appearance.

Figure 5: Text-to-3D comparison showing how ELSA3D satisfies both broad category and fine prompt constraints.

Ablation and Analysis

• Anchor Token Mechanisms: Removing anchors or using dense cross-attention both degrade quality and dramatically increase computational cost, validating the utility of sparse, routed anchors.
• Scale-aware Routing: Forcing every anchor to attend to a single (coarse or fine) scale or using random scale assignment reduces performance, confirming the value of learned, scale-aware assignment.
• Elastic Computation: Dynamic block skipping and adaptive MLP width retain near-maximum quality with halved computational resources.

Implications and Future Directions

ELSA3D represents a step toward structured, sparse, and interpretable cross-modal reasoning in unified 3D models. The framework’s elastic routing and semantic-geometric scale coupling enable explicit, efficient grounding—a critical advancement for scaling to higher-resolution assets and more complex scene-level tasks.

Potential extensions include:

• Incorporation of dynamic scene and multi-object interactions.
• Progressive refinement for ultra-high-fidelity 3D asset generation.
• Integration within embodied agent systems for language-driven 3D manipulation and planning.

These directions point toward models that not only generate rich 3D content but also natively support grounding, compositional reasoning, and efficient inference, underlying future multimodal AI systems.

Conclusion

ELSA3D advances the field of unified 3D understanding and generation by embedding scale-aware, elastic, and dynamically routed semantic-geometric anchoring into a single transformer backbone. The approach yields superior cross-modal alignment, greater interpretability, and marked efficiency gains, establishing a new state of the art in both generative and reasoning-based 3D tasks. These results motivate further exploration of structured routing and adaptive computation for multimodal foundation models.