Are We Ready for RL in Text-to-3D Generation? A Progressive Investigation

Abstract: Reinforcement learning (RL), earlier proven to be effective in large language and multi-modal models, has been successfully extended to enhance 2D image generation recently. However, applying RL to 3D generation remains largely unexplored due to the higher spatial complexity of 3D objects, which require globally consistent geometry and fine-grained local textures. This makes 3D generation significantly sensitive to reward designs and RL algorithms. To address these challenges, we conduct the first systematic study of RL for text-to-3D autoregressive generation across several dimensions. (1) Reward designs: We evaluate reward dimensions and model choices, showing that alignment with human preference is crucial, and that general multi-modal models provide robust signal for 3D attributes. (2) RL algorithms: We study GRPO variants, highlighting the effectiveness of token-level optimization, and further investigate the scaling of training data and iterations. (3) Text-to-3D Benchmarks: Since existing benchmarks fail to measure implicit reasoning abilities in 3D generation models, we introduce MME-3DR. (4) Advanced RL paradigms: Motivated by the natural hierarchy of 3D generation, we propose Hi-GRPO, which optimizes the global-to-local hierarchical 3D generation through dedicated reward ensembles. Based on these insights, we develop AR3D-R1, the first RL-enhanced text-to-3D model, expert from coarse shape to texture refinement. We hope this study provides insights into RL-driven reasoning for 3D generation. Code is released at https://github.com/Ivan-Tang-3D/3DGen-R1.

• The paper introduces AR3D-R1, a two-step generative pipeline that employs hierarchical reward ensembles to optimize both global geometry and fine-grained part details.
• It demonstrates that step-specific reward shaping boosts semantic alignment and geometric consistency, achieving up to a 2.1-point CLIP score improvement.
• The study utilizes the Hi-GRPO strategy for multi-stage reinforcement learning, outperforming direct mesh token optimization in complex text-to-3D generation tasks.

RL Paradigms for Text-to-3D Generation: A Progressive Evaluation

Introduction

Text-to-3D generation has witnessed rapid advances in data curation, parametric representations, and learning systems, yet the integration of reinforcement learning (RL) into 3D autoregressive generative models remains nascent. This work critically investigates whether current RL infrastructures and reward designs are suited to text-to-3D, introducing the AR3D-R1 framework with a hierarchical group-relative policy optimization (Hi-GRPO) paradigm. The study systematically benchmarks reward schemas, multi-level granularity, and group-based optimization—demonstrating their effects on geometry, composition, and semantic alignment in 3D autoregressive models. The analysis emphasizes the failures and successes of direct reward transfer from 2D and LLM RL settings, while highlighting task-specific complexities unique to 3D content creation.

Background and Context

Recent 2D and LLM paradigms have established RL as a principle for driving advanced reasoning, planning, and content fidelity. In LLMs, rule-based rewards, chain-of-thought (CoT) mechanisms, and group-aware RL (e.g., GRPO, DPO) are critical for robust multi-step reasoning. Successful adoptions in text-to-image have advanced semantic planning and generation quality via iterative token-level feedback and reward feedback from powerful vision-LLMs.

3D generative modeling, in contrast, must resolve far higher structural and appearance complexities: spatial consistency, multiview coherence, and fine-grained part localization are all aggravated by the ill-posed mapping from textual description to geometric/compositional details. Two-stage and diffusion-based 3D systems have advanced quality, but suffer from compounding errors and immense computation. Autoregressive 3D transformers, such as MeshGPT and ShapeLLM-Omni, represent a promising direction, making discrete token-based policies tractable for RL optimization.

Yet, attempts to directly transpose 2D RL protocols to 3D show limited gains and poorly address the multimodal, hierarchical requirements of 3D reward shaping. This gap motivates the study's systematic investigation of RL strategy and reward engineering for 3D content.

Methodology: AR3D-R1 and Hi-GRPO

The core contribution is the AR3D-R1 pipeline, architected around ShapeLLM-Omni as the base generator and using a two-step semantic-to-visual reasoning process for hierarchical planning and mesh token prediction.

Two-Step Generation Decomposition

• Step 1: Generates high-level semantic reasoning tokens for global geometric planning, then coarse mesh tokens—decoded to a preliminary mesh via a VQVAE.
• Step 2: Conditioned on Step 1, generates visual reasoning tokens for local appearance and component-level structure, finalizing with mesh tokens that are further decoded.

Hierarchical Reward Ensemble

The reward ensemble is the key innovation, explicitly aligned with the two-step process and integrating multiple perspectives:

• Human Preference Model (HPS V2.1): 6-view text-image similarity measuring overall plausibility.
• Unified Reward Model: Multi-scoring (prompt alignment, logic, style) per view, normalized over assessment dimensions.
• 2D Multi-Modal Consistency (Qwen2.5-VL): Categorical and appearance consistency, decomposed into color, material, texture.
• 3D Part-wise Assessment (ShapeLLM): Mesh-to-point-cloud conversion with per-component existence and completeness from prompt-parsed object lists.

All individual reward components are normalized by dimension and aggregated, with the Step 2 reward also supervising Step 1 through a weighting parameter $\lambda$. This ensemble is calculated within prompt groups to stabilize variance across batch samples.

Policy Optimization

The policy gradient is optimized by Hi-GRPO, which extends group-relative updates with two-step decoupled clipping (asymmetric thresholding for promoting exploration while maintaining stability), token-level averaging, and independent KL divergence regularization per generation stage.

Empirical Results and Visualization

Reward Engineering Ablation

Quantitative comparison using the Toys4k dataset reveals that naive transfer of step-2 (final object) rewards to optimally align both geometric and appearance properties is ineffective; step-specific rewards—especially those that target the coarser/planning stage—are crucial for robust mesh structure and semantic correctness. Integrating component-level rewards further enhances part positioning and structural accuracy, resulting in a 2.1-point CLIP score lift when step-specific alignment is enforced.

RL Paradigm Efficacy

GRPO with textual reasoning guidance outperforms direct mesh token optimization by 0.9 CLIP points, confirming the necessity of multistage reasoning even in RL settings. Hi-GRPO’s hierarchical integration leads to the highest geometric and text prompt consistency—with significant degradation in both qualitative and quantitative metrics when either stage's reward system is removed.

Figure 1: Visualization Results of Small Objects for Different RL Paradigms.

Figure 2: Visualization Results of Large Objects for Different RL Paradigms.

Visual results demonstrate that hierarchical RL with step-specific reward shaping produces 3D shapes with superior global form and significantly improved fine-grained appearance, especially for objects with intricate part segmentation or complex structural dependencies.

Theoretical and Practical Implications

The findings establish that 3D content generation requires RL infrastructures that:

• Decompose multiscale generation into independent, hierarchically supervised subtasks.
• Integrate reward signals specific to geometry, appearance, and part completeness—leveraging both 2D and point-cloud/3D evaluation models.
• Normalize and ensemble multi-dimensional evaluations to prevent any single signal from dominating or destabilizing optimization.

Practically, these insights adjust the design of training protocols for future 3D generative models, emphasizing the inadequacy of 2D RL reward schemas and the need for 3D-aware, modularized supervision. They provide a foundation for scalable, multitask 3D LLMs supporting not just generation, but also understanding and editing tasks.

Limitations and Future Directions

Current limitations include reliance on external reward models (e.g., LMMs and preference models trained on 2D data), computational overhead for dense multi-view/point-cloud assessments, and potential difficulty scaling to scenes rather than single objects.

Future research should address:

• End-to-end learned 3D reward models leveraging geometric and physical constraints.
• More efficient inference for large-scale autoregressive 3D policies.
• Studying reward hacking and robustness under open-domain, long-tail prompts.
• Transferring insights to other structured outputs beyond 3D meshes, such as interactive environments or embodied AI tasks.

Conclusion

This work substantiates the critical role of hierarchical, multi-dimensional reward engineering and group-based policy optimization in RL for autoregressive text-to-3D generation. The AR3D-R1/Hi-GRPO approach delineates the design space for future 3D creators, demonstrating that deep integration of semantic planning, geometric reasoning, and part-level evaluation is essential to progress beyond the current quality plateau of 2D-inspired RL schemas.