Sel3DCraft: Interactive Text-to-3D System

• Sel3DCraft is an interactive system that advances text-to-3D generation by integrating dual-branch retrieval and generative pathways.
• It employs a multi-view hybrid scoring methodology combining low-level metrics and high-level MLLM-based semantic evaluations for precise 3D output consistency.
• The system features an interactive visual analytics suite that provides actionable feedback, reducing prompt iterations by over 66% and creation time by 70%.

Sel3DCraft is an interactive visual prompt engineering system designed to advance text-to-3D (T23D) generation by providing a structured, guided workflow for exploring and refining 3D content creation. The system addresses core limitations of prior T23D systems—most notably, the unpredictability and inefficiency of blind prompt iteration—by integrating retrieval and generation pathways, employing a multi-view hybrid scoring methodology, and delivering actionable, visual analytics for prompt refinement. Sel3DCraft has been demonstrated to significantly improve both the efficiency and quality of T23D content workflows by surpassing traditional black-box solutions in supporting designer creativity and control (Xiang et al., 1 Aug 2025).

1. Dual-Branch Architecture: Retrieval and Generation

Sel3DCraft introduces a dual-branch pipeline that simultaneously explores candidate 3D content through complementary retrieval and generative mechanisms:

• The retrieval branch sources 3D shapes, associated images, and text prompt variants from a large-scale 3D repository via a model such as OpenShape, which unifies the embedding of text, images, and 3D geometry in a joint representation space.
• The generative branch uses LLMs to expand or augment user input, generating a suite of variant prompts. These variants are rendered as 2D images using a text-to-image (T2I) engine, which are then used with a 3D synthesis model (e.g., TripoSR) to obtain multi-view 3D renderings.

This approach greatly expands the candidate solution space. Users interact with a diverse array of outputs, enabling direct comparison, selection, and iterative refinement rather than being limited to sparse, sequential generations. The two branches are integrated in a joint embedding space via the fusion function:

$$F_\mathrm{fs}(T, I, S) = \mathcal{Z}(T_\mathrm{PC}, T^*_\mathrm{T2I}, S^*_\mathrm{3D})$$

where $T$ is the original text prompt, $\mathcal{Z}$ is the joint embedding operator, $T_\mathrm{PC}$ represents the retrieved prompt candidates, $T^*_\mathrm{T2I}$ the augmented text for T2I synthesis, and $S^*_\mathrm{3D}$ the optimized 3D model.

2. Multi-View Hybrid Scoring and Semantic Evaluation

A core technical challenge in T23D is assessing multi-view and semantic consistency in 3D outputs. Sel3DCraft’s multi-view hybrid scoring combines low-level computational metrics with high-level, MLLM-based perceptual assessments:

• Low-Level Metrics:
  • Color Consistency: Computed by comparing Lab color histograms across multiple rendered views using the Bhattacharyya coefficient:

  $$f_\mathrm{color} = \frac{1}{N} \sum_{i=1}^{N} \mathrm{BC}(H_\mathrm{Lab}^{(i)}, \bar{H}_\mathrm{Lab})$$

  where $H_\mathrm{Lab}^{(i)}$ is the Lab histogram for the $i$-th view, $\bar{H}_\mathrm{Lab}$ the average histogram, and $N$ the number of views. - Lighting Consistency: An analogous procedure is applied using lighting statistics. - CLIP Scores: Both base CLIP and CLIP-I scores are calculated. The former measures cosine similarity between text and image embeddings (text–visual alignment), while the latter measures inter-view consistency, i.e., semantic similarity across different views.

• High-Level Metrics:

  • A suite of multimodal LLMs (MLLMs) are employed as semantic judges, evaluating candidates on properties such as text-image alignment, 3D plausibility, texture-geometry consistency, and the quality of fine details.
  • The evaluation leverages step-by-step, structured prompting to induce expert-human–aligned reasoning processes in the MLLM.
  • Outputs are aggregated into an 8-dimensional semantic score vector for each candidate, providing transparent, multidimensional feedback to the user.

3. Interactive Visual Analytics Suite

Sel3DCraft introduces a prompt-driven, multi-modal visual analytics environment to facilitate human-in-the-loop refinement:

• Image Browser:

Presents a satellite chart: a central frontal view of the model surrounded by rotated perspectives (e.g., 45° increments). Clustering of views indicates 3D plausibility; a tight cluster corresponds to high quality.

• Hybrid Scoring Heatmaps:

Renders heatmaps derived from hybrid semantic scores, visually flagging regions of defect—such as texture inconsistency, lighting anomalies, or structural implausibility—in red.

• Multi-Line Semantic Trend Chart:

Offers temporal feedback on the eight semantic evaluation dimensions, allowing deficiency trends to be tracked across prompt revisions.

• Text Exploration View:

Features a “treemap wordle” (keywords encoded by frequency and quality) and an interactive keyword contribution map inspired by Sankey diagrams. This design visualizes the quantitative effect of altering specific lexical prompt elements on the generative quality, enabling data-driven prompt optimization.

The suite is tightly linked to the underlying scoring system, ensuring prompt modifications are reflected instantaneously in candidate evaluation and visualization.

4. Metrics and Mathematical Formulations

Sel3DCraft introduces and operationalizes several key mathematical scoring constructs:

• Fusion Function:

$$F_\mathrm{fs}(T, I, S) = \mathcal{Z}(T_\mathrm{PC}, T^*_\mathrm{T2I}, S^*_\mathrm{3D})$$

defines the multimodal embedding of prompt, image, and 3D candidate.

• 3D-Friendliness Score:

$$f_{3Df}(I) = w_O \cdot (1 - \frac{O}{O_\mathrm{max}}) + w_\mathrm{IoU} \cdot \mathrm{IoU} + w_\mathrm{BB} \cdot \mathrm{IoU_{BB}}$$

with $w_O, w_\mathrm{IoU}, w_\mathrm{BB}$ tuned empirically to correlate with expert judgment; $O$ is the center offset, $\mathrm{IoU}$ the standard segmentation overlap, and $\mathrm{IoU_{BB}}$ the bounding box intersection over union.

• Color Consistency:

$$f_\mathrm{color} = \frac{1}{N} \sum_{i=1}^{N} \mathrm{BC}(H_\mathrm{Lab}^{(i)}, \bar{H}_\mathrm{Lab})$$

where $\mathrm{BC}$ refers to the Bhattacharyya coefficient.

No additional statistics, pseudocode, or alternative metrics are introduced beyond those declared in the source data.

5. Workflow Enhancements, Empirical Evaluation, and Impact

Sel3DCraft’s design has demonstrated substantial improvements in T23D workflows:

• Model creation time is reduced by over 70%.
• The number of prompt iterations to achieve satisfactory results is decreased by more than 66%.
• Quality ratings, as assessed by domain experts, are significantly higher in controlled comparisons with conventional T23D systems.

The primary impact lies in transforming T23D from a “black-box” trial-and-error process to a guided, feedback-driven, human-computer collaborative exploration, yielding both time efficiency and generative quality. The system’s transparency and prompt steering facilitate creative workflows, supporting professional design and content authoring at scale.

6. Relationship to S3D3C and Broader Applicability

Sel3DCraft’s dual-branch strategy leverages large-scale 3D repositories such as the Sketchfab 3D Creative Commons Collection (S3D3C) (Spiess et al., 24 Jul 2024) for its retrieval pathway. The technical heterogeneity and metadata-rich nature of S3D3C (including textures, animation, materials, and category annotations) make it especially well suited for Sel3DCraft’s multi-modal candidate sourcing, high-fidelity evaluation, and diverse application domains.

Broader applications for Sel3DCraft are evident in contexts where prompt-driven, controllable, and multi-modal 3D content creation is required—such as digital content production, virtual and augmented reality, gaming, and cultural heritage digitization. The system is positioned as a testbed and operational pipeline for state-of-the-art prompt engineering methodologies targeting the next generation of user-empowered, T23D-driven creative industries.