Carve3D: RL Finetuning for 3D Consistency

• Carve3D is a reinforcement-learning finetuning method that enhances multi-view consistency in text-to-3D diffusion models.
• It employs a reconstruction-based reward, calculating LPIPS between generated views and NeRF-rendered images to assess consistency.
• The approach incorporates KL regularization to balance fidelity to the pretrained SFT model while improving reconstruction quality.

Carve3D is a reinforcement-learning finetuning (RLFT) algorithm for improving the multi-view consistency of text-to-3D precursor diffusion models, introduced in “Carve3D: Improving Multi-view Reconstruction Consistency for Diffusion Models with RL Finetuning” (Xie et al., 2023). Rather than carving geometry directly, it optimizes a multi-view diffusion model against a reconstruction-based reward, Multi-view Reconstruction Consistency (MRC), that measures whether a sparse-view NeRF reconstructed from generated views can render back images matching those views at the same camera poses. In broader technical usage, “Carve3D” can also function as a descriptive label for carve-based 3D workflows in reconstruction, editing, visualization, and shape manipulation, but the formal proper noun denotes the RLFT method and its resulting model, Carve3DM (Xie et al., 2023).

1. Problem setting and conceptual placement

Carve3D addresses a bottleneck in text-to-3D systems based on multi-view diffusion. In that regime, a text prompt is first converted into a small set of images of the same object from different viewpoints, and those images are then lifted into a 3D representation—here, a NeRF—using a feed-forward sparse-view reconstructor. The paper positions this paradigm as attractive because 2D text-to-image diffusion models are trained on internet-scale data and therefore carry strong semantic priors, whereas direct 3D generative models are limited by the size and quality of datasets such as Objaverse and Objaverse-XL. However, supervised finetuning (SFT) on multi-view renderings does not fully solve cross-view inconsistency and can induce a tradeoff: longer SFT improves consistency but increasingly biases the model toward the narrower 3D finetuning distribution, degrading realism, diversity, detail, and sometimes prompt fidelity (Xie et al., 2023).

The method is explicitly framed as an alignment procedure analogous to the pretraining $\rightarrow$ SFT $\rightarrow$ RLHF/RLFT pipeline in LLMs. In this view, SFT teaches the model the multi-view generation task, but RLFT is needed to optimize the desired behavior—3D consistency—without continuing to imitate a limited supervised dataset. Carve3D therefore uses self-generated samples: the model produces candidate multi-view outputs from prompts, those outputs are scored by a reconstruction-based reward, and the model is updated to increase that reward while remaining close to the base SFT model (Xie et al., 2023).

A common misconception is to interpret the name as denoting volumetric space carving or mesh subtraction. In the formal sense of the 2023 paper, Carve3D is neither a visual-hull algorithm nor a geometric editing operator. Its “carving” is objective-level: it sculpts the behavior of a multi-view diffusion prior so that generated views can be jointly explained by a downstream 3D reconstruction model.

2. Multi-view Reconstruction Consistency

The central technical contribution is the MRC metric. Given a set of generated views and their camera poses, Carve3D reconstructs a NeRF using a sparse-view Large Reconstruction Model (LRM), renders that NeRF back at the same viewpoints, crops both the original and rendered views to a foreground square bounding box, resizes those crops, and computes LPIPS between matched pairs. The average LPIPS over views is the MRC score, with lower values indicating better multi-view consistency. In mathematical form, the paper gives the faithful reconstruction

$$\text{MRC}(x_0) = \frac{1}{V}\sum_{v=1}^{V} \operatorname{LPIPS}\!\left( \operatorname{Resize}\!\left(\operatorname{Crop}(I_v, B_v)\right), \operatorname{Resize}\!\left(\operatorname{Crop}(\hat I_v, B_v)\right) \right),$$

where $I_v$ is the generated view, $\hat I_v$ is the render of the reconstructed NeRF at the same pose, and $B_v$ is the square foreground bounding box (Xie et al., 2023).

The metric operationalizes consistency through the reconstruction-rendering loop. If the views are mutually compatible in geometry, identity, and appearance, a sparse-view NeRF can explain them jointly and re-render them closely. If they are inconsistent, reconstruction must compromise, producing floaters, broken geometry, blurred details, or incompatible parts, and the LPIPS discrepancy rises. The paper validates this behavior by controlled perturbation experiments: starting from four consistent renders, it corrupts one view by inpainting, azimuth rotation, or elevation rotation and shows that MRC increases monotonically with inconsistency strength.

Bounding-box normalization is essential. The paper reports that whole-image LPIPS can be reward-hacked by shrinking the foreground object against a mostly white background. Cropping to the smallest square foreground box and resizing before LPIPS computation prevents this failure mode; removing that normalization causes RL-trained models to reduce object size drastically and eventually collapse toward blank-background outputs. The paper also compares LPIPS with L1, L2, PSNR, and SSIM, concluding that LPIPS provides smoother and more monotonic inconsistency curves for diffusion-style failures.

3. Reinforcement-learning finetuning formulation

Carve3D casts the denoising process as a Markov Decision Process. The paper defines

$$s_t=(c, t, x_t),\quad a_t=x_{t-1},\quad \pi(a_t|s_t)=p_\theta(x_{t-1}|c, t, x_t),$$

with terminal reward

$$r(s_t, a_t)= \begin{cases} r(x_0,c) & \text{if } t=0,\ 0 & \text{otherwise}, \end{cases} \qquad r(x_0,c) = -\text{MRC}(x_0).$$

Here $c$ is the text prompt, $x_T$ is the initial noise, and $\rightarrow$0 is the final tiled multi-view sample (Xie et al., 2023).

To reduce variance, rewards are normalized per prompt: $\rightarrow$1 The paper uses the score-function variant of DDPO rather than the importance-sampled PPO-style variant, reporting that DDPO$\rightarrow$2 is too unstable in this setting and yields high-variance reward curves and training failures. The selected estimator is

$\rightarrow$3

Carve3D also adds KL regularization toward the base SFT model, approximating trajectory-level KL by averaging per-timestep log-probability differences. That KL term is itself normalized per prompt and combined with the reward in

$\rightarrow$4

with $\rightarrow$5 and $\rightarrow$6. Functionally, the KL term behaves as a trust-region-style regularizer that discourages drift from the base model. The paper shows that removing KL causes faster divergence and degraded identity and texture, even when reward improvement remains similar.

4. Architecture, training procedure, and empirical results

Carve3D is applied to Instant3D-10K, itself built from SDXL. The generator uses the SDXL denoising UNet, described as 2.6B parameters, and outputs a $\rightarrow$7 image containing four tiled $\rightarrow$8 views in a $\rightarrow$9 layout. Inference uses 100 denoising steps and classifier-free guidance scale 5.0. For reconstruction inside the RL loop, the method uses sparse-view LRM, chosen because it reconstructs from only 4–6 views and is orders of magnitude faster than optimization-based NeRF fitting. Optimization is parameter-efficient: LoRA of rank 4 is applied to the UNet, the base weights remain frozen in fp16, and the LoRA weights are trained in fp32 with mixed precision. The reported training setup uses AdamW with learning rate $$\text{MRC}(x_0) = \frac{1}{V}\sum_{v=1}^{V} \operatorname{LPIPS}\!\left( \operatorname{Resize}\!\left(\operatorname{Crop}(I_v, B_v)\right), \operatorname{Resize}\!\left(\operatorname{Crop}(\hat I_v, B_v)\right) \right),$$0, $$\text{MRC}(x_0) = \frac{1}{V}\sum_{v=1}^{V} \operatorname{LPIPS}\!\left( \operatorname{Resize}\!\left(\operatorname{Crop}(I_v, B_v)\right), \operatorname{Resize}\!\left(\operatorname{Crop}(\hat I_v, B_v)\right) \right),$$1, $$\text{MRC}(x_0) = \frac{1}{V}\sum_{v=1}^{V} \operatorname{LPIPS}\!\left( \operatorname{Resize}\!\left(\operatorname{Crop}(I_v, B_v)\right), \operatorname{Resize}\!\left(\operatorname{Crop}(\hat I_v, B_v)\right) \right),$$2, $$\text{MRC}(x_0) = \frac{1}{V}\sum_{v=1}^{V} \operatorname{LPIPS}\!\left( \operatorname{Resize}\!\left(\operatorname{Crop}(I_v, B_v)\right), \operatorname{Resize}\!\left(\operatorname{Crop}(\hat I_v, B_v)\right) \right),$$3, and weight decay $$\text{MRC}(x_0) = \frac{1}{V}\sum_{v=1}^{V} \operatorname{LPIPS}\!\left( \operatorname{Resize}\!\left(\operatorname{Crop}(I_v, B_v)\right), \operatorname{Resize}\!\left(\operatorname{Crop}(\hat I_v, B_v)\right) \right),$$4, on 48 A100 80GB GPUs, with total batch size 768; one run to 55 epochs takes 16.5 hours, and early stopping is triggered at KL threshold $$\text{MRC}(x_0) = \frac{1}{V}\sum_{v=1}^{V} \operatorname{LPIPS}\!\left( \operatorname{Resize}\!\left(\operatorname{Crop}(I_v, B_v)\right), \operatorname{Resize}\!\left(\operatorname{Crop}(\hat I_v, B_v)\right) \right),$$5 (Xie et al., 2023).

Prompt selection is unusually small-scale. The paper curates a training set of only 30 prompts by first identifying prompts on which the base model has low reward, then generating additional prompts with ChatGPT-4 that resemble the “low-reward” regime, and finally selecting the lowest-reward prompts for RLFT. This design is presented as evidence that reward-guided finetuning can generalize beyond a very small prompt set when the base model already has a strong pretrained prior.

On the DreamFusion prompt benchmark, evaluated over 414/415 testing prompts with four outputs per prompt, Carve3DM achieves the best reported MRC among the compared models:

Model	MRC
MVDream	0.1222
Instant3D-10K	0.0892
Instant3D-20K	0.0795
Instant3D-100K	0.0685
Carve3DM	0.0606
Zero123++	0.0700
SyncDreamer	0.1018

These results support two distinct claims. First, longer SFT does improve consistency: Instant3D-100K outperforms Instant3D-20K and Instant3D-10K. Second, Carve3DM outperforms even Instant3D-100K despite starting from Instant3D-10K and using RLFT on only 30 training prompts, indicating that explicit reward optimization is more efficient than continued SFT for this objective. In a user study with 15 participants on 20 unseen prompts, 68.33% of preferences favored Carve3DM for 3D consistency, while prompt alignment was roughly tied with the base model (Xie et al., 2023).

5. Broader carve-based 3D methods and terminological neighbors

Outside the proper-noun usage of the 2023 RLFT method, “Carve3D” also resonates with a wider family of carve-based 3D techniques. In medical visualization, “AnatomyCarve” introduces “segment-aware clipping,” in which one or more clipping spheres are assigned per-segment Boolean masks so that clipping is jointly conditioned on geometry and anatomical label. For each voxel $$\text{MRC}(x_0) = \frac{1}{V}\sum_{v=1}^{V} \operatorname{LPIPS}\!\left( \operatorname{Resize}\!\left(\operatorname{Crop}(I_v, B_v)\right), \operatorname{Resize}\!\left(\operatorname{Crop}(\hat I_v, B_v)\right) \right),$$6, the paper defines

$$\text{MRC}(x_0) = \frac{1}{V}\sum_{v=1}^{V} \operatorname{LPIPS}\!\left( \operatorname{Resize}\!\left(\operatorname{Crop}(I_v, B_v)\right), \operatorname{Resize}\!\left(\operatorname{Crop}(\hat I_v, B_v)\right) \right),$$7

realizing clipping as segment-conditioned opacity nulling in direct volume rendering; the paper explicitly notes that “Carve3D” is a reasonable way to think about the method, even though the actual term is “segment-aware clipping” (Titov et al., 8 Jul 2025).

In reconstruction, the carve metaphor appears in several distinct forms. EvAC3D is a continuous, event-based contour-carving or visual-hull method: it classifies Apparent Contour Events (ACE), casts tangent rays at event timestamps, accumulates a voxel count field via 3D Bresenham traversal, and extracts a surface with Marching Cubes (Wang et al., 2023). ObjectCarver uses user clicks, SAM-based mask propagation, NeuS scene reconstruction, compactness loss, and overlap regularization to separate a scene into per-object SDFs; each object SDF is initialized as a copy of the full scene SDF and then “cut off” into an individual object while avoiding inappropriate carving-out under occlusion (Hassena et al., 2024). CarveNet formulates point-cloud completion as “Point-block Carving”: a uniformly sampled 3D point block is merged with the partial observation, and exclusive convolution determines which candidate points should be carved away to yield the complete shape (Guo et al., 2021).

In editing and augmentation, the same family extends to explicit mesh subtraction, content-aware resizing, and surface-detail generation. “Image Sculpting” supports carving by placing a user-specified mold against a reconstructed mesh and excising the overlapping region before diffusion-based 2D enhancement (Yenphraphai et al., 2024). Efficient 3D Seam Carving (E3SC) generalizes 2D seam carving to volumetric occupancy, SDF, or TSDF grids by searching for low-energy 2D sheet-like seams and removing or inserting them progressively for content-aware shape resizing (Chen et al., 2024). “SuperCarver” performs texture-consistent 3D geometry super-resolution by predicting detail-rich normal maps from multi-view renders of a coarse textured mesh and then optimizing a fixed-SDF, moving-grid distance-field deformation so that rendered normals match those predicted normals across views (Zhang et al., 12 Mar 2025).

These neighboring systems indicate that “carving” is not a single algorithmic primitive but a recurring technical motif. Depending on context, it may denote selective reveal in volume rendering, tangent-ray accumulation in visual-hull reconstruction, visibility-aware object separation, mesh-domain subtraction, seam-based volumetric resizing, point-block pruning, or inverse-rendered surface refinement.

6. Limitations, interpretation, and significance

Carve3D’s limitations are closely tied to its reward design. MRC depends on the sparse-view reconstruction model: if LRM reconstructs imperfectly even from perfectly consistent inputs, then the reward is biased and nonzero on ideal data. The paper explicitly notes that this makes MRC a consistency signal as judged through a particular reconstructor rather than a pure oracle of 3D correctness (Xie et al., 2023).

This dependence has methodological consequences. The reward can favor images that are easier for the reconstructor to explain, which the paper associates with some loss of high-frequency detail relative to the base model. Computational cost is also substantial, because every RL sample requires a full 100-step denoising trajectory plus a NeRF reconstruction-rendering pass. Credit assignment is indirect, since reward is terminal rather than stepwise, which helps explain the instability of PPO-style sample reuse in this setting.

Even with those caveats, Carve3D is significant for three reasons. First, it supplies a practical consistency metric that can function both as an evaluator and as a training reward. Second, it demonstrates that multi-view diffusion models can improve beyond the limitations of their SFT datasets through self-generated RLFT. Third, it reframes text-to-3D precursor training as an alignment problem: SFT is necessary to induce the task, but RLFT is used to optimize the intended behavior while preserving the broader visual prior of the underlying text-to-image model. A plausible implication is that, for multi-view diffusion, the pretraining $$\text{MRC}(x_0) = \frac{1}{V}\sum_{v=1}^{V} \operatorname{LPIPS}\!\left( \operatorname{Resize}\!\left(\operatorname{Crop}(I_v, B_v)\right), \operatorname{Resize}\!\left(\operatorname{Crop}(\hat I_v, B_v)\right) \right),$$8 SFT $$\text{MRC}(x_0) = \frac{1}{V}\sum_{v=1}^{V} \operatorname{LPIPS}\!\left( \operatorname{Resize}\!\left(\operatorname{Crop}(I_v, B_v)\right), \operatorname{Resize}\!\left(\operatorname{Crop}(\hat I_v, B_v)\right) \right),$$9 RLFT sequence may be as structurally important as it has become in LLMs, even though the optimization target here is reconstruction consistency rather than human preference (Xie et al., 2023).