UniFi3D: A Unified Framework for 3D Representations

• UniFi3D is a unified benchmarking framework for evaluating 3D representations, integrating generation and reconstruction processes for fair comparison.
• It standardizes preprocessing, compression with autoencoders, diffusion-based generation, and mesh reconstruction to isolate the impact of representation choices.
• Empirical findings show that reconstruction errors critically constrain generative performance, highlighting trade-offs in quality, efficiency, and generalization.

UniFi3D, as represented by "Unifi3D: A Study on 3D Representations for Generation and Reconstruction in a Common Framework" (Wiedemann et al., 2 Sep 2025), is a unified benchmarking and research framework for 3D representation learning that studies how different 3D formats perform when used in the full generation + reconstruction pipeline. Rather than comparing methods in isolation, it standardizes the entire workflow so that representation choices can be evaluated fairly and jointly across quality, computational efficiency, and generalization performance. Its central empirical claim is that reconstruction errors significantly impact overall performance, underscoring the need to evaluate generation and reconstruction jointly.

1. Scope, motivation, and research questions

Unifi3D is organized around a specific methodological problem: 3D generation methods are often incomparable because they differ not only in representation, but also in preprocessing, compression, decoder architecture, diffusion model, and metrics (Wiedemann et al., 2 Sep 2025). The framework therefore asks four linked questions: which 3D representations are best for reconstruction, which are best for generation, how preprocessing and reconstruction errors affect generative quality, and what trade-offs exist among quality, efficiency, and generalization.

This emphasis distinguishes the framework from single-model leaderboards. Its main idea is to put different representations into the same pipeline so that the effect of representation choice can be isolated more cleanly. The framework is open-source, with code released at https://github.com/isl-org/unifi3d (Wiedemann et al., 2 Sep 2025).

A central consequence of this design is that generation is not treated as an autonomous endpoint. The framework’s findings argue that if a representation reconstructs poorly, a generative model built on it is already disadvantaged. This suggests that Unifi3D is best understood not primarily as a new 3D generator, but as a comparative infrastructure for studying how representational decisions propagate through the entire 3D pipeline.

2. Representations and the standardized pipeline

The benchmark focuses on several common 3D representations: voxel grids, signed distance functions (SDFs), point-cloud or set-based representations via 3DShape2VecSet, triplanes, NeRFs, and Dual Octrees (Wiedemann et al., 2 Sep 2025). The paper also surveys related work using other representations such as meshes, occupancy fields, sparse voxel hierarchies, and hybrid schemes, but the experimental benchmark is restricted to those six families.

Representation	Characterization in the framework
Voxel grids	regular 3D grids, easy for CNNs, but memory-heavy
SDFs	continuous geometry via signed distance to surface
Point clouds / 3DShape2VecSet	flexible and detailed, good for implicit surface prediction
Triplanes	efficient hybrid explicit-implicit encoding
NeRFs	implicit radiance/density fields adapted to generation
Dual Octrees	hierarchical adaptive volumetric representation

The standardized pipeline has four stages. First, meshes are converted into a chosen representation through preprocessing steps such as point sampling, ray casting, closest-point distance queries, SDF computation, voxelization, octree construction, point cloud sampling, or multi-view image generation for NeRF-style encodings. The authors emphasize that preprocessing is crucial and often underreported in prior work (Wiedemann et al., 2 Sep 2025).

Second, the chosen representation is compressed into a latent space with an autoencoder family, specifically AE, VAE, or VQ-VAE. The encoder–decoder pair is trained on the reconstruction path

$$\text{mesh} \rightarrow \text{representation} \rightarrow \text{latent} \rightarrow \text{representation} \rightarrow \text{mesh}.$$

The encoder usually ends with LayerNorm to make latents better suited for diffusion.

Third, a generative model is trained over the latents, mainly using diffusion models. The framework uses DiT and U-Net diffusion, with DiT working well for tokenizable or structured latents and U-Net often better for grid-like representations (Wiedemann et al., 2 Sep 2025).

Fourth, the generated latent is decoded back to the target 3D representation and then converted into a mesh, often with Marching Cubes. For NeRFs, the density field is also converted to a mesh via Marching Cubes. This explicit inclusion of mesh reconstruction is one of the framework’s defining features, because it prevents evaluation from ending at an intermediate latent or field representation.

3. Evaluation methodology and formal metrics

Unifi3D evaluates representations along three axes: quality, computational efficiency, and generalization (Wiedemann et al., 2 Sep 2025). Quality is measured both for reconstruction and for unconditional generation. For reconstruction, the primary mesh-based metrics are Chamfer Distance (CD), F-score, and Normal Consistency (NC). For unconditional generation, the framework uses Coverage (COV), Minimum Matching Distance (MMD), and 1-Nearest Neighbor Accuracy (1-NNA), and it supplements these with a user study modeled with the Bradley–Terry ranking framework.

The appendix provides the formal metric definitions. For point sets $X$ and $Y$, Chamfer Distance is

$$\mathrm{CD}(X,Y) = \frac{1}{|X|}\sum_{x\in X} \min_{y\in Y}\|x-y\|_2 + \frac{1}{|Y|}\sum_{y\in Y} \min_{x\in X}\|y-x\|_2.$$

F-score combines precision and recall at threshold $\tau$:

$F(\tau) = \frac{2P(\tau)R(\tau)}{P(\tau)+R(\tau)},$

with $\tau = 0.0125$ in the main experiments.

For generation, Coverage and Minimum Matching Distance are defined as

$$\mathrm{COV}(S_g, S_r) = \frac{|\{\arg\min_{Y\in S_r} D(X,Y)\mid X\in S_g\}|}{|S_r|}$$

and

$$\mathrm{MMD}(S_g, S_r) = \frac{1}{|S_r|}\sum_{Y\in S_r}\min_{X\in S_g} D(X,Y),$$

where the distance $D$ is also instantiated with Chamfer Distance. The 1-NNA statistic is

$X$0

and the paper notes that a perfect generator is near $X$1.

Computational efficiency is measured through latent size, memory footprint, and inference or decoding runtime. Generalization is tested by transferring encoders across categories, for example training on chairs and testing on airplanes. This OOD protocol is not ancillary; it is used to assess whether a representation is robust beyond the training class (Wiedemann et al., 2 Sep 2025).

4. Empirical findings and representation trade-offs

The strongest reconstruction performance in the benchmark is achieved by SDF autoencoders (Wiedemann et al., 2 Sep 2025). On ShapeNet, SDF AE obtains an F-score of about $X$2, a CD of about $X$3, and an NC of about $X$4. Within the framework’s comparative logic, this makes SDFs the strongest choice when accurate mesh recovery is prioritized.

For unconditional generation, the best distributional scores come from DualOctree + VAE + UNet, with COV $X$5, MMD $X$6, and 1-NNA $X$7 (Wiedemann et al., 2 Sep 2025). At the same time, the user study ranks SDF-based outputs higher in human preference. One common misconception addressed by the paper is therefore that metric-leading generation necessarily coincides with perceptual preference. In this benchmark, distributional metrics and human judgments do not fully agree.

The framework also reports that standard autoencoders with LayerNorm often outperform VAEs. The authors suggest that LayerNorm makes AE latents behave more like diffusion-friendly distributions (Wiedemann et al., 2 Sep 2025). This is a notable result because it runs against a common assumption that VAE-style regularization is intrinsically more suitable for latent generative modeling.

The paper’s most consequential decomposition concerns error propagation. Reconstruction errors contribute about $X$8 of MMD, while compression errors contribute about $X$9 of MMD. It also reports a Pearson correlation of $Y$0 between MMD and reconstruction CD, and a correlation of $Y$1 between compression and reconstruction error (Wiedemann et al., 2 Sep 2025). These values underpin the paper’s central thesis: poor reconstruction places a ceiling on generative performance, so generation-only evaluation can hide representation-induced losses.

5. Preprocessing, mesh fidelity, and benchmark design principles

A substantial part of Unifi3D concerns preprocessing. The paper identifies non-watertight meshes as a major source of ambiguity for SDF or voxel conversion, because inside–outside labeling becomes unstable (Wiedemann et al., 2 Sep 2025). Common remedies such as Manifold or Mesh2SDF can make shapes watertight, but may thicken geometry, distort thin structures, and introduce bias.

To address this, the framework introduces a flood-fill based method to define inside and outside without thickening the mesh. The stated purpose is to reduce artificial geometry distortion and better preserve the original mesh when converting to SDF (Wiedemann et al., 2 Sep 2025). This is important because otherwise preprocessing artifacts become entangled with the representation’s actual strengths and weaknesses.

Several benchmark-design recommendations follow directly from this analysis. The authors recommend measuring errors against the original mesh, not only against the preprocessed intermediate. They also argue that mesh conversion should always be included in evaluation, because comparing only intermediate representations can be misleading. For unconditional metrics, they recommend more than 200 samples for stable COV, MMD, and 1-NNA estimates (Wiedemann et al., 2 Sep 2025).

The generalization study reinforces these design concerns. When trained on chairs and tested on airplanes, most representations generalize fairly well, but NeRFs and DualOctrees struggle more OOD (Wiedemann et al., 2 Sep 2025). The paper interprets this as evidence that compact or highly specialized latent structures may overfit category-specific geometry.

6. Position within unified 3D research

Unifi3D belongs to a broader wave of work organized around “unified” 3D modeling, but its role within that wave is distinct. UniForward unifies 3D scene and semantic field reconstruction via feed-forward Gaussian Splatting from only sparse-view images (Tian et al., 11 Jun 2025). UniSH unifies metric-scale 3D scene and human reconstruction in a feed-forward pass (Li et al., 3 Jan 2026). PnP-U3D combines autoregression with diffusion for unified 3D understanding and generation (Chen et al., 3 Feb 2026). UniUGG presents a unified understanding-and-generation framework for 3D modalities built around a geometric-semantic encoder, an LLM, and a latent diffusion spatial decoder (Xu et al., 16 Aug 2025). UNITE unifies semantic segmentation, instance embeddings, open-vocabulary features, affordance, and articulation prediction for 3D scenes from RGB images (Koch et al., 16 Dec 2025). Uni3D, by contrast, explores unified 3D representation at scale as a 3D foundation model aligned to image-text features (Zhou et al., 2023).

Against this background, Unifi3D is not primarily a feed-forward end-task architecture for scene reconstruction, human reconstruction, generation, or semantic reasoning. Its contribution is a common framework for evaluating how representations behave across preprocessing, compression, generation, and mesh reconstruction. This suggests that it occupies a methodological layer beneath many of the newer “unified” systems: it provides a way to compare representational substrates and pipeline decisions that those systems often inherit implicitly.

Its overall takeaway is therefore narrower than a universal model but broader than a benchmark table. The framework argues that 3D representations should not be judged only by generative sample quality; their ability to reconstruct meshes, compress faithfully, generalize to new shapes, and support efficient decoding matters just as much (Wiedemann et al., 2 Sep 2025). In that sense, UniFi3D formalizes a comparative research program in which reconstruction error is not a side issue, but a determinant of downstream generative performance.