Mosaic-SDF: Local SDF Patches for 3D Generation

• Mosaic-SDF is a boundary-adaptive 3D shape representation that uses overlapping local SDF patches to approximate a shape's signed distance function.
• It efficiently captures geometric details by focusing on local volumetric grids near shape boundaries, reducing redundancy compared to dense voxel grids.
• The method integrates a simple tensorial structure with Transformer-based architectures, offering parameter efficiency and fast reconstruction for generative modeling.

Searching arXiv for the Mosaic-SDF paper and closely related work to ground the article in the cited literature. Mosaic-SDF (M-SDF) is a 3D shape representation for forward-based generative modeling that approximates a shape’s signed distance function by a set of local volumetric grids placed near the shape boundary. Rather than storing a single global voxel grid or fitting one implicit network per shape, it represents a shape as a “mosaic” of overlapping local SDF patches with a simple matrix form compatible with Transformer-based architectures. The representation was introduced for class-conditioned and text-conditioned 3D generation, with the stated design goals of preprocess efficiency, parameter efficiency, and simple tensorial structure (Yariv et al., 2023).

1. Design rationale and representational premise

The motivation for M-SDF is the observation that 3D shape geometry is concentrated near the surface boundary, while most ambient volume is empty. In this setting, dense volumetric grids scale cubically with resolution and store many redundant empty voxels; point clouds are simple to preprocess but do not directly encode a smooth surface or full signed distance field; meshes lack a simple fixed tensorial structure; and neural fields represented by per-shape MLP weights are expensive to fit and difficult to use directly in generative modeling. M-SDF is positioned as a representation that satisfies three explicit principles simultaneously: efficient preprocessing over large datasets, a favorable approximation-quality versus parameter-count tradeoff, and a simple tensorial form suitable for modern neural architectures (Yariv et al., 2023).

The underlying geometric object is the signed distance function of a shape $\Omega \subset \mathbb{R}^3$,

$$F_\Omega(x) = \begin{cases} -d(x,\partial \Omega) & x \in \Omega \ d(x,\partial \Omega) & x \notin \Omega \end{cases},$$

with

$$d(x,\partial \Omega)=\min_{y\in\partial \Omega}\|x-y\|_2.$$

M-SDF does not store $F_\Omega$ globally on a full lattice. Instead, it constructs a boundary-adaptive local covering and approximates the SDF only where geometric signal is concentrated (Yariv et al., 2023).

This choice places M-SDF between dense global discretizations and fully implicit per-shape neural representations. A plausible implication is that the method is best understood as a boundary-focused local-field representation whose utility derives as much from representation design as from the downstream generative model.

2. Formal definition of the representation

M-SDF represents a shape by a collection of $n$ local SDF patches,

$X = \{ (p_i, s_i, V_i)\}_{i\in [n]},$

where $p_i \in \mathbb{R}^3$ is the center of the $i$-th local volume, $s_i \in \mathbb{R}$ is its local scale, and $V_i \in \mathbb{R}^{k\times k\times k}$ is a small volumetric grid of SDF values. In the reported experiments, the local grid resolution is $$F_\Omega(x) = \begin{cases} -d(x,\partial \Omega) & x \in \Omega \ d(x,\partial \Omega) & x \notin \Omega \end{cases},$$0, so each patch stores $$F_\Omega(x) = \begin{cases} -d(x,\partial \Omega) & x \in \Omega \ d(x,\partial \Omega) & x \notin \Omega \end{cases},$$1 scalar values (Yariv et al., 2023).

Each local grid is defined over the origin-centered cube $$F_\Omega(x) = \begin{cases} -d(x,\partial \Omega) & x \in \Omega \ d(x,\partial \Omega) & x \notin \Omega \end{cases},$$2. Let $$F_\Omega(x) = \begin{cases} -d(x,\partial \Omega) & x \in \Omega \ d(x,\partial \Omega) & x \notin \Omega \end{cases},$$3 denote the trilinear interpolant of $$F_\Omega(x) = \begin{cases} -d(x,\partial \Omega) & x \in \Omega \ d(x,\partial \Omega) & x \notin \Omega \end{cases},$$4, with the convention

$$F_\Omega(x) = \begin{cases} -d(x,\partial \Omega) & x \in \Omega \ d(x,\partial \Omega) & x \notin \Omega \end{cases},$$5

The global approximated SDF is then

$$F_\Omega(x) = \begin{cases} -d(x,\partial \Omega) & x \in \Omega \ d(x,\partial \Omega) & x \notin \Omega \end{cases},$$6

The weights satisfy a partition-of-unity constraint,

$$F_\Omega(x) = \begin{cases} -d(x,\partial \Omega) & x \in \Omega \ d(x,\partial \Omega) & x \notin \Omega \end{cases},$$7

with

$$F_\Omega(x) = \begin{cases} -d(x,\partial \Omega) & x \in \Omega \ d(x,\partial \Omega) & x \notin \Omega \end{cases},$$8

Thus each local patch contributes only inside its own cube, and overlapping predictions are blended locally (Yariv et al., 2023).

The effective represented domain is

$$F_\Omega(x) = \begin{cases} -d(x,\partial \Omega) & x \in \Omega \ d(x,\partial \Omega) & x \notin \Omega \end{cases},$$9

where

$$d(x,\partial \Omega)=\min_{y\in\partial \Omega}\|x-y\|_2.$$0

This makes the representation explicitly boundary-focused rather than volumetrically exhaustive (Yariv et al., 2023).

For generative modeling, the collection is flattened into a matrix

$$d(x,\partial \Omega)=\min_{y\in\partial \Omega}\|x-y\|_2.$$1

Each row contains the patch center, scale, and flattened local grid. With the default setting $$d(x,\partial \Omega)=\min_{y\in\partial \Omega}\|x-y\|_2.$$2, $$d(x,\partial \Omega)=\min_{y\in\partial \Omega}\|x-y\|_2.$$3,

$$d(x,\partial \Omega)=\min_{y\in\partial \Omega}\|x-y\|_2.$$4

so the representation is

$$d(x,\partial \Omega)=\min_{y\in\partial \Omega}\|x-y\|_2.$$5

This corresponds to about $$d(x,\partial \Omega)=\min_{y\in\partial \Omega}\|x-y\|_2.$$6K parameters per shape. The rows are treated as an unordered set, so the representation is invariant to row permutation up to relabeling (Yariv et al., 2023).

3. Construction pipeline and fitting procedure

The preprocessing pipeline converts each mesh independently into an M-SDF representation. Before fitting, each mesh is scaled to fit inside $$d(x,\partial \Omega)=\min_{y\in\partial \Omega}\|x-y\|_2.$$7 and made watertight using the method/library of Huang et al. and Point Cloud Utils. The paper notes that this watertight manifold assumption is standard in implicit shape methods and supports mesh extraction (Yariv et al., 2023).

Patch centers $$d(x,\partial \Omega)=\min_{y\in\partial \Omega}\|x-y\|_2.$$8 are initialized by farthest point sampling over the surface $$d(x,\partial \Omega)=\min_{y\in\partial \Omega}\|x-y\|_2.$$9, which distributes local cubes across the boundary. The initial scales are set uniformly,

$F_\Omega$0

where $F_\Omega$1 is the minimal value achieving full boundary coverage,

$F_\Omega$2

In practice, the supplementary material states that this is estimated by densely sampling the surface and finding the minimal distance from the dense sample set to the initialized centers (Yariv et al., 2023).

The initial local grids are obtained by direct SDF sampling,

$F_\Omega$3

where $F_\Omega$4 is the $F_\Omega$5 unit grid inside $F_\Omega$6. This yields an immediately usable approximation before optimization (Yariv et al., 2023).

Fine-tuning then optimizes patch parameters to better match both SDF values and SDF gradients. The loss is

$F_\Omega$7

with

$F_\Omega$8

and

$F_\Omega$9

The supervision points are sampled on and near the surface, and the paper states that this loss follows DeepSDF / IGR-style fitting ideas (Yariv et al., 2023).

The reported fitting settings are concrete: 300K points sampled on the surface, 200K near-surface points obtained by Gaussian perturbation with variance $n$0, 16K points per optimization step, 1K Adam steps, learning rate $n$1, and in the main experiments $n$2, $n$3, $n$4. The fitting procedure is reported to take less than 2 minutes on a single NVIDIA A100 GPU per shape, and the preprocessing is described as highly parallelizable because each shape is processed independently (Yariv et al., 2023).

4. Role in generative modeling

M-SDF is used as the shape representation for a Flow Matching generative model. Given dataset pairs

$n$5

where $n$6 is the M-SDF representation and $n$7 is a condition, the model learns a velocity field

$n$8

Sampling is defined by the ODE

$n$9

starting from Gaussian noise $X = \{ (p_i, s_i, V_i)\}_{i\in [n]},$0 with $X = \{ (p_i, s_i, V_i)\}_{i\in [n]},$1 and integrating to time $X = \{ (p_i, s_i, V_i)\}_{i\in [n]},$2 (Yariv et al., 2023).

The training path uses Conditional Optimal Transport interpolation,

$X = \{ (p_i, s_i, V_i)\}_{i\in [n]},$3

with $X = \{ (p_i, s_i, V_i)\}_{i\in [n]},$4, so

$X = \{ (p_i, s_i, V_i)\}_{i\in [n]},$5

With classifier-free guidance training, the objective is

$X = \{ (p_i, s_i, V_i)\}_{i\in [n]},$6

where $X = \{ (p_i, s_i, V_i)\}_{i\in [n]},$7, $X = \{ (p_i, s_i, V_i)\}_{i\in [n]},$8, and $X = \{ (p_i, s_i, V_i)\}_{i\in [n]},$9 is either the condition or the null token (Yariv et al., 2023).

The velocity model is a Transformer without positional encoding, which makes it permutation equivariant over the row-set representation. The paper reports 24 Transformer layers, 16 attention heads, hidden dimension 1024, and total model size 328M parameters. Each row of $p_i \in \mathbb{R}^3$0 is used as a token (Yariv et al., 2023).

Conditioning is implemented in two regimes. For class-conditioned generation on ShapeNetCore-v2 / 3D Warehouse, each class has a learned embedding of dimension 128, projected to 1024. For text-conditioned generation, the model uses a pretrained T5 text model with embedding size 768 and maximum sequence length 32; each text token is linearly projected to 1024 before entering the Transformer (Yariv et al., 2023).

At inference, classifier-free guidance is applied through

$p_i \in \mathbb{R}^3$1

The paper reports $p_i \in \mathbb{R}^3$2 for class-conditional generation in the main results and $p_i \in \mathbb{R}^3$3 for text-conditioned generation. An ablation in the supplement finds that larger $p_i \in \mathbb{R}^3$4 tends to make shapes more common or average-looking and often degrades distribution metrics (Yariv et al., 2023).

5. Reconstruction, efficiency, and empirical results

Surface reconstruction from M-SDF proceeds by evaluating

$p_i \in \mathbb{R}^3$5

on a 3D query grid and extracting the zero level set with Marching Cubes. The representation is flexible at inference because $p_i \in \mathbb{R}^3$6 can be sampled at any desired extraction resolution; the paper contrasts this with methods whose extraction is tied to a fixed generated grid or requires a Transformer forward pass during function evaluation (Yariv et al., 2023).

In representation evaluation, M-SDF is compared to 3D-Grid, Triplane, and INR. The reported finding is that M-SDF gives the best approximation quality for a given parameter budget, is much faster than INR, and that 3D grids are faster but significantly worse in approximation per parameter. At a budget of 355K parameters, the paper reports rough fitting times of about 1 minute for M-SDF, about 30 minutes for INR, about 0.5 minute for 3D Grid, and about 6 minutes for Triplane; the implementation section states that the full fine-tuned version used in experiments takes under 2 minutes per shape (Yariv et al., 2023).

For class-conditioned generation, M-SDF is compared with 3DILG, Neural Wavelet, and 3DShape2VecSet using FPD, KPD, COV, MMD, and 1-NNA. On the five largest classes—airplane, car, chair, sofa, and table—the paper states that M-SDF is best or second-best on essentially all metrics, and often clearly best on FPD/KPD and distributional metrics. Specific examples reported from Table 1 include: for airplane, best FPD $p_i \in \mathbb{R}^3$7, best KPD $p_i \in \mathbb{R}^3$8, best COV-EMD $p_i \in \mathbb{R}^3$9, best MMD-EMD $i$0, and best 1-NNA; for car, best performance across all reported metrics; for chair, best on FPD/KPD, COV-EMD, MMD-CD/EMD, and 1-NNA, with 3DShape2VecSet slightly better in COV-CD; and for table, best on FPD/KPD, COV-EMD, and 1-NNA, with Neural Wavelet slightly better on MMD-CD (Yariv et al., 2023).

The data regime is also unusually large for direct 3D generative modeling. The class-conditioned experiments use about 50K polygonal meshes across 55 categories from ShapeNetCore-v2 / 3D Warehouse. The text-conditioned experiments use about 600K caption-shape pairs from Luo et al., and the paper presents qualitative text-to-3D results rather than a quantitative metric table (Yariv et al., 2023).

Extraction-time results further emphasize practical deployability. The reported total extraction times for M-SDF are $i$1 s at $i$2, $i$3 s at $i$4, $i$5 s at $i$6, and $i$7 s at $i$8. In the same comparison, 3DILG is reported at $i$9, $s_i \in \mathbb{R}$0, $s_i \in \mathbb{R}$1, and $s_i \in \mathbb{R}$2 s respectively; S2VS at $s_i \in \mathbb{R}$3, $s_i \in \mathbb{R}$4, and out-of-memory at $s_i \in \mathbb{R}$5 and $s_i \in \mathbb{R}$6; and Neural Wavelet at $s_i \in \mathbb{R}$7 s at $s_i \in \mathbb{R}$8 only. This suggests that M-SDF’s locality is useful not only during training but also during surface extraction (Yariv et al., 2023).

6. Relation to neighboring methods and naming ambiguities

The term “Mosaic-SDF” has specific meaning in 3D generative modeling and should be distinguished from several differently motivated methods that are only adjacent in spirit. In pathology, “Selection of Distinct Morphologies” introduces an unsupervised WSI patch-selection method whose output is called a montage; it is contrasted with Yottixel’s mosaic, but the paper does not introduce the name Mosaic-SDF or M-SDF. The resemblance is terminological rather than methodological (Shafique et al., 2023).

Within 3D reconstruction, SDFoam combines an explicit Voronoi Diagram / Delaunay tessellation with a global neural signed distance field. The method partitions space into Voronoi cells and derives cell density from a shared SDF, which the paper describes as introducing metric-consistent isosurfaces and biasing near-surface Voronoi faces toward the zero level set. This suggests a cellular or “mosaic-like” decomposition of space, but SDFoam is not a bank of independent local SDFs; it is a global SDF coordinated with an explicit Voronoi scaffold (Rech et al., 18 Dec 2025).

In robotics, “Learning Deep SDF Maps Online for Robot Navigation and Exploration” maintains a sequence of local neural SDF maps with learned confidence and composes them into a global field through confidence-based selection or softmax blending. This is conceptually close to a submap or mosaic neural SDF system, but it is event-driven by convergence and novelty rather than built from fixed local grids near a static object boundary (Camps et al., 2022).

For exact extraction, TetraSDF is relevant because it preserves continuous piecewise affine structure by composing a ReLU MLP with a multi-resolution tetrahedral positional encoder using barycentric interpolation. TetraSDF therefore addresses a different problem: precise analytic extraction of the exact zero-level set of a neural SDF. This suggests a complementary axis of comparison with M-SDF, namely local decomposition plus extraction fidelity rather than local decomposition plus generative efficiency (Oh et al., 20 Nov 2025).

7. Limitations and prospective directions

The limitations stated for M-SDF are primarily representational. The method currently encodes only the SDF, not texture, color, or lighting. The paper also notes that the current Transformer input pipeline uses a simple linear layer to embed each local grid, and suggests that convolutional processing or autoencoders could improve resolution and feature reuse. In addition, the model is permutation equivariant over local patches but not orientation equivariant; the authors suggest local coordinate frames as a possible extension (Yariv et al., 2023).

The preprocessing assumptions also matter. Like many implicit-surface methods, M-SDF relies on watertight preprocessing and surface extraction assumptions, and the supplementary material notes that thin structures can be impacted by watertight conversion. This does not negate the reported empirical performance, but it constrains the kinds of raw geometry that can be ingested without additional processing (Yariv et al., 2023).

A broader methodological limitation is that M-SDF’s strengths are tied to a particular balance among locality, simplicity, and tensor compatibility. The method is explicitly designed to be simpler to preprocess than shared latent implicit methods such as 3DILG and 3DShape2VecSet, and more boundary-aware than dense grids or point clouds. This suggests that M-SDF occupies a specific niche: it is not primarily an exact-meshing framework, a radiance-field model, or an online mapping system, but a boundary-adaptive SDF representation for large-scale generative learning (Yariv et al., 2023).

In that sense, the defining formula

$s_i \in \mathbb{R}$9

captures the central idea precisely: a 3D shape is represented as a mosaic of overlapping local SDF grids whose union covers the boundary and whose weighted blend approximates the full signed distance function.