3D Proxy Construction: Methods & Applications

• 3D proxy construction is a technique that represents complex 3D scenes using simplified geometry and learnable features.
• It utilizes methods like sparse meshes, voxel grids, and hierarchical clusters for efficient data fusion and neural integration.
• This approach enhances controllability, real-time editing, and multi-modal integration for applications such as animation and scene synthesis.

3D proxy construction is a foundational paradigm in contemporary 3D vision, graphics, and neural rendering pipelines. The core principle is to represent complex 3D objects or scenes using a set of simplified geometric or structural carriers—"proxies"—that mediate between raw high-dimensional data (e.g., images, point clouds) and downstream tasks such as animation, editing, synthesis, scene understanding, or efficient neural encoding. Proxies are realized via coarse meshes, hierarchical points or nodes, semantic bounding boxes, voxel grids, or analytically defined primitives, and can be coupled to implicit or learnable per-proxy feature embeddings for appearance, semantics, or motion. This intermediary construction enables explicit geometric control, interpretable manipulation, and efficient mapping to neural architectures, with significant advantages over purely neural or volumetric approaches in terms of controllability, editability, and resource efficiency.

1. Proxy Representation Paradigms

Several proxy construction paradigms have emerged, each dictated by application constraints, source input modality, and desired trade-off between fidelity and control.

• Sparse proxy meshes and point sets: For controllable animation from monocular signals, as in 3DProxyImg, a lightweight triangle mesh or coarse point set is extracted, typically by fusing partial geometry from monocular depth estimation (e.g., VGGT) with full meso-scale shape from a generative prior (e.g., Hunyuan3D). Each proxy vertex stores a compact, learnable feature vector for implicit appearance (Zhu et al., 17 Dec 2025).
• Hierarchically organized proxies: HPR3D introduces a multi-level tree of proxy nodes, with the bottom level sampling mesh vertices and interior levels summarizing larger spatial regions. This enables efficient, scale-adaptive interpolation and editing, and supports both geometry and appearance by associating each node with learnable features decoded by a shared small MLP (Wang et al., 16 Jul 2025).
• Proxy volumes and voxel grids: Coin3D leverages user-assembled primitive-based proxies, voxelized into binary occupancy grids and subsequently extended to high-dimensional feature volumes through 3D convolutional adapters, serving as spatial control for diffusion-based 3D generation (Dong et al., 2024).
• Semantic and geometric proxy boxes: In ControlRoom3D, and for vision-language reasoning (Proxy3D), proxies take the form of axis-aligned bounding boxes or semantic instance clusters in 3D space, either as layout scaffolds or as compact, semantically-grouped clusters distilled from feature-enriched point clouds (Schult et al., 2023, Jiang et al., 8 May 2026).
• Proxy-based point association: SP²T and MPPNet construct spatially distributed proxies (regular grids or local object-aligned points), with explicit associations between input points and proxies to facilitate efficient hierarchical attention or temporal feature fusion in point cloud analysis and object detection (Wan et al., 2024, Chen et al., 2022).
• Spherical or adaptive proxies for haptic interaction: For real-time haptic rendering of point clouds, adaptive spherical proxies, whose radii are determined by local density via kernel density estimation, enable stable contact simulation without requiring explicit surface meshing (Kumari et al., 2020).

2. Proxy Construction Pipelines and Algorithms

The pipeline for 3D proxy construction is highly structured and task-specific, but general principles recur across methods:

• Data fusion and alignment: Combination of monocular or multiview depth, generative shape priors, object detection, and ICP-based rigid alignment produce a spatially consistent proxy shape, as in mesh-based proxies for single-image animation (Zhu et al., 17 Dec 2025).
• Downsampling and hierarchical clustering: Adaptive error-guided clustering (octree, K-means) reduces full-resolution geometry or feature clouds to manageable proxy sets; hierarchical organization supports multi-scale querying and manipulation (Wang et al., 16 Jul 2025, Jiang et al., 8 May 2026).
• Voxelization and masking: Conversion of primitive-based assemblies to voxel occupancy grids or binary masks enables neural integration and spatial conditioning in volumetric generative models (Dong et al., 2024).
• Semantic clustering: Assignment of low-level descriptors (from 2D or 3D encoders) to semantic labels, grouping, and spatial clustering are used to form proxies that summarize and compress scene representations for VLMs or reasoning (Jiang et al., 8 May 2026).
• Proxy-to-data or point-proxy association: Regular grid sampling and association schemes (vertex-based for each 3D cell, or object-local for temporal alignment) facilitate sparse attention or inter-frame correspondence (Wan et al., 2024, Chen et al., 2022).
• Adaptive sizing: Proxy parameters (e.g., sphere radius for haptic proxies) are adapted in real time using local density/gap estimation to ensure robustness to input sparsity or variable resolution (Kumari et al., 2020).

3. Neural Integration and Optimization Objectives

Once established, proxies interface tightly with neural architectures—either as structural carriers for feature learning or as targets for supervision and regularization.

• Feature learning at proxies: Attach per-proxy implicit features, optimized to match input views or appearance via MLP decoders, as in 3DProxyImg or HPR3D, often augmenting with positional encoding for high-frequency variation (Zhu et al., 17 Dec 2025, Wang et al., 16 Jul 2025).
• Proxy feature fusion and attention: SP²T leverages sparse proxy attention between point and proxy features, augmented by learnable table-based relative biases for efficient and precise feature aggregation across the input (Wan et al., 2024).
• Hierarchical message passing: Features at different proxy levels are fused top-down (HPR3D), and in temporal sequences proxies serve as "couriers" for multi-frame feature intertwining and sequence aggregation (MPPNet) (Wang et al., 16 Jul 2025, Chen et al., 2022).
• Neural adapters and control strength: Volumetric proxies are injected into diffusion pipelines via specialized 3D UNet adapters, with user-tunable interpolation between data-driven and proxy-driven synthesis (Dong et al., 2024).
• Losses and regularization: Training objectives include view-reconstruction (MSE/SSIM), multi-view consistency via Score Distillation Sampling (SDS), Laplacian smoothness to prevent geometric artifacts, mask alignment, and geometry-aware losses (e.g., normal consistency, size regularization) (Zhu et al., 17 Dec 2025, Wang et al., 16 Jul 2025, Lahlali et al., 2024, Schult et al., 2023).
• Curriculum alignment: For VLMs, multi-stage alignment is used to bridge symbolic/semantic proxy representations and real-geometry-based proxies, leveraging supervised and instruction-tuning objectives (Jiang et al., 8 May 2026).

4. Applications: Animation, Editing, Generation, and Reasoning

Proxy-based construction is instrumental in diverse applications:

• Controllable animation and deformation: The separation of proxy-driven geometry from implicit appearance, as in 3DProxyImg and HPR3D, enables interactive deformation via handle-based mesh editing, PBD, or skeleton-based rigging, with appearance coherently re-synthesized from fixed learned features (Zhu et al., 17 Dec 2025, Wang et al., 16 Jul 2025).
• Precise video editing: Shape-for-Motion employs time-consistent proxies to propagate user edits—using dual-path geometry and texture alignment—across frames, allowing for physically consistent, photorealistic video edits by steering the results of video diffusion models with proxy-based control signals (Liu et al., 27 Jun 2025).
• Interactive generation and local editing: Coin3D's proxy volume permits tightly bounded local edits, enabling precise and responsive updates to 3D assets with immediate preview and edit consistency (Dong et al., 2024).
• Scene layout and semantic alignment: ControlRoom3D guarantees layout plausibility in generative scene synthesis by constraining all stages of rendering, inpainting, and mesh reconstruction to a user-editable semantic bounding box proxy, preventing geometric or semantic inconsistencies (Schult et al., 2023).
• Efficient haptic interaction and LOD control: Adaptive proxies, whose radius is modulated by local density, allow robust haptic rendering and variable resolution object exploration without mesh construction overhead (Kumari et al., 2020).
• 3D reasoning and VLM alignment: Proxy3D clusters scene features into compact proxies for seamless integration with vision-language backbones, dramatically improving efficiency and spatial consistency in 3D question answering and grounding (Jiang et al., 8 May 2026).
• Hybrid representations and deformation: UniMGS binds Gaussians to mesh faces for coherent joint deformation and single-pass rasterization, unifying mesh and 3DGS pipelines for graphics applications, including occlusion-correct rendering and mesh-driven manipulation (Xiao et al., 27 Jan 2026).

5. Performance, Scalability, and Trade-offs

The adoption of proxies introduces favorable computational and usability properties, but necessitates careful design:

• Compactness and editability: Hierarchical or clustered proxies (HPR3D, Proxy3D) yield orders-of-magnitude parameter reduction versus fully volumetric approaches, while supporting drag-and-edit manipulation and real-time control (Wang et al., 16 Jul 2025, Jiang et al., 8 May 2026).
• Quality-control scaling: The proxy count controls the fidelity/complexity trade-off—lower thresholds concentrate representation in high-curvature or semantically salient regions. Empirically, Chamfer Distance and view-consistency improve sub-linearly with proxy count (Wang et al., 16 Jul 2025).
• Speed and interactivity: Mesh simplification and point/voxel proxying, along with algorithmic optimizations (binary search for grid sizes, per-tuple sparse attention, k-NN for feature fusion), enable interactive preview and editing capable of sub-second response times, even in generative settings (Coin3D, SP²T) (Dong et al., 2024, Wan et al., 2024).
• Adaptation to multiple modalities: Pipelines support raw image, point cloud, mesh, or semantic label inputs; proxies are robust to source modality and favor integration with neural modules and haptic devices alike (Zhu et al., 17 Dec 2025, Kumari et al., 2020).
• Limitations: Purely proxy-based pipelines may require careful feature learning (e.g., explicit positional encoding, Laplacian regularization) to match the appearance fidelity of high-capacity neural fields, especially in real-time or resource-limited scenarios (Zhu et al., 17 Dec 2025, Wang et al., 16 Jul 2025).

6. Comparative Table of Key Approaches

Method	Proxy Structure	Application Domain
3DProxyImg (Zhu et al., 17 Dec 2025)	Sparse triangle mesh + per-vertex implicit features	Single-image animation, interactive editing
Coin3D (Dong et al., 2024)	User-constructed shape primitives, voxelized grid	Generative 3D asset design, local editing
HPR3D (Wang et al., 16 Jul 2025)	Multilevel tree of proxies on mesh/interior	Reconstruction, editing, compact representation
ControlRoom3D (Schult et al., 2023)	Semantic bounding boxes ("proxy room")	Scene layout, room synthesis
Proxy3D (Jiang et al., 8 May 2026)	Clustered patch-level semantic/geometric proxies	Vision-language modeling, spatial reasoning
SP²T (Wan et al., 2024)	Uniform 3D grid of proxies, vertex-based sampling	Point cloud understanding, transformers
UniMGS (Xiao et al., 27 Jan 2026)	Mesh faces as anchor for Gaussian splats	Unified mesh + neural field rendering
Shape-for-Motion (Liu et al., 27 Jun 2025)	Time-consistent mesh proxy from video	Video editing with 3D consistency

The diversity of proxy construction techniques demonstrates their central role in bridging explicit 3D geometry with high-dimensional neural feature spaces, delivering controllability, interpretability, and efficiency in modern 3D vision and graphics systems.