UP2You: Tuning-Free 3D Human Reconstruction

• UP2You is a tuning-free framework enabling fast, high-fidelity 3D reconstruction of clothed human portraits from unconstrained 2D photos.
• It employs a data rectifier paradigm and a pose-correlated feature aggregation module to standardize multi-view inputs, enhancing both geometric accuracy and texture realism.
• The framework supports advanced applications like arbitrary pose control and multi-garment virtual try-on, achieving efficient 3D avatar synthesis in approximately 1.5 minutes per person.

UP2You is a tuning-free framework designed for fast, high-fidelity 3D reconstruction of clothed human portraits from unconstrained, in-the-wild 2D photo collections. Differing from previous approaches that require controlled captures or plentiful manual intervention, UP2You standardizes highly variable images—diverse in pose, viewpoint, cropping, and occlusion—into clean, orthogonal multi-view representations in a single forward pass. This enables practical, scalable photo-to-3D avatar synthesis, supporting arbitrary pose control and multi-garment virtual try-on. UP2You achieves substantial improvements in geometric accuracy and texture realism while operating in approximately 1.5 minutes per person, with both models and code publicly released to advance research in unconstrained human reconstruction (Cai et al., 29 Sep 2025).

1. Framework Architecture and Data Rectifier Paradigm

UP2You departs from optimization-based text-to-3D methodologies such as DreamBooth and Score Distillation Sampling (SDS), which entail slow, iterative refinements and may rely on precise body templates or aligned multi-view captures. Instead, UP2You introduces a data rectifier paradigm. Raw, cluttered input photos—subject to arbitrary pose, viewpoint, cropping, or partial occlusion—are transformed into a standardized set of orthogonal view images and SMPL-X normal maps (encoding both pose and camera geometry) via a single forward inference pass.

This process rectifies input heterogeneity and enables effective usage of mature multi-view mesh carving and texture baking techniques. The rectified views serve as canonical references for downstream shape and texture reconstruction without necessitating online tuning, facilitating rapid 3D synthesis directly from amateur or casual photo collections.

2. Pose-Correlated Feature Aggregation (PCFA) Module

UP2You’s innovation centers around the pose-correlated feature aggregation (PCFA) module, designed for scalable multi-reference information fusion. For each target pose (represented as a SMPL-X normal map), PCFA computes a set of pixel-wise correlation maps assessing the relevance of every reference image. Feature extraction utilizes backbone networks including ReferenceNet and DINOv2 for detailed appearance encoding.

Correlation between query features (from the pose-conditioned image encoder) and projected reference features is evaluated via transformer-based attention mechanisms. The computation follows two principal steps:

• Attention matrix generation:

$$A^{i} = \frac{Q_i \times (K^{(\text{ref})})^T}{\sqrt{d}}$$

where $Q_i$ are pose-conditioned query features, $K^{(\text{ref})}$ are projected reference features, and $d$ is the feature dimension.

• Final correlation map aggregation:

$$A^{i} = \text{ReLU}(\text{AvgPool}(\text{mean}(A^{i})))$$

Top-$k$ most relevant features are selected (via sorting and interpolation), ensuring only informative references are propagated. Notably, the memory footprint remains nearly constant despite scaling to large numbers of input images, distinguishing PCFA from naïve concatenation approaches.

3. Multi-Reference Shape Prediction and Mesh Reconstruction

A perceiver-based multi-reference shape predictor replaces reliance on pre-captured body templates. Rectified multi-view images and associated SMPL-X normals feed into the shape predictor, enabling direct 3D geometry estimation in the presence of missing regions, collapsed limbs, or misalignments typical of unconstrained input.

The pipeline generates six orthogonal views for mesh carving and texture synthesis. Although increasing the number of reference images improves both geometric and texture fidelity, PCFA ensures a scalable approach without ballooning GPU resource consumption.

Performance metrics demonstrate substantial gains over previous methods (e.g., PuzzleAvatar, AvatarBooth):

• On PuzzleIOI dataset: Chamfer distance ↓15%, P2S error ↓18%.
• On 4D-Dress dataset: PSNR ↑21%, LPIPS ↓46%.

4. Texture Reconstruction, Fidelity, and Limitations

Texture generation leverages the aggregated feature set from PCFA, supporting sharp, consistent, and identity-preserving results. Texture realism is validated via objective metrics (PSNR, LPIPS) and qualitative analyses that highlight improvements in spatial alignment, color accuracy, and detail retention. The rectified pipeline ensures that unconstrained photos are appropriately reprojected to minimize distortions.

A plausible implication is that relying on only six canonical views for texture reconstruction can occasionally introduce misalignment artifacts, particularly on less visible regions (e.g., backside). The authors identify integration of video or dense-view synthesis models as a future solution.

5. Applications: Pose Control and Multi-Garment Try-On

UP2You’s conditioning on SMPL-X normal maps enables arbitrary pose modulation for reconstructed avatars. Users can generate orthogonal-view images of the same identity in novel poses, supporting animation, game character creation, or biomechanical studies.

Multi-garment 3D virtual try-on is supported without additional fine-tuning: upper and lower garments extracted from different photos can be swapped and rendered while maintaining overall subject identity. The disentanglement of pose, shape, and texture within the pipeline affords robust garment transfer across varied input photos.

6. Resource Efficiency and Practical Deployment

UP2You achieves high computational efficiency, completing full avatar reconstruction within approximately 1.5 minutes per person. The nearly constant memory profile enabled by PCFA allows practitioners to scale input reference counts without corresponding increases in GPU consumption.

The pipeline is compatible with real-world deployment settings such as casual photo album digitization or social media-based avatar synthesis, requiring no controlled input, template collection, or prolonged optimization. Open-source release of both models and code aims to encourage rapid adoption and extension in both academic and commercial domains.

7. Future Directions and Open Challenges

Future work will address:

• Reducing dependence on 3D training data, potentially via semi-supervised, unsupervised, or large-scale video-based architectures.
• Refining texture alignment through integration of dense or continuous view synthesis models.
• Streamlining the multi-stage inference pipeline into a single feed-forward architecture, potentially mitigating sequential error accumulation and accelerating generation.
• Expanding the open dataset and benchmarks for unconstrained 3D human reconstruction tasks to promote reproducibility and progress.

In summary, UP2You establishes a new operating point for unconstrained, high-fidelity, tuning-free 3D human reconstruction by combining a novel data rectifier paradigm, pose-correlated feature aggregation, and efficient multi-reference inference (Cai et al., 29 Sep 2025). It supports advanced applications in pose control, garment swapping, and practical avatar synthesis, charting a course for future research in unconstrained human photogrammetry.