MSMA: Multi-Scale Fusion for 3D Face Reconstruction

• The paper introduces a robust MSMA architecture that fuses hierarchical CNN features and assigns 3DMM parameters to specific feature scales for improved single-image face reconstruction.
• It demonstrates competitive performance with a point-to-plane RMSE of 1.64 mm on MICC Florence and a 1.77 mm error on FaceWarehouse, validating its effectiveness under varied conditions.
• The method employs adaptive multi-scale fusion and MLKA-based attention to capture both global shape and fine texture details, enhancing robustness against pose, occlusions, and illumination changes.

Searching arXiv for the named papers and closely related work to ground the article. Multi-Scale Feature Fusion with Multi-Attribute (MSMA) denotes a weakly supervised framework for 3D face reconstruction from unconstrained single images that combines multi-resolution feature aggregation with attribute-specific prediction of 3D Morphable Model (3DMM) parameters. In the formulation reported for unconstrained face reconstruction, MSMA is designed for images captured “in the wild” under pose variation, occlusions, illumination changes, expression changes, grayscale inputs, and low resolution, and it seeks to estimate identity / shape, expression, texture, pose, and illumination from a single 2D image by fusing hierarchical CNN features and routing different facial attributes to the scales most relevant to them (Cao, 15 Sep 2025). In a broader methodological sense, the term also aligns with a line of work in which feature fusion is treated as an adaptive, scale-aware, and attribute-sensitive process rather than a fixed aggregation operator, as exemplified by attentional fusion mechanisms and joint multi-scale, multi-attribute retrieval models (Dai et al., 2020, Bao et al., 2022).

1. Conceptual definition and problem scope

In the reported 3D face reconstruction setting, MSMA addresses single-image reconstruction under unconstrained conditions. The target image may exhibit pose variation, occlusions, illumination changes, expression changes, grayscale inputs, and low resolution. The central motivation is that existing methods often use a single feature representation to predict all facial attributes, whereas different attributes depend on different levels of visual information: identity / expression benefit from global semantic structure, texture / illumination / translation need fine-grained local detail, and pose requires a mix of global and local cues (Cao, 15 Sep 2025).

This motivates two coupled design choices. First, multi-scale feature fusion is used to preserve both coarse structure and local detail. Second, multi-attribute learning is used so that each facial parameter is predicted from the scale most relevant to it. In the specific MSMA architecture, these ideas are operationalized through a Multi-Scale Fusion (MSF) module and an MLK Attention-Based Multi-Attribute (MAMA) module.

A common misconception is to treat “multi-attribute” here as generic semantic attribute classification. In the MSMA face-reconstruction formulation, the attributes are the parameter groups of a standard 3DMM pipeline: identity / shape, expression, texture, pose, and illumination. A plausible implication is that “attribute” in this context refers to reconstruction-relevant latent factors rather than only human-interpretable labels such as age or gender.

2. Representational basis in 3DMM and scale-specific prediction

MSMA uses the standard 3D Morphable Model formulation, in which a face is represented by low-dimensional parameters for identity / shape $\boldsymbol{\alpha}_{id}$, expression $\boldsymbol{\beta}_{exp}$, texture $\boldsymbol{\gamma}_{tex}$, pose $\mathbf{R}, \mathbf{t}$, and illumination $\boldsymbol{\delta}_{lig}$ (Cao, 15 Sep 2025). The shape and texture representations are written as

$$\mathbf{S}(\boldsymbol{\alpha},\boldsymbol{\beta}_{exp})=\overline{\mathbf{S}}+\mathbf{A}_{id}\boldsymbol{\alpha}_{id}+\mathbf{A}_{exp}\boldsymbol{\beta}_{exp}$$

and

$$\mathbf{T}(\boldsymbol{\gamma}_{tex})=\overline{\mathbf{T}}+\mathbf{A}_{tex}\boldsymbol{\gamma}_{tex}.$$

For pose, the formulation uses weak-perspective / perspective projection:

$$\mathbf{V}_{2d} = \mathbf{P_r} * (\mathbf{R} * \mathbf{S} + \mathbf{t}),$$

and for illumination it uses spherical harmonics:

$$\mathbf{T}_{sh} = \mathbf{T}(\boldsymbol{\gamma}_{tex}) \odot \sum_{k=1}^9 \boldsymbol{\delta}_{lig} \Psi_k(\mathbf{n}).$$

The scale-specific assignment of facial parameters is a defining part of the method. After fusion, different resolutions are used for different regressors:

Feature scale	Predicted attributes	Reported role
High-resolution $\mathbf{F}_1$	lighting $\boldsymbol{\beta}_{exp}$0, translation $\boldsymbol{\beta}_{exp}$1	fine spatial detail
Mid-resolution $\boldsymbol{\beta}_{exp}$2	texture $\boldsymbol{\beta}_{exp}$3, pose $\boldsymbol{\beta}_{exp}$4	mixed global-local cues
Low-resolution $\boldsymbol{\beta}_{exp}$5	identity $\boldsymbol{\beta}_{exp}$6, expression $\boldsymbol{\beta}_{exp}$7	global semantic structure

This mapping is written explicitly as

$\boldsymbol{\beta}_{exp}$8

The paper also gives the generalized form

$\boldsymbol{\beta}_{exp}$9

followed by concatenation:

$\boldsymbol{\gamma}_{tex}$0

This design distinguishes MSMA from architectures that infer all facial coefficients from a single pooled representation. A plausible implication is that the framework embeds an inductive bias about the visual scale at which different 3DMM parameters are most recoverable.

3. Multi-scale fusion architecture

The full MSMA pipeline is: Input image $\boldsymbol{\gamma}_{tex}$1 backbone CNN $\boldsymbol{\gamma}_{tex}$2 multi-scale features $\boldsymbol{\gamma}_{tex}$3 MSF fusion $\boldsymbol{\gamma}_{tex}$4 MLK attention refinement $\boldsymbol{\gamma}_{tex}$5 attribute-specific regression heads $\boldsymbol{\gamma}_{tex}$6 3DMM parameters $\boldsymbol{\gamma}_{tex}$7 differentiable rendering $\boldsymbol{\gamma}_{tex}$8 weakly supervised losses (Cao, 15 Sep 2025).

The method builds on a ResNet backbone and keeps intermediate features from multiple stages: $\boldsymbol{\gamma}_{tex}$9 as high-resolution, fine spatial detail; $\mathbf{R}, \mathbf{t}$0 and $\mathbf{R}, \mathbf{t}$1 as intermediate features; and $\mathbf{R}, \mathbf{t}$2 as a low-resolution, high-level semantic feature. The retained fusion feature sizes are reported as $\mathbf{R}, \mathbf{t}$3, $\mathbf{R}, \mathbf{t}$4, and $\mathbf{R}, \mathbf{t}$5.

The Multi-Scale Fusion module aligns features from different resolutions and fuses them by upsampling / downsampling, channel alignment, and element-wise addition:

$\mathbf{R}, \mathbf{t}$6

The alignment rule is

$\mathbf{R}, \mathbf{t}$7

and

$\mathbf{R}, \mathbf{t}$8

Fusion is then performed as

$\mathbf{R}, \mathbf{t}$9

The retained fused intermediate features are $\boldsymbol{\delta}_{lig}$0, $\boldsymbol{\delta}_{lig}$1, and $\boldsymbol{\delta}_{lig}$2, which represent balanced spatial + semantic information, mid-level semantics, and global structure, respectively. This places MSMA within a broader family of methods that treat fusion as an adaptive process over heterogeneous feature sources rather than as blind addition or concatenation. In the attentional feature fusion literature, the corresponding critique is that fixed operators are content-agnostic and cannot adapt weights to the actual object/content present in the image, especially when fused features differ in semantics and scale (Dai et al., 2020).

4. MAMA and large-kernel attention

After MSF, the fused features are refined using MLKA / LKA-style self-attention inside the MLK Attention-Based Multi-Attribute module (Cao, 15 Sep 2025). The role of MLKA is to enlarge the effective receptive field, capture long-range dependencies, refine fused multi-scale features, and improve structural coherence and detail recovery.

The input feature map $\boldsymbol{\delta}_{lig}$3 is split into three parts, $\boldsymbol{\delta}_{lig}$4, $\boldsymbol{\delta}_{lig}$5, and $\boldsymbol{\delta}_{lig}$6, corresponding to kernel sizes $\boldsymbol{\delta}_{lig}$7, $\boldsymbol{\delta}_{lig}$8, and $\boldsymbol{\delta}_{lig}$9. Each branch uses depthwise convolution, dilated convolution, and pointwise $$\mathbf{S}(\boldsymbol{\alpha},\boldsymbol{\beta}_{exp})=\overline{\mathbf{S}}+\mathbf{A}_{id}\boldsymbol{\alpha}_{id}+\mathbf{A}_{exp}\boldsymbol{\beta}_{exp}$$0 convolution:

$$\mathbf{S}(\boldsymbol{\alpha},\boldsymbol{\beta}_{exp})=\overline{\mathbf{S}}+\mathbf{A}_{id}\boldsymbol{\alpha}_{id}+\mathbf{A}_{exp}\boldsymbol{\beta}_{exp}$$1

The outputs are concatenated and fused with the depthwise features:

$$\mathbf{S}(\boldsymbol{\alpha},\boldsymbol{\beta}_{exp})=\overline{\mathbf{S}}+\mathbf{A}_{id}\boldsymbol{\alpha}_{id}+\mathbf{A}_{exp}\boldsymbol{\beta}_{exp}$$2

The refined output is then

$$\mathbf{S}(\boldsymbol{\alpha},\boldsymbol{\beta}_{exp})=\overline{\mathbf{S}}+\mathbf{A}_{id}\boldsymbol{\alpha}_{id}+\mathbf{A}_{exp}\boldsymbol{\beta}_{exp}$$3

where $$\mathbf{S}(\boldsymbol{\alpha},\boldsymbol{\beta}_{exp})=\overline{\mathbf{S}}+\mathbf{A}_{id}\boldsymbol{\alpha}_{id}+\mathbf{A}_{exp}\boldsymbol{\beta}_{exp}$$4 is a $$\mathbf{S}(\boldsymbol{\alpha},\boldsymbol{\beta}_{exp})=\overline{\mathbf{S}}+\mathbf{A}_{id}\boldsymbol{\alpha}_{id}+\mathbf{A}_{exp}\boldsymbol{\beta}_{exp}$$5 convolution and $$\mathbf{S}(\boldsymbol{\alpha},\boldsymbol{\beta}_{exp})=\overline{\mathbf{S}}+\mathbf{A}_{id}\boldsymbol{\alpha}_{id}+\mathbf{A}_{exp}\boldsymbol{\beta}_{exp}$$6 is a learnable parameter.

The architectural logic resembles a general trend in multi-scale fusion research. In fashion retrieval, for example, multi-scale fused representations are refined by context-aware attention extraction and then distributed across global, part-informed, and local detailed branches, with joint attribute recognition providing auxiliary supervision (Bao et al., 2022). This suggests a broader interpretation of MSMA as a design pattern: fused multi-scale features are most effective when subsequent modules preserve branch specificity rather than collapsing all information into a single descriptor.

5. Weakly supervised training objectives

MSMA is trained in a weakly supervised manner using the input image, 2D facial landmarks, a differentiable renderer, reconstruction/image-based losses, and coefficient regularization (Cao, 15 Sep 2025). The total loss is

$$\mathbf{S}(\boldsymbol{\alpha},\boldsymbol{\beta}_{exp})=\overline{\mathbf{S}}+\mathbf{A}_{id}\boldsymbol{\alpha}_{id}+\mathbf{A}_{exp}\boldsymbol{\beta}_{exp}$$7

The photometric loss uses the rendered image $$\mathbf{S}(\boldsymbol{\alpha},\boldsymbol{\beta}_{exp})=\overline{\mathbf{S}}+\mathbf{A}_{id}\boldsymbol{\alpha}_{id}+\mathbf{A}_{exp}\boldsymbol{\beta}_{exp}$$8:

$$\mathbf{S}(\boldsymbol{\alpha},\boldsymbol{\beta}_{exp})=\overline{\mathbf{S}}+\mathbf{A}_{id}\boldsymbol{\alpha}_{id}+\mathbf{A}_{exp}\boldsymbol{\beta}_{exp}$$9

where $$\mathbf{T}(\boldsymbol{\gamma}_{tex})=\overline{\mathbf{T}}+\mathbf{A}_{tex}\boldsymbol{\gamma}_{tex}.$$0 is the reprojected face region and $$\mathbf{T}(\boldsymbol{\gamma}_{tex})=\overline{\mathbf{T}}+\mathbf{A}_{tex}\boldsymbol{\gamma}_{tex}.$$1 is a face mask; the mask helps reduce occlusion effects such as glasses.

The perceptual loss uses ArcFace embeddings:

$$\mathbf{T}(\boldsymbol{\gamma}_{tex})=\overline{\mathbf{T}}+\mathbf{A}_{tex}\boldsymbol{\gamma}_{tex}.$$2

where $$\mathbf{T}(\boldsymbol{\gamma}_{tex})=\overline{\mathbf{T}}+\mathbf{A}_{tex}\boldsymbol{\gamma}_{tex}.$$3 is the ArcFace feature extractor.

The landmark reprojection loss uses 68 detected landmarks:

$$\mathbf{T}(\boldsymbol{\gamma}_{tex})=\overline{\mathbf{T}}+\mathbf{A}_{tex}\boldsymbol{\gamma}_{tex}.$$4

where $$\mathbf{T}(\boldsymbol{\gamma}_{tex})=\overline{\mathbf{T}}+\mathbf{A}_{tex}\boldsymbol{\gamma}_{tex}.$$5 is the detected 2D landmark, $$\mathbf{T}(\boldsymbol{\gamma}_{tex})=\overline{\mathbf{T}}+\mathbf{A}_{tex}\boldsymbol{\gamma}_{tex}.$$6 is the projected 2D landmark from the reconstructed 3D face, $$\mathbf{T}(\boldsymbol{\gamma}_{tex})=\overline{\mathbf{T}}+\mathbf{A}_{tex}\boldsymbol{\gamma}_{tex}.$$7 for inner mouth and $$\mathbf{T}(\boldsymbol{\gamma}_{tex})=\overline{\mathbf{T}}+\mathbf{A}_{tex}\boldsymbol{\gamma}_{tex}.$$8 for others, and $$\mathbf{T}(\boldsymbol{\gamma}_{tex})=\overline{\mathbf{T}}+\mathbf{A}_{tex}\boldsymbol{\gamma}_{tex}.$$9.

The 3DMM coefficient regularization term is

$$\mathbf{V}_{2d} = \mathbf{P_r} * (\mathbf{R} * \mathbf{S} + \mathbf{t}),$$0

with $$\mathbf{V}_{2d} = \mathbf{P_r} * (\mathbf{R} * \mathbf{S} + \mathbf{t}),$$1, $$\mathbf{V}_{2d} = \mathbf{P_r} * (\mathbf{R} * \mathbf{S} + \mathbf{t}),$$2, and $$\mathbf{V}_{2d} = \mathbf{P_r} * (\mathbf{R} * \mathbf{S} + \mathbf{t}),$$3.

The reflectance loss is

$$\mathbf{V}_{2d} = \mathbf{P_r} * (\mathbf{R} * \mathbf{S} + \mathbf{t}),$$4

where

$$\mathbf{V}_{2d} = \mathbf{P_r} * (\mathbf{R} * \mathbf{S} + \mathbf{t}),$$5

and $$\mathbf{V}_{2d} = \mathbf{P_r} * (\mathbf{R} * \mathbf{S} + \mathbf{t}),$$6 is the binary face mask. The combined regularization term is

$$\mathbf{V}_{2d} = \mathbf{P_r} * (\mathbf{R} * \mathbf{S} + \mathbf{t}),$$7

with $$\mathbf{V}_{2d} = \mathbf{P_r} * (\mathbf{R} * \mathbf{S} + \mathbf{t}),$$8 and $$\mathbf{V}_{2d} = \mathbf{P_r} * (\mathbf{R} * \mathbf{S} + \mathbf{t}),$$9.

The training setup uses PyTorch, Nvdiffrast, Adam, a learning rate of $$\mathbf{T}_{sh} = \mathbf{T}(\boldsymbol{\gamma}_{tex}) \odot \sum_{k=1}^9 \boldsymbol{\delta}_{lig} \Psi_k(\mathbf{n}).$$0, batch size $$\mathbf{T}_{sh} = \mathbf{T}(\boldsymbol{\gamma}_{tex}) \odot \sum_{k=1}^9 \boldsymbol{\delta}_{lig} \Psi_k(\mathbf{n}).$$1, learning-rate decay by a factor of $$\mathbf{T}_{sh} = \mathbf{T}(\boldsymbol{\gamma}_{tex}) \odot \sum_{k=1}^9 \boldsymbol{\delta}_{lig} \Psi_k(\mathbf{n}).$$2 every 10 epochs, input size $$\mathbf{T}_{sh} = \mathbf{T}(\boldsymbol{\gamma}_{tex}) \odot \sum_{k=1}^9 \boldsymbol{\delta}_{lig} \Psi_k(\mathbf{n}).$$3, and NVIDIA Tesla V100 hardware. Training data are derived from LFW and VGGFace2; after filtering and landmark detection, about 50,000 single-face images are used.

6. Empirical performance, relation to prior fusion work, and limitations

The reported evaluation uses MICC Florence and FaceWarehouse, with qualitative testing on MoFA-Test, AFLW2000-3D, and a custom-collected dataset (Cao, 15 Sep 2025). The metrics are point-to-plane RMSE (mm) for MICC Florence and point-to-point RMSE (mm) for FaceWarehouse. Baselines include RingNet, CPEM, Yu et al. / Deep3DFace, Tran et al., 3DDFA-V2, MGCNet, Booth et al., MS-SFN, PRNet, and FML.

Dataset / setting	Reported MSMA result	Context in the paper
MICC Florence, Cooperative	$$\mathbf{T}_{sh} = \mathbf{T}(\boldsymbol{\gamma}_{tex}) \odot \sum_{k=1}^9 \boldsymbol{\delta}_{lig} \Psi_k(\mathbf{n}).$$4	competitive with or better than most baselines
MICC Florence, Indoor	$$\mathbf{T}_{sh} = \mathbf{T}(\boldsymbol{\gamma}_{tex}) \odot \sum_{k=1}^9 \boldsymbol{\delta}_{lig} \Psi_k(\mathbf{n}).$$5	close to 3DDFA-V2
MICC Florence, Outdoor	$$\mathbf{T}_{sh} = \mathbf{T}(\boldsymbol{\gamma}_{tex}) \odot \sum_{k=1}^9 \boldsymbol{\delta}_{lig} \Psi_k(\mathbf{n}).$$6	outperforms all methods except Booth et al.
FaceWarehouse	$$\mathbf{T}_{sh} = \mathbf{T}(\boldsymbol{\gamma}_{tex}) \odot \sum_{k=1}^9 \boldsymbol{\delta}_{lig} \Psi_k(\mathbf{n}).$$7 mm	best reported among compared methods

On MICC Florence, the paper notes that 3DDFA-V2 is slightly better on cooperative and indoor, with $$\mathbf{T}_{sh} = \mathbf{T}(\boldsymbol{\gamma}_{tex}) \odot \sum_{k=1}^9 \boldsymbol{\delta}_{lig} \Psi_k(\mathbf{n}).$$8 and $$\mathbf{T}_{sh} = \mathbf{T}(\boldsymbol{\gamma}_{tex}) \odot \sum_{k=1}^9 \boldsymbol{\delta}_{lig} \Psi_k(\mathbf{n}).$$9, but emphasizes that MSMA is very close and slightly better on outdoor than several methods. On FaceWarehouse, the mean geometric reconstruction error for MSMA is $\mathbf{F}_1$0 mm, compared with $\mathbf{F}_1$1 mm for FML.

Qualitatively, the paper reports improvements in reconstruction of open mouths, preservation of jawline / cheekbones, handling of head pose variation, recovery of identity-related facial details, texture realism under lighting changes, robustness to occlusion including hair, sunglasses, and bangs, and expression fidelity. It repeatedly emphasizes preservation of facial symmetry, mouth geometry, skin tone realism, wrinkle and fold detail, and consistency under grayscale / stylized / low-light inputs. Ablation on CelebA suggests that removing MSF and multi-attribute learning causes poor handling of occluded eyes, oversmoothed cheek textures under uneven illumination, and flattened smiling expressions with missing nasolabial folds.

Within the broader literature, MSMA can be understood as part of a methodological progression. Attentional Feature Fusion argues that feature fusion should be dynamic, selective, and scale-aware rather than fixed and linear, and its Multi-Scale Channel Attention Module combines local and global context inside attention (Dai et al., 2020). MMFL-Net, in turn, combines bidirectional multi-scale semantic-spatial fusion with multi-task attribute recognition and product ID supervision for cross-domain fashion retrieval (Bao et al., 2022). MSMA extends this general logic into 3D face reconstruction by tying feature scale directly to 3DMM attribute groups and by using MLKA to refine fused representations before regression.

The limitations described in the discussion are correspondingly specific. The method remains challenged by very extreme occlusion, severe pose / illumination combinations, dependence on weak supervision and face detection quality, and potential residual errors in very fine texture or heavily stylized inputs. The paper also notes that it is not always best in every metric on every dataset; on MICC Florence outdoor, some supervised/optimization-heavy methods are slightly stronger. This indicates that MSMA’s principal contribution is not universal dominance across all settings, but a structured demonstration that multi-scale fusion and multi-attribute prediction can improve robustness and detail in weakly supervised single-image 3D face reconstruction.