Factorized-Head Decoder (FHD) Overview

• FHD is defined as an architecture that decomposes the decoding map into multiple specialized heads operating on a shared latent representation.
• In point-cloud reconstruction, FHD improves global structural recovery by partitioning outputs into disjoint subsets, thereby reducing overfitting and enhancing metrics like CD and EMD.
• For proactive forensics, FHD decouples watermark extraction into regression and classification heads, enabling joint optimization and robust manipulation detection.

Searching arXiv for the cited papers to ground the article in current sources. arXiv search: (Alonso et al., 25 May 2025) Factorized-Head Decoder (FHD) denotes a decoder architecture in which the decoding map is decomposed into multiple heads that operate on a shared representation but produce distinct outputs or solve distinct sub-tasks. In the literature, the term is used explicitly for a proactive forensics decoder that splits watermark extraction into regression and classification heads (Wu et al., 26 Feb 2026), and it also describes, in functional terms, a point-cloud reconstruction decoder whose output mapping is factorized into parallel heads that each generate a subset of the final point set (Alonso et al., 25 May 2025). Taken together with work on factorized neural layers, multi-head speech decoders, and factorized face rendering, the literature suggests that FHD is best understood as a family of decoder factorizations rather than a single canonical module (Khodak et al., 2021).

1. Defining characteristics

At its most general, an FHD separates a decoder into multiple mappings conditioned on a common upstream representation. The factorization may occur in different ways. In point cloud reconstruction, the decoder is decomposed into $M$ parallel heads $\{\phi_i\}_{i=1}^M$, each receiving the same latent code $z$ and outputting a subset of points, with the final point cloud formed by concatenation (Alonso et al., 25 May 2025). In proactive forensics, a shared convolutional backbone produces a feature $F_{shared}$, which is then split into a landmark regression head and an identifier classification head, yielding $\text{FHD}(I'_{wm}) = (\hat{W}_L,\hat{W}_{ID})$ (Wu et al., 26 Feb 2026).

This shared-backbone-plus-specialized-heads structure gives FHD two recurrent properties across domains. First, it preserves a common latent or feature space, so the heads are not independent models. Second, it assigns each head a restricted decoding responsibility, either by output partitioning or by task decomposition. In the point-cloud setting, this restriction is implemented as a disjoint output indexing over points; in the forensic setting, it is implemented as a distinction between continuous regression and bitwise classification (Alonso et al., 25 May 2025, Wu et al., 26 Feb 2026).

A common misconception is to treat FHD as interchangeable with multi-head attention. The available papers instead place factorization at the decoder stage. The point-cloud paper explicitly describes factorization in the output mapping $D(z)=\bigcup_{i=1}^M D_i(z)$, while the forensic paper factorizes post-backbone decoding into two FC heads. This suggests that FHD is primarily an output-side architectural decomposition, even when it is conceptually related to multi-head attention or mixture-of-experts (Alonso et al., 25 May 2025, Wu et al., 26 Feb 2026).

2. Point-cloud reconstruction formulation

In "Less is More: Efficient Point Cloud Reconstruction via Multi-Head Decoders" (Alonso et al., 25 May 2025), FHD appears as a direct reinterpretation of the proposed multi-head decoder for point cloud autoencoders. The paper’s central claim is that making the decoder deeper is not necessarily beneficial: beyond a modest depth, deeper single-head decoders tend to overfit and generalize worse, especially on metrics reflecting global structure such as EMD, HD, and F1. The proposed alternative is to factorize the decoder into multiple heads that exploit the redundancy of point clouds.

The architectural pattern is explicit. A standard encoder produces a single latent vector $z$. A single-head decoder implements

$Q = D(z) \in \mathbb{R}^{K \times 3},$

with $K=2048$. The multi-head version replicates the decoder into $M$ parallel heads, all consuming the same $\{\phi_i\}_{i=1}^M$0, but each outputting only $\{\phi_i\}_{i=1}^M$1 points. In the experiments, $\{\phi_i\}_{i=1}^M$2, so each head produces

$\{\phi_i\}_{i=1}^M$3

and the final reconstruction is

$\{\phi_i\}_{i=1}^M$4

The paper describes the heads as responsible for disjoint subsets of output points via fixed indexing, with no learned assignment or gating, and with no weight sharing across heads (Alonso et al., 25 May 2025).

The training objective is also factorized. Training uses Chamfer Distance only, but for $\{\phi_i\}_{i=1}^M$5 heads the loss is the average of the per-head CDs against the full ground-truth shape rather than the CD on the concatenated output. For two heads, each head is therefore individually forced to approximate the entire shape using only 1024 points. The paper states that there is no explicit diversity regularizer or orthogonality term; diversity arises implicitly because each head must cover the full shape under a reduced point budget (Alonso et al., 25 May 2025).

Empirically, the paper reports that multi-head decoders almost always outperform single-head baselines across Light-AE, Deep-AE, and PTv3 encoders on ModelNet40 and ShapeNetPart. The average improvement of multi-head over single-head is reported as +2.73% CD, +22.57% EMD, +1.68% HD, and +1.48% F1 on ModelNet40, and +3.37% CD, +17.14% EMD, +2.42% HD, and +1.23% F1 on ShapeNetPart. Example results include ModelNet40 with Light-AE and a 3-layer decoder, where the single-head model gives CD $\{\phi_i\}_{i=1}^M$6, EMD $\{\phi_i\}_{i=1}^M$7, HD $\{\phi_i\}_{i=1}^M$8, F1 $\{\phi_i\}_{i=1}^M$9, while the 2-head version gives CD $z$0, EMD $z$1, HD $z$2, F1 $z$3 (Alonso et al., 25 May 2025).

The broader significance of this formulation lies in what the paper identifies as the mechanism of improvement. Each head produces fewer points but is trained against the full target, which increases the “semantic density” of points and improves global structural coverage. The paper argues that performance gains are not explained by sheer parameter count, since metric-versus-parameter plots align more closely with depth than with raw parameter number. In this interpretation, FHD regularizes by decomposing one large output mapping into several smaller but independently trained mappings (Alonso et al., 25 May 2025).

3. Unified proactive forensics formulation

In "All in One: Unifying Deepfake Detection, Tampering Localization, and Source Tracing with a Robust Landmark-Identity Watermark" (Wu et al., 26 Feb 2026), FHD is the explicit watermark extraction module of the LIDMark framework. The framework embeds a 152-dimensional landmark-identity watermark

$z$4

where $z$5 contains normalized coordinates of 68 facial landmarks and $z$6 is a 16-dimensional bipolar identifier derived from a SHA-256 hash of the image filename. The decoder must recover this heterogeneous payload from images that may have undergone common distortions or deepfake manipulations (Wu et al., 26 Feb 2026).

The architecture is a multi-task shared-backbone decoder. A convolutional backbone, comprising multiple ConvBlocks, a SEResNetDecoder, and a SEResNet module, extracts a shared latent representation $z$7 from the manipulated watermarked image $z$8. This shared feature is flattened and fed into two parallel FC heads: a landmark regression head that outputs $z$9, and an identifier classification head that outputs the 16-dimensional identifier logits $F_{shared}$0. The paper states that this factorized design is crucial because it decouples landmark regression and identifier classification while allowing joint optimization under a shared feature extractor (Wu et al., 26 Feb 2026).

The associated loss is likewise factorized by task. The landmark component minimizes a mean point-wise Euclidean distance over the 68 normalized landmark pairs, while the identifier component uses

$F_{shared}$1

These are combined as

$F_{shared}$2

with $F_{shared}$3 during pre-training on common distortions and $F_{shared}$4 during fine-tuning on deepfake manipulations. The decoder is trained jointly with the watermark encoder under generator objectives that also include imperceptibility, adversarial, generative-consistency, and stability terms (Wu et al., 26 Feb 2026).

The regression head underlies the framework’s detection and localization mechanism. The decoded landmarks $F_{shared}$5 are treated as intrinsic landmarks, while extrinsic landmarks are obtained by running face-alignment on the same manipulated image. Their discrepancy is measured by Average Euclidean Distance, and the paper reports an AUC of 0.9388 for distinguishing benign from malicious operations, with an optimal threshold $F_{shared}$6 pixels by Youden’s $F_{shared}$7. This same intrinsic-extrinsic consistency check is then applied region-wise for tampering localization (Wu et al., 26 Feb 2026).

The classification head underlies source tracing via Bit Error Rate. On CelebA-HQ under common distortions, the framework reports 0.27% BER at $F_{shared}$8 and 0.10% BER at $F_{shared}$9; under deepfakes, average BER across five models is 3.33% and 2.55% at the same resolutions; on cross-dataset LFW, average BER is 4.06% and 2.76% (Wu et al., 26 Feb 2026). Ablation results further show that disabling either task loss strongly degrades the corresponding output, while joint optimization yields the best landmark accuracy and acceptable BER, supporting the paper’s claim that both the multi-task loss and the factorized-head design are necessary for a good trifunctional balance (Wu et al., 26 Feb 2026).

4. Relation to factorized neural layers and optimization

A different but closely related meaning of factorization is supplied by "Initialization and Regularization of Factorized Neural Layers" (Khodak et al., 2021). That paper does not define FHD as a named decoder module, but it provides the broader mathematical context in which head-specific decoder matrices can themselves be factorized. A layer weight is parameterized as

$\text{FHD}(I'_{wm}) = (\hat{W}_L,\hat{W}_{ID})$0

with low-rank factorization as the special case $\text{FHD}(I'_{wm}) = (\hat{W}_L,\hat{W}_{ID})$1. The paper explicitly notes that such factorized layers occur in compressed model training, knowledge distillation, and multi-head self-attention architectures (Khodak et al., 2021).

This perspective is relevant to FHD in two ways. First, head-specific projections or decoder-output matrices can be treated as factorized matrices. Second, the paper studies how to make such factorizations behave like their well-tuned unfactorized counterparts. It proposes spectral initialization, defined by taking the rank-$\text{FHD}(I'_{wm}) = (\hat{W}_L,\hat{W}_{ID})$2 SVD of an unfactorized weight $\text{FHD}(I'_{wm}) = (\hat{W}_L,\hat{W}_{ID})$3 and setting

$\text{FHD}(I'_{wm}) = (\hat{W}_L,\hat{W}_{ID})$4

so that $\text{FHD}(I'_{wm}) = (\hat{W}_L,\hat{W}_{ID})$5 reconstructs the best rank-$\text{FHD}(I'_{wm}) = (\hat{W}_L,\hat{W}_{ID})$6 approximation to $\text{FHD}(I'_{wm}) = (\hat{W}_L,\hat{W}_{ID})$7 in Frobenius norm (Khodak et al., 2021).

The paper also distinguishes naive weight decay on factors from Frobenius decay on the recomposed product. For a factorized weight $\text{FHD}(I'_{wm}) = (\hat{W}_L,\hat{W}_{ID})$8, standard decay on $\text{FHD}(I'_{wm}) = (\hat{W}_L,\hat{W}_{ID})$9 behaves like nuclear-norm regularization on $D(z)=\bigcup_{i=1}^M D_i(z)$0, whereas Frobenius decay regularizes $D(z)=\bigcup_{i=1}^M D_i(z)$1, which more closely matches the unfactorized objective. The stated principle is to design optimization routines for factorized models that are as close as possible to those of standard layers (Khodak et al., 2021).

The empirical results provide a theoretical backdrop for compressed or low-rank FHD implementations. The paper reports that SI and FD improve low-rank convolutional networks, translation models, and BERT pre-training, and that both schemes applied to multi-head attention lead to improved performance on translation and unsupervised pre-training. This suggests that when an FHD uses factorized head weights rather than only multiple heads, the relevant optimization question is not merely architectural decomposition but also whether initialization and regularization are imposed on the product matrix rather than on the factors themselves (Khodak et al., 2021).

5. Historical antecedents and adjacent architectures

The historical antecedents of FHD include multi-head decoder designs in speech recognition. "Multi-Head Decoder for End-to-End Speech Recognition" (Hayashi et al., 2018) proposes replacing attention-level integration with decoder-level integration. Each attention head receives its own decoder state $D(z)=\bigcup_{i=1}^M D_i(z)$2, computes its own context vector $D(z)=\bigcup_{i=1}^M D_i(z)$3, and updates its own recurrent state using an independent LSTM. The final distribution is then

$D(z)=\bigcup_{i=1}^M D_i(z)$4

This architecture is not named FHD in the paper, but it is a direct instance of decoder factorization across heads (Hayashi et al., 2018).

The paper further introduces heterogeneous multi-head decoding, in which different heads use different attention functions, including dot-product, additive, location-based, and coverage attention. On the Corpus of Spontaneous Japanese, the 4-head heterogeneous configuration HMHD (2×Loc+2×Cov) reports the best CER across all three evaluation tasks: 10.4, 7.7, and 8.9, improving over single-head and conventional multi-head attention baselines. The paper interprets this as evidence that different heads capture different speech or linguistic contexts within a shared encoder-decoder framework (Hayashi et al., 2018).

A distinct adjacent architecture appears in face reenactment. "FSRT: Facial Scene Representation Transformer for Face Reenactment from Factorized Appearance, Head-pose, and Facial Expression Features" (Rochow et al., 2024) uses a transformer decoder that predicts the color of a query pixel from a query representation

$D(z)=\bigcup_{i=1}^M D_i(z)$5

conditioned on a set-latent appearance representation $D(z)=\bigcup_{i=1}^M D_i(z)$6 from source image(s), so that

$D(z)=\bigcup_{i=1}^M D_i(z)$7

The paper explicitly factorizes appearance, head pose, and facial expression into separate carriers, although it does not use the term FHD as a formal module name (Rochow et al., 2024).

These antecedents show that decoder factorization has appeared in at least three distinct modes: parallel recurrent decoders over shared encoder features in ASR, pixel-query rendering from factorized face latents in reenactment, and explicit multi-task heads over a shared convolutional decoder backbone in forensics. A plausible implication is that FHD is better characterized by where factorization is imposed—on outputs, tasks, or latent factors—than by any fixed choice of layer type or modality (Hayashi et al., 2018, Rochow et al., 2024).

6. Limitations, misconceptions, and open directions

The current literature imposes several important constraints on how FHD should be interpreted. First, no paper establishes a single optimal number of heads across settings. The point-cloud study evaluates only $D(z)=\bigcup_{i=1}^M D_i(z)$8 and states that the optimal number of heads likely depends on point cloud density and redundancy; the forensic study fixes two heads because the payload itself is heterogeneous; the speech study uses four heads; FSRT factorizes through latent carriers rather than an explicit count of decoding heads (Alonso et al., 25 May 2025, Wu et al., 26 Feb 2026, Hayashi et al., 2018, Rochow et al., 2024).

Second, FHD does not necessarily imply explicit head specialization mechanisms. In the point-cloud case there is no attention, assignment network, or gating, and specialization is emergent under fixed output indexing and independent parameters. In the forensic case the specialization is explicit because the targets are heterogeneous. This suggests that specialization in FHD may be either architectural and supervised, or implicit and optimization-driven, depending on the task (Alonso et al., 25 May 2025, Wu et al., 26 Feb 2026).

Third, the benefits of factorization are not unconditional. The point-cloud paper notes that observed depth trends can change with details such as batch normalization in the decoder, even though multi-head remains beneficial. The forensic paper identifies attribute-based global edits such as StarGAN-v2 as particularly difficult, reporting BER of 8.47% on CelebA-HQ and 8.96% on LFW at $D(z)=\bigcup_{i=1}^M D_i(z)$9. FSRT reports limitations for out-of-distribution expressions and extreme head turns, and notes that background and hair details may be less sharp than in direct CNN image decoders (Alonso et al., 25 May 2025, Wu et al., 26 Feb 2026, Rochow et al., 2024).

Finally, a common oversimplification is to view FHD as merely a way to add parameters. The point-cloud paper explicitly argues that performance gains are not explained by parameter count alone, while the forensic paper contrasts a single-backbone factorized-head design with dual-decoder baselines and reports a whole-model complexity of 5.815M parameters at $z$0, versus SepMark’s 28.31M and WaveGuard’s 34.81M. The available evidence therefore supports a narrower interpretation: the central effect of FHD is structural decomposition of decoding, not parameter inflation by itself (Alonso et al., 25 May 2025, Wu et al., 26 Feb 2026).

Across these works, the most consistent open directions are systematic scaling of the number of heads, stronger mechanisms for controlling or visualizing specialization, extension to additional tasks such as shape completion, segmentation, and video forensics, and improved robustness to unseen manipulations or generative models. The literature therefore treats FHD not as a completed design pattern but as an extensible decoder principle whose value depends on how factorization is matched to the geometry, task structure, and robustness requirements of the problem domain (Alonso et al., 25 May 2025, Wu et al., 26 Feb 2026).