Supervision-by-Hallucination-and-Transfer (SHT)

• SHT is a weakly-supervised framework that integrates a Dual Hallucination Learning Network (DHLN) and a Facial Pose Transfer Network (FPTN) to address low-resolution facial landmark detection.
• The DHLN jointly enhances face super-resolution and landmark heatmap regression by fusing pose-aware features through a multi-stream architecture.
• The framework leverages both labeled low-resolution data and unlabeled high-resolution images, achieving improved landmark precision and robustness across various poses and occlusions.

Supervision-by-Hallucination-and-Transfer (SHT) is a weakly-supervised learning framework designed to robustly and precisely detect facial landmarks in low-resolution images. SHT integrates two mutually enhanced modules: the Dual Hallucination Learning Network (DHLN), which fuses face hallucination and landmark heatmap regression, and the Facial Pose Transfer Network (FPTN), which refines both hallucinated faces and heatmaps by transferring pose cues—all without requiring additional manual landmark annotations on large high-resolution datasets. This design addresses the challenges of limited image resolution, scarce high-precision labels, and the cross-dependence of hallucination and localization tasks (Wan et al., 19 Jan 2026).

1. Motivation and Conceptual Foundations

Facial landmark detection (FLD) in unconstrained, low-resolution scenarios is heavily limited by the loss of high-frequency spatial detail caused by pooling, strided convolution, and low-quality imaging (e.g., 16×16 or 64×64 faces upsampled to 128×128). High-resolution annotations are both expensive and prone to imprecision, while traditional face hallucination (super-resolution) techniques are typically decoupled from the FLD task. SHT is conceived as a solution that embeds hallucination into the FLD backbone, enforces mutual benefits between hallucination and localization, and leverages large unlabeled high-resolution face collections for additional weak supervision.

Key contributions include:

• The Dual Hallucination Learning Network (DHLN), which tightly couples a hallucination stream and a landmark heatmap stream for mutual enhancement.
• The Facial Pose Transfer Network (FPTN), which uses the outputs of DHLN (hallucinated faces and predicted heatmaps) as pseudo-ground-truth for cross-pose transfer, facilitating further refinement.
• Utilization of both labeled, low-resolution datasets and unlabeled, high-resolution data without extra landmark label requirements, enabling improved generalization and annotation-efficient training.

2. Dual Hallucination Learning Network (DHLN)

DHLN comprises two parallel but interacting streams: one for face hallucination (super-resolution) and one for landmark heatmap estimation.

2.1 Input and Output Representations

• Input: $I^{LR} \in \mathbb{R}^{H \times W \times 3}$ (e.g., $64 \times 64 \times 3$).
• Ground truth (for labeled data): $I^{HR} \in \mathbb{R}^{sH \times sW \times 3}$, $H^* \in \mathbb{R}^{h \times w \times K}$ (heatmaps for $K$ landmarks).
• Outputs: $I^{SR}$ (super-resolved face), $H$ (predicted landmark heatmaps).

2.2 Face Hallucination Stream

This stream consists of four SRBlocks that incrementally increase spatial detail. A critical fusion step employs pose attention: each Hourglass block in the heatmap stream produces a feature map $P_t$; a residual branch ($RB(P_t)$) is processed via a sigmoid to create a pose attention map. This is fused with the current SRBlock output $Q_t$ as:

$Q_t' = Q_t + Q_t \odot \sigma(RB(P_t))$

where $\odot$ is element-wise multiplication and $\sigma$ is the sigmoid function, enabling high-resolution details focused on facial structure.

2.3 Landmark Heatmap Hallucination Stream

Backbone architecture is a four-stage Stacked Hourglass Network (SHN). At each stage $t$, feature $P_t$ is concatenated with pose-conditioned $Q_t'$ from the hallucination stream and processed via a $1\times1$ convolution:

$$P_t' = P_t + \mathrm{Conv}_{1\times1}\big([P_t \| Q_t']\big)$$

where $\|$ denotes channel concatenation. This design ensures progressive mutual enhancement.

2.4 DHLN Loss Function

For $N$ labeled samples, DHLN is optimized via a weighted sum:

$$L_{DH} = \sum_{i=1}^N \left[ \gamma_1 \| H_i - H_i^* \|_2^2 + \gamma_2 \| I_i^{SR} - I_i^{HR} \|_1 + \gamma_3 \| G(I_i^{SR}) - G(I_i^{HR}) \|_1 \right]$$

where $G(I)$ is the image gradient (using finite differences). Hyper-parameters: $\gamma_1=1$, $\gamma_2=\gamma_3=0.01$ for labeled data. The three loss components target heatmap accuracy, image reconstruction, and edge/contour structure, respectively.

3. Facial Pose Transfer Network (FPTN)

FPTN is a generative network that transfers facial appearance and structure between poses, using only hallucinated faces and predicted heatmaps.

3.1 Architecture and Components

• Inputs: Condition face $I_{con}$, corresponding heatmap $H_{con}$, and target heatmap $H_{tar}$.
• Generator: A PATB backbone that manipulates local spatial patches to match the target pose.
• Discriminators: Two PatchGAN-based networks—$D_A$ for appearance consistency and $D_S$ for shape/landmark consistency.

3.2 Loss Functions

The objective for FPTN includes:

• Adversarial loss:

$$L_{GAN} = \mathbb{E}_{\text{real}}\left[\log\left(D_A(I_{con}\|I_{tar}) \cdot D_S(H_{tar}\|I_{tar})\right)\right] + \mathbb{E}_{\text{fake}}\left[\log\left((1-D_A(I_{con}\|I_{ger})) \cdot (1-D_S(H_{tar}\|I_{ger}))\right)\right]$$

• Pixel-wise L1 loss: $L_{L1} = \| I_{ger} - I_{tar} \|_1$
• Perceptual L1 (on VGG-19 Conv1_2 features): $L_{perL1} = \| \phi(I_{ger}) - \phi(I_{tar}) \|_1$

The combined loss is:

$$L_{PT} = \lambda_1 L_{GAN} + \lambda_2 L_{L1} + \lambda_3 L_{perL1}$$

with $\lambda_1 = 0.05$, $\lambda_2 = \lambda_3 = 0.01$.

4. End-to-End SHT Learning Framework

SHT combines DHLN and FPTN in a unified training strategy utilizing both labeled and unlabeled data.

• Labeled: For labeled pairs $$\{(I_{n,j}, I_{n,k}, I_{n,j}^{HR}, I_{n,k}^{HR}, H_{n,j}^*, H_{n,k}^*)\}$$, DHLN computes $I^{SR}$ and $H$, and FPTN is applied bidirectionally $(j \to k$ and $k \to j)$.
• Unlabeled: For unlabeled pairs, the same pipeline is used, but with heatmap supervision removed in DHLN (i.e., $\gamma_1 = 0$).

The total weakly-supervised loss:

$$L_{SHT} = \sum_{n=1}^N [L_{DH}(n) + L_{PT}(I_{n,j}^{SR}, H_{n,j}, H_{n,k}) + L_{PT}(I_{n,k}^{SR}, H_{n,k}, H_{n,j})] + \sum_{m=1}^M [L'_{DH}(m) + L_{PT}(I_{m,j}^{SR}, H_{m,j}, H_{m,k}) + L_{PT}(I_{m,k}^{SR}, H_{m,k}, H_{m,j})]$$

where $L'_{DH}$ omits heatmap loss for unlabeled data.

5. Training Regimen and Implementation Details

• Batch size: 16 (8 pairs per batch).
• Pretraining: DHLN and FPTN are separately pretrained on labeled FLD datasets using their respective supervised losses.
• Integration and fine-tuning: Pretrained DHLN and FPTN are merged, followed by fine-tuning on labeled data with the full SHT loss.
• Weak supervision: Final optimization leverages large-scale unlabeled collections (such as CelebA and 300VW) with $\gamma_1=0$ for these samples.
• Platform: PyTorch, single RTX 4090 GPU.

6. Empirical Evaluation and Quantitative Performance

SHT is evaluated on benchmark face hallucination and FLD datasets, including CelebA, Helen, 300W, AFLW, and WFLW. Metrics include PSNR, SSIM, NME (various normalizations), AUC (area under CED), and FR (failure rate).

6.1 Face Hallucination

Method	CelebA PSNR↑	CelebA SSIM↑	CelebA NME_wid↓	Helen PSNR↑	Helen SSIM↑	Helen NME_wid↓
SRResNet	25.82	0.7369	–	25.30	0.7297	–
FSRNet	26.48	0.7718	0.1430	25.90	0.7759	0.3723
DIC	27.37	0.7962	0.1320	26.69	0.7933	0.3674
DHLN (ours)	27.84	0.8153	0.1279	27.07	0.8177	0.3218
SHT (ours)	28.14	0.8238	0.1246	27.48	0.8233	0.3171
SHT-v (+300VW)	28.79	0.8377	0.1198	27.95	0.8310	0.3102

6.2 Facial Landmark Detection (300W, NME_io%)

Method	Common↓	Challenging↓	Fullset↓
HR-Net	2.87	5.15	3.32
AWing	2.72	4.52	3.07
SHN (256x256)	3.11	6.23	3.72
DHLN-M (ours)	2.74	4.78	3.14
SHT-M (ours)	2.57	4.23	2.90
SHT-M-i (+CelebA)	2.46	4.07	2.78
SHT-M-v (+300VW)	2.50	4.14	2.82

6.3 AFLW and WFLW Results

On AFLW (NME_diag%, AUC^7_box):

Method	Full↓	Frontal↓	AUC^7_box↑
LUVLi	1.39	1.19	68.0
SCPAN	1.31	1.10	69.8
SHT-M (ours)	1.21	1.06	70.1
SHT-M-i (+CelebA)	1.09	0.96	72.4

On WFLW (NME_io%, AUC^10_io, FR^10_io%):

Method	NME↓	AUC↑	FR↓
AWing	4.36	0.572	2.84
STAR	4.02	0.605	2.32
SHT-M (ours)	4.03	0.605	2.48
SHT-M-i (+CelebA)	3.92	0.621	2.36

Ablation Observations

• Removal of L_grad drops PSNR by ~0.05 dB and increases NME by ~0.002.
• Compared to a single-stream Hallucination approach, DHLN improves (PSNR: +2.1 dB; SSIM: +0.10 on CelebA), and SHT further exceeds DHLN.
• Incorporation of additional unlabeled data (CelebA/300VW) provides consistent (0.1–0.2%) performance gains in landmark precision.

7. Significance and Scalability

SHT establishes a robust paradigm for combining face hallucination and geometric landmark localization in a mutually beneficial, end-to-end weakly-supervised framework, applicable in data-constrained and resolution-challenged contexts. Its design enables the utilization of unlabeled high-resolution image/video corpora for further incremental performance improvements without reliance on costly manual annotations, and demonstrates robustness under severe pose variation, occlusion, and noisy annotations (Wan et al., 19 Jan 2026).