Skin-PAViT: Adaptive Vision for Skin Analysis

• The paper introduces Skin-PAViT, which integrates skin texture and positional priors into a frozen ViT backbone for remote skin assessment.
• It employs frequency filtering with band-pass techniques and symmetric contrastive regularization to refine predictions for hydration and TEWL metrics.
• Results demonstrate improved MAE and R² compared to standard CNN and transformer baselines, enabling precise patch-level estimations and full-face heatmap reconstructions.

Searching arXiv for the specified papers and related skin-domain Vision Transformer work. Skin-Prior Adaptive Vision Transformer, or Skin-PAViT, is a Vision Transformer-based regression model for remote facial skin assessment from smartphone-captured selfie images. It was introduced for estimating two quantitative indicators of skin barrier function—skin hydration (SH) and trans-epidermal water loss (TEWL)—from facial patches, and for reconstructing full-face heatmaps of estimated skin condition (Soh et al., 8 Sep 2025). The defining characteristic of the method is that it does not treat the task as generic image regression. Instead, it uses a frozen ImageNet-pretrained ViT-B backbone augmented with trainable modules that inject skin-specific texture priors and facial position priors, together with a symmetric-based contrastive regularization designed to reduce bias induced by annotation imbalance (Soh et al., 8 Sep 2025).

1. Problem setting and conceptual basis

Skin-PAViT is motivated by the observation that SH and TEWL are clinically meaningful indicators of skin barrier function, yet they are ordinarily measured using dedicated instruments such as the Corneometer for SH and the VapoMeter for TEWL (Soh et al., 8 Sep 2025). In the reported formulation, the objective is to estimate these quantities remotely from selfie facial images and to visualize the result as a face heatmap, thereby shifting the task from point-based measurement to image-based dense estimation (Soh et al., 8 Sep 2025).

The method is premised on four limitations of a standard ViT regression model for this domain. First, skin assessment depends on subtle local texture, including wrinkles, pores, roughness, dryness, and redness. Second, the SH/TEWL label distribution is imbalanced, with skew toward common ranges. Third, the same facial patch appearance can depend on its position on the face, so the problem is better modeled as $\hat{y} = f(x,d)$, where $x$ is the patch and $d$ is the facial region ID. Fourth, the dataset is relatively small, making full fine-tuning of ViT susceptible to overfitting and underfitting issues (Soh et al., 8 Sep 2025).

Within that framing, Skin-PAViT can be understood as a skin-prior and adaptive transformer because it inserts task-specific prompt-like modules into a frozen backbone rather than retraining the entire model. This suggests that adaptation is achieved by constraining where trainable capacity is introduced: local texture is handled by texture-oriented modules, and facial-region dependence is handled by explicit position conditioning (Soh et al., 8 Sep 2025).

2. Data acquisition, annotation, and patch construction

The system in which Skin-PAViT is embedded comprises four stages: data collection, anchor estimation and patch cropping, SH/TEWL prediction with Skin-PAViT, and heatmap visualization (Soh et al., 8 Sep 2025). The data were collected from 336 Chinese panelists using two image modalities: selfie images captured with an iPhone XR and VISIA facial images captured by VISIA-CR. Images were taken from front and side views and under different lighting conditions. For selfie images these conditions were natural, white, and yellow light, whereas VISIA images were collected in controlled modes such as Standard 2 and Cross-polarized (Soh et al., 8 Sep 2025).

Ground-truth physiological measurements were obtained using Corneometer CM825 for SH and VapoMeter SWL5001JT for TEWL. A set of 37 facial anchor points was defined per face. Sticker markers were placed to help localize measurement points on sticker-labeled images, and measurements were collected after a standardized facial washing and waiting protocol: wash face, wait 20 minutes, capture facial image, and measure SH/TEWL at point locations (Soh et al., 8 Sep 2025).

Because the labels are tied to specific facial locations, the pipeline first localizes corresponding regions on non-sticker images. A pretrained DAD-3DHeads detector extracts 68 facial landmarks,

$$\mathcal{P} = \{\boldsymbol{p}_j \in \mathbb{R}^2 \mid j=1,\dots,68\}.$$

For sticker-labeled images, sticker centroids are computed via color segmentation. A PointNet model $h$ is then trained to map facial landmarks to anchor centroids: $\mathcal{C}_i = h(\mathcal{P}_i),$ where

$$\mathcal{C}_i = \{\boldsymbol{c}_j \in \mathbb{R}^2 \mid j=1,\dots,M\}.$$

From each estimated centroid, a square skin patch is cropped: $$\boldsymbol{x}_{i,d_j} = \boldsymbol{I}_i[\boldsymbol{c}_{d_j,0}-r:\boldsymbol{c}_{d_j,0}+r,\; \boldsymbol{c}_{d_j,1}-r:\boldsymbol{c}_{d_j,1}+r].$$ Patch size is standardized using sticker size: for Selfie, $r=70$ yielding $140 \times 140$ patches; for VISIA, $x$0 yielding $x$1 patches (Soh et al., 8 Sep 2025).

The resulting dataset is defined as

$x$2

where $x$3 is the facial image, $x$4 the anchor IDs, $x$5 the anchor coordinates, and $x$6 the SH/TEWL labels (Soh et al., 8 Sep 2025). Subjects were split by skin tone distribution, measured by ITA°, into 235 training and 101 testing panelists. Each panelist contributed 9 selfie facial images and 6 VISIA facial images, producing 40,185 training and 17,271 testing selfie patches, and 26,790 training and 11,514 testing VISIA patches (Soh et al., 8 Sep 2025).

3. Architectural design of Skin-PAViT

Skin-PAViT is built on an ImageNet-pretrained ViT-B backbone that remains frozen during training, while lightweight trainable modules are inserted for task adaptation (Soh et al., 8 Sep 2025). The architecture comprises a Texture Adaptive Module (TAM), Position Adapters (PAs), the frozen ViT backbone, and a regression head (Soh et al., 8 Sep 2025). This is described as a PEFT-style design, but the defining adaptation is specifically tailored to regression on skin patches and explicitly encodes local skin texture, frequency-highlighted details, and facial location (Soh et al., 8 Sep 2025).

The Texture Adaptive Module contains two components. The Prior Texture Module (PTM) is a small CNN intended to capture local spatial context and fine texture. It uses successive convolution and pooling layers with channel progression 48, 96, 192, 384, and 768. Its output feature map is reduced to $x$7, then flattened to $x$8 (Soh et al., 8 Sep 2025). Texture Adapters (TAs) are MLPs that transform this output into ViT-compatible token dimension: $x$9 These 49 tokens are used as prompts and are prepended to the ViT input sequence for each transformer layer (Soh et al., 8 Sep 2025).

Before the patch is sent to PTM, the method applies a band-pass filter in the frequency domain to highlight textures such as wrinkles and pores. For an image $d$0, the transformation is

$d$1

followed by a binary mask

$d$2

and filtered texture

$d$3

The filtered texture $d$4 is concatenated with the original input and passed to PTM (Soh et al., 8 Sep 2025). The reported ablations state that low-pass overly blurs texture, high-pass introduces too much noise, and band-pass preserves the right amount of skin structure (Soh et al., 8 Sep 2025).

Position Adapters encode facial-region identity explicitly. They take a one-hot encoded anchor ID as input, produce positional prompt token(s) via an MLP, and concatenate these with the texture prompt tokens (Soh et al., 8 Sep 2025). The concatenation yields 50 prompt tokens: 49 from texture and 1 from position encoding/adaptation (Soh et al., 8 Sep 2025). The stated rationale is that cheeks, eyelids, jawline, and nose area have different SH/TEWL statistics, and symmetric regions often have related distributions (Soh et al., 8 Sep 2025).

The ViT backbone itself remains a standard ImageNet-pretrained transformer using multi-head self-attention. However, unlike a vanilla ViT regression model, each layer receives additional prompt tokens from TAM and PAs, which direct attention toward skin-specific cues. The output is then passed through a standard regression head to predict a scalar SH or TEWL value, with the target normalized to $d$5 during training (Soh et al., 8 Sep 2025).

4. Learning objective, imbalance handling, and training protocol

The optimization objective combines MSE regression loss with a symmetric-based contrastive loss (Soh et al., 8 Sep 2025). The contrastive term is defined on a latent feature $d$6 and its symmetric counterpart $d$7: $d$8 Here, $d$9 denotes similarity in feature space, $$\mathcal{P} = \{\boldsymbol{p}_j \in \mathbb{R}^2 \mid j=1,\dots,68\}.$$0 is a temperature parameter, and $$\mathcal{P} = \{\boldsymbol{p}_j \in \mathbb{R}^2 \mid j=1,\dots,68\}.$$1 is the set of latent features (Soh et al., 8 Sep 2025). The purpose of this term is to encourage symmetric regions from the same subject to have similar latent representations, reflecting the observation that symmetric facial regions tend to share similar SH/TEWL distributions (Soh et al., 8 Sep 2025).

The total objective is

$$\mathcal{P} = \{\boldsymbol{p}_j \in \mathbb{R}^2 \mid j=1,\dots,68\}.$$2

with

$$\mathcal{P} = \{\boldsymbol{p}_j \in \mathbb{R}^2 \mid j=1,\dots,68\}.$$3

Within the reported interpretation, MSE provides standard regression supervision, while the contrastive component combats imbalance by aligning symmetric latent features and discouraging bias toward dominant label ranges (Soh et al., 8 Sep 2025).

Training also uses standard augmentation—horizontal flip, vertical flip, rotation, random erasing, and random cropping—applied with probability 0.5 (Soh et al., 8 Sep 2025). Because selfie images vary in illumination, the method further introduces lighting augmentation on color saturation, contrast, brightness, and sharpness through blending of original and degraded images: $$\mathcal{P} = \{\boldsymbol{p}_j \in \mathbb{R}^2 \mid j=1,\dots,68\}.$$4 where $$\mathcal{P} = \{\boldsymbol{p}_j \in \mathbb{R}^2 \mid j=1,\dots,68\}.$$5 is the original image, $$\mathcal{P} = \{\boldsymbol{p}_j \in \mathbb{R}^2 \mid j=1,\dots,68\}.$$6 the degraded image, and $$\mathcal{P} = \{\boldsymbol{p}_j \in \mathbb{R}^2 \mid j=1,\dots,68\}.$$7 the magnitude (Soh et al., 8 Sep 2025).

The implementation uses PyTorch on an NVIDIA A5000 GPU with Adam, learning rate $$\mathcal{P} = \{\boldsymbol{p}_j \in \mathbb{R}^2 \mid j=1,\dots,68\}.$$8, CosineAnnealingLR, 50 epochs, batch size 16, and labels scaled to $$\mathcal{P} = \{\boldsymbol{p}_j \in \mathbb{R}^2 \mid j=1,\dots,68\}.$$9 (Soh et al., 8 Sep 2025). A plausible implication is that the design aims to keep the backbone stable while concentrating optimization on the newly inserted skin-specific modules.

5. Empirical performance and ablation structure

Evaluation is reported using MAE and $h$0, with MAE also broken down by many-shot, medium-shot, and few-shot regimes because of the imbalanced label distribution (Soh et al., 8 Sep 2025). The anchor estimation stage achieved average error rate 0.77 on Selfie and 0.78 on VISIA, where

$h$1

with $h$2 denoting sticker centroid, $h$3 predicted anchor, and $h$4 the sticker radius. Values below 1 indicate that the predicted centroid falls within the sticker radius and is therefore sufficient for cropping (Soh et al., 8 Sep 2025).

The main regression results are summarized below.

Modality	Target	MAE (All)	$h$5
Selfie	TEWL	2.28	0.264
Selfie	SH	8.97	0.205
VISIA	TEWL	2.19	0.348
VISIA	SH	8.83	0.263

The method is reported to outperform CNN and standard transformer baselines, including ViT-B, ResNet-18/50, VGG-16, EfficientNet-B0, ConvNeXt-B, Swin-B, and PVT-M, achieving best or near-best overall MAE and best $h$6, with especially strong improvement on medium-shot and few-shot samples (Soh et al., 8 Sep 2025). The paper interprets this as evidence that the architecture handles the imbalanced label distribution more effectively than generic baselines (Soh et al., 8 Sep 2025).

The ablation study is organized as incremental configurations: A, ViT + TAM; B, A + Frequency Filtering; C, B + PAs; D, C + Data Augmentation; E, D + Symmetric-based Contrastive Learning (Soh et al., 8 Sep 2025). The reported findings are structurally consistent across these additions. TAM alone is comparable to fine-tuned ViT while using fewer trainable parameters. Frequency filtering improves texture representation, with band-pass performing best and improving $h$7 for both SH and TEWL. Position Adapters produce a significant improvement, consistent with facial-region-specific SH/TEWL distributions. Data augmentation improves generalization and reduces MAE for many medium/few-shot samples. The final addition of symmetric contrastive learning yields the best-performing configuration, reducing few-shot and medium-shot error further and producing the strongest overall $h$8 (Soh et al., 8 Sep 2025).

6. Visualization, interpretation, and relation to adjacent skin-domain transformers

After patch-wise prediction, Skin-PAViT reconstructs full-face heatmaps (Soh et al., 8 Sep 2025). For SH, the visualization range is 0–90 with midpoint 45: below 40 is shown as red and interpreted as dry, 40–50 as white and normal, and above 50 as blue and well-hydrated. For TEWL, the range is 0–30 with midpoint 15: below 15 is blue and interpreted as healthy barrier, whereas above 15 is red and interpreted as compromised barrier or dry skin. Missing regions are linearly interpolated (Soh et al., 8 Sep 2025). The reported qualitative outcome is that Skin-PAViT produces heatmaps closer to ground truth than a fully fine-tuned ViT, especially for subjects whose values are far from the average distribution (Soh et al., 8 Sep 2025).

Interpretability in Skin-PAViT is not formulated in the same way as in diagnostic classifiers that explicitly constrain feature selection, but it is still structured rather than purely post hoc. The model makes spatially localized predictions tied to anchor-defined facial regions, it conditions predictions on location, and it externalizes output as region-wise SH/TEWL maps (Soh et al., 8 Sep 2025). This suggests that its interpretive axis is anatomical and physiological rather than class-discriminative.

A useful comparison within skin-domain transformer design is the contemporaneous "IViT: A Novel Interpretable Visual Transformer for Skin Disease Detection" (Li et al., 22 Jun 2026). IViT addresses multi-class skin disease recognition rather than SH/TEWL regression, and it builds interpretability through discrete QP feature selection, class-feature assignment, and activation localization (Li et al., 22 Jun 2026). By contrast, Skin-PAViT addresses remote regression from facial patches and encodes inductive structure through skin-texture prompts, position prompts, and symmetry-based regularization (Soh et al., 8 Sep 2025). The comparison clarifies that “adaptive” and “skin-prior” in Skin-PAViT refer primarily to domain-conditioned representation learning for regression, not to QP-constrained intrinsic interpretability of the kind used in IViT (Li et al., 22 Jun 2026).

A common misconception is that Skin-PAViT is simply a standard ViT with light prompt tuning. The reported design is narrower and more domain-specific: TAM introduces a CNN-based local texture prior, band-pass frequency filtering highlights wrinkles and pores before prompt construction, and PAs encode anchor identity explicitly (Soh et al., 8 Sep 2025). Another misconception is that the model performs full-face dense prediction directly from an image. In the reported pipeline, prediction is patch-based, indexed by anchor locations, and the full-face map is reconstructed afterward with interpolation (Soh et al., 8 Sep 2025).

7. Limitations, deployment considerations, and research significance

The reported limitations are explicit. First, the selfie domain is harder than VISIA because of uncontrolled lighting, face size variation, and pose variation (Soh et al., 8 Sep 2025). Second, the selfie dataset covers only three lighting conditions, so additional uncontrolled real-world data would improve robustness (Soh et al., 8 Sep 2025). Third, SH is harder to predict than TEWL because SH has weaker correlation with skin texture, whereas the method focuses primarily on texture and may therefore miss cues related to skin tone and radiance (Soh et al., 8 Sep 2025). Fourth, the authors suggest multimodal extension, including non-RGB modalities such as infrared and multimodal fusion (Soh et al., 8 Sep 2025). Fifth, the dataset is not publicly released because it contains PII, and the images are masked in the paper (Soh et al., 8 Sep 2025).

From a deployment perspective, the method is framed as suitable for smartphone-based skin assessment, consumer self-monitoring, full-face region-wise visualization, and accessible AI-driven skincare analysis (Soh et al., 8 Sep 2025). It can operate from a single view even though training used three views per subject (Soh et al., 8 Sep 2025). These claims should not be conflated with clinical equivalence to direct instrumentation: the system estimates SH and TEWL from images, whereas the ground truth is measured by dedicated devices (Soh et al., 8 Sep 2025).

In summary, Skin-PAViT designates a frozen ImageNet-pretrained ViT regression framework augmented with a skin-texture prompt pipeline and facial position prompts, optimized with MSE plus symmetry-based contrastive regularization, for patch-level estimation of SH and TEWL and reconstruction of full-face heatmaps from facial images (Soh et al., 8 Sep 2025). Its significance lies in the way it reformulates remote skin assessment as a structured vision problem in which local texture, facial topology, and label imbalance are treated as first-class modeling constraints rather than incidental nuisances (Soh et al., 8 Sep 2025).