Focal-RegionFace: Region-Based Facial Analysis

• Focal-RegionFace is a vision-language model that generates localized, multi-attribute descriptions from user-specified facial regions.
• It leverages progressive fine-tuning with LoRA adapters to integrate spatial visuals and language signals, enhancing region-specific analysis.
• The approach supports applications in dermatology, affective computing, and human–robot interaction by offering interpretable, region-level feedback.

Focal-RegionFace is a vision-LLM architecture designed to generate and recognize fine-grained, multi-attribute natural language descriptions for arbitrarily selected focal regions on human faces. The approach directly addresses the underexplored task of FaceFocalDesc: conditioning on a face image and a user-specified region-of-interest (ROI), the system produces interpretive descriptions that enumerate local facial muscle movements (action units, AUs), express inferred emotional states, and offer age-related skin cues specific to the designated region. This spatially localized and multi-attribute framework advances beyond traditional face captioning by enabling interpretable, region-level analysis relevant for domains such as dermatology, affective computing, and detailed interactive feedback (Zheng et al., 1 Jan 2026).

1. Problem Definition and Motivation

The core objective of FaceFocalDesc is to enable a model, given a face image $I$ and an ROI $B$, to output both (a) a paragraph $D_{ROI}$ describing the local AUs, emotion, and age cues visible in $B$, and (b) categorical predictions $P_{AU}$ (active action units), $P_{Emo}$ (emotion category), and $P_{Age}$ (age bin). Formally, the following mappings are defined:

• $$D_{ROI} = \text{Decoder}(\text{Encoder}_{VL}(I, B))$$
• $$(P_{AU}, P_{Emo}, P_{Age}) = \text{Classifier}(\text{Encoder}_{VL}(I, B), D_{ROI})$$

This problem is underexplored because existing face captioning models predominantly address global facial attributes or generate single-attribute descriptions (e.g., “smiling”) rather than generating detailed, region-focused textual and categorical feedback. In real-world applications—cosmetics, medical dermatology, and human–robot interaction—such localization and attribute disentanglement are critical for interpretable, actionable analysis. Region-level multi-attribute descriptions bridge the gap between opaque black-box classifiers and human-readable, spatially-grounded feedback.

2. MFRF Dataset: Construction and Statistics

Research progress is enabled by the Multimodal Face Region-Focal (MFRF) dataset, assembled from BP4D (AU labels), Aff-Wild2 and RAF-DB (emotion labels), and UTKFace (age annotations), further enriched with custom region-level annotations and human-refined descriptions.

• Images: 10,000 for training and 1,000 for testing, sampled from the four source datasets.
• Regions: Each face is annotated with 12 ROIs, strategically placed to overlap (≥80% IoU) with the face bounding polygon.
• Per-region annotations:
  • Action Units: Transferred from global ground truth if ≥60% of canonical AU area overlaps a given ROI.
  • Emotion: One categorical label from seven classes.
  • Age: Assigned to one of 12 bins ([0–4], [5–9], …, 60+).
• Natural language descriptions: GPT-4o generates initial drafts via staged prompting (Contextual Focus → Region Constraint → Structured Generation), followed by refinement from human experts for clinical and descriptive accuracy.

Split	Images	Regions/image	Total ROI samples	Region labels	Text pairs
Train	10,000	12	120,000	AU/Emo/Age	60,000
Test	1,000	12	12,000	AU/Emo/Age	12,000

The dataset supports comprehensive training and robust, fine-grained evaluation for FaceFocalDesc scenarios.

3. Focal-RegionFace Model and Progressive Fine-Tuning

Focal-RegionFace utilizes Qwen2.5-VL as its vision-language foundation—a two-tower transformer architecture. The model's backbone parameters are frozen; adaptation is achieved by inserting LoRA adapters at critical cross-attention layers, enabling efficient fine-tuning for spatially localized, attribute-specific reasoning.

• Embedding: The vision encoder maps face images $I$ to feature maps $V \in \mathbb{R}^{H \times W \times d_v}$. The ROI feature $z_v = \text{RoIAlign}(V, B)$ captures spatially localized visual content. Tokens $w_{1:t}$ are embedded as $z_t^{(0)}$.
• Cross-modal fusion: Multi-head cross-attention integrates $z_v$ (visual) and $z_t$ (textual) for downstream tasks.

Progressive Fine-Tuning Stages:

1. Global Perception: Multi-task learning of AU (multi-label), emotion (7-way softmax), and age (12-bin softmax) on the whole face. Composite loss $$L_{I} = \lambda_1 L_{AU} + \lambda_2 L_{Emo} + \lambda_3 L_{Age}$$.
2. Region-aware Vision-Language Alignment: Joint input of full-face and ROI; maximization of the log-likelihood for human-authored regional paragraphs $D_{ROI}$.
3. Masked Region-Focal Alignment: Non-ROI regions are grayed out, further refining model attention to the target area under captioning loss.
4. Multi-region Guided Recognition: Multi-region input with per-region caption-based prompts; joint prediction of AU, emotion, and age for each region.

Fine-tuning proceeds in sequence: Stage I → II → III → IV, with each stage targeting distinct aspects of spatial grounding, language generation, and multi-attribute reasoning.

4. Multi-Attribute Description Generation and Interpretability

At inference, for a given $(I,B)$ tuple, the model:

• Extracts the region embedding $z_v$.
• Prefaces the LLM decoder with a task prompt (e.g., “<Task: Describe AU/Emo/Age in this box>”).
• Autoregressively generates $D_{ROI}$, the region-specific paragraph, according to $P(D_{ROI}) = \prod_{t=1}^{T} P(w_t|z_v, w_{<t})$.

Attention visualization demonstrates that cross-attention heads focus tightly within the selected ROI, substantiating the model’s ability for spatially-resolved, interpretable output.

Qualitative examples highlight the model’s descriptive capacity:

• For a box around the left eye: Identification of crow’s‐feet wrinkles (AU6), partial cheek lift, and age-consistent skin texture, inferring "genuine smiling" and an age estimate.
• For a box on the nasolabial fold: Detection of AU12 activation (zygomatic major), skin elasticity, and nuanced emotional valence.

5. Evaluation Metrics

Assessment employs both traditional and innovative region-level metrics.

Traditional Metrics:

• NLP quality: BERTScore precision/recall/F₁, Grammar Issues (GI), Expert Rating (ER)
• Recognition: AU F₁ score, emotion classification accuracy, age classification accuracy

Region-Level MLLM-Based Metrics (evaluated by models such as Gemini-2.5-Pro and GPT-4o):

• Cls: Correctness of AU, emotion, and age recognition within the region
• Det: Richness and granularity of facial detail in text
• Flu: Fluency and coherence of region description
• Box: Degree of relevance to ROI
• Sem: Semantic alignment between the generated text and ground-truth region content
• Win %: Fraction of ROIs where the model outperforms all baselines under the above criteria

Metric	Meaning	Range
Cls	AU/Emotion/Age classification accuracy (per region)	0–100
Det	Facial detail richness	0–100
Flu	Text fluency/coherence	0–100
Box	Task relevance to ROI	0–100
Sem	Text–image semantic alignment	0–100
Win %	ROI-level model supremacy	0–100

6. Experimental Results and Ablations

Closed-Source MLLM Evaluation:

Focal-RegionFace achieves top performance on all five MLLM-judged region-level metrics and Win %, substantially surpassing Qwen2.5-VL, Gemini3, Deepseek-Janus-Pro, and Llama3.2-Vision.

Model	Cls	Det	Flu	Box	Sem	Win %	Rank
Qwen2.5-VL	52.7	47.4	74.5	73.2	51.9	13.5	2
Gemini3	59.0	47.7	71.4	76.5	58.0	12.4	3
Deepseek-Janus-Pro	44.3	13.8	79.8	80.1	43.8	1.7	5
Llama3.2-Vision	51.0	33.2	74.3	68.7	46.0	4.9	4
Focal-RegionFace	70.5	82.9	93.8	91.8	74.7	67.6	1

NLP Metric Comparison:

Focal-RegionFace yields superior BERTScore F₁ (76.0), minimum grammar errors (0.43), and highest expert rating (86.7%).

Multi-Attribute Recognition (Region vs. Full Face):

The model demonstrates leading performance for AU F₁, emotion accuracy, and age accuracy across both ROI and global settings.

Fine-Tuning Stage Ablations:

Ablation experiments confirm that Stage II (region-aware alignment) dramatically improves language scores, Stage III enhances spatial focus, and Stage IV optimizes attribute recognition.

7. Significance, Implications, and Future Directions

Focal-RegionFace establishes a new paradigm for spatially-grounded, attribute-rich facial analysis via the FaceFocalDesc framework, introducing:

1. A benchmark dataset (MFRF) for arbitrarily localized, multi-attribute face analysis.
2. A progressive, four-stage vision-language fine-tuning protocol building spatial awareness and multi-attribute reasoning.
3. State-of-the-art empirical results validated by both automated and region-level human preference metrics.

This demonstrates that region-level modeling significantly improves the interpretability and richness of facial analysis, particularly for applications requiring localizable feedback (e.g., clinical dermatology or nuanced affect recognition). A plausible implication is that the described progressive fine-tuning strategy and hybrid annotation pipeline will influence subsequent work on spatially-resolved, multi-attribute vision-language understanding, both on the face and beyond (Zheng et al., 1 Jan 2026).