Vision-Language Model for Face Verification

• The paper introduces VerLM, a model that fuses CNN-based visual encoding with GPT-2 decoding to verify faces and produce human-interpretable textual rationales.
• The model leverages a cross-projection fusion layer with a learned separator token, achieving improved metrics like METEOR and BERTScore over standard pipelines.
• A three-stage training regimen—including unimodal pretraining, mapper training, and end-to-end fine-tuning—ensures robust multimodal adaptation and enhanced system transparency.

A Vision-LLM (VLM) for face verification is a multimodal neural system that, given a pair of facial images, determines whether they depict the same individual and generates human-interpretable textual rationales for its decision. Recent advancements in this paradigm address the opacity of conventional face verification pipelines by combining convolutional visual inference with natural language explanation generation, thereby enhancing system transparency and user trust (Hannan et al., 5 Jan 2026).

1. Model Architecture and Encoding Pipeline

The VerLM architecture integrates visual and linguistic modalities to jointly enable face matching and explanation synthesis. The model consists of the following primary components:

1. Vision Encoder: A convolutional neural network (CNN), by default VGGFace (yielding 99.65% accuracy on LFW), processes each input image to generate a fixed-length embedding of dimension $h$ for a batch size $b$ (output shape $[b,h]$). CASIA-WebFace can optionally be used as the encoder, with a reported 99.05% accuracy.
2. Image Projection Layer: Each $[b,h]$ embedding is linearly projected into a vector of dimension $k = s \cdot d$, where $s$ is the number of prefix tokens and $d$ is the decoder’s hidden size. The result, reshaped to $[b,s,d]$, is appended with a learnable constant token $c$ and passed through a lightweight Transformer. Clipping out $c$ yields $s$ visual tokens.
3. Text Embedder and Text Projection: A GPT-2 tokenizer encodes the user’s natural language prompt (including explanation style) to $[b,t,d]$. The same constant token $c$ is appended, followed by transformation and clipping to produce the text-conditioning sequence.
4. Cross-Projection Fusion Layer: Inspired by audio-based ADIFF, image token sequences for the two images are concatenated with a learnable separator token (EOS), yielding $[b,2s+1,d]$. This is further concatenated with the text condition ($[b,t,d]$) to form $[b,2s+t+1,d]$. The sequence is fed to an additional Transformer to explicitly model the visual/textual pairwise relations.
5. Decoder-only LLM: A GPT-2 decoder (Base, Medium, or Large) autoregressively generates the output explanation, conditioned on the fused cross-modal representation.

Mathematical formulation:

Let $v_1,v_2 \in \mathbb{R}^d$ be image embeddings, projected to $V_1,V_2 \in \mathbb{R}^{s \times d}$. Let $\text{SEP} \in \mathbb{R}^d$ be a learned separator token; $P \in \mathbb{R}^{t \times d}$ is the projected prompt. The fused input is:

$$X = \text{Transformer}(\text{Concat}([V_1; \text{SEP}; V_2; P])) \in \mathbb{R}^{n \times d}$$

where $n = 2s + t + 1$. The LLM attends to $X$ and produces tokens $y_1,...,y_m$.

2. Training Objectives and Optimization

VerLM is optimized with respect to both the verification accuracy and linguistic quality of explanations.

• Verification Loss: The CNN vision encoder may be frozen after unimodal pretraining or (optionally) fine-tuned with a contrastive or triplet loss. By default, the encoder is frozen post-training on a large face dataset (e.g., VGGFace).
• Explanation Loss: The primary supervised objective is explanation generation, optimized using a diversity-regularized cross-entropy loss:

$L = L_\text{CE} - \lambda \cdot H$

with

$$L_\text{CE} = -\sum_{b=1}^B \sum_{t=1}^T \log p(y_{b,t} \mid X)$$

and entropy regularizer

$$H = -\frac{1}{B \cdot T} \sum_{b,t,v} p_{b,t,v} \log(p_{b,t,v} + \epsilon)$$

where $p_{b,t,v}$ is the softmax probability for token $v$, and $\lambda$ ($=0.1$ by default) controls text diversity.

• Training Regimen: Training is conducted in three stages:
  • Stage 1 (Unimodal Pretraining): Separate pretraining of the vision encoder and decoder.
  • Stage 2 (Mapper Training): Only projection and cross-projection layers are trained; encoders/decoders are frozen.
  • Stage 3 (End-to-End Fine-tuning): All modules are fine-tuned (lower learning rate for backbone to control forgetting).

3. Explanation Styles and Generation

VerLM enables two complementary forms of explanation for its face matching decisions, selectable via user prompt:

• Concise Explanations: Short summaries listing the three to five facial attributes most salient to the match/non-match determination (e.g., “same almond-shaped eyes, matching jawline; minor hair-color difference”).
• Comprehensive Explanations: Extended paragraphs detailing all observed similarities and differences (e.g., “both images share warm tan complexion, straight nose, and high cheekbones; however, hair texture, eyebrow thickness, and eye color differ”).

User input prompts the desired explanation style, which is encoded in the text conditioning vector. Ground-truth examples were annotated from two datasets: Dataset 1 (mean 53 words, concise) and Dataset 2 (mean 122 words, comprehensive), with explanations generated by Llama 3.2 VLM and subsequently human-verified (Hannan et al., 5 Jan 2026).

4. Datasets, Preprocessing, and Training Details

• Data Source: Faces sampled from a subset of VGGFace2, yielding 79,771 image pairs (7,689 same-identity).
• Annotations: Image captions were initially generated by Llama 3.2 VLM, with face pairs randomly combined and labeled as match/non-match. Both explanation styles were produced and human-verified.
• Preprocessing: Images were resized to $224 \times 224$ pixels and normalized; no augmentation was applied to retain fine identity cues.
• Hyperparameters:
  • Stage 1: Backbone/GPT-2 pretrained separately.
  • Stage 2: Mapper trained for 30 epochs, Adam optimizer ($lr=1 \times 10^{-4}$), batch size 64, Cosine Annealing with Warm Restarts.
  • Stage 3: End-to-end fine-tuned for 10–20 epochs ($lr=1 \times 10^{-5}$), same scheduler/batch.

5. Evaluation and Empirical Results

VerLM was evaluated against both quantitative and qualitative benchmarks.

• Metrics: METEOR, BLEU, and BERTScore were computed on held-out splits.
  • Dataset 1: METEOR 0.3986 (+7.6% over OneDiff), BLEU 0.1557, BERTScore 0.9039.
  • Dataset 2: METEOR 0.3548, BLEU 0.1338, BERTScore 0.9004.
• Ablation Findings:
  • “Mapper-then-finetune” training is superior to “mapper-only” or “full end-to-end.”
  • Inclusion of the separator token (SEP) improved METEOR by ~1.4 points; eliminating cross-projection reduced METEOR by ~6 points.
  • Scaling the GPT-2 decoder from Base to Large yields incremental gains (METEOR up to 0.4098; BERTScore up to 0.9047).
  • The VGGFace encoder outperformed CASIA-WebFace by 2–3 METEOR points.
• Qualitative Analysis:
  • VerLM outputs explanations that systematically identify subtle facial distinctions, such as jawline curvature, eyebrow shape, and skin tone, and present them in a manner congruent with human reasoning.
  • This approach surpasses traditional saliency heatmaps in interpretability (Hannan et al., 5 Jan 2026).

6. Contributions, Limitations, and Future Directions

VerLM exhibits multiple advancements:

• First architecture to jointly perform face verification and generate natural language explanations in multiple styles.
• Pioneers the adaptation of an audio-difference cross-projection mechanism (including a learned separator token) to the visual domain.
• Introduces a three-stage training pipeline that preserves pretrained capabilities while facilitating deep cross-modal adaptation.

Limitations involve reliance on global image embeddings without spatial/region-level grounding, limited coverage to VGGFace2-style data, absence of broader evaluations (e.g., occluded/pose variants), and lack of user studies on explanation quality. Future directions include scaling to larger, instruction-tuned LLMs (e.g., Flan-T5, LLaMA), region-level attention, and assessing real-world applicability in human-centered evaluation (Hannan et al., 5 Jan 2026).

---

The VerLM architecture demonstrates that transparent and trustworthy face verification is achievable by fusing robust visual encoding with conditional natural language reasoning, yielding outputs that enhance accountability and bias detection in biometric systems.