---
title: Point-Supervised Facial Expression Spotting
url: https://www.emergentmind.com/topics/point-supervised-facial-expression-spotting-p-fes
type: topic
---

# Point-Supervised Facial Expression Spotting

Point-Supervised Facial Expression Spotting (P-FES) addresses the problem of automatically identifying temporal intervals corresponding to facial expression instances—specifically macro-expressions (MaE) and micro-expressions (ME)—in extended, untrimmed facial video streams. Departing from traditional fully-supervised paradigms that demand dense frame-level or temporal boundary annotations, P-FES leverages sparse point-level supervision, requiring only a single timestamp inside each expression instance during training. This reduces annotation overhead while still yielding temporally precise and semantically valid expression proposals in real-world facial video analysis tasks [2511.16952].

## 1. Problem Definition and Motivation

Given an untrimmed video $V = \{v_1, \ldots, v_T\}$ of $T$ frames, and a set of $N$ point annotations $Y = \{(p_i, y_i)\}_{i=1}^{N}$—where $p_i$ denotes a single timestamp within the $i$-th expression and $y_i \in \{0,1\}^C$ is a multi-hot vector encoding $C$ classes (commonly, $C=2$ for MaE and ME)—the task is to predict a set of temporal proposals $\{(s_j, e_j, c_j)\}$ with onset $s_j$, offset $e_j$, and class $c_j \in \{\mathrm{MaE}, \mathrm{ME}\}$. Point supervision vastly reduces annotation costs and circumvents the ambiguities inherent in interval boundary determination. The primary challenge is to extrapolate from sparse signals to accurate temporal localization across varying expression durations and intensities [2511.16952].

## 2. Gaussian-Based Instance-Adaptive Intensity Modeling

To address the shortcomings of hard assignment pseudo-labeling schemes and the complex temporal signatures of facial expressions, P-FES introduces Gaussian-based Instance-Adaptive Intensity Modeling (GIM). This module models the expression intensity as a Gaussian-shaped function peaking at the apex and smoothly decaying outward, tailoring intensity supervision on a per-instance basis.

Given snippet-level features $\mathbf{x}_j \in \mathbb{R}^D$ and temporal intensity scores $a_j$, the GIM process operates per expression annotation as follows:

- **Pseudo-Apex Estimation**: Within a search region $I_i = [p_i - k_c/4, p_i + k_c/4]$ (where $k_c$ is the estimated duration for class $c$), the pseudo-apex frame $v_i^{\mathrm{apex}} = \operatorname{argmax}_{j \in I_i} a_j$ provides a feature mean $\mu_i = x_{v_i^{\mathrm{apex}}}$.
- **Duration and Spread**: The active duration $L_i$ is obtained by counting frames with $a_j > \theta$ in $J_i = [v_i^{\mathrm{apex}} - k_c/2, v_i^{\mathrm{apex}} + k_c/2]$, which is then symmetrically expanded by a factor $\delta$ to define $K_i$, the region for Gaussian modeling. The standard deviation $\sigma_i$ is computed as the root mean square distance in feature space around $\mu_i$ over $K_i$.
- **Instance-Adapted Soft Labels**: For each ${j \in K_i}$, soft intensity pseudo-labels are defined as
  \[
  \hat{a}_j = \exp \left( -\frac{ \| x_j - \mu_i \|^2 }{ 2 \sigma_i^2 } \right)
  \]
  yielding a smooth, instance-specific intensity profile peaking at the apex and providing a continuous signal for supervision [2511.16952].

## 3. Network Architecture and Supervision Branches

The overall P-FES framework utilizes a two-branch network grounded on overlapping temporal snippets processed via optical flow and a backbone such as SpotFormer. The shared feature tensor $\mathbf{F} \in \mathbb{R}^{T \times D}$ feeds two functionally distinct branches:

- **Class-Agnostic Expression Intensity Branch**: An MLP regressor maps features to intensity scores $a \in \mathbb{R}^T$. Supervision signals include the GIM-derived mean squared error loss $L_\mathrm{GIM}$, $\ell_1$ normalization $L_\mathrm{norm}$, a reward for high-confidence frames $L_\mathrm{reward}$, and a temporal smoothness regularizer $L_\mathrm{smooth}$.
- **Class-Aware Apex Classification Branch**: An MLP classifier predicts per-frame class scores $S \in \mathbb{R}^{T \times C}$, supervised via focal loss on pseudo-apex and pseudo-neutral frames, distinguishing between MaE and ME at the expression apex.

An intensity-aware contrastive (IAC) loss is integrated to encourage feature discriminability and neutral-expression separation. This contrastive term leverages weights based on intensity similarities, contrasting expression and neutral frames according to their soft pseudo-labels [2511.16952].

Total training loss aggregates these terms:
\[
L = L_\mathrm{GIM} + L_\mathrm{apex} + L_\mathrm{reward} + \lambda_1 L_\mathrm{smooth} + \lambda_2 L_\mathrm{norm} + \lambda_3 L_\mathrm{IAC}
\]

## 4. Inference Procedure and Proposal Generation

For inference, the dual-branch model processes the test video to yield temporal intensity scores $\{a_t\}$ and class logits $\{S_{t,c}\}$:

1. Consecutive frames with $a_t$ above adaptively selected thresholds are grouped as candidate proposals.
2. Each proposal $(s, e)$ undergoes:
   - Apex detection: $t^* = \operatorname{argmax}_{t \in [s,e]} a_t$.
   - Class determination: $c = \operatorname{argmax}_{c} S_{t^*,c}$, subject to $S_{t^*,c} > 0.5$.
   - Outer-Inner Contrastive (OIC) scoring: calculates proposal saliency by contrasting average intensity within vs. outside the segment.
3. Overlaps are suppressed via class-wise Soft-NMS to finalize the set of detections [2511.16952].

## 5. Experimental Protocols and Evaluation

Evaluation adheres to standardized datasets and metrics for facial expression spotting:

- **Datasets**: SAMM-LV (147 videos, 343 MaE + 159 ME, subsampled to 30 fps), CAS(ME)$^2$ (98 videos, 300 MaE + 57 ME), and CAS(ME)$^3$ (over 2231 MaE + 285 ME with ME clips filtered for <15 frames).
- **Validation**: Leave-one-subject-out cross-validation with a true positive definition of IoU $\geq 0.5$.
- **Performance**: Main results are summarized as follows (Table 1 and 2; F$_1$ scores):

| Method                  | SAMM-LV (All) | CAS(ME)$^2$ (All) | CAS(ME)$^3$ (All) |
|-------------------------|:-------------:|:-----------------:|:-----------------:|
| Fully-supervised        | 0.4401        | 0.4841            | 0.2559            |
| LAC                     | 0.3223        | 0.3598            | 0.2146            |
| TSP-Net                 | 0.2703        | 0.3358            | 0.1484            |
| Deng et al. 2025        | 0.3587        | 0.4000            | 0.2273            |
| **Ours (P-FES+GIM+IAC)**| **0.3705**    | **0.4023**        | **0.2335**        |

Ablation studies and qualitative visualizations confirm that instance-adaptive soft labeling (GIM) and two-branch optimization outperform hard labeling and fused structures, particularly in ME spotting. The approach delivers smooth, neutral-suppressed intensity curves and tight feature clustering around Gaussian-shaped expressions [2511.16952].

## 6. Comparative Context and Recent Advances

P-FES with GIM expands the methodological repertoire for point-supervised facial expression spotting, contrasting with anchor-based systems (e.g., LGSNet) and interval regression approaches. PESFormer [2410.18695] provides an alternative by leveraging Direct Timestamp Encoding (DTE)—binary supervision per snippet—within a vision transformer backbone, forgoing anchor parameterization and interval regression. PESFormer demonstrates significant gains in recall, precision, and F$_1$ by operationalizing dense snippet-wise classification with direct point (timestamp) supervision and zero-padding for interval preservation.

This broader shift toward point-supervised learning in temporal localization tasks is motivated by the sparse nature of micro-expressions and the annotational impracticalities of precise boundary definition, making P-FES and DTE-driven approaches influential for automatic facial expression analysis in unconstrained environments [2511.16952, 2410.18695].

## 7. Code Availability and Reproducibility

The codebase and pretrained weights for the P-FES method with GIM and IAC loss are publicly available at https://github.com/KinopioIsAllIn/GIM, supporting reproducibility and algorithmic benchmarking in facial expression spotting research [2511.16952].

Source: https://www.emergentmind.com/topics/point-supervised-facial-expression-spotting-p-fes