Affective State Label (ASL) Overview

• Affective State Label (ASL) is a framework that represents human affect using various annotation paradigms such as discrete emotion tags, dimensional coordinates, ordinal change labels, and open-ended latent representations.
• It integrates diverse data acquisition protocols including expert annotation, retrospective self-report, and controlled elicitation from modalities like sign language and physiological signals.
• ASL modeling spans direct prediction, representation learning, and preference-based affect change estimation, underscoring its role in shaping robust multimodal emotion inference.

Searching arXiv for papers on affective state labels and related formulations. Searching arXiv for "affective state label" and closely related recent formulations. Affective State Label (ASL) denotes a representation assigned to a person’s affective condition, but its operational meaning varies substantially across subfields. In recent work, ASL has referred to discrete emotion tags, dimensional coordinates such as valence–arousal–dominance, continuous trajectories over time, retrospective self-reports tied to video review, experimentally induced condition labels, and latent affective representations learned without explicit supervision. This heterogeneity is not merely terminological. It reflects different assumptions about what counts as affect, how labels are elicited, whether affect is best modeled categorically or dimensionally, and whether annotation should target instantaneous states, global judgments, or affective dynamics (Chua et al., 20 May 2025). A precise account of ASL therefore requires attention to task formulation, annotation protocol, representation space, and the relation between labels and downstream inference.

1. Conceptual scope and definitional variants

In multimodal sign-language work, ASL has been introduced as a benchmark problem over American Sign Language videos: given a sign video, predict its overall sentiment, identify which emotions are present, and explain the visual cues that support those judgments (Chua et al., 20 May 2025). In that setting, the label space is explicitly bifurcated between global valence and specific discrete emotions. Sentiment is treated as a global valence judgment, while emotion is treated as a more specific affective category that may occur with intensity and may include multiple simultaneous states (Chua et al., 20 May 2025).

In collaborative-learning research, ASL refers instead to the verbal or self-reported labels that participants assign to their own cognitive-affective states while retrospectively reviewing a video of their collaborative task (Anindho et al., 1 Jul 2025). Here the labels are not externally assigned class tags but retrospective reports collected through a video-stimulated recall procedure. The operational emphasis is temporal localization of self-experience rather than recognition from sensory input.

In multimodal physiological feature-selection work, the term is effectively equivalent to multi-dimensional emotional labels, specifically continuous or multi-label representations such as valence, arousal, and dominance (Xu et al., 8 Aug 2025). There, ASL is not a single categorical variable but a label matrix shared with incomplete multimodal physiological features through a learned latent structure.

Other papers use ASL-like constructions without the exact phrase. LaScA treats affective state labeling as prediction of affective change—whether valence or arousal goes up or down between consecutive windows—rather than framewise regression of absolute affect values (Pinitas et al., 8 Apr 2026). PREFAB treats ASLs as time-continuous, subjective, and relative annotations of arousal during gameplay, and argues that only affective inflection regions need be directly labeled (Moon et al., 20 Jan 2026). A-Situ represents affective labels as paired valence–arousal coordinates accumulated into an affective curve over time (Kim et al., 2019).

This suggests that ASL is best understood as a family of affect annotation formalisms rather than a single standardized ontology. A plausible implication is that apparent disagreements about ASL often arise from mismatched assumptions about unit of analysis: clip-level judgment, event-level self-report, frame-level dynamics, or latent state representation.

2. Label spaces: categorical, dimensional, ordinal, and open-ended

Recent ASL formulations span several distinct representational regimes.

In EmoSign, the affective labels for ASL videos consist of a single overall sentiment label and a set of emotion labels (Chua et al., 20 May 2025). Sentiment is annotated on a 7-point scale from strongly negative to strongly positive, numerically represented as $-3, -2, -1, 0, +1, +2, +3$, and the paper also evaluates a coarse 3-class version: negative, neutral, positive (Chua et al., 20 May 2025). Emotion annotation is multi-label in spirit during data collection: annotators rate the presence and intensity of 10 emotion categories on a 0–3 scale, where 0 means “not present at all” and 3 means “extremely present.” The 10 categories are joy, excited, surprise (positive), surprise (negative), worry, sadness, fear, disgust, frustration, and anger (Chua et al., 20 May 2025). For single-label emotion classification, joy and excited are merged into happiness because their Jaccard similarity was 0.81, and the benchmark label set becomes happiness, surprise (positive), surprise (negative), worry, sadness, fear, disgust, frustration, anger, and neutral (Chua et al., 20 May 2025).

In structured retrospective self-report for collaborative learning, the survey label options were Confused, Disengaged, Curious, Optimistic, Frustrated, Conflicted, and Surprised, with an open-ended “Other” option (Anindho et al., 1 Jul 2025). The paper explicitly states that these labels are not treated as mutually exclusive, and participants were allowed to report multiple labels at the same time (Anindho et al., 1 Jul 2025).

In physiological emotion recognition, ASL may instead be multi-dimensional. ASLSL uses the valence-arousal-dominance framework and writes the label matrix as $Y \in \{0,1\}^{k\times n}$, described as a multi-dimensional emotional label matrix (Xu et al., 8 Aug 2025). In A-Situ, labels are pairs in valence–arousal space, and the system output at each time is “a set of two emotional points” over that space (Kim et al., 2019). LaScA further reduces the target to directional change labels: $$y_{t,t+1} = \begin{cases} 1 & \text{if } a_{t+1} > a_t \ 0 & \text{otherwise.} \end{cases}$$ This reformulates ASL as ordinal or preference-based dynamics prediction rather than absolute-state estimation (Pinitas et al., 8 Apr 2026).

Open-ended formulations also exist. MASIVE defines affective state identification over a practically unbounded set of affective states, including any terms that humans use to describe their experiences of feeling, and reports more than 1,000 unique affective states each in English and Spanish (Deas et al., 2024). That work frames the task as masked span prediction rather than closed-set classification.

These formulations show that ASL can denote at least four label families: discrete categorical states, dimensional affect coordinates, ordinal change labels, and open-vocabulary affective expressions. This suggests that transfer across ASL datasets is intrinsically nontrivial because the supervision geometry differs across tasks.

3. Annotation and label acquisition protocols

ASL acquisition protocols vary as much as label spaces.

EmoSign uses a structured annotation pipeline over 200 ASL utterance clips from ASLLRP, annotated by 3 ASL-native signers with professional interpreting experience (Chua et al., 20 May 2025). The researchers first selected 200 utterances after manually inspecting captions for emotional expressiveness and also used VADER sentiment analysis on the captions to help choose a spread of positive and negative examples (Chua et al., 20 May 2025). Annotators received a training session before starting, the study was IRB-approved under NUS-IRB-2024-1081, annotation was implemented in Qualtrics, annotators could skip clips, and each clip received between 1 and 3 annotations depending on skips (Chua et al., 20 May 2025). For each video, tasks were completed in the order sentiment analysis, emotion classification, and free-response description of emotional cues, followed by a confidence rating on a 0–100 scale (Chua et al., 20 May 2025). Final labels were derived by majority vote, with ties broken by the label from the annotator with the highest self-reported confidence (Chua et al., 20 May 2025).

Collaborative-learning ASLs are elicited through retrospect cued recall. After completing the collaborative task, participants watched a video recording of their own group interaction in an interactive application, were prompted at specific points to report their cognitive-affective state, and selected one or more labels from the survey or used “Other” to type a free-text label (Anindho et al., 1 Jul 2025). Two reporting modes were used: self-caught reporting, where the participant voluntarily opens the survey, and probe-caught reporting, where the survey opens automatically after a period of idleness (Anindho et al., 1 Jul 2025). The probe interval was 60 seconds, and after any report the video skipped forward by 1 second before resuming (Anindho et al., 1 Jul 2025). Participants indicated whether a state was onset or ongoing and could estimate the duration of ongoing states (Anindho et al., 1 Jul 2025).

PREFAB proposes low-budget retrospective self-annotation by targeting affective inflection regions rather than full-session labeling (Moon et al., 20 Jan 2026). Each detected inflection point becomes the center of a 5-second clip with 2.5 seconds before and 2.5 seconds after; overlapping regions are merged (Moon et al., 20 Jan 2026). The method then asks users to label only those selected clips and reconstructs the remainder by interpolation (Moon et al., 20 Jan 2026).

In experimentally induced physiological ASL, labels may be assigned by protocol rather than by self-report. The fNIRS study operationalizes meditation, amusement, and cognitive load through guided meditation music, stand-up comedy, and a pen-and-paper Fibonacci sequence writing task, respectively, each for four minutes after a two-minute baseline and separated by one-minute dead bands (Ghosh, 2024). The paper explicitly treats these induced conditions as affective labels in the ASL sense (Ghosh, 2024).

These protocols illustrate three major acquisition regimes: expert annotation from observable behavior, retrospective self-report from memory-supported review, and protocol-driven label assignment via controlled elicitation. A plausible implication is that disagreement between ASL datasets can originate as much from acquisition method as from label ontology.

4. Signal grounding and affective cues

A central issue in ASL is what evidence supports label assignment.

In sign-language affect labeling, the primary cues are non-manual markers: facial expressions, head movements, mouth shapes, and body movements (Chua et al., 20 May 2025). EmoSign reports concrete examples including furrowed brows, pursed lips, squinted eyes, head thrusts, head tilting, changes in head orientation, mouth shapes like an “O,” tongue out, puffed lips, shoulder raising, and full-body tilting (Chua et al., 20 May 2025). Annotators also noted sign-level modifications such as larger or smaller sign size, faster or slower signing speed, repetition, and finger-spelling used for emphasis (Chua et al., 20 May 2025). Context matters because eye gaze shifts, changes in signing space, and body orientation can indicate narrative switching or perspective changes, and without context the emotion can be ambiguous (Chua et al., 20 May 2025).

EASLT, though not framed as an ASL labeling paper, provides a related continuous affective-state representation in sign language translation. It extracts facial ROIs, encodes them with a ViT fine-tuned on FER2013, and uses linear interpolation to yield a smooth representation $\hat{\mathbf{Z}}_e$ of the signer’s affective state (Tu et al., 7 Jan 2026). The affective stream is then used as a semantic anchor to recalibrate manual sign features through Emotion-Aware Fusion (Tu et al., 7 Jan 2026). This indicates a broader tendency in sign-language research to treat affect as a first-class semantic signal rather than auxiliary metadata.

Physiological ASL relies on very different evidence. The fNIRS study uses hemoglobin-derived features from a 16-channel device: HbO, HbR, total hemoglobin $HbT = HbO + HbR$, and oxygen consumption $Oxy = HbO - HbR$, yielding 64 features per sample after preprocessing (Ghosh, 2024). A-Situ derives affective labels from motion, motivation, and contentment computed from egocentric image sequences, accelerometer data, and temporal persistence in the situation, then maps them into valence and arousal (Kim et al., 2019).

Latent-affect approaches use structured intermediate representations instead of direct labels. ES4R explicitly models turn-level affective states and dialogue-level affective dynamics from speech using intra-turn and inter-turn attention, but does so without explicit emotion supervision (Gao et al., 16 Jan 2026). In ASL terms, it is closer to affective state estimation or representation learning than to supervised hard labeling (Gao et al., 16 Jan 2026).

Taken together, these works show that ASL is inseparable from modality-specific evidence models. In sign language, the difficulty lies partly in the fact that the same facial and bodily signals serve both grammatical and emotional functions (Chua et al., 20 May 2025). In physiology and speech, the difficulty lies in converting indirect correlates into reliable affective representations.

5. Modeling paradigms

Recent ASL research supports several modeling strategies.

One paradigm uses labels as direct prediction targets. EmoSign evaluates sentiment analysis and single-label emotion classification from ASL videos, with and without captions (Chua et al., 20 May 2025). The fNIRS study trains neural networks over engineered physiological features for three-class classification in person-specific, group-level, and subject-independent settings (Ghosh, 2024). SMSAT similarly treats labels as three auditory-condition classes interpreted as calmness-related affective states: Normal/Silence, Music, and Spiritual Meditation (Suleman et al., 1 May 2025).

A second paradigm uses labels to guide representation learning or feature selection. ASLSL assumes that similar features share similar emotional labels and learns a shared latent structure $U$ connecting incomplete multimodal physiological features and multi-dimensional emotional labels (Xu et al., 8 Aug 2025). Its final objective combines reconstruction loss for incomplete modalities, label reconstruction loss, graph manifold regularization, and $l_{2,1}$-norm sparsity on the projection matrix: $$\min_{Q^{(v)}, \alpha^{(v)}, M, U} \sum_{v = 1}^{m} \left(\alpha^{(v)}\right)^{\gamma} \left[ \left\|\left(X^{(v)} - Q^{(v)} U^T\right) S^{(v)}\right\|_F^2 + \lambda \left\|Y - M U^T\right\|_F^2 + \eta \operatorname{Tr}\left(U^T L_Y U\right) + \delta \left\|Q^{(v)}\right\|_{2,1} \right]$$ subject to the stated nonnegativity and simplex constraints (Xu et al., 8 Aug 2025). In this formulation, the label matrix is not merely an output target; it defines which physiological features are emotionally meaningful.

LSGMER adopts a comparable principle for multimodal emotion recognition. It introduces learnable label embeddings $L_s$ as emotion prototypes or anchors and uses them as attention queries over audio and text features: $Y \in \{0,1\}^{k\times n}$0 followed by weighted residual fusion and Joint Objective Optimization with an Attribution-Prediction Consistency Constraint (Shao et al., 7 Apr 2025). Here, affective labels function internally as representation-shaping anchors.

A third paradigm treats ASL as preference learning over affective dynamics. LaScA predicts directional affect change between consecutive windows (Pinitas et al., 8 Apr 2026). PREFAB uses a Siamese preference-learning model with Ordinal Cross-Entropy to predict whether arousal increases, decreases, or stays equal between paired gameplay segments: $Y \in \{0,1\}^{k\times n}$1 and

$Y \in \{0,1\}^{k\times n}$2

with cutpoints set to $Y \in \{0,1\}^{k\times n}$3 (Moon et al., 20 Jan 2026).

These paradigms indicate that ASL can serve as target, structural prior, or relative supervision. This suggests that model comparison across ASL papers should be conditioned on the supervisory role of the label rather than on surface terminology alone.

6. Evaluation regimes and empirical properties

ASL research uses markedly different evaluation metrics depending on formulation.

EmoSign reports accuracy and weighted F1 for sentiment, weighted accuracy and weighted F1 for emotion classification, and Krippendorff’s alpha for annotation agreement (Chua et al., 20 May 2025). Sentiment agreement is 0.738, average agreement across emotion labels is 0.593, and emotion-specific alphas vary from 0.699 for joy to 0.119 for surprise_neg (Chua et al., 20 May 2025). The authors explicitly note that positive emotions had higher agreement than negative ones (Chua et al., 20 May 2025). Benchmark results show that ASL-video affect prediction remains difficult, especially from video alone. For 3-class sentiment, the best video-only result is GPT-4o with weighted accuracy 40.72 and weighted F1 24.43, whereas with captions GPT-4o reaches 52.13 wAcc and 76.72 wF1 and AffectGPT reaches 56.18 wAcc and 64.37 wF1 (Chua et al., 20 May 2025). For single-label emotion classification, the best video-only weighted accuracy and weighted F1 remain low—11.50 / 20.76 for GPT-4o and 13.01 / 22.02 for MiniGPT4—while captions substantially improve scores (Chua et al., 20 May 2025).

PREFAB evaluates selective ASL annotation through region-level F1, a time-efficiency gap $Y \in \{0,1\}^{k\times n}$4, and user-study reconstruction metrics including CCC, Spearman’s $Y \in \{0,1\}^{k\times n}$5, and DTW similarity (Moon et al., 20 Jan 2026). Across nine games, PREFAB achieved the best F1 and the closest TE to ground truth on average, with average $Y \in \{0,1\}^{k\times n}$6 (Moon et al., 20 Jan 2026). In the user study, reported averages were CCC = 0.67, Spearman $Y \in \{0,1\}^{k\times n}$7 = 0.69, and DTW similarity = 0.82 (Moon et al., 20 Jan 2026).

The fNIRS study uses mean classification accuracy under three deployment assumptions. It reports 83.04% for the individual model, 84.39% for the group model, and 60.57% for the subject-independent LOSO model (Ghosh, 2024). The group-model confusion matrix indicates especially strong classification of cognitive load and more confusion between meditation and amusement (Ghosh, 2024).

LaScA evaluates affect-change prediction with subject-independent accuracy and Wilcoxon signed-rank significance testing. It reports that language conditioning yields consistent improvements on Aff-Wild2 and SEWA, with gains especially strong for valence on SEWA (Pinitas et al., 8 Apr 2026).

The empirical pattern across these papers is consistent: ASL is feasible, but reliability and performance depend strongly on representation choice, modality completeness, and contextual support. In sign-language affect recognition, captions provide crucial disambiguating context (Chua et al., 20 May 2025). In retrospective annotation, preview improves confidence without degrading quality (Moon et al., 20 Jan 2026). In physiology, subject-independent generalization remains harder than within-person or pooled-subject modeling (Ghosh, 2024).

7. Recurrent challenges and research directions

Several technical difficulties recur across recent ASL literature.

A first challenge is ambiguity of the label space itself. EmoSign separates sentiment from emotion because clips with similar valence can instantiate distinct states such as worry and fear, and because surprise can have either positive or negative polarity depending on context (Chua et al., 20 May 2025). MASIVE argues that fixed small taxonomies fail to capture the open-ended vocabulary with which people actually describe their feelings (Deas et al., 2024). These works point in different directions—taxonomy refinement versus open-vocabulary modeling—but converge on the inadequacy of oversimplified label inventories.

A second challenge is temporal fidelity. Retrospective ASL collection improves ecological validity but introduces recall bias and uncertain alignment between reported state and original moment (Anindho et al., 1 Jul 2025). PREFAB addresses this by focusing annotation on affective inflection regions and adding a preview mechanism (Moon et al., 20 Jan 2026). LaScA addresses related noise by predicting affective change rather than absolute values (Pinitas et al., 8 Apr 2026). A plausible implication is that ordinal or event-centered formulations may be more robust than dense absolute annotation in unconstrained settings.

A third challenge is modality incompleteness and contextual dependency. ASLSL is motivated by the fact that real multimodal physiological data are often incomplete due to channel noise, electrode detachment, and acquisition or environmental issues (Xu et al., 8 Aug 2025). EmoSign shows that visual cues alone are insufficiently disambiguating for many ASL affect labels, while captions can substantially improve prediction (Chua et al., 20 May 2025). EASLT likewise treats facial affect as a semantic anchor because manual signals alone often do not uniquely determine meaning (Tu et al., 7 Jan 2026).

A fourth challenge concerns the status of labels inside the model. Some systems predict labels directly; others use them as latent anchors, semantic prototypes, or supervisory priors (Shao et al., 7 Apr 2025, Xu et al., 8 Aug 2025). This suggests a broader shift from viewing ASL as passive supervision toward viewing it as structured knowledge that can shape representation geometry.

Overall, recent work portrays ASL as an evolving interface between affect theory, annotation design, and multimodal machine learning. The term now spans discrete, dimensional, ordinal, open-ended, and latent formulations. What unifies these usages is not a single ontology, but the attempt to operationalize affect in a form that is sufficiently structured for computation while remaining faithful to modality, context, and temporal dynamics.