LaScA: Language-Conditioned Scalable Modelling of Affective Dynamics

Abstract: Predicting affect in unconstrained environments remains a fundamental challenge in human-centered AI. While deep neural embeddings dominate contemporary approaches, they often lack interpretability and limit expert-driven refinement. We propose a novel framework that uses LLMs (LMs) as semantic context conditioners over handcrafted affect descriptors to model changes in Valence and Arousal. Our approach begins with interpretable facial geometry and acoustic features derived from structured domain knowledge. These features are transformed into symbolic natural-language descriptions encoding their affective implications. A pretrained LM processes these descriptions to generate semantic context embeddings that act as high-level priors over affective dynamics. Unlike end-to-end black-box pipelines, our framework preserves feature transparency while leveraging the contextual abstraction capabilities of LMs. We evaluate the proposed method on the Aff-Wild2 and SEWA datasets for affect change prediction. Experimental results show consistent improvements in accuracy for both Valence and Arousal compared to handcrafted-only and deep-embedding baselines. Our findings demonstrate that semantic conditioning enables interpretable affect modelling without sacrificing predictive performance, offering a transparent and computationally efficient alternative to fully end-to-end architectures

• The paper proposes a deterministic, language-conditioned framework that maps handcrafted facial and audio features to affect-aware semantic embeddings.
• It demonstrates statistically significant improvements in predicting arousal and valence across diverse datasets using adaptive cue selection and refined semantic lexicon.
• The approach offers a scalable, computationally efficient alternative to deep architectures, enabling real-time and interpretable affect modelling.

Language-Conditioned Scalable Modelling of Affective Dynamics: An Expert Review of LaScA

Framework Overview and Motivation

LaScA introduces a deterministic, language-conditioned framework for affect dynamics modelling, emphasizing transparency, interpretability, and computational efficiency over traditional end-to-end deep architectures. The method leverages handcrafted facial geometry and audio descriptors, extracted using domain-informed protocols, and augments them with a fixed semantic lexicon derived offline from LLMs. Each behavioral feature (e.g., facial blendshape coefficient or MFCC) is mapped to a concise, affect-aware linguistic description via a reproducible LLM prompt. For each temporal window, salient features are selected through Otsu-based thresholding, their semantic labels are composed into structured templates, and the resulting text is encoded using a frozen sentence transformer to yield high-level semantic embeddings. These embeddings are fused with the original feature vectors, generating a multimodal representation for downstream affect change prediction using a lightweight preference-learning MLP.

Figure 1: The LaScA pipeline—features are filtered for salience, mapped to fixed semantic descriptions, encoded via a frozen LLM, fused, and evaluated by a preference-learning MLP for affective change.

Methodological Innovations and Technical Implementation

LaScA's hybrid approach decouples signal transparency from contextual abstraction. Handcrafted feature extraction ensures domain-aligned, compact representations, minimizing entanglement between behavioral cues and predicted affect. Salience estimation via Otsu's threshold delivers adaptive cue selection in the presence of inter-subject variability. The deterministic semantic lexicon provides stable, reproducible priors, circumventing stochasticity and drift common to dynamic LLM prompting.

Segment-level language-conditioned representations are constructed as concatenated templates derived from active features, enabling deterministic formatting compatible with sentence-transformer tokenization. Five distinct pretrained sentence encoders (MPNet, DistilRoBERTa, MiniLM, DistilBERT) are evaluated as frozen backbones, each providing semantic context embedding irrespective of dataset specificity. Fusion with handcrafted features results in rich, modality-agnostic representations. The preference learner is a constrained MLP, trained exclusively on pairwise differences between consecutive segments where affective change exceeds a threshold. The approach exploits ordinal, pairwise supervision, reducing label noise associated with absolute affect estimation and aligning with recent paradigms in affective computing.

Empirical Results and Numerical Evidence

Experiments on Aff-Wild2 and SEWA DB, covering in-the-wild and cross-cultural conversational settings, demonstrate statistically significant accuracy gains for both arousal and valence prediction across visual, audio, and multimodal configurations. Language-conditioned fusion yields peak accuracies of up to 0.75 (arousal, Aff-Wild2), 0.74 (valence, Aff-Wild2), and 0.83 (valence, SEWA DB) in subject-independent protocols, outperforming or matching state-of-the-art deep embedding models (SwinFace, MAE-Face, MMA-DFER, HiCMAE). Longer temporal windows (5s vs. 3s) and higher affect change thresholds both favor semantic grounding, underscoring the importance of extended context.

Ablation studies reveal that lexicon quality is critical: models leveraging an affect-aware LLM lexicon consistently outperform those using feature-name-only mappings, with up to 2% improvements for arousal and consistent gains for valence. The impact of semantic conditioning is more pronounced in conversational and less constrained scenarios (SEWA DB), and in modalities where handcrafted features exhibit limited discriminative power.

Figure 2: Frequency distribution of unique multimodal prompts evidences the diversity and compositional richness in semantic template construction.

Computational Efficiency and Scalability

LaScA is architected for computational tractability. All encoders are frozen; only the preference learner is trainable, with parameter counts in the 129k–230k range. Sentence-transformer backbones contribute semantic priors without increasing inference latency beyond 140 ms per sample on commodity hardware. This efficiency enables deployment in real-time, low-resource environments, and supports reproducible experimentation without the stochasticity and overhead of large-scale deep models.

Implications and Future Directions

Theoretically, LaScA bridges explainable AI and affective computing by integrating semantic knowledge with domain-grounded behavioral descriptors. Practically, it offers a scalable, transparent alternative to opaque deep architectures, facilitating real-time deployment, domain transfer, and expert-driven refinement. Modality-agnostic fusion and deterministic lexicon generation enhance robustness in environments with annotation noise, cultural variability, or ambiguous context.

Future expansions could investigate selective adaptation of backbone encoders, dynamic or multilingual lexicon generation for cross-cultural transfer, alternative salience mechanisms (learned gating), and sequence models for long-range affective trajectory modelling. Extension to discrete emotion categories or higher-dimensional affect states is tractable given the modularity of the template construction and semantic encoding approach.

Conclusion

LaScA presents an efficient, interpretable, and reproducible solution for modelling affective dynamics in unconstrained environments by leveraging language-conditioned representations over handcrafted descriptors. Empirical evidence confirms consistent predictive improvements, particularly in multimodal and less structured settings, establishing LaScA as a viable, modality-agnostic framework for affect modelling under real-world conditions (2604.07193).