Titans-as-a-Layer: Test-Time Memory for Conversational Speech Emotion Recognition

Abstract: Speech emotion recognition (SER) is commonly formulated as utterance-level classification, although conversational emotion depends on a speaker's usual vocal range and the emotional context established by previous utterances. Speech-LLMs provide strong pretrained acoustic and semantic representations, and can adapts them to SER labels via finetune, but this mechanism still missing per-dialogue state. We study whether test-time neural memory can supply this missing context while leaving the large audio LLMs (LALMs) backbone intact. Building on Titans, we introduce a plug-and-play Memory-as-a-Layer (MAL) adapter that writes dialogue history into a small neural memory and reads it back as an audio-token-aligned residual update, avoiding changes to the host model's token positions. Across different audio LLMs and emotion recognition datasets evaluations, our design improves SER performs across different evaluation metrics, supporting test-time memory as a residual contextual mechanism for conversational SER.

• The paper introduces a test-time neural memory module (MAL adapter) that captures dialogue history without disrupting pretrained representations.
• The paper demonstrates that integrating MAL with LoRA yields improvements up to 1.83% in WF1 and boosts macro-F1 scores across multiple datasets.
• The paper outlines a sequential training process where MAL is optimized after LoRA adaptation, ensuring efficient, stateful SER without architectural modifications.

Test-Time Neural Memory for Speech Emotion Recognition: The Titans-as-a-Layer Approach

Problem Setting and Motivation

Contemporary speech emotion recognition (SER), especially in conversational contexts, requires not only the detection of emotional cues from isolated utterances but also an understanding of the evolving affective state across dialogue history. The paralinguistic expressions that denote emotion—pitch, intensity, rate—must be interpreted relative to a speaker's baseline and in reference to preceding conversational turns. Despite advances in large audio LLMs (LALMs) that robustly encode acoustic and semantic information, current SER pipelines routinely operate at the single-utterance level, neglecting historical context and speaker-specific baselines. Conventional adaptation via LoRA enables task-specific fine-tuning with large models, but the learned updates are static, and fail to model dialogue-dependent state or adapt to a speaker’s expressive range over a session. Expanding context windows or prepending explicit history tokens can disrupt pretrained position encoding and computational efficiency.

Proposed Method: MAL Adapter with Titans Neural Memory

The paper introduces a Memory-as-a-Layer (MAL) adapter, building on the Titans framework, to address the absence of dialogue-level contextualization in frozen, LoRA-tuned LALMs. MAL is a plug-and-play neural module that, at inference time, maintains an internally-updated, compact neural memory state representing dialogue history. Key architectural features include:

• Non-invasive Integration: MAL injects memory-derived residuals aligned to audio-token positions in every transformer block, preserving token ordering and avoiding sequence length augmentation. This obviates the need for prepending memory tokens or shifting positional indices.
• Dialogue-Specific State: The memory state is reset at dialogue boundaries and updated incrementally per utterance during inference, allowing the model to condition emotion classification decisions on both task-general knowledge and dialogue-specific history.
• Sequential Two-Stage Training: The model is first adapted to SER with LoRA while MAL is detached, establishing the decision boundary for emotion labels. Subsequently, all model and adapter parameters except those of MAL are frozen, and MAL is trained to optimize context-sensitive refinement, compelling the memory to act as a pure state-tracking mechanism.

Ablative analysis across memory integration variants—memory as prepended context (MAC), memory as gating (MAG), and MAL—demonstrate numerically superior performance using the additive MAL strategy (WF1: MAL 57.48% vs. MAC 49.90% and MAG 50.32% on IEMOCAP with UltraVox v0.4).

Experimental Results and Analysis

Experiments are conducted on three standard datasets: IEMOCAP, MELD, and Multidialog, each representing increasing conversational complexity and participant diversity. The core evaluation compares baseline backbones (Qwen2-Audio, Audio Flamingo 3, UltraVox v0.4), their LoRA-adapted versions, and Titans+LoRA (i.e., LoRA plus MAL).

Numerical highlights from the results include:

• On IEMOCAP, Titans+LoRA consistently outperforms both frozen and LoRA-fine-tuned models across all backbones, with improvements of up to 1.60% WA, 2.59% UA, and 1.83% WF1 over LoRA alone.
• On MELD and Multidialog, Titans+LoRA delivers higher Weighted F1 and Macro-F1 scores across challenging multi-speaker, multi-party settings. For instance, for UltraVox v0.4 on Multidialog, Macro-F1 rises from 30.74% (LoRA baseline) to 33.32% with memory.
• Gains are observed not only in overall accuracy but also in metrics sensitive to class imbalance (e.g., Macro-F1), indicating improved recognition of underrepresented emotions.

The results robustly establish that test-time neural memory is a principled complement to parameter-efficient task adaptation—the modular memory mechanism systematically enhances representation of cross-utterance dependencies without necessitating disruptive architectural changes to the LALM backbone.

Implications and Future Perspectives

The work demonstrates that a Titans-inspired, residual-form memory branch can be employed at inference to inject dialogue-dependent state into frozen LALM pipelines for SER. This approach preserves the interface and inductive biases of host models, while efficiently modeling temporal and speaker dynamics external to the training regime. The additive structure of MAL prevents interference with the host backbone’s pretrained representations or the LoRA-adapted task boundary.

The implications are twofold:

• Practical: Deploying memory-augmented SER systems in resource-constrained or privacy-sensitive environments is facilitated by the low parameter overhead and the lack of requirement for full fine-tuning or architectural modification. The methodology is extendable to other sequence classification domains where stateful contextualization is critical, such as dialogue act recognition and mental health assessment.
• Theoretical: The explicit separation of static task adaptation and dynamic state encoding posits a modular decomposition principle for transfer learning in large sequence models. This design pattern may generalize to multi-modal or continual learning scenarios, enabling robust dialog- or session-aware inference across domains.

Potential directions for future work include generalizing to broader backbone and dataset combinations, systematic comparison with alternative neural memory architectures, and integration into real-world conversational AI pipelines. Open problems pertain to optimal memory capacity scaling, handling of cross-session generalization, and the interface between memory and long-context transformers.

Conclusion

This paper establishes that Titans-style test-time neural memory, operationalized as a plug-in Memory-as-a-Layer adapter, confers clear, consistent benefits for conversational speech emotion recognition using LALMs. The additive, dialogue-specific memory mechanism is both numerically effective and architecturally non-intrusive. The findings suggest a principled pathway toward robust, stateful sequence modeling in the presence of immutable, pretrained backbones.