THAMA Fusion in EmoFake Detection

• THAMA is a tensor-based fusion mechanism that leverages Tucker and Hadamard operations to effectively combine embeddings from dual multilingual speech models.
• It processes representations through tailored convolutional and projection layers, followed by explicit bilinear interaction modeling to capture complementary features.
• Empirical results show THAMA significantly reduces Equal Error Rates compared to simple concatenation, demonstrating improved performance in both in-domain and cross-lingual EmoFake Detection.

THAMA, expanded as Tucker–HAdaMArd Conjunction, is a fusion mechanism proposed for combining representations from multiple foundation models, especially multilingual speech foundation models, in EmoFake Detection (EFD). In the formulation reported on arXiv, THAMA is a parametric, tensor-based fusion layer that takes embeddings from two foundation models, projects them into a common latent space, models their high-order interactions via a Tucker-style bilinear core tensor, applies a Hadamard product to emphasize informative feature interactions, and then feeds the fused representation to a binary classifier for fake-versus-non-fake emotion detection (Phukan et al., 16 Jul 2025).

1. Definition and research setting

The task context for THAMA is EmoFake Detection, where the input is a raw speech waveform $x(t)$ and the output is a binary label $y \in \{0,1\}$, with the manipulation affecting the emotional state while preserving the content and speaker identity. The paper situates this problem in settings where prosodic cues such as pitch, tone, intensity, and timbre are altered only subtly, making detection difficult both within a language and in cross-lingual transfer. The reported experiments use the EmoFake dataset of Zhao et al., with English and Chinese subsets and five emotions—Neutral, Happy, Angry, Sad, and Surprise—where fake samples are generated by emotional voice conversion models (Phukan et al., 16 Jul 2025).

THAMA is introduced against two linked observations. First, multilingual speech foundation models are reported to be especially effective for EFD because their pre-training across many languages yields robust representations of prosodic variability. Second, simple fusion operations such as concatenation or averaging do not explicitly model cross-model interactions. THAMA addresses that gap by serving as a learned fusion layer for two foundation-model branches. Its intended advantage is not merely dimensional aggregation, but explicit modeling of complementary relationships between the features extracted by different encoders. In the paper’s framing, this is particularly relevant when the paired models differ in architecture, pretraining corpus, and learning objective, as with XLS-R, Whisper, and MMS.

2. Architectural formulation

THAMA operates on two high-level representations, $\mathbf{X}_1 \in \mathbb{R}^{d_1}$ and $\mathbf{X}_2 \in \mathbb{R}^{d_2}$, produced by two foundation models. Before fusion, each branch is passed through a small convolutional block consisting of 3 × 1D convolution layers with 64, 128, 256 filters, kernel size 3, ReLU activation, with each conv layer followed by max pooling with pool size 2. Flattening yields transformed vectors $\mathbf{X}_1' \in \mathbb{R}^{d_1'}$ and $\mathbf{X}_2' \in \mathbb{R}^{d_2'}$ (Phukan et al., 16 Jul 2025).

These flattened vectors are projected into a shared fused space: $$\mathbf{F}_1 = \mathbf{W}_1 \mathbf{X}_1', \quad \mathbf{F}_2 = \mathbf{W}_2 \mathbf{X}_2',$$ where $\mathbf{W}_1 \in \mathbb{R}^{d_f \times d_1'}$ and $\mathbf{W}_2 \in \mathbb{R}^{d_f \times d_2'}$. The projection normalizes the dimensionalities and learns a space in which cross-model interactions are meaningful. The central tensor operation then uses a core tensor $$\mathbf{T} \in \mathbb{R}^{d_f \times d_f \times d_f}$$ to define the fused representation $y \in \{0,1\}$0: $y \in \{0,1\}$1 The paper also writes this compactly as

$y \in \{0,1\}$2

The Tucker-style interpretation is important to the design rationale. The tensor core is used to capture high-order interactions between the two projected feature vectors, so that the contribution of feature $y \in \{0,1\}$3 from one model and feature $y \in \{0,1\}$4 from the other can be learned jointly for each fused output dimension $y \in \{0,1\}$5. The paper presents this as more expressive than concatenation while remaining parameter-efficient relative to a naive dense interaction tensor.

After tensor fusion, THAMA applies a self-Hadamard product: $y \in \{0,1\}$6 equivalently $y \in \{0,1\}$7. This element-wise squaring is explicitly motivated as a sharpening step that amplifies strong, distinctive interactions and relatively suppresses weak or noisy components. The fused vector is then sent to a fully connected classifier with a Dense layer with 128 units + ReLU, a Dense layer with 64 units + ReLU, and an output layer for binary classification (sigmoid).

3. End-to-end pipeline and cooperative multilingual speech foundation models

The reported instantiation of THAMA uses frozen foundation models as front-end extractors. Audio is resampled to 16 kHz for speech foundation models and 16/24 kHz for music foundation models, and the paper uses average pooling over time on the final hidden layer to obtain per-utterance embeddings. Typical embedding dimensions reported are 768, 1024, 1280, and 512, depending on the model family (Phukan et al., 16 Jul 2025).

The investigated model families include monolingual SFMs—WavLM, Unispeech-SAT, wav2vec 2.0, and HuBERT—multilingual SFMs—XLS-R 300M, Whisper base, and MMS 1B—a speaker-recognition model, x-vector, and music foundation models from the MERT family plus music2vec-v1. Within this setting, “cooperative” SFMs refers to pairing two foundation models and allowing THAMA to learn their synergistic behavior through explicit fusion rather than simple stacking.

The full processing chain is: preprocessing; dual feature extraction from two FMs; branch-specific convolutional processing; projection to the common fused space; Tucker fusion; Hadamard sharpening; and binary classification. Training uses binary cross-entropy,

$y \in \{0,1\}$8

with Adam optimizer, batch size 32, learning rate $y \in \{0,1\}$9, and training for up to 100 epochs with dropout, learning rate reduction, and early stopping. The paper states that THAMA is trained end-to-end excluding the frozen FMs. It also reports that THAMA introduces between 5.5M and 10M trainable parameters, depending on the input dimensions and the chosen $\mathbf{X}_1 \in \mathbb{R}^{d_1}$0.

The baseline fusion method is feature concatenation after the convolutional block and flattening. The conceptual distinction is explicit: concatenation increases dimension without directly parameterizing interactions, whereas THAMA separately projects each branch, applies a bilinear/tensor interaction, and then applies a non-linear element-wise squaring.

4. Empirical performance in in-domain and out-domain EFD

The evaluation metric is Equal Error Rate (EER). The paper reports that multilingual SFMs significantly outperform monolingual SFMs, x-vector, and music foundation models in both in-domain and out-domain settings. Among individual models, MMS and XLS-R achieve the lowest in-domain EERs in the CNN back-end setting: English: MMS 1.69%, XLS-R 1.80%; Chinese: MMS 2.04%, XLS-R 2.09%. Music foundation models are reported to perform poorly, with EERs often above 10%, indicating that those representations are not well suited to speech emotion manipulation detection (Phukan et al., 16 Jul 2025).

Fusion with THAMA improves on concatenation across most strong model pairs. For XLS-R + MMS, the reported in-domain EERs are 1.00 for concatenation versus 0.89 for THAMA in English, and 1.17 versus 1.03 in Chinese. For Whisper + MMS, the English EER is 1.38 with concatenation and 1.24 with THAMA, while the Chinese EER is 1.75 versus 1.46. For Whisper + XLS-R, the English EER is 1.67 versus 1.53, and the Chinese EER is 1.83 versus 1.65. This pattern suggests that the gains are largest when the paired branches are already strong multilingual encoders and thus provide complementary information that benefits from explicit interaction modeling.

The paper also evaluates cross-lingual performance for the best pair, XLS-R + MMS. In E(TR)–C(TE), THAMA yields 3.01% EER versus 3.33% for concatenation. In C(TR)–E(TE), THAMA yields 4.21% versus 4.86%. These results are reported as improvements over both the individual multilingual SFMs and the concatenation baseline. Relative to the prior SOTA by Zhao et al., which achieved 3.65% EER in English and 8.34% in Chinese for in-domain EFD, THAMA with XLS-R + MMS reaches 0.89% and 1.03% respectively.

The paper does not present a separate “Tucker only” versus “Hadamard only” ablation. However, it does provide indirect evidence through concatenation-versus-THAMA comparisons, single-model-versus-fused comparisons, and t-SNE visualizations of raw FM embeddings. The reported t-SNE plots show clearer class clusters for multilingual SFMs such as MMS and XLS-R than for MERT-v1-330M or WavLM, offering a representational explanation for why THAMA is especially effective when built on multilingual SFM pairs.

5. Computational profile, implementation, and limitations

THAMA’s computational profile is described as modest relative to the underlying foundation models, whose parameter counts range from tens of millions to 1B. The fusion layer itself comprises the projection matrices $\mathbf{X}_1 \in \mathbb{R}^{d_1}$1, the core tensor $\mathbf{X}_1 \in \mathbb{R}^{d_1}$2, and the downstream FCN layers. The paper’s argument for parameter efficiency rests on Tucker-style structuring of the interaction tensor rather than a fully dense, unconstrained fusion parameterization (Phukan et al., 16 Jul 2025).

The implementation scope is also deliberately narrow. THAMA is evaluated only on EFD, using the EmoFake dataset with English and Chinese data and manipulations generated by seven voice conversion models. The paper explicitly notes several limitations: dataset restriction to EmoFake; language coverage restricted to two languages despite the multilingual pretraining of the upstream encoders; absence of detailed ablations isolating the Tucker and Hadamard components; and the fact that performance on other tasks is not empirically validated in the paper.

At the same time, the paper explicitly suggests broader applicability. Because THAMA is a generic two-branch FM fusion layer operating on embeddings, it is presented as adaptable to other speech or audio tasks where multiple SFMs are available and cross-model complementarities are expected. The cited examples include Automatic Speech Recognition, Speech Emotion Recognition, broader audio deepfake detection, spoofing detection challenges, and multimodal settings with appropriate architectural adjustments. These are presented as future directions rather than validated outcomes.

6. Terminological ambiguity in adjacent arXiv usage

The acronym THAMA is not uniformly used across recent arXiv papers. In the EFD paper, THAMA is explicitly defined as Tucker–HAdaMArd Conjunction, and that explicit definition anchors the term most precisely (Phukan et al., 16 Jul 2025).

By contrast, the remote sensing paper "Threshold Attention Network for Semantic Segmentation of Remote Sensing Images" does not explicitly use the acronym THAMA. Its formal terminology is Threshold Attention Mechanism (TAM), Threshold Attention Network (TANet), Attentional Feature Enhancement Module (AFEM), and Threshold Attention Pyramid Pooling (TAPP). The accompanying details state that interpreting “THAMA” there is best done informally as “THreshold Attention Mechanism + Architecture,” but that the technical referents in the paper are TAM and TANet rather than THAMA (Long et al., 14 Jan 2025).

A second nearby source makes the distinction even sharper. "On the Transients Handler for the Cherenkov Telescope Array Observatory" describes the Transients Handler (TH) as a subsystem of CTAO’s Array Control and Data Acquisition, but its full text does not mention the acronym THAMA anywhere. That paper focuses on alert ingestion, filtering, scheduling, persistent queues, MongoDB-backed databases, STS integration, and VOEvent/Kafka-based alert handling, not on any concept named THAMA (Collins et al., 28 Aug 2025).

Accordingly, the term has a clear, explicit meaning in current EFD literature—Tucker–HAdaMArd Conjunction—but a broader literature search around similar strings encounters at least two sources where the acronym is either absent or only an informal reinterpretation of different technical terms. For precision, THAMA should therefore be used explicitly for the tensor-based fusion mechanism in EFD unless a source defines it otherwise.