Hierarchical Fusion for Code-Switched Speech

• The paper demonstrates that hierarchical feature fusion optimizes modality order to achieve up to 98.75% accuracy in ASD detection.
• It details a multi-stage integration of acoustic, linguistic, and paralinguistic features via Transformer encoders for improved code-switching analysis.
• Empirical results and ablation studies confirm that proper fusion order and LID injection timing are critical for robust ASR and clinical speech tasks.

Hierarchical feature fusion for code-switched speech refers to the structured integration of multi-modal features extracted from speech data exhibiting intra-utterance language changes. The method is designed to address the multi-faceted nature of code-switched speech, which combines acoustic, linguistic, and paralinguistic streams. Hierarchical fusion systematically fuses these modalities in ordered stages—often using deep neural architectures, particularly Transformer Encoders—such that lower-level and higher-level information are integrated with attention to modality complementarity and the temporal progression of speech cues. Recent applications include early autism spectrum disorder (ASD) detection from code-mixed children’s speech (Akhtar et al., 2024), and language tagging and recognition in automatic speech recognition (ASR) systems for Mandarin-English utterances (Yang et al., 2024). Both demonstrate that the order and granularity of feature fusion stages are critical determinants of system performance.

1. Multimodal Feature Extraction in Code-Switched Speech

Feature fusion for code-switched speech targets three principal modalities:

• Acoustic features: Captured via Mel Frequency Cepstral Coefficients (MFCCs, 39-dimensional, commonly extracted in 25 ms frames with 10 ms hop) and pre-trained representations such as XLSR-53 wav2vec2 (1024-dimensional), encoding detailed time-frequency speech properties.
• Linguistic features: Consisting of hand-crafted code-mixing statistics (average word length, sentence length, speech rate, English/Hindi ratio, and switch count) and contextual embeddings output by pretrained multilingual BERT (mBERT) (768-dimensional).
• Paralinguistic features: TRILLsson (512-dimensional) embeddings designed to capture non-verbal and prosodic cues including intonation, stress, and rhythm.

In ASR contexts, additional frame-aligned representations are generated at multiple network depths for phonetic and semantic coding, with explicit segment-level and posterior-level language identification (LID) information (Akhtar et al., 2024, Yang et al., 2024).

2. Hierarchical Fusion Architectures and Modality Order

Hierarchical feature fusion architectures proceed in multiple fusion steps, with the order of integration being a fundamental variable. For ASD detection with code-switched speech (Akhtar et al., 2024), the process is formalized as:

• Initial projection of each feature stream (acoustic, linguistic, paralinguistic) into time-aligned matrices $A_i \in \mathbb{R}^{T \times d_a}$, $L_i \in \mathbb{R}^{T \times d_l}$, $P_i \in \mathbb{R}^{T \times d_p}$.
• Ordered fusion: for example, $f_A$ and $f_L$ are merged via a Transformer encoder yielding intermediate $H_1$, then concatenated with $f_P$ and passed through a second Transformer encoder to obtain a final utterance representation $H_2$:

$$\begin{align*} f_L &= \text{TEnc}_L(L_i), \quad f_A = \text{TEnc}_A(A_i) \ H_1 &= \text{TEnc}_1(\text{Concat}(f_A, f_L)) \ f_P &= \text{TEnc}_P(P_i) \ H_2 &= \text{TEnc}_2(\text{Concat}(H_1, f_P)) \end{align*}$$

Experiments demonstrate that the order—acoustic + linguistic first, paralinguistic last—maximizes discriminative power (98.75% accuracy and 97.19% macro-F1 for ASD detection) compared to alternative orderings, with up to 2% accuracy variance observed depending on fusion sequence. This order reflects the speech processing hierarchy: lexical/phonological content precedes prosodic nuance (Akhtar et al., 2024).

In code-switching ASR (Yang et al., 2024), hierarchical fusion involves staged integration of LID posteriors into stacked Transformer layers, with injection at intermediate layers (e.g., layers 3 or 6 in a 9- or 15-layer encoder). This enables language-specific expert encoders to receive both shared and language-wised fused representations, with mixture-of-experts gating for final output combination.

3. Mathematical Formulation of Hierarchical Fusion

Transformer blocks are central to hierarchical fusion in both fields:

• Transformer Encoder Block: Applies multi-head self-attention:

$Q = X W_Q, \quad K = X W_K, \quad V = X W_V$

$L_i \in \mathbb{R}^{T \times d_l}$0

• Outputs are normalized and post-processed through feed-forward and residual connections.
• Flat vs Hierarchical Fusion: Flat fusion concatenates all features and passes them jointly to a classifier; hierarchical fusion sequences the integration steps, allowing intermediate representations and staged learning.
• In ASR: Language posteriors $L_i \in \mathbb{R}^{T \times d_l}$1 extracted by $L_i \in \mathbb{R}^{T \times d_l}$2, are projected and injected at subsequent layers: $L_i \in \mathbb{R}^{T \times d_l}$3. A mixture-of-experts gating mechanism computes final output fusion per frame: $L_i \in \mathbb{R}^{T \times d_l}$4 (Yang et al., 2024).
• Non-Peaky CTC Loss: Training losses favor soft alignments at language change points:

$L_i \in \mathbb{R}^{T \times d_l}$5

improving sensitivity to fine-grained language boundaries.

4. Datasets, Preprocessing, and Workflows

Hierarchical feature fusion requires carefully annotated and preprocessed code-switched corpora:

For ASD detection (Akhtar et al., 2024):

• CoSAm Corpus: 62 recordings (159.75 min) from 30 ASD and 31 neurotypical children (3–13 y), English/Hindi code-mixing.
• Audio processing: Silence removal, 25 ms Hamming windows, MFCC extraction via OpenSMILE; XLSR-53 features for deep audio representations.
• Text preprocessing: Whisper for ASR transcripts, mBERT WordPiece tokenization, statistical code-mixing features.

For ASR (Yang et al., 2024):

• SEAME Corpus: ~115 hours Mandarin–English code-switched speech. LID-aligned for ground truth evaluation.
• Hierarchical workflows: Input frames are processed through shared and expert encoders, with intermediate LID injection and MoE gating.

Training uses categorical cross-entropy or CTC/NPC for ASR, Adam optimizer, and 5-fold cross-validation for classification tasks.

5. Empirical Results, Ablations, and Comparative Analysis

Hierarchical fusion provides robust improvements over baseline and flat fusion methods:

Fusion Method/Modality	Accuracy (%)	Macro-F1 (%)	Setting
Acoustic Only	93.48	91.50	ASD, single-modality
Linguistic Only	87.90	83.50	ASD, single-modality
Paralinguistic Only	97.67	96.50	ASD, single-modality
Flat (A+L+P)	96.74	97.00	ASD, concatenation
Hierarch. (A+L → P, mBERT)	98.75	97.19	ASD, hierarchical fusion
D-MoE (ASR)	21.0*	N/A	ASR, MER DevMAN (%)
Proposed Hierarchical (ASR)	18.5*	N/A	ASR, MER DevMAN (%)

*ASR MER: lower is better; other entries: higher is better

Flat fusion sometimes underperforms single-modality paralinguistic models due to feature dilution (e.g., 96.74% vs. 97.67% accuracy in ASD). Hierarchical fusion unlocks complementary information, with performance dependent on fusion order. In ASR, replacing standard CTC with non-peaky CTC loss and injecting LID at a single optimal depth yields tangible reductions in mixed error rate (MER), benefiting code-switch sensitivity (Akhtar et al., 2024, Yang et al., 2024).

Ablation studies confirm that fusion sequence selection is not a trivial hyperparameter: swapping fusion steps can elicit up to 2% swing in classification accuracy. For ASR, LID injection at the carefully chosen intermediate layer outperforms all-at-once or late-injection alternatives.

6. Theoretical Rationale and Implications

The sequencing of feature fusion stages is grounded in the hierarchical organization of speech: lower-level acoustics and lexical cues precede prosodic and affective overlays. In the ASD detection task, integrating acoustic and linguistic modalities before paralinguistic features leverages the core content and phonological form as a substrate for subsequent prosodic discrimination. Analogously, in code-switching ASR, injecting LID posteriors at intermediate network depths encourages segmental language homogeneity, facilitating chunk-aware language-specific processing and reducing phonemic ambiguity at switch points.

This suggests hierarchical, order-sensitive integration is preferable for a wide range of code-switched tasks beyond ASD detection and ASR, such as emotion recognition or speaker verification, especially when signal, content, and affect are naturally stratified (Akhtar et al., 2024, Yang et al., 2024).

7. Applications and Future Directions

Hierarchical feature fusion enables high-accuracy, scalable screening tools for clinical applications (e.g., near-99% accurate ASD flagging from code-mixed child speech) and enhances robustness in multilingual ASR with rapid code-switches (e.g., state-of-the-art MER on SEAME). The methodology’s emphasis on modality order, intermediate representation learning, and per-frame language-awareness is applicable wherever speech processing demands disentangling interleaved linguistic systems at multiple temporal and semantic scales. A plausible implication is that future work will expand these hierarchies to include even richer prosodic, syntactic, or pragmatic embeddings, and explore their transferability across typologically divergent code-switched language pairs (Akhtar et al., 2024, Yang et al., 2024).