UniSpeech-BERT Multimodal Configuration

• The paper introduces a dual-stream transformer architecture that integrates UniSpeech for acoustic and BERT for contextual phoneme representations to detect mispronunciations.
• It implements early, intermediate, and late fusion strategies, with intermediate fusion achieving superior accuracy on diverse datasets.
• The design enhances cross-modal alignment, supporting robust, speaker-independent pronunciation assessment in CALL systems for Quranic recitation.

The UniSpeech-BERT multimodal configuration defines a dual-stream transformer-based model for integrating acoustic and textual modalities, optimized for phoneme-level mispronunciation detection in Arabic speech and specifically Quranic recitation. This architecture leverages UniSpeech for acoustic representations and BERT for contextualized phoneme representations, unifying these via structured fusion mechanisms. The design enables precise modeling of linguistic and phonetic context, facilitating robust detection of pronunciation errors and supporting the development of speaker-independent, multimodal Computer-Aided Language Learning (CALL) systems (Kucukmanisa et al., 21 Nov 2025).

1. Multimodal Transformer Architecture

The configuration comprises parallel acoustic and textual streams, each powered by a 12-layer transformer:

• Acoustic stream: A pretrained UniSpeech Transformer (768 hidden size, 12 heads) processes 4-second, 16 kHz waveforms segmented into 25 ms frames, generating a time series of embeddings:

$$x \in \mathbb{R}^{64000} \ \rightarrow \ H_a = \mathrm{UniSpeechEncoder}(x) \in \mathbb{R}^{T \times d_a}, \qquad T=160,\ d_a=768.$$

• Textual stream: The same audio segment is transcribed to a phoneme sequence $p_1,\dots,p_L$ using Whisper, which is then tokenized and encoded with a multilingual BERT-base (12 layers, $d_t=768$):

$$H_t = \mathrm{BERT}(p_1, \dots, p_L) \in \mathbb{R}^{L \times d_t}.$$

• Projection heads: Each modality output is linearly projected (FC layer with ReLU and dropout) to a shared 256-dimensional space:

$$\tilde{H}_a = \mathrm{ReLU}\left( H_a W_a^{(0)} + b_a^{(0)} \right), \qquad \tilde{H}_t = \mathrm{ReLU}\left( H_t W_t^{(0)} + b_t^{(0)} \right),\quad W_{·}^{(0)} \in \mathbb{R}^{768 \times 256}.$$

This architecture forms the basis for subsequent multimodal fusion strategies (Kucukmanisa et al., 21 Nov 2025).

2. Fusion Strategies for Multimodal Integration

Three distinct fusion paradigms are implemented for combining acoustic ($A = \tilde{H}_a$) and textual ($T = \tilde{H}_t$) features:

• Early Fusion:

Normalize average-pooled features from both modalities using LayerNorm, concatenate, and process through a multi-layer perceptron (MLP) for classification:

$$\hat{a} = \mathrm{LayerNorm}(a), \quad \hat{t} = \mathrm{LayerNorm}(t),$$

$$h_{\mathrm{EF}} = \mathrm{ReLU}([\,\hat a;\hat t\,]\,W^{(\mathrm{EF})}+b^{(\mathrm{EF})}) \in \mathbb{R}^{512}$$

Two additional FC+ReLU layers (512→128→29) produce final logits.

• Intermediate Fusion:

Each modality is individually transformed through a bottleneck of 128 units and then concatenated:

$$a' = \mathrm{ReLU}(a W_a^{(\mathrm{mid})} + b_a^{(\mathrm{mid})}), \quad t' = \mathrm{ReLU}(t W_t^{(\mathrm{mid})} + b_t^{(\mathrm{mid})}),$$

$$h_{\mathrm{IF}} = \mathrm{ReLU}([\,a';t'\,]\,W^{(\mathrm{IF})} + b^{(\mathrm{IF})}) \in \mathbb{R}^{256}$$

Output processed by FC to 29 classes.

• Late Fusion:

Independent classifiers are trained for each modality; final logits are projected and concatenated before passing through a final classifier:

$z_a, z_t \in \mathbb{R}^{29}$

$$a'' = \mathrm{ReLU}(z_a W_a^{(\mathrm{LF})} + b_a^{(\mathrm{LF})}), \quad t'' = \mathrm{ReLU}(z_t W_t^{(\mathrm{LF})} + b_t^{(\mathrm{LF})})$$

$$h_{\mathrm{LF}} = \mathrm{ReLU}([\,a'';t''\,]\,W^{(\mathrm{LF2})} + b^{(\mathrm{LF2})}) \in \mathbb{R}^{128}$$

The comparative evaluation of these strategies enables optimized cross-modal representation alignment (Kucukmanisa et al., 21 Nov 2025).

3. Embedding Extraction and Feature Construction

• UniSpeech: Final-layer features ($H_a^{(L)}$) are sampled at a 25 ms frame rate. For classification, either a global average is taken or stacked frame-level features are used.
• BERT: The [CLS] token’s final-layer embedding is extracted, followed by LayerNorm and linear projection to 256 dimensions:

$$t = \mathrm{LayerNorm}(H_t[\mathrm{CLS}]) W_t^{(0)} + b_t^{(0)} \in \mathbb{R}^{256}$$

No additional pooling is performed on the [CLS] vector. These representation choices facilitate effective phoneme-level modeling (Kucukmanisa et al., 21 Nov 2025).

4. Training Protocol and Implementation Details

• Loss: Standard cross-entropy over 29 phoneme classes:

$\mathcal{L} = -\sum_{i=1}^{29} y_i \log \hat{p}_i$

• Optimization: AdamW with learning rate $3 \times 10^{-5}$, weight decay $0.01$; batch size 8; dropout $0.1$.
• Schedule: Maximum 30 epochs, early stopping (patience 3) on validation loss.
• Evaluation: Fivefold cross-validation on training splits (Dataset A and B); selected weights evaluated on held-out test sets.
• Implementation specifics:
  • UniSpeech: First 6 layers frozen during late fusion fine-tuning.
  • BERT: Standard multilingual base (12 layers); WordPiece tokenizer adapted for phonemes.
  • Projection heads: FC $768 \rightarrow 256$, dropout $0.1$.
  • Classifier: 2–3 layer MLP, dimensions 512→128→29 for early/intermediate fusion; 128→29 for late.
  • Data: 1015 samples, 29 Arabic phonemes (8 “Hafiz” sounds), 11 reciters + YouTube; Dataset A (YouTube train, Hafiz test), Dataset B (randomized split across sources) (Kucukmanisa et al., 21 Nov 2025).

5. Evaluation Metrics and Empirical Results

The following metrics are used per class, then macro-averaged:

$$\mathrm{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}, \quad \mathrm{Precision} = \frac{TP}{TP + FP}, \quad \mathrm{Recall} = \frac{TP}{TP + FN}, \quad F_1 = 2 \cdot \frac{\mathrm{Precision} \cdot \mathrm{Recall}}{\mathrm{Precision} + \mathrm{Recall}}$$

Dataset	Fusion	Accuracy	Precision	Recall	F1
Dataset A	Early/Inter	0.966	0.969	0.966	0.965
Dataset A	Late	0.957	0.959	0.957	0.957
Dataset B	Early	0.970	0.974	0.970	0.970
Dataset B	Intermediate	0.985	0.988	0.985	0.985
Dataset B	Late	0.956	0.964	0.956	0.955

Intermediate fusion outperforms all alternatives, especially under data heterogeneity in Dataset B. This demonstrates the critical role of bottleneck-based feature alignment in multimodal representations for challenging speech tasks (Kucukmanisa et al., 21 Nov 2025).

6. Relation to Broader Multimodal Transformer Paradigms

The feature-level dual-encoder strategy in UniSpeech-BERT contrasts with methods such as the Multimodal Adaptation Gate (MAG) introduced in “Integrating Multimodal Information in Large Pretrained Transformers” (Rahman et al., 2019). MAG injects nonverbal (acoustic, visual) features as additively gated perturbations into selected transformer layers, computed as:

$\bar{h}^{(j)}_t = h^{(j)}_t + \alpha \Delta h_t$

where $\Delta h_t$ is a sum of modality-conditioned, element-wise gated projections of the nonverbal features. MAG’s design allows per-token alignment and incremental integration at multiple transformer depths, and is shown to yield stable 1–2% accuracy gains in multimodal sentiment analysis benchmarks.

Both approaches leverage pretrained unimodal transformers, use projection heads for modality alignment, and implement feature interaction via network-level fusion or cross-modal gating. The UniSpeech-BERT architecture, however, focuses on strict dual-stream fusion and separately benchmarked early/intermediate/late fusion strategies for frame-level classification, as demanded by fine-grained phoneme mispronunciation tasks. A plausible implication is that such fine temporal alignment and explicit projection-based fusion are favorable where temporal granularity and speaker independence are required, as in Quranic pronunciation assessment (Kucukmanisa et al., 21 Nov 2025, Rahman et al., 2019).

7. Impact and Applications

The UniSpeech-BERT multimodal configuration provides a framework for developing robust, speaker-independent pronunciation assessment systems deployable in CALL contexts, notably for Quranic recitation and related educational settings. Its pronounced accuracy and generalization on diverse, phoneme-rich datasets supports the feasibility of multimodal transformer architectures in high-stakes, granular speech diagnostics. The explicit comparison of fusion strategies informs best practice for multimodal system design, with intermediate fusion validated as optimal for fine-grained phonological error detection. This architecture contributes foundational modeling techniques extensible to broader tasks in multimodal speech processing and language learning platforms (Kucukmanisa et al., 21 Nov 2025).