Multi-task Learning is Not Enough: Representational Entanglement in Dual-output Second Language Speech Recognition

Abstract: Second-language (L2) speech recognition often requires transcriptions of pronunciations and intended meanings. Multi-task learning (MTL) is a natural approach because it assumes that shared representations benefit both outputs. However, this paper shows that this assumption does not hold across Korean and English. MTL improves meaning but degrades surface transcription, especially in English, where the degradation scales with surface-meaning divergence measured by Levenshtein edit distance. Encoder analysis links these patterns to encoder-level entanglement, with Korean preserving distinct task representations while English produces nearly identical ones. Cross-task decoder analysis shows that the meaning dual-output decoder adapts with a unique representation, while the surface dual-output decoder remains constrained by the encoder. These findings motivate the design of MTL frameworks that mitigate encoder-level entanglement to reduce surface degradation in dual-output L2 automatic speech recognition.

• The paper demonstrates that multi-task learning in dual-output L2 ASR causes encoder-level representational entanglement, differentially impacting surface and meaning transcription.
• It reveals that while dual-output models enhance meaning transcription, they significantly degrade surface transcription accuracy in English L2 speech.
• The study employs CKA analysis to expose language-specific encoder divergence and recommends structured disentanglement methods for robust speech recognition.

Multi-task Learning Limitations Due to Representational Entanglement in Dual-output L2 Speech Recognition

Background and Motivation

Second-language (L2) automatic speech recognition (ASR) systems often require simultaneous recovery of surface-level (verbatim) pronunciations and intended meaning (canonical transcription) from a single acoustic input. While multi-task learning (MTL) frameworks intuitively seem well-suited for such dual-output (DO) tasks—by sharing representations across tasks and potentially improving generalization—the assumption that both outputs benefit from joint training had not been systematically validated across languages. The paper critically examines this assumption for Korean and English L2 speech and reveals language-dependent limitations of MTL, focusing on representational entanglement at the encoder level.

Experimental Framework

English L2 speech (produced by native Korean speakers) and Korean L2 speech (produced by native Chinese and Japanese speakers) datasets were used. Surface-meaning transcription divergence was quantified using the Levenshtein edit distance (ED), with syllable-level metrics for Korean and word-level metrics for English, confirming comparable distributions across languages.

Single-output (SO) baselines (Conformer, Whisper-base, Whisper-small) were trained for surface and meaning tasks individually. The DO configuration combined a shared Conformer encoder with dual Transformer decoders targeting each transcription. Critically, auxiliary CTC heads—essential for alignment—remained identical across setups, ensuring differences arose solely from the presence of the second decoder and joint optimization.

Numerical Results and Empirical Observations

All models demonstrated that surface-level transcription remains intrinsically more challenging than meaning extraction, regardless of architecture or parameter scaling. However, the central empirical finding is that MTL improves meaning transcription while degrading surface transcription—an effect that is markedly larger in English.

Strong numerical results include:

• Surface transcription for English DO Conformer exhibited a significant error rate increase: from 13.78% (SO) to 15.08% (DO), while Korean saw only a marginal increase (11.14% SO, 11.34% DO).
• Meaning accuracy improved in both languages under MTL: e.g., Korean DO Conformer dropped from 1.60% (SO) to 0.77% (DO), English from 3.87% to 3.19%.
• For English, degradation in surface transcription scaled monotonically with surface-meaning ED: peaking to a +6.72% gap for high divergence cases (ED > 10).

These results directly challenge the uniform advantage posited for MTL in DO ASR and establish a language-specific, divergence-dependent trade-off.

Mechanistic Representational Analysis

To probe the root cause, the paper employs Centered Kernel Alignment (CKA) to quantify representational similarity across encoder and decoder layers.

Encoder analysis revealed:

• Korean SO encoders rapidly diverge after early layers, indicating task-specific separation. DO encoders align strongly with the surface task.
• English SO encoders remain highly similar throughout most layers, suggesting encoder-level task entanglement—they fail to meaningfully differentiate tasks even when trained independently.

Decoder analysis elucidates adaptation dynamics:

• Korean DO decoders maintain alignment with their respective SO baselines, echoing the encoder’s separability.
• English decoders show a distinctive asymmetry: the DO meaning decoder establishes a unique representation, diverging from SO meaning and even DO surface; this enables some meaning improvement. Conversely, the DO surface decoder remains constrained by the entangled encoder and cannot mitigate degradation.

Contradictory claim: Decoder-level adaptation is insufficient to correct for encoder-level entanglement; surface transcription suffers disproportionately as meaning diverges from surface realization.

Implications and Directions for Future Research

The findings provide a formal, quantitative basis for reevaluating MTL’s role in DO L2 ASR, particularly for languages like English where surface and meaning diverge substantially. Practically, deploying MTL in feedback-driven L2 ASR systems risks impairing pronunciation modeling precisely when divergence is greatest—a domain where accurate surface recovery is most critical.

Implications for model architecture:

• Encoder-level disentanglement mechanisms are required to mitigate cross-task interference and preserve surface transcription integrity.
• Candidates for future architectural enhancements include sparse decomposition strategies (to enforce orthogonality), adversarial regularization, and gating mechanisms.
• Extending representational analysis beyond CKA (e.g., using mutual information or probe-based metrics) should further guide practical disentanglement.

Theoretical implications:

• Joint optimization in the presence of shared but distinct linguistic content may promote representational collapse, especially when the divergence between output tasks is substantial and domain-specific (e.g., English phonological reduction, L2 pronunciation variability).

Advancing DO L2 ASR thus requires rethinking shared representation paradigms and adopting language- and task-sensitive approaches for multi-task optimization.

Conclusion

The paper formally demonstrates that MTL in dual-output L2 ASR produces asymmetric, language-dependent trade-offs, rooted in encoder-level representational entanglement. Decoder adaptation is insufficient to compensate, especially in English, where surface-meaning divergence is high. These insights motivate the development of structured MTL methods targeting encoder disentanglement, with broader applicability to multi-task speech and language modeling (2606.06065).