Dual-Output L2 Speech Recognition

• The paper shows that naive multi-task learning with a shared encoder improves meaning transcription while degrading surface transcription due to encoder-level representational entanglement.
• Dual-output L2 ASR maps a single second-language speech signal into both surface-level and meaning-oriented transcriptions, highlighting a direct trade-off between phonetic detail and canonical normalization.
• Empirical evaluations on Korean and English datasets reveal that the severity of surface degradation and meaning improvement varies with language-dependent divergence and encoder entanglement.

Searching arXiv for recent and foundational papers relevant to dual-output L2 ASR, dual-decoder architectures, and L2/accent-aware ASR. Search query: "dual-output second language speech recognition representational entanglement dual-output L2 ASR" Dual-output L2 automatic speech recognition denotes an ASR setting in which a single second-language speech signal is mapped to two distinct textual outputs: a surface-level transcription, representing what was actually pronounced, and a meaning-oriented transcription, representing the intended canonical form or standard orthography. In the formulation studied by "Multi-task Learning is Not Enough: Representational Entanglement in Dual-output Second Language Speech Recognition" (Cho et al., 4 Jun 2026), the input is a log mel-spectrogram $X \in \mathbb{R}^{T \times 80}$, and the central question is whether a shared-encoder multi-task learning (MTL) architecture can support both outputs without degrading either. The paper shows that naïve MTL is not uniformly beneficial: it improves meaning-oriented recognition in both Korean and English, but degrades surface transcription, especially in English, and links this pattern to encoder-level representational entanglement (Cho et al., 4 Jun 2026).

1. Task formulation and representational targets

Dual-output L2 ASR is defined by the simultaneous prediction of two token sequences from one acoustic input. The surface-level transcription $y^{\text{surf}}$ is the verbatim spoken form, including reductions, mispronunciations, and L1-influenced realizations. The meaning-oriented transcription $y^{\text{mean}}$ is the intended canonical written form, ignoring pronunciation deviations. Both are tokenized over a vocabulary $\mathcal{V}$; for Korean, the relevant units are character-level syllables, while for English the divergence analysis is word-level, although ASR evaluation is reported with character error rate (CER) (Cho et al., 4 Jun 2026).

This task departs from conventional single-output ASR because the two supervision streams are neither independent nor identical. They share high-level linguistic content, but they diverge precisely where L2 production differs from canonical form. A common assumption is that related tasks should benefit from shared representations under MTL. The principal result of (Cho et al., 4 Jun 2026) is that this assumption does not hold uniformly across languages: shared representations can improve meaning-oriented decoding while simultaneously harming faithful surface reconstruction.

A recurrent misconception is that the two outputs merely duplicate orthographic and phonetic views of the same utterance. The evidence in (Cho et al., 4 Jun 2026) indicates a stricter distinction. Surface transcription is not treated as an auxiliary pronunciation label; it is a first-class recognition target whose preservation depends on the encoder retaining fine-grained acoustic-phonological detail. Meaning transcription, by contrast, rewards abstraction away from pronunciation variation. The two outputs therefore impose partially conflicting representational pressures on a shared acoustic encoder.

2. Data, divergence structure, and evaluation regime

The comparative study in (Cho et al., 4 Jun 2026) uses two AI-Hub L2 datasets. The Korean corpus, “Educational Korean Audio Data Recorded by Native (L1) Chinese and Japanese Speakers,” contains 41,803 utterances, partitioned into 33,442 train, 4,180 dev, and 4,181 test. The English corpus, “Educational English Audio Data Recorded by L1 Korean Speakers,” contains 72,022 utterances, partitioned into 57,616 train, 7,199 dev, and 7,207 test. In both corpora, each utterance has aligned surface and meaning transcriptions (Cho et al., 4 Jun 2026).

A key variable is surface–meaning divergence, quantified by Levenshtein edit distance (ED) between the two transcription forms. Korean uses character-level syllable ED; English uses word-level ED. The distributions are non-trivial but broadly similar. In Korean, ED $=0$ accounts for 38.8%, ED $=1\text{–}3$ for 35.4%, ED $=4\text{–}10$ for 20.6%, and ED $\ge 11$ for 5.2%. In English, ED $=0$ accounts for 26.1%, ED $=1\text{–}3$ for 40.3%, ED $y^{\text{surf}}$0 for 29.5%, and ED $y^{\text{surf}}$1 for 4.2% (Cho et al., 4 Jun 2026).

Language	Corpus size and split	SO vs DO Conformer CER (%)
Korean	41,803; 33,442 / 4,180 / 4,181	SO: surface 11.14, meaning 1.60; DO: surface 11.34, meaning 0.77
English	72,022; 57,616 / 7,199 / 7,207	SO: surface 13.78, meaning 3.87; DO: surface 15.08, meaning 3.19

CER is the evaluation metric for both tasks. The study trains for 50 epochs with batch size 8, uses AdamW with weight decay 0.01, applies SpecAugment, and decodes with beam search of size 5. The learning rate is $y^{\text{surf}}$2 for Conformer and $y^{\text{surf}}$3 for Whisper fine-tuning, and the reported experiments were run on a single NVIDIA RTX 3090 (Cho et al., 4 Jun 2026).

These corpus statistics matter because the subsequent language contrast cannot be reduced to a trivial difference in divergence frequency. The paper explicitly notes that the two corpora have broadly similar ED distributions, yet exhibit markedly different MTL behavior. This suggests that distributional similarity in divergence magnitude is not sufficient to guarantee similar representational dynamics (Cho et al., 4 Jun 2026).

3. Architectural formulation: single-output and dual-output models

The baseline comparison in (Cho et al., 4 Jun 2026) is between single-output (SO) models, trained separately for surface or meaning transcription, and a dual-output (DO) model trained jointly. All Conformer-based models are end-to-end hybrid CTC–attention ASR systems. Each SO system uses a Conformer encoder, a single Transformer decoder, and an auxiliary CTC head on encoder outputs. Separate SO models are trained for Korean surface, Korean meaning, English surface, and English meaning. Whisper-base and Whisper-small are also fine-tuned as SO baselines (Cho et al., 4 Jun 2026).

The DO model uses one shared Conformer encoder, two separate Transformer decoders, and one auxiliary CTC head on the encoder output. The crucial design choice is that the auxiliary CTC head is trained on surface-level targets, on the rationale that surface transcriptions preserve the monotonic frame-token alignment assumed by CTC. The dual loss combines CTC on surface targets with attention losses for the surface and meaning decoders, using weights fixed by pre-experiments: $y^{\text{surf}}$4 (Cho et al., 4 Jun 2026).

The main empirical comparison is therefore controlled: for each language and task, DO and SO models are contrasted under otherwise similar capacity and training conditions. The result is asymmetric. In Korean, the DO Conformer produces a small surface degradation and a large meaning improvement. In English, the DO Conformer produces a larger surface degradation and a smaller but still clear meaning improvement (Cho et al., 4 Jun 2026).

This model class belongs to a broader family of shared-encoder, multiple-decoder architectures. "Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translation" (Le et al., 2020) provides a directly related precedent in which a single speech encoder feeds two decoders, one for ASR and one for speech translation, with explicit cross-decoder interaction. A plausible implication is that dual-output L2 ASR can be situated within a general dual-decoder design space, but (Cho et al., 4 Jun 2026) shows that simple parameter sharing is insufficient when the two outputs compete for incompatible encoder structure.

4. Empirical trade-offs between surface and meaning recognition

The central quantitative result is that MTL improves meaning but degrades surface. For Korean, the SO Conformer achieves 11.14 CER on surface and 1.60 on meaning, while the DO Conformer achieves 11.34 on surface and 0.77 on meaning. For English, the SO Conformer achieves 13.78 on surface and 3.87 on meaning, while the DO Conformer achieves 15.08 on surface and 3.19 on meaning. Thus the absolute surface degradation is $y^{\text{surf}}$5 in Korean and $y^{\text{surf}}$6 in English, while the meaning improvement is $y^{\text{surf}}$7 in Korean and $y^{\text{surf}}$8 in English (Cho et al., 4 Jun 2026).

Across all reported models, surface transcription is consistently harder than meaning transcription, and scaling model size improves both tasks but does not remove the surface–meaning gap. The Whisper SO baselines reinforce this pattern. In Korean, Whisper-base yields surface 10.05 and meaning 4.62, while Whisper-small yields surface 6.76 and meaning 0.54. In English, Whisper-base yields surface 11.39 and meaning 0.55, while Whisper-small yields surface 11.20 and meaning 0.27 (Cho et al., 4 Jun 2026).

The divergence-stratified analysis sharpens the surface–meaning trade-off. Test utterances are binned by ED, and the paper computes the CER gap

$y^{\text{surf}}$9

For Korean, surface $y^{\text{mean}}$0 remains small and shows no clear monotonic dependence on ED, ranging from $y^{\text{mean}}$1 to $y^{\text{mean}}$2 for ED $y^{\text{mean}}$3 and becoming slightly negative for ED $y^{\text{mean}}$4; meaning $y^{\text{mean}}$5 improves modestly, reaching about $y^{\text{mean}}$6 at ED $y^{\text{mean}}$7. For English, by contrast, surface $y^{\text{mean}}$8 increases monotonically with ED, from $y^{\text{mean}}$9 at ED $\mathcal{V}$0 to $\mathcal{V}$1 at ED $\mathcal{V}$2, while meaning $\mathcal{V}$3 decreases monotonically from $\mathcal{V}$4 to $\mathcal{V}$5 over the same bins (Cho et al., 4 Jun 2026).

The English result is especially important because it shows that the penalty is not a generic MTL effect but a divergence-sensitive one. The greater the difference between actual pronunciation and intended form, the more the shared encoder appears to favor meaning-oriented abstraction at the expense of surface fidelity. This directly contradicts any strong claim that related-output MTL should automatically help both outputs.

5. Representational entanglement and encoder–decoder dynamics

The explanatory core of (Cho et al., 4 Jun 2026) is representational entanglement, especially at the encoder level. The paper defines encoder-level entanglement as the situation in which encoder representations for the two tasks are not separable and become nearly identical despite different supervision targets. More generally, representational entanglement denotes a lack of task-specific structure in the learned latent space (Cho et al., 4 Jun 2026).

To analyze this, the study uses Centered Kernel Alignment (CKA) following Kornblith et al. (2019), comparing layer-wise similarity across SO and DO encoders and decoders. In Korean, the encoder CKA between the surface-only and meaning-only SO models drops sharply in deeper layers, from 0.95 at Layer 0 to 0.45 at Layer 9, showing that the two SO encoders learn distinct task-specific representations. The DO encoder remains highly similar to the surface SO encoder in deep layers, reaching 0.81 at Layer 11, while maintaining moderate similarity to the meaning SO encoder, 0.62 at Layer 11 (Cho et al., 4 Jun 2026).

In English, the encoder story is different. The CKA between the surface-only and meaning-only SO encoders remains high through most layers, including 0.89 at Layer 3 and 0.75 at Layer 6, falling to 0.40 only at the final layer. The paper interprets this as evidence that even separately trained SO encoders already learn very similar representations, which is a signature of intrinsic encoder-level entanglement (Cho et al., 4 Jun 2026).

Decoder analysis adds an asymmetry. In Korean, SO decoders diverge into task-specific spaces, and DO decoders remain closer to their same-task SO counterparts than to cross-task ones. In English, the DO meaning decoder departs strongly from the SO meaning decoder: $\mathcal{V}$6 collapses to 0.24 at the final layer, while the cross-task similarity $\mathcal{V}$7 is higher at 0.44 at Layer 7. The paper interprets this as the meaning decoder developing a fundamentally different representation, almost as if it is bypassing or reinterpreting the encoder’s entangled structure. By contrast, the surface DO decoder remains more constrained by the encoder and by the CTC objective (Cho et al., 4 Jun 2026).

The resulting picture is not that both decoders fail equally. Rather, decoder-level adaptation is asymmetric. Meaning decoding can partially escape entanglement; surface decoding cannot recover fine-grained distinctions that the encoder has already collapsed. This is why the paper argues that decoder adaptation alone cannot fully compensate for encoder-level entanglement.

6. Language contrast and relation to adjacent research directions

The Korean–English contrast in (Cho et al., 4 Jun 2026) is empirical first and interpretive second. Empirically, Korean shows small surface degradation, strong meaning improvement, and weak ED dependence. English shows larger surface degradation, moderate meaning improvement, and strong ED-dependent trade-offs. Representationally, Korean exhibits clearer task separation in both SO encoders and SO decoders, whereas English exhibits stronger encoder entanglement even before MTL is introduced (Cho et al., 4 Jun 2026).

The paper suggests, without deeply theorizing linguistics, that Korean orthography may be more systematic and closer to phonemic structure, while English orthography may be deeper and more non-phonemic. This suggests that English surface and canonical forms place more divergent demands on a shared encoder. The study explicitly notes that similar ED distributions across languages do not yield similar MTL behavior, so the contrast is not purely distributional (Cho et al., 4 Jun 2026).

Adjacent work indicates several neighboring design directions. "On using 2D sequence-to-sequence models for speech recognition" (Bahar et al., 2019) formulates ASR with a 2DLSTM over a grid indexed by input time and output position, rather than explicit attention. This suggests an alternative way to model alignment-sensitive structure when surface realization departs from canonical form. "Incorporating L2 Phonemes Using Articulatory Features for Robust Speech Recognition" (Wang et al., 2023) uses a unified phoneme inventory, articulatory feature analysis, and LF-MMI with multiple pronunciation candidates to model Korean-accented English; this indicates that explicit pronunciation-variant modeling can improve L2 ASR without the standard L1–L2 performance trade-off. "Synthetic Audio Generation Framework for Air Traffic Control Speech Recognition" (Bagat et al., 19 Jun 2026) shows that accent conversion, voice conversion, and acoustical simulation can improve recognition in a heavily L2 domain, especially when synthetic L1→L2 accent diversity is added. "Balancing ASR and diarization in end-to-end LLMs for multi-talker speech recognition" (Zheng et al., 11 Jun 2026) demonstrates that multi-output speech systems may require architectural separation and loss-balancing mechanisms rather than undifferentiated sharing.

These works do not solve dual-output L2 ASR in the sense of (Cho et al., 4 Jun 2026), but together they mark out a broader design space: explicit alignment modeling, explicit pronunciation modeling, synthetic accent augmentation, and carefully balanced multi-output architectures.

7. Design implications and open research directions

The principal conclusion of (Cho et al., 4 Jun 2026) is that multi-task learning is not enough for dual-output L2 ASR when implemented as a shared encoder with dual decoders and a surface CTC head. The paper explicitly motivates several alternatives intended to mitigate encoder-level entanglement: partial sharing or split encoders, disentanglement or orthogonality constraints, task-specific bottlenecks or adapters, sparse decomposition or gating mechanisms, and adversarial training (Cho et al., 4 Jun 2026).

These proposals remain architectural directions rather than implemented solutions in the paper. Their common purpose is to preserve a shared acoustic substrate while preventing the encoder from collapsing surface-sensitive and meaning-sensitive structure into a single blended representation. In practical terms, the recommendation is not to abandon joint modeling, but to impose stronger structural constraints on how sharing occurs.

A second implication is methodological. The paper recommends monitoring performance as a function of surface–meaning divergence, for example through ED bins, rather than reporting only aggregate CER. It also shows the value of representation analysis such as layer-wise CKA for diagnosing whether a model is learning task-specific or entangled structure (Cho et al., 4 Jun 2026). This suggests that future work on dual-output L2 ASR should treat representation analysis as part of model evaluation rather than as a post hoc interpretive add-on.

A final implication is conceptual. Dual-output L2 ASR is not simply a case of “two related outputs from one encoder.” It is a setting in which one output rewards preservation of surface phonological detail and the other rewards normalization toward intended form. The evidence in (Cho et al., 4 Jun 2026) shows that these objectives can compete directly inside the encoder. The open problem is therefore not whether dual-output L2 ASR is feasible, but how to engineer encoder structure so that surface fidelity and meaning recovery can coexist without one being systematically sacrificed to the other.