Dual-Path Decoder Training

• Dual-path decoder training is a framework that employs two complementary decoder branches sharing a common encoder to address varied task requirements.
• It encompasses designs like task-specialized, reverse-order, complementary restoration, and teacher/student decoders, each utilizing distinct loss coupling and feature transfer mechanisms.
• Empirical evidence across fields such as segmentation, ASR, and speech restoration demonstrates enhanced accuracy and robustness through this dual-decoding strategy.

Searching arXiv for recent and relevant papers on dual-path / dual-decoder training. arXiv search query: dual decoder training shared encoder auxiliary decoder attention arXiv Dual-path decoder training denotes a family of encoder–decoder training schemes in which a shared upstream representation is decoded by two coupled branches rather than by a single decoder. In the recent literature, the term does not identify one standardized algorithm. Instead, it covers several recurrent motifs: task-specialized parallel decoders, forward/backward sequence decoders, teacher/student decoders that separate differentiable and non-differentiable paths, and paired decoders for complementary targets such as masking versus mapping or magnitude versus complex-spectrum reconstruction. Early usage appears in DDNet, where a dual-path decoder separates occlusion boundary recovery from occlusion orientation recovery and combines multi-scale supervision with an Orthogonal Orientation Regression loss (Feng et al., 2019). Subsequent work broadened the pattern across segmentation, speech restoration, ASR, watermarking, and code-mixed language modeling (Tomar et al., 2020, Watzel et al., 2020, Sun et al., 2024, Baral et al., 19 May 2025).

1. Conceptual scope and main design families

The literature uses “dual-path decoder” in at least four distinct senses. First, two decoders may solve different but related tasks from a common encoder, as in DDANet’s segmentation and grayscale-reconstruction branches or PlutoNet’s main and auxiliary segmentation decoders (Tomar et al., 2020, Erol et al., 2022). Second, two decoders may encode opposite factorization orders over the same target sequence, as in forward/backward ASR training with left-to-right and right-to-left decoders (Watzel et al., 2020). Third, two decoders may represent complementary restoration operators, such as masking and mapping in the DM network or complex masking and complex spectral mapping in D2Former (Yang et al., 2024, Zhao et al., 2023). Fourth, dual decoding may be purely a training-time device, with only one branch retained or used operationally at inference, as in PlutoNet, the Regularized Forward-Backward Decoder, and END$^2$ (Erol et al., 2022, Watzel et al., 2020, Sun et al., 2024).

Family	Representative papers	Core coupling mechanism
Task-specialized auxiliary decoding	DDANet (Tomar et al., 2020), PlutoNet (Erol et al., 2022)	Auxiliary branch supervises or regularizes main branch
Reverse-order sequence decoding	Regularized Forward-Backward Decoder (Watzel et al., 2020)	Forward/backward CE plus output regularization
Complementary restoration targets	DM (Yang et al., 2024), D2Former (Zhao et al., 2023), Uformer (Fu et al., 2021)	Masking/mapping or complex/magnitude fusion
Teacher/student robustness decoding	END$^2$ (Sun et al., 2024)	Stop-gradient distorted path plus feature alignment
Synchronized bilingual decoding	CMLFormer (Baral et al., 19 May 2025)	Decoder–decoder cross-attention

A plausible implication is that “dual-path decoder training” is best treated as an umbrella term for coupled decoder optimization rather than as a fixed architectural template. This reading is reinforced by DPCFCS-Net, which explicitly uses dual-path processing in the bottleneck rather than proposing a specialized dual-path decoder training algorithm (Wang, 2023).

2. Shared architectural principles

Most instances share a single encoder or latent backbone and delay branching until relatively late in the model. DDANet uses one shared encoder with 4 encoder blocks and two decoders with 4 decoder blocks each; the main branch predicts the polyp mask, while the auxiliary branch reconstructs a grayscale image and emits an attention map that modulates the segmentation path (Tomar et al., 2020). PlutoNet likewise uses a shared EfficientNetB0 encoder and two heterogeneous decoders: a modified partial decoder for multi-scale salient detail and a lightweight auxiliary decoder built from multiplicative interactions over high-level features (Erol et al., 2022). In multilingual ASR, a Conformer encoder feeds a phoneme decoder, a grapheme decoder, a CTC branch, and an encoder-side language classifier (N, 2021).

A second recurring principle is decoder specialization rather than redundancy. In the DM network, the two magnitude decoders perform different operations—mask estimation for suppression and direct spectral mapping for restoration—even though they “share the same structure and parameters, except for the final activation function” (Yang et al., 2024). D2Former separates complex masking and complex spectral mapping into two fully complex dual-path decoders over a common encoder–conformer trunk, then fuses their outputs as $\hat{S}=\alpha \hat{S}'+\beta \hat{S}''$ (Zhao et al., 2023). Uformer makes the specialization domain-explicit: a complex-branch decoder predicts a CRM-like output, and a magnitude-branch decoder predicts an IRM-like mask, with layerwise cross-domain fusion in both encoder and decoder (Fu et al., 2021).

A third design axis is whether both decoders survive into inference. Some models keep both branches active at test time, including DM2, D2Former, Uformer, and CMLFormer (Yang et al., 2024, Zhao et al., 2023, Fu et al., 2021, Baral et al., 19 May 2025). Others are training-only regularizers: PlutoNet discards the auxiliary decoder, the forward-backward ASR model removes the backward decoder during decoding, and END$^2$ uses teacher/student dual decoders to train robustness while relying on the clean teacher path for encoder gradients (Erol et al., 2022, Watzel et al., 2020, Sun et al., 2024).

3. Optimization patterns and loss design

The dominant optimization pattern is joint end-to-end training with branch-specific supervision. DDANet applies binary cross-entropy plus Dice loss to the segmentation output and binary cross-entropy to grayscale reconstruction, thereby training both branches simultaneously without any reported alternating schedule (Tomar et al., 2020). PlutoNet uses a Dice-style supervised loss on the main decoder output, a Dice-style consistency loss between main and auxiliary predictions, and a total objective $L=L_s+\alpha L_c$; the auxiliary branch is therefore regularized primarily through agreement rather than by its own direct ground-truth loss (Erol et al., 2022). In the multilingual Conformer, the combined objective is $$\mathcal{L}=\alpha \mathcal{L}_{ctc}+\beta \mathcal{L}_{pr}+\gamma \mathcal{L}_{gr}+\pi \mathcal{L}_{lid}$$ with reported weights $\alpha=0.3$, $\beta=0.5$, $\gamma=0.5$, and $\pi=10.0$ (N, 2021).

A second pattern couples decoder outputs explicitly rather than merely summing task losses. The Regularized Forward-Backward Decoder augments weighted forward and backward cross-entropies with an agreement term,

$^2$0

using an $^2$1 regularizer for equal-length character sequences and soft-DTW for unequal-length BPE sequences (Watzel et al., 2020). END$^2$2 aligns teacher and student decoder features on a hypersphere while keeping the distorted branch under stop-gradient, with total loss

$^2$3

defaulting to $^2$4, $^2$5, and $^2$6 (Sun et al., 2024).

A third pattern is hard routing: only one decoder is updated per example. In the sentiment-controlled response model, the loss is $^2$7 with $^2$8 for positive labels and $^2$9 for negative labels, so the non-target decoder is inactive for that sample (Wu et al., 2019). This is structurally different from joint multitask training because branch competition is eliminated at the sample level.

Finally, some works rely on staged optimization. The forward-backward ASR model pretrains a forward decoder, then trains a backward decoder with the encoder frozen, and only then jointly optimizes the combined architecture (Watzel et al., 2020). By contrast, DDANet, PlutoNet, DM, D2Former, Uformer, CMLFormer, and the multilingual Conformer are described as jointly optimized from the outset (Tomar et al., 2020, Yang et al., 2024, Zhao et al., 2023, Fu et al., 2021, Baral et al., 19 May 2025, N, 2021).

4. Inter-branch communication mechanisms

Decoder coupling varies from weak shared supervision to direct cross-branch information flow. DDANet implements unidirectional feature transfer: the auxiliary autoencoder branch applies a $\hat{S}=\alpha \hat{S}'+\beta \hat{S}''$0 convolution and sigmoid to produce an attention map, multiplies that map with segmentation decoder features, and feeds the result into the next segmentation decoder block (Tomar et al., 2020). PlutoNet instead couples at the prediction level: the main and auxiliary decoders produce segmentation maps $\hat{S}=\alpha \hat{S}'+\beta \hat{S}''$1 and $\hat{S}=\alpha \hat{S}'+\beta \hat{S}''$2, and consistency is enforced directly between those outputs rather than through intermediate feature exchange (Erol et al., 2022).

In restoration models, coupling frequently occurs through skip-style guidance. DM2 uses the masking decoder’s enhanced magnitude as a learnable skip connection into the mapping decoder output, effectively replacing harmful direct input-to-output skips in distorted conditions. The paper describes the integrated output as the mapping estimate plus $\hat{S}=\alpha \hat{S}'+\beta \hat{S}''$3 times the masking estimate, and sets $\hat{S}=\alpha \hat{S}'+\beta \hat{S}''$4 so the final output comes from the skip-fused mapping path rather than from external averaging (Yang et al., 2024). D2Former uses a different interaction: both decoders share the encoder/backbone, produce distinct complex outputs, and are fused only at the final enhanced spectrum by fixed weights $\hat{S}=\alpha \hat{S}'+\beta \hat{S}''$5 and $\hat{S}=\alpha \hat{S}'+\beta \hat{S}''$6 (Zhao et al., 2023). Uformer is more tightly coupled, with branch interaction at every hybrid layer through explicit complex–magnitude fusion and again at the final magnitude estimate through averaging of complex-derived and magnitude-derived magnitudes (Fu et al., 2021).

Sequence models also exhibit stronger decoder–decoder coupling. CMLFormer introduces synchronized decoder cross-attention,

$\hat{S}=\alpha \hat{S}'+\beta \hat{S}''$7

so the base-language and mixing-language decoders run in lockstep and exchange same-layer hidden states before feed-forward processing (Baral et al., 19 May 2025). The forward-backward ASR model does not cross-attend, but its regularizer forces alignment between forward and backward decoder outputs, which serves a comparable coordination role (Watzel et al., 2020). END$\hat{S}=\alpha \hat{S}'+\beta \hat{S}''$8 pushes this idea into robustness training: clean and distorted decoders are kept structurally identical, their intermediate vectors are projected by a shared bias-free layer, normalized, and aligned by cosine similarity, while momentum updating and role swapping transfer robustness from the student path back to the teacher path (Sun et al., 2024).

These designs suggest a useful technical distinction between architectural dual decoding and purely loss-level dual decoding. In some models the second decoder changes the first decoder’s hidden-state trajectory; in others it only changes the gradient field.

5. Empirical evidence across domains

The evidence for dual-path decoder training is strongest when a paper includes either controlled ablations or direct single-decoder comparisons. In polyp segmentation, DDANet reports on Kvasir-SEG a DSC of 0.8576, mIoU of 0.7800, recall of 0.8880, precision of 0.8643, and FPS of 69.59; on an unseen challenge dataset it reports DSC 0.7874, mIoU 0.7010, recall 0.7987, precision 0.8577, and FPS 70.23 (Tomar et al., 2020). PlutoNet provides a more targeted ablation: on Kvasir, adding decoder consistency changes Dice from 0.8839 to 0.8954 and precision from 0.9250 to 0.9559, while on ColonDB precision rises from 0.7923 to 0.8845 even though Dice decreases from 0.7178 to 0.6935, indicating a more conservative false-positive profile (Erol et al., 2022).

Speech restoration papers provide especially explicit decoder-combination evidence. In the DM network, the naive unified model $\hat{S}=\alpha \hat{S}'+\beta \hat{S}''$9 yields PESQ 2.67 and LSD 3.95, whereas DM1 improves to PESQ 2.60 and LSD 2.37, and DM2 further improves to PESQ 2.61, STOI 0.92, SRMR 9.81, and LSD 2.24; the masking-to-mapping skip therefore improves bandwidth restoration without sacrificing the denoising role of masking (Yang et al., 2024). D2Former reports its best trade-off at $^2$0 with WB-PESQ 3.43, CSIG 4.66, CBAK 3.94, COVL 4.15, STOI 0.96, and 0.87M parameters (Zhao et al., 2023). Uformer reports DNSMOS 3.6032 on the blind test set of the Interspeech 2021 DNS Challenge and attributes gains to hybrid complex–magnitude interaction plus encoder–decoder attention (Fu et al., 2021).

In sequence modeling, the forward-backward decoder provides a clean comparison between auxiliary dual decoding and explicitly regularized dual decoding. On TEDLIUMv2, forward-only character WER drops from 17.32 to 15.94 under Dual Decoder Reg; on LibriSpeech character test-clean and test-other improve from 7.72 to 7.02 and from 21.63 to 20.95, respectively (Watzel et al., 2020). In multilingual ASR, the Conformer grapheme baseline reports weighted average WER 28.10, adding the phoneme decoder gives 27.67, and adding language identification yields 26.70, which supports the claim that the second decoder acts as useful auxiliary supervision rather than mere parameter inflation (N, 2021). In non-differentiable watermarking, END$^2$1 reports ACC 94.55, PSNR 45.62, and SSIM 0.9897 under random distortions, and its ablation shows ACC rising from 78.93 with none of the dual-decoder mechanisms to 94.55 when feature alignment, momentum updating, and swapping learning are combined (Sun et al., 2024).

Not all papers isolate the effect equally well. DDANet and D2Former report strong system-level results but do not provide full factorial ablations separating decoder count, coupling type, and loss weighting (Tomar et al., 2020, Zhao et al., 2023). The resulting evidence is persuasive at the architecture level but less definitive at the level of causal mechanism.

6. Methodological caveats and open questions

A persistent misconception is that dual-path decoder training necessarily implies two equally important inference branches. This is false in several prominent instances. PlutoNet’s auxiliary decoder is “only needed for training,” the forward-backward ASR model removes the right-to-left decoder during decoding, and the multilingual dual-decoder Conformer reports final inference from CTC plus a 3-gram LLM rather than from either decoder output (Erol et al., 2022, Watzel et al., 2020, N, 2021). Conversely, DM2, D2Former, Uformer, and CMLFormer depend on both decoder paths at test time (Yang et al., 2024, Zhao et al., 2023, Fu et al., 2021, Baral et al., 19 May 2025).

A second misconception is that “dual-path” always refers to decoder branching. DPCFCS-Net explicitly places the dual-path mechanism in the intermediate enhancement layer and states that it does not introduce a specially trained dual-path decoder (Wang, 2023). SepTDA similarly combines dual-path mixture modeling with a transformer decoder-based attractor and triple-path refinement; its contribution is joint training of a dual-path front end with a decoder-based attractor, not a decoder split into two paths (Lee et al., 2024). This suggests terminological caution is necessary when comparing papers across subfields.

A third issue is under-specification. DDANet does not report exact total-loss weights or batch size; PlutoNet does not report the numeric value of the consistency weight $^2$2; DM does not print explicit branch-loss equations; D2Former does not provide standalone mask-only and mapping-only baselines under identical settings (Tomar et al., 2020, Erol et al., 2022, Yang et al., 2024, Zhao et al., 2023). These omissions complicate strict reproducibility and make cross-paper comparisons partly qualitative.

The most defensible general conclusion is therefore limited but clear: dual-path decoder training is most technically justified when the two branches embody genuinely different inductive biases—detail versus semantics, past versus future context, suppression versus generation, clean versus distorted decoding, or constituent-language translations—and when their interaction is made explicit through loss coupling, feature transfer, or synchronized attention. A plausible implication is that the success of the paradigm depends less on decoder multiplicity per se than on whether the second path contributes complementary information that a single decoder would represent poorly or only inefficiently.