Multi-Encoder Transformer Architectures

• Multi-Encoder Transformer is a model architecture employing multiple encoder modules to process diverse inputs independently before fusion, preserving complementary representations.
• It supports varied encoder designs—parallel, hierarchical, or modality-specific—that address challenges in multilingual translation, speech recognition, and multimodal tasks.
• Empirical results demonstrate improved metrics like BLEU and WER, while trade-offs in compute and parameter sharing depend on the chosen fusion strategy.

Searching arXiv for papers on multi-encoder transformer architectures and closely related formulations. A Multi-Encoder Transformer is a family of Transformer architectures in which more than one encoder is deployed to process an input representation before fusion or prediction. In the literature, this includes same-source parallel encoders, language-specific encoders, hierarchical token-to-sentence-to-document stacks, tree-structured partially shared encoders, and dual encoders specialized for different dependency axes such as space and time. The common design motive is to avoid forcing a single encoder to collapse heterogeneous evidence too early, and instead to preserve multiple representational pathways that can be fused, reused, or specialized according to modality, language family, temporal context, or task structure (Burtsev et al., 2021, Galoğlu et al., 2023, Zhou et al., 2020, Guan et al., 22 Sep 2025, Wei et al., 2024).

1. Conceptual scope and definitional boundaries

Across the literature, “multi-encoder” does not denote a single canonical topology. In multilingual generation, it can mean separate or partially shared encoder stacks for different target languages, as in the Transformer Encoder Tree (TET), where paths from a shared root to language-specific leaves constitute different encoder stacks (Guan et al., 22 Sep 2025). In code-switching ASR, it denotes symmetric language-specific encoders whose outputs are fused in a shared decoder (Zhou et al., 2020). In document representation, it denotes distinct encoder modules operating at different granularities, such as a lower multilingual sentence encoder and an upper document transformer in HMDE (Galoğlu et al., 2023). In reinforcement learning, it denotes a Dual-Transformer Encoder with a Spatial Transformer and a Temporal Transformer (Wei et al., 2024).

The term is also used in architectures that process the same source sequence through multiple encoders with different inductive biases. The heterogeneous Multi-Encoder Transformer for NMT replaces a standard single encoder with up to five encoders—Self-Attention, LSTM, ConvS2S, Static Expansion, and FNet—arranged side-by-side and fed the same tokenized source sequence (Hu et al., 2023). Multi-Stream Transformers are closely related but explicitly distinguished from conventional multi-encoder systems: multi-encoder architectures in prior literature typically handle multi-source inputs, whereas Multi-Stream encodes a single source by branching after an initial joint layer into parallel streams that do not interact until the final merge (Burtsev et al., 2021).

This distinction separates multi-encoder models from several adjacent paradigms. They are not equivalent to mixture-of-experts, because many of the cited systems use fixed streams or branches and no routing; they are not equivalent to single-encoder plus multi-decoder multilingual systems, because the encoder side itself is replicated or hierarchically partitioned; and they are not restricted to multimodal fusion, since several examples process one source sequence through multiple same-source encoders or repeated passes (Guan et al., 22 Sep 2025, Hu et al., 2023, Gao et al., 2020).

2. Recurrent architectural patterns

Pattern	Representative system	Structural property
Same-source parallel branching	Multi-Stream Transformer (Burtsev et al., 2021)	Branch after one joint layer; merge only at the end
Heterogeneous same-source encoders	Multi-Encoder Transformer for NMT (Hu et al., 2023)	Base, LSTM, ConvS2S, Static Expansion, FNet in parallel
Language-specific encoders	MED Transformer (Zhou et al., 2020)	English and Mandarin encoders with shared decoder
Hierarchical granularity split	HMDE (Galoğlu et al., 2023)	Lower sentence encoder plus shallow document transformer
Tree-structured sharing	TET (Guan et al., 22 Sep 2025)	Root, internal, and leaf encoder paths reused by language family
Spatial/temporal decomposition	DTPPO (Wei et al., 2024)	Spatial Transformer followed by Temporal Transformer
Modality-specific encoders	MiTREE (Chen et al., 2024); M-TabNet (Mursil et al., 20 Apr 2025)	Separate encoders per modality, then joint fusion
Multi-pass tied encoders	MPT (Gao et al., 2020)	Repeated encoder passes with tied parameters and inter-pass routing

These patterns differ mainly in where specialization is introduced and how much parameter sharing is retained. Multi-Stream uses independent parameters in each stream and no cross-stream computation until the final merge (Burtsev et al., 2021). MED likewise uses two symmetric language-specific encoders and two language-specific source-target attention modules in the decoder (Zhou et al., 2020). By contrast, MPT repeats the encoder stack along a new multi-pass dimension while tying parameters across passes, so compute increases with the number of passes but parameter count does not (Gao et al., 2020). TET occupies an intermediate position: early and intermediate layers are shared along common root-to-branch paths, while leaf nodes are language-specific (Guan et al., 22 Sep 2025).

Hierarchical multi-encoder designs introduce specialization by granularity rather than by language or modality. HMDE initializes a lower transformer with LaBSE, takes the transformed representation of the special beginning-of-sequence token as the sentence embedding, prepends a document-level BOS token, and contextualizes sentence embeddings with a 2-layer upper transformer. This yields a token-to-sentence-to-document decomposition rather than a single flat encoder over all document tokens (Galoğlu et al., 2023). MiTREE and M-TabNet apply the same principle to multimodal inputs: each modality is encoded separately at its native structure, then fused downstream (Chen et al., 2024, Mursil et al., 20 Apr 2025).

3. Fusion mechanisms and information flow

The defining operation in a Multi-Encoder Transformer is not merely duplication of encoders, but the rule by which their outputs are combined. In Multi-Stream Transformers, the encoder is decomposed into an input layer, parallel streams, and a final output layer. If $Z_{\mathrm{in}}$ is the shared input-layer output and $Z_s$ is the output of stream $s$, then the merge is

$$Z_{\mathrm{merge}}=\sum_{s=1}^{S} Z_s + Z_{\mathrm{in}},$$

followed by

$$Z_{\mathrm{out}} = L_{\mathrm{out}}(Z_{\mathrm{merge}}).$$

The skip from $Z_{\mathrm{in}}$ to $Z_{\mathrm{merge}}$ is a simple residual addition with no gating or learned weights, and the paper reports that this improves learnability and allows the final layer to factor in the initial joint representation (Burtsev et al., 2021).

Other systems fuse encoder outputs inside the decoder. In the MED Transformer for code-switching ASR, the decoder attends to each language-specific encoder separately with language-specific multi-head attention modules, forms residual outputs $RC^{(\mathrm{Eng})}$ and $RC^{(\mathrm{Man})}$, and uses simple mean fusion,

$$\mathrm{MidLyr}=\frac{RC^{(\mathrm{Eng})}+RC^{(\mathrm{Man})}}{2}.$$

The fusion is therefore decoder-side and residual-level, rather than a shared encoder-memory sum (Zhou et al., 2020). MEL for Transformer-based ASR performs training-time weighted fusion of two encoder-decoder attention outputs,

$Z_s$0

but uses only one encoder at inference, so the multi-encoder effect is confined to training while test-time runtime and parameter count remain unchanged (Lohrenz et al., 2021).

A separate class of models uses cross-attention or hierarchical reuse rather than direct addition. The multi-scale cross-attention event-classification architecture deploys two jet-substructure encoders and one kinematics encoder, then integrates them with an additional transformer encoder with cross-attention heads, where queries come from jet-derived summaries and keys and values from the kinematic encoder (Hammad et al., 2023). DTPPO does not use cross-attention between its two encoders; instead, the Temporal Transformer consumes the sequence of Spatial Transformer outputs, and the actor uses the Temporal Transformer output with a residual self-observation shortcut (Wei et al., 2024). TET does not fuse branches at all during encoding: root activations are reused by all branches beneath the root, internal node activations are reused by all leaves under that node, and each leaf computes its remaining layers and a CTC distribution in parallel (Guan et al., 22 Sep 2025).

MPT generalizes information flow in another direction. It repeats the encoder stack along a multi-pass dimension, permits inter-pass edges only from earlier to later passes to preserve a DAG, and considers both soft weighted aggregation and hard searched connections between layers of adjacent passes. This suggests that “multi-encoder” can include parameter-tied repeated encoders when the main objective is to let earlier layers in later passes process information in light of deeper representations from earlier passes (Gao et al., 2020).

4. Objectives, optimization, and computational properties

Training objectives vary with the downstream task. Machine translation systems in the cited NMT work use standard sequence-to-sequence cross-entropy, with label smoothing in the heterogeneous multi-encoder NMT study (Burtsev et al., 2021, Hu et al., 2023). TET is encoder-only and non-autoregressive, trained with CTC, using per-language heads over vocabulary plus blank and updating shared layers once per language in each minibatch (Guan et al., 22 Sep 2025). MED fine-tunes on code-switching data with a joint CTC+attention objective,

$Z_s$1

with $Z_s$2 (Zhou et al., 2020). HMDE pretrains with a cross-lingual contrastive objective using Wikipedia-derived positives, in-batch negatives, and category-based hard negatives (Galoğlu et al., 2023). DTPPO combines the PPO clipped surrogate, critic loss, entropy term, and a dynamic-predictor MSE term in

$Z_s$3

with $Z_s$4, $Z_s$5, and $Z_s$6 (Wei et al., 2024). MiTREE uses binary cross-entropy for multi-label encounter-rate prediction over 670 bird species (Chen et al., 2024), while M-TabNet predicts birth weight by regression and derives low-birth-weight classification by thresholding the regression output at 2500 grams (Mursil et al., 20 Apr 2025).

The computational effect of multiple encoders depends on how sharing is arranged. Multi-Stream keeps total encoder depth fixed, so parameter count and FLOPs remain comparable to the single-stream baseline at the same depth, and the additional merge is a negligible elementwise sum (Burtsev et al., 2021). HMDE reduces flat token-level quadratic cost by applying the lower encoder independently to sentences and then a shallow transformer over sentence vectors, yielding complexity roughly $Z_s$7 rather than $Z_s$8 (Galoğlu et al., 2023). TET reduces redundant multilingual computation by sharing intermediate paths: for 8 languages and depth 6 per language, separate per-language encoders require $Z_s$9 layer applications, whereas the TET tree uses 24 layer applications to produce all 8 outputs (Guan et al., 22 Sep 2025).

Other variants trade efficiency for specialization. The heterogeneous multi-encoder NMT model increases parameters and GFLOPs as encoders are added: in low-resource setups, the single-encoder model has $s$0 GFLOPs and $s$1M parameters, whereas the quintuple-encoder has $s$2 GFLOPs and $s$3M (Hu et al., 2023). MED roughly doubles encoder-side parameters and decoder cross-attention parameters relative to a single-encoder Transformer, because two encoders and two language-specific source-target attention modules are active (Zhou et al., 2020). MPT preserves parameter memory through weight tying but multiplies encoder compute by the number of passes (Gao et al., 2020). MEL is unusual in that the additional encoder exists only during training; at inference, only the magnitude feature encoder is used, so runtime and parameters match the single-stream baseline (Lohrenz et al., 2021).

5. Empirical behavior across domains

In machine translation, multi-encoder variants consistently target either ambiguity preservation or heterogeneous inductive bias. On WMT-14 DE–EN, the 4-layer Multi-Stream 2(1) + skip model reaches BLEU-4 19.46, compared with 18.99 for the baseline Transformer, and the 6-layer Multi-Stream 4(1) + skip reaches 25.63, compared with 24.65 for the baseline; the paper also reports faster convergence and a more even distribution of attention-pattern types (Burtsev et al., 2021). The heterogeneous Multi-Encoder Transformer reports that dual-encoder systems improve over the single encoder across all tested datasets, and that the best gain appears in low-resource Spanish–English with 34.38 BLEU for the quadruple-encoder against 27.22 for the single encoder, a maximum increase of 7.16 BLEU; Galician–English likewise improves from 13.03 to 18.38 (Hu et al., 2023). MPT shows that Base Transformer equipped with multi-pass routing can reach 28.40 BLEU on En-De and 41.80 on En-Fr with Base-level parameter counts, matching or surpassing Large Transformer in the reported comparison (Gao et al., 2020). TET, aimed at multilingual MT and ST, reports Multi30K average WER 16.9 for TET versus 35.6 for TEnc/lang and 24.9 for TEnc/all, and Tatoeba average WER 46.7 for TET versus 57.2 and 54.2, with explicit gains for Romanian and Polish in the Tatoeba setting (Guan et al., 22 Sep 2025).

In speech recognition, the main empirical pattern is that language- or stream-specific encoders help when the shared input space is heterogeneous. On SEAME, the MED Transformer with MOL+LM obtains 16.7 TER on eval_man and 23.1 on eval_sge, improving over the main single-encoder baseline at 18.6 and 25.9 by 10.2% and 10.8% relative (Zhou et al., 2020). MEL reaches 3.40% WER on WSJ eval92 with LM in the MEL-t-Fusion-Late configuration, compared with the prior best Transformer-based 4.20% WER cited in the paper, a 19% relative reduction. On LibriSpeech, MEL-t-Fusion-Late reduces WER from 4.05 to 3.34 on test clean and from 8.14 to 7.15 on test other relative to the standard magnitude-only Transformer (Lohrenz et al., 2021).

In document representation and retrieval, the principal advantage comes from hierarchical decomposition. HMDE reaches average 86.8 accuracy on cross-lingual topical document classification across non-English target languages, compared with 86.1 for mLongformer, and average MAP 0.249 on cross-lingual document retrieval, compared with 0.186 for Vanilla LaBSE and 0.109 for mLongformer. The paper emphasizes that the model generalizes to languages unseen during document-level pretraining because the lower encoder is massively multilingual (Galoğlu et al., 2023).

In multimodal and scientific settings, multi-encoder structures are used to preserve native resolutions, ecological context, or modality-specific sparsity. MiTREE encodes satellite imagery, pedologic rasters, bioclimate rasters, and ecoregion labels separately, and on SatBird Summer reaches MAE 2.070, MSE 0.630, Top-10 47.380%, Top-30 66.609%, and Top-k 67.821%, outperforming the ResNet18 baseline at MAE 2.134, MSE 0.642, Top-10 46.561%, Top-30 65.744%, and Top-k 67.121%; on Winter it reaches MAE 1.621 and MSE 0.430 against 1.658 and 0.450 for ResNet18 (Chen et al., 2024). DTPPO reports Average Transfer Reward 256.19 in Scene-I at 10% obstacle density, compared with 168.39 for MAPPO, and Average Collision Penalty 2.56 in Scene-II at 50% density, compared with 5.80 for MAPPO (Wei et al., 2024). M-TabNet, a multimodal tabular multi-encoder model, reports MAE = 122.3 g and $s$4 on the in-house cohort, with sensitivity = 97.55% and specificity = 94.48%; on the external IEEE dataset it reports MAE = 105.4 g and $s$5 (Mursil et al., 20 Apr 2025).

6. Misconceptions, limitations, and open directions

A common misconception is that a Multi-Encoder Transformer necessarily means multi-source input. The literature shows otherwise. Multi-Stream, the heterogeneous NMT model, HMDE, and MPT all process a single source through multiple encoders, streams, or passes; what varies is whether the purpose is ambiguity preservation, inductive-bias diversification, hierarchical abstraction, or feedback-like re-encoding (Burtsev et al., 2021, Hu et al., 2023, Galoğlu et al., 2023, Gao et al., 2020). Another misconception is that multi-encoder designs are equivalent to MoE. TET explicitly contrasts its static tree topology with MoE and adapter sharing, and the heterogeneous NMT system likewise uses parallel encoders with simple sum or concatenation rather than routing (Guan et al., 22 Sep 2025, Hu et al., 2023).

The main limitations arise from fusion simplicity, topology choice, and compute scaling. Multi-Stream without the skip can underperform and converge slower in deeper settings, while baseline-with-skip training instability appears after 10–15 epochs in the 4-layer case (Burtsev et al., 2021). In the heterogeneous NMT study, quintuple-encoder models often reduce BLEU, and FNet combinations are consistently poor under naive fusion (Hu et al., 2023). MED ablations show that language-specific decoder attentions alone are insufficient; improvements come primarily from language-disentangled encoder-side features (Zhou et al., 2020). TET-Rnd performs worse on average than linguistically grouped TET, and the model can underperform on some high-resource languages such as German and Italian in Tatoeba, indicating that sharing and specialization must be balanced (Guan et al., 22 Sep 2025). HMDE depends on sentence segmentation quality and inherits language coverage limits from LaBSE (Galoğlu et al., 2023). MiTREE reports that gains from the ecoregion encoder are modest but consistent (Chen et al., 2024). M-TabNet identifies a related failure mode in single-encoder multimodal tabular learning: attention can over-prioritize one data type and neglect others (Mursil et al., 20 Apr 2025).

The open research directions in the cited work converge on learned routing and more adaptive fusion. TET proposes dynamic trees and learned routing, sparse MoE within branches, semi-autoregressive heads, and tree optimization (Guan et al., 22 Sep 2025). The heterogeneous NMT study suggests learned gates or attention over encoders to replace naive sum or concatenation (Hu et al., 2023). DTPPO points to cross-attention or bidirectional fusion between spatial and temporal pathways, hierarchical encoders, and adaptive neighbor selection (Wei et al., 2024). The event-classification model notes that bidirectional fusion and explicit hierarchical tokens are natural next steps (Hammad et al., 2023). Taken together, this suggests that the central unresolved issue is no longer whether multiple encoders can help, but how to allocate specialization, sharing, and routing so that complementary structure is preserved without incurring destructive interference or unnecessary cost.