---
title: Multi-Encoder Transformer Architectures
url: https://www.emergentmind.com/topics/multi-encoder-transformer
type: topic
---

# Multi-Encoder Transformer Architectures

Searching arXiv for recent papers on multi-encoder transformer architectures and closely related formulations.
A Multi-Encoder Transformer is a family of Transformer architectures in which more than one encoder is deployed to process an input representation before fusion or prediction. In the literature, this includes same-source parallel encoders, language-specific encoders, hierarchical token-to-sentence-to-document stacks, tree-structured partially shared encoders, and dual encoders specialized for different dependency axes such as space and time. The common design motive is to avoid forcing a single encoder to collapse heterogeneous evidence too early, and instead to preserve multiple representational pathways that can be fused, reused, or specialized according to modality, language family, temporal context, or task structure [2107.10342] [2305.07016] [2006.10414] [2509.17930] [2410.15205].

## 1. Conceptual scope and definitional boundaries

Across the literature, “multi-encoder” does not denote a single canonical topology. In multilingual generation, it can mean separate or partially shared encoder stacks for different target languages, as in the Transformer Encoder Tree (TET), where paths from a shared root to language-specific leaves constitute different encoder stacks [2509.17930]. In code-switching ASR, it denotes symmetric language-specific encoders whose outputs are fused in a shared decoder [2006.10414]. In document representation, it denotes distinct encoder modules operating at different granularities, such as a lower multilingual sentence encoder and an upper document transformer in HMDE [2305.07016]. In reinforcement learning, it denotes a Dual-Transformer Encoder with a Spatial Transformer and a Temporal Transformer [2410.15205].

The term is also used in architectures that process the same source sequence through multiple encoders with different inductive biases. The heterogeneous Multi-Encoder Transformer for NMT replaces a standard single encoder with up to five encoders—Self-Attention, LSTM, ConvS2S, Static Expansion, and FNet—arranged side-by-side and fed the same tokenized source sequence [2312.15872]. Multi-Stream Transformers are closely related but explicitly distinguished from conventional multi-encoder systems: multi-encoder architectures in prior literature typically handle multi-source inputs, whereas Multi-Stream encodes a single source by branching after an initial joint layer into parallel streams that do not interact until the final merge [2107.10342].

This distinction separates multi-encoder models from several adjacent paradigms. They are not equivalent to mixture-of-experts, because many of the cited systems use fixed streams or branches and no routing; they are not equivalent to single-encoder plus multi-decoder multilingual systems, because the encoder side itself is replicated or hierarchically partitioned; and they are not restricted to multimodal fusion, since several examples process one source sequence through multiple same-source encoders or repeated passes [2509.17930] [2312.15872] [2009.11382].

## 2. Recurrent architectural patterns

| Pattern | Representative system | Structural property |
|---|---|---|
| Same-source parallel branching | Multi-Stream Transformer [2107.10342] | Branch after one joint layer; merge only at the end |
| Heterogeneous same-source encoders | Multi-Encoder Transformer for NMT [2312.15872] | Base, LSTM, ConvS2S, Static Expansion, FNet in parallel |
| Language-specific encoders | MED Transformer [2006.10414] | English and Mandarin encoders with shared decoder |
| Hierarchical granularity split | HMDE [2305.07016] | Lower sentence encoder plus shallow document transformer |
| Tree-structured sharing | TET [2509.17930] | Root, internal, and leaf encoder paths reused by language family |
| Spatial/temporal decomposition | DTPPO [2410.15205] | Spatial Transformer followed by Temporal Transformer |
| Modality-specific encoders | MiTREE [2412.18995]; M-TabNet [2504.15312] | Separate encoders per modality, then joint fusion |
| Multi-pass tied encoders | MPT [2009.11382] | Repeated encoder passes with tied parameters and inter-pass routing |

These patterns differ mainly in where specialization is introduced and how much parameter sharing is retained. Multi-Stream uses independent parameters in each stream and no cross-stream computation until the final merge [2107.10342]. MED likewise uses two symmetric language-specific encoders and two language-specific source-target attention modules in the decoder [2006.10414]. By contrast, MPT repeats the encoder stack along a new multi-pass dimension while tying parameters across passes, so compute increases with the number of passes but parameter count does not [2009.11382]. TET occupies an intermediate position: early and intermediate layers are shared along common root-to-branch paths, while leaf nodes are language-specific [2509.17930].

Hierarchical multi-encoder designs introduce specialization by granularity rather than by language or modality. HMDE initializes a lower transformer with LaBSE, takes the transformed representation of the special beginning-of-sequence token as the sentence embedding, prepends a document-level BOS token, and contextualizes sentence embeddings with a 2-layer upper transformer. This yields a token-to-sentence-to-document decomposition rather than a single flat encoder over all document tokens [2305.07016]. MiTREE and M-TabNet apply the same principle to multimodal inputs: each modality is encoded separately at its native structure, then fused downstream [2412.18995] [2504.15312].

## 3. Fusion mechanisms and information flow

The defining operation in a Multi-Encoder Transformer is not merely duplication of encoders, but the rule by which their outputs are combined. In Multi-Stream Transformers, the encoder is decomposed into an input layer, parallel streams, and a final output layer. If $Z_{\mathrm{in}}$ is the shared input-layer output and $Z_s$ is the output of stream $s$, then the merge is
$$
Z_{\mathrm{merge}}=\sum_{s=1}^{S} Z_s + Z_{\mathrm{in}},
$$
followed by
$$
Z_{\mathrm{out}} = L_{\mathrm{out}}(Z_{\mathrm{merge}}).
$$
The skip from $Z_{\mathrm{in}}$ to $Z_{\mathrm{merge}}$ is a simple residual addition with no gating or learned weights, and the paper reports that this improves learnability and allows the final layer to factor in the initial joint representation [2107.10342].

Other systems fuse encoder outputs inside the decoder. In the MED Transformer for code-switching ASR, the decoder attends to each language-specific encoder separately with language-specific multi-head attention modules, forms residual outputs $RC^{(\mathrm{Eng})}$ and $RC^{(\mathrm{Man})}$, and uses simple mean fusion,
$$
\mathrm{MidLyr}=\frac{RC^{(\mathrm{Eng})}+RC^{(\mathrm{Man})}}{2}.
$$
The fusion is therefore decoder-side and residual-level, rather than a shared encoder-memory sum [2006.10414]. MEL for Transformer-based ASR performs training-time weighted fusion of two encoder-decoder attention outputs,
$$
y_\ell=\lambda\, y_\ell^{(\mathrm{mag})} + (1-\lambda)\, y_\ell^{(\mathrm{phase})},
$$
but uses only one encoder at inference, so the multi-encoder effect is confined to training while test-time runtime and parameter count remain unchanged [2104.00120].

A separate class of models uses cross-attention or hierarchical reuse rather than direct addition. The multi-scale cross-attention event-classification architecture deploys two jet-substructure encoders and one kinematics encoder, then integrates them with an additional transformer encoder with cross-attention heads, where queries come from jet-derived summaries and keys and values from the kinematic encoder [2401.00452]. DTPPO does not use cross-attention between its two encoders; instead, the Temporal Transformer consumes the sequence of Spatial Transformer outputs, and the actor uses the Temporal Transformer output with a residual self-observation shortcut [2410.15205]. TET does not fuse branches at all during encoding: root activations are reused by all branches beneath the root, internal node activations are reused by all leaves under that node, and each leaf computes its remaining layers and a CTC distribution in parallel [2509.17930].

MPT generalizes information flow in another direction. It repeats the encoder stack along a multi-pass dimension, permits inter-pass edges only from earlier to later passes to preserve a DAG, and considers both soft weighted aggregation and hard searched connections between layers of adjacent passes. This suggests that “multi-encoder” can include parameter-tied repeated encoders when the main objective is to let earlier layers in later passes process information in light of deeper representations from earlier passes [2009.11382].

## 4. Objectives, optimization, and computational properties

Training objectives vary with the downstream task. Machine translation systems in the cited NMT work use standard sequence-to-sequence cross-entropy, with label smoothing in the heterogeneous multi-encoder NMT study [2107.10342] [2312.15872]. TET is encoder-only and non-autoregressive, trained with CTC, using per-language heads over vocabulary plus blank and updating shared layers once per language in each minibatch [2509.17930]. MED fine-tunes on code-switching data with a joint CTC+attention objective,
$$
\mathcal{L}_{\mathrm{MOL}}=\lambda\,\mathcal{L}_{\mathrm{CTC}}+(1-\lambda)\,\mathcal{L}_{\mathrm{Attention}},
$$
with $\lambda=0.3$ [2006.10414]. HMDE pretrains with a cross-lingual contrastive objective using Wikipedia-derived positives, in-batch negatives, and category-based hard negatives [2305.07016]. DTPPO combines the PPO clipped surrogate, critic loss, entropy term, and a dynamic-predictor MSE term in
$$
l_{\mathrm{DTPPO}}=\delta_1 l_{\mathrm{actor}}+\delta_2 l_{\mathrm{critic}}+\delta_3 l_{\mathrm{pred}},
$$
with $\delta_1=1$, $\delta_2=1$, and $\delta_3=1\mathrm{e}{-2}$ [2410.15205]. MiTREE uses binary cross-entropy for multi-label encounter-rate prediction over 670 bird species [2412.18995], while M-TabNet predicts birth weight by regression and derives low-birth-weight classification by thresholding the regression output at 2500 grams [2504.15312].

The computational effect of multiple encoders depends on how sharing is arranged. Multi-Stream keeps total encoder depth fixed, so parameter count and FLOPs remain comparable to the single-stream baseline at the same depth, and the additional merge is a negligible elementwise sum [2107.10342]. HMDE reduces flat token-level quadratic cost by applying the lower encoder independently to sentences and then a shallow transformer over sentence vectors, yielding complexity roughly $O(S\cdot L^2)+O(S^2)$ rather than $O((S\cdot L)^2)$ [2305.07016]. TET reduces redundant multilingual computation by sharing intermediate paths: for 8 languages and depth 6 per language, separate per-language encoders require $8\times 6=48$ layer applications, whereas the TET tree uses 24 layer applications to produce all 8 outputs [2509.17930].

Other variants trade efficiency for specialization. The heterogeneous multi-encoder NMT model increases parameters and GFLOPs as encoders are added: in low-resource setups, the single-encoder model has $\omega=5.9$ GFLOPs and $\theta=26$M parameters, whereas the quintuple-encoder has $\omega=16.0$ GFLOPs and $\theta=64$M [2312.15872]. MED roughly doubles encoder-side parameters and decoder cross-attention parameters relative to a single-encoder Transformer, because two encoders and two language-specific source-target attention modules are active [2006.10414]. MPT preserves parameter memory through weight tying but multiplies encoder compute by the number of passes [2009.11382]. MEL is unusual in that the additional encoder exists only during training; at inference, only the magnitude feature encoder is used, so runtime and parameters match the single-stream baseline [2104.00120].

## 5. Empirical behavior across domains

In machine translation, multi-encoder variants consistently target either ambiguity preservation or heterogeneous inductive bias. On WMT-14 DE–EN, the 4-layer Multi-Stream 2(1) + skip model reaches BLEU-4 19.46, compared with 18.99 for the baseline Transformer, and the 6-layer Multi-Stream 4(1) + skip reaches 25.63, compared with 24.65 for the baseline; the paper also reports faster convergence and a more even distribution of attention-pattern types [2107.10342]. The heterogeneous Multi-Encoder Transformer reports that dual-encoder systems improve over the single encoder across all tested datasets, and that the best gain appears in low-resource Spanish–English with 34.38 BLEU for the quadruple-encoder against 27.22 for the single encoder, a maximum increase of 7.16 BLEU; Galician–English likewise improves from 13.03 to 18.38 [2312.15872]. MPT shows that Base Transformer equipped with multi-pass routing can reach 28.40 BLEU on En-De and 41.80 on En-Fr with Base-level parameter counts, matching or surpassing Large Transformer in the reported comparison [2009.11382]. TET, aimed at multilingual MT and ST, reports Multi30K average WER 16.9 for TET versus 35.6 for TEnc/lang and 24.9 for TEnc/all, and Tatoeba average WER 46.7 for TET versus 57.2 and 54.2, with explicit gains for Romanian and Polish in the Tatoeba setting [2509.17930].

In speech recognition, the main empirical pattern is that language- or stream-specific encoders help when the shared input space is heterogeneous. On SEAME, the MED Transformer with MOL+LM obtains 16.7 TER on eval\_man and 23.1 on eval\_sge, improving over the main single-encoder baseline at 18.6 and 25.9 by 10.2% and 10.8% relative [2006.10414]. MEL reaches 3.40% WER on WSJ eval92 with LM in the MEL-t-Fusion-Late configuration, compared with the prior best Transformer-based 4.20% WER cited in the paper, a 19% relative reduction. On LibriSpeech, MEL-t-Fusion-Late reduces WER from 4.05 to 3.34 on test clean and from 8.14 to 7.15 on test other relative to the standard magnitude-only Transformer [2104.00120].

In document representation and retrieval, the principal advantage comes from hierarchical decomposition. HMDE reaches average 86.8 accuracy on cross-lingual topical document classification across non-English target languages, compared with 86.1 for mLongformer, and average MAP 0.249 on cross-lingual document retrieval, compared with 0.186 for Vanilla LaBSE and 0.109 for mLongformer. The paper emphasizes that the model generalizes to languages unseen during document-level pretraining because the lower encoder is massively multilingual [2305.07016].

In multimodal and scientific settings, multi-encoder structures are used to preserve native resolutions, ecological context, or modality-specific sparsity. MiTREE encodes satellite imagery, pedologic rasters, bioclimate rasters, and ecoregion labels separately, and on SatBird Summer reaches MAE 2.070, MSE 0.630, Top-10 47.380%, Top-30 66.609%, and Top-k 67.821%, outperforming the ResNet18 baseline at MAE 2.134, MSE 0.642, Top-10 46.561%, Top-30 65.744%, and Top-k 67.121%; on Winter it reaches MAE 1.621 and MSE 0.430 against 1.658 and 0.450 for ResNet18 [2412.18995]. DTPPO reports Average Transfer Reward 256.19 in Scene-I at 10% obstacle density, compared with 168.39 for MAPPO, and Average Collision Penalty 2.56 in Scene-II at 50% density, compared with 5.80 for MAPPO [2410.15205]. M-TabNet, a multimodal tabular multi-encoder model, reports MAE = 122.3 g and $R^2 = 0.9432$ on the in-house cohort, with sensitivity = 97.55% and specificity = 94.48%; on the external IEEE dataset it reports MAE = 105.4 g and $R^2 = 0.9502$ [2504.15312].

## 6. Misconceptions, limitations, and open directions

A common misconception is that a Multi-Encoder Transformer necessarily means multi-source input. The literature shows otherwise. Multi-Stream, the heterogeneous NMT model, HMDE, and MPT all process a single source through multiple encoders, streams, or passes; what varies is whether the purpose is ambiguity preservation, inductive-bias diversification, hierarchical abstraction, or feedback-like re-encoding [2107.10342] [2312.15872] [2305.07016] [2009.11382]. Another misconception is that multi-encoder designs are equivalent to MoE. TET explicitly contrasts its static tree topology with MoE and adapter sharing, and the heterogeneous NMT system likewise uses parallel encoders with simple sum or concatenation rather than routing [2509.17930] [2312.15872].

The main limitations arise from fusion simplicity, topology choice, and compute scaling. Multi-Stream without the skip can underperform and converge slower in deeper settings, while baseline-with-skip training instability appears after 10–15 epochs in the 4-layer case [2107.10342]. In the heterogeneous NMT study, quintuple-encoder models often reduce BLEU, and FNet combinations are consistently poor under naive fusion [2312.15872]. MED ablations show that language-specific decoder attentions alone are insufficient; improvements come primarily from language-disentangled encoder-side features [2006.10414]. TET-Rnd performs worse on average than linguistically grouped TET, and the model can underperform on some high-resource languages such as German and Italian in Tatoeba, indicating that sharing and specialization must be balanced [2509.17930]. HMDE depends on sentence segmentation quality and inherits language coverage limits from LaBSE [2305.07016]. MiTREE reports that gains from the ecoregion encoder are modest but consistent [2412.18995]. M-TabNet identifies a related failure mode in single-encoder multimodal tabular learning: attention can over-prioritize one data type and neglect others [2504.15312].

The open research directions in the cited work converge on learned routing and more adaptive fusion. TET proposes dynamic trees and learned routing, sparse MoE within branches, semi-autoregressive heads, and tree optimization [2509.17930]. The heterogeneous NMT study suggests learned gates or attention over encoders to replace naive sum or concatenation [2312.15872]. DTPPO points to cross-attention or bidirectional fusion between spatial and temporal pathways, hierarchical encoders, and adaptive neighbor selection [2410.15205]. The event-classification model notes that bidirectional fusion and explicit hierarchical tokens are natural next steps [2401.00452]. Taken together, this suggests that the central unresolved issue is no longer whether multiple encoders can help, but how to allocate specialization, sharing, and routing so that complementary structure is preserved without incurring destructive interference or unnecessary cost.

Source: https://www.emergentmind.com/topics/multi-encoder-transformer