Asymmetric Dual-Encoder (ADE)

• ADE is a dual-encoder design principle employing non-identical branches, where each encoder is tailored for specific roles or modalities.
• It is applied across various domains such as speech recognition, remote sensing, graph auto-encoding, QA retrieval, and single-cell RNA-seq integration.
• Empirical studies show that strategic encoder asymmetry and fusion rules can enhance performance, efficiency, and interpretability in complex tasks.

Asymmetric Dual-Encoder (ADE) denotes a class of dual-branch architectures in which the two encoders are intentionally non-identical and are assigned different representational, modal, directional, or optimization roles. In the cited arXiv literature, ADE appears in joint close-talk/far-talk automatic speech recognition, RGB-DSM remote-sensing semantic segmentation, directed-graph auto-encoding, question-answer retrieval, and single-cell RNA-seq integration (Weninger et al., 2021, Ye et al., 22 Jul 2025, Kollias et al., 2022, Dong et al., 2022, Yan et al., 18 May 2026). Across these settings, asymmetry is implemented through unequal encoder capacity, distinct parameterization, role-specific message passing, encoder selection, or one-way alignment from a more stable stream to a noisier stream. This suggests that ADE is best understood as a design principle rather than as a single canonical model family.

1. Terminological scope and recurrent structure

The principal uses of the term in the cited literature are summarized below (Weninger et al., 2021, Ye et al., 22 Jul 2025, Kollias et al., 2022, Dong et al., 2022, Yan et al., 18 May 2026).

Domain	Asymmetric branches or roles	Integration or decoder
ASR	single-channel CT encoder; 16-channel FT encoder with neural beamformer	encoder selection; hard or soft selection
Remote sensing	Swin-Base RGB encoder; Swin-Small DSM encoder	channel matching, APF, DA
Directed graphs	source embeddings; target embeddings	asymmetric inner product
QA retrieval	distinct question and passage towers	cosine similarity after projection
scRNA-seq integration	Variant stream; Anchor stream	Align-Refine-Fuse with bounded gating

In the question-answering formulation of Dong et al., ADE is explicitly contrasted with the Siamese Dual Encoder (SDE): SDE shares parameters across two encoders, whereas ADE uses two distinctly parameterized encoders (Dong et al., 2022). In the remote-sensing and speech settings, asymmetry is tied to the differing information content and sensing conditions of the two inputs; in directed graphs it is tied to the source-versus-target roles induced by edge direction; and in scHelix it is tied to the distinction between domain-invariant Anchors and domain-sensitive Variants (Ye et al., 22 Jul 2025, Weninger et al., 2021, Kollias et al., 2022, Yan et al., 18 May 2026).

A common misconception is that ADE always means a “heavier” branch and a “lighter” branch. The cited literature is more varied. In AMMNet, the selected Base/Small configuration uses the same stage depths, $[2,2,18,2]$, for RGB and DSM and realizes asymmetry mainly through channel widths; in directed graphs, asymmetry lies in the exchange rules and decoder; and in QA retrieval, it lies in non-shared encoder and projection parameters rather than in modality-specific depth allocation (Ye et al., 22 Jul 2025, Kollias et al., 2022, Dong et al., 2022).

2. Modality-specialized ADEs

In Weninger et al., the ADE for joint close-talk (CT) and far-talk (FT) speech recognition couples a single-channel CT encoder with a multi-channel FT encoder and an encoder-selection network (Weninger et al., 2021). The common front end uses raw waveform sampled at 16 kHz, on-the-fly STFT, Mel-filterbank extraction, 80-dim log-Mel features, frame shift 10 ms, window 32 ms, cepstral mean normalization, and per-utterance batch norm. The CT branch consumes single-channel log-Mel features and applies 6 layers of bidirectional LSTM with 512 units per direction, with frame decimation after layers 2 and 4 for a total down-sampling factor of 8. The FT branch starts from a 16-channel waveform, applies STFT, a Spatial Filtering neural beamformer with $C=16$ channels and $D=11$ look-directions, converts the enhanced signal to log-Mel features, and then uses the same 6-layer bLSTM stack. The selector takes the time-aligned feature streams $$\bigl[x^{(\mathrm{CT})}_{1:T};x^{(\mathrm{FT})}_{1:T}\bigr]\in\mathbb R^{T\times(80+80)}$$, processes them with two TDNN layers with 256 hidden units, a 256-unit unidirectional LSTM, additive attention to a single 256-dim vector, and a final 2-way softmax that yields $\{q_{\mathrm{CT}},q_{\mathrm{FT}}\}$. The fused representation is

$$e=q_{\mathrm{CT}}\,e^{(\mathrm{CT})}+q_{\mathrm{FT}}\,e^{(\mathrm{FT})},$$

and the overall loss is

$$\mathcal L=\mathcal L_{\mathrm{ASR}}+\lambda\,\mathcal L_{\mathrm{sel}}.$$

The ADE module in "AMMNet: An Asymmetric Multi-Modal Network for Remote Sensing Semantic Segmentation" places asymmetry at the level of encoder capacity allocation (Ye et al., 22 Jul 2025). The RGB branch uses a Swin-Base variant with 4 stages, blocks per stage $[2,2,18,2]$, and channel widths $[128,256,512,1024]$. The DSM branch uses a Swin-Small variant with 4 stages, the same block pattern $[2,2,18,2]$, and channel widths $C=16$0. Channel Matching (CM) projects each DSM feature map of width $C=16$1 to the corresponding RGB width $C=16$2 via $C=16$3 conv $C=16$4 BN $C=16$5 ReLU. The formal constraints are $C=16$6 and $C=16$7 for stages $C=16$8; in the chosen configuration, the depths are equal while $C=16$9 at every stage, and the total capacity satisfies $D=11$0.

In scHelix, asymmetry is imposed even earlier, at the input level (Yan et al., 18 May 2026). Genes are partitioned into an Anchor set

$D=11$1

and a Variant set

$D=11$2

with default thresholds $D=11$3. Each cell input is decomposed as $D=11$4, so no Anchor features leak into the Variant stream. The two streams then pass through sparse diffusion encoders built from learned sparse gene-gene graphs $D=11$5, stop-gradient caching, and multi-scale low-pass and high-pass propagation. The asymmetric Align-Refine-Fuse protocol first aligns the Variant stream to the Anchor topology, then refines the Anchor stream using bounded residual gating, and finally performs HyperFusion with an adaptive gate. The one-way nature of the alignment is explicit: stop-gradient on the Anchor projection ensures that the model does not pull the Anchor stream toward noisy Variant features.

Taken together, these modality-specialized ADEs allocate representational capacity according to information density, noise characteristics, or stability. The remote-sensing paper states this directly for RGB versus DSM, and scHelix states it directly for Anchors versus Variants; the speech model operationalizes the same principle through single-channel CT processing versus multi-channel FT beamforming (Ye et al., 22 Jul 2025, Yan et al., 18 May 2026, Weninger et al., 2021).

3. Directed-role ADE for graphs

"Directed Graph Auto-Encoders" formulates ADE, termed DiGAE in that paper, as an auto-encoder for directed graphs in which each node $D=11$6 receives two latent vectors: a source embedding $D=11$7 and a target embedding $D=11$8 (Kollias et al., 2022). The asymmetry is intrinsic to edge direction. Source vectors summarize a node’s role as an outgoing “hub,” while target vectors summarize its role as an incoming “authority.”

Let $D=11$9 be the directed adjacency matrix, $$\bigl[x^{(\mathrm{CT})}_{1:T};x^{(\mathrm{FT})}_{1:T}\bigr]\in\mathbb R^{T\times(80+80)}$$0, and let $$\bigl[x^{(\mathrm{CT})}_{1:T};x^{(\mathrm{FT})}_{1:T}\bigr]\in\mathbb R^{T\times(80+80)}$$1 and $$\bigl[x^{(\mathrm{CT})}_{1:T};x^{(\mathrm{FT})}_{1:T}\bigr]\in\mathbb R^{T\times(80+80)}$$2 be the out-degree and in-degree diagonal matrices including self-loops. With degree exponents $$\bigl[x^{(\mathrm{CT})}_{1:T};x^{(\mathrm{FT})}_{1:T}\bigr]\in\mathbb R^{T\times(80+80)}$$3, the directed normalization is

$$\bigl[x^{(\mathrm{CT})}_{1:T};x^{(\mathrm{FT})}_{1:T}\bigr]\in\mathbb R^{T\times(80+80)}$$4

One directed convolutional layer updates the source and target streams as

$$\bigl[x^{(\mathrm{CT})}_{1:T};x^{(\mathrm{FT})}_{1:T}\bigr]\in\mathbb R^{T\times(80+80)}$$5

Thus, source states aggregate target states of out-neighbors, and target states aggregate source states of in-neighbors. The decoder is asymmetric as well:

$$\bigl[x^{(\mathrm{CT})}_{1:T};x^{(\mathrm{FT})}_{1:T}\bigr]\in\mathbb R^{T\times(80+80)}$$6

so $$\bigl[x^{(\mathrm{CT})}_{1:T};x^{(\mathrm{FT})}_{1:T}\bigr]\in\mathbb R^{T\times(80+80)}$$7 estimates the likelihood of a directed edge $$\bigl[x^{(\mathrm{CT})}_{1:T};x^{(\mathrm{FT})}_{1:T}\bigr]\in\mathbb R^{T\times(80+80)}$$8.

Training uses binary cross-entropy reconstruction on observed edges $$\bigl[x^{(\mathrm{CT})}_{1:T};x^{(\mathrm{FT})}_{1:T}\bigr]\in\mathbb R^{T\times(80+80)}$$9 and an equal-sized set of randomly sampled non-edges $\{q_{\mathrm{CT}},q_{\mathrm{FT}}\}$0, plus L2 weight decay on the encoder parameters. The model therefore differs from multimodal ADEs in a fundamental way: the two encoders do not correspond to two sensors or two documents, but to two directional roles over the same graph. The paper further assigns clear interpretability to the dual embeddings by analogy to HITS and SVD: source vectors behave like hub representations and target vectors like authority representations, and their magnitudes correlate empirically with out-degree, in-degree, hub score, and PageRank/authority.

4. ADE in question answering and retrieval

Dong et al. study ADE directly as a retrieval architecture for question answering and information retrieval (Dong et al., 2022). Given a question $\{q_{\mathrm{CT}},q_{\mathrm{FT}}\}$1 and an answer candidate $\{q_{\mathrm{CT}},q_{\mathrm{FT}}\}$2, the two Transformer towers are denoted $\{q_{\mathrm{CT}},q_{\mathrm{FT}}\}$3 and $\{q_{\mathrm{CT}},q_{\mathrm{FT}}\}$4. Mean pooling yields

$\{q_{\mathrm{CT}},q_{\mathrm{FT}}\}$5

followed by learned projections

$\{q_{\mathrm{CT}},q_{\mathrm{FT}}\}$6

Pairs are scored by cosine similarity,

$\{q_{\mathrm{CT}},q_{\mathrm{FT}}\}$7

and optimized with the in-batch softmax contrastive loss

$\{q_{\mathrm{CT}},q_{\mathrm{FT}}\}$8

The paper distinguishes five architectures: SDE, vanilla ADE, ADE-STE, ADE-FTE, and ADE-SPL. In vanilla ADE, the two encoders share no parameters: $\{q_{\mathrm{CT}},q_{\mathrm{FT}}\}$9 and $$e=q_{\mathrm{CT}}\,e^{(\mathrm{CT})}+q_{\mathrm{FT}}\,e^{(\mathrm{FT})},$$0 are independent copies of the same pre-trained Transformer, and $$e=q_{\mathrm{CT}}\,e^{(\mathrm{CT})}+q_{\mathrm{FT}}\,e^{(\mathrm{FT})},$$1 and $$e=q_{\mathrm{CT}}\,e^{(\mathrm{CT})}+q_{\mathrm{FT}}\,e^{(\mathrm{FT})},$$2 are independent projection matrices. For T5.1.1-base, with hidden size $$e=q_{\mathrm{CT}}\,e^{(\mathrm{CT})}+q_{\mathrm{FT}}\,e^{(\mathrm{FT})},$$3, 12 layers, and 12 heads, the total ADE parameter count is reported as approximately $$e=q_{\mathrm{CT}}\,e^{(\mathrm{CT})}+q_{\mathrm{FT}}\,e^{(\mathrm{FT})},$$4M. ADE-STE shares the token embedding matrix while keeping the Transformer stacks and projections separate; ADE-FTE freezes the token embedding matrix while keeping the Transformer stacks and projections separate; ADE-SPL keeps the token embedding and Transformer stacks distinct but uses a single shared projection matrix $$e=q_{\mathrm{CT}}\,e^{(\mathrm{CT})}+q_{\mathrm{FT}}\,e^{(\mathrm{FT})},$$5 for both towers.

The central empirical result is that SDE performs significantly better than vanilla ADE, while ADE-SPL closes the gap and in many cases matches or slightly outperforms SDE. On MS MARCO, the reported MRR values are $$e=q_{\mathrm{CT}}\,e^{(\mathrm{CT})}+q_{\mathrm{FT}}\,e^{(\mathrm{FT})},$$6 for SDE, $$e=q_{\mathrm{CT}}\,e^{(\mathrm{CT})}+q_{\mathrm{FT}}\,e^{(\mathrm{FT})},$$7 for ADE, and $$e=q_{\mathrm{CT}}\,e^{(\mathrm{CT})}+q_{\mathrm{FT}}\,e^{(\mathrm{FT})},$$8 for ADE-SPL. On open-domain NQ, TopExact at $$e=q_{\mathrm{CT}}\,e^{(\mathrm{CT})}+q_{\mathrm{FT}}\,e^{(\mathrm{FT})},$$9 is $$\mathcal L=\mathcal L_{\mathrm{ASR}}+\lambda\,\mathcal L_{\mathrm{sel}}.$$0 for SDE, $$\mathcal L=\mathcal L_{\mathrm{ASR}}+\lambda\,\mathcal L_{\mathrm{sel}}.$$1 for ADE, and $$\mathcal L=\mathcal L_{\mathrm{ASR}}+\lambda\,\mathcal L_{\mathrm{sel}}.$$2 for ADE-SPL. The t-SNE analysis provides the mechanism proposed by the authors: vanilla ADE, ADE-STE, and ADE-FTE produce two almost disjoint clusters for questions and answers, whereas SDE and ADE-SPL produce a single mixed cloud. The paper’s explanation is that shared projection enforces a common coordinate system, making cosine similarity meaningful across towers.

5. Objectives, fusion rules, and inference regimes

The ADE literature spans several distinct objective families. In the speech model, the optimization target is end-to-end ASR loss—either cross-entropy with attention or RNN-T loss—with an optional supervised encoder-selection term:

$$\mathcal L=\mathcal L_{\mathrm{ASR}}+\lambda\,\mathcal L_{\mathrm{sel}}.$$3

The selector outputs $$\mathcal L=\mathcal L_{\mathrm{ASR}}+\lambda\,\mathcal L_{\mathrm{sel}}.$$4, and gradients flow through the soft-selection operation into both sub-encoders, the beamformer, and the selector itself. Inference can use hard selection, with $$\mathcal L=\mathcal L_{\mathrm{ASR}}+\lambda\,\mathcal L_{\mathrm{sel}}.$$5, or soft selection, with $$\mathcal L=\mathcal L_{\mathrm{ASR}}+\lambda\,\mathcal L_{\mathrm{sel}}.$$6 (Weninger et al., 2021).

In directed graphs, the training objective is reconstruction, not discrimination: ADE learns source and target embeddings by binary cross-entropy on true edges and sampled non-edges, regularized by $$\mathcal L=\mathcal L_{\mathrm{ASR}}+\lambda\,\mathcal L_{\mathrm{sel}}.$$7 (Kollias et al., 2022). In retrieval, the objective is contrastive rather than reconstructive, and inference consists of nearest-neighbor ranking under cosine similarity in the projected space (Dong et al., 2022).

AMMNet places ADE at the first stage of a larger segmentation pipeline (Ye et al., 22 Jul 2025). Its forward pass returns two aligned feature pyramids—RGB features and channel-matched DSM features—which are then consumed by the Asymmetric Prior Fuser (APF) and the Distribution Alignment (DA) module. scHelix similarly embeds ADE inside a broader pipeline: the dual-stream sparse diffusion encoder is followed by Align, Refine, and Fuse operations. The alignment loss is explicitly given as

$$\mathcal L=\mathcal L_{\mathrm{ASR}}+\lambda\,\mathcal L_{\mathrm{sel}}.$$8

where $$\mathcal L=\mathcal L_{\mathrm{ASR}}+\lambda\,\mathcal L_{\mathrm{sel}}.$$9 is stop-gradient; refinement updates the Anchor embedding by

$[2,2,18,2]$0

with $[2,2,18,2]$1 elementwise; and fusion produces

$[2,2,18,2]$2

through HyperFusion with an adaptive gate (Yan et al., 18 May 2026).

These formulations show that asymmetry can enter at multiple points: in the encoders themselves, in the rule that combines branch outputs, in the decoder, or in the optimization path. This suggests that ADE is less a single algorithm than a recurring strategy for constraining how information moves between two non-equivalent representations.

6. Empirical behavior, interpretability, and limitations

The empirical record is mixed in a task-dependent way. In speech recognition, Weninger et al. report that the proposed dual-encoder architecture obtains up to $[2,2,18,2]$3 relative WER reduction when using both CT and FT input, compared to the best single-encoder system trained and tested in matched condition (Weninger et al., 2021). For LAS on manually segmented utterances, the best single-encoder matched CT system yields $[2,2,18,2]$4 WER on CT and $[2,2,18,2]$5 on FT, single-encoder plus SF beamformer on joint input yields $[2,2,18,2]$6, dual-encoder hard selection yields $[2,2,18,2]$7, and dual-encoder soft selection yields $[2,2,18,2]$8. For the Conformer Transducer, CT-only yields $[2,2,18,2]$9, FT-only with SF yields $[128,256,512,1024]$0, single-encoder SF(CT;FT) yields $[128,256,512,1024]$1, and dual-encoder soft selection yields $[128,256,512,1024]$2 with significant $[128,256,512,1024]$3 under a matched-pairs test. The same work reports graceful degradation under random time shifts up to $[128,256,512,1024]$4 ms, especially with shift-aware training.

In remote sensing, the AMMNet ADE contributes to both efficiency and accuracy (Ye et al., 22 Jul 2025). The Base+Small ADE design is reported with $[128,256,512,1024]$5 GFLOPs, $[128,256,512,1024]$6M parameters, $[128,256,512,1024]$7 MB GPU memory, and $[128,256,512,1024]$8 mIoU. A typical symmetric baseline, FTransUNet, reports $[128,256,512,1024]$9 GFLOPs, $[2,2,18,2]$0M parameters, $[2,2,18,2]$1 MB, and $[2,2,18,2]$2 mIoU. In the component ablation, enabling ADE alone raises mOA from $[2,2,18,2]$3 to $[2,2,18,2]$4 and mIoU from $[2,2,18,2]$5 to $[2,2,18,2]$6; in concert with APF and DA, the final mIoU reaches $[2,2,18,2]$7, a $[2,2,18,2]$8 gain over the all-disabled baseline. In scHelix, the asymmetric dual-stream design outperforms both symmetric dual-encoders and classical batch-correction on Human Pancreas, Failing Heart, and Immune datasets as measured by the scIB Overall score $[2,2,18,2]$9, with an average $C=16$00 gain over the second-best baseline; the reported scores are $C=16$01 versus $C=16$02 on Human Pancreas, $C=16$03 versus $C=16$04 on Failing Heart, and $C=16$05 versus $C=16$06 on Immune (Yan et al., 18 May 2026).

In directed graphs, ADE is reported to surpass competing GCN-based directed-link autoencoders by large margins in AUC/AP on multiple datasets (Kollias et al., 2022). On CoraML, ADE-1L achieves $C=16$07 and $C=16$08, versus Gravity GAE’s $C=16$09 and $C=16$10; on CiteSeer, ADE-1L achieves $C=16$11 and $C=16$12, versus Gravity GAE’s $C=16$13 and $C=16$14. The single-layer variant is also reported as $C=16$15–$C=16$16 faster than baselines. Here, interpretability is not incidental but built into the model: the dual embeddings are explicitly associated with hub-like and authority-like behavior.

The main counterexample to any universal claim of ADE superiority comes from QA retrieval (Dong et al., 2022). There, vanilla ADE lags behind SDE by $C=16$17–$C=16$18 MRR points, and the paper’s conclusion is not that asymmetry is uniformly advantageous, but that asymmetry without a shared projection layer produces misaligned embedding spaces. This is an important limitation for the broader concept. Another common misconception is that asymmetry itself is the performance source; the cited evidence is more specific. In QA, projection sharing is decisive. In speech, soft selection outperforms hard selection. In AMMNet, ADE’s standalone gains are small but become more consequential when combined with APF and DA. In scHelix, the benefit depends on the full Align-Refine-Fuse protocol, including bounded gating and self-distillation ablations. A plausible implication is that ADE is most effective when the architectural asymmetry is matched to a concrete source of heterogeneity—sensor geometry, modality sparsity, graph directionality, or batch-sensitive gene structure—rather than imposed as an abstract preference for non-shared towers.