Asymmetric Dual Encoders (ADE)

Updated 15 December 2025

Asymmetric Dual Encoders (ADE) are dual-tower neural architectures with distinct parameters tailored for modality-specific tasks and efficient resource utilization.
They employ deeper encoders for dense modalities and pruned, low-latency encoders for queries, resulting in significant reductions in memory use and computational load.
Empirical results show that methods like shared projection layers and post-training alignment (KALE) mitigate embedding misalignment and maintain retrieval accuracy.

An Asymmetric Dual Encoder (ADE) is a dual-tower neural architecture in which the two encoders—responsible for mapping different input modalities, roles, or tasks (such as queries vs. documents, or RGB vs. DSM in multimodal segmentation)—are not restricted to share parameters, depth, or width. This asymmetry is introduced to address modality or task-specific representational demands, enable architectural efficiency in resource-constrained scenarios, and facilitate computational acceleration, with applications spanning information retrieval, question answering, model compression, and remote sensing semantic segmentation (Dong et al., 2022, Campos et al., 2023, Ye et al., 22 Jul 2025).

1. Architectural Foundations and Mathematical Formulation

In the canonical information retrieval setting, consider a question $q$ and a set of candidate answers or documents $A$ . A dual-encoder system computes—via two (potentially distinct) embedding functions $f_q$ and $f_a$ —representations $h_q=f_q(q)\in\mathbb{R}^d$ and $h_a=f_a(a)\in\mathbb{R}^d$ , later scored by e.g., cosine similarity. In a fully asymmetric realization, every parameter, from input embedding layers, transformer stacks, to output projections, is distinct across the two encoders (“vanilla ADE”) (Dong et al., 2022).

The loss employed during training is a batched contrastive softmax:

$L_i = -\log \frac{\exp(\mathrm{sim}(f_q(q_i), f_a(a_i)) / \tau)}{\sum_{j=1}^B \exp(\mathrm{sim}(f_q(q_i), f_a(a_j)) / \tau)}$

where $\tau$ is a temperature scalar, $\mathrm{sim}$ denotes normalized dot product, and $B$ is the batch size. The total loss is averaged over the batch.

In asymmetric bi-encoder designs for resource-optimized retrieval, the two encoders may differ in architecture: for example, $eq$ (the query encoder) is pruned to fewer transformer layers than $ed$ (the context/document encoder) (Campos et al., 2023). In multi-modal settings, the ADE module consists of a deep, wide encoder for the denser modality and a shallow encoder for the sparser one; feature dimension mismatch is resolved by a trainable linear projection (“channel matching”) (Ye et al., 22 Jul 2025).

2. Design Rationale and Efficiency Trade-offs

Dual encoders traditionally used parameter sharing (“Siamese Dual Encoder”, SDE). However, for tasks involving genuinely asymmetric inputs—queries and documents, audio and text, or multi-modal signals—strict sharing can bottleneck performance or inflate resource usage. ADEs are motivated by:

Modality/task-specific complexity: Dense contextual modalities (e.g., RGB imagery) benefit from deeper, higher-capacity encoders; structured/scalar modalities (e.g., DSM maps) suffice with lightweight models (Ye et al., 22 Jul 2025).
Computational efficiency: In IR and QA, document/context encoding is typically done offline, while query encoding must be performed at low latency per request. Pruning the query-side encoder yields direct QPS improvements with negligible impact on overall retrieval quality (Campos et al., 2023).
Avoidance of architectural redundancy: In multi-modal segmentation, ADEs reduce redundant computation and memory, yielding up to 70% memory and 36% FLOP reductions compared to two identical encoders (Ye et al., 22 Jul 2025).

Empirical analysis of ADEs for QA retrieval reveals that full asymmetry in all components (input token embeddings, encoder stacks, projection layers) leads to sub-optimal alignment of the two embedding spaces, harming retrieval performance (Dong et al., 2022). The following variations are introduced:

ADE-STE: Sharing the token embedder (trainable).
ADE-FTE: Sharing and freezing the token embedder.
ADE-SPL: Sharing the projection layer only.

Among these, ADE-SPL—where only the final dense layer mapping encoded vectors to retrieval space is shared—substantially narrows (and sometimes closes) the performance gap with fully Siamese DEs. Visualization via t-SNE confirms that separate projections lead to disjoint clusters for queries and answers, while shared projections collapse the distributions into a coherent semantic space, improving match scoring.

For asymmetry at the architectural level (e.g., differing number of transformer layers), the Kullback–Leibler Alignment of Embeddings (KALE) is introduced (Campos et al., 2023). After pruning the query encoder, KALE minimizes the KL divergence between the output distributions of the pruned and full query encoders, with the document encoder’s parameters held fixed. Unlike standard knowledge distillation, KALE enables rapid alignment post–index construction, obviating re-encoding all documents and facilitating flexible deployment.

4. Application Domains and Empirical Outcomes

Information Retrieval and QA

On the MS MARCO, OpenNQ, and MultiReQA benchmarks, baseline ADE lags SDE by 1–2 MRR points; ADE-STE/FTE recover minor losses, while ADE-SPL matches or exceeds SDE (Dong et al., 2022). For hardware efficiency, aggressive pruning of the query encoder in an ADE reduces latency by 3–6× at the cost of <2% top-k recall drop, provided KALE is used for post-pruning alignment (Campos et al., 2023). Best-performing configurations combine a full context encoder with a 3-layer pruned query encoder, matching full BERT-Base recall at ≈3× query throughput.

AMMNet demonstrates the generality of the ADE principle to vision tasks (Ye et al., 22 Jul 2025). Its ADE module, using a Swin-Base encoder for RGB and a Swin-Small encoder for DSM, achieves functions:

36% reduction in FLOPs,
70% reduction in memory,
mIoU improvement from 84.23% (dual Base) to 87.56% (ADE).

Empirical ablations confirm that the Base+Small pairing outperforms both heavier and lighter symmetric configurations.

Model	FLOPs (G)	Params (M)	Memory (MB)	mIoU (%)
Symmetric Base+Base	45.21	160.88	3463	84.23
ADE (Base+Small)	28.82	151.26	1026	87.56

5. Module Interactions and Embedding Space Alignment

ADE’s asymmetric towers often introduce a misalignment of the two embedding spaces, observable as spatially disjoint clusters in low-dimensional projections (Dong et al., 2022). Empirical findings suggest:

Non-shared projections: Disjoint clusters, requiring the model to “bridge” spaces at scoring time.
Shared projections: Embedded points from both towers are intermixed, yielding a single coherent semantic space for relevance estimation.

In AMMNet, ADEs interoperate with Asymmetric Prior Fuser (APF) and Distribution Alignment (DA) modules. The APF leverages semantic priors for refined fusion; DA minimizes a KL-type loss between feature distributions, further enhancing cross-modal compatibility (Ye et al., 22 Jul 2025).

6. Trade-offs, Recommendations, and Empirical Guidelines

Observed trade-offs and practical recommendations for ADEs include:

Compression vs. performance: Aggressive query encoder pruning is feasible provided alignment (e.g., KALE) is performed (Campos et al., 2023).
Symmetry vs. efficiency: Assigning more capacity to the context (or “harder”/richer modality) encoder is preferable over symmetric compression for a fixed compute or parameter budget.
Projection sharing: Whenever encoder asymmetry is architectural rather than semantic, sharing the final projection is essential to restore semantic alignment and maximize retrieval accuracy (Dong et al., 2022).
Fusion and alignment: ADEs fundamentally reduce memory and compute without sacrificing accuracy, particularly when paired with appropriately designed fusion and alignment modules in multi-modal architectures (Ye et al., 22 Jul 2025).

A plausible implication is that ADE configurations with shared projection layers or efficient post-hoc alignment mechanisms generalize well across domains where input asymmetry (in role, modality, or complexity) is intrinsic, provided the embedding space is forced to align through explicit mechanisms or architectural sharing.

7. Empirical Ablation, Limitations, and Open Directions

Empirical ablation studies confirm that the benefit of ADE arises from both the removal of redundancy and improved feature allocation to modality-specific towers. However, vanilla ADEs with no alignment or projection sharing show persistent misalignment in embedding space and measurable accuracy deficits (Dong et al., 2022, Ye et al., 22 Jul 2025). The effectiveness of alignment strategies and optimal degree of asymmetry remain sensitive to the task (retrieval vs. segmentation), size of recall sets, and underlying modality characteristics. Post-training alignment methods such as KALE mitigate most, but not all, capacity-induced quality loss under extreme model pruning (Campos et al., 2023).

Continued exploration is warranted on:

Dynamic parameter-sharing strategies that operate beyond static projection sharing.
Improved modality-specific fusion and alignment mechanisms in multi-modal ADE settings.
Extension of ADEs to non-transformer and non-Euclidean architectures.

ADEs serve as a versatile architectural template for resource-efficient, modality- or task-adaptive representation learning in diverse settings spanning text, vision, and multi-modal integration.