Shared Encoder Architectures

• Shared encoder architectures are neural network designs that use a single encoder module to learn common representations for multiple tasks, languages, or modalities.
• They enable parameter sharing that improves data efficiency and supports specialization via separate decoders or heads.
• Empirical studies demonstrate enhanced throughput, lower latency, and robust generalization across NLP, computer vision, and multimodal applications.

A shared encoder architecture is a neural network design paradigm in which a single encoder module processes input representations for multiple tasks, languages, modalities, or styles. The outputs of this shared encoder are used either directly for prediction, or routed to task-/style-/modality-specific decoders or heads. This strategy is motivated by the need to efficiently capture common, reusable features while allowing downstream specialization where necessary. Shared encoder architectures have become foundational across natural language processing, computer vision, multimodal learning, and operator approximation, enabling improvements in data efficiency, parameter sharing, and adaptivity in real-world systems.

1. Core Principles and Formulations

A shared encoder is typically a deep neural network module (e.g., Transformer, CNN, or MLP) whose parameters are updated using supervision from a union of tasks, domains, or modalities. If $E_\theta$ denotes the shared encoder mapping from inputs $x$ to latent space $\mathbf{h}$, and $\mathcal{T}$ represents a set of downstream tasks, the canonical pattern is:

$$\mathbf{h} = E_\theta(x), \qquad \hat{y}_t = \mathrm{Dec}_t(\mathbf{h}),$$

where $\mathrm{Dec}_t$ is a task- or domain-specific decoder (possibly just a shallow head).

This architecture generalizes to scenarios such as:

• Style adaptation with both shared (“generic”) and private (“specific”) encoders in text generation (Zhang et al., 2018).
• Multi-task computer vision, where a single encoder provides hierarchical features for depth, flow, segmentation, etc. (Jiang et al., 2019, Laboyrie et al., 24 Jan 2025).
• Multimodal processing with a transformer-based backbone, using learnable modality features to distinguish inputs (Roy et al., 3 Mar 2025).
• Multilingual and multi-domain models, where a shared encoder supports language-specific decoders (Tang et al., 2018, Escolano et al., 2020).

The architectural essence is a tight coupling of shared low-level representation learning with specialized or adaptive readout mechanisms.

2. Design Methodologies: Specialization, Adaptivity, and Control

2.1 Shared and Private Parameterization

Mixed shared/private parameterization enables models to disentangle “universal” features from task- or domain-specific signals. In the SHAPED framework (Zhang et al., 2018), a shared encoder $f_\mathrm{enc}^s$ captures general language characteristics, while private encoders/decoders $f_\mathrm{enc}^z$, $f_\mathrm{dec}^z$ capture style-specific signals. During generation, the private and shared decoder states are concatenated, then mapped to outputs via a head $g$, yielding

$$p(o_t|x, y_{1:t-1}, z) = \mathrm{softmax}\left( g\left([s_t^z, s_t^s]\right) \right).$$

2.2 Mixture Models for On-the-Fly Adaptation

When explicit domain/style labels are unavailable, mixture-of-experts or mixture-of-decoders approaches use a classifier $p(z|x)$ to weight outputs:

$$p(o_t|x, y_{1:t-1}) = \sum_{d=1}^{|D|} p(o_t|x, y_{1:t-1}, z=d)p(z=d|x)$$

This allows dynamic adaptation even for unseen domains (Zhang et al., 2018).

2.3 Slimmable and Dynamically Controllable Encoders

For multi-task compute control, encoder and decoder channel widths are treated as runtime-controllable hyperparameters. With slimmable networks, CUDA-efficient width search, and configuration-invariant distillation, users can set task importance and compute budgets post-deployment, and a search algorithm finds feasible width schedules that maximize performance for prioritized tasks (Aich et al., 2023):

$$\omega_i = \omega_{\min} + (\omega_{\max} - \omega_{\min}) \cdot \tau_i \qquad \text{(task-specific decoder width)}$$

3. Domains of Application

Language and Multilingual NLP

Shared multilingual encoders pretrained via translation produce language-agnostic embeddings, enabling transfer to low-resource settings where labeled data are scarce. These encoders serve as the backbone for STS and transfer learning frameworks, producing higher performance and robustness versus translation-based baselines, especially for noisy, user-generated data (Tang et al., 2018). In universal MT models, shared parameterization across all languages supports zero-shot translation, but requires retraining when adding new languages. Modular variants with language-specific encoders/decoders enable fast incremental extension with joint interlingua constraints (Escolano et al., 2020).

Multi-task and Multimodal Vision

In scene flow or dense prediction, architectures such as SENSE (Jiang et al., 2019) share a ResNet-derived backbone across optical flow, disparity, occlusion, and segmentation. Banks of features (feature banks, sampling banks) further share and fuse global context into decoder modules, enabling both top-down and bottom-up information propagation for pixelwise predictions (Laboyrie et al., 24 Jan 2025). In multimodal medical AI, a transformer-based shared encoder with learnable modality features, possibly augmented with modality-specific layers, achieves greater generalization in low-data regimes compared to separate encoders (Roy et al., 3 Mar 2025).

Retrieval and Dual Encoding

In question answering and image-text retrieval, Siamese dual encoders—where both query and candidate branches share all parameters—yield more aligned embedding spaces, demonstrably improving retrieval metrics over asymmetric dual branches. Sharing only projection layers mitigates distribution misalignment while enabling some architectural freedom (Dong et al., 2022). For multi-encoder cases (LoopITR (Lei et al., 2022)), dual and cross-encoders are trained interactively within a shared system, using teacher-student distillation and hard negative mining.

4. Technical Benefits and Trade-offs

Efficiency, Generalization, and Flexibility

Shared encoder architectures impose strong inductive biases towards generalizable representations, regularize parameter growth, and reduce computational and memory footprints. In on-device or edge environments, encoder-decoder SLMs show up to 47% lower first-token latency and 4.7x throughput advantage relative to decoder-only models, enabled by one-time input processing and encoder-decoder separation of understanding/generation phases (Elfeki et al., 27 Jan 2025). For multi-scale object detection and segmentation, reusing shared modules with attention mechanisms and pooling strategies improves adaptive performance (notably for small objects) without significant parameter increase (Li et al., 2020, Silva et al., 2021).

Task Specialization and Modularity

The main limitation is that excessive parameter sharing can dilute task-specific information or induce negative transfer, especially under severe task heterogeneity. Hybrid strategies—incorporating both shared and private layers (Zhang et al., 2018, Roy et al., 3 Mar 2025) or slimmable decoders (Aich et al., 2023)—can mitigate this. Evolutionary or search-based approaches optimize shared decoder configurations to match user- or scenario-specific performance constraints. Modularity, as seen in language-specific decoders or frame-group decoders for video INRs (Escolano et al., 2020, Rangarajan et al., 29 Jun 2025), allows extensibility and efficient adaptation.

5. Empirical Results Across Benchmarks

Numerous empirical studies demonstrate the benefits of shared encoder architectures across domains:

Domain / Task	Key Metric Improvement	Reference
Multilingual STS (ES-ES, AR-AR)	Pearson up to 0.825 (vs. 0.711 FastText)	(Tang et al., 2018)
Scene Flow (MPI Sintel, KITTI)	State-of-the-art EPE with 13M params	(Jiang et al., 2019)
One-stage object detection (COCO)	+3.8% AP over MNC ResNet-101 baseline	(Li et al., 2020)
Medical multimodal retrieval (low-data)	Recall increase >80% (vs. CLIP)	(Roy et al., 3 Mar 2025)
SLMs on edge devices	47% lower first-token latency, 4.7x throughput	(Elfeki et al., 27 Jan 2025)
Video INR encoding	20–30× speed-up, competitive PSNR/SSIM	(Rangarajan et al., 29 Jun 2025)

Improvements typically result from:

• Efficient knowledge sharing across related tasks/domains
• Regularization against overfitting
• Aggressive parallelism enabled by shared/frozen encoders
• More robust adaptation to new domains, small sample sizes, or unseen styles

6. Theoretical Advances and Universality

The universal operator approximation theorem (Gödeke et al., 31 Mar 2025) provides a rigorous mathematical underpinning for the generality of shared encoder-decoder architectures. If the input/output spaces possess the encoder-decoder approximation property (EDAP), then any continuous operator $G : X \to Y$ can be uniformly approximated on every compact subset using architectures of the form:

$G_n = D_n^Y \circ \varphi_n \circ E_n^X,$

with a shared encoder $E_n^X$, universal function approximator $\varphi_n$, and decoder $D_n^Y$. Crucially, the approximating sequence is independent of the compact set, a stronger universality property than previous operator learning frameworks and instantiated in DeepONets, BasisONets, and related neural operator classes.

7. Implications and Outlook

A recurring observation in neural news recommenders (Iana et al., 2 Oct 2024) and similar domains is that architectural complexity beyond a threshold does not yield significant performance gains. Representational similarity metrics (CKA) and retrieval overlap (Jaccard) indicate that simpler shared encoder models can achieve nearly identical outputs to more elaborate designs. This supports the widespread adoption of shared encoder architectures in practice, favoring efficiency, ease of deployment, and extensibility.

Shared encoder paradigms will likely play a central role in future multi-domain, multi-task, and multi-modal systems as requirements for data efficiency, transferability, and adaptability continue to escalate. With advances in theoretical analysis, as well as empirical validation in diverse application settings, shared encoders establish a robust, principled, and efficient foundation for modern AI architectures.