ARLISS: Adapter-Based Unified Model

• The paper demonstrates that integrating compact adapter modules into frozen transformers drastically reduces trainable parameters while maintaining competitive performance across diverse tasks.
• The methodology employs strategic adapter insertion after attention and feedforward layers, utilizing LoRA, fusion, and task-specific prompts for efficient parameter updates.
• Empirical evaluations reveal significant improvements, such as an 18.4% average gain and notable error reductions, across speech and multi-modal telecom benchmarks.

The Adapter-Based Unified Model, referred to as ARLISS in the spoken language processing literature, is a modular, parameter-efficient framework for multi-task learning that inserts compact, trainable adapter modules into a frozen transformer backbone. ARLISS enables a single neural architecture to tackle a diverse range of tasks—such as automatic speech recognition (ASR), phoneme recognition, intent classification, slot filling, and emotion recognition—by plugging task-specific adapters into a pretrained encoder–decoder and optimizing only the adapters for each downstream objective. This approach contrasts with conventional multi-head or multi-decoder strategies by dramatically reducing the number of trainable parameters, improving parameter sharing, and preserving the foundational model’s generalization capacity. Adapter-based methodologies have also been extended to multi-modal and telecommunications domains, where they bridge distinct input modalities (e.g., radio and text) to a unified LLM backbone, enabling a single model to perform heterogeneous physical-layer tasks with minimal adaptation overhead (Suresh et al., 2024, Jiao et al., 15 May 2025).

1. Core Model Architecture and Adapter Placement

ARLISS employs a frozen pretrained backbone (e.g., wav2vec 2.0-large for spoken language processing, or domain-specific encoders for telecom applications) as the base feature extractor. Adapters are inserted into each transformer block—one after the multi-head attention block and one after the feedforward sublayer—across both the encoder and decoder stacks. In the canonical spoken language setting, the backbone comprises 24 wav2vec 2.0 encoder layers and a randomly initialized 6-layer transformer decoder, with adapters inserted throughout. Only these adapter modules, and the decoder during initial warm-up, are updated on downstream tasks; the principal backbone remains fixed (Suresh et al., 2024).

In the telecom multi-modal regime (AI2MMUM), two frozen encoders extract universal embeddings from radio modality inputs (e.g., channel state information or physical environment data), which are projected via a modality-bridging adapter into the LLM’s embedding space. These embeddings are concatenated with processed instruction tokens and passed through the LLM transformer, where low-rank adapters are applied to query and key projections in every self-attention block. The LLM’s output is routed through lightweight task-specific heads for final prediction (Jiao et al., 15 May 2025).

2. Mathematical Characterization of Adapter Modules

Adapter modules in ARLISS are parameter-light, bottlenecked MLPs with residual connections. For hidden size $d$ and bottleneck $k$, let $x \in \mathbb{R}^d$ denote the input:

• Down-projection: $h = \mathrm{LayerNorm}(x)$,
• Bottleneck: $z = \mathrm{ReLU}(W_{\downarrow} h)$ with $W_{\downarrow} \in \mathbb{R}^{k \times d}$,
• Up-projection: $u = W_{\uparrow} z$ with $W_{\uparrow} \in \mathbb{R}^{d \times k}$,
• Residual output: $\mathrm{Adapter}(x) = x + u$.

In multi-modal architectures, the modality-bridging adapter maps frozen encoder output $h_{radio} \in \mathbb{R}^{1 \times 128}$ to the LLM embedding space using $k$0, $k$1, prior to layer normalization.

For transformer adaptation, ARLISS employs Low-Rank Adaptation (LoRA) by adding a trainable low-rank update $k$2 to the frozen weight $k$3 in each transformer query/key projection, with $k$4, $k$5, and $k$6, yielding $k$7. Only $k$8 and $k$9 are updated during fine-tuning, limiting parameter growth (Jiao et al., 15 May 2025).

3. Multi-Task Training Paradigms and Task Conditioning

ARLISS supports flexible adapter composition to enable unified or task-specialized predictions:

• Single-task adapters: One dedicated adapter per layer per task.
• Stacking: New adapters are added on top of the base stack, with bottom adapters frozen and reused.
• Fusion ("Fast Fusion"): Independently trained adapters for each task are linearly combined via learned gating coefficients at each layer: $x \in \mathbb{R}^d$0, where $x \in \mathbb{R}^d$1 are gating scalars or vectors.

For conditional multi-task learning, downstream tasks are encoded either by prompt (spoken SLP) or via task keywords and learnable prefix tokens (AI2MMUM). In the telecom domain, task-aware instruction tokens combine fixed keywords (corresponding to each specific task) and a trainable prefix of three embedding vectors, together forming a task-specific context that shapes the model’s attention and downstream feature extraction. This composite prompt is updated jointly with LoRA adapter parameters and task-level output heads (Suresh et al., 2024, Jiao et al., 15 May 2025).

Losses are tailored per task (e.g., mean squared error for regression, cross-entropy or focal loss for classification), and mini-batching is typically organized by task to ensure isolated gradient updates for task-exclusive adapters and prompt parameters.

4. Parameter Efficiency and Incremental Scalability

ARLISS significantly improves parameter efficiency versus conventional model-per-task or multi-decoder setups. Each adapter module introduces $x \in \mathbb{R}^d$2262k parameters per module with $x \in \mathbb{R}^d$3 and $x \in \mathbb{R}^d$4, or $x \in \mathbb{R}^d$515.7M parameters for a full stack of adapters in a 30-layer backbone, which is substantially fewer than a full model or multi-decoder fine-tuning approach. For example, on the SUPERB suite of SLP tasks, ARLISS requires only 89.3% of the parameters compared to the baseline multi-decoder (6 tasks: 113.1M vs. 126.6M parameters), and for expanded setups (9 tasks), only 53.6% (135.6M vs. 252.8M) (Suresh et al., 2024).

In stacking or fusion settings, parameter sharing further reduces the incremental cost of adding new tasks. By learning only task-specific adapters and leverage gating, storage and training overhead are minimized, and the core model remains unchanged, enhancing system maintainability and scalability.

5. Empirical Evaluation Across Multiple Domains

For spoken language, ARLISS achieves competitive or superior results relative to existing solutions, with average performance improvements of 18.4% across five standardized tasks (LibriSpeech ASR, Phoneme Recognition, IEMOCAP Emotion Recognition, FSC Intent Classification, SNIPS Slot Filling) (Suresh et al., 2024). Notable gains include a phoneme recognition error rate (PER) reduction to 2.4%, IEMOCAP emotion recognition accuracy increase to 68.2%, and slot filling F1 increase to 95.4% with an 11.8 CER.

For multi-modal telecom applications, ARLISS-style adapters in the AI2MMUM model achieved state-of-the-art results on unseen WAIR-D and DeepMIMO benchmarks. Direct positioning CDF90 errors improved from ~2.3 m (best non-LLM baseline) to ~1.8 m, LOS/NLOS classification improved from 92% to 96%, MIMO precoding accuracy (SGCS) increased by 10%, beam-selection top-1 accuracy increased from 78% to 86%, and path-loss RMSE dropped by 15% (Jiao et al., 15 May 2025).

Ablation studies reveal that omitting LoRA, trainable prefix embeddings, or frozen encoders systematically degrades performance, confirming that the adapter-based approach is foundational to both accuracy and flexible multi-modality.

6. Training Regimens and Hyperparameter Configurations

Adapters are trained using Adam or AdamW optimizers with distinct learning rates for LoRA, bridging adapters, prefix embeddings, and output heads. For AI2MMUM, the bridging adapter dimension is 128 to 4096, LoRA rank $x \in \mathbb{R}^d$6, prefix length is 3 tokens, and batch size is 32 per GPU. Training includes a linear warmup phase and cosine decay, with gradient steps typically totaling 20,000 for telecom models. In all cases, the main backbone and embedding layers are strictly frozen; only adapters, task prompt parameters, and output heads are updated (Suresh et al., 2024, Jiao et al., 15 May 2025).

7. Significance and Broader Implications

The adapter-based unified model paradigm enables rapid adaptation to new tasks or modalities with minimal overhead. By decoupling the task-conditioned components from the frozen general-purpose encoder–decoder, ARLISS accommodates multi-task and multi-modal deployments, scalable expansion, and simplified deployment pipelines, without compromising core model generalization or inflating parameter counts. This design is particularly advantageous in settings where resource constraints, incremental update requirements, or cross-domain generalization are paramount.

The success of ARLISS across speech, language, and multi-modal telecom benchmarks suggests that adapter-based methods constitute a generalizable approach to scalable, maintainable, and high-performing unified models in a range of data domains and architectures (Suresh et al., 2024, Jiao et al., 15 May 2025).