Attribute-oriented Mixture of Adapter (AoMoA)

• AoMoA is a parameter-efficient design that distributes adaptation across multiple small experts, each specialized for different inherent input attributes.
• It utilizes explicit attribute tokens and dual routing mechanisms to separate spatial and spectral subspaces, thereby reducing gradient conflicts.
• Empirical results in remote sensing, audio-language modeling, and person retrieval show AoMoA outperforms single dense adapters under similar PEFT budgets.

Attribute-oriented Mixture of Adapter (AoMoA) denotes a parameter-efficient fine-tuning design in which multiple lightweight adapter experts are organized, interpreted, or routed according to attribute structure rather than collapsed into a single monolithic adaptation pathway. In the literature provided here, the term appears explicitly in SpectralX, where spectral images are decomposed into spatial and spectral attribute tokens and then processed by an Attribute-oriented Mixture of Adapter inside a frozen remote-sensing foundation model (Zhang et al., 3 Aug 2025). Closely related work across audio-language modeling, text-based person retrieval, remote sensing, text-to-image control, and large-language-model PEFT shows the same broader principle in explicit or implicit form: heterogeneous inputs contain multiple factors, small experts can specialize to different subspaces, and a router or analogous mechanism can decide how those experts contribute (Lei et al., 6 Jan 2026, Liu et al., 6 Mar 2025, Cao et al., 6 Jun 2025).

1. Conceptual definition and scope

AoMoA is best understood as a specialization of mixture-of-adapters PEFT in which the organizing axis is an “attribute.” In SpectralX, the attributes are not metadata such as region or sensor labels; they are explicitly defined as the two inherent factors extracted from spectral imagery itself: spatial local details and spectral long-range band dependencies (Zhang et al., 3 Aug 2025). The resulting module is therefore “attribute-oriented” in two senses at once: the inputs are attribute tokens, and routing is specialized by attribute branch.

The broader literature shows that AoMoA can also be read more loosely as a design pattern rather than a single fixed module. Some systems are explicit, with named attribute branches and dedicated routers, as in SpectralX. Others are implicit or emergent: the model is trained without attribute labels, but routing behavior later aligns with factor structure such as speech versus music, semantic versus paralinguistic information, or different aspects of person knowledge (Lei et al., 6 Jan 2026, Liu et al., 6 Mar 2025). This suggests that AoMoA currently functions both as a named architecture and as an interpretive lens for related adapter-mixture systems.

A central motivation recurs across these works. Dense single adapters are repeatedly described as inadequate when the input distribution is heterogeneous. In audio-language alignment, a single projector is said to induce “gradient conflict” because speech, music, and environmental sounds require contradictory updates (Lei et al., 6 Jan 2026). In spectral remote sensing, simply resetting the patch projection layer is presented as insufficient for the RGB-to-spectral gap because spectral imagery contains both spatial structure and rich band-wise dependencies (Zhang et al., 3 Aug 2025). In text-based person retrieval, a single adapter is described as too coarse for clothing, accessories, colors, and other fine-grained person semantics (Liu et al., 6 Mar 2025). AoMoA addresses this family of problems by distributing adaptation across multiple small modules and conditioning their use on attribute structure.

2. Canonical explicit formulation in SpectralX

The clearest explicit AoMoA instantiation in the supplied literature is SpectralX (Zhang et al., 3 Aug 2025). Its full pipeline has three conceptual stages. Stage 1 is spectral modality adaptation under an MAE-style masked-reconstruction objective with a frozen RSFM backbone. Stage 2 performs downstream semantic segmentation using UperNet and an additional Attribute-refined Adapter. Stage 3 is deployment on unseen scenes. AoMoA is the main PEFT mechanism in Stage 1.

AoMoA operates on attribute tokens produced by HyperT. HyperT first uses a lightweight CNN, then local self-attention for spatial attributes and global self-attention for spectral attributes, yielding spatial semantic features $\mathbf{Z}_{spa} \in \mathbb{R}^{784 \times 512}$, spectral semantic features $\mathbf{Z}_{spe} \in \mathbb{R}^{512 \times 784}$, and final attribute tokens $\mathbf{T}_{att} \in \mathbb{R}^{196 \times 1024}$ with $L=196$ (Zhang et al., 3 Aug 2025). AoMoA then splits these tokens channel-wise into

$$\mathbf{T}_{spa} \in \mathbb{R}^{196 \times 512}, \qquad \mathbf{T}_{spe} \in \mathbb{R}^{196 \times 512}.$$

Architecturally, AoMoA contains a down-projection matrix, an attribute-specific router bank $\{ \mathcal{R}_{spa}, \mathcal{R}_{spe} \}$, an attribute-shared adapter bank $$\{ \mathcal{A}_1,\mathcal{A}_2,\ldots,\mathcal{A}_{N_a} \}$$, and an up-projection matrix, with $N_a=4$ and $r=512$ (Zhang et al., 3 Aug 2025). For the spatial branch, routing is defined as

$$\mathcal{R}_{spa}(\mathbf{T}_{spa}) = \mathrm{softmax}\Bigl( TopK(\mathbf{T}_{spa}\mathbf{W}^{0}_{gate}) + \mathcal{N}(0,1)\cdot \mathrm{softplus}(\mathbf{T}_{spa}\mathbf{W}^{0}_{noise}) \Bigr),$$

with sparse top-$\mathbf{Z}_{spe} \in \mathbb{R}^{512 \times 784}$0 selection using $\mathbf{Z}_{spe} \in \mathbb{R}^{512 \times 784}$1. The routed output is

$\mathbf{Z}_{spe} \in \mathbb{R}^{512 \times 784}$2

and the spectral branch is treated analogously. The two streams are concatenated, projected back up, scaled by $\mathbf{Z}_{spe} \in \mathbb{R}^{512 \times 784}$3, and injected into the transformer FFN residual: $\mathbf{Z}_{spe} \in \mathbb{R}^{512 \times 784}$4

Insertion is selective rather than ubiquitous. With a ViT-L backbone of 24 transformer blocks, AoMoA is inserted into the FFN of the 6th, 12th, 18th, and 24th layers in both encoder and decoder during Stage 1 (Zhang et al., 3 Aug 2025). In Stage 2, the decoder is removed, the encoder keeps AoMoA, and the Are-adapter is added. This placement makes AoMoA a layer-wise modality-adaptation mechanism rather than a full-transformer MoE.

The key property of this formulation is that expert knowledge is shared, but expert selection is attribute-specific. A standard adapter would apply one bottleneck transformation regardless of whether variation is spatial or spectral. AoMoA instead first decomposes token channels into attribute subspaces and then uses separate routers to choose from a shared adapter bank. That is the explicit core of the approach.

3. Explicit, implicit, and adjacent AoMoA variants

The literature surrounding AoMoA spans several levels of explicitness. Some methods define attributes directly; others rely on latent specialization; still others are adjacent designs that illuminate the same modularization problem.

Work	Specialization axis	Relation to AoMoA
SpectralX (Zhang et al., 3 Aug 2025)	Spatial vs. spectral attributes	Explicit AoMoA with attribute-specific routers over a shared adapter bank
MoE-Adapter for large audio-LLMs (Lei et al., 6 Jan 2026)	Speech, music, sound; semantic vs. paralinguistic factors	Implicit AoMoA-like adapter bank at the audio–LLM bridge
DM-Adapter (Liu et al., 6 Mar 2025)	Distinct aspects of person knowledge under domain-aware routing	Domain-aware rather than explicitly attribute-aware, but close in fine-grained specialization
Zero-shot TTS with mixture of adapters (Fujita et al., 2024)	Speaker characteristics inferred from speaker embeddings	Latent attribute specialists selected from speaker-conditioned routing
Earth-Adapter (Hu et al., 8 Apr 2025)	Spatial, low-frequency, high-frequency components	Frequency-oriented special case of attribute-structured adapter mixing
Att-Adapter (Cho et al., 15 Mar 2025)	Joint continuous multi-attribute control	Unified multi-attribute adapter rather than a mixture of adapter experts

Beyond these, several works are best read as supporting infrastructure for AoMoA rather than direct instances of it. SMoA learns sub-adapters that capture different bias-related patterns and shows that examples with similar bias types tend to activate similar sub-adapters, making it a latent-factor precursor (Liu et al., 2023). MixDA separates frozen general-domain FFNs from domain-specific adapters and dynamically fuses them through a gate, which is architecturally close to an attribute-specialized memory design (Diao et al., 2023). Soft-MoA demonstrates that adapters themselves can act as experts under soft slot-based routing, giving a scalable PEFT substrate for future attribute-aware routing (Cappellazzo et al., 2024). Heterogeneous MoA shows that structurally different PEFT experts can be mixed collaboratively with sigmoid routing, which is directly relevant when attributes are non-exclusive (Cao et al., 6 Jun 2025). This suggests that AoMoA should be treated as a family of adapter-mixture designs whose explicitness varies with the availability of attribute structure.

4. Routing, specialization, and optimization dynamics

A defining property of AoMoA-style systems is conditional rather than uniform adaptation. SpectralX uses attribute-specific sparse top-$\mathbf{Z}_{spe} \in \mathbb{R}^{512 \times 784}$5 routing with noisy logits to avoid some adapters being always selected while others are ignored (Zhang et al., 3 Aug 2025). DM-Adapter routes tokens to six experts with Top-$\mathbf{Z}_{spe} \in \mathbb{R}^{512 \times 784}$6 and further biases routing through learnable domain-aware prompts, modifying the gating term from $\mathbf{Z}_{spe} \in \mathbb{R}^{512 \times 784}$7 to $\mathbf{Z}_{spe} \in \mathbb{R}^{512 \times 784}$8 (Liu et al., 6 Mar 2025). Earth-Adapter uses a softmax router over three experts—spatial, low-frequency, and high-frequency—after DFT-based decomposition (Hu et al., 8 Apr 2025). Soft-MoA replaces hard routing with token-to-slot and slot-to-token soft assignments, and heterogeneous MoA replaces softmax competition with sigmoid collaboration so that multiple structurally different experts can contribute simultaneously (Cappellazzo et al., 2024, Cao et al., 6 Jun 2025).

The literature also converges on a small set of recurrent failure modes. In audio-language alignment, dense shared adapters exhibit persistent negative gradient cosine similarities across heterogeneous categories, while the MoE-Adapter shifts these toward positive values; the text specifically notes that Music–Speech improves to $\mathbf{Z}_{spe} \in \mathbb{R}^{512 \times 784}$9 (Lei et al., 6 Jan 2026). The same paper reports negative cross-task Gradient Influence Scores for dense adapters and predominantly positive influence under sparse routing, which the authors interpret as transforming destructive interference into constructive updates. In DM-Adapter, a separate load-balancing loss is introduced to mitigate routing imbalance (Liu et al., 6 Mar 2025). In homogeneous MoE-LoRA settings, representation collapse and expert redundancy are identified as major problems, motivating orthogonal expert representations in OMoE and heterogeneous expert banks in MoA (Feng et al., 17 Jan 2025, Cao et al., 6 Jun 2025).

A second recurring lesson concerns the granularity of specialization. Several papers favor token-level rather than sample-level behavior. SpectralX routes attribute-token halves at selected transformer layers (Zhang et al., 3 Aug 2025). The audio MoE-Adapter repeatedly states that it routes audio tokens to specialized experts (Lei et al., 6 Jan 2026). DM-Adapter uses token-level expert weighting in both vision and text branches (Liu et al., 6 Mar 2025). Soft-MoA likewise performs token-level soft routing through latent slots rather than instance-level selection, and its appendix reports token-level routing as clearly better than instance-level routing (Cappellazzo et al., 2024). This suggests that AoMoA is most natural when attribute variation is local within the representation stream rather than only global at the sample level.

Capacity allocation is an additional orthogonal concern. HILO argues that mixture-of-adapter systems should vary both expert count and rank across layers, not just the number of experts (Cong et al., 6 Feb 2025). OMoE argues that expert diversity must be actively promoted rather than assumed to arise from multiplicity alone (Feng et al., 17 Jan 2025). Heterogeneous MoA argues that structural diversity of experts can itself serve as a specialization mechanism (Cao et al., 6 Jun 2025). A plausible implication is that AoMoA is not only about what attributes are represented, but also about where and with what capacity each attribute-sensitive expert is instantiated.

5. Empirical behavior, efficiency, and modality-specific evidence

Within explicit AoMoA, the strongest direct evidence comes from SpectralX. In the key ablation table, adding AoMoA on top of HyperT raises WHUOHS mIoU from 49.5 to 55.1 without Stage 1, and from 51.7 to 58.8 with Stage 1; on DFC2020, the corresponding gains are 59.6 to 63.5 and 61.5 to 65.9 (Zhang et al., 3 Aug 2025). The paper further remarks that when AoMoA is applied to multiple transformer blocks, it achieves about a 2% improvement over full fine-tuning on WHUOHS and DFC2020 in the no-domain-gap setting. Efficiency is a central part of the claim: full fine-tuning of ScaleMAE uses 303.4M trainable parameters, whereas the SpectralX configuration with HyperT + AoMoA + Are-adapter uses 11.8M, and AoMoA itself accounts for most of that PEFT budget.

Implicit AoMoA-like evidence is broad. In large audio-LLMs, replacing a dense audio projector with a sparse MoE-Adapter improves MMSU from $\mathbf{T}_{att} \in \mathbb{R}^{196 \times 1024}$0 to $\mathbf{T}_{att} \in \mathbb{R}^{196 \times 1024}$1, OBQA from $\mathbf{T}_{att} \in \mathbb{R}^{196 \times 1024}$2 to $\mathbf{T}_{att} \in \mathbb{R}^{196 \times 1024}$3, and MMAU from $\mathbf{T}_{att} \in \mathbb{R}^{196 \times 1024}$4 to $\mathbf{T}_{att} \in \mathbb{R}^{196 \times 1024}$5 under a comparable total parameter budget of about 94.4M (Lei et al., 6 Jan 2026). The same paper reports emergent modality-aware specialization on MMAU categories Speech, Sound, and Music, including the finding that “no expert is jointly specialized for both Speech and Music,” while Sound acts as an “Acoustic Bridge.” In text-based person retrieval, DM-Adapter raises average R@1 across datasets from 63.89 for a single MLP-Adapter to 64.94 when Sparse Mixture-of-Adapters, load balancing, and the Domain-Aware Router are combined (Liu et al., 6 Mar 2025). In zero-shot TTS, sparse MoA improves over both a small backbone and a larger medium model while keeping near-small-model inference speed; Proposed(s) uses 15M inference parameters versus 42M for the medium baseline and runs at real-time factor 0.0148 versus 0.0286 (Fujita et al., 2024). In remote sensing, Earth-Adapter reports +9.0 mIoU over Rein on DA and +3.1 mIoU on DG by mixing spatial, low-frequency, and high-frequency adapters (Hu et al., 8 Apr 2025).

Att-Adapter is not a mixture-of-adapter bank in the strict sense, but it is a strong unified multi-attribute alternative. On latent-control benchmarks it reports CR 67.87 single and 90.39 multi, versus 26.73 and 31.35 for paired ConceptSlider, and DIS 29.8% single and 32.3% multi, versus 17.5% and 19.8% for the same baseline (Cho et al., 15 Mar 2025). Its role in the AoMoA landscape is instructive: some multi-attribute problems may be handled by one shared stochastic controller rather than a bank of experts, especially when all attributes belong to one coherent domain.

Across these results, three empirical patterns recur. First, multiple small experts often outperform a single dense adapter under similar PEFT budgets. Second, sparse or soft conditional activation can reduce active compute without erasing gains. Third, the most convincing benefits appear when the input space is obviously heterogeneous: spectral remote sensing, multi-category audio, person retrieval, or continuous multi-attribute image control.

6. Misconceptions, limitations, and open directions

A common misconception is that AoMoA necessarily requires one adapter per explicitly labeled attribute. The literature does not support such a narrow reading. SpectralX is explicit, with spatial and spectral branches (Zhang et al., 3 Aug 2025). The audio MoE-Adapter, DM-Adapter, SMoA, and the TTS MoA are attribute-oriented primarily in effect: specialization emerges from routing statistics, benchmark categories, or speaker-conditioned behavior rather than from direct attribute supervision (Lei et al., 6 Jan 2026, Liu et al., 6 Mar 2025, Liu et al., 2023, Fujita et al., 2024). This suggests that explicit supervision is one route to AoMoA, not its only form.

A second misconception is that all “mixture of adapters” methods remain mixtures at inference. MoSA splits a standard adapter into non-overlapping sparse modules during training and then merges them into a single dense adapter for deployment (Zhang et al., 2023). Shortcut-Rerouted Adapter Training routes confounding factors through auxiliary modules during training and removes those modules at inference (Goyal et al., 23 Oct 2025). Att-Adapter uses one unified multi-attribute adapter rather than a bank of experts (Cho et al., 15 Mar 2025). These methods are adjacent rather than identical, but they clarify the design boundary of AoMoA.

The main limitations are also consistent. Many systems remain weak on explicit semantics: the audio MoE-Adapter is not supervised by attribute labels and does not define a known attribute taxonomy (Lei et al., 6 Jan 2026). SpectralX models only two predefined attribute types, spatial and spectral, and does not study temporal, sensor-specific, or region-specific factors (Zhang et al., 3 Aug 2025). DM-Adapter is domain-aware, not attribute-aware in a strict sense (Liu et al., 6 Mar 2025). Heterogeneous MoA improves specialization structurally but does not guarantee that experts correspond to interpretable attributes (Cao et al., 6 Jun 2025). OMoE promotes geometric diversity through orthogonality, but orthogonality alone does not ensure semantic decomposition (Feng et al., 17 Jan 2025).

The surveyed literature points toward several open directions. One is the move from latent specialization to explicit taxonomies and metadata-conditioned routing. Another is the combination of shared adapters for cross-attribute invariants with attribute-specific experts for conflicting factors, a possibility suggested by both SpectralX’s shared bank and the audio finding that Sound can bridge Speech and Music (Zhang et al., 3 Aug 2025, Lei et al., 6 Jan 2026). A third is the integration of orthogonal enhancements—heterogeneous experts, hierarchical rank allocation, or orthogonalized expert outputs—into explicitly attribute-structured systems (Cao et al., 6 Jun 2025, Cong et al., 6 Feb 2025, Feng et al., 17 Jan 2025). A plausible implication is that future AoMoA systems will combine three layers of design: explicit attribute structure, conditional routing, and capacity/diversity mechanisms that prevent collapse while preserving transfer.

Taken together, AoMoA is best characterized not as a single settled architecture but as a research program in structured adapter specialization. Its explicit form is already instantiated in SpectralX. Its implicit forms appear across heterogeneous audio, vision, language, and generative-control settings. Its central claim is consistent across those contexts: when inputs entangle multiple factors, parameter-efficient adaptation becomes more reliable when those factors are decoupled across lightweight experts and reassembled conditionally rather than forced through one shared adapter.