Primary-Auxiliary Fusion Module

• Primary-Auxiliary Fusion Module is an architectural component that fuses auxiliary features with primary task representations to improve inference accuracy.
• It employs diverse fusion mechanisms—such as cross-attention, gating, and element-wise multiplication—to directly condition and reshape primary signals.
• This design framework enhances robustness, regularization, and performance across domains like ASR, emotion recognition, and deepfake detection.

Searching arXiv for the primary paper and related uses of primary–auxiliary fusion across domains. A Primary-Auxiliary Fusion Module is an architectural component in multi-task or multimodal systems that integrates representations associated with a primary task and an auxiliary task so that the auxiliary signal can directly influence the main prediction pathway. In the most explicit formulation in speech recognition, predicted articulatory trajectories are injected into an automatic speech recognition pipeline via a cross-attention block placed before the decoder, so that articulatory information functions both as an auxiliary supervision target and as a pseudo-input to the recognizer (Attia et al., 1 Oct 2025). Across recent work, closely related modules appear in multimodal emotion recognition, transparent object tracking, deepfake detection, medical image segmentation, object detection, traffic imputation, image fusion, and bimanual manipulation. Although the fusion operators differ—cross-attention, gating, element-wise multiplication, concatenation with projection, or channel mixing—the shared pattern is the use of an auxiliary representation to reshape, regularize, or condition the primary representation rather than merely adding an extra loss term.

1. Conceptual definition and scope

In the narrow sense exemplified by articulation-informed ASR, the Primary-Auxiliary Fusion Module is the stage where the model’s predicted auxiliary features are reintroduced into the main inference stream. The paper “Articulation-Informed ASR: Integrating Articulatory Features into ASR via Auxiliary Speech Inversion and Cross-Attention Fusion” describes two coupled roles for articulatory information: an Auxiliary Task, where the model jointly predicts text and articulatory trajectories, and a Primary-Auxiliary Fusion Module, where predicted vocal tract variables are injected into the recognition pipeline just before decoding through a cross-attention fusion block (Attia et al., 1 Oct 2025).

A broader reading across the cited literature suggests that the term denotes a family of designs in which the auxiliary branch is not limited to regularizing training through an additional loss. Instead, the auxiliary branch produces features, priors, heatmaps, or action tokens that are fused with the primary branch at an intermediate stage. This suggests a functional distinction between ordinary multi-task learning and primary-auxiliary fusion: in the latter, the auxiliary stream directly participates in representation formation for the main output.

The “primary” signal is task-dependent. In ASR it is the acoustic recognition stream; in deepfake detection it is the supervised RGB classification stream; in transparent object tracking it is the pre-trained tracker backbone; in bimanual manipulation it is the arm assigned to the core operation; in traffic imputation it is the set of internal spatio-temporal dependencies; and in infrared-visible fusion it is the visible branch while infrared and prior-based cues play an auxiliary role (Reddy et al., 2 Jan 2026). The “auxiliary” signal is correspondingly diverse: speech inversion targets, self-supervised texture reconstruction, transparency-aware features, external traffic attributes, or role-conditioned action information.

2. Canonical formulation in articulation-informed ASR

The most direct and formally specified Primary-Auxiliary Fusion Module in the provided corpus appears in articulation-informed ASR. The model pipeline is: raw acoustic waveform, a Wav2Vec2.0 backbone, a speech inversion layer that predicts articulatory trajectories, a cross-attention fusion module, and finally a CTC layer for grapheme prediction (Attia et al., 1 Oct 2025). This creates a two-path structure in which the same encoder supports both the primary recognition objective and the auxiliary articulatory prediction objective.

Within this design, the keys and values are acoustic embeddings from the Wav2Vec2.0 encoder, with $K, V \in \mathbb{R}^{T \times d}$, while the queries are predicted vocal tract variables after feed-forward mapping, $Q \in \mathbb{R}^{T \times d'}$. The articulatory trajectories are temporally downsampled to match the stride and output rate of the acoustic embeddings at 50 Hz (Attia et al., 1 Oct 2025). The cross-attention follows the standard transformer form:

$$Q = \mathrm{Linear}_Q(\hat{X}_{\mathrm{TV}}), \quad K = \mathrm{Linear}_K(X_{\mathrm{acoustic}}), \quad V = \mathrm{Linear}_V(X_{\mathrm{acoustic}})$$

and

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V.$$

The operational interpretation given in the source is that the fused representation at each step is constructed by querying the acoustic representations with the predicted articulatory features, thereby weighting the acoustic input in light of articulatory context (Attia et al., 1 Oct 2025). The block is implemented as a single cross-attention block before the CTC layer, although the description notes that it can be stacked.

Training is joint. The ASR branch uses CTC loss, and the speech inversion branch uses MAE (Mean Absolute Error) loss between predicted and reference vocal tract variables. The two are combined with a multi-task objective that may be weighted manually or with uncertainty-based weighting:

$$\mathcal{L}_{\mathrm{total}} = \frac{1}{\sigma_{\mathrm{CTC}}^2}\mathcal{L}_{\mathrm{CTC}} + \frac{1}{2\sigma_{\mathrm{MAE}}^2}\mathcal{L}_{\mathrm{MAE}} + \log \sigma_{\mathrm{CTC}} + \log \sigma_{\mathrm{MAE}}.$$

In this formulation, the auxiliary stream contributes both through representation fusion and through explicit supervision (Attia et al., 1 Oct 2025).

3. Core design patterns across architectures

Across the papers on arXiv, primary-auxiliary fusion modules recur in several architectural patterns.

The first pattern is cross-attention-based conditioning. In ASR, articulatory predictions query acoustic embeddings (Attia et al., 1 Oct 2025). In multimodal emotion recognition, a K-layer multi-head cross-attention module lets text attend to audio and audio attend to text, while auxiliary tasks are added to alleviate insufficient fusion between modalities; empirically, $K=2$ yields best performance on IEMOCAP (Sun et al., 2023). In PA-BiCoop, the auxiliary decoder contains a cross-attention block that aligns action tokens from the primary arm with visual and linguistic context so that auxiliary-arm prediction is directly conditioned on the primary arm’s action (Qicheng et al., 26 Jun 2026). These examples share a transformer-style mechanism in which one stream selectively reads from another.

The second pattern is feature-level multiplicative or gated fusion. Fusion-SSAT fuses features from a self-supervised auxiliary reconstruction branch and a primary deepfake detection branch using element-wise multiplication,

$z = f'(\tilde{L}(v_i)) \odot f(R(v_j)),$

and uses the fused vector for final classification (Reddy et al., 2 Jan 2026). PAST uses a cross-gated module in which auxiliary embeddings and primary hidden vectors modulate one another through sigmoid gates, producing auxiliary hidden representations from mixtures of external and intrinsic signals (Hu et al., 17 Nov 2025). EAPFusion uses FiLM-based cross-modal gating followed by channel shuffling and channel-wise mixing so that infrared and visible features are modulated before channel-level fusion (Sun et al., 3 May 2026). In these systems, the auxiliary stream does not merely append information; it scales or filters the primary stream.

The third pattern is concatenation plus projection into a target latent space. In transparent object tracking, primary tracker features, auxiliary transparency features, and an optional query embedding are concatenated at each spatial position, processed by a transformer encoder, and then projected by an MLP back into the fixed latent space expected by the downstream tracker (Garigapati et al., 2023). This pattern is used specifically to preserve compatibility with fully pre-trained transformer-based trackers that are sensitive to latent-space perturbations.

The fourth pattern is multi-branch fusion in feature pyramids or dual-stream decoders. In SS-MAF for hard exudate segmentation, segmentation and super-resolution features are concatenated, aligned with a $1 \times 1$ convolution to a hidden space of dimension $d=32$, processed through a split spatial convolutional layer, and then reweighted by task-specific attention (Zhang et al., 2022). In MAF-YOLO and MHAF-YOLO, shallow backbone features and deeper neck features are fused through Superficial Assisted Fusion (SAF) and Advanced Assisted Fusion (AAF) modules within a Multi-Branch Auxiliary FPN (Yang et al., 2024). This suggests a spatially structured version of primary-auxiliary fusion in which one stream supplies detail-rich or semantically rich support to another across scales.

4. Functional roles of the auxiliary branch

One recurrent role of the auxiliary branch is to provide complementary representational content that the primary stream does not encode robustly on its own. In articulation-informed ASR, the auxiliary stream supplies articulatory trajectories that had been “once sidelined in ASR research” but are reintroduced through modern architectures (Attia et al., 1 Oct 2025). In SS-MAF, super-resolution is explicitly described as bringing in helpful detailed features for tiny lesion and boundary detection (Zhang et al., 2022). In EAPFusion, infrared contributes thermal saliency while visible imaging contributes fine textures, and the fusion design is explicitly motivated by the need to highlight targets while preserving details (Sun et al., 3 May 2026).

A second role is regularization and representation shaping. Fusion-SSAT places strong emphasis on the auxiliary reconstruction task, using

$$L = \lambda \cdot L_{\mathrm{cls}} + (1-\lambda)\cdot L_{\mathrm{rec}}$$

with $Q \in \mathbb{R}^{T \times d'}$0, meaning a 0.9 weight on the reconstruction auxiliary task, to emphasize representation learning via the auxiliary task (Reddy et al., 2 Jan 2026). Self-MI uses auxiliary unimodal tasks and Contrastive Predictive Coding to maximize the mutual information between unimodal representations and the multimodal fusion representation, thereby encouraging the fused representation to preserve informative unimodal content (Nguyen et al., 2023). In these cases, the auxiliary branch shapes the learned latent space as much as it supplements it.

A third role is disambiguation under missing, noisy, or weakly observed conditions. PAST distinguishes primary patterns, derived from intrinsic relationships among traffic nodes, from auxiliary patterns, influenced by timestamps and node attributes, and lets the two interact via shared hidden vectors under an ensemble self-supervised framework (Hu et al., 17 Nov 2025). In transparent object tracking, transparency-aware auxiliary features compensate for the weakness of general-purpose learned features when object appearance is reflective or adaptive (Garigapati et al., 2023). In head detection, the ALFF branch predicts a heatmap to reinforce target perception in complex scenes with large-scale variation (Chen et al., 2023).

A fourth role is coordination in structured decision-making. PA-BiCoop adopts dynamic primary-auxiliary arm differentiation, where the primary arm handles core operations and the auxiliary arm provides support, with the auxiliary action predicted in the primary arm’s coordinate system (Qicheng et al., 26 Jun 2026). Here, the primary-auxiliary distinction is not only representational but functional, and the fusion mechanism encodes coordinated division of labor.

5. Representative module forms

Domain	Primary signal	Auxiliary signal	Fusion mechanism
ASR (Attia et al., 1 Oct 2025)	Acoustic embeddings	Predicted vocal tract variables	Cross-attention with TVs as queries and acoustic embeddings as keys/values
MER (Sun et al., 2023)	Audio-text fusion stream	Two auxiliary classification tasks	K-layer multi-head cross-attention plus joint auxiliary losses
Transparent tracking (Garigapati et al., 2023)	Tracker backbone features	Transparency features	Concatenation, transformer encoder, MLP projection to fixed latent space
Deepfake detection (Reddy et al., 2 Jan 2026)	Full RGB features	Masked-input LDP-informed features	Element-wise multiplication at embedding level
Traffic imputation (Hu et al., 17 Nov 2025)	Primary spatio-temporal patterns	Embedded external features	Bidirectional cross-gating and shared hidden vectors
Infrared-visible fusion (Sun et al., 3 May 2026)	Visible features	Infrared and intrinsic priors	FiLM gating, channel shuffle/interleaving, local channel mixing
Bimanual manipulation (Qicheng et al., 26 Jun 2026)	Primary arm action tokens	Auxiliary arm conditioning on primary action	Cross-attention in auxiliary decoder

This comparison shows that “fusion” is not a single operator but a design locus. The operator may be attention, gating, multiplication, concatenation with projection, or structured channel interaction. A plausible implication is that the defining criterion of a Primary-Auxiliary Fusion Module is less the specific algebraic form than the architectural fact that auxiliary information participates in intermediate prediction formation for the primary task.

6. Training strategies and optimization couplings

Primary-auxiliary fusion is commonly paired with joint optimization. In the ASR case, CTC loss and MAE loss are optimized together, optionally using uncertainty-based weighting (Attia et al., 1 Oct 2025). In Fusion-SSAT, classification and reconstruction losses are linearly combined, with the auxiliary branch receiving stronger emphasis through the chosen value of $Q \in \mathbb{R}^{T \times d'}$1 (Reddy et al., 2 Jan 2026). In Self-MI, the main multimodal regression objective is optimized together with auxiliary unimodal losses, while CPC-based mutual-information maximization supplies additional alignment pressure (Nguyen et al., 2023).

A second recurring strategy is two-stage or staged training to prevent degeneration of the fusion pathway. Transparent object tracking explicitly proposes a two-step training strategy because the fusion module might otherwise learn to pass through original backbone features and ignore transparency cues. The first step trains with transparency-only input to force latent-space alignment; the second step trains with both original and transparency features while freezing the rest of the tracker (Garigapati et al., 2023). This addresses a common failure mode of primary-auxiliary fusion: if the primary stream is already strong, the auxiliary stream may be ignored unless optimization is structured to make auxiliary use necessary.

A third strategy is self-supervision or pseudo-supervision for the auxiliary branch. In articulation-informed ASR, reference articulatory trajectories come from offline speech inversion rather than direct articulatory measurement (Attia et al., 1 Oct 2025). In Fusion-SSAT, the auxiliary task reconstructs Local Directional Pattern texture maps from heavily masked RGB video inputs (Reddy et al., 2 Jan 2026). In PAST, the system predicts synthetically masked values in an ensemble self-supervised setting (Hu et al., 17 Nov 2025). These examples indicate that primary-auxiliary fusion is often especially attractive where direct labels for the auxiliary phenomenon are scarce or unavailable.

7. Empirical effects, advantages, and limitations

The empirical rationale for Primary-Auxiliary Fusion Modules is typically improved robustness, generalization, or low-resource performance. In articulation-informed ASR, experiments on LibriSpeech show consistent improvements over strong transformer-based baselines, particularly under low-resource conditions (Attia et al., 1 Oct 2025). In multimodal emotion recognition, adding auxiliary tasks to cross-attention fusion improves over vanilla fusion, and the best result is 78.42% Weighted Accuracy (WA) and 79.71% Unweighted Accuracy (UA) with $Q \in \mathbb{R}^{T \times d'}$2 and both auxiliary tasks (Sun et al., 2023). In Fusion-SSAT, feature fusion of self-supervised and primary-task representations improves cross-dataset generalizability, with AUC = 0.9613 in-domain on FF++ and AUC = 0.8930 averaged across cross-domain settings (Reddy et al., 2 Jan 2026).

The advantages are domain-specific but structurally similar. Auxiliary representations can supply complementary cues, counter unimodal dominance, retain small-object or boundary information, and improve coordination or context sensitivity. The PAST results attribute gains to the complementary use of intrinsic and external patterns, reporting that imputation accuracy outperforms seven baselines by up to 26.2% in RMSE and 31.6% in MAE across 27 missing-data conditions (Hu et al., 17 Nov 2025). PA-BiCoop reports that dynamic primary-auxiliary arm differentiation outperforms state-of-the-art baselines by 48% on average in RLBench2 simulation tasks and by over 50% on average in real world tasks (Qicheng et al., 26 Jun 2026).

At the same time, the corpus identifies several limitations and design tensions. Transparent tracking emphasizes that adding new features can perturb a fixed latent space and break pre-trained trackers unless the fusion module explicitly projects back into the expected feature distribution (Garigapati et al., 2023). EAPFusion argues that coarse auxiliary semantics can create a granularity mismatch, making it difficult to highlight targets and preserve details simultaneously (Sun et al., 3 May 2026). These observations caution against a simplistic view that more auxiliary information is automatically beneficial. This suggests that the effectiveness of a Primary-Auxiliary Fusion Module depends heavily on alignment: temporal alignment in ASR, latent-space alignment in tracking, scale alignment in detection and fusion, or role alignment in robotic cooperation.

In aggregate, the literature portrays the Primary-Auxiliary Fusion Module as a general architectural answer to a recurrent problem: the primary stream alone is often insufficiently invariant, insufficiently detailed, or insufficiently coordinated for the target task. By injecting auxiliary information into the representation pathway rather than only into the loss, these modules seek to turn auxiliary learning into a direct mechanism of inference.