---
title: Multimodal Unified Attention Network (MUAN)
url: https://www.emergentmind.com/topics/multimodal-unified-attention-network-muan
type: topic
---

# Multimodal Unified Attention Network (MUAN)

The Multimodal Unified Attention Network (MUAN) is a neural network architecture for vision-and-language tasks that integrates intra-modal (within-modality) and inter-modal (cross-modality) reasoning through a unified attention framework. By using a stack of deep, identical “Unified Attention” (UA) blocks, MUAN simultaneously computes self-attention and co-attention between textual and visual features, enabling robust multimodal interaction. The architecture has demonstrated state-of-the-art or near SOTA results on visual question answering (VQA) and visual grounding benchmarks without requiring task-specific modules or highly engineered tricks [1908.04107].

## 1. Architecture Overview

MUAN comprises $L$ identical Unified Attention (UA) blocks stacked in depth. The initial UA block ingests two input feature sets:
- $X\in\mathbb R^{m\times d_x}$: sequence of $m$ word embeddings for text
- $Y\in\mathbb R^{n\times d_y}$: set of $n$ visual region features (e.g., from a CNN)

These are projected (possibly through learned projections $\mathrm{FC}_x$ and $\mathrm{FC}_y$) into a common $d$-dimensional space and concatenated:
$$
Z^{(0)} = \begin{bmatrix} \mathrm{FC}_x(X) \\ \mathrm{FC}_y(Y) \end{bmatrix} \in \mathbb{R}^{(m+n)\times d}
$$
Each of the $L$ UA blocks processes $Z^{(\ell)}$ and outputs $Z^{(\ell+1)}$ of the same shape. At the network’s output, features are split back into their text and image components as necessary for downstream heads.

## 2. Unified Attention Mechanism

Each UA block performs self-attention over the concatenated multimodal features. Query ($Q$), Key ($K$), and Value ($V$) matrices are projected from $Z$:
$$
Q = Z W^Q,\quad K = Z W^K,\quad V = Z W^V
$$
Standard attention is computed as:
$$
A = \mathrm{softmax}\left(\frac{Q K^\top}{\sqrt{d}}\right),\quad Z' = A V
$$
The partitioning of $Z$ as $[\text{text}; \text{image}]$ enables $A$ to encode:
- $A_{TT}$: text–text (intra-modal)
- $A_{VV}$: image–image (intra-modal)
- $A_{TV}$: text-to-image (inter-modal)
- $A_{VT}$: image-to-text (inter-modal)

MUAN further inserts a learned gating mechanism (“Gated Self-Attention”, GSA) for additive selectivity:
$$
U_q = Q W^{qg},\quad U_k = K W^{kg}
$$
$$
M = \sigma((U_q \odot U_k)W^g) \in \mathbb{R}^{(m+n)\times 2}
$$
$M$ is split into $M_q$, $M_k$ (row-wise scaling masks), and applied channel-wise:
$$
\tilde Q = Q \odot M_q,\quad \tilde K = K \odot M_k
$$
Attention is then recomputed as $A^g = \mathrm{softmax}((\tilde Q \tilde K^\top)/\sqrt{d})$ and $Z' = A^g V$. GSA enables strong suppression of irrelevant tokens or regions.

## 3. Core Components: Multihead, Normalization, and Feed-Forward

Each UA block contains:
- **Multihead Gated Attention**: $Q$, $K$, $V$ are split into $h$ heads (typically $h=8$), independently attended and concatenated, with shared output projection $W^O$.
- **Residual Connections + LayerNorm**: Residual update is combined with LayerNorm:
  $$
  H = \mathrm{LayerNorm}(Z + \mathrm{MultiHeadGSA}(Z))
  $$
- **Feed-Forward Network (FFN)**: Two-layer MLP ($4d$ hidden size, ReLU, Dropout=0.1), also wrapped with residual + LayerNorm:
  $$
  H' = \mathrm{LayerNorm}(H + \mathrm{FFN}(H))
  $$
Default per-block hyperparameters: $d=768$, $h=8$, gating-projection dimension $d_g=96$, inner FFN dimension $3072$, dropout 0.1.

## 4. Parameterization and Layer Specialization

Parameterization is strictly per-layer: All projections ($W^Q, W^K, W^V, W^O, W^{qg}, W^{kg}, W^g$) and FFN layers are unique within each UA block. Only the first UA block includes explicit $\mathrm{FC}_x$ and $\mathrm{FC}_y$ projections that map raw text/image features to the unified $d$-dimensional space; all subsequent layers treat input as a single modality.

## 5. Objective Functions for Downstream Tasks

### Visual Question Answering (VQA)
After $L$ stacked UA blocks, attended text features $Z_T^{(L)}$ are extracted. A dummy $[\mathsf{ans}]$ token is prepended to each question; its embedding $z^{(L)}_{\mathsf{ans}}$ is used for answer classification:
- For multiple annotators (soft targets), binary cross-entropy loss:
  $$
  \mathcal{L}_\mathrm{VQA} = \sum_{i=1}^k \left[ y_i\log\sigma(p_i) + (1-y_i)\log(1-\sigma(p_i)) \right]
  $$
- For single ground-truth answers, standard softmax cross-entropy is used.

### Visual Grounding
For each region proposal $z_i$, two heads are applied: scalar score $s_i$ and 4-dimensional bounding box regression $t_i$.
- **Ranking loss**: Kullback-Leibler divergence between predicted region scores and targets.
- **Box-regression loss**: Smooth-$L_1$ loss.
- **Total loss**: $\mathcal{L}_\mathrm{VG} = \mathcal{L}_\mathrm{rank} + \lambda \mathcal{L}_\mathrm{reg}$ ($\lambda=0.5$).

## 6. Experimental Setup and Results

Training uses Adam optimizer ($\beta_1=0.9$, $\beta_2=0.99$), batch size 64, and staged learning rate schedule.

**VQA-v2 Dataset**:
- Image features: Bottom-Up attention (Faster-RCNN on Visual Genome), $n\approx36$, 2048-D.
- Question length $m=14$ (+1 dummy), answer vocab $k=3129$.
- Best MUAN: $L=10$ UA blocks, $d=768$, $h=8$.
- Test-std (train+val+VG for training): 71.10% overall accuracy. Previous best single-model (MCAN): ~70.9%.

**CLEVR Dataset**:
- Image features: $14\times14$ ResNet-101 grids (“res4b22”), $n=196$, 1024-D.
- Max question length 43, answer vocab 28.
- Result: 98.7% accuracy (FiLM and MAC: 98.9%).

**RefCOCO / RefCOCO+ / RefCOCOg (Visual Grounding):**
- $n=100$ region proposals (Faster-RCNN or Mask-RCNN), feature dim 4096 (visual/spatial).
- Query length $m=15$.
- Results using Mask-RCNN on COCO:

| Dataset           | MUAN (%) | Prior SOTA (%) |
|-------------------|----------|---------------|
| RefCOCO testA     | 82.8     | 81.1          |
| RefCOCO testB     | 78.6     | 70.0          |
| RefCOCO+ val      | 68.9     | 56.0          |
| RefCOCOg val      | 71.0     | 66.6          |

These metrics show MUAN meets or exceeds prior SOTA on both VQA and visual grounding tasks using a unified, stackable attention formulation, without auxiliary architectural modifications [1908.04107].

## 7. Context and Significance

Prior to MUAN, state-of-the-art multimodal models for VQA and visual grounding used co-attention to couple modality pairs but lacked mechanisms for intra-modal context sharing. By recasting both intra- and inter-modal reasoning as a single attention operation on concatenated features and augmenting this with learned gating, MUAN provides a generic, stackable module that generalizes standard transformer-style architectures to the multimodal regime. This suggests that strong multimodal fusion and deep reasoning in joint space can be achieved without explicit modality-specific architectural “bells and whistles.” The improvement in both VQA and grounding aligns with the architectural hypothesis that unified, deep attention enables synergetic multimodal feature interaction [1908.04107].

Source: https://www.emergentmind.com/topics/multimodal-unified-attention-network-muan