---
title: Mora-Conditioned Fusion
url: https://www.emergentmind.com/topics/mora-conditioned-fusion
type: topic
---

# Mora-Conditioned Fusion

Mora-Conditioned Fusion refers to a class of fusion mechanisms that incorporate mora-level, modality-level, or linguistic-unit-level information as conditioning variables or adaptation axes in neural architectures. In recent literature, the term has precise instantiations in both speech processing—where moras are explicit phonological units central to Japanese accent structure—and cross-modal adaptation contexts, where "MoRA-conditioned fusion" denotes conditional, modality-aware low-rank adaptation for robust feature integration. These mechanisms enable models to dynamically exploit or ignore specific unit representations or modalities, leading to sensitivity to fine-grained errors or robust performance under missing input conditions.

## 1. Mora-Conditioned Fusion in Speech Quality Assessment

In pitch-accent-focused speech quality assessment, as exemplified by PASQA, mora-conditioned fusion leverages linguistic unit (mora) information to align text-derived accent structure with acoustic representations. The mechanism proceeds as follows [2606.20137]:

- **Mora Representation and Embedding**: Input text is segmented into a sequence of moras (using, e.g., MeCab), each indexed $\ell=1,\dots,L$ and mapped to learnable embeddings $e_\ell \in \mathbb{R}^{d_m}$, with rotary positional encoding (RoPE) augmenting positional information.
- **Contextualization**: A lightweight Transformer encoder processes the embedding sequence, outputting contextualized mora representations $\mathbf{M} \in \mathbb{R}^{L\times d_m}$.
- **Cross-Attention Fusion**: Self-supervised acoustic features $\mathbf{A} \in \mathbb{R}^{T\times d_a}$ from a wav2vec 2.0 front end are fused with moras via multi-head cross-attention. Query matrices are computed per frame from $\mathbf{A}$, while key and value matrices are obtained from $\mathbf{M}$. The output $\mathbf{F} = \mathrm{LayerNorm}(\mathbf{A} + O)$, with $O$ the concatenated attention heads post-projection, constitutes the mora-conditioned acoustic representation at each frame.
- **Downstream Heads**: The fused representation feeds:
  - An accent-quality prediction MLP head (using Bradley–Terry ranking and regression loss),
  - A frame-level accent-error head,
  - A speaker-invariant classifier branch via gradient reversal.

This module explicitly aligns temporal-acoustic events with moraic structure, enabling enhanced detection of localized pitch-accent errors.

## 2. Mathematical Formulation

The core of mora-conditioned fusion in PASQA is cross-modal attention, instantiated as:
\[
Q_i = \mathbf{A} W^Q_i ,\quad
K_i = \mathbf{M} W^K_i ,\quad
V_i = \mathbf{M} W^V_i
\]
for each attention head $i=1, ..., H$ (typically $H=4$).

Head outputs are computed by:
\[
\mathrm{head}_i = \mathrm{softmax}(Q_i K_i^\top / \sqrt{d_k}) V_i
\]
with all heads concatenated and linearly projected. This yields integrated frame-wise features that reflect both the acoustic and mora-level contexts.

The residual connection and layer normalization post-fusion further stabilize training and ensure gradient flow.

## 3. Integration into End-to-End Architectures

Mora-conditioned fusion is typically embedded mid-pipeline, downstream of frame-level acoustic encoding and upstream of score prediction heads. In PASQA [2606.20137], the complete network component flow is:

| Stage                       | Output Dimensions              | Key Components                          |
|-----------------------------|-------------------------------|------------------------------------------|
| wav2vec2.0                  | $\mathbb{R}^{T \times 768}$   | Acoustic frame features                  |
| Mora Embedding + RoPE       | $\mathbb{R}^{L \times 256}$   | Tokenized mora sequence contextualized   |
| Cross-Attention Fusion      | $\mathbb{R}^{T \times 768}$   | Multi-head fusion of acoustic + mora     |
| Accent-Quality Head         | $1$                           | Score prediction via MLP layers          |
| Frame-Error Head            | $T$                           | Framewise error via MLP                  |
| Speaker Classifier (GRL)    | $C$                           | Adversarial speaker prediction           |

Residual ablation experiments demonstrate that removing mora-conditioned fusion leads to absolute drops of 5–6 points in order accuracy and 3–8 points in SRCC, highlighting its role in improving sensitivity to accent error severity [2606.20137].

## 4. Mora-Conditioned Fusion in Modality-Aware Low-Rank Adaptation

In multi-modal adaptation contexts, MoRA-conditioned fusion refers not to linguistic moras, but to a mechanism that adapts backbone multimodal networks (such as CLIP or ViLT) using conditioning on input modality presence [2511.06225, 2408.09064]. The fusion mechanism consists of:

- **Modality-Specific Low-Rank Updates**: Each modality $i$ (vision, text, etc.) receives trainable low-rank down-projection $A_i$ and up-projection $B_i$ matrices; adaptation occurs only if modality $i$ is present as governed by indicator $\delta_i$.
- **Shared/Branch-Common Adaptation**: Optionally, MoRA introduces shared parameter matrices whose effect is gated (zeroed) if the corresponding modality is missing.
- **Additive Fusion**: Adaptation vectors $a = \sum_i \delta_i B_i (A_i x_i)$ are summed and injected as increments into early transformer layers, immediately modifying the representation space according to modality context.
- **Computational Efficiency**: MoRA modules typically introduce $<1.6\%$ of the parameters of full fine-tuning but achieve competitive or superior F1 under missing modality regimes [2408.09064].

This approach generalizes to $N$ modalities, with binary indicators for presence/absence gating the adaptation.

## 5. Training Objectives and Loss Functions

The fusion output supports diversified loss landscapes as required by the application domain:

- **In PASQA [2606.20137]**:
  - Bradley–Terry loss encourages correct prediction ordering by accent error rate:
    \[
    \mathcal{L}_{\mathrm{BT}} = -\sum_{i,j : y_i>y_j} \log \sigma(\hat{y}_i-\hat{y}_j)
    \]
  - $L_1$ and framewise cross-entropy losses jointly optimize rating regression and error localization.
  - A speaker-classification loss with gradient reversal regularizes representations for invariance.

- **In MoRA [2511.06225, 2408.09064]**:
  - Standard cross-entropy is used for final predictions.
  - Low-rank parameter regularization constrains adaptation complexity.
  - The fusion parameters (e.g., $A_i$, $B_i$, $S^v$, $S^t$) are updated exclusively, leaving the backbone frozen.

## 6. Quantitative Impact and Ablations

Mora-conditioned fusion yields quantifiable improvements:

| Setting                          | Without Fusion | With Fusion | $\Delta$ Order Acc. / F1 |
|-----------------------------------|---------------|-------------|--------------------------|
| PASQA Seen-Speaker Order Acc.     | 0.695         | 0.754       | +0.059                   |
| PASQA Unseen-Speaker Order Acc.   | 0.735         | 0.785       | +0.050                   |
| MoRA (CXR F1, 100/30 Input)       | 26.19         | 27.48       | +1.29                    |
| MoRA (ODIR F1, 100/30 Input)      | 58.96         | 60.94       | +1.98                    |

These increments establish mora-conditioned or MoRA-conditioned fusion as critical to robust performance under fine-grained error and missing modality conditions.

## 7. Implementation Parameters

- **PASQA [2606.20137]**:
  - wav2vec2.0 (acoustic base, $d_a=768$)
  - Mora embedding $d_m=256$, RoPE
  - Transformer encoder (1 layer, 4 heads, $d_{ff}=512$)
  - Cross-attention (4 heads, $d_k=64$), residual fusion, dropout $0.1$
  - Head MLPs ($64$ hidden units), SGD optimizer ($1e$‐3, momentum $0.9$), batch size $16$
- **MoRA [2511.06225, 2408.09064]**:
  - Intrinsic rank $r=4$–$16$, block-level LoRA injection
  - Gated adaptation per modality using binary indicators
  - Parameter cost $<$1.6% of full model, parameter-efficient for N modalities; inference-time fusion folded into backbone weights

## Summary

Mora-Conditioned Fusion encompasses attention-based and low-rank adaptation mechanisms that condition neural feature fusion on the presence, sequence, or nature of language or modality units. In speech applications, it enables precise mapping of linguistic structure onto acoustic signals, producing heightened sensitivity to accent errors. In multimodal settings, it facilitates robust adaptation under incomplete input while minimizing trainable parameter footprint. Ablation analyses consistently demonstrate the necessity of such conditioned fusion for preserving ordering, correlation, and accuracy under challenging evaluation regimes [2606.20137, 2511.06225, 2408.09064].

Source: https://www.emergentmind.com/topics/mora-conditioned-fusion