Mora-Conditioned Fusion

• Mora-Conditioned Fusion is a technique that conditions neural feature integration on fine-grained linguistic or modality-specific signals for improved model sensitivity.
• It employs cross-attention mechanisms to align acoustic features with mora embeddings, enabling precise mapping of pitch accent and error localization.
• The approach also leverages modality-aware low-rank adaptation to maintain robust performance even under missing input conditions while minimizing parameter overhead.

Mora-Conditioned Fusion refers to a class of fusion mechanisms that incorporate mora-level, modality-level, or linguistic-unit-level information as conditioning variables or adaptation axes in neural architectures. In recent literature, the term has precise instantiations in both speech processing—where moras are explicit phonological units central to Japanese accent structure—and cross-modal adaptation contexts, where "MoRA-conditioned fusion" denotes conditional, modality-aware low-rank adaptation for robust feature integration. These mechanisms enable models to dynamically exploit or ignore specific unit representations or modalities, leading to sensitivity to fine-grained errors or robust performance under missing input conditions.

1. Mora-Conditioned Fusion in Speech Quality Assessment

In pitch-accent-focused speech quality assessment, as exemplified by PASQA, mora-conditioned fusion leverages linguistic unit (mora) information to align text-derived accent structure with acoustic representations. The mechanism proceeds as follows (Kawamura et al., 18 Jun 2026):

• Mora Representation and Embedding: Input text is segmented into a sequence of moras (using, e.g., MeCab), each indexed $\ell=1,\dots,L$ and mapped to learnable embeddings $e_\ell \in \mathbb{R}^{d_m}$, with rotary positional encoding (RoPE) augmenting positional information.
• Contextualization: A lightweight Transformer encoder processes the embedding sequence, outputting contextualized mora representations $\mathbf{M} \in \mathbb{R}^{L\times d_m}$.
• Cross-Attention Fusion: Self-supervised acoustic features $\mathbf{A} \in \mathbb{R}^{T\times d_a}$ from a wav2vec 2.0 front end are fused with moras via multi-head cross-attention. Query matrices are computed per frame from $\mathbf{A}$, while key and value matrices are obtained from $\mathbf{M}$. The output $\mathbf{F} = \mathrm{LayerNorm}(\mathbf{A} + O)$, with $O$ the concatenated attention heads post-projection, constitutes the mora-conditioned acoustic representation at each frame.
• Downstream Heads: The fused representation feeds:
  • An accent-quality prediction MLP head (using Bradley–Terry ranking and regression loss),
  • A frame-level accent-error head,
  • A speaker-invariant classifier branch via gradient reversal.

This module explicitly aligns temporal-acoustic events with moraic structure, enabling enhanced detection of localized pitch-accent errors.

2. Mathematical Formulation

The core of mora-conditioned fusion in PASQA is cross-modal attention, instantiated as: $$Q_i = \mathbf{A} W^Q_i ,\quad K_i = \mathbf{M} W^K_i ,\quad V_i = \mathbf{M} W^V_i$$ for each attention head $i=1, ..., H$ (typically $e_\ell \in \mathbb{R}^{d_m}$0).

Head outputs are computed by: $e_\ell \in \mathbb{R}^{d_m}$1 with all heads concatenated and linearly projected. This yields integrated frame-wise features that reflect both the acoustic and mora-level contexts.

The residual connection and layer normalization post-fusion further stabilize training and ensure gradient flow.

3. Integration into End-to-End Architectures

Mora-conditioned fusion is typically embedded mid-pipeline, downstream of frame-level acoustic encoding and upstream of score prediction heads. In PASQA (Kawamura et al., 18 Jun 2026), the complete network component flow is:

Stage	Output Dimensions	Key Components
wav2vec2.0	$e_\ell \in \mathbb{R}^{d_m}$2	Acoustic frame features
Mora Embedding + RoPE	$e_\ell \in \mathbb{R}^{d_m}$3	Tokenized mora sequence contextualized
Cross-Attention Fusion	$e_\ell \in \mathbb{R}^{d_m}$4	Multi-head fusion of acoustic + mora
Accent-Quality Head	$e_\ell \in \mathbb{R}^{d_m}$5	Score prediction via MLP layers
Frame-Error Head	$e_\ell \in \mathbb{R}^{d_m}$6	Framewise error via MLP
Speaker Classifier (GRL)	$e_\ell \in \mathbb{R}^{d_m}$7	Adversarial speaker prediction

Residual ablation experiments demonstrate that removing mora-conditioned fusion leads to absolute drops of 5–6 points in order accuracy and 3–8 points in SRCC, highlighting its role in improving sensitivity to accent error severity (Kawamura et al., 18 Jun 2026).

4. Mora-Conditioned Fusion in Modality-Aware Low-Rank Adaptation

In multi-modal adaptation contexts, MoRA-conditioned fusion refers not to linguistic moras, but to a mechanism that adapts backbone multimodal networks (such as CLIP or ViLT) using conditioning on input modality presence (Zhao et al., 9 Nov 2025, Shi et al., 2024). The fusion mechanism consists of:

• Modality-Specific Low-Rank Updates: Each modality $e_\ell \in \mathbb{R}^{d_m}$8 (vision, text, etc.) receives trainable low-rank down-projection $e_\ell \in \mathbb{R}^{d_m}$9 and up-projection $\mathbf{M} \in \mathbb{R}^{L\times d_m}$0 matrices; adaptation occurs only if modality $\mathbf{M} \in \mathbb{R}^{L\times d_m}$1 is present as governed by indicator $\mathbf{M} \in \mathbb{R}^{L\times d_m}$2.
• Shared/Branch-Common Adaptation: Optionally, MoRA introduces shared parameter matrices whose effect is gated (zeroed) if the corresponding modality is missing.
• Additive Fusion: Adaptation vectors $\mathbf{M} \in \mathbb{R}^{L\times d_m}$3 are summed and injected as increments into early transformer layers, immediately modifying the representation space according to modality context.
• Computational Efficiency: MoRA modules typically introduce $\mathbf{M} \in \mathbb{R}^{L\times d_m}$4 of the parameters of full fine-tuning but achieve competitive or superior F1 under missing modality regimes (Shi et al., 2024).

This approach generalizes to $\mathbf{M} \in \mathbb{R}^{L\times d_m}$5 modalities, with binary indicators for presence/absence gating the adaptation.

5. Training Objectives and Loss Functions

The fusion output supports diversified loss landscapes as required by the application domain:

• In PASQA (Kawamura et al., 18 Jun 2026):
  • Bradley–Terry loss encourages correct prediction ordering by accent error rate:

  $\mathbf{M} \in \mathbb{R}^{L\times d_m}$6 - $\mathbf{M} \in \mathbb{R}^{L\times d_m}$7 and framewise cross-entropy losses jointly optimize rating regression and error localization. - A speaker-classification loss with gradient reversal regularizes representations for invariance.

• In MoRA (Zhao et al., 9 Nov 2025, Shi et al., 2024):

  • Standard cross-entropy is used for final predictions.
  • Low-rank parameter regularization constrains adaptation complexity.
  • The fusion parameters (e.g., $\mathbf{M} \in \mathbb{R}^{L\times d_m}$8, $\mathbf{M} \in \mathbb{R}^{L\times d_m}$9, $\mathbf{A} \in \mathbb{R}^{T\times d_a}$0, $\mathbf{A} \in \mathbb{R}^{T\times d_a}$1) are updated exclusively, leaving the backbone frozen.

6. Quantitative Impact and Ablations

Mora-conditioned fusion yields quantifiable improvements:

Setting	Without Fusion	With Fusion	$\mathbf{A} \in \mathbb{R}^{T\times d_a}$2 Order Acc. / F1
PASQA Seen-Speaker Order Acc.	0.695	0.754	+0.059
PASQA Unseen-Speaker Order Acc.	0.735	0.785	+0.050
MoRA (CXR F1, 100/30 Input)	26.19	27.48	+1.29
MoRA (ODIR F1, 100/30 Input)	58.96	60.94	+1.98

These increments establish mora-conditioned or MoRA-conditioned fusion as critical to robust performance under fine-grained error and missing modality conditions.

7. Implementation Parameters

• PASQA (Kawamura et al., 18 Jun 2026):
  • wav2vec2.0 (acoustic base, $\mathbf{A} \in \mathbb{R}^{T\times d_a}$3)
  • Mora embedding $\mathbf{A} \in \mathbb{R}^{T\times d_a}$4, RoPE
  • Transformer encoder (1 layer, 4 heads, $\mathbf{A} \in \mathbb{R}^{T\times d_a}$5)
  • Cross-attention (4 heads, $\mathbf{A} \in \mathbb{R}^{T\times d_a}$6), residual fusion, dropout $\mathbf{A} \in \mathbb{R}^{T\times d_a}$7
  • Head MLPs ($\mathbf{A} \in \mathbb{R}^{T\times d_a}$8 hidden units), SGD optimizer ($\mathbf{A} \in \mathbb{R}^{T\times d_a}$9‐3, momentum $\mathbf{A}$0), batch size $\mathbf{A}$1
• MoRA (Zhao et al., 9 Nov 2025, Shi et al., 2024):
  • Intrinsic rank $\mathbf{A}$2–$\mathbf{A}$3, block-level LoRA injection
  • Gated adaptation per modality using binary indicators
  • Parameter cost $\mathbf{A}$41.6% of full model, parameter-efficient for N modalities; inference-time fusion folded into backbone weights

Summary

Mora-Conditioned Fusion encompasses attention-based and low-rank adaptation mechanisms that condition neural feature fusion on the presence, sequence, or nature of language or modality units. In speech applications, it enables precise mapping of linguistic structure onto acoustic signals, producing heightened sensitivity to accent errors. In multimodal settings, it facilitates robust adaptation under incomplete input while minimizing trainable parameter footprint. Ablation analyses consistently demonstrate the necessity of such conditioned fusion for preserving ordering, correlation, and accuracy under challenging evaluation regimes (Kawamura et al., 18 Jun 2026, Zhao et al., 9 Nov 2025, Shi et al., 2024).