DashFusion: Dual-Stream Hierarchical Bottleneck Fusion

• The paper presents an innovative framework that resolves temporal misalignment and semantic heterogeneity using dual-stream alignment and hierarchical bottleneck fusion.
• It employs cross-modal attention and supervised contrastive learning to integrate text, audio, and visual features efficiently.
• Empirical evaluations show that DashFusion outperforms baseline fusion schemes on sentiment analysis benchmarks while balancing accuracy and computational cost.

Dual-stream Alignment with Hierarchical Bottleneck Fusion (DashFusion) refers to an advanced multimodal representation learning framework designed to address the integration and alignment of heterogeneous data modalities, prominently in tasks such as multimodal sentiment analysis. The DashFusion architecture unifies dual-stream temporal-semantic alignment with a computationally efficient, progressive bottleneck fusion strategy, and has shown empirical effectiveness on multiple benchmark datasets (Wen et al., 5 Dec 2025). Closely related variants apply similar principles to event-based recognition with event cameras (Yuan et al., 2023). The following summarizes the methodology, core principles, architectural choices, and experimental findings of DashFusion and related bottleneck fusion models.

1. Conceptual Foundations and Motivation

DashFusion is constructed to resolve two primary obstacles in multimodal learning: temporal misalignment and semantic heterogeneity. Temporal misalignment arises from differing sampling rates and asynchronous dynamics between modalities (e.g., text, visual, audio), making it difficult to synchronize features at the word or frame level. Semantic heterogeneity refers to the disjoint feature spaces induced by distinct unimodal encoders, complicating downstream fusion. Naïve strategies such as token concatenation or dense self-attention across all inputs either lead to sub-optimal integration or excessive quadratic computational overhead. To address these, DashFusion employs a dual-stream alignment module (temporal and semantic) and a hierarchical bottleneck fusion mechanism to realize both accurate prediction and computational efficiency (Wen et al., 5 Dec 2025).

2. Dual-Stream Alignment: Temporal and Semantic Synchronization

Temporal Alignment via Cross-Modal Attention

Textual features $X_t \in \mathbb{R}^{T_t \times d}$ serve as the central reference for alignment. Cross-modal attention projects audio ($X_a$) and visual ($X_v$) features onto the text timeline, allowing frame-level correspondence:

$$\alpha_{i,j} = \frac{\exp(Q_{t,i} K_{s,j}^\top/\sqrt{d})}{\sum_{j'} \exp(Q_{t,i} K_{s,j'}^\top/\sqrt{d})}, \qquad X_{s \to t, i} = \sum_j \alpha_{i,j} V_{s,j}$$

with $Q_t = X_t W_Q, K_s = X_s W_K, V_s = X_s W_V$. The fused aligned representation is:

$$H = X_t + \mathrm{CA}(X_t, X_a) + \mathrm{CA}(X_t, X_v)$$

Semantic Alignment via Contrastive Learning

Semantic alignment ensures modal representations share a coherent feature space. The NT-Xent (normalized temperature-scaled cross-entropy) loss is applied with:

$$\ell_{\mathrm{sem}}(t,i) = -\log \frac{\exp(\mathrm{sim}(z_t, z_i)/\tau)}{\sum_k \exp(\mathrm{sim}(z_t, z_k)/\tau)}$$

where $z_t$ (anchor) is the text embedding for a video, $z_i$ the paired audio or visual feature, $z_k$ negatives from the batch, $\mathrm{sim}(u,v) = u \cdot v/\|u\|\|v\|$, and $\tau$ is a temperature hyperparameter (Wen et al., 5 Dec 2025).

3. Supervised Contrastive Learning Enhancement

DashFusion further extends innate contrastive learning with supervised information. Positives $P(i)$ are chosen as samples with the same sentiment label and high feature similarity; negatives $N(i)$ incorporate both hard negatives (intra-class but dissimilar) and all inter-class instances. The supervised contrastive objective is:

$$\ell_{\mathrm{sup}}(i) = -\frac{1}{|P(i)|} \sum_{p \in P(i)} \log \frac{\exp(\mathrm{sim}(z_i, z_p)/\tau)}{\sum_{a \in P(i) \cup N(i)} \exp(\mathrm{sim}(z_i, z_a)/\tau)}$$

This loss is applied both on unimodal pooled features and the temporally-aligned multimodal feature $H$.

4. Hierarchical Bottleneck Fusion

Multi-Level Bottleneck Design

DashFusion introduces $p$ learnable bottleneck tokens at the outermost layer, halving their number at each layer $l$: $p^l = p/2^l$. At each layer, multimodal tokens $H^{l-1}$ are processed by a Transformer:

$B^l = \mathrm{Transformer}(H^{l-1})[0:p^l]$

Bidirectional cross-modal updates entail:

1. Fusion from modalities to bottleneck:

$$\hat B^l = \mathrm{LayerNorm}\left(B^l + \sum_{m \in \{t,v,a\}} \mathrm{CA}(B^l, X_m^{l-1})\right) + \mathrm{FFN}(\cdot)$$

2. Distribution from bottleneck back to modalities:

$$X_m^l = \mathrm{LayerNorm}(X_m^{l-1} + \mathrm{CA}(X_m^{l-1}, \hat B^l)) + \mathrm{FFN}(\cdot)$$

A simplified perspective equates $B^l$ to a projection of pooled modal features:

$b^l = W_b^l [h_t; h_a; h_v] + b$

Hierarchical reduction of token count per stage maintains a favorable trade-off between accuracy and $O((p^l)^2)$ computational cost, sustaining salient sentiment and discarding modal noise (Wen et al., 5 Dec 2025).

A closely related variant, "Learning Bottleneck Transformer for Event Image-Voxel Feature Fusion based Classification," employs a two-stage fusion: first compressing dense image tokens to bottleneck form, then fusing with voxel-based features via a similar bottleneck Transformer. This also leverages dual-stream alignment, but omits auxiliary alignment losses, using only cross-entropy for supervision (Yuan et al., 2023).

5. Detailed Architecture and Training Protocols

DashFusion adopts a BERT-base text encoder (final embedding projected to $d=128$), with audio/visual Transformers (dimension $d=128$, 2 layers for CMU-MOSI/CH‐SIMS, 3 for CMU-MOSEI). Cross-modal attention employs 4 heads. The hierarchical bottleneck fusion module operates over $L$ layers ($L=2$ or $3$), with initial $p=8$ bottleneck tokens.

Training uses Adam (no weight decay), learning rates of $5\times10^{-5}$ for CMU-MOSI/CH-SIMS and $2\times10^{-5}$ for CMU-MOSEI, batch sizes of 16/8 (epochs 100/25), contrastive loss weight $\lambda=0.2$, and $\tau=0.5$ (Wen et al., 5 Dec 2025).

The analogous event-based model in (Yuan et al., 2023) uses two parallel backbones (a ResNet-18 based vision Transformer and a Structured GNN), with bottleneck tokens bridging the streams through two layers of multi-head self-attention (FusionFormer), and is supervised with cross-entropy.

6. Empirical Evaluation and Ablation Studies

DashFusion demonstrates superior or state-of-the-art performance on CMU-MOSI, CMU-MOSEI, and CH-SIMS sentiment analysis datasets:

Dataset	Acc-2	F1	Acc-7	MAE	Corr
MOSI	85.82	84.17	45.63	0.709	0.796
MOSEI	86.30	82.70	53.12	0.524	0.784
CH-SIMS	79.21	79.39	--	0.416	0.601

Ablation studies confirm that hierarchical bottleneck fusion outperforms alternative fusion schemes (flat bottleneck, simple concat, self-attention, and cross-modal attention) in both accuracy and efficiency. For example, on CH-SIMS:

Fusion Type	F1	Acc-5	MAE	MAdds
Concat	77.46	42.67	0.430	0
Concat+Self-Attn	79.52	44.08	0.424	324M
Cross-modal Attn	78.53	39.95	0.456	73M
Flat Bottleneck	78.56	42.89	0.433	162M
HBF	79.39	44.24	0.412	145M

Ablations further separate the effects of removing dual-stream alignment (degrades F1 from 79.39 to 76.37), omitting temporal or semantic information, or not applying supervised contrastive loss (Wen et al., 5 Dec 2025).

Related models for event-based recognition achieve state-of-the-art accuracy, e.g., 99.6% on ASL-DVS and competitive results on N-MNIST (Yuan et al., 2023).

7. Discussion, Limitations, and Future Directions

DashFusion achieves a balance between predictive power and efficiency via progressive bottlenecking, which maintains key sentiment cues and constrains attention costs. However, reliance on text as the central alignment reference limits generality in low-text scenarios. Handling missing or corrupted modalities, and extending alignment to anchor on non-linguistic modalities, remain open challenges.

Potential directions include broader use in multimodal emotion recognition, video-QA, and sample-adaptive bottleneck sizing. The event-based classification variant additionally highlights the versatility of dual-stream bottleneck fusion in domains beyond sentiment analysis, substantiating the broader applicability of these architectural principles (Wen et al., 5 Dec 2025, Yuan et al., 2023).