Dual-stream Unified Encoding and Transformation

• DUET is a multimodal fusion paradigm that employs dual-stream encoding to reconcile heterogeneous signals from different modalities.
• It integrates separate semantic and generative streams with multi-branch transformation, ensuring robust performance in both synthesis and analysis tasks.
• Empirical studies in Janus and MotionDuet demonstrate significant improvements in FID, retrieval, and overall cross-modal fidelity.

Dual-stream Unified Encoding and Transformation (DUET) is a multimodal fusion paradigm designed to reconcile heterogeneous signals from distinct modality pathways—such as vision and language, or video and text—through unified latent representations. DUET’s architectural principles underpin advanced frameworks in both image-centric multimodal modeling (Janus (Wu et al., 17 Oct 2024)) and motion-generation systems (MotionDuet (Zhang et al., 22 Nov 2025)), establishing DUET as a keystone strategy for achieving state-of-the-art cross-modal understanding and generative fidelity.

1. Conceptual Framework and Motivation

DUET formalizes a dual-pathway approach for encoding heterogeneous modalities, targeting two main conflicts in unified multimodal architectures: (1) the information granularity mismatch between semantic understanding and low-level generation, and (2) the unreliability and scale disparities present in direct modality concatenation or attention. In settings such as Janus, DUET decouples visual encoding into “understanding” and “generation” streams. In MotionDuet, DUET amalgamates semantic text prompts and spatio-temporal video cues, ensuring that generated outputs faithfully reflect both the high-level intent and fine-grained distribution of real-world signals.

The explicit separation of modality branches followed by a unified transformation resolves competing requirements—semantic abstraction versus physical detail—observed in prior single-stream models. This suggests that DUET’s design inherently supports robust performance on both discriminative (e.g., classification, Q&A) and generative (e.g., synthesis, prediction) tasks.

2. Formal Definitions and Architectural Summary

DUET operates by (1) independently encoding modality-specific signals, (2) projecting these into a common latent dimension, (3) fusing or transforming the signals via unified processing, and (4) delivering a composite representation suitable for downstream autoregressive or diffusion-based decoders.

In Janus (Wu et al., 17 Oct 2024):

• Visual Encoding: Partitioned into a semantic pathway (SigLIP) and a generative pathway (VQ tokenizer). SigLIP produces $U = [u_1,\ldots,u_{N_u}], u_j \in \mathbb{R}^{d_u}$, VQ yields discrete codebook tokens $z \in \{1,\ldots,K\}^{h_g \times w_g}$, embedded to $v_j \in \mathbb{R}^d$.
• Adapters: MLPs map each visual stream to the LLM embedding dimension $d$.
• Unified Input: Concatenation of text embeddings $e_i^t$, adapted SigLIP tokens $a_j^u$, and VQ embeddings $v_j$, with positional encodings.
• Processing: A single causal-mask Transformer operates over the fused sequence.

In MotionDuet (Zhang et al., 22 Nov 2025):

• Encoders: CLIP for text prompts ($T \in \mathbb{R}^{d_t}$), VideoMAE for video features ($V \in \mathbb{R}^{T \times d_v}$).
• Projection: Both streams mapped to shared latent ($R_o = W_o T + b_o$, $R_b = W_b V + b_b$, $R_{fusion} = R_o + R_b$).
• Multi-branch Transformation:
  • Residual identity, 1D FFT branch, convolutional branch, and Dynamic Mask Mechanism (DMM).
  • Four outputs concatenated and projected to $d_h$ via a two-layer feed-forward network with residual connections.
• Downstream Conditioning: Fused latent injected into the motion decoder for sample generation.

3. Multi-branch Fusion and Dynamic Selection

A hallmark of DUET is its multi-stream transformation after initial fusion. In MotionDuet, this includes:

• Residual Branch: Preserves element-wise fusion unchanged.
• FFT Branch: Models periodicity—vital for capturing global motion cycles—by applying temporal FFT, a learnable magnitude filter, and inverse FFT.
• Convolutional Branch: Refines local geometric and temporal details using Conv1D, LayerNorm, GELU.
• DMM: Implements a binary selection mask via L2 distance between modality-specific latent projections, allowing dynamic fallback to the most reliable signal at each timestep and channel.

The concatenation of these branches, followed by linear projection, enables DUET to support scenarios where certain modalities are absent or noisy, maintaining consistent output quality. This multi-path approach also facilitates disentanglement between modalities, further aiding interpretability and fine-grained control.

4. Unified Training Objectives and Coupling

DUET models employ end-to-end training, aligning all streams through unified objectives tailored to their context.

• Janus Objective: Autoregressive cross-entropy loss over the concatenated token sequence, treating image and text tokens equally, with no task-specific reweighting.
• MotionDuet Objective: Sum of multimodal denoising loss $\mathcal{L}_{MLD}$ and DASH loss $\mathcal{L}_{DASH}$, which aligns predicted motion trajectories with both token and pairwise statistics of the video-derived features. This multimodal coupling mitigates the distribution gap between video and text representations.

The training protocol in Janus comprises three explicit stages: adapter initialization (frozen encoders, rapid alignment), unified pretraining (full sequence mixing, encoders frozen), and instruction fine-tuning (dialogue and comprehension specialization). MotionDuet centralizes the fusion and transformation modules, with DASH loss coupling directly to DUET’s output to preserve distributional alignment.

5. Empirical Performance and Ablations

Both Janus and MotionDuet empirically validate DUET’s efficacy through comprehensive benchmarks and isolation studies.

Tabulated Results

Model/Module	FID (↓)	Retrieval (R@3, ↑)	POPE (↑)	VQA (↑)	GQA (↑)
Janus DUET	8.53	-	87.0	77.3	59.1
Show-o (1.3B)	9.24	-	73.8	59.3	48.7
DALL·E 2 (6.5B)	10.39	-	-	-	-
MotionDuet-only	0.192	0.742	-	-	-
MotionDuet DUET	0.101	0.755	-	-	-
+DASH	0.084	0.764	-	-	-

Adding DUET in MotionDuet lowers FID from 0.192 to 0.101 (+47% improvement), and R@3 increases from 0.742 to 0.755 (+1.3%). Further coupling with DASH yields FID 0.084 and R@3 0.764. In Janus, DUET leverages SigLIP+VQ decoupling to boost POPE and MMB scores by more than 20 points over single-stream VQ, while also delivering leading FID on MSCOCO-30K.

Ablation studies also demonstrate that sharing a single encoder (e.g., VQ only) degrades understanding scores but keeps FID competitive, highlighting the need for dual-stream separation, as implemented in DUET, to achieve joint optimality.

6. Modal Flexibility and Guidance Mechanisms

DUET’s framework inherently supports scenarios where modalities are non-uniform or missing. In MotionDuet, auto-guidance dynamically balances textual and visual signals at inference by combining outputs from strong and weakened DUET conditionings, parameterized by a guidance scale $\omega$.

Notably, DMM in DUET provides resilience to uneven modality reliability (noisy video or ill-defined text), systematically selecting the most informative branch at each timestep. This flexible design is particularly relevant in open-domain, multi-task, or zero-shot applications where input quality can vary.

7. Connections, Implications, and Prospective Extensions

DUET’s dual-pathway separation advances prior unified models, such as Chameleon, by directly addressing the granularity mismatch and information bottlenecks. Its cross-domain formalism—entrenched in both transformer-based (Janus (Wu et al., 17 Oct 2024)) and diffusion-based (MotionDuet (Zhang et al., 22 Nov 2025)) architectures—indicates broad applicability. A plausible implication is that future multimodal systems will increasingly adopt dual (or multi)-stream encoding and transformation paradigms, extending DUET-type fusion mechanisms to 3+ modalities and even continuous information flows.

In summary, Dual-stream Unified Encoding and Transformation establishes a reproducible, empirically validated blueprint for multimodal fusion, robust against signal heterogeneity, and foundational for next-generation generative and understanding models.