DSA-Tokenizer: Hierarchical Flow Matching

• DSA-Tokenizer is a hierarchical decoder that integrates semantic tokens via input adapters and acoustic tokens through cross-attention, extending traditional flow-matching methods.
• It employs multi-tier attention and structured constraint enforcement to fuse linguistic and acoustic information, ensuring controlled token generation.
• Empirical evaluations show improved metrics in speech synthesis and tokenization, with enhanced prosodic naturalness and reduced word error rates compared to conventional models.

A Hierarchical Flow-Matching Decoder is a generative modeling component that extends the flow-matching paradigm by introducing multi-level, structured conditioning, multi-tier attention, or hierarchical constraints into the forward ODE-based sampling process. These decoders have been developed to address challenges in tasks ranging from speech synthesis and tokenization to physically-constrained time-series generation and multi-modal data modeling. Their defining features include hierarchical data representations and/or constraint enforcement, specialized architectural fusion schemes, and loss functions tailored for distinct forms of alignment or domain compliance (Wang et al., 27 Dec 2025, Okita, 9 Oct 2025, Zhang et al., 14 Jan 2026, Zhang et al., 17 Jul 2025).

1. Core Principles and Motivation

Hierarchical Flow-Matching Decoders generalize the flow-matching approach, which defines a neural velocity field $v_t$ aligning synthetic and real data distributions via ODE integration along a linear path between noise and data. Hierarchical extensions are motivated by:

• Multi-granularity in data: For agglutinative languages, speech, or multimodal signals, different linguistic or representational levels (e.g., phoneme, syllable, prosody for speech; semantic vs. acoustic tokens; physical law levels) impact the output.
• Improved support for multi-modality: Standard flow matching may fail to reproduce multi-modal vector fields due to averaging effects; hierarchical decomposition and rectified heads mitigate this.
• Domain-specific structure: Incorporating physics-based, linguistic, or semantic/instance constraints is necessary for statistical consistency and interpretability.

These decoders systematically align multiple data or constraint tiers, either through architectural fusions (attention, multi-stream injection) or loss function design (multi-tier contrastive, physically-informed regularization), yielding generative models with improved fidelity, alignment, and controllability (Wang et al., 27 Dec 2025, Zhang et al., 14 Jan 2026, Okita, 9 Oct 2025, Zhang et al., 17 Jul 2025).

2. Architectural Patterns

Hierarchical Flow-Matching Decoders adopt several distinctive architectures, typically in the context of a non-autoregressive ODE-based backbone. The principal variants, as instantiated in recent research, are summarized below.

Study	Hierarchy Type	Backbone	Hierarchical Fusion/Constraint
ManchuTTS (Wang et al., 27 Dec 2025)	Phoneme/Syllable/Prosody (text)	DiT (8-layer Transformer)	Cross-modal hierarchical attention in coarse-to-fine order
DSA-Tokenizer (Zhang et al., 14 Jan 2026)	Semantic/Acoustic (token streams)	DiT (22 blocks)	Semantic via adapter at input; acoustic via cross-attention in every block
FNO-Guided CFM (Okita, 9 Oct 2025)	Physical law levels	UNet + FNO modules	Parallel FNO blocks implementing conservation, dynamics, boundary, empirical constraints
HRF (Zhang et al., 17 Jul 2025)	State/velocity/acceleration (dynamical)	UNet/DiT/MLP	Multi-level ODEs, rectified velocity field at each level with mini-batch couplings

In each case, hierarchical information is injected progressively—either by coarse-to-fine cross-modal attention (linguistic hierarchy), dual-stream fusion (semantic/acoustic), FNO-based constraint enforcement (physics), or stacked ODEs (state/velocity/joint derivatives) (Wang et al., 27 Dec 2025, Zhang et al., 14 Jan 2026, Okita, 9 Oct 2025, Zhang et al., 17 Jul 2025).

3. Mathematical Formalism

Hierarchical Flow-Matching builds upon the interpolation-based flow matching objective: for target sample $x_1$, noise $x_0 \sim \mathcal{N}(0, I)$, the interpolation path is $x_t = (1-t)x_0 + t x_1$, $t \in [0,1]$. The canonical loss is

$$\mathcal{L}_{\rm FM}(\theta) = \mathbb{E}_{t, x_0, x_1, c} \left[ \| v_t(x_t | c; \theta) - (x_1 - x_0) \|^2 \right]$$

where $v_t(x_t|c;\theta)$ is the neural velocity parameterized by a backbone model, conditioning $c$ encodes the hierarchical information (e.g., concatenated or summed embedding sequences aligned to $x_t$) (Wang et al., 27 Dec 2025, Zhang et al., 14 Jan 2026, Okita, 9 Oct 2025, Zhang et al., 17 Jul 2025).

Hierarchical elaborations include:

• Multi-tier contrastive losses: For each level $k$ (e.g., phoneme/syllable/prosody), a contrastive term $\ell_k$ is defined to encourage acoustic features to be most similar to the matching tier embedding and dissimilar to negatives, forming a total loss $\mathcal{L}_{\rm HC} = \sum_k \lambda_k \ell_k$ (Wang et al., 27 Dec 2025).
• Physics-informed constraints: Four constraint terms (conservation $\mathcal{C}_1$, dynamics $\mathcal{C}_2$, boundary $\mathcal{C}_3$, empirical $\mathcal{C}_4$), each regularized with time-dependent weights $\lambda_k(t)$, are accumulated as additional losses parallel to the main flow matching objective (Okita, 9 Oct 2025).
• Multi-level ODEs and rectification: For hierarchies of flows (state, velocity, etc.), each level $\ell$ integrates its own ODE, rectifying the predicted velocity by an additive modal correction $\Delta^{(\ell)}$ to better fit multi-modal vector fields (Zhang et al., 17 Jul 2025).

4. Training, Inference, and Optimization Regimes

Training and inference protocols adapt the core flow matching pipeline to support hierarchical fusion and constraint enforcement.

• Data preparation: Input signals are decomposed along hierarchical axes, e.g., multi-tier text units for speech, semantic/acoustic tokens, or physical parameter vectors.
• Conditioning alignment: Each tier's embedding is upsampled/repeated to the acoustic or integration grid length; alignment is achieved by simple replication, monotonic alignment, or CNN adapters depending on the domain (Wang et al., 27 Dec 2025, Zhang et al., 14 Jan 2026).
• Fusion mechanisms: Hierarchical attention or fusion is applied block-wise. For instance, ManchuTTS applies three consecutive cross-attention stages for phoneme, syllable, then prosody within each block (Wang et al., 27 Dec 2025). In DSA-Tokenizer, semantic tokens are injected via CNN adapters at input while acoustic tokens are injected in every transformer block via cross-attention (Zhang et al., 14 Jan 2026).
• Loss computation: Flow matching loss is computed per batch over interpolated paths. For hierarchical decoders, auxiliary losses (multi-tier contrastive, constraint residuals) are added per their respective formulations.
• Inference: ODE integration, often via Euler or Runge-Kutta steps, maps noise to output domain. For constraint-regularized models, FNO-guided corrections are extracted at every time step and subtracted from the neural velocity before integration (Okita, 9 Oct 2025).

Ablation studies confirm that hierarchical guidance, contrastive alignment, and physically-informed regularization yield significant improvements in task metrics (e.g., AWPA, prosodic scores, FID, and WER) (Wang et al., 27 Dec 2025, Zhang et al., 14 Jan 2026, Okita, 9 Oct 2025).

5. Applications and Empirical Performance

Hierarchical Flow-Matching Decoders have demonstrated effectiveness across speech, time-series, and image domains.

Speech Synthesis and Tokenization: In ManchuTTS, the hierarchical decoder achieves MOS 4.52 on a 5.2-hour subset, with 31% improvement in agglutinative word pronunciation and 27% gain in prosodic naturalness over baselines. DSA-Tokenizer attains UTMOS ~3.6 and WER ~6.7% in cross-utterance recombination tasks, far surpassing prior rigid or partially disentangled schemes (Wang et al., 27 Dec 2025, Zhang et al., 14 Jan 2026).

Physics-informed Generation: FNO-guided hierarchical decoders achieve 16.3% improvement in generation quality, 46% reduction in physics violations, and 18.5% gain in predictive accuracy for physical time-series. Constraint stratification prevents unphysical sample trajectories without sacrificing data likelihood (Okita, 9 Oct 2025).

Multi-modal Data Modeling: Hierarchical rectified flow matching enables low-NFE, high-fidelity sampling of multi-modal synthetic and visual distributions even in the presence of complex velocity structures (Zhang et al., 17 Jul 2025). Mini-batch OT couplings at each hierarchy level induce more tractable training objectives and improved sample diversity.

6. Comparative Analysis and Limitations

Comparisons with prior decoders highlight systematic advantages of hierarchical flow-matching strategies:

• Decoupled information fusion: Unlike single-stream or GAN-based dual-stream decoders, hierarchical approaches enable flexible, length-agnostic fusion, supporting inpainting and recombination without rigid alignment constraints (Zhang et al., 14 Jan 2026).
• Constraint compliance: Hierarchically regularized decoders capture macroscopic invariants and boundary structure, in contrast to baseline generative models that lack explicit physical awareness (Okita, 9 Oct 2025).
• Sample efficiency and inference latency: While deeper or more complex hierarchical decoders (e.g., DSA-Tokenizer’s 22-layer DiT) can incur increased inference time, early adoption of coupling methods (mini-batch OT) or hardware-efficient backbones mitigate runtime costs (Zhang et al., 14 Jan 2026, Zhang et al., 17 Jul 2025).

Identified limitations include higher model complexity, increased parameter count, and slower generation per sample when compared to GAN or shallow RVQ decoders. In scenarios prioritizing minimal latency, further refinement or hybridization with lightweight architectures is desirable.

7. Practical Implementation and Hyperparameters

Specific hyperparameter regimes are established for different use-cases:

• Model size: Ranges from ~86M parameters in ManchuTTS (8×512) to over 22×1024 in DSA-Tokenizer.
• Training regime: Optimizers like AdamW, learning rates in the $10^{-4}$–$10^{-5}$ range, 32–400k warmup steps, and batch sizes between $10^2$–$10^4$ depending on data dimensionality.
• Component-specific choices: Hierarchical attention, cross-attention, positionwise FFNs, and normalization schemes (LayerNorm, AdaNorm) are task-tuned.
• Efficiency: INT8 quantization yields 3× real-time synthesis even for large models on edge devices (e.g., Jetson Orin Nano); inference speed of 0.12 RTF and first-token latency of 86 ms are reported on RTX-4090 (Wang et al., 27 Dec 2025).
• Mini-batch coupling: Mini-batch OT sizes of 50–500 for data-level, 20–100 for velocity-level; coupling strength hyperparameter $\lambda$ in the range 1–10; number of ODE function evaluations (NFE) between 1–100 per hierarchy level (Zhang et al., 17 Jul 2025).

The implementations are reproducible based on the provided descriptions and equations; code for HRF (Hierarchical Rectified Flow Matching) is publicly available (Zhang et al., 17 Jul 2025).

---

The Hierarchical Flow-Matching Decoder framework integrates multi-tier representation, fusion, or constraint injection into ODE-driven generative processes, offering state-of-the-art performance across speech, physical, and multi-modal data domains through principled architectural and optimization innovations (Wang et al., 27 Dec 2025, Okita, 9 Oct 2025, Zhang et al., 14 Jan 2026, Zhang et al., 17 Jul 2025).