Unimodal Aggregation (UMA) in ASR

• UMA is a CTC-compatible mechanism that segments acoustic streams via learned unimodal weight patterns to form explicit token-like representations.
• It employs a scalar weight predictor with local extrema detection to aggregate frame-level hidden states and improve token alignment.
• Extensions such as streaming UMA for early termination and UMA-Split for multilingual support enhance token emission and reduce latency.

Searching arXiv for the core UMA papers and closely related extensions. arxiv_search(query="(Fang et al., 2023) OR (Fang et al., 2024) OR (Fang et al., 18 Sep 2025) unimodal aggregation speech recognition", max_results=10, sort_by="relevance") arxiv_search(query="Unimodal Aggregation for CTC-based Speech Recognition", max_results=5, sort_by="relevance") arxiv_search(query="Mamba for Streaming ASR Combined with Unimodal Aggregation", max_results=5, sort_by="relevance") arxiv_search(query="UMA-Split unimodal aggregation English Mandarin non-autoregressive speech recognition", max_results=5, sort_by="relevance") Unimodal Aggregation (UMA) is a CTC-compatible segment-and-integrate mechanism for automatic speech recognition in which an encoder produces framewise hidden representations together with scalar aggregation weights, and local minima of the weight sequence are used to segment the acoustic stream into token-like spans that are then pooled into shorter representations for downstream decoding (Fang et al., 2023). In the original formulation, UMA was proposed for non-autoregressive ASR to make token formation explicit rather than leaving it entirely to framewise CTC spikes; subsequent work adapted it to streaming ASR with a Mamba encoder and convolutional lookahead, and later extended it with a split module so that one UMA segment can support up to two tokens, which was introduced to make the method work in both Mandarin and English (Fang et al., 2024, Fang et al., 18 Sep 2025).

1. Conceptual basis and development

The original UMA formulation addresses a standard CTC difficulty: ASR targets are tokens, but the model predicts on acoustic frames, so the system must implicitly align, aggregate, and spike token evidence over a much longer frame sequence (Fang et al., 2023). UMA inserts an explicit intermediate stage between encoder and decoder. The encoder outputs framewise hidden states $h_t$, a scalar weight predictor computes $\alpha_t=\text{Sigmoid}(\text{Linear}(h_t))$, and local minima of $\alpha_t$ are treated as boundaries between token-like segments. The segment between adjacent valleys is then integrated into one vector before CTC is applied on the shortened sequence (Fang et al., 2023).

The defining assumption is that frames belonging to one text token should have a unimodal weight pattern: the weights should first increase and then decrease over that token span. Under that assumption, valleys mark token boundaries and peaks mark the strongest internal token activity (Fang et al., 2023). The original paper explicitly notes that the method is currently suitable for monosyllabic languages with clear acoustic boundaries, such as Mandarin Chinese (Fang et al., 2023).

Later work extended the same principle in two distinct directions. In streaming ASR, UMA was placed after a Mamba encoder and a convolutional lookahead layer so that valley detection could serve as an explicit online trigger for token emission (Fang et al., 2024). In multilingual non-autoregressive ASR, UMA-Split retained the original unimodal aggregation stage but added a split module so that each aggregated frame can map to multiple tokens, which was introduced to handle English BPE tokenization and very short token spans (Fang et al., 18 Sep 2025).

Paper	Setting	Main UMA contribution
(Fang et al., 2023)	CTC-based non-autoregressive ASR	Original segmentation-and-aggregation formulation
(Fang et al., 2024)	Streaming ASR with Mamba	Streaming-style UMA and Early Termination
(Fang et al., 18 Sep 2025)	English and Mandarin NAR ASR	Split module enabling one UMA unit to emit up to two tokens

2. Core mechanism

In the original formulation, the encoder transforms the acoustic input $\mathbf{x}$ into a downsampled hidden sequence $\mathbf{h}=(h_1,\dots,h_{T'})$, with downsampling factor $4$ (Fang et al., 2023). A scalar weight is then predicted at each encoder position: $\alpha_t=\text{Sigmoid}(\text{Linear}(h_t)).$ A time index $t$ is defined as a valley when

$$\alpha_t \le \alpha_{t-1}\quad \text{and}\quad \alpha_t \le \alpha_{t+1}.$$

The first and last positions are forced to be valleys, and the valley indices are denoted by $\tau_i$ (Fang et al., 2023).

The aggregation step integrates the hidden states between adjacent valleys by weighted averaging: $\alpha_t=\text{Sigmoid}(\text{Linear}(h_t))$0 This produces a shortened sequence $\alpha_t=\text{Sigmoid}(\text{Linear}(h_t))$1 whose length $\alpha_t=\text{Sigmoid}(\text{Linear}(h_t))$2 is near token-level but is not constrained to equal the target length $\alpha_t=\text{Sigmoid}(\text{Linear}(h_t))$3 (Fang et al., 2023). The paper emphasizes a specific overlap rule: segment $\alpha_t=\text{Sigmoid}(\text{Linear}(h_t))$4 uses $\alpha_t=\text{Sigmoid}(\text{Linear}(h_t))$5, so adjacent segments share two overlap frames. Those overlap frames are described as “very important” because they allow neighboring token units to send gradients into the boundary region and let valley positions move during training (Fang et al., 2023).

A common misunderstanding is to treat UMA as a threshold-based firing process akin to CIF. That is not how the method is defined in these papers. In the streaming-style formulation, the boundary rule remains local: the model predicts scalar weights $\alpha_t=\text{Sigmoid}(\text{Linear}(h_t))$6, detects valleys and peaks by comparing neighboring weights, and then aggregates the completed segment between two valleys (Fang et al., 2024). The streaming paper explicitly distinguishes this from cumulative thresholds, monotonic halting probabilities, forced boundary labels, and separate supervised boundary classifiers (Fang et al., 2024).

3. Architectural integration and training

The original UMA architecture is encoder–UMA–decoder–CTC. The encoder may be a Transformer, Conformer, or E-Branchformer; the decoder is a Transformer-encoder-like non-autoregressive self-attention network; and the training objective remains CTC on the shortened sequence (Fang et al., 2023). Because the aggregated sequence is much shorter than the frame-level sequence, the paper reports that after UMA the sequence length is about one-fifth of the frame-level length, which reduces decoder cost and contributes to lower real-time factor (Fang et al., 2023).

UMA does not require boundary annotation, forced alignment, or an auxiliary unimodality loss. The original paper states that “there is no explicit constraint being put on the aggregation weights $\alpha_t=\text{Sigmoid}(\text{Linear}(h_t))$7, and the CTC loss automatically learns to assign unimodal aggregation weights to the feature frames of each token” (Fang et al., 2023). This makes UMA an end-to-end CTC-trained module rather than a separately supervised segmentation stage.

The same paper also integrates self-conditioned CTC into the UMA framework. The reported implementation inserts self-conditioned intermediate layers at encoder layers $\alpha_t=\text{Sigmoid}(\text{Linear}(h_t))$8, $\alpha_t=\text{Sigmoid}(\text{Linear}(h_t))$9, and $\alpha_t$0, and decoder layers $\alpha_t$1 and $\alpha_t$2, with final-layer loss weight $\alpha_t$3 and each intermediate-layer loss weight $\alpha_t$4 (Fang et al., 2023). The stated motivation is complementary to UMA itself: UMA addresses representation granularity and segmentation, while self-conditioning addresses token dependency and conditioning (Fang et al., 2023).

In the streaming Mamba system, the architecture is explicitly described as: input acoustic features, two causal $\alpha_t$5D convolution layers with stride $\alpha_t$6, Mamba encoder, convolutional lookahead layer, UMA, $\alpha_t$7-layer causal self-attention decoder, and CTC training objective (Fang et al., 2024). There, UMA sits after the Mamba encoder and lookahead layer and before the decoder, and the whole model is trained end-to-end with CTC loss (Fang et al., 2024).

4. Streaming-style UMA and Early Termination

In streaming ASR, UMA is used not only for frame-to-token aggregation but also for output triggering. After the Mamba encoder and optional lookahead, UMA receives a frame-level speech embedding sequence $\alpha_t$8, predicts scalar weights $\alpha_t$9 with a Linear-Sigmoid network, and detects both valleys and peaks: $\mathbf{x}$0 for valleys, and

$\mathbf{x}$1

for peaks (Fang et al., 2024).

The streaming trigger rule is valley-based. When the current time reaches a UMA valley, that valley is interpreted as the endpoint of one text token; frames from the previous valley to the current valley are aggregated into a token embedding; one decoder step is run; and one text token is emitted (Fang et al., 2024). The aggregation formula in this streaming-style version is

$\mathbf{x}$2

The paper frames this as an explicit solution to “endpoint detection for triggering token output” in streaming ASR (Fang et al., 2024).

On top of streaming UMA, the paper proposes Early Termination (ET), an inference-only strategy that adds an earlier emission attempt at the UMA peak (Fang et al., 2024). The stated logic has three possible outcomes at a peak: if the decoder outputs the correct token, latency decreases; if it outputs blank, accuracy and latency remain effectively unchanged; if it outputs a wrong token, recognition errors increase (Fang et al., 2024). The reported conclusion is therefore not that ET is uniformly better, but that it usually improves the CER–latency trade-off by leveraging the peak–valley structure already learned by UMA (Fang et al., 2024).

5. Empirical behavior and language dependence

On Mandarin benchmarks, the original UMA paper reports that UMA improves over regular CTC and shortens the sequence length while remaining non-autoregressive (Fang et al., 2023). On AISHELL-1, the reported test CER is $\mathbf{x}$3 for UMA and $\mathbf{x}$4 for UMA plus self-condition, compared with $\mathbf{x}$5 for regular CTC and $\mathbf{x}$6 for self-conditioned CTC; the reported real-time factor is $\mathbf{x}$7 for UMA, compared with $\mathbf{x}$8 for CTC and $\mathbf{x}$9 for self-conditioned CTC (Fang et al., 2023). On AISHELL-2, UMA reports $\mathbf{h}=(h_1,\dots,h_{T'})$0 CER across android, ios, and mic, with UMA plus self-condition giving $\mathbf{h}=(h_1,\dots,h_{T'})$1 (Fang et al., 2023). On HKUST, plain UMA is encoder-dependent: with a Transformer encoder, UMA is $\mathbf{h}=(h_1,\dots,h_{T'})$2 versus $\mathbf{h}=(h_1,\dots,h_{T'})$3 for CTC, but with Conformer and E-Branchformer it improves over both CTC and self-conditioned CTC, and UMA plus self-condition is best in all three encoder families reported (Fang et al., 2023).

The streaming paper reports consistent gains from adding UMA to several streaming encoders. On AISHELL-1, Causal Transformer CTC is $\mathbf{h}=(h_1,\dots,h_{T'})$4 CER and Causal Transformer plus UMA is $\mathbf{h}=(h_1,\dots,h_{T'})$5; Chunk Conformer CTC is $\mathbf{h}=(h_1,\dots,h_{T'})$6 and Chunk Conformer plus UMA is $\mathbf{h}=(h_1,\dots,h_{T'})$7; Mamba CTC is $\mathbf{h}=(h_1,\dots,h_{T'})$8, while Mamba plus UMA without lookahead is $\mathbf{h}=(h_1,\dots,h_{T'})$9 and with $4$0 ms lookahead is $4$1 (Fang et al., 2024). The same paper reports that after UMA the sequence length is reduced to about one-fifth of the frame-level length, and that ET further reduces average token latency, for example from $4$2 ms to $4$3 ms on AISHELL-1 with Mamba UMA and $4$4 lookahead, with CER changing from $4$5 to $4$6 (Fang et al., 2024).

The later English–Mandarin extension identifies a limitation of the original method: Mandarin characters correspond to complete long syllables, but in English a single syllable may be tokenized into multiple fine-grained tokens, and some tokens span fewer than $4$7 acoustic frames, which can fail to form unimodal weights (Fang et al., 18 Sep 2025). UMA-Split addresses this by adding a split module after a low-rate encoder. If $4$8 is the low-rate representation, the split outputs are

$4$9

This doubles the CTC time axis from $\alpha_t=\text{Sigmoid}(\text{Linear}(h_t)).$0 to $\alpha_t=\text{Sigmoid}(\text{Linear}(h_t)).$1 and allows one UMA-aggregated frame to emit $\alpha_t=\text{Sigmoid}(\text{Linear}(h_t)).$2, $\alpha_t=\text{Sigmoid}(\text{Linear}(h_t)).$3, or $\alpha_t=\text{Sigmoid}(\text{Linear}(h_t)).$4 non-blank tokens (Fang et al., 18 Sep 2025).

The empirical evidence reported for this extension is explicitly language-sensitive. On AISHELL-1, UMA-Split reports test CER $\alpha_t=\text{Sigmoid}(\text{Linear}(h_t)).$5, while the same model without the split module reports $\alpha_t=\text{Sigmoid}(\text{Linear}(h_t)).$6; the reported $\alpha_t=\text{Sigmoid}(\text{Linear}(h_t)).$7-non-blank split ratio is $\alpha_t=\text{Sigmoid}(\text{Linear}(h_t)).$8, which is consistent with the claim that Mandarin rarely needs two-token emissions per UMA segment (Fang et al., 18 Sep 2025). On LibriSpeech with BPE $\alpha_t=\text{Sigmoid}(\text{Linear}(h_t)).$9, the reported token rate is $t$0 tps, the post-UMA frame rate is $t$1 fps, the non-blank split ratio is $t$2, and the $t$3-non-blank split ratio is $t$4; the resulting WER is $t$5 for the base model and $t$6 for the large model (Fang et al., 18 Sep 2025). The paper further reports that character-token experiments on LibriSpeech failed because the token rate was $t$7 tokens/sec, which was described as too high to form effective UMA weights (Fang et al., 18 Sep 2025).

6. Relation to other methods, limitations, and terminological scope

Within ASR, UMA differs from regular CTC by explicitly constructing token-like representations before final prediction, rather than relying entirely on framewise hidden states and spike formation (Fang et al., 2023). It also differs from CIF-style methods in that the streaming formulation does not use a cumulative threshold or a monotonic halting probability; instead, segmentation is driven by local extrema of the learned weight curve (Fang et al., 2024). Relative to self-conditioned CTC, the original paper presents the two methods as complementary rather than interchangeable (Fang et al., 2023).

The main limitations reported in the ASR papers are structural rather than purely implementation-specific. The original formulation is stated to be currently suitable mainly for monosyllabic languages with clear acoustic boundaries, such as Mandarin Chinese (Fang et al., 2023). The same paper notes that there is no hard guarantee of one segment per token, that one token may still be split or multiple tokens may merge, that segmentation mistakes can increase deletion errors, and that there is no explicit unimodality constraint beyond the pressure induced by CTC (Fang et al., 2023). The streaming paper adds that ET can introduce errors when a premature peak-based emission is wrong (Fang et al., 2024). The English extension shows that token granularity is a critical variable: coarser BPE vocabularies reduce the need for two-token split emissions and improve WER, whereas very high token rates can make UMA ineffective (Fang et al., 18 Sep 2025).

A further point of clarification concerns terminology. In speech-recognition research, UMA denotes unimodal aggregation (Fang et al., 2023). The same acronym is used differently elsewhere on in federated edge learning, “UMA” denotes unsourced massive access (Qiao et al., 2023), and in atomistic machine learning, it denotes Universal Models for Atoms (Wood et al., 30 Jun 2025). In the ASR literature summarized here, however, UMA refers specifically to the learned unimodal weight mechanism that segments acoustic frames by valleys and aggregates them into token-level representations before CTC decoding (Fang et al., 2023).