Speech-RATE: Modeling and Control in Speech

• Speech-RATE is defined as the average number of phonemic units produced per unit time, measured using various methods (phonemes, syllables, tokens, or frames).
• Controllable TTS systems leverage sentence-level conditioning, attention-based duration prediction, and total-duration-aware models to adjust speech rate for natural and intelligible output.
• Disentanglement techniques mitigate unintended alterations in prosody, ensuring that modifications in speech rate do not adversely affect pitch, speaker similarity, or other stylometric attributes.

Searching arXiv for recent and foundational papers on speech rate, speaking-rate control, and related assessment tasks. Speech-RATE denotes the average number of phonemic units produced per unit time. In one formalization, for an utterance of duration $T$ seconds containing $N$ phonemes, the speaking rate is $r = \frac{N}{T}$ in phonemes/sec; in practice, one study measured speaking rate in “syllables/sec” via Praat’s Syllable Nuclei script, which closely tracks phoneme counts (Yeh et al., 2018). Across contemporary speech technology, the notion appears in several closely related forms: utterance-level or sentence-level rate for speech synthesis control, articulation rate excluding pauses for sociophonetic analysis, and fluency-related temporal predictors for clinical assessment. The literature further distinguishes global tempo from rhythmic patterns, defined as duration distributions for realizations of different phonemes within different phonetic structures (Yeh et al., 2018).

1. Definitions, measurement, and related temporal quantities

Several papers define speech rate as a corpus- or utterance-level scalar derived from linguistic unit counts and duration. In rhythm-flexible voice conversion, speaking rate is the average number of phonemic units produced per unit time, with $r = \frac{N}{T}$ for an utterance of duration $T$ seconds containing $N$ phonemes (Yeh et al., 2018). In speech recognition, rate of speech (ROS) is the average number of phonetic units per second in an utterance; from forced alignment one computes $$\mathrm{ROS} = \frac{\text{number of phones in the utterance}}{\text{utterance duration in seconds}}$$, and in practice forms the scalar feature $r = \frac{M}{T}$ in phones/sec (Zeng et al., 2015). In non-autoregressive speed-control TTS, the ground-truth speaking rate in frames per token is computed as $SR = \frac{\mathrm{length}(y)}{T}$, where $y$ is the raw waveform of an utterance and $N$0 is the number of input tokens (phonemes) (Bandekar et al., 2023). In sentence-level controllable TTS, the sentence-level speaking-rate scalar is defined as $N$1, where $N$2 is the number of input phonemes, $N$3 is the length of the corresponding acoustic sequence in frames after removal of leading/trailing silence, and $N$4 is a scaling constant set to 100 in experiments (Bae et al., 2020).

These definitions differ mainly in the temporal denominator and the unit counted. Some formulations use seconds, some use acoustic frames, and some use text-token counts rather than directly aligned phones. This suggests that “Speech-RATE” is operational rather than uniquely standardized: it is adapted to the control or analysis interface of the model under study.

The literature also distinguishes speech rate from articulation rate and from rhythm. In a broad literature review over spontaneous English speech, “speech rate” refers to the total number of linguistic units produced per second of recorded time, including both phonation and silent pauses, whereas articulation rate is the number of syllables produced per second of signal-active time, explicitly excluding pauses longer than 150 ms (Tanner et al., 2024). In the voice-conversion literature, rhythmic patterns are characterized by the statistical distribution of durations of individual phonemes or phoneme classes within an utterance, and target-speaker rhythm is assessed by how closely transformed posteriorgrams approach the target speaker’s phoneme-duration histograms (Yeh et al., 2018). Clinical fluency work uses still other temporal proxies: pseudo-syllable rate, speech ratio, rate of silent breaks, and standard deviation of pseudo-syllable length (Fontan et al., 2023).

2. Sentence-level and duration-based control in text-to-speech

A major research line treats Speech-RATE as a controllable variable in TTS. One approach, speed-controllable TTS (SCTTS), conditions an end-to-end synthesis model on a single sentence-level scalar rather than on per-phoneme durations. The speaking-rate value is the ratio of the number of input phonemes to the length of input speech, and it is injected into the text encoder through a learnable embedding concatenated to each phoneme embedding; all subsequent modules remain unchanged (Bae et al., 2020). During preprocessing, phoneme sequence length $N$5 is known from text, coarse-mel spectrogram length $N$6 is measured after VAD-based trimming of silence, and no further normalization is applied. For inference, the “normal” rate is set to the average $N$7 over the training set, and fast or slow speech is generated by multiplying that average by a factor greater than or less than 1, or by directly setting $N$8 (Bae et al., 2020). The training loss is $N$9, where $r = \frac{N}{T}$0 is $r = \frac{N}{T}$1 loss plus binary divergence and $r = \frac{N}{T}$2 decays linearly from 1 to 0 over training; no explicit auxiliary loss on speaking rate is used (Bae et al., 2020).

A second line conditions duration prediction itself on speaking rate. In speaking-rate attention-based duration prediction for speed control TTS, a FastSpeech-style duration predictor is augmented with an SR feature derived from $r = \frac{N}{T}$3 through a 256-dimensional dense layer, and a scaled dot-product attention mechanism injects that SR representation into the duration-feature sequence (Bandekar et al., 2023). Two insertion schemes are reported: SRA-b applies the attention before the two convolution layers of the duration predictor, and SRA-e applies it after those layers and just before the final dense layer (Bandekar et al., 2023). The baseline rescales predicted token durations linearly as $r = \frac{N}{T}$4, whereas the SR-attention block enables token-dependent, non-linear duration changes (Bandekar et al., 2023).

A third formulation makes total duration an explicit conditioning target. Total-duration-aware (TDA) duration models predict phoneme durations from text input and an additional input of the total target duration. The predictor is aware of $r = \frac{N}{T}$5 through a per-phoneme scalar sequence $r = \frac{N}{T}$6, where $r = \frac{N}{T}$7 indicates whether a phoneme duration is to be predicted or given (Eskimez et al., 2024). At inference, a length-regulator step rescales predicted masked durations by $r = \frac{N}{T}$8 so that their sum matches the target exactly (Eskimez et al., 2024). The regression version is trained with MSE in the log-duration domain over masked positions, and an end-to-end variant adds a regularization term encouraging the sum to match $r = \frac{N}{T}$9 (Eskimez et al., 2024).

These three families embody distinct control granularities. Sentence-level conditioning supplies one scalar per utterance, SR-aware duration prediction conditions the duration model on a global rate feature, and TDA models constrain durations by a total-duration target. A plausible implication is that current TTS systems increasingly treat Speech-RATE not as a post hoc waveform transformation, but as a first-class conditioning signal inside the generative duration pathway.

3. Disentanglement, naturalness, and prosodic side effects

A recurring issue is that changing speaking rate can unintentionally alter other attributes such as pitch, timbre, intelligibility, or speaker similarity. SCTTS addresses this with a global style token-based style encoder. In SCTTS-GST, a reference encoder and multi-head attention over style tokens produce a style embedding that is concatenated with the text-plus-SR encoding and projected before attention, with the stated purpose of ensuring that other style attributes, such as pitch, remain unaffected when rate changes (Bae et al., 2020). The paper reports that expressive datasets often exhibit correlations between SR and other prosodic attributes, such as $r = \frac{N}{T}$0, and that the GST encoder disentangles these attributes by capturing pitch, energy, and other style factors except speaking rate, which is externally provided by $r = \frac{N}{T}$1 (Bae et al., 2020).

The SR-attention duration-prediction work reports a similar phenomenon from the duration-modeling side. Mean pitch remains far more stable across speaking-rate factors for SRA models than for the FastSpeech baseline, indicating better disentanglement of pitch and rate (Bandekar et al., 2023). It also reports that plotting individual phoneme durations against the rate factor shows strictly linear scaling for the baseline and non-linear, token-dependent curves for SRA-e, which reflect human-like variation (Bandekar et al., 2023).

TDA duration models broaden the problem formulation beyond duration matching. The 2024 work explicitly frames total-duration control as crucial for applications, while noting that the impact of adjusting the speech rate on speech quality, such as intelligibility and speaker characteristics, had been underexplored. Its empirical results show that proposed TDA duration models achieve better intelligibility and speaker similarity for various speech rate configurations than baseline models (Eskimez et al., 2024). Under a $r = \frac{N}{T}$2 rate configuration, baseline regression+LR shows WER rising from 2.7 to 15.1 and SIM dropping sharply, whereas TDA regression+LR improves WER to 7.4 and restores SIM from approximately 0.50 to approximately 0.67 (Eskimez et al., 2024).

The evidence across these studies supports a narrow technical conclusion: naive global duration scaling tends to conflate Speech-RATE with other prosodic or speaker-related properties, while explicit conditioning and disentangling mechanisms reduce that coupling. This does not imply complete independence of prosodic factors, but it does suggest that rate control is most effective when the model has a separate representational pathway for non-rate style information.

4. Comparative performance of rate-control methods

The comparative TTS literature consistently contrasts sentence-level or condition-aware methods with uniform duration scaling. In SCTTS, listening tests on fast-, normal-, and slow-speed speech show that the model generates more natural speech than phoneme duration control approaches that increase or decrease duration at the same rate for the entire sentence, especially in the case of slow-speed speech (Bae et al., 2020). At slow speed on a 5-point MOS scale, PDC-TTS and FastSpeech are approximately $r = \frac{N}{T}$3, SCTTS is approximately $r = \frac{N}{T}$4, and SCTTS-GST is approximately $r = \frac{N}{T}$5; at fast speed, all models perform comparably at approximately 4.0 (Bae et al., 2020). Objective mel-spectrograms show that SCTTS applies variable stretching, lengthening vowels and pauses more, whereas phoneme-level methods uniformly stretch all phones, causing choppy or unnatural artifacts especially at slow rates (Bae et al., 2020).

The SR-attention duration-prediction study reports objective SR error and subjective MOS gains over a FastSpeech baseline. Baseline FastSpeech has SR error of approximately 0.8–1.2 frames/token at extreme factors, whereas RS-SRA-e and RS-SRA-b reduce error by approximately 15–30% across $r = \frac{N}{T}$6 (Bandekar et al., 2023). In naturalness MOS, the baseline scores 3.98 $r = \frac{N}{T}$7, RS-SRA-e 4.13 $r = \frac{N}{T}$8, RS-SRA-b 4.18 $r = \frac{N}{T}$9, and SRA-FT2 4.35 $T$0 (Bandekar et al., 2023). The fine-tuning result is notable because SRA-FT2 freezes only the encoder and still matches or exceeds from-scratch performance after 100 epochs with batch size 24 and Adam at learning rate $T$1 (Bandekar et al., 2023).

The TDA paper extends the comparison to regression, flow-matching, and MaskGIT-based duration modeling.

Model	Cost	FDD at 1× rate
regression+LR	1×	$T$2
FM+LR	32×	$T$3
MaskGIT+LR	32×	$T$4

MaskGIT+LR achieves the best FDD at 1× rate, while TDA conditioning is reported to recover strong rate control and preserve intelligibility and speaker similarity at faster rates (Eskimez et al., 2024). Subjectively at $T$5, 58.7% of samples are preferred for intelligibility and 53.9% for speaker similarity when comparing TDA regression+LR against regression+LR (Eskimez et al., 2024).

At the vocoder level, speaking-rate-controllable HiFi-GAN uses a differentiable interpolation layer to warp either mel-spectrograms or hidden features by a factor $T$6, with $T$7 (Xin et al., 2022). No additional training is performed; a pretrained universal HiFi-GAN is frozen and the interpolation layer is activated only at inference (Xin et al., 2022). Objective MCD results show mel-spectrogram interpolation outperforming WSOLA in all reported conversion directions, and the paper states that warping mel-spectrograms by image scaling obtained the best performance among all proposed methods (Xin et al., 2022). Real-Time Factor on RTX2080Ti is approximately 0.01 for HiFi-GAN alone, approximately 0.05 for WSOLA + HiFi-GAN, and 0.01–0.02 for linear-mel interpolation + HiFi-GAN, with negligible inference-time overhead and unchanged model parameters and GPU memory (Xin et al., 2022).

5. Speech-RATE beyond TTS: recognition, fluency, and quality assessment

Speech-RATE is also used as side information for recognition and as a measurement target in assessment tasks. In DNN-based ASR, adding the ROS scalar to frame-stacked filterbank features yields a 441-dimensional input before splicing or 451 dimensions if injected before splicing, followed by LDA projection to 200 dimensions and a 4-hidden-layer network with 1,200 sigmoid units per layer and a 3,656-unit output layer (Zeng et al., 2015). The DNN is trained with frame-level cross-entropy, and the ROS-aware system improves WER for slow and fast speech compared with a DNN without ROS. On a spontaneous Mandarin test set divided into Slow, Normal, and Fast, the DNN baseline gives WER 45.71, 28.04, 31.22, and total 30.03, while DNN + ROS-feature gives 44.92, 28.05, 29.54, and total 29.53 (Zeng et al., 2015). Combined with HMM transition adaptation, the system achieves 44.76 for slow and 29.08 for fast speech (Zeng et al., 2015). The interpretation is that ROS variation affects both temporal and spectral or static properties of speech, and that ROS injection makes the spectral model ROS-aware (Zeng et al., 2015).

In clinical fluency assessment, the problem is not generation but automatic measurement. An entirely acoustic pipeline segments read speech using the forward–backward divergence algorithm, clusters segments into pseudo-syllables and silent breaks, and computes four predictors: pseudo-syllable rate $T$8, standard deviation of pseudo-syllable duration $T$9, speech ratio $N$0, and rate of silent breaks $N$1 (Fontan et al., 2023). A multiple linear regression model using $N$2, $N$3, and $N$4 after stepwise selection yields $N$5; adding a repetition-sensitive predictor $N$6 produces $N$7 (Fontan et al., 2023). Under leave-one-speaker-out validation over 34 participants, sentence-level MLR reaches RMSE $N$8 and $N$9, participant-level MLR reaches RMSE $$\mathrm{ROS} = \frac{\text{number of phones in the utterance}}{\text{utterance duration in seconds}}$$0 and $$\mathrm{ROS} = \frac{\text{number of phones in the utterance}}{\text{utterance duration in seconds}}$$1, and with $$\mathrm{ROS} = \frac{\text{number of phones in the utterance}}{\text{utterance duration in seconds}}$$2 the sentence-level MLR improves to RMSE $$\mathrm{ROS} = \frac{\text{number of phones in the utterance}}{\text{utterance duration in seconds}}$$3, $$\mathrm{ROS} = \frac{\text{number of phones in the utterance}}{\text{utterance duration in seconds}}$$4, and participant-level to RMSE $$\mathrm{ROS} = \frac{\text{number of phones in the utterance}}{\text{utterance duration in seconds}}$$5, $$\mathrm{ROS} = \frac{\text{number of phones in the utterance}}{\text{utterance duration in seconds}}$$6 (Fontan et al., 2023). Here Speech-RATE functions as one constituent of a broader fluency construct rather than a standalone endpoint.

A distinct but related use of rate information appears in speech quality assessment. HighRateMOS is a non-intrusive MOS model that explicitly considers sampling rate rather than speaking rate, supporting 16 kHz, 24 kHz, and 48 kHz via a learnable sampling-rate embedding table (Ren et al., 27 Jun 2025). The system ensembles three model variants that use wav2vec 2.0 embeddings, multi-scale CNN spectral features, and optionally MFCCs, with the final score $$\mathrm{ROS} = \frac{\text{number of phones in the utterance}}{\text{utterance duration in seconds}}$$7 (Ren et al., 27 Jun 2025). Compared to a single-rate baseline, system-level SRCC improves by +27.5% and KTAU by +53.9%, to absolute SRCC 0.955 and KTAU 0.842 (Ren et al., 27 Jun 2025). Although this work concerns sampling-rate awareness rather than Speech-RATE in the temporal-prosodic sense, it illustrates the broader methodological pattern of conditioning predictive models on rate-related metadata to reduce distribution shift.

6. Rhythm, articulation, and target-specific temporal structure

Speech-RATE is not only a scalar control variable; it is also embedded in finer temporal organization. In rhythm-flexible voice conversion without parallel data, a sequence-to-sequence Cycle-GAN over phoneme posteriorgram sequences removes the length constraint of fixed-rate methods. The decoder is autoregressive and attention-based, allowing variable-length outputs and hence rhythm flexibility (Yeh et al., 2018). Objective evaluation shows that the converted speaking rate shifts close to the target’s mean ± standard deviation, whereas a baseline phonetic-posteriorgram method remains locked at the source rate, and phoneme-duration histograms for selected phonemes match the target speaker’s rhythmic patterns better than the fixed-rate baseline (Yeh et al., 2018). This suggests that a speaker’s temporal style is not exhausted by average rate alone.

An articulatory analogue appears in AstNet, an encoder-decoder model with location-sensitive attention for transforming articulatory movements between neutral, fast, and slow speaking rates (Singh et al., 2020). Attention replaces DTW alignment and models duration variations across speaking rates, while the decoder uses a stop token to determine output duration (Singh et al., 2020). For phoneme-specific analysis, only 7 of 50 phones differ significantly between predicted and original target durations in neutral-to-fast conversion, and only 1 of 50 does so in neutral-to-slow conversion, according to paired $$\mathrm{ROS} = \frac{\text{number of phones in the utterance}}{\text{utterance duration in seconds}}$$8-tests at $$\mathrm{ROS} = \frac{\text{number of phones in the utterance}}{\text{utterance duration in seconds}}$$9 (Singh et al., 2020). The model also captures amplitude differences quantified by SDAT, reflecting reduced range in fast speech and increased range in slow speech (Singh et al., 2020). Relative improvement of AstNet fine-tuned over a prior baseline ranges from approximately 2–4.3% for neutral-to-fast and 3–10% for neutral-to-slow on DTW distance (Singh et al., 2020).

A 2026 intelligibility study sharpens the distinction between global slowing and targeted temporal manipulation. Reverse-correlation experiments with minimal-pair contrasts reveal a “scissor” shape in temporal weighting kernels: from about 800 ms to 300 ms before the target vowel, slower context biases perception toward the phonetically shorter or faster vowel, while from about 100 ms to 0 ms before the target, slower speech biases perception toward the phonetically longer or slower vowel (Tuttösí et al., 31 Mar 2026). The resulting kernel is reported to be remarkably stable within individuals and across L1-English and L2-English listeners with French, Mandarin, and Japanese L1s (Tuttösí et al., 31 Mar 2026). A data-driven MatchaTTS-based algorithm then applies segment-level rate multipliers around target words. On single-target sentences, baseline WER is 38.4% and the proposed targeted model achieves 29.5%; on double-target sentences, baseline WER is 24.3% and the proposed model achieves 15.2% (Tuttösí et al., 31 Mar 2026). Yet MOS judgments favor global slowdown, despite its lower objective intelligibility (Tuttösí et al., 31 Mar 2026). This directly challenges the common assumption that making all speech slower is necessarily the best accessibility strategy.

7. Distributional structure and broader implications

Speech-RATE is shaped by the distribution of training or observational data. In SR-attention TTS, four ways of collecting 45 h of multi-speaker data are compared: random selection, tail-first selection, and two narrow subsets defined by 1.5 or 1 standard deviation around the mean (Bandekar et al., 2023). Random-selection SRA yields the lowest SR error across most factors greater than 0.8, tail-first selection performs poorly at slow rates, and single-speaker models trained on LJSpeech have significantly higher SR errors than multi-speaker random-selection models (Bandekar et al., 2023). The authors interpret this as evidence that wider SR training support aids controllability (Bandekar et al., 2023).

In spontaneous English speech, articulation rate is modeled across 13 corpora with a Bayesian distributional multilevel model. Utterance length has the largest effect: articulation rate increases sharply from very short utterances of 3–5 syllables to medium lengths and then asymptotes for very long utterances, with posterior median predictions rising by approximately 2.1 to 2.4 syllables/sec from the shortest to the longest observed utterances (Tanner et al., 2024). Age and gender also modulate articulation rate, but with much smaller magnitudes: $r = \frac{M}{T}$0 with 95% credible interval $r = \frac{M}{T}$1, and $r = \frac{M}{T}$2 with 95% credible interval $r = \frac{M}{T}$3 (Tanner et al., 2024). The paper states that female speakers are about 0.21–0.25 syll/sec slower than male speakers holding utterance length and age constant, while age corresponds to about 0.7–0.8 syll/sec decrease from youngest to oldest speakers over the full age span (Tanner et al., 2024). Variability in the length-effect trajectories across corpora is negligible, and speakers differ little in their within-speaker length-effect slopes (Tanner et al., 2024).

Taken together, these results argue against treating Speech-RATE as a single monolithic attribute. The empirical record supports at least four levels of organization: a global scalar useful for conditioning and adaptation; token- or phoneme-level duration distributions that determine rhythm; interactions with other prosodic or speaker attributes that require disentanglement; and higher-level structural constraints, such as utterance length effects, that dominate social covariates in spontaneous speech. A plausible implication is that effective modeling of Speech-RATE requires both coarse control variables and mechanisms that preserve or manipulate fine-grained temporal structure.