Speaking-Rate Modification (SRM)

Updated 5 July 2026

Speaking-Rate Modification (SRM) is the controlled alteration of speech timing to adjust speed while preserving pitch, timbre, and intelligibility.
It encompasses both classic methods like WSOLA and phase vocoder techniques and modern neural approaches that manipulate feature representations.
SRM research informs applications in TTS, voice conversion, and ASR by addressing the trade-offs between global scaling and context-sensitive, non-uniform adjustments.

Speaking-rate modification (SRM) denotes the controlled alteration of speech timing so that an utterance becomes faster or slower while preserving, as far as possible, pitch, timbre, speaker identity, intelligibility, and other prosodic attributes. In the literature, SRM appears under several closely related formulations: waveform-level time-scale modification (TSM), segmental duration modification, duration control in neural text-to-speech (TTS), rhythm conversion in voice conversion (VC), and timing adaptation for automatic speech recognition (ASR). The operational definition of “speaking rate” is task-dependent: it may be “the average number of phonemes produced within some unit time,” phones per second, a sentence-level ratio such as $r=\lambda \frac{P}{T}$ , or a learned duration-state distribution over aligned speech units (Yeh et al., 2018, Zeng et al., 2015, Bae et al., 2020, Torgashov et al., 13 Mar 2026).

1. Scope, formalizations, and control variables

At the signal-processing level, SRM is commonly formalized as time warping. A generic model is

$y(t)=x\bigl(g(t)\bigr),$

where $g(t)$ is a monotonic time-warping function; global SRM uses a constant factor, whereas local or segmental SRM uses a piecewise function defined over selected intervals (Jang et al., 6 Jul 2025). In this framework, “arbitrary modification” means that “any region of the signal can be changed by specifying the starting and ending time for modification or the target duration of the specified interval,” and “any number of intervals can be modified at the same time” (Jang et al., 6 Jul 2025). A closely related statement in neural TSM defines the goal as altering “the duration of an input speech waveform $x(t)$ by a speed factor $\alpha$ , without changing its perceived pitch or degrading intelligibility” (Wisnu et al., 3 Oct 2025).

The control representation varies substantially across model classes. Classical TSM systems use explicit scaling factors such as $\beta$ , $\alpha$ , or $s$ ; neural TTS systems may use a sentence-level speaking-rate value, an utterance-level rate embedding, or explicit duration tokens; and some cross-lingual or emotional systems do not expose a user-facing rate parameter at all, instead modeling speaking rate implicitly through phoneme durations or latent style variables (Bae et al., 2020, Bandekar et al., 2023, Li et al., 2023). Rhythm-oriented work further expands SRM beyond a single scalar: “rhythmic patterns” are treated as duration distributions for different phoneme or segment types, and speaking rate becomes one component of a broader temporal style representation (Yeh et al., 2018, Hajal et al., 17 Jan 2025).

Paradigm	Control representation	Representative work
Waveform or segmental TSM	$\beta(t)$ , $\alpha$ , $y(t)=x\bigl(g(t)\bigr),$ 0	(Jang et al., 6 Jul 2025, Wisnu et al., 3 Oct 2025, Sapkota et al., 18 Jun 2026)
Duration-predictive neural synthesis	$y(t)=x\bigl(g(t)\bigr),$ 1, SR embeddings, duration tokens	(Bae et al., 2020, Bandekar et al., 2023, Torgashov et al., 13 Mar 2026)
Rhythm or style transfer	phoneme/word durations, GST/LST, Gamma duration distributions	(Li et al., 2023, Hajal et al., 17 Jan 2025, Prabhu et al., 15 Aug 2025)

2. Waveform-level and segmental time-scale modification

Classical SRM is dominated by SOLA-family methods and phase-vocoder variants. In segmental duration modification, the central relation is $y(t)=x\bigl(g(t)\bigr),$ 2, where $y(t)=x\bigl(g(t)\bigr),$ 3 is the analysis hop and $y(t)=x\bigl(g(t)\bigr),$ 4 the synthesis hop; WSOLA and SOLAFS then use waveform-similarity or normalized cross-correlation criteria to place overlapping frames so that pitch contour and short-time spectral structure are preserved as durations change (Jang et al., 6 Jul 2025). In an evaluation with a segment slowed by factor 1.5 in $y(t)=x\bigl(g(t)\bigr),$ 5 s and another sped up by factor 0.5 in $y(t)=x\bigl(g(t)\bigr),$ 6 s, WSOLA produced the smallest PSD difference, $y(t)=x\bigl(g(t)\bigr),$ 7, and energy loss $y(t)=x\bigl(g(t)\bigr),$ 8 dB, compared with SOLAFS at $y(t)=x\bigl(g(t)\bigr),$ 9 and $g(t)$ 0 dB, and phase vocoder at $g(t)$ 1 and $g(t)$ 2 dB (Jang et al., 6 Jul 2025). The reported conclusion is correspondingly narrow and practical: for segmental SRM on speech, WSOLA best preserves energy and PSD, SOLAFS is a computationally cheaper alternative, and the evaluated phase-vocoder configuration is least robust (Jang et al., 6 Jul 2025).

A separate signal-processing line treats SRM as short-time Fourier transform modification with iterative phase reconstruction. In dysarthric-ASR augmentation, speaking-rate modification is implemented with RTISI-LA; the paper specifies a scaling factor $g(t)$ 3 in the range $g(t)$ 4, with modified frame length $g(t)$ 5 and hop size $g(t)$ 6 (Sapkota et al., 18 Jun 2026). This is explicitly presented as an in-domain augmentation because dysarthric speech is often “slurred, slow, or hard-to-understand,” and speaking rate is therefore not a generic nuisance variable but a central acoustic correlate of severity (Sapkota et al., 18 Jun 2026).

Neural vocoders have shifted SRM from waveform post-processing to feature-domain warping. “Speaking-Rate-Controllable HiFi-GAN Using Feature Interpolation” inserts a differentiable interpolation layer into HiFi-GAN and compares mel-level and hidden-feature warping using both bandlimited resampling and image-style linear interpolation (Xin et al., 2022). The reported empirical result is that “warping mel-spectrograms by image scaling obtained the best performance among all proposed methods,” that the approach “outperforms a baseline time-scale modification algorithm in speech naturalness,” and that rate control can be added “without losing computational efficiency” (Xin et al., 2022). A later neural TSM model, STSM-FiLM, conditions the decoder on a continuous factor $g(t)$ 7 via FiLM and trains against WSOLA-generated targets; across encoder-decoder variants, WavLM-HiFi-GAN obtains the highest average MOS, 4.40, slightly above WSOLA at 4.33, while STFT-HiFi-GAN gives the best PESQ, 2.034, and STOI, 0.894 (Wisnu et al., 3 Oct 2025).

3. Duration modeling, neural synthesis, and rate transfer

In end-to-end TTS, a prominent line of work replaces post-hoc scaling with explicit conditioning during alignment and duration prediction. “Speaking Speed Control of End-to-End Speech Synthesis using Sentence-Level Conditioning” defines sentence-level speaking rate as

$g(t)$ 8

where $g(t)$ 9 is the number of input phonemes and $x(t)$ 0 the mel-spectrogram length after removing silence (Bae et al., 2020). The scalar $x(t)$ 1 is replicated across the text sequence and concatenated to text embeddings, allowing the model to control speaking speed “while retaining other speech attributes, such as the pitch,” especially when combined with GST-based style conditioning (Bae et al., 2020). The explicit contrast with duration-scaling baselines is that the model does not require phoneme-level duration labels, and the reported listening tests show more natural slow speech than systems that “increase or decrease duration at the same rate for the entire sentence” (Bae et al., 2020).

A more localized variant is “Speaking rate attention-based duration prediction for speed control TTS,” which inserts speaking-rate conditioning inside the duration predictor of a FastSpeech-style non-autoregressive model (Bandekar et al., 2023). It defines $x(t)$ 2, constructs an SR embedding, and uses attention between duration features and SR features so that duration prediction becomes token- and context-dependent rather than a uniform multiplicative scaling of all phonemes (Bandekar et al., 2023). The reported SR factors span $x(t)$ 3, and the best fine-tuned model, RS-SRA-FT2, achieves overall MOS 4.35 versus 3.98 for FastSpeech pace control (Bandekar et al., 2023). This supports a recurring result in neural SRM: non-uniform duration control is preferable to global duration multiplication when perceptual naturalness is a primary objective.

Cross-lingual and style-transfer systems often place speaking rate inside richer prosodic representations. In automatic dubbing, “Joint Multi-scale Cross-lingual Speaking Style Transfer” does not expose an explicit user-controllable rate parameter; instead, speaking rate is modeled implicitly through FastSpeech 2 durations and through learned global and local style representations, GST and LST, which condition the duration predictor (Li et al., 2023). The duration-only baseline is described as “an explicit SRM mechanism,” but the proposed latent style transfer outperforms it: for en→zh, MOS improves from 3.923 to 4.123 and preference from 25.83% to 65.47%; for zh→en, MOS is 4.005 versus 3.992 and preference 43.99% versus 40.41% (Li et al., 2023). Related VC work removes fixed-length constraints by transforming phoneme posteriorgram sequences with sequence-to-sequence Cycle-GAN, so that converted speaking rate and phoneme duration distributions shift toward the target speaker without parallel data (Yeh et al., 2018). A simpler any-to-any VC strategy applies WSOLA after disentangled conversion, with a global scaling factor $x(t)$ 4 derived from source and target phoneme rates, thereby increasing speaking-rate similarity with respect to the target speaker (Kuhlmann et al., 2022).

Duration control also appears in emotion conversion and articulatory modeling. In “Enhancing In-the-Wild Speech Emotion Conversion with Resynthesis-based Duration Modeling,” durations are predicted in log-space for HuBERT unit runs conditioned on speaker and continuous arousal; low-arousal outputs are longer and slower, and high-arousal outputs are shorter and faster, with the strongest contrast reported for the L1 variant, $x(t)$ 5 s (Prabhu et al., 15 Aug 2025). In the articulatory domain, AstNet treats SRM as mapping neutral articulatory trajectories to fast or slow ones with attention-based duration transformation and smoother predicted movements, removing the need for DTW alignment and improving both duration and extent of articulatory motion (Singh et al., 2020). A streaming generalization appears in VoXtream2, where duration control is performed over six duration tokens via online distribution matching,

$x(t)$ 6

enabling dynamic speaking-rate control “updated mid-utterance on the fly,” with a usable operating range of about 1–6 SPS and 74 ms first-packet latency (Torgashov et al., 13 Mar 2026).

4. ASR, dysarthria, and robustness to rate variation

SRM has a parallel history in ASR, where the objective is often not perceptual naturalness but robustness to rate mismatch. “Learning Speech Rate in Speech Recognition” defines ROS as phones per second and argues that abnormal ROS changes “not only the dynamic but also the static property of speech signals” (Zeng et al., 2015). The DNN baseline on a Mandarin spontaneous-speech test set reports WER 45.71% for slow speech, 28.04% for normal speech, and 31.22% for fast speech; augmenting the acoustic model with a 1-dimensional ROS feature changes these to 44.92%, 28.05%, and 29.54%, and combining DNN-ROS compensation with HMM transition adaptation gives 44.76% for slow and 29.08% for fast speech (Zeng et al., 2015). The implication is narrow but important: rate mismatch is not just a duration problem.

For pathological speech, recent work reframes SRM as rhythm conversion rather than simple global scaling. “Unsupervised Rhythm and Voice Conversion of Dysarthric to Healthy Speech for ASR” models speech as distributions of three segment classes—sonorants, obstruents, and silences—derived from clustered WavLM-Large features and a dynamic-programming segmenter (Hajal et al., 17 Jan 2025). Speaking rate is approximated as sonorants-per-second, and two conversion regimes are studied: a global SR-like factor $x(t)$ 7, and fine-grained duration mapping through Gamma distributions for each segment type (Hajal et al., 17 Jan 2025). The reported finding is that rhythm conversion, with or without kNN-VC, improves Whisper ASR especially for more severe dysarthria, and that rhythm normalization is more important than voice matching for this task (Hajal et al., 17 Jan 2025).

Severity-specific augmentation studies reach a related conclusion from a different angle. In “Improving End-to-End Speech Recognition for Dysarthric Speech through In-Domain Data Augmentation,” the best WERs are achieved with SRM $x(t)$ 8 for low severity, 9.02%, and medium severity, 38.11%, while pitch modification is slightly better for high severity at 55.15%; the corresponding best SRM result for high severity is 55.80% at $x(t)$ 9 (Sapkota et al., 18 Jun 2026). The paper’s own summary is explicit: “speaking rate modification (SRM) proved most effective for decoding low and medium severity speech, while pitch modification (PM) was the most beneficial for recognizing high severity speech” (Sapkota et al., 18 Jun 2026).

5. Intelligibility, fluency, and perceptual consequences

A distinct branch of SRM aims at fluency enhancement rather than literal rate control. “Increase Apparent Public Speaking Fluency By Speech Augmentation” does not uniformly time-scale speech; it removes filled pauses, classifies silences as fluent or disfluent, and shortens disfluent pauses toward the median of fluent silence durations in the same recording (Das et al., 2018). The resulting timing changes increase Speech Rate from 165.3571 to 186.241, Articulation Rate from 171.0986 to 186.241, Phonation-Time Ratio from 58.865 to 65.570, Mean Length of Runs from 0.400 to 0.495, and reduce Mean Length of Pauses from 0.654 s to 0.365 s and Filled Pauses per Minute from 3.659 to 1.762 (Das et al., 2018). In this formulation, SRM is pause-aware and disfluency-aware rather than segment-uniform.

Recent perceptual work further complicates the common assumption that global slowing improves intelligibility. “Covertly improving intelligibility with data-driven adaptations of speech timing” derives reverse-correlation kernels showing that the temporal influence of speaking rate before a target vowel contrast has a “scissor-like pattern,” with contrastive distal and congruent proximal effects, and that this pattern is stable across L1-English listeners and L2-English listeners with French, Mandarin, and Japanese L1s (Tuttösí et al., 31 Mar 2026). When translated into a MatchaTTS duration-control algorithm, the proposed targeted slowing reduces tense-word WER from 60.2% to 29.5% in single-target sentences and overall WER from 24.3% to 15.2% in double-target sentences, while participants nevertheless often judge global slowing or target-wide stretching as clearer even when those manipulations increase comprehension errors (Tuttösí et al., 31 Mar 2026). The documented dissociation is therefore not between speed and quality alone, but between subjective clarity judgments and actual lexical comprehension.

6. Limitations, recurring debates, and research directions

The literature does not converge on a single control interface. Some systems expose explicit, continuous rate parameters— $\alpha$ 0, $\alpha$ 1, $\alpha$ 2, or SPS—whereas others embed speaking rate inside duration predictors, duration-state histograms, or latent style variables (Bae et al., 2020, Sapkota et al., 18 Jun 2026, Torgashov et al., 13 Mar 2026, Li et al., 2023). This creates a persistent trade-off between precision and naturalness. Explicit control offers direct user manipulation and easier synchronization, but latent or style-mediated control often yields better prosodic integration. In automatic dubbing, for example, the paper explicitly notes that there is “no direct duration/rate control interface” and no explicit utterance-duration matching loss, even though style-based transfer improves perceived timing (Li et al., 2023).

A second debate concerns global versus non-uniform scaling. Uniform duration scaling remains attractive because it is simple, real-time capable, and often sufficient for moderate factors. Yet multiple papers report that human speaking-rate change is not uniform across phonemes, segment classes, or time windows (Bae et al., 2020, Bandekar et al., 2023, Tuttösí et al., 31 Mar 2026). This suggests that direct duration multiplication is a useful baseline, but not a complete model of natural SRM. The same point reappears in pathological-speech ASR, where three-class rhythm conversion outperforms generic time-stretching, and in dubbing, where latent local style transfer outperforms duration-only transfer (Hajal et al., 17 Jan 2025, Li et al., 2023).

Extreme-rate behavior remains a general weakness. HiFi-GAN interpolation, STSM-FiLM, and streaming duration-state controllers all report degradation or instability at aggressive slow or fast settings, especially when operating outside the trained factor range or when long-duration states accumulate (Xin et al., 2022, Wisnu et al., 3 Oct 2025, Torgashov et al., 13 Mar 2026). Dataset bias is equally recurrent: several systems are trained on clean speech, single speakers, or constrained rate ranges, while dysarthric and in-the-wild settings require broader temporal variability and more reliable segmentation (Hajal et al., 17 Jan 2025, Wisnu et al., 3 Oct 2025). A plausible implication is that future SRM systems will combine explicit duration objectives with richer, context-sensitive representations: the cross-lingual dubbing work explicitly suggests adding a rate scalar to MST-FastSpeech 2 and introducing duration alignment losses; the dysarthric rhythm-conversion work suggests more fine-grained segment categories; and the duration-predictor work points toward explicit SR losses or richer SR embeddings (Li et al., 2023, Hajal et al., 17 Jan 2025, Bandekar et al., 2023).

Across these strands, a stable conclusion emerges. SRM is no longer adequately described as uniform playback acceleration or deceleration. In current research, it is a family of timing-control problems spanning waveform TSM, duration prediction, rhythm transfer, pathological-speech normalization, and intelligibility optimization. The central methodological shift is from hard “copy duration” strategies toward learned, distribution-aware, and context-sensitive control of when speech should be slower, when it should be faster, and which parts of the utterance should bear those changes (Li et al., 2023, Wisnu et al., 3 Oct 2025, Tuttösí et al., 31 Mar 2026).