---
title: Speaking-Rate Modification (SRM)
url: https://www.emergentmind.com/topics/speaking-rate-modification-srm
type: topic
---

# Speaking-Rate Modification (SRM)

Speaking-rate modification (SRM) denotes the controlled alteration of speech timing so that an utterance becomes faster or slower while preserving, as far as possible, pitch, timbre, speaker identity, intelligibility, and other prosodic attributes. In the literature, SRM appears under several closely related formulations: waveform-level time-scale modification (TSM), segmental duration modification, duration control in neural text-to-speech (TTS), rhythm conversion in voice conversion (VC), and timing adaptation for automatic speech recognition (ASR). The operational definition of “speaking rate” is task-dependent: it may be “the average number of phonemes produced within some unit time,” phones per second, a sentence-level ratio such as \(r=\lambda \frac{P}{T}\), or a learned duration-state distribution over aligned speech units [1808.03113] [1506.00799] [2007.15281] [2603.13518].

## 1. Scope, formalizations, and control variables

At the signal-processing level, SRM is commonly formalized as time warping. A generic model is
\[
y(t)=x\bigl(g(t)\bigr),
\]
where \(g(t)\) is a monotonic time-warping function; global SRM uses a constant factor, whereas local or segmental SRM uses a piecewise function defined over selected intervals [2507.04264]. In this framework, “arbitrary modification” means that “any region of the signal can be changed by specifying the starting and ending time for modification or the target duration of the specified interval,” and “any number of intervals can be modified at the same time” [2507.04264]. A closely related statement in neural TSM defines the goal as altering “the duration of an input speech waveform \(x(t)\) by a speed factor \(\alpha\), without changing its perceived pitch or degrading intelligibility” [2510.02672].

The control representation varies substantially across model classes. Classical TSM systems use explicit scaling factors such as \(\beta\), \(\alpha\), or \(s\); neural TTS systems may use a sentence-level speaking-rate value, an utterance-level rate embedding, or explicit duration tokens; and some cross-lingual or emotional systems do not expose a user-facing rate parameter at all, instead modeling speaking rate implicitly through phoneme durations or latent style variables [2007.15281] [2310.08846] [2305.05203]. Rhythm-oriented work further expands SRM beyond a single scalar: “rhythmic patterns” are treated as duration distributions for different phoneme or segment types, and speaking rate becomes one component of a broader temporal style representation [1808.03113] [2501.10256].

| Paradigm | Control representation | Representative work |
|---|---|---|
| Waveform or segmental TSM | \(\beta(t)\), \(\alpha\), \(s\) | [2507.04264], [2510.02672], [2606.19797] |
| Duration-predictive neural synthesis | \(r=\lambda \frac{P}{T}\), SR embeddings, duration tokens | [2007.15281], [2310.08846], [2603.13518] |
| Rhythm or style transfer | phoneme/word durations, GST/LST, Gamma duration distributions | [2305.05203], [2501.10256], [2508.11535] |

## 2. Waveform-level and segmental time-scale modification

Classical SRM is dominated by SOLA-family methods and phase-vocoder variants. In segmental duration modification, the central relation is \(S_S=\beta S_A\), where \(S_A\) is the analysis hop and \(S_S\) the synthesis hop; WSOLA and SOLAFS then use waveform-similarity or normalized cross-correlation criteria to place overlapping frames so that pitch contour and short-time spectral structure are preserved as durations change [2507.04264]. In an evaluation with a segment slowed by factor 1.5 in \([2,3]\) s and another sped up by factor 0.5 in \([4.5,5]\) s, WSOLA produced the smallest PSD difference, \(0.0743\), and energy loss \(-1.52\) dB, compared with SOLAFS at \(0.1220\) and \(-26.15\) dB, and phase vocoder at \(0.1106\) and \(55.82\) dB [2507.04264]. The reported conclusion is correspondingly narrow and practical: for segmental SRM on speech, WSOLA best preserves energy and PSD, SOLAFS is a computationally cheaper alternative, and the evaluated phase-vocoder configuration is least robust [2507.04264].

A separate signal-processing line treats SRM as short-time Fourier transform modification with iterative phase reconstruction. In dysarthric-ASR augmentation, speaking-rate modification is implemented with RTISI-LA; the paper specifies a scaling factor \(s\) in the range \(0.5 \le s \le 2\), with modified frame length \(L=256\cdot s\) and hop size \(S=\frac{L}{4}\) [2606.19797]. This is explicitly presented as an in-domain augmentation because dysarthric speech is often “slurred, slow, or hard-to-understand,” and speaking rate is therefore not a generic nuisance variable but a central acoustic correlate of severity [2606.19797].

Neural vocoders have shifted SRM from waveform post-processing to feature-domain warping. “Speaking-Rate-Controllable HiFi-GAN Using Feature Interpolation” inserts a differentiable interpolation layer into HiFi-GAN and compares mel-level and hidden-feature warping using both bandlimited resampling and image-style linear interpolation [2204.10561]. The reported empirical result is that “warping mel-spectrograms by image scaling obtained the best performance among all proposed methods,” that the approach “outperforms a baseline time-scale modification algorithm in speech naturalness,” and that rate control can be added “without losing computational efficiency” [2204.10561]. A later neural TSM model, STSM-FiLM, conditions the decoder on a continuous factor \(\alpha\) via FiLM and trains against WSOLA-generated targets; across encoder-decoder variants, WavLM-HiFi-GAN obtains the highest average MOS, 4.40, slightly above WSOLA at 4.33, while STFT-HiFi-GAN gives the best PESQ, 2.034, and STOI, 0.894 [2510.02672].

## 3. Duration modeling, neural synthesis, and rate transfer

In end-to-end TTS, a prominent line of work replaces post-hoc scaling with explicit conditioning during alignment and duration prediction. “Speaking Speed Control of End-to-End Speech Synthesis using Sentence-Level Conditioning” defines sentence-level speaking rate as
\[
r=\lambda \frac{P}{T},
\]
where \(P\) is the number of input phonemes and \(T\) the mel-spectrogram length after removing silence [2007.15281]. The scalar \(r\) is replicated across the text sequence and concatenated to text embeddings, allowing the model to control speaking speed “while retaining other speech attributes, such as the pitch,” especially when combined with GST-based style conditioning [2007.15281]. The explicit contrast with duration-scaling baselines is that the model does not require phoneme-level duration labels, and the reported listening tests show more natural slow speech than systems that “increase or decrease duration at the same rate for the entire sentence” [2007.15281].

A more localized variant is “Speaking rate attention-based duration prediction for speed control TTS,” which inserts speaking-rate conditioning inside the duration predictor of a FastSpeech-style non-autoregressive model [2310.08846]. It defines \(SR=\frac{\text{length}(y)}{\text{length}(tokens)}\), constructs an SR embedding, and uses attention between duration features and SR features so that duration prediction becomes token- and context-dependent rather than a uniform multiplicative scaling of all phonemes [2310.08846]. The reported SR factors span \(\{0.54,0.66,0.77,0.83,0.89,1.00,1.11,1.17,1.23,1.34,1.46\}\), and the best fine-tuned model, RS-SRA-FT2, achieves overall MOS 4.35 versus 3.98 for FastSpeech pace control [2310.08846]. This supports a recurring result in neural SRM: non-uniform duration control is preferable to global duration multiplication when perceptual naturalness is a primary objective.

Cross-lingual and style-transfer systems often place speaking rate inside richer prosodic representations. In automatic dubbing, “Joint Multi-scale Cross-lingual Speaking Style Transfer” does not expose an explicit user-controllable rate parameter; instead, speaking rate is modeled implicitly through FastSpeech 2 durations and through learned global and local style representations, GST and LST, which condition the duration predictor [2305.05203]. The duration-only baseline is described as “an explicit SRM mechanism,” but the proposed latent style transfer outperforms it: for en→zh, MOS improves from 3.923 to 4.123 and preference from 25.83% to 65.47%; for zh→en, MOS is 4.005 versus 3.992 and preference 43.99% versus 40.41% [2305.05203]. Related VC work removes fixed-length constraints by transforming phoneme posteriorgram sequences with sequence-to-sequence Cycle-GAN, so that converted speaking rate and phoneme duration distributions shift toward the target speaker without parallel data [1808.03113]. A simpler any-to-any VC strategy applies WSOLA after disentangled conversion, with a global scaling factor \(\alpha=\frac{r_{\text{tgt}}}{r_{\text{src}}}\) derived from source and target phoneme rates, thereby increasing speaking-rate similarity with respect to the target speaker [2209.01978].

Duration control also appears in emotion conversion and articulatory modeling. In “Enhancing In-the-Wild Speech Emotion Conversion with Resynthesis-based Duration Modeling,” durations are predicted in log-space for HuBERT unit runs conditioned on speaker and continuous arousal; low-arousal outputs are longer and slower, and high-arousal outputs are shorter and faster, with the strongest contrast reported for the L1 variant, \(\Delta_{1-7}=0.37\) s [2508.11535]. In the articulatory domain, AstNet treats SRM as mapping neutral articulatory trajectories to fast or slow ones with attention-based duration transformation and smoother predicted movements, removing the need for DTW alignment and improving both duration and extent of articulatory motion [2006.03107]. A streaming generalization appears in VoXtream2, where duration control is performed over six duration tokens via online distribution matching,
\[
W(d)=\exp\Big(\beta\big(\log_{10}(P_{\text{target}}(d))-\log_{10}(P_{\text{acc}}(d))\big)\Big),
\]
enabling dynamic speaking-rate control “updated mid-utterance on the fly,” with a usable operating range of about 1–6 SPS and 74 ms first-packet latency [2603.13518].

## 4. ASR, dysarthria, and robustness to rate variation

SRM has a parallel history in ASR, where the objective is often not perceptual naturalness but robustness to rate mismatch. “Learning Speech Rate in Speech Recognition” defines ROS as phones per second and argues that abnormal ROS changes “not only the dynamic but also the static property of speech signals” [1506.00799]. The DNN baseline on a Mandarin spontaneous-speech test set reports WER 45.71% for slow speech, 28.04% for normal speech, and 31.22% for fast speech; augmenting the acoustic model with a 1-dimensional ROS feature changes these to 44.92%, 28.05%, and 29.54%, and combining DNN-ROS compensation with HMM transition adaptation gives 44.76% for slow and 29.08% for fast speech [1506.00799]. The implication is narrow but important: rate mismatch is not just a duration problem.

For pathological speech, recent work reframes SRM as rhythm conversion rather than simple global scaling. “Unsupervised Rhythm and Voice Conversion of Dysarthric to Healthy Speech for ASR” models speech as distributions of three segment classes—sonorants, obstruents, and silences—derived from clustered WavLM-Large features and a dynamic-programming segmenter [2501.10256]. Speaking rate is approximated as sonorants-per-second, and two conversion regimes are studied: a global SR-like factor \(r=\frac{SR_t}{SR_s}\), and fine-grained duration mapping through Gamma distributions for each segment type [2501.10256]. The reported finding is that rhythm conversion, with or without kNN-VC, improves Whisper ASR especially for more severe dysarthria, and that rhythm normalization is more important than voice matching for this task [2501.10256].

Severity-specific augmentation studies reach a related conclusion from a different angle. In “Improving End-to-End Speech Recognition for Dysarthric Speech through In-Domain Data Augmentation,” the best WERs are achieved with SRM \((s=0.8)\) for low severity, 9.02%, and medium severity, 38.11%, while pitch modification is slightly better for high severity at 55.15%; the corresponding best SRM result for high severity is 55.80% at \(s=1.2\) [2606.19797]. The paper’s own summary is explicit: “speaking rate modification (SRM) proved most effective for decoding low and medium severity speech, while pitch modification (PM) was the most beneficial for recognizing high severity speech” [2606.19797].

## 5. Intelligibility, fluency, and perceptual consequences

A distinct branch of SRM aims at fluency enhancement rather than literal rate control. “Increase Apparent Public Speaking Fluency By Speech Augmentation” does not uniformly time-scale speech; it removes filled pauses, classifies silences as fluent or disfluent, and shortens disfluent pauses toward the median of fluent silence durations in the same recording [1812.03415]. The resulting timing changes increase Speech Rate from 165.3571 to 186.241, Articulation Rate from 171.0986 to 186.241, Phonation-Time Ratio from 58.865 to 65.570, Mean Length of Runs from 0.400 to 0.495, and reduce Mean Length of Pauses from 0.654 s to 0.365 s and Filled Pauses per Minute from 3.659 to 1.762 [1812.03415]. In this formulation, SRM is pause-aware and disfluency-aware rather than segment-uniform.

Recent perceptual work further complicates the common assumption that global slowing improves intelligibility. “Covertly improving intelligibility with data-driven adaptations of speech timing” derives reverse-correlation kernels showing that the temporal influence of speaking rate before a target vowel contrast has a “scissor-like pattern,” with contrastive distal and congruent proximal effects, and that this pattern is stable across L1-English listeners and L2-English listeners with French, Mandarin, and Japanese L1s [2603.30032]. When translated into a MatchaTTS duration-control algorithm, the proposed targeted slowing reduces tense-word WER from 60.2% to 29.5% in single-target sentences and overall WER from 24.3% to 15.2% in double-target sentences, while participants nevertheless often judge global slowing or target-wide stretching as clearer even when those manipulations increase comprehension errors [2603.30032]. The documented dissociation is therefore not between speed and quality alone, but between subjective clarity judgments and actual lexical comprehension.

## 6. Limitations, recurring debates, and research directions

The literature does not converge on a single control interface. Some systems expose explicit, continuous rate parameters—\(r\), \(s\), \(\alpha\), or SPS—whereas others embed speaking rate inside duration predictors, duration-state histograms, or latent style variables [2007.15281] [2606.19797] [2603.13518] [2305.05203]. This creates a persistent trade-off between precision and naturalness. Explicit control offers direct user manipulation and easier synchronization, but latent or style-mediated control often yields better prosodic integration. In automatic dubbing, for example, the paper explicitly notes that there is “no direct duration/rate control interface” and no explicit utterance-duration matching loss, even though style-based transfer improves perceived timing [2305.05203].

A second debate concerns global versus non-uniform scaling. Uniform duration scaling remains attractive because it is simple, real-time capable, and often sufficient for moderate factors. Yet multiple papers report that human speaking-rate change is not uniform across phonemes, segment classes, or time windows [2007.15281] [2310.08846] [2603.30032]. This suggests that direct duration multiplication is a useful baseline, but not a complete model of natural SRM. The same point reappears in pathological-speech ASR, where three-class rhythm conversion outperforms generic time-stretching, and in dubbing, where latent local style transfer outperforms duration-only transfer [2501.10256] [2305.05203].

Extreme-rate behavior remains a general weakness. HiFi-GAN interpolation, STSM-FiLM, and streaming duration-state controllers all report degradation or instability at aggressive slow or fast settings, especially when operating outside the trained factor range or when long-duration states accumulate [2204.10561] [2510.02672] [2603.13518]. Dataset bias is equally recurrent: several systems are trained on clean speech, single speakers, or constrained rate ranges, while dysarthric and in-the-wild settings require broader temporal variability and more reliable segmentation [2501.10256] [2510.02672]. A plausible implication is that future SRM systems will combine explicit duration objectives with richer, context-sensitive representations: the cross-lingual dubbing work explicitly suggests adding a rate scalar to MST-FastSpeech 2 and introducing duration alignment losses; the dysarthric rhythm-conversion work suggests more fine-grained segment categories; and the duration-predictor work points toward explicit SR losses or richer SR embeddings [2305.05203] [2501.10256] [2310.08846].

Across these strands, a stable conclusion emerges. SRM is no longer adequately described as uniform playback acceleration or deceleration. In current research, it is a family of timing-control problems spanning waveform TSM, duration prediction, rhythm transfer, pathological-speech normalization, and intelligibility optimization. The central methodological shift is from hard “copy duration” strategies toward learned, distribution-aware, and context-sensitive control of when speech should be slower, when it should be faster, and which parts of the utterance should bear those changes [2305.05203] [2510.02672] [2603.30032].

Source: https://www.emergentmind.com/topics/speaking-rate-modification-srm