Incremental Machine Speech Chain (IMSC)

• IMSC is a neural framework enabling real-time integration of incremental ASR and TTS, reducing delay by up to 9× through iterative, block-wise processing.
• It employs attention-based models with joint cross-entropy and L2 loss optimization, yielding significant improvements in accuracy under low-resource conditions.
• The framework supports continual learning and multi-task extensions by leveraging generative replay and gradient projection to mitigate catastrophic forgetting.

The Incremental Machine Speech Chain (IMSC) is a neural framework that enables real-time, streaming interaction between automatic speech recognition (ASR) and text-to-speech synthesis (TTS), and supports downstream processing such as machine translation and continual learning. It addresses the latency and feedback limitations of traditional machine speech chains, emulating human-like capabilities for listening while speaking, and provides mechanisms for learning under low supervision and in sequential or multi-task domains. The IMSC paradigm has been realized in several key research efforts, which propose differentiable, block-wise, and symbol-synchronous processing loops, with joint training routines and empirical analyses quantifying the trade-off between latency, accuracy, and generalizability (Novitasari et al., 2020, Tyndall et al., 2024, Sudoh et al., 2020).

1. Core Architecture and Processing Workflow

The IMSC integrates an Incremental ASR (ISR) and an Incremental TTS (ITTS) in a closed, short-term feedback loop. ISR processes input audio in small blocks, emitting partial transcript tokens at each step. Each partial hypothesis is immediately consumed by ITTS, which reconstructs corresponding spectrogram frames in real time (Novitasari et al., 2020). The chain operates iteratively as follows:

• Data Chunking: The input utterance $X$ of $S$ frames is split into $N = S/W$ blocks, with each block $X_n$ of $W$ frames (commonly $W=8$, i.e., $0.14\,{\rm s}$). The target text $Y$ of length $T$ is partitioned into $N$ segments $Y_1, ..., Y_N$, typically aligned using attention transfers from a non-incremental ASR teacher.
• Iterative Block Loop:

1. ISR consumes block $X_n$ (optionally with look-back and look-ahead), updates its RNN state, and greedily emits a partial token sequence $\hat{Y}_n$ with a special end-of-block marker. 2. ITTS receives $\hat{Y}_n$ (with context), synthesizes block output $\hat{X}_n$ (approx. $W$ mel-spectrogram frames), and halts on emitting a stop flag. 3. Symmetrically, gold $Y_n$ can be fed to ITTS to produce synthetic $X'_n$ for further ISR training.

This loop enables real-time echoing of partial transcripts and spectrograms, significantly reducing system delay compared to non-incremental pipelines.

2. Mathematical Formulation and Loss Functions

The ISR is an attention-based sequence-to-sequence model that, at block $n$, models the conditional probability: $$p(\hat{Y}_n \mid X_{1:n}) = \prod_{t=1}^{K_n} p(y_t \mid y_{<t}, \mathbf{c}_t),$$ where $\mathbf{c}_t$ is the attention context over encoder outputs for the current (and possibly surrounding) frames, and the RNN decoder state is recurrent across blocks. Greedy decoding is employed to minimize latency, though shallow beam search is feasible at cost of increased block delay.

The incremental ASR loss for block $n$ is the cross-entropy: $$\mathcal{L}_{\rm ISR}^{(n)} = -\sum_{t=1}^{K_n} \log p(y_t \mid y_{<t}, X_{1:n}),$$ averaged over all $N$ blocks to yield the total ISR loss $\mathcal{L}_{\rm ISR}$.

The ITTS module, also encoder-decoder based, synthesizes $W_n$ frames from incoming partial text: $$p(\hat{X}_n \mid \hat{Y}_{1:n}) = \prod_{t=1}^{W_n} p(x_t \mid x_{<t}, \mathbf{h}_t),$$ with RNN state $\mathbf{h}_t$ attending over character embeddings. At each step, four mel-spectrogram frames are emitted, until the stop flag or minimum frame requirement is reached. The loss is a framewise $L_2$ reconstruction: $$\mathcal{L}_{\rm TTS}^{(n)} = \left\| X_n - \hat{X}_n \right\|_2^2,$$ again averaged per utterance as $\mathcal{L}_{\rm TTS}$.

To optimize the speech chain, the joint objective

$$\mathcal{L}_{\rm chain} = \mathcal{L}_{\rm ISR} + \mathcal{L}_{\rm TTS}$$

is minimized by alternating gradient updates between ISR and ITTS.

3. Training Protocols and Streaming Algorithms

The IMSC training proceeds in two phases:

• Independent Supervised Training: ISR and ITTS are first trained separately, with full input/target pairs. Typical datasets include Wall Street Journal SI-84 for ASR/TTS, with chain training on additional unlabeled audio/text such as SI-200 (Novitasari et al., 2020).
• Joint Chain Training: Using both labeled and unlabeled data, ISR and ITTS generate pseudo-labeled outputs for each other in a dual-loop configuration (ISR→ITTS and ITTS→ISR). Teacher-forcing yields a substantial character error rate (CER) gain relative to greedily feeding previous predictions.

Streaming efficiency is achieved by minimal block sizes and judicious use of context:

• ISR employs 4 main, 2 look-back, 4 look-ahead blocks (context window $\sim1.4\,{\rm s}$), but the critical delay is dominated by look-ahead ($\sim0.56\,{\rm s}$) and decoding.
• ITTS typically waits for $\sim30$ characters before generating speech, with per-block context.

4. Continual Learning and Generative Replay within IMSC

IMSC generalizes beyond paired data to support continual learning using the speech chain loop as a replay mechanism (Tyndall et al., 2024). By embedding a TTS module in the ASR/TTS loop and using Gradient Episodic Memory (GEM), IMSC mitigates catastrophic forgetting:

• GEM Integration: For each new speech recognition task, TTS synthesizes pseudo-speech from earlier task transcripts, which is stored in memory to support replay-based consolidation.
• Training Regimen: The process encompasses supervised pretraining, semi-supervised mutual learning with pseudo-data, and sequential continual learning using episodic memory buffers per task.
• Gradient Projection: During each update, GEM solves a quadratic program to ensure updated gradients do not increase past-task loss, using gradient projections derived from the replay memory.

Experiments on LJ Speech show that IMSC+GEM achieves substantially lower CER compared to standard fine-tuning or multitask baselines—for instance, 15.5% CER in noisy conditions with 30/70 labeled/unlabeled ratio, improving with more labeled data.

5. Latency, Accuracy, and Empirical Trade-offs

Empirical analyses show that IMSC achieves dramatic reductions in end-to-end system latency:

Setting	ISR Delay	TTS Wait	ASR CER (%)	TTS $L_2$ Loss
Non-incremental baseline	7.88 s	103 chars	7.27 (nat-sp)	0.77
Incremental (IMSC)	0.84 s	30 chars	9.43 (nat-sp)	0.79

The system thus achieves a $9\times$ speed-up in ASR output and $3.4\times$ in TTS emission, at a $2$-point CER and $0.02$ $L_2$ degradation (Novitasari et al., 2020). Teacher-forcing during chain training provides an additional $\sim2\%$ absolute CER reduction. Chain training, as compared to fully supervised baselines, yields up to $40\%$ relative CER improvement in low-resource settings.

In continual learning, the IMSC+GEM combination demonstrates AVG backward transfer (BWT) $\approx4.7\%$ and minimal forward transfer penalties (FWT $\approx-0.3\%$), outperforming simple fine-tuning and pure multitask paradigms (Tyndall et al., 2024).

6. Extensions to Language Translation and End-to-End Modalities

The IMSC principle generalizes to more complex, multi-stage processing pipelines, such as simultaneous speech-to-speech translation (Sudoh et al., 2020). In such systems:

• ISR emits source-language subword streams in fixed-length blocks.
• An Incremental MT (IMT) module applies a wait-$k$ prefix-to-prefix policy to stream translation tokens.
• ITTS incrementally synthesizes output waveforms, often employing phrase-based segmentation for synthesis.

Module-level average processing delays are: ISR, $1.24\,{\rm s}$; IMT, $2.04\,{\rm s}$; ITTS, $3.22\,{\rm s}$ (total Ear-Voice Span $3$–$4\,{\rm s}$). Module-wise error rates are reported for ISR (CER: $22\%$–$20\%$), IMT (BLEU-4: $3.5$–$4.9$), and ITTS (MOS: $2.34$–$2.55$), reflecting challenges in latency-accuracy trade-off for end-to-end systems.

7. Limitations, Scalability, and Research Directions

Key limitations of current IMSC realizations include:

• Scalability with respect to the number of sequential tasks is constrained by memory buffer size and TTS synthesis quality (Tyndall et al., 2024).
• The fidelity of generative replay is bottlenecked by TTS quality—poor synthesizer outputs may degrade ASR retention.
• Current experiments focus on control settings (e.g., noise-augmented monolingual data); extension to multilingual, cross-domain, or speaker-adaptive scenarios remains an open problem.

Future research directions include adaptive memory allocation, hybrid replay-regularization strategies (incorporating EWC or knowledge distillation), and joint training of the entire chain to optimize latency-quality objectives.

IMSC frameworks provide a fully differentiable, real-time processing architecture that reduces delay while preserving accuracy, enables robust semi-supervised and continual learning, and is extensible to streaming scenarios such as machine translation and spoken-dialog systems (Novitasari et al., 2020, Tyndall et al., 2024, Sudoh et al., 2020).