--- title: TDS-ConvNets for Low-Latency Speech Recognition url: https://www.emergentmind.com/topics/tds-convnets type: topic --- # TDS-ConvNets for Low-Latency Speech Recognition Time-Depth Separable Convolutional Networks (TDS-ConvNets) are a class of fully convolutional architectures specifically designed for high-throughput, low-latency end-to-end speech recognition. Developed with the goal of providing efficient online acoustic modeling, TDS-ConvNets employ “time–depth separable” 1D convolutional residual blocks, together with Connectionist Temporal Classification (CTC) loss and optimized beam search decoding. This approach results in significant improvements in inference speed and latency, while maintaining or surpassing competitive word error rates (WER) in both clean and noisy speech recognition scenarios [2001.09727]. ## 1. Mathematical Formulation of the TDS Block A TDS block is defined as a residual 1D convolution that is both time- and depth-separable. The structural parameters are denoted as TDS($c, k_w, w, r$), where $c$ is the number of channel-groups, $w$ is the width of each group (so total channels $C = c \cdot w$), $k_w$ is the temporal kernel size, and $r$ is the right padding (number of future frames used for context). The block comprises the following stages: - **Layer normalization** is applied along the channel and group width axes, with statistics computed for each individual time-step: \[ \mu_t = \frac{1}{C} \sum_{i=1}^C X_{t,i}, \quad \sigma_t^2 = \frac{1}{C} \sum_{i=1}^C (X_{t,i} - \mu_t)^2, \quad \widetilde X_{t,i} = \gamma_i \frac{X_{t,i} - \mu_t}{\sqrt{\sigma_t^2 + \epsilon}} + \beta_i. \] - **Grouped 1D convolution over time:** The normalized input is split into $c$ contiguous groups. Each group $g$ undergoes temporal convolution using group-specific kernel $K^{(g)}$ with asymmetric padding $(p_\text{left} = k_w - 1 - r,\, p_\text{right} = r)$: \[ Y^{(g)}_{t, 1\ldots w} = \sum_{u=0}^{k_w-1} \widetilde{X}_{t - (k_w-1-r) + u, [g\cdot w+1 : g\cdot w+w]} \cdot K^{(g)}[u]. \] The outputs for all groups are concatenated across the channel dimension. - **Pointwise (1×1) convolution:** The output $Y \in \mathbb{R}^{T \times C}$ is transformed by a learnable matrix $W \in \mathbb{R}^{C \times C}$ and bias $b \in \mathbb{R}^C$: \[ Z_t = Y_t W + b. \] - **Residual connection and ReLU:** \[ \mathrm{TDS\!-\!Block}(X) = \mathrm{ReLU}(Z + X). \] Factorization into grouped and pointwise convolutions significantly reduces parameter count compared to dense temporal convolution. Asymmetric padding, with small $r$, restricts the required future context, trading off accuracy for lower latency. ## 2. Model Architecture and Layerwise Specifications The TDS-ConvNet acoustic encoder consists of a sequence of convolutional and TDS blocks, processing 80-dimensional log-Mel filterbank features sampled at 10 ms intervals. The architecture is as follows: | Layer(s) | Specification | Output Shape | |------------------|-----------------------------------------------------------------------|----------------------------------| | 0 (Input) | 80-dim log-Mel filterbank, 10 ms stride | $T \times 80$ | | 1 | 1×1 conv (80→256), $k=5$, $s=2$, pad=2 | $T\!/\!2 \times 256$ | | 2–3 | 2× TDS($c=10, k_w=9, w=80, r=1$), no subsample | $T\!/\!2 \times 800$ | | 4 | 1×1 conv (800→512), $k=5$, $s=2$, pad=2 | $T\!/\!4 \times 512$ | | 5–7 | 3× TDS($c=19, k_w=9, w=27, r=1$), no subsample | $T\!/\!4 \times 513$ | | 8 | 1×1 conv (512→512), $k=5$, $s=2$, pad=2 | $T\!/\!8 \times 512$ | | 9–12 | 4× TDS($c=23, k_w=9, w=22, r=1$), no subsample | $T\!/\!8 \times 512$ | | 13 | 1×1 conv (512→512), $k=1$, $s=1$, pad=0 | $T\!/\!8 \times 512$ | | 14–18 | 5× TDS($c=27, k_w=9, w=19, r=1$), no subsample | $T\!/\!8 \times 513$ | | 19 (Output) | Linear (512→N), log-softmax, CTC | $T\!/\!8 \times N$ | The model contains approximately 104 million parameters, with an overall 8× subsampling factor due to three stride-2 convolutions. Future context per TDS block is one frame ($r = 1 \Rightarrow$ 10 ms), accumulating to about 250 ms across all TDS layers. ## 3. Temporal Dynamics, Receptive Field, and Latency The effective receptive field is governed by the progression of convolutional and TDS layers: \[ R_0 = 1,\quad R_\ell = R_{\ell-1} + (k_\ell - 1) \prod_{i<\ell} s_i, \] with $k_\ell$ and $s_i$ as the kernel size and stride, respectively, of layer $\ell$. For the three stride-2 convolutions ($k=5, s=2$) and all TDS blocks ($k=9, s=1$), the full receptive field is approximately 10 seconds (i.e., 1000 frames at 10 ms interval). Future context (right padding) only accumulates across the $L$ TDS layers: \[ \mathrm{future\_dependency} = \sum_{\ell \in \{\text{TDS}\}} r_\ell \times \prod_{i<\ell} s_i \approx 250\,\text{ms}. \] Constraining $r_\ell$ to small values—here, $r = 1$—dramatically reduces total model-induced latency. Operational end-to-end latency is the sum of acoustic model future context (~250 ms), decoder delay, and audio chunking overhead. ## 4. CTC Objective and Sequence Decoding The encoder output is projected to $N$ vocabulary logits per frame, followed by log-softmax. The sequence is trained with Connectionist Temporal Classification (CTC), with loss \[ \mathcal{L}_{\mathrm{CTC}}(x, y) = -\ln \sum_{\pi \in \mathcal{A}(y, T)} \prod_{t=1}^T p(\pi_t \mid x), \] where $\mathcal{A}(y, T)$ is the set of all length-$T$ alignments (with insertion of blank tokens) that collapse to $y$. Standard forward–backward dynamic programming is used, without modification of the canonical CTC formulation. Online decoding employs a prefix beam search algorithm over the CTC output, incorporating an $n$-gram language model using simple log-linear score interpolation. ## 5. Performance Metrics and Trade-Offs TDS-ConvNet systems are benchmarked against strong low-latency baselines (LC-BLSTM + LF-MMI and LC-BLSTM + RNN-T), with the following results: | Metric | LC-BLSTM + LF-MMI | LC-BLSTM + RNN-T | TDS conv + CTC | |-------------------------------|-------------------|------------------|---------------| | Parameters | 80 M | 60 M | 104 M | | Inference precision | INT8 | INT8 | FP16 | | WER (vid-clean) | 14.10% | 13.93% | 13.19% | | WER (vid-noisy) | 22.15% | 22.58% | 21.16% | | Throughput (sec audio/sec) | 55 | 64 | 147 | | RTF@40 streams | 0.70 | 0.60 | 0.26 | | User-perceived latency (40 str)| 1.18 s | – | 1.09 s | Ablation on future context vs WER demonstrates that reducing future context from 5 s to 0.25 s incurs only a ~4% relative WER degradation (vid-clean: 12.65% → 13.19%; vid-noisy: 20.44% → 21.16%) while reducing model latency by an order of magnitude. TDS-ConvNet achieves approximately three times the throughput of an optimized hybrid baseline, with significantly reduced real-time factor (RTF). ## 6. Decoder Optimizations and Practical Considerations The optimized wav2letter++ CTC beam-search decoder is further enhanced by two additional pruning strategies: 1. **Acoustic-pruning:** During beam expansion, retain only the top-$K$ tokens by local acoustic score ($K=50$). 2. **Blank-pruning:** If $p(\texttt{blank} \mid x_t) > 0.95$, only the blank arc is extended. Combined with the encoder's 8× subsampling, decoding constitutes approximately 5% of total inference time. **Training and inference notes:** - Training is performed with SpecAugment and local mean/variance normalization (window: 300 frames, ≈3 s) to enable online normalization. - Inference leverages FB GEMM for efficient mixed-precision (FP16) group convolutions. - Chunk size (e.g., 750 ms) and the number of concurrent streams (e.g., 40–60) can be tuned to balance latency and throughput. The overall architecture—fully convolutional and devoid of recurrent connections—enables streaming-friendly operation, often with higher throughput and lower latency than traditional RNN-based ASR systems, while holding competitive or superior recognition accuracy [2001.09727]. Source: https://www.emergentmind.com/topics/tds-convnets