---
title: 'TF-LocoFormer: Dual-Path Audio Transformer'
url: https://www.emergentmind.com/topics/locoformer
type: topic
---

# TF-LocoFormer: Dual-Path Audio Transformer

Searching arXiv for recent papers on "LocoFormer" and "TF-Locoformer" to ground the article in current literature.
Searching arXiv for "LocoFormer" exact term and related variants.
LocoFormer most commonly denotes a family of Transformer-based architectures that combine self-attention with explicit local modeling by convolution in time–frequency (TF) dual-path source separation; in this literature, the canonical instantiation is TF-Locoformer, short for “Transformer with LOcal modeling by COnvolution” [2408.03440]. In the audio domain, TF-Locoformer operates on short-time Fourier transform (STFT) features, alternates sequence modeling over frequency and time axes, and uses convolutional feed-forward networks (ConvSwiGLU) to encode locality while multi-head self-attention (MHSA) captures global dependencies [2408.03440]. Subsequent work has used TF-Locoformer as a backbone for efficient long-sequence separation, conditional unified source separation, positional-encoding studies, and unsupervised music-source-separation pre-training [2508.19528] [2507.11435] [2504.19605] [2505.07631]. The name should be distinguished from the unrelated “Locformer” for temporal moment localization in video [2112.10066] and the later robotics model “LocoFormer: Generalist Locomotion via Long-context Adaptation” [2509.23745].

## 1. Terminological scope and naming

In the speech and audio-separation literature, LocoFormer refers to “Transformer with local modeling by convolution,” and TF-Locoformer is its TF-domain dual-path realization [2408.03440]. The defining premise is that pure Transformer blocks in TF dual-path architectures underperform if local modeling is not explicitly enforced, whereas TF-Locoformer combines self-attention for global pattern modeling with convolutional feed-forward networks before and after self-attention to inject strong local modeling [2408.03440].

This naming convention has been reused in several later works. “FLASepformer: Efficient Speech Separation with Gated Focused Linear Attention Transformer” treats TF-Locoformer as a baseline and introduces FLA-TFLocoformer by replacing TF-Locoformer’s temporal MHSA with Gated Focused Linear Attention [2508.19528]. “FasTUSS: Faster Task-Aware Unified Source Separation” uses TF-Locoformer as the architectural core of TUSS, a conditional model for multiple audio source-separation tasks [2507.11435]. “A Comparative Study on Positional Encoding for Time-frequency Domain Dual-path Transformer-based Source Separation Models” takes TF-Locoformer as the base architecture for a systematic analysis of positional encoding and length extrapolation [2504.19605]. In music source separation, “Is MixIT Really Unsuitable for Correlated Sources?” uses a band-split TF-Locoformer, denoted BS-Locoformer, as the separator [2505.07631].

A separate paper, “LocFormer: Enabling Transformers to Perform Temporal Moment Localization on Long Untrimmed Videos With a Feature Sampling Approach,” uses the spelling “LocFormer” rather than “LocoFormer” and addresses video grounding rather than audio separation [2112.10066]. Another distinct usage appears in “LocoFormer: Generalist Locomotion via Long-context Adaptation,” where the term designates a long-context Transformer-XL locomotion controller for legged and wheeled robots [2509.23745]. In current arXiv usage, therefore, the term is polysemous; in audio research, however, it generally designates the TF-Locoformer lineage [2408.03440].

## 2. Core architecture in TF-domain source separation

TF-Locoformer operates on complex STFT inputs and performs alternating frequency and temporal sequence modeling via Locoformer blocks, producing complex spectra for each separated source followed by inverse STFT [2408.03440]. The mixture is represented in the TF domain as
\[
\bm{X} \in \mathbb{R}^{2 \times T \times F},
\]
where the leading \(2\) corresponds to real and imaginary parts, \(T\) is the number of frames, and \(F\) is the number of frequency bins [2408.03440]. An encoder maps this input to
\[
\bm{Z} = \mathrm{gLN}(\mathrm{Conv2D}(\bm{X})) \in \mathbb{R}^{D \times T \times F},
\]
after which the model alternates between frequency-path and temporal-path processing [2408.03440].

The dual-path structure is central. In the frequency-path step, \(Z\) is viewed as \(T\) sequences of length \(F\); in the temporal-path step, it is viewed as \(F\) sequences of length \(T\) [2408.03440]. In each path, one Locoformer block applies the sequence update
\[
\bm{Z} \xleftarrow{} \bm{Z} + \mathrm{ConvSwiGLU}(\bm{Z}) / 2,
\]
\[
\bm{Z} \xleftarrow{} \bm{Z} + \mathrm{MHSA}(\mathrm{Norm}(\bm{Z})),
\]
\[
\bm{Z} \xleftarrow{} \bm{Z} + \mathrm{ConvSwiGLU}(\bm{Z}) / 2,
\]
with ConvSwiGLU and MHSA acting along the active sequence axis [2408.03440]. This macaron-style placement of convolutional feed-forward sublayers is not incidental: the paper reports that removing the first ConvSwiGLU or replacing SwiGLU with Swish degrades SI-SNRi and SDRi, indicating that both the pre/post-attention local modeling and the gating are material to performance [2408.03440].

ConvSwiGLU replaces the standard Transformer FFN’s two linear layers with a Conv1D followed by a Deconv1D and uses a SwiGLU activation:
\[
\bm{Z} \xleftarrow{} \mathrm{Norm}(\bm{Z}),
\]
\[
\bm{Z} \xleftarrow{} \mathrm{Swish}(\mathrm{Conv1D}(\bm{Z})) \otimes \mathrm{Conv1D}(\bm{Z}),
\]
\[
\bm{Z} \xleftarrow{} \mathrm{Deconv1D}(\bm{Z}).
\]
The convolution is local along time or frequency depending on the active path, while attention remains global over that sequence [2408.03440]. This decomposition underlies the model’s name: locality is handled explicitly by convolution, permitting attention to focus on long-range dependencies.

TF-Locoformer also introduces RMSGroupNorm, which normalizes each TF bin’s \(D\)-dimensional feature vector in groups and reduces to RMSNorm when \(G=1\) [2408.03440]. The authors report consistent gains over RMSNorm across model sizes, which they interpret as improved disentanglement of per-bin features [2408.03440]. In later derivative models, RMSGroupNorm remains part of the TF-Locoformer design, including FLA-TFLocoformer, where the Gated MLP normalization uses RMSGroupNorm [2508.19528].

## 3. Modeling locality, positional information, and long-sequence behavior

The architectural rationale of TF-Locoformer is that self-attention and convolution play distinct but complementary roles. In the base formulation, self-attention provides global pattern modeling across time or frequency sequences, while the ConvSwiGLU feed-forward networks supply the local inductive bias that RNN-based TF models had previously provided [2408.03440]. The 2025 positional-encoding study makes this separation more explicit by characterizing TF-Locoformer as a model in which “locality is injected by ConvSwiGLU,” whereas MHSA is the global mechanism over the chosen path dimension [2504.19605].

TF-Locoformer originally uses rotary positional encoding (RoPE) along the active sequence axis [2408.03440]. The positional-encoding study systematically compares absolute sinusoidal PE, KERPLE, RoPE, and no positional encoding (NoPE) within TF-Locoformer and BS-Locoformer [2504.19605]. Its key finding is two-part: when handling sequences that are the same length as or shorter than those seen during training, models with positional encodings achieve better performance; however, models without positional encoding exhibit superior length extrapolation, and this trend is particularly pronounced when the model contains convolutional layers [2504.19605].

The mechanistic explanation offered there is that convolution acts as an implicit positional encoder through zero-padding and local receptive fields, allowing NoPE to function well and generalize to longer sequences and unseen sampling rates [2504.19605]. On WHAMR!, the study reports parity between reproduced TF-Locoformer (S) and TF-Locoformer-NoPE (S), both at 17.8 SI-SDRi / 16.1 SDRi, and reports TF-Locoformer-NoPE (M) at 18.8 / 17.1 versus 18.5 / 16.9 for the prior TF-Locoformer (M) [2504.19605]. This suggests that, within this architecture family, explicit positional encoding is not invariably required for strong performance, provided the convolutional pathway remains intact.

The same study also reports that NoPE training was approximately \(16\%\) faster than RoPE in one WHAMR! setting, reflecting reduced overhead in attention computation without positional rotations or bias handling [2504.19605]. A plausible implication is that the locality encoded by ConvSwiGLU does not merely complement attention; in some regimes it partially substitutes for explicit positional structure.

## 4. Efficiency variants: from TF-Locoformer to FLA-TFLocoformer

A major limitation of the base TF-Locoformer is that MHSA remains quadratic in sequence length, with \(O(F^2)\) frequency attention and \(O(T^2)\) temporal attention [2408.03440]. “FLASepformer” addresses this by introducing Focused Linear Attention and building FLASepformer with linear complexity for efficient speech separation [2508.19528]. One of its two variants, FLA-TFLocoformer, specifically modifies TF-Locoformer by replacing the MHSA in Temporal Modeling with Gated Focused Linear Attention while keeping Frequency Modeling and Conv-SwiGLU unchanged [2508.19528].

The replacement is narrowly targeted. TF-Locoformer’s STFT-based front end, convolutional frequency-modeling blocks, and Conv-SwiGLU feed-forward components are retained; only the temporal global-attention module is replaced [2508.19528]. This matters because TF-Locoformer already shortens the raw sequence through STFT and already encodes local TF structure through convolution; FLA-TFLocoformer therefore preserves the original local/global decomposition while changing the temporal global operator to one with linear complexity [2508.19528].

The paper presents the comparison in terms of similarity functions. Softmax attention uses
\[
\mathrm{Sim}(Q, K) = \exp(QK^{T}/\sqrt{d}),
\]
with \(O(N^2)\) time complexity in sequence length \(N\) [2508.19528]. Linear attention replaces this with
\[
\mathrm{Sim}(Q, K) = \phi(Q)\phi(K)^{T},
\]
and linearizes computation by computing \(\sum_j \phi(K_j)^T V_j\) first, thereby avoiding explicit construction of \(QK^T\) [2508.19528]. Focused Linear Attention sharpens vanilla linear attention through the Focused Function
\[
\phi_{p}(x) = f_{p}(\mathrm{RELU}(x)), \qquad f_{p}=\frac{||x||}{||x^{**p}||}x^{**p},
\]
with focused power \(p=3\) in FLA-TFLocoformer, and adds a depthwise convolutional augmentation
\[
O = \phi(Q)\phi(K)^{T}V + \mathrm{DWC1d}(V),
\]
with kernel size \(k=7\), to enhance local information and compensate for rank limitations [2508.19528].

A lightweight Gated MLP is inserted after FLA. In FLA-TFLocoformer it uses RMSGroupNorm, a linear layer, and a nonlinearity to compute a gate that is multiplied element-wise with the FLA output [2508.19528]. The paper states that this gating improves performance and stabilizes global feature interaction [2508.19528].

Empirically, FLA-TFLocoformer closely matches TF-Locoformer on WSJ0-2Mix while providing linear-complexity benefits [2508.19528]. On WSJ0-2Mix with dynamic mixing, TF-Locoformer baselines report 22.8/23.0, 24.6/24.7, and 25.1/25.2 SI-SNRi/SDRi for S/M/L, while FLA-TFLocoformer reports 22.8/22.9, 24.4/24.5, and 24.8/24.9 [2508.19528]. On WHAMR!, TF-Locoformer-M reports 18.5/16.9 and FLA-TFLocoformer-M reports 18.7/17.0; on Libri2Mix-360, TF-Locoformer-M reports 22.1/22.2 and FLA-TFLocoformer-M reports 22.2/22.4 [2508.19528]. The paper interprets these results as maintaining or slightly surpassing TF-Locoformer on noisy, reverberant, and larger-corpus settings while delivering better scalability [2508.19528].

## 5. TF-Locoformer as the backbone of conditional and unified source separation

TF-Locoformer is also used as the core sequence model in Task-Aware Unified Source Separation (TUSS), where a single conditional model performs speech enhancement, speech separation, music source separation, cinematic audio source separation, and sound event separation [2507.11435]. In this setting, the architecture remains TF-domain and dual-path, but prompts are prepended to the encoded mixture along the time axis:
\[
Z' = [P; Z] \in \mathbb{R}^{D \times (N+T) \times F},
\]
where \(P=[p_1,\ldots,p_N] \in \mathbb{R}^{D \times N}\) are learned prompt embeddings [2507.11435].

The cross-prompt module consists of TF-Locoformer blocks operating on \(Z'\), so attention is bidirectional across prompts and mixture frames [2507.11435]. The output is split into prompt and mixture encodings, and per-prompt conditioning is performed by element-wise product,
\[
\tilde{Z}_n = \tilde{Z} \odot \tilde{p}_n,
\]
before a further stack of TF-Locoformer blocks and a decoder produce \(\hat{s}_n \in \mathbb{R}^{L}\) for each requested source [2507.11435]. The paper emphasizes that pointwise Conv-SwiGLU is used in the cross-prompt temporal path to encourage robustness to prompt order [2507.11435]. It also states that RoPE is beneficial and that a start-of-sentence token is inserted between prompts and mixture in \(Z'\) for better performance [2507.11435].

“FasTUSS” studies the performance–complexity trade-off of this TF-Locoformer-based conditional model [2507.11435]. Its profiling result is that for short audio chunks, convolution dominates compute, whereas MHSA becomes comparable only for long sequences [2507.11435]. Accordingly, the final efficiency interventions leave MHSA intact and re-parameterize the Conv-SwiGLU pathway: remove FFN1, increase convolution stride from \(S=1\) to \(S=4\), and optionally use grouped convolutions with channel shuffle [2507.11435]. The two final models are FasTUSS-11.7G, with 8.9M parameters and 11.7G MAC, and FasTUSS-8.3G, with 7.5M parameters and 8.3G MAC [2507.11435]. Relative to the original TUSS M5 baseline at 43.1G MAC and 11.1M parameters, they reduce operations by approximately \(73\%\) and \(81\%\) with average SNR drops of \(0.4\) dB and \(1.2\) dB, respectively [2507.11435].

This line of work is significant because it shows that TF-Locoformer is not merely a task-specific separator. Its dual-path TF design, explicit local modeling, and prompt-compatible Transformer structure make it suitable for conditional, multi-task audio separation as well [2507.11435]. A plausible implication is that the original motivation for replacing RNNs—parallelizability, scalability, and versatility—extends directly to prompt-conditioned audio generation and extraction tasks.

## 6. Extensions to music source separation and unsupervised pre-training

In music source separation, TF-Locoformer appears in a band-split form. “Is MixIT Really Unsuitable for Correlated Sources?” uses BS-Locoformer, an encoder–separator–decoder system in the STFT domain where the stereo mixture \(X \in \mathbb{R}^{2M \times T \times F}\) is partitioned into \(Q=62\) non-overlapping subbands, encoded to \(Z \in \mathbb{R}^{D \times T \times Q}\), processed by TF-Locoformer blocks, and decoded into complex masks per band [2505.07631]. In this study, positional encoding is removed, following later analysis that positional encodings are not necessary for this TF-dyadic architecture [2505.07631].

The paper investigates MixIT-based unsupervised pre-training on in-the-wild music from the Free Music Archive, using mixture-of-mixtures training with \(N=12\) outputs during pre-training [2505.07631]. The MixIT objective is given as
\[
L_{\mathrm{MixIT}} = \min_{A \in \{0,1\}^{2\times N}} \sum_{b=1}^{2} \ell(x_b, \hat{x}_b),
\]
with thresholded SNR loss
\[
\ell_{\mathrm{SNR}}(y, \hat{y}) = -10 \log_{10} \left( \frac{\|y\|^2}{\|y - \hat{y}\|^2 + \tau \|y\|^2} \right), \qquad \tau = 10^{-3},
\]
and a relaxed least-squares solver followed by column-wise projection used to obtain an efficient near-optimal assignment matrix \(A\) [2505.07631].

After supervised fine-tuning on MUSDB18-HQ, MixIT pre-training improves BS-Locoformer across all model sizes [2505.07631]. For the medium model, average cSDR/uSDR improves from 9.04 / 8.75 to 9.46 / 9.30; for the large model, from 9.58 / 9.21 to 9.90 / 9.72 [2505.07631]. The paper also reports a frozen-separator probe in which only the encoder/decoder are fine-tuned: the medium model rises from 3.68 to 7.27 uSDR with MixIT pre-training [2505.07631]. The authors interpret this as evidence that the pre-trained separator has learned to separate instruments even before full supervised training adapts stem semantics [2505.07631].

This music-source-separation study also situates TF-Locoformer relative to other baselines. Unlike time-domain models such as Demucs or Conv-TasNet, BS-Locoformer operates entirely on TF features with band splitting, complex masking, and transformer blocks that separately model time and frequency with local convolutional FFNs [2505.07631]. The result is a TF-domain architecture that is compatible both with supervised stem separation and with source-agnostic unsupervised pre-training.

## 7. Performance profile, limitations, and broader context

Across the original speech-separation benchmarks, TF-Locoformer meets or exceeds state of the art with an RNN-free architecture [2408.03440]. On WSJ0-2Mix, TF-Locoformer-S reports 22.0/22.1 dB SI-SNRi/SDRi without dynamic mixing and 22.8/23.0 with dynamic mixing; TF-Locoformer-M reports 23.6/23.8 and 24.6/24.7; TF-Locoformer-L reports 24.2/24.3 and 25.1/25.2 [2408.03440]. On Libri2Mix, TF-Locoformer-M reports 22.1/22.2, and on DNS 2020 it reports SI-SNR 23.3 dB, STOI 98.8, and PESQ-WB 3.72 [2408.03440]. On WHAMR!, TF-Locoformer-S reports 17.4/15.9 and TF-Locoformer-M 18.5/16.9 [2408.03440]. These results established TF-Locoformer as a strong TF-domain baseline subsequently reused by multiple 2025 studies.

The model’s principal technical limitation is quadratic attention scaling with sequence length [2408.03440]. Later work addresses this in two different ways. FLA-TFLocoformer replaces temporal MHSA with linear attention to obtain \(O(N)\) temporal complexity while preserving near-parity separation quality [2508.19528]. FasTUSS instead argues that, for short chunks in TUSS, most compute resides in convolution rather than MHSA, and therefore reduces cost primarily by modifying Conv-SwiGLU rather than attention [2507.11435]. These approaches are complementary rather than contradictory: one targets long-sequence attention bottlenecks, the other short-sequence convolutional MAC concentration.

Another limitation concerns reproducibility details in derivative models. In FLASepformer, the explicit gating formula is not provided, and the authors note that activation choice and gate dimensionality may affect reproducibility and performance nuances [2508.19528]. In the positional-encoding study, only four positional-encoding methods are evaluated, so generalization to alternatives such as ALiBi or learned relative schemes remains open [2504.19605]. In the MixIT pre-training study, STFT specifics are not enumerated, and the paper notes that mixture-consistency constraints are not applied [2505.07631].

Within the broader nomenclature of arXiv, LocoFormer is therefore best understood as a context-dependent label. In audio research, it designates a TF-domain dual-path Transformer family in which convolutional FFNs provide local modeling and attention supplies global context [2408.03440]. In video grounding, “Locformer” instead denotes a constant-memory Transformer with stochastic bucket-wise feature sampling for temporal moment localization [2112.10066]. In robotics, “LocoFormer” denotes a long-context Transformer-XL policy trained with PPO on procedurally generated robots for omni-bodied locomotion adaptation [2509.23745]. These models share a Transformer lineage and a concern with long-context reasoning or efficiency, but they are architecturally and application-wise distinct. For audio source separation, however, TF-Locoformer remains the central and historically defining usage of the term [2408.03440].

Source: https://www.emergentmind.com/topics/locoformer