TF-LocoFormer: Dual-Path Audio Transformer

• LocoFormer is a family of Transformer-based architectures for TF-domain audio separation that fuse global self-attention with convolutional local modeling.
• The model alternates between frequency and temporal sequence processing using ConvSwiGLU for local feature encoding and MHSA for capturing long-range dependencies.
• Extensions like FLA-TFLocoformer and FasTUSS adapt the base design for efficient, conditional, and music source separation tasks in diverse acoustic scenarios.

Searching arXiv for papers on "LocoFormer" and "TF-Locoformer" to ground the article in current literature. Searching arXiv for "LocoFormer" exact term and related variants. LocoFormer most commonly denotes a family of Transformer-based architectures that combine self-attention with explicit local modeling by convolution in time–frequency (TF) dual-path source separation; in this literature, the canonical instantiation is TF-Locoformer, short for “Transformer with LOcal modeling by COnvolution” (Saijo et al., 2024). In the audio domain, TF-Locoformer operates on short-time Fourier transform (STFT) features, alternates sequence modeling over frequency and time axes, and uses convolutional feed-forward networks (ConvSwiGLU) to encode locality while multi-head self-attention (MHSA) captures global dependencies (Saijo et al., 2024). Subsequent work has used TF-Locoformer as a backbone for efficient long-sequence separation, conditional unified source separation, positional-encoding studies, and unsupervised music-source-separation pre-training (Wang et al., 27 Aug 2025, Paissan et al., 15 Jul 2025, Saijo et al., 28 Apr 2025, Saijo et al., 12 May 2025). The name should be distinguished from the unrelated “Locformer” for temporal moment localization in video (Rodriguez-Opazo et al., 2021) and the later robotics model “LocoFormer: Generalist Locomotion via Long-context Adaptation” (Liu et al., 28 Sep 2025).

1. Terminological scope and naming

In the speech and audio-separation literature, LocoFormer refers to “Transformer with local modeling by convolution,” and TF-Locoformer is its TF-domain dual-path realization (Saijo et al., 2024). The defining premise is that pure Transformer blocks in TF dual-path architectures underperform if local modeling is not explicitly enforced, whereas TF-Locoformer combines self-attention for global pattern modeling with convolutional feed-forward networks before and after self-attention to inject strong local modeling (Saijo et al., 2024).

This naming convention has been reused in several later works. “FLASepformer: Efficient Speech Separation with Gated Focused Linear Attention Transformer” treats TF-Locoformer as a baseline and introduces FLA-TFLocoformer by replacing TF-Locoformer’s temporal MHSA with Gated Focused Linear Attention (Wang et al., 27 Aug 2025). “FasTUSS: Faster Task-Aware Unified Source Separation” uses TF-Locoformer as the architectural core of TUSS, a conditional model for multiple audio source-separation tasks (Paissan et al., 15 Jul 2025). “A Comparative Study on Positional Encoding for Time-frequency Domain Dual-path Transformer-based Source Separation Models” takes TF-Locoformer as the base architecture for a systematic analysis of positional encoding and length extrapolation (Saijo et al., 28 Apr 2025). In music source separation, “Is MixIT Really Unsuitable for Correlated Sources?” uses a band-split TF-Locoformer, denoted BS-Locoformer, as the separator (Saijo et al., 12 May 2025).

A separate paper, “LocFormer: Enabling Transformers to Perform Temporal Moment Localization on Long Untrimmed Videos With a Feature Sampling Approach,” uses the spelling “LocFormer” rather than “LocoFormer” and addresses video grounding rather than audio separation (Rodriguez-Opazo et al., 2021). Another distinct usage appears in “LocoFormer: Generalist Locomotion via Long-context Adaptation,” where the term designates a long-context Transformer-XL locomotion controller for legged and wheeled robots (Liu et al., 28 Sep 2025). In current arXiv usage, therefore, the term is polysemous; in audio research, however, it generally designates the TF-Locoformer lineage (Saijo et al., 2024).

2. Core architecture in TF-domain source separation

TF-Locoformer operates on complex STFT inputs and performs alternating frequency and temporal sequence modeling via Locoformer blocks, producing complex spectra for each separated source followed by inverse STFT (Saijo et al., 2024). The mixture is represented in the TF domain as

$\bm{X} \in \mathbb{R}^{2 \times T \times F},$

where the leading $2$ corresponds to real and imaginary parts, $T$ is the number of frames, and $F$ is the number of frequency bins (Saijo et al., 2024). An encoder maps this input to

$$\bm{Z} = \mathrm{gLN}(\mathrm{Conv2D}(\bm{X})) \in \mathbb{R}^{D \times T \times F},$$

after which the model alternates between frequency-path and temporal-path processing (Saijo et al., 2024).

The dual-path structure is central. In the frequency-path step, $Z$ is viewed as $T$ sequences of length $F$; in the temporal-path step, it is viewed as $F$ sequences of length $T$ (Saijo et al., 2024). In each path, one Locoformer block applies the sequence update

$2$0

$2$1

$2$2

with ConvSwiGLU and MHSA acting along the active sequence axis (Saijo et al., 2024). This macaron-style placement of convolutional feed-forward sublayers is not incidental: the paper reports that removing the first ConvSwiGLU or replacing SwiGLU with Swish degrades SI-SNRi and SDRi, indicating that both the pre/post-attention local modeling and the gating are material to performance (Saijo et al., 2024).

ConvSwiGLU replaces the standard Transformer FFN’s two linear layers with a Conv1D followed by a Deconv1D and uses a SwiGLU activation: $2$3

$2$4

$2$5

The convolution is local along time or frequency depending on the active path, while attention remains global over that sequence (Saijo et al., 2024). This decomposition underlies the model’s name: locality is handled explicitly by convolution, permitting attention to focus on long-range dependencies.

TF-Locoformer also introduces RMSGroupNorm, which normalizes each TF bin’s $2$6-dimensional feature vector in groups and reduces to RMSNorm when $2$7 (Saijo et al., 2024). The authors report consistent gains over RMSNorm across model sizes, which they interpret as improved disentanglement of per-bin features (Saijo et al., 2024). In later derivative models, RMSGroupNorm remains part of the TF-Locoformer design, including FLA-TFLocoformer, where the Gated MLP normalization uses RMSGroupNorm (Wang et al., 27 Aug 2025).

3. Modeling locality, positional information, and long-sequence behavior

The architectural rationale of TF-Locoformer is that self-attention and convolution play distinct but complementary roles. In the base formulation, self-attention provides global pattern modeling across time or frequency sequences, while the ConvSwiGLU feed-forward networks supply the local inductive bias that RNN-based TF models had previously provided (Saijo et al., 2024). The 2025 positional-encoding study makes this separation more explicit by characterizing TF-Locoformer as a model in which “locality is injected by ConvSwiGLU,” whereas MHSA is the global mechanism over the chosen path dimension (Saijo et al., 28 Apr 2025).

TF-Locoformer originally uses rotary positional encoding (RoPE) along the active sequence axis (Saijo et al., 2024). The positional-encoding study systematically compares absolute sinusoidal PE, KERPLE, RoPE, and no positional encoding (NoPE) within TF-Locoformer and BS-Locoformer (Saijo et al., 28 Apr 2025). Its key finding is two-part: when handling sequences that are the same length as or shorter than those seen during training, models with positional encodings achieve better performance; however, models without positional encoding exhibit superior length extrapolation, and this trend is particularly pronounced when the model contains convolutional layers (Saijo et al., 28 Apr 2025).

The mechanistic explanation offered there is that convolution acts as an implicit positional encoder through zero-padding and local receptive fields, allowing NoPE to function well and generalize to longer sequences and unseen sampling rates (Saijo et al., 28 Apr 2025). On WHAMR!, the study reports parity between reproduced TF-Locoformer (S) and TF-Locoformer-NoPE (S), both at 17.8 SI-SDRi / 16.1 SDRi, and reports TF-Locoformer-NoPE (M) at 18.8 / 17.1 versus 18.5 / 16.9 for the prior TF-Locoformer (M) (Saijo et al., 28 Apr 2025). This suggests that, within this architecture family, explicit positional encoding is not invariably required for strong performance, provided the convolutional pathway remains intact.

The same study also reports that NoPE training was approximately $2$8 faster than RoPE in one WHAMR! setting, reflecting reduced overhead in attention computation without positional rotations or bias handling (Saijo et al., 28 Apr 2025). A plausible implication is that the locality encoded by ConvSwiGLU does not merely complement attention; in some regimes it partially substitutes for explicit positional structure.

4. Efficiency variants: from TF-Locoformer to FLA-TFLocoformer

A major limitation of the base TF-Locoformer is that MHSA remains quadratic in sequence length, with $2$9 frequency attention and $T$0 temporal attention (Saijo et al., 2024). “FLASepformer” addresses this by introducing Focused Linear Attention and building FLASepformer with linear complexity for efficient speech separation (Wang et al., 27 Aug 2025). One of its two variants, FLA-TFLocoformer, specifically modifies TF-Locoformer by replacing the MHSA in Temporal Modeling with Gated Focused Linear Attention while keeping Frequency Modeling and Conv-SwiGLU unchanged (Wang et al., 27 Aug 2025).

The replacement is narrowly targeted. TF-Locoformer’s STFT-based front end, convolutional frequency-modeling blocks, and Conv-SwiGLU feed-forward components are retained; only the temporal global-attention module is replaced (Wang et al., 27 Aug 2025). This matters because TF-Locoformer already shortens the raw sequence through STFT and already encodes local TF structure through convolution; FLA-TFLocoformer therefore preserves the original local/global decomposition while changing the temporal global operator to one with linear complexity (Wang et al., 27 Aug 2025).

The paper presents the comparison in terms of similarity functions. Softmax attention uses

$T$1

with $T$2 time complexity in sequence length $T$3 (Wang et al., 27 Aug 2025). Linear attention replaces this with

$T$4

and linearizes computation by computing $T$5 first, thereby avoiding explicit construction of $T$6 (Wang et al., 27 Aug 2025). Focused Linear Attention sharpens vanilla linear attention through the Focused Function

$T$7

with focused power $T$8 in FLA-TFLocoformer, and adds a depthwise convolutional augmentation

$T$9

with kernel size $F$0, to enhance local information and compensate for rank limitations (Wang et al., 27 Aug 2025).

A lightweight Gated MLP is inserted after FLA. In FLA-TFLocoformer it uses RMSGroupNorm, a linear layer, and a nonlinearity to compute a gate that is multiplied element-wise with the FLA output (Wang et al., 27 Aug 2025). The paper states that this gating improves performance and stabilizes global feature interaction (Wang et al., 27 Aug 2025).

Empirically, FLA-TFLocoformer closely matches TF-Locoformer on WSJ0-2Mix while providing linear-complexity benefits (Wang et al., 27 Aug 2025). On WSJ0-2Mix with dynamic mixing, TF-Locoformer baselines report 22.8/23.0, 24.6/24.7, and 25.1/25.2 SI-SNRi/SDRi for S/M/L, while FLA-TFLocoformer reports 22.8/22.9, 24.4/24.5, and 24.8/24.9 (Wang et al., 27 Aug 2025). On WHAMR!, TF-Locoformer-M reports 18.5/16.9 and FLA-TFLocoformer-M reports 18.7/17.0; on Libri2Mix-360, TF-Locoformer-M reports 22.1/22.2 and FLA-TFLocoformer-M reports 22.2/22.4 (Wang et al., 27 Aug 2025). The paper interprets these results as maintaining or slightly surpassing TF-Locoformer on noisy, reverberant, and larger-corpus settings while delivering better scalability (Wang et al., 27 Aug 2025).

5. TF-Locoformer as the backbone of conditional and unified source separation

TF-Locoformer is also used as the core sequence model in Task-Aware Unified Source Separation (TUSS), where a single conditional model performs speech enhancement, speech separation, music source separation, cinematic audio source separation, and sound event separation (Paissan et al., 15 Jul 2025). In this setting, the architecture remains TF-domain and dual-path, but prompts are prepended to the encoded mixture along the time axis: $F$1 where $F$2 are learned prompt embeddings (Paissan et al., 15 Jul 2025).

The cross-prompt module consists of TF-Locoformer blocks operating on $F$3, so attention is bidirectional across prompts and mixture frames (Paissan et al., 15 Jul 2025). The output is split into prompt and mixture encodings, and per-prompt conditioning is performed by element-wise product,

$F$4

before a further stack of TF-Locoformer blocks and a decoder produce $F$5 for each requested source (Paissan et al., 15 Jul 2025). The paper emphasizes that pointwise Conv-SwiGLU is used in the cross-prompt temporal path to encourage robustness to prompt order (Paissan et al., 15 Jul 2025). It also states that RoPE is beneficial and that a start-of-sentence token is inserted between prompts and mixture in $F$6 for better performance (Paissan et al., 15 Jul 2025).

“FasTUSS” studies the performance–complexity trade-off of this TF-Locoformer-based conditional model (Paissan et al., 15 Jul 2025). Its profiling result is that for short audio chunks, convolution dominates compute, whereas MHSA becomes comparable only for long sequences (Paissan et al., 15 Jul 2025). Accordingly, the final efficiency interventions leave MHSA intact and re-parameterize the Conv-SwiGLU pathway: remove FFN1, increase convolution stride from $F$7 to $F$8, and optionally use grouped convolutions with channel shuffle (Paissan et al., 15 Jul 2025). The two final models are FasTUSS-11.7G, with 8.9M parameters and 11.7G MAC, and FasTUSS-8.3G, with 7.5M parameters and 8.3G MAC (Paissan et al., 15 Jul 2025). Relative to the original TUSS M5 baseline at 43.1G MAC and 11.1M parameters, they reduce operations by approximately $F$9 and $$\bm{Z} = \mathrm{gLN}(\mathrm{Conv2D}(\bm{X})) \in \mathbb{R}^{D \times T \times F},$$0 with average SNR drops of $$\bm{Z} = \mathrm{gLN}(\mathrm{Conv2D}(\bm{X})) \in \mathbb{R}^{D \times T \times F},$$1 dB and $$\bm{Z} = \mathrm{gLN}(\mathrm{Conv2D}(\bm{X})) \in \mathbb{R}^{D \times T \times F},$$2 dB, respectively (Paissan et al., 15 Jul 2025).

This line of work is significant because it shows that TF-Locoformer is not merely a task-specific separator. Its dual-path TF design, explicit local modeling, and prompt-compatible Transformer structure make it suitable for conditional, multi-task audio separation as well (Paissan et al., 15 Jul 2025). A plausible implication is that the original motivation for replacing RNNs—parallelizability, scalability, and versatility—extends directly to prompt-conditioned audio generation and extraction tasks.

6. Extensions to music source separation and unsupervised pre-training

In music source separation, TF-Locoformer appears in a band-split form. “Is MixIT Really Unsuitable for Correlated Sources?” uses BS-Locoformer, an encoder–separator–decoder system in the STFT domain where the stereo mixture $$\bm{Z} = \mathrm{gLN}(\mathrm{Conv2D}(\bm{X})) \in \mathbb{R}^{D \times T \times F},$$3 is partitioned into $$\bm{Z} = \mathrm{gLN}(\mathrm{Conv2D}(\bm{X})) \in \mathbb{R}^{D \times T \times F},$$4 non-overlapping subbands, encoded to $$\bm{Z} = \mathrm{gLN}(\mathrm{Conv2D}(\bm{X})) \in \mathbb{R}^{D \times T \times F},$$5, processed by TF-Locoformer blocks, and decoded into complex masks per band (Saijo et al., 12 May 2025). In this study, positional encoding is removed, following later analysis that positional encodings are not necessary for this TF-dyadic architecture (Saijo et al., 12 May 2025).

The paper investigates MixIT-based unsupervised pre-training on in-the-wild music from the Free Music Archive, using mixture-of-mixtures training with $$\bm{Z} = \mathrm{gLN}(\mathrm{Conv2D}(\bm{X})) \in \mathbb{R}^{D \times T \times F},$$6 outputs during pre-training (Saijo et al., 12 May 2025). The MixIT objective is given as

$$\bm{Z} = \mathrm{gLN}(\mathrm{Conv2D}(\bm{X})) \in \mathbb{R}^{D \times T \times F},$$7

with thresholded SNR loss

$$\bm{Z} = \mathrm{gLN}(\mathrm{Conv2D}(\bm{X})) \in \mathbb{R}^{D \times T \times F},$$8

and a relaxed least-squares solver followed by column-wise projection used to obtain an efficient near-optimal assignment matrix $$\bm{Z} = \mathrm{gLN}(\mathrm{Conv2D}(\bm{X})) \in \mathbb{R}^{D \times T \times F},$$9 (Saijo et al., 12 May 2025).

After supervised fine-tuning on MUSDB18-HQ, MixIT pre-training improves BS-Locoformer across all model sizes (Saijo et al., 12 May 2025). For the medium model, average cSDR/uSDR improves from 9.04 / 8.75 to 9.46 / 9.30; for the large model, from 9.58 / 9.21 to 9.90 / 9.72 (Saijo et al., 12 May 2025). The paper also reports a frozen-separator probe in which only the encoder/decoder are fine-tuned: the medium model rises from 3.68 to 7.27 uSDR with MixIT pre-training (Saijo et al., 12 May 2025). The authors interpret this as evidence that the pre-trained separator has learned to separate instruments even before full supervised training adapts stem semantics (Saijo et al., 12 May 2025).

This music-source-separation study also situates TF-Locoformer relative to other baselines. Unlike time-domain models such as Demucs or Conv-TasNet, BS-Locoformer operates entirely on TF features with band splitting, complex masking, and transformer blocks that separately model time and frequency with local convolutional FFNs (Saijo et al., 12 May 2025). The result is a TF-domain architecture that is compatible both with supervised stem separation and with source-agnostic unsupervised pre-training.

7. Performance profile, limitations, and broader context

Across the original speech-separation benchmarks, TF-Locoformer meets or exceeds state of the art with an RNN-free architecture (Saijo et al., 2024). On WSJ0-2Mix, TF-Locoformer-S reports 22.0/22.1 dB SI-SNRi/SDRi without dynamic mixing and 22.8/23.0 with dynamic mixing; TF-Locoformer-M reports 23.6/23.8 and 24.6/24.7; TF-Locoformer-L reports 24.2/24.3 and 25.1/25.2 (Saijo et al., 2024). On Libri2Mix, TF-Locoformer-M reports 22.1/22.2, and on DNS 2020 it reports SI-SNR 23.3 dB, STOI 98.8, and PESQ-WB 3.72 (Saijo et al., 2024). On WHAMR!, TF-Locoformer-S reports 17.4/15.9 and TF-Locoformer-M 18.5/16.9 (Saijo et al., 2024). These results established TF-Locoformer as a strong TF-domain baseline subsequently reused by multiple 2025 studies.

The model’s principal technical limitation is quadratic attention scaling with sequence length (Saijo et al., 2024). Later work addresses this in two different ways. FLA-TFLocoformer replaces temporal MHSA with linear attention to obtain $Z$0 temporal complexity while preserving near-parity separation quality (Wang et al., 27 Aug 2025). FasTUSS instead argues that, for short chunks in TUSS, most compute resides in convolution rather than MHSA, and therefore reduces cost primarily by modifying Conv-SwiGLU rather than attention (Paissan et al., 15 Jul 2025). These approaches are complementary rather than contradictory: one targets long-sequence attention bottlenecks, the other short-sequence convolutional MAC concentration.

Another limitation concerns reproducibility details in derivative models. In FLASepformer, the explicit gating formula is not provided, and the authors note that activation choice and gate dimensionality may affect reproducibility and performance nuances (Wang et al., 27 Aug 2025). In the positional-encoding study, only four positional-encoding methods are evaluated, so generalization to alternatives such as ALiBi or learned relative schemes remains open (Saijo et al., 28 Apr 2025). In the MixIT pre-training study, STFT specifics are not enumerated, and the paper notes that mixture-consistency constraints are not applied (Saijo et al., 12 May 2025).

Within the broader nomenclature of arXiv, LocoFormer is therefore best understood as a context-dependent label. In audio research, it designates a TF-domain dual-path Transformer family in which convolutional FFNs provide local modeling and attention supplies global context (Saijo et al., 2024). In video grounding, “Locformer” instead denotes a constant-memory Transformer with stochastic bucket-wise feature sampling for temporal moment localization (Rodriguez-Opazo et al., 2021). In robotics, “LocoFormer” denotes a long-context Transformer-XL policy trained with PPO on procedurally generated robots for omni-bodied locomotion adaptation (Liu et al., 28 Sep 2025). These models share a Transformer lineage and a concern with long-context reasoning or efficiency, but they are architecturally and application-wise distinct. For audio source separation, however, TF-Locoformer remains the central and historically defining usage of the term (Saijo et al., 2024).