AVFSNet: Audio-Visual Flexible-Speaker Network

• AVFSNet is an audio-visual speech separation model that flexibly handles an unknown number of speakers using a novel separation-before-counting approach with multi-scale audio encoding.
• It employs a parallel per-speaker branch architecture that integrates visual lip cues with a Branchformer-based audio encoder to robustly extract individual speech signals.
• The system jointly optimizes waveform reconstruction and speaker counting through multi-task learning, achieving state-of-the-art results on VoxCeleb2 and LRS2.

Searching arXiv for the AVFSNet paper and closely related context. First, the target paper. Searching for arXiv ID (Zhang et al., 17 Jul 2025). AVFSNet, short for Audio-Visual Flexible-Speaker Network, is an audio-visual speech separation model designed for monaural mixtures containing an unknown and variable number of active speakers under noisy, real-world acoustic conditions. It combines a Branchformer-based multi-scale audio encoder, a parallel per-speaker audio-visual separation architecture, and a speaker counting head trained jointly with separation in a “separation-before-counting” paradigm. The system is introduced in “AVFSNet: Audio-Visual Speech Separation for Flexible Number of Speakers with Multi-Scale and Multi-Task Learning” (Zhang et al., 17 Jul 2025), where it is evaluated on VoxCeleb2 and LRS2 and reported to achieve state-of-the-art results across multiple evaluation metrics.

1. Problem formulation and motivation

AVFSNet addresses the setting in which a monaural mixture is modeled as

$x(t) = \sum_{i=1}^N s_i(t) + n(t),$

where $s_i(t)$ are speech signals from an unknown number of active speakers, $N$ is unknown at test time, and $n(t)$ is noise. The accompanying video provides lip sequences for all $M$ visible speakers,

$$\mathcal{V} = \{v_1, v_2, \ldots, v_M\}, \quad N \le M.$$

The objective is twofold: to separate each active speaker waveform $\{s_i(t)\}_{i=1}^N$ and to estimate the number of active speakers $N$ without prior knowledge of that number (Zhang et al., 17 Jul 2025).

The formulation explicitly targets a limitation of many strong speech separation and audio-visual separation systems, including ConvTasNet, Sepformer, VisualVoice, AV-ConvTasNet, and AV-Sepformer, which assume a fixed known number of speakers. In that regime, output dimensionality, loss computation, and decoder design remain fixed. When $N$ is unknown, however, a system must either produce variable-sized outputs or map a fixed maximal output set to an unknown true speaker set while suppressing phantom outputs. AVFSNet is proposed against this backdrop of flexible-number separation, where prior approaches often rely on recursive extraction, count-then-separate strategies with multiple decoders, or fixed upper-bound max-speaker separation; the paper identifies error accumulation, limited scalability, noisy phantom outputs, and weak robustness in real acoustic environments as central issues (Zhang et al., 17 Jul 2025).

The model also assumes that no new speakers enter during the spoken segment and that the number of active speakers does not exceed the number of visible faces. This makes the visual stream structurally central: the number of candidate output branches is determined by visible speakers rather than by a fixed architectural speaker count. A plausible implication is that AVFSNet reframes flexible-source separation as a visually indexed candidate-selection problem rather than as a purely acoustic cardinality-estimation problem.

2. System architecture

AVFSNet consists of an audio encoder, a visual encoder, a parallel separator, an audio decoder, and a speaker counting module (Zhang et al., 17 Jul 2025). The audio encoder comprises a Conv1D frontend followed by a Branchformer-based dual-branch encoder. The visual encoder uses a 3D CNN and ResNet18 lip embedding pipeline followed by a temporal convolutional network. The separator is instantiated once per visible speaker using shared weights and is based on AV-Sepformer. The decoder reconstructs waveforms with transposed Conv1D. The counting head performs binary classification on each estimated mask to infer whether the corresponding branch represents an active speaker.

The high-level processing sequence is fixed. The mixed waveform $x(t)$ is encoded into audio features $s_i(t)$0. For each visible speaker $s_i(t)$1, the lip sequence $s_i(t)$2 is encoded into $s_i(t)$3. The audio representation is chunked and replicated across branches, and each branch combines replicated audio features with one speaker-specific visual stream to estimate a mask $s_i(t)$4, from which an estimated waveform $s_i(t)$5 is decoded. The counting head then maps $s_i(t)$6 to a presence probability $s_i(t)$7. At inference, a threshold on $s_i(t)$8 determines which separated outputs are retained (Zhang et al., 17 Jul 2025).

This branch-wise design differs from recursive or fixed-decoder strategies because each candidate speaker is processed independently through a shared-weight separation module. The architecture therefore scales with the number of visual candidates rather than with a predeclared speaker count. The paper characterizes this as handling unknown $s_i(t)$9 by a parallel per-speaker branch that scales with the number of visual candidate speakers (Zhang et al., 17 Jul 2025).

3. Multi-scale audio encoding and visual feature extraction

The audio input is a single-channel waveform $N$0 sampled at 16 kHz. A Conv1D frontend produces

$N$1

with kernel size $N$2, stride $N$3, and feature dimension $N$4 (Zhang et al., 17 Jul 2025). A Branchformer encoder then generates the audio representation $N$5 through two parallel branches.

The first branch is an MHSA branch that captures global long-range temporal dependencies. After LayerNorm and linear projections to queries, keys, and values,

$N$6

multi-head attention is applied with $N$7 heads and head dimension $N$8:

$N$9

The second branch is a cgMLP branch for local dependency modeling. After LayerNorm and linear up-projection to $n(t)$0 followed by GELU,

$n(t)$1

the hidden representation is split along channels,

$n(t)$2

A depth-wise Conv1D is applied after normalization to one half,

$n(t)$3

and the two halves are combined with element-wise gating,

$n(t)$4

A linear down-projection returns to the original feature dimension:

$n(t)$5

The two branches are fused with a learnable scalar $n(t)$6:

$n(t)$7

The paper presents this as a multi-scale representation combining global and local structure without requiring multiple separate encoders (Zhang et al., 17 Jul 2025).

On the visual side, lip movement sequences are processed at 25 FPS. The visual encoder uses a Conv3D plus ResNet18 backbone pretrained on lip-reading, followed by a multi-layer TCN to model temporal dynamics. For speaker $n(t)$8, the output is

$n(t)$9

with $M$0 aligned to the chunk index of the audio representation (Zhang et al., 17 Jul 2025). This temporal alignment is enforced by chunking the audio encoder output into overlapping segments of length $M$1 and hop $M$2, yielding

$M$3

where $M$4 is chosen to match the temporal dimension of the visual features. The paper states that cross-modal attention then uses visual features as queries over audio chunks, guaranteeing time alignment at the chunk level (Zhang et al., 17 Jul 2025).

4. Parallel separation and audio-visual fusion

The separator adopts a dual-path chunking strategy analogous to DPRNN and Sepformer. The encoded audio features are padded and segmented into overlapping chunks of length $M$5 with hop $M$6:

$M$7

For $M$8 potential speakers, this tensor is replicated $M$9 times, and each replica is paired with one speaker-specific visual feature stream. Each branch computes

$$\mathcal{V} = \{v_1, v_2, \ldots, v_M\}, \quad N \le M.$$0

Overlap-add reconstruction converts chunk-wise mask estimates back to

$$\mathcal{V} = \{v_1, v_2, \ldots, v_M\}, \quad N \le M.$$1

Each branch is independent at the output level but shares the same AV separation module weights across candidate speakers (Zhang et al., 17 Jul 2025).

The separation module is based on AV-Sepformer and contains three parts. The IntraTransformer operates on the chunked representation:

$$\mathcal{V} = \{v_1, v_2, \ldots, v_M\}, \quad N \le M.$$2

The CrossModalTransformer performs cross-attention with visual features as queries and audio features as keys and values. With learnable projections $$\mathcal{V} = \{v_1, v_2, \ldots, v_M\}, \quad N \le M.$$3, $$\mathcal{V} = \{v_1, v_2, \ldots, v_M\}, \quad N \le M.$$4, and $$\mathcal{V} = \{v_1, v_2, \ldots, v_M\}, \quad N \le M.$$5,

$$\mathcal{V} = \{v_1, v_2, \ldots, v_M\}, \quad N \le M.$$6

This is the main audio-visual fusion point and is described as focusing on audio components consistent with lip motion (Zhang et al., 17 Jul 2025). The InterTransformer then models longer-term dependencies:

$$\mathcal{V} = \{v_1, v_2, \ldots, v_M\}, \quad N \le M.$$7

Waveform reconstruction uses transposed convolution:

$$\mathcal{V} = \{v_1, v_2, \ldots, v_M\}, \quad N \le M.$$8

where $$\mathcal{V} = \{v_1, v_2, \ldots, v_M\}, \quad N \le M.$$9 denotes element-wise multiplication across channels and time (Zhang et al., 17 Jul 2025). The separation path therefore remains entirely mask-based, but the masks are conditioned on speaker-specific visual evidence. This suggests that visual cues function as branch selectors within a shared acoustic latent space rather than as an auxiliary post hoc filter.

5. Multi-task learning and flexible-speaker inference

Training is organized into three stages (Zhang et al., 17 Jul 2025). In Stage 1, the audio encoder, visual encoder, separator, and decoder are trained in a single-branch configuration to maximize SI-SNR between the estimated and ground-truth target speech. The paper gives the SI-SNR loss for a single utterance as

$\{s_i(t)\}_{i=1}^N$0

with

$\{s_i(t)\}_{i=1}^N$1

In Stage 2, the backbone is frozen and the speaker counting module is pretrained. Positive and negative audio-visual pairs are constructed depending on whether the mixture contains the speaker whose lips are supplied. The counting module takes mask $\{s_i(t)\}_{i=1}^N$2 as input, processes it through convolution and max pooling, applies multi-head self-attention over time, compresses the resulting features into a vector $\{s_i(t)\}_{i=1}^N$3, and predicts a probability

$\{s_i(t)\}_{i=1}^N$4

The training loss is binary cross-entropy:

$\{s_i(t)\}_{i=1}^N$5

where $\{s_i(t)\}_{i=1}^N$6 is the ground-truth speaker-presence label.

In Stage 3, the entire model is unfrozen and jointly fine-tuned with a multi-task objective using learnable uncertainty weights:

$\{s_i(t)\}_{i=1}^N$7

The paper states that this allows separation and counting to co-adapt so that estimated masks are simultaneously useful for waveform reconstruction and discriminable for speaker presence classification (Zhang et al., 17 Jul 2025).

At inference, the model first extracts lip sequences for all visible speakers, producing $\{s_i(t)\}_{i=1}^N$8 candidates. For each candidate, it outputs a separated waveform $\{s_i(t)\}_{i=1}^N$9 and an existence probability $N$0. The final output set is obtained by thresholding:

$N$1

where $N$2 is a predefined threshold, and the number of active speakers is the number of branches satisfying $N$3 (Zhang et al., 17 Jul 2025). The paper describes this as “separation-before-counting.” Unlike architectures with a hard speaker-cap upper bound, AVFSNet has no hard architectural upper bound because one branch can in principle be instantiated for each detected face; the practical upper bound is set by the number of tracked speakers and computational resources.

6. Experimental protocol and empirical findings

The reported experiments use VoxCeleb2 and LRS2 (Zhang et al., 17 Jul 2025). VoxCeleb2 is described as containing approximately 1 million clips, more than 2000 hours of material, about 6000 speakers, and real noise. The training setup uses 800 training speakers and 118 evaluation speakers, split into 59 validation and 59 test speakers, all disjoint from training; all utterances exceed 4 seconds. Mixture sets include 20k training mixtures, 5k validation mixtures, and 3k test mixtures, with 2-speaker, 3-speaker, and combined 2/3-speaker test sets. Mixtures are generated with random SNR in $N$4 dB between target and interfering speech. LRS2 is used to construct 3000-sample 2-speaker and 3-speaker test sets for cross-dataset generalization.

Optimization uses Adam, Distributed Data Parallel on 2 × RTX 3090, Automatic Mixed Precision, validation-based learning-rate scheduling, gradient clipping, and early stopping (Zhang et al., 17 Jul 2025). The paper compares a “Combining” strategy, which mixes 2- and 3-speaker data during training, with a “Transfer” strategy, which pretrains on 2-speaker mixtures and fine-tunes on 3-speaker mixtures. Transfer is reported to work vastly better: on LRS2, SI-SDRi is 6.43 dB for combining versus 13.56 dB for transfer, and on VoxCeleb2, 6.36 dB versus 12.95 dB.

For 2-speaker separation, AVFSNet is compared with AV-ConvTasNet, AV-DPRNN, MuSE, AV-Sepformer, and SEANet. On LRS2-2mix, AVFSNet attains SI-SDRi/SDRi/PESQ of 14.34/14.13/2.51, compared with 13.05/13.52/2.25 for SEANet and 12.64/12.97/2.20 for AV-Sepformer. On VoxCeleb2-2mix, AVFSNet records 12.93/12.97/2.49, compared with 12.77/13.13/2.34 for SEANet and 11.96/12.29/2.25 for AV-Sepformer (Zhang et al., 17 Jul 2025).

For 3-speaker separation, the paper distinguishes models trained only on 2-speaker mixtures from models trained by transfer learning. On LRS2-3mix, AVFSNet improves from 6.81/7.04/1.36 in the non-transfer setting to 13.56/13.96/1.83 under transfer, while AV-Sepformer improves from 6.49/6.65/1.35 to 12.22/12.63/1.70. On VoxCeleb2-3mix, AVFSNet improves from 6.59/6.90/1.44 to 12.95/13.49/1.89, whereas AV-Sepformer improves from 6.43/6.70/1.42 to 11.59/12.15/1.76 (Zhang et al., 17 Jul 2025). The paper interprets this as indicating stronger adaptation to increasing mixture complexity.

Ablation studies isolate the effects of the encoder and counting module. In encoder comparisons on VoxCeleb2, a conventional convolutional encoder yields SI-SDRi 11.96 and STOI 0.88 on 2mix and SI-SDRi 10.68 and STOI 0.76 on 3mix; a multi-kernel attention multi-scale encoder yields approximately 12.03 on 2mix and 10.95 on 3mix; the Branchformer encoder yields 12.93 with STOI 0.89 on 2mix and 12.95 with STOI 0.81 on 3mix (Zhang et al., 17 Jul 2025). In flexible-number comparisons on VoxCeleb2-2&3mix, Conv encoder plus attractor counting gives SI-SDRi 8.64, STOI 0.57, SCA 59.87%; Conv encoder plus proposed counting gives 9.08, 0.59, 64.62%; Branchformer plus attractor counting gives 10.45, 0.68, 75.79%; and AVFSNet, combining Branchformer with the proposed counting head, gives 10.88, 0.71, 84.74%.

Noise robustness is examined across input SNR ranges from $N$5 dB. The paper reports that AVFSNet outperforms AV-DPRNN in all SNR intervals. In the lowest interval, $N$6 dB, AV-DPRNN achieves SI-SDR 0.11 and SI-SDRi 15.14 dB, whereas AVFSNet attains SI-SDR 4.73 and SI-SDRi 19.76 dB (Zhang et al., 17 Jul 2025). Speaker-count extrapolation experiments extend from 2 to 10 speakers. For SI-SDRi, the paper reports 14.00 for AVFSNet at 2 speakers, 10.85 at 4 speakers, and 2.69 at 10 speakers; corresponding AV-Sepformer results are 13.20, 9.88, and 2.32, and AV-DPRNN results are 10.45, 7.70, and 1.90 (Zhang et al., 17 Jul 2025). Even though performance declines with cardinality, the paper emphasizes that gains remain positive at 10 speakers.

7. Significance, constraints, and limitations

Within the scope of the reported experiments, AVFSNet combines three ideas that are usually treated separately in prior work: multi-scale audio encoding, visually indexed parallel separation, and mask-based speaker counting (Zhang et al., 17 Jul 2025). Its contribution is not merely the addition of a counting head to a separator; rather, counting is performed on estimated masks, which allows the separation process itself to define the representation on which activity decisions are made. This suggests a tighter coupling between source extraction and cardinality inference than in regression-based or attractor-based counting schemes.

The model’s reported robustness is tied to both modalities. The paper attributes part of its performance under low SNR and high overlap to the visual stream, which provides a modality independent of acoustic noise, and part of its performance across speaker counts and SNR ranges to the Branchformer encoder, whose MHSA and cgMLP branches capture global and local structure, respectively (Zhang et al., 17 Jul 2025). Cross-dataset evaluation from VoxCeleb2 to LRS2 is presented as evidence of decent generalization across domains and unseen speakers.

The paper also identifies several limitations. AVFSNet relies on visible lips and faces, so performance may degrade with occlusion, off-camera speakers, or tracking errors. Its compute cost is high because the architecture uses parallel branches together with a Transformer-based backbone, especially when the number of visible speakers $N$7 is large. Although there is no hard architectural upper bound on speaker count, performance decreases as the number of speakers grows, even if SI-SDRi remains positive at 10 speakers. Domain shifts beyond VoxCeleb2 and LRS2, including low-resolution surveillance settings, different languages, and varying camera viewpoints, are not fully studied (Zhang et al., 17 Jul 2025).

These constraints delimit the current scope of the approach. A plausible implication is that AVFSNet is best understood as a flexible-speaker architecture for visually grounded separation rather than as a universal unconstrained source-separation model. Within that domain, it advances the treatment of unknown speaker cardinality by aligning output multiplicity with detected visual candidates and by using joint optimization of separation and counting to suppress phantom outputs (Zhang et al., 17 Jul 2025).