Picopatch: Evaluating Speaker-Switch Protocols

• Picopatch is an umbrella term for evaluation protocols that deliberately manipulate speaker continuity to assess model sensitivity to speaker-specific cues.
• It encompasses diverse methodologies including dyadic interaction control, speaker leakage tests, multi-turn consistency evaluation, and source-speaker tracing.
• Practical applications range from improving ASR robustness to fine-tuning TTS adaptations, emphasizing the separation of task-relevant signals from confounding speaker attributes.

Searching arXiv for papers on “speaker-switch test” and closely related formulations. I’m checking arXiv for papers that define or use “speaker-switch” as an evaluation protocol or control. Speaker-switch test denotes a family of evaluation procedures in speech, dialogue, speaker recognition, and speech generation research in which speaker continuity, speaker pairing, or speaker identity is deliberately perturbed, controlled, or queried. In recent arXiv literature, the term is used for a principled negative control on dyadic interaction representations, a size-matched protocol for exposing speaker leakage in clinical speech models, source-speaker verification on voice-converted speech, multi-turn speaker-consistency judgment in dialogue audio, and an interactive comparison mechanism in human-in-the-loop TTS adaptation. A plausible unifying characterization is that the test asks whether a system is responding to the intended relational or task-specific signal, or to speaker-specific shortcuts such as identity cues, overlap, or memorized voice characteristics (Nilabh et al., 1 Jun 2026, Yeh et al., 15 Apr 2026, Wang et al., 2024, Lee et al., 7 Jan 2026, Udagawa et al., 2022).

1. Terminological scope and principal variants

The label is not used in a single standardized sense. Instead, it denotes several technically distinct protocols that share a common diagnostic role.

Usage	Core operation	Primary target
Dyadic interaction control (Nilabh et al., 1 Jun 2026)	Replace one speaker’s turns with those from an unrelated speaker from a different conversation	Test whether a representation encodes dyad-specific coadaptation
Speaker leakage control (Yeh et al., 15 Apr 2026)	Compare speaker-independent and speaker-overlapped training sets with identical training size and a shared test set	Test whether performance depends on speaker overlap
Source-speaker tracing (Wang et al., 2024)	Decide whether two converted utterances originate from the same source speaker	Recover latent source identity under voice conversion
Multi-turn consistency judgment (Lee et al., 7 Jan 2026)	Detect or localize an inconsistent turn inside a sequence of turns claimed to belong to one speaker	Evaluate speaker consistency across dialogue turns
Adaptation under speaker change (Li et al., 2024)	Evaluate adapted ASR when the test speaker differs from the adaptation speaker	Measure robustness to speaker changes
Human comparison in TTS adaptation (Udagawa et al., 2022)	Let a listener switch among candidate synthetic voices while an utterance loops	Support perceptual search in speaker-embedding space

This heterogeneity suggests that speaker-switch test is best treated as an umbrella term for controlled interventions on speaker continuity. In some formulations the intervention destroys a relation that should matter, such as dyadic coadaptation. In others it removes a confound, such as speaker overlap. In others still it probes whether source identity survives manipulation, or whether a judge can detect a switched turn.

2. Dyadic conversational interaction as a controlled perturbation

The most explicit formalization appears in "Breaking the Pair: Evaluating Dyadic Interaction via Speaker Switching" (Nilabh et al., 1 Jun 2026). That work asks whether a dyadic representation captures interaction between two people, or merely the individual signatures of the two speakers. Its core representation is the Dyadic Distance Matrix (DDM), defined for turn embeddings $\mathbf{a}_1,\dots,\mathbf{a}_{|\mathbf{a}|}$ from speaker $A$ and $\mathbf{b}_1,\dots,\mathbf{b}_{|\mathbf{b}|}$ from speaker $B$ as

$$M_{i,j} = 1 - \frac{\mathbf{a}_i^\top \mathbf{b}_j}{\|\mathbf{a}_i\|_2\,\|\mathbf{b}_j\|_2}.$$

Each entry is a cosine distance between one turn from $A$ and one turn from $B$, so the matrix contains all pairwise cross-speaker similarities over the entire conversation rather than only adjacent-turn relations. The switched control preserves $A_1$’s turn sequence from conversation $\mathcal{C}_1=(A_1,B_1)$ and replaces $B_1$ with an unrelated speaker $A$0 from a different conversation $A$1:

$A$2

The paper states that this preserves each speaker’s turn-level embedding distribution, the general turn structure, and—by matching conversations by approximate turn count—the overall matrix dimensions and broad marginal statistics, while disrupting the original dyad’s co-adaptation.

The experimental setting uses CANDOR, described as a large corpus of naturalistic dyadic conversations with over 1,600 hours of speech from more than 1,300 speakers. Conversations are segmented into sentence-level turns using Cliffhanger, and sessions with fewer than 20 turns per speaker are discarded. A cross-corpus comparison uses LibriSpeech to probe the effect of read speech. Four embedding types are evaluated: wav2vec 2.0, x-vector/ECAPA-TDNN, openSMILE/GeMAPS, and all-MiniLM. For wav2vec 2.0, x-vector, and openSMILE, per-speaker $A$3-normalization is applied before DDM construction. DDMs are resized to $A$4 and standardized. Classification is binary, real versus switched, with a ResNet-50 backbone as the main model and a three-layer CNN and MLP as baselines. Training uses Adam with learning rate $A$5, batch size $A$6, early stopping, and a $A$7 split by conversation. Evaluation uses accuracy, macro-F1, and equal error rate.

The quantitative pattern is sharply modality-dependent. On CANDOR, semantic DDMs are maximally discriminable: all-MiniLM with ResNet-50 reaches $A$8 and $A$9. The same backbone on CANDOR acoustic and structural representations is much weaker: wav2vec 2.0 gives $\mathbf{b}_1,\dots,\mathbf{b}_{|\mathbf{b}|}$0 and $\mathbf{b}_1,\dots,\mathbf{b}_{|\mathbf{b}|}$1, x-vector gives $\mathbf{b}_1,\dots,\mathbf{b}_{|\mathbf{b}|}$2 and $\mathbf{b}_1,\dots,\mathbf{b}_{|\mathbf{b}|}$3, and openSMILE gives $\mathbf{b}_1,\dots,\mathbf{b}_{|\mathbf{b}|}$4 and $\mathbf{b}_1,\dots,\mathbf{b}_{|\mathbf{b}|}$5. On LibriSpeech, the same acoustic representations become much easier to distinguish: wav2vec 2.0 ResNet-50 reaches $\mathbf{b}_1,\dots,\mathbf{b}_{|\mathbf{b}|}$6, x-vector achieves perfect accuracy with all models, and openSMILE reaches $\mathbf{b}_1,\dots,\mathbf{b}_{|\mathbf{b}|}$7 with ResNet-50 and up to $\mathbf{b}_1,\dots,\mathbf{b}_{|\mathbf{b}|}$8 with the MLP. The paper interprets the CANDOR-versus-LibriSpeech gap as evidence about prosodic variability: read speech makes identity-like structure stable and easy to detect, whereas natural conversation leaves subtler cross-speaker adaptation after normalization.

GradCAM analysis gives the test an additional interpretive layer. For all-MiniLM, real conversations show strong activation near the DDM diagonal, corresponding to semantically similar temporally proximate turns; in the switched condition, this diagonal structure largely disappears. For wav2vec 2.0 and x-vector, activations are more global and less diagonal, which the authors connect to longer-timescale acoustic accommodation. The broader implication is that speaker switching is functioning as a negative control: success on real-versus-switched discrimination indicates sensitivity to dyad-specific structure rather than only marginal speaker properties.

3. Speaker leakage, overlap control, and speaker-independent evaluation

A second major formulation uses speaker switching as an evaluation control for leakage. "Who is Speaking or Who is Depressed? A Controlled Study of Speaker Leakage in Speech-Based Depression Detection" (Yeh et al., 15 Apr 2026) constructs a size-matched speaker-switch evaluation protocol on DAIC-WOZ to isolate speaker overlap while keeping training size fixed. From the standard 189-subject subset, the study obtains 6,545 participant-only speech segments. Speakers are split into a control group of 151 speakers with 5,117 segments and a target group of 38 speakers with 1,428 segments. The target-group segments are divided within each speaker into two equal halves: 714 segments form a shared test set, and 714 segments can optionally be added to training. To equalize training size, 4,403 segments from the control group form subcontrol A. Two training sets, each with 5,117 segments, are then built: Training Set A is speaker-independent, with no test speaker present in training; Training Set B is speaker-overlapped, with 4,403 control segments plus 714 subtarget segments.

This construction makes the test highly controlled: the training size and the test set are identical across conditions, and the only variable is speaker overlap. The models evaluated are Wav2Vec-Linear Probing, XLSR-eGeMAPS Concatenation, and Wav2Vec-SLS, each in original and Domain-Adversarial Neural Network (DANN) versions. Metrics are Depression Macro F1, Depression Classification Accuracy, and Speaker Identification Accuracy. The speaker-identification chance level for the 38-speaker target group is stated as $\mathbf{b}_1,\dots,\mathbf{b}_{|\mathbf{b}|}$9.

The central result is that speaker overlap dramatically inflates depression performance. For the fine-tuned Wav2Vec-Linear Probing Original model, the speaker-independent condition gives $B$0 and accuracy $B$1, whereas the speaker-overlapped condition gives $B$2 and accuracy $B$3, a gap of $B$4 percentage points. The DANN version reduces but does not remove the effect: $B$5 versus $B$6, a gap of $B$7 points. Wav2Vec-SLS shows the same pattern: the fine-tuned original model moves from $B$8 to $B$9, and the fine-tuned DANN model from $$M_{i,j} = 1 - \frac{\mathbf{a}_i^\top \mathbf{b}_j}{\|\mathbf{a}_i\|_2\,\|\mathbf{b}_j\|_2}.$$0 to $$M_{i,j} = 1 - \frac{\mathbf{a}_i^\top \mathbf{b}_j}{\|\mathbf{a}_i\|_2\,\|\mathbf{b}_j\|_2}.$$1. By contrast, XLSR-eGeMAPS shows lower overall depression accuracy and speaker identification near chance, including speaker-ID values of $$M_{i,j} = 1 - \frac{\mathbf{a}_i^\top \mathbf{b}_j}{\|\mathbf{a}_i\|_2\,\|\mathbf{b}_j\|_2}.$$2, $$M_{i,j} = 1 - \frac{\mathbf{a}_i^\top \mathbf{b}_j}{\|\mathbf{a}_i\|_2\,\|\mathbf{b}_j\|_2}.$$3, $$M_{i,j} = 1 - \frac{\mathbf{a}_i^\top \mathbf{b}_j}{\|\mathbf{a}_i\|_2\,\|\mathbf{b}_j\|_2}.$$4, and $$M_{i,j} = 1 - \frac{\mathbf{a}_i^\top \mathbf{b}_j}{\|\mathbf{a}_i\|_2\,\|\mathbf{b}_j\|_2}.$$5.

The paper’s interpretation is explicit: models with strong speaker discrimination also tend to show inflated depression accuracy under speaker-overlapped evaluation, whereas models with near-chance speaker ID show only moderate depression performance. This supports the claim that depression-related and identity-related information are highly entangled in current speech representations. Under overlapped splits, apparent clinical success can partly be "who is speaking detection." In this use, the speaker-switch test is not a perturbation of conversation structure but a controlled removal of speaker leakage.

4. Multi-turn consistency, localization, and speech-generation evaluation

A third line of work treats speaker switching as a consistency problem over dialogue turns. "SpeakerSleuth: Evaluating Large Audio-LLMs as Judges for Multi-turn Speaker Consistency" (Lee et al., 7 Jan 2026) defines three tasks. Detection asks whether a set of turns from one target speaker is consistent. Localization asks which turn is inconsistent, or predicts None. Discrimination asks which of three candidate audios best matches the target speaker for a masked turn. The benchmark contains 606 unique dialogues, 1,818 total evaluation instances, 152 speakers, and 10.2 hours of audio, drawn from Bazinga, AMI, Behavior-SD, and DailyTalk. Each dialogue yields three scenarios: S1 fully consistent, S2 gender switch via FreeVC, and S3 similar-speaker switch using ECAPA-TDNN embeddings with highest cosine similarity excluding the target speaker.

The results show that large audio-LLMs struggle to use speaker switching as a stable consistency test. Detection accuracies across S1/S2/S3 vary sharply by model. GPT-4o-audio reports $$M_{i,j} = 1 - \frac{\mathbf{a}_i^\top \mathbf{b}_j}{\|\mathbf{a}_i\|_2\,\|\mathbf{b}_j\|_2}.$$6, Gemini2.5-Flash-Lite reports $$M_{i,j} = 1 - \frac{\mathbf{a}_i^\top \mathbf{b}_j}{\|\mathbf{a}_i\|_2\,\|\mathbf{b}_j\|_2}.$$7, and MiniCPM-o-2.6 reports $$M_{i,j} = 1 - \frac{\mathbf{a}_i^\top \mathbf{b}_j}{\|\mathbf{a}_i\|_2\,\|\mathbf{b}_j\|_2}.$$8. The best overall detection accuracy is reported as about $$M_{i,j} = 1 - \frac{\mathbf{a}_i^\top \mathbf{b}_j}{\|\mathbf{a}_i\|_2\,\|\mathbf{b}_j\|_2}.$$9 for Gemini-2.5-Flash-Lite, while localization F1 on S2 and S3 is typically below $A$0. The paper states that when other interlocutors’ turns are provided as text, performance degrades dramatically because models prioritize textual coherence over acoustic cues, failing to detect even obvious gender switches. Discrimination is much easier: GPT-4o achieves $A$1 discrimination accuracy, and speaker embedding baselines reach roughly $A$2–$A$3 average accuracy. This establishes a strong distinction between relative acoustic matching and absolute speaker-consistency judgment.

Speech generation work uses related protocols to quantify or operationalize speaker consistency. "Automatic Evaluation of Speaker Similarity" (Kamil et al., 2022) addresses speaker leakage in multi-speaker TTS by predicting MUSHRA speaker-similarity scores from speaker embeddings. The dataset contains 51 evaluation cycles, 13 target speakers, 354 systems, 18,493 examples, 788 listeners, and 730,308 individual scores. A human split-half upper bound gives Pearson correlation $A$4. The best automatic configuration—GE2E embeddings, an MLP regressor, a Mahalanobis distance-inspired loss, and training on individual scores—reaches Pearson correlation $A$5, accuracy $A$6, and RMSE $A$7. At sub-utterance level on 1-second segments, it reports Pearson $A$8 and accuracy $A$9. In this setting, the speaker-switch test becomes a continuous speaker-similarity proxy for detecting identity drift.

"Human-in-the-loop Speaker Adaptation for DNN-based Multi-speaker TTS" (Udagawa et al., 2022) turns speaker switching into an interface. The system plays a single utterance in a loop while a user switches among multiple candidate synthetic voices, with switching allowed for each phoneme. This supports sequential line search in a 16-dimensional speaker-embedding space over 30 SLS steps, with each adaptation taking about 15–25 minutes. The paper reports that the method can achieve comparable performance to conventional speaker adaptation in objective and subjective evaluations even when reference speech is not used as the input of a speaker encoder directly. Here, speaker switching is neither a classifier control nor a benchmark perturbation, but a perceptual comparison mechanism for navigating speaker-embedding space.

5. Source-speaker tracing and robustness under speaker changes

In source-speaker tracing, the switch is hidden rather than explicit. "Speaker Contrastive Learning for Source Speaker Tracing" (Wang et al., 2024) defines the problem as source speaker verification against voice conversion: given two converted speech samples, determine whether they originate from the same source speaker. If a source utterance $B$0 is converted to sound like target speaker $B$1, the result is denoted $B$2. The training scheme has three phases: Phase 1 trains on Librispeech train-clean only; Phase 2 fine-tunes on SSTC converted speech plus Librispeech source speech; Phase 3 trains only on converted speech using speaker AAM-Softmax loss and a speaker contrastive loss. With converted embedding $B$3, source-speaker embedding set $B$4, true source embedding $B$5, and temperature $B$6, the contrastive loss is

$B$7

and the final loss is

$B$8

with $B$9 and $A_1$0 negative distractors. On the SSTC test set, MFA-Conformer gives $A_1$1 EER, MFA-Conformer + adapter gives $A_1$2, ResNet293 gives $A_1$3, and ResNet293 + $A_1$4 gives $A_1$5, a $A_1$6 absolute EER reduction over ResNet293 and a $A_1$7 absolute improvement over the official baseline.

Automatic speech recognition research treats speaker switching as a robustness stressor for adaptation. "Speaker-Smoothed kNN Speaker Adaptation for End-to-End ASR" (Li et al., 2024) argues that fine-tuning can perform well when the adaptation and test speaker match, but can fail under speaker changes. Speaker-Smoothed kNN keeps the pretrained ASR model fixed, builds a datastore of token-level hidden states and labels, attaches utterance-level x-vectors to datastore entries, and uses speaker similarity to dynamically set a retrieval temperature

$A_1$8

and interpolation weight

$A_1$9

In the in-domain setting, the baseline gives single-speaker CER $\mathcal{C}_1=(A_1,B_1)$0 and multi-speaker CER $\mathcal{C}_1=(A_1,B_1)$1. Fine-tuning improves the single-speaker test to $\mathcal{C}_1=(A_1,B_1)$2 but degrades the multi-speaker test to $\mathcal{C}_1=(A_1,B_1)$3. Plain kNN gives $\mathcal{C}_1=(A_1,B_1)$4, while Speaker-Smoothed kNN gives $\mathcal{C}_1=(A_1,B_1)$5. In the all-domain setting, Speaker-Smoothed kNN reports $\mathcal{C}_1=(A_1,B_1)$6 on the single-speaker test and $\mathcal{C}_1=(A_1,B_1)$7 on the multi-speaker test, while Fine-tune + Ours gives $\mathcal{C}_1=(A_1,B_1)$8. The paper explicitly states that the method performs comparably to fine-tuning without the associated performance degradation during speaker changes.

Earlier adaptive ASR work operationalized related unseen-speaker conditions through cluster selection. "Speaker Cluster-Based Speaker Adaptive Training for Deep Neural Network Acoustic Modeling" (Chu et al., 2016) clusters training speakers with Ward’s hierarchical clustering using normalized i-vectors and an inner-product similarity,

$\mathcal{C}_1=(A_1,B_1)$9

then matches an unseen test speaker to the nearest cluster by

$B_1$0

With 10 speaker clusters and the first layer as the speaker-dependent layer, the method improves WER from $B_1$1 for the in-domain SI-DNN to $B_1$2 for the in-domain SAT-DNN, a relative $B_1$3 reduction. This is not called a speaker-switch test in the same sense as later work, but it addresses the same operational problem: model behavior when the speaker at test time differs from the speaker conditions seen during adaptation.

6. Related notions, analogues, and conceptual boundaries

Speaker-switch testing is closely related to classical speaker change detection, but the two are not identical. "Speaker Change Detection Using Features through A Neural Network Speaker Classifier" (Ge et al., 2017) studies real-time SCD by transforming speech into 200-dimensional likelihood vectors over in-domain speakers and comparing consecutive intervals with a distance metric. Using synthesized conversations from out-of-domain TIMIT speakers, it reports 100% file-level classification accuracy at a 200-speaker size given speech duration of at least 0.97 seconds. For change detection, the reported test-set results are $B_1$4 and $B_1$5 at 1 second, and $B_1$6 and $B_1$7 at 2 seconds, with the conclusion stating that it captures close to 97% of the changes by comparing the current second of speech with the previous second. This is boundary detection, not a control against leakage or a consistency benchmark, but it supplies a direct antecedent.

Work on spoken language change detection makes the comparison explicit. "Spoken language change detection inspired by speaker change detection" (Mishra et al., 2023) states that LCD and SCD are structurally similar boundary-finding tasks, yet humans require a larger duration around the change point and language-specific prior exposure for LCD as compared to SCD. Increasing the analysis window length in the unsupervised distance-based approach yields a relative performance improvement of 29.1% on the synthetic code-switched dataset and 2.4% on the practical code-switched dataset. With language-specific modeling, the x-vector detector reaches 87.01% IDR on the synthetic corpus and 52.59% IDR on the practical corpus. A plausible implication is that speaker-switch methodology generalizes to other change-point problems when the main question is how much local versus longer-range context is needed to disambiguate a boundary.

A further extension treats within-speaker condition changes as switch-like nuisance factors. "Emotion Invariant Speaker Embeddings for Speaker Identification with Emotional Speech" (Sarma et al., 2020) addresses emotional mismatch by learning a mapping from emotion-dependent i-vectors into an emotion-invariant space, reporting average accuracy of 87.9% for an averaged multi-emotion i-vector baseline, 90.1% for EINV-Test, and 90.5% for EINV-Pair, an absolute gain of 2.6%. "LASPA: Language Agnostic Speaker Disentanglement with Prefix-Tuned Cross-Attention" (Menon et al., 2 Jun 2025) addresses the case where the same speaker changes language, arguing that speaker embeddings entangle linguistic information. On VoxCeleb1-B, ResNet-L baseline EER is 5.96 and LASPA ResNet-L gives 2.38; on NISP-B, ECAPA baseline is 20.60 and LASPA ECAPA is 11.95. These are not speaker-switch tests in the narrow sense of replacing one speaker with another, but they are structurally related: the system must preserve speaker identity under a controlled change in the speech signal.

Taken together, these variants delimit the concept. In its narrowest sense, a speaker-switch test deliberately swaps or withholds speaker identity to create a negative control or a consistency challenge. In a broader sense, it is any protocol that asks whether a model’s representation survives a controlled change in speaker continuity, speaker overlap, or speaker-conditioned nuisance variation. The literature therefore treats speaker switching not as a single benchmark, but as a methodological device for separating genuine task structure from speaker-specific confounds.