SpeakerCard-1M: Bilingual Evidence-Grounded SV Corpus

• The paper introduces SpeakerCard-1M as a bilingual speaker verification corpus with 1.78M utterance-level captions that adds a structured textual evidence layer to in-the-wild data.
• It employs a tool-first, LLM-last pipeline that leverages acoustic probes and schema-driven prompts to clearly separate stable speaker traits from dynamic utterance states.
• The resource supports natural-language speaker search and cross-modal retrieval, offering robust evaluation protocols for attribute-sensitive verification.

SpeakerCard-1M is a bilingual, speaker-centric annotation corpus for evidence-grounded speaker verification (SV), introduced to make SV systems more interpretable and queryable in natural language while remaining grounded in actual acoustic evidence. It is derived from VoxCeleb1, VoxCeleb2, CN-Celeb1, and CN-Celeb2, and the “-1M” suffix refers specifically to the 1.78M utterance-level captions contained in the release rather than to the number of speakers or cards. The resource adds a structured textual layer to in-the-wild SV data, with field-level provenance, a schema that separates relatively stable speaker traits from utterance-level states, bilingual Speaker Cards rendered independently in English and Chinese, and evaluation protocols designed for cross-modal retrieval and attribute-sensitive verification (Peng et al., 2 Jun 2026).

1. Conceptual motivation and problem setting

Modern SV systems rely on speaker embeddings that support same/different similarity scoring but are difficult to interpret or query in natural language. The motivating gap is that existing SV corpora provide speaker IDs but no language layer, while most existing speech-text corpora focus on controllable synthesis, expressive speech captioning, or utterance-level descriptions rather than speaker-level supervision for in-the-wild speaker recognition (Peng et al., 2 Jun 2026).

The corpus is framed around “evidence-grounded SV.” In this formulation, the objective is not to replace speaker embeddings with text, but to add a language-accessible evidence layer tied to structured acoustic probes. This design is intended to support tasks such as natural-language speaker search and attribute-sensitive matching while preserving traceability to measurable acoustic evidence.

A central principle is the separation between relatively stable speaker traits and utterance-level states. Stable traits are properties intended to support speaker-level identity descriptions, including gender, age, timbre, pitch band, accent, and language-related characteristics. States are session- or utterance-dependent conditions such as emotion, channel, speaking rate, and environment. The paper emphasizes that this distinction is enforced programmatically in the schema and prompts rather than being left to a LLM to infer. This separation is presented as one of the main points of novelty relative to prior speaker profiling and speaker-text retrieval work.

2. Source corpora, filtering, and data representation

SpeakerCard-1M is built from four established in-the-wild SV datasets: VoxCeleb1, VoxCeleb2, CN-Celeb1, and CN-Celeb2. Audio from these sources is normalized into a unified utterance manifest with fields

$$\{\texttt{utt\_id}, \texttt{speaker\_id}, \texttt{corpus}, \texttt{language\_prior}, \texttt{wav\_path}, \texttt{duration}, \texttt{sample\_rate}\}.$$

Corrupted files are removed during ingestion. For speaker-level aggregation, speakers with fewer than three valid utterances or less than 30 seconds of cumulative speech are excluded from speaker-profile construction, although their utterances can still be retained for utterance-level captioning (Peng et al., 2 Jun 2026).

After filtering, the unified manifest contains 1,783,791 utterances from 10,188 speakers, totaling 4,069.1 hours. The release includes the annotation layer rather than raw audio, because it inherits source-corpus licensing terms. The released components comprise cards, probe evidence, splits, triplets, and protocols.

The paper gives a precise account of the retained corpus statistics:

Component	Count
Speakers after filtering	10,188
Utterances with captions	1,783,791
Total audio hours	4,069.1
Candidate card records	61,128
Post-QC card records	56,692
English card records	28,254
Chinese card records	28,438
Caption fields per card	4
Total caption fields	226,768
Speaker-disjoint training triplets	189,201

The train/validation/test split is speaker-ID-disjoint at the source-corpus level. The paper also notes that post-hoc identity deduplication against external overlap lists was not performed, so duplicate real-world identities across different VoxCeleb IDs may still exist.

3. Tool-first, LLM-last construction pipeline

The construction pipeline is explicitly “tool-first, LLM-last.” Perception of acoustic properties is delegated to specialized automatic probes, and the LLM is used only at the final verbalization stage. The paper states that the LLM never sees the raw audio and is not trusted to infer traits directly from speech; instead, it verbalizes already-structured evidence (Peng et al., 2 Jun 2026).

Ten acoustic probe families are applied to each utterance, covering six trait dimensions and four state dimensions. The six trait dimensions are gender, age, timbre, pitch, accent, and language ID. The four state dimensions are emotion, channel, speaking rate, and environment. The probe setup is field-specific:

• Gender and age: from a speech-based age/gender model.
• Language ID: from a multilingual model.
• Speaking rate: from a multilingual estimator.
• Channel and environment: derived from language-independent acoustic extractors.
• Timbre and emotion: from English-trained voice-quality and SER models applied cross-lingually.
• Accent: from separate English and Chinese accent classifiers, selected by source corpus.

Pitch is treated specially because it is central to later evaluation. It is estimated by consensus across three independent $F0$ systems: Praat autocorrelation via Parselmouth, torchcrepe with confidence threshold $0.5$, and RMVPE. All are restricted to the $75$–$500$ Hz range. For each utterance, each system produces a median $F0$; these are combined into a “canonical $F0$” by median-of-medians, and then binned into five pitch levels.

Probe outputs are stored as structured evidence rather than only final labels. For each utterance and field, the record includes the predicted label or continuous value, the probe’s native per-utterance confidence, the model identifier and revision, and the source utterance ID. The paper repeatedly stresses “field-level provenance”: traits in a final Speaker Card remain traceable to the underlying utterance-level probe outputs.

The release reports average per-utterance probe confidence by field. Language ID is reported at $0.99$, gender at $0.96$, accent/pitch/environment/channel around $0.75$–$F0$0, and age/timbre/emotion at $F0$1–$F0$2. The authors explicitly warn that these confidence values are meaningful only within a field and are not calibrated across different probe types.

4. Trait-state schema, aggregation, and bilingual Speaker Cards

Speaker-level aggregation is applied only to trait fields. For a given speaker, categorical traits are aggregated by confidence-weighted voting. Continuous or ordinal traits, such as pitch, are summarized with robust statistics such as median and MAD, and discretized into bins only when needed for verbalization. State evidence is not aggregated into a speaker identity profile; it remains attached to individual utterances as contextual metadata (Peng et al., 2 Jun 2026).

The paper defines an intra-speaker self-consistency measure $F0$3 for speaker $F0$4 and field $F0$5. For a single-label categorical field, $F0$6 is the fraction of utterances whose per-utterance prediction matches the aggregated speaker-level label. For continuous fields, an analogous coefficient-of-variation-style statistic is used. The authors state that $F0$7 measures self-consistency rather than ground-truth accuracy. It is used as a stability indicator. For example, a speaker is flagged as low_confidence when gender stability falls below $F0$8, i.e.

$F0$9

Only $0.5$0 of speakers meet this low-confidence condition, and the median self-consistency for the other categorical trait fields is $0.5$1.

The paper also reports an external audit on 200 speakers sampled from the English AC-Verify gallery. On that subset, the probes achieve $0.5$2 gender accuracy and $0.5$3 accuracy on 4-way age-band classification, with $0.5$4 chance level. For pitch, the authors rely on the three-system consensus and release all three per-utterance $0.5$5 estimates so that users can inspect disagreements; a human-labeled pitch audit is deferred to future work.

Speaker Cards are generated only after aggregation. The LLM receives a serialized schema containing structured trait fields, confidence and stability metadata, and, when needed, a summarized utterance-state context. Prompting enforces which fields may appear in which style and how uncertain fields should be hedged. The paper states explicitly that the “hedging and trait/state inclusion rules” are enforced by the prompt schema rather than left to the model.

Each Speaker Card record contains four textual styles:

1. detailed: combines speaker traits with a representative utterance-state context.
2. identity_only: contains only trait fields and excludes emotion, channel, environment, and speaking rate.
3. technical_report: a semi-structured rendering that includes numerical estimates and per-field confidence.
4. short_query: a concise retrieval-oriented description.

A “Speaker Card record” is defined as one speaker-language-paraphrase item, and each record contains all four styles. English and Chinese cards are generated independently from the same schema using separate EN and ZH prompts rather than by translating one language into the other. For each $0.5$6 pair, Qwen2.5-72B-Instruct is called three times at temperature $0.5$7 with deterministic per-call seeds, and in each call the model jointly emits all four caption styles as a single JSON object. Thus each speaker has $0.5$8 languages $0.5$9 $75$0 paraphrase variants $75$1 records before quality control, each with $75$2 caption fields.

5. Quality control, triplets, and benchmark protocols

The quality-control pipeline has four stages: malformed-output detection; near-duplicate filtering using MinHash-LSH over character 5-grams with Jaccard threshold $75$3, $75$4 hashes, and $75$5 bands; state-leakage checking to ensure identity_only cards do not mention forbidden state terms; and evidence consistency scoring with the multilingual NLI model MoritzLaurer/mDeBERTa-v3-base-mnli-xnli (Peng et al., 2 Jun 2026).

Cards with entailment below $75$6 are rejected, and those in $75$7 are retained but flagged. Retained cards average $75$8 entailment with their structured premises, while rejected cards have median entailment $75$9 and flagged cards $500$0. The near-duplicate filter removed $500$1 cross-speaker duplicates in the release. Out of $500$2 candidate records, $500$3 are retained, yielding a $500$4 acceptance rate. QC is applied at the record level: either all four styles for a speaker-language-paraphrase record are kept, or the entire record is rejected.

The release also includes speaker-ID-disjoint triplets for contrastive training. Negatives are mined in two types. Easy negatives differ in coarse attributes. Hard negatives share coarse traits such as gender, age band, and accent, but differ in finer traits such as pitch band and timbre, with less restrictive fallbacks if needed. This hard-negative construction is used later in both retrieval training and attribute-conditioned verification.

Three evaluation protocols are defined.

First, standard speaker verification uses cosine scoring between enrollment and test embeddings on VoxCeleb1-O, E, and H, with EER reported.

Second, bidirectional Speaker-Text Retrieval comprises two directions: text query to speaker/audio gallery (T2S-R) and speaker/audio query to text gallery (S2T-R). A held-out gallery of size $500$5 is built from the test split, with $500$6 for English on VoxCeleb and $500$7 for the Chinese clean split on CN-Celeb. Each gallery speaker has an audio anchor defined as the mean embedding of three deterministic enrollment utterances and a text anchor defined as the mean embedding of the speaker’s three identity_only paraphrase variants in the query language. Metrics are Recall@1, Recall@5, and Recall@10.

Third, Attribute-Conditioned Verification (AC-Verify) is a 2-way forced-choice task in which the system must choose the better matching card for a given audio sample. It has two variants. In CF, the positive is the speaker’s true identity_only card and the distractor is a counterfactual rewrite with exactly one trait changed: gender, age range, accent, or pitch level. Counterfactual cards are generated offline by Qwen3-32B-Instruct at temperature $500$8 under a minimal-edit prompt that preserves every unflipped trait phrase verbatim, yielding a “style-symmetric” protocol. In Hard, the distractor is a speaker-ID-disjoint hard-negative card mined to share coarse traits while differing in finer traits.

6. Modeling setup and empirical findings

The main baseline is a dual encoder with an audio tower and a text tower mapped into a shared 256-dimensional space. The audio tower is WavLM-Base with MHFA pooling, initialized from a WeSpeaker SV checkpoint. The text tower is BGE-M3 with attention-masked mean pooling. The base encoders are frozen; only MHFA and projection heads on the audio side and the projection head on the text side are trained (Peng et al., 2 Jun 2026).

A cascade baseline is also defined. It has no learned cross-modal audio tower. Instead, it applies the same probes and the same LLM verbalization pipeline to the query audio, produces identity_only text, and matches that text to speaker-level card text in the BGE-M3 text embedding space. The paper presents this as an architecture-matched control for testing whether the learned audio tower contributes more than probe-then-text conversion alone.

The training regimes are threefold. The audio-only reference trains only the audio tower with AAM-Softmax:

$500$9

The main dual-task “balanced” model is trained on identity_only cards with one positive text, one easy negative, and one hard negative per anchor. Its loss is

$F0$0

with AAM warmup for 1,000 steps. Training uses speaker-balanced batches of $F0$1 speakers $F0$2 $F0$3 utterances per GPU, totaling $F0$4 audio and $F0$5 text items across $F0$6 GPUs, for $F0$7 steps, with MUSAN and RIR augmentation at $F0$8.

A retrieval-specialized model is initialized from a retrieval-push checkpoint and further trained for 3,000 steps using technical_report as the main positive, plus an additional template-rendered positive and template-normalized negatives. It adds a hard-margin ranking term:

$F0$9

with ranking margin $F0$0, weight $F0$1, start step $F0$2, and warmup $F0$3 steps.

On VoxCeleb1-O/E/H, the reported EER results are:

• ECAPA-TDNN: $F0$4
• WavLM-Base SV: $F0$5
• Ours (audio only): $F0$6
• Ours (balanced): $F0$7
• Ours (retrieval-specialized): $F0$8

The paper interprets the increase from $F0$9 to $0.99$0 on VoxCeleb1-O as a $0.99$1 absolute degradation associated with joint audio-text training.

On the 1K-speaker English retrieval benchmark, the cascade baseline yields T2S-R values of $0.99$2 and S2T-R values of $0.99$3. The balanced dual encoder yields T2S-R values of $0.99$4 and S2T-R values of $0.99$5. The retrieval-specialized model reaches $0.99$6 on T2S-R and $0.99$7 on S2T-R. The particularly large cascade-versus-dual gap on S2T-R@10, $0.99$8 versus $0.99$9, is used to argue that learned audio embeddings capture speaker-level aggregation information that is not fully recovered by probing a single utterance and verbalizing it.

On AC-Verify in the Vox-only setting, the balanced model scores $0.96$0 on CF and $0.96$1 on Hard, while the retrieval-specialized model scores $0.96$2 on CF and $0.96$3 on Hard. The paper highlights a trade-off: retrieval optimization improves retrieval metrics while harming attribute-sensitive verification, especially on pitch.

Against eight recent audio LLMs evaluated zero-shot under the style-symmetric counterfactual protocol, the dual encoder achieves per-trait AC-Verify results of Gender $0.96$4, Accent $0.96$5, Age $0.96$6, Pitch $0.96$7, Aggregate CF $0.96$8, and Hard $0.96$9. For pitch, the compared models range from $0.75$0 to $0.75$1: Audio Flamingo 3 ($0.75$2), Qwen2-Audio-7B-Instruct ($0.75$3), Qwen3-Omni-30B-A3B-Instruct ($0.75$4), MiMo-Audio-7B-Instruct ($0.75$5), Kimi-Audio-7B-Instruct ($0.75$6), Gemini 2.5 Flash ($0.75$7), Gemini 3.5 Flash ($0.75$8), and GPT audio mini ($0.75$9). The dual encoder’s $F0$00 on pitch exceeds the strongest listed audio LLM by $F0$01 absolute points.

Text-view ablations show that the balanced checkpoint performs best on retrieval when evaluated with identity_only, which matches training: T2S@10 $F0$02 and S2T@10 $F0$03, versus detailed at $F0$04 and $F0$05, technical_report at $F0$06 and $F0$07, and short_query at $F0$08 and $F0$09. A more consequential ablation removes schema enforcement by training on detailed cards instead of identity_only while keeping hyperparameters and seed fixed. This reduces T2S@10 from $F0$10 to $F0$11, S2T@10 from $F0$12 to $F0$13, CF from $F0$14 to $F0$15, and Hard from $F0$16 to $F0$17. The paper presents this as a clear empirical validation of trait-state separation.

7. Bilingual behavior, limitations, and research significance

The bilingual and cross-lingual experiments use three training settings: EN-only, ZH-only, and bilingual. Evaluation is carried out on the 1K-speaker English VoxCeleb gallery and a 144-speaker clean Chinese CN-Celeb gallery (Peng et al., 2 Jun 2026).

The reported results are asymmetric. EN-only transfers reasonably to Chinese retrieval, with EN $F0$18 ZH T2S@10 $F0$19, S2T@10 $F0$20, CF $F0$21, and Hard $F0$22. ZH-only transfers poorly to English, with ZH $F0$23 EN T2S@10 $F0$24, S2T@10 $F0$25, CF $F0$26, and Hard $F0$27. In-language performance is higher: EN $F0$28 EN gives T2S@10 $F0$29, S2T@10 $F0$30, CF $F0$31, and Hard $F0$32, while ZH $F0$33 ZH gives T2S@10 $F0$34, S2T@10 $F0$35, CF $F0$36, and Hard $F0$37. Bilingual training preserves English fairly well and substantially improves Chinese: Bilingual $F0$38 EN gives T2S@10 $F0$39, S2T@10 $F0$40, CF $F0$41, and Hard $F0$42, while Bilingual $F0$43 ZH gives T2S@10 $F0$44, S2T@10 $F0$45, CF $F0$46, and Hard $F0$47.

The paper concludes that coarse cross-modal matching transfers better than fine attribute reasoning, since AC-Verify CF falls substantially on unseen-language tests relative to in-language evaluation. A plausible implication is that cross-lingual alignment is more robust for speaker-level matching than for fine-grained contradiction detection.

The limitations are stated explicitly. Annotation quality is probe-bounded: gender, language ID, environment, and channel are relatively reliable, whereas accent, timbre, and emotion are noisier in in-the-wild data, especially when English-trained models are applied cross-lingually. The cards are therefore not to be treated as gold-standard human labels. Confidence and stability metadata are included so that users can filter or reweight uncertain attributes. A larger human pitch audit is still missing. Split disjointness is only at the source speaker-ID level, external identity overlap across datasets was not removed, and hard-negative difficulty is not calibrated with human judgments. Language coverage is limited to English and Chinese.

The intended-use guidance is equally explicit. The authors state that the technical_report style is for interpretability research rather than forensic speaker characterization; they do not endorse the corpus for identification, surveillance, or voice cloning; and they release annotations rather than audio. Within those limits, SpeakerCard-1M functions as a large-scale testbed for natural-language speaker querying, evidence-grounded cross-modal retrieval, and attribute-conditioned verification. Its principal contribution is not only scale—10,188 speakers, 56,692 post-QC Speaker Card records, 1,783,791 utterance captions, and 189,201 speaker-disjoint training triplets—but also the combination of probe-derived evidence, provenance-preserving storage, schema-constrained generation, trait-state separation, and LLM verbalization rather than LLM inference.