SpeakerCard-1M: An Evidence-Grounded Speaker Card Corpus for In-the-Wild Speaker Verification

Abstract: Modern speaker verification (SV) systems rely on speaker embeddings that are effective but difficult to interpret or query in natural language. Most existing speech-text corpora target controllable synthesis or utterance-level captioning, and provide limited speaker-level supervision for in-the-wild speaker recognition. This paper introduces SpeakerCard-1M, a bilingual speaker-centric resource for evidence-grounded SV, derived from VoxCeleb1/2 and CN-Celeb1/2, where the "-1M" suffix refers to the 1.78M utterance-level captions contained in the release. We adopt a tool-first, LLM-last approach: ten acoustic probes produce field-level evidence, the evidence is aggregated into speaker profiles under a schema that separates relatively stable traits from utterance-level states, and bilingual Speaker Cards are rendered by a constrained LLM that sees only the structured fields. The release includes 56.7K Speaker Card records over 10.2K speakers, 1.78M utterance-level captions, and speaker-ID-disjoint hard-negative triplets. We further define two SV-oriented cross-modal protocols, bidirectional Speaker-Text Retrieval (T2S-R / S2T-R) and Attribute-Conditioned Verification (AC-Verify), and compare a dual-encoder baseline against recent audio LLMs under a zero-shot forced-choice setting. Joint audio-text training increases VoxCeleb1-O EER by 0.31% absolute over the audio-only baseline. Under a style-symmetric LLM-generated counterfactual protocol, eight recent audio LLMs (7B-30B+ parameters, both open- and closed-source) score 49-77% on pitch-level AC-Verify under two-way forced choice, compared with 88.66% reached by our dual encoder.

• The paper introduces SpeakerCard-1M, a large-scale bilingual corpus for in-the-wild speaker verification that employs explicit trait/state separation and evidence aggregation.
• It details a robust construction pipeline using acoustic probing, confidence-weighted voting, and bilingual LLM-based verbalization for generating diversified speaker cards.
• Empirical results demonstrate minimal speaker verification degradation in joint audio-text training while highlighting current audio LLMs' limitations in fine-grained trait discrimination, particularly for pitch.

Evidence-Grounded Speaker Cards for In-the-Wild Speaker Verification: SpeakerCard-1M

Motivation and Background

Interpretability and searchable supervision are significant bottlenecks in current speaker verification (SV) systems: speaker embeddings achieve high discrimination performance, but do not allow element-level inspection or natural language querying for the presence, absence, or contradiction of specific speaker or utterance properties. Existing datasets and retrieval systems target controllable synthesis or utterance-level captioning rather than speaker-level descriptive grounding, and most lack explicit, field-level evidence for speaker traits and utterance states.

SpeakerCard-1M (2606.03283) directly addresses these deficiencies by providing an evidence-grounded, bilingual (EN/zh) corpus derived from VoxCeleb1/2 and CN-Celeb1/2, focusing on verifiable, structured, and queryable supervision at both the speaker and utterance level. The corpus enforces trait/state separation and retains explicit provenance from a suite of acoustic probes, supporting both interpretability and robust cross-modal evaluation.

Construction Pipeline

The corpus construction adheres to a pipeline guided by a tool-first, LLM-last principle that ensures faithful extraction and aggregation of evidence, and delegates only structured verbalization to LLMs. The stages are:

1. Ingestion of utterances from VoxCeleb1/2 and CN-Celeb1/2, with filtering to ensure sufficient speaker-level data.
2. Acoustic probing with ten off-the-shelf systems to extract six speaker traits (gender, age, timbre, pitch, accent, language ID) and four state-level attributes (emotion, channel, speaking rate, environment).
3. Aggregation of trait fields by confidence-weighted voting, with state fields retained per-utterance.
4. Speaker Card verbalization via a constrained LLM, generating four caption styles (detailed, identity_only, technical_report, short_query) in both English and Chinese, with three paraphrase variants per (speaker, language) tuple.
5. Quality control including malformed-output detection, MinHash-LSH near-duplicate filtering, trait/state leakage checks, and entailment validation with NLI models, producing a high-confidence release.

   Figure 1: The SpeakerCard-1M pipeline: evidence ingestion, trait/state probing, aggregation, bilingual LLM verbalization, and rigorous quality control.

A total of 56.7K SpeakerCard records (covering 10.2K speakers) and 1.78M utterance-level captions are released, augmented with explicit provenance metadata and hard-negative triplets for contrastive training.

Technical Contributions

Trait-State Schema and Evidence Aggregation

A key innovation is the programmatic enforcement of trait/state separation: stable (identity) traits are aggregated over utterances via confidence-weighted voting, while highly variable session-level states are preserved as utterance-local. Field-level metadata, including per-probe confidence, native model ID, and per-field stability, are maintained throughout. This explicit schema prevents both uncontrolled hallucination by the LLM and leakage of ephemeral states into identity supervision.

Bilingual, Style-Diverse Caption Generation

Speaker Cards are generated in four styles targeting different retrieval, verification, and query needs:

• detailed: combines identity traits with representative states
• identity_only: speaker traits only
• technical_report: structured, numeric, fieldwise summary
• short_query: succinct description for retrieval

Generation is performed independently for EN and ZH, with three paraphrase variants per (speaker, language), supporting robust training and evaluation under textual variation.

Protocols for Evidence-Grounded Evaluation

Two protocols—bidirectional speaker–text retrieval (T2S-R, S2T-R) and attribute-conditioned verification (AC-Verify)—are introduced for systematic, cross-modal benchmarking:

• T2S-R / S2T-R: Measures retrieval when querying with either text or audio anchor in a gallery of speakers.
• AC-Verify: 2-way forced choice for matching audio to a SpeakerCard vs. a single-trait-contradicted or hard-negative card; counterfactuals are generated via minimally-edited LLM scripts ensuring style symmetry.

Baseline Systems and Empirical Results

The primary baseline is a dual-encoder system: a WavLM-Base audio tower, frozen except for the final projection head, paired with a BGE-M3 text tower, trained with combinations of InfoNCE and ArcFace (AAM-Softmax) loss. The encoders map audio and text into a 256D shared space.

• Speaker verification performance: EER on VoxCeleb1-O/E/H is 1.07%/0.91%/2.07% for the balanced dual-task system (vs. 0.76%/0.79%/1.58% for the audio-only baseline), indicating just a 0.31% worst-case absolute degradation from joint audio–text training.
• Retrieval: T2S@10 and S2T@10 in the primary setting reach 24.8% and 25.5%, respectively.
• Probe→LLM cascade: Control system (probe outputs verbalized via LLM, retrieval in text-only space) performs notably worse, especially for S2T retrieval (9.3% vs. 25.5%), demonstrating that audio embeddings encode speaker aggregation information not recoverable from single-utterance probe verbiage alone.
• Attribute-Conditioned Verification: On LLM-generated counterfactuals, the dual encoder achieves 93.84% aggregate CF accuracy, with 88.66% on pitch counterfactuals. The strongest open/closed-source Audio LLMs (7B–30B+ parameters) still fail to reliably reject pitch-level counterfactuals, topping out at 76.99%. Thus, fine-grained acoustic trait reasoning remains a systematic deficiency in current LLMs.

Experimental Insights

• Schema enforcement is crucial: Removing explicit trait–state schema enforcement produces a 6.9% drop in AC-Verify CF performance and a 4.5% hit to Hard trials, confirming the importance of programmatic attribute partitioning for robust evidence-grounded discrimination.
• Tradeoff between SV and retrieval: Retrieval-optimized checkpoints increase retrieval metrics at a cost to SV and fine-grained trait discrimination, particularly for pitch.
• Bilingual training effects: Cross-lingual transfer is asymmetric; English-to-Chinese transfer is stronger due to greater coverage in English, while bilingual training lifts Chinese performance without harming English results. AC-Verify attribute-level generalization is language-specific.
• Trait hierarchy in LLMs: Current audio LLMs achieve high accuracy for gender (94–98%), moderate for accent/age, but all underperform on pitch (49.2–76.99%); Qwen2-Audio is at chance for all traits.

Implications and Theoretical Significance

SpeakerCard-1M provides a high-signal, schema-enforced resource for advancing evidence-grounded SV, challenging current audio-capable LLMs with cross-modal, fine-grained, and attribute-contradicting tests unattainable with previous datasets. The explicit trait/state separation and per-field evidence provenance set a new standard for verifiable, interpretable SV supervision. For SV foundation models, the work illuminates both the promise of joint audio–text training (minimal SV loss, enhanced interpretability) and the limitations of current LLMs (consistent failures on high-frequency acoustic attributes and fine clausal contradiction).

Practically, the resource enables the development and robust evaluation of systems that must respond to natural language queries about speaker characteristics under real-world acoustic variation. Theoretically, it motivates further work on explicit evidence tracing, schema enforcement, and discriminative learning for attributes that challenge large pretrained models' inductive biases.

Prospects for Future Research

• Human-in-the-loop and crowdsourced trait auditing: Complementing probe-bound trait assignment with curated human ratings, especially for pitch, timbre, and accent.
• Richer cross-lingual and code-switched speaker cards: Leveraging the bilingual architecture for multi-lingual, multi-modal systems.
• Tool-augmented and hybrid inference: Integrating acoustic tools at inference-time for LLMs, akin to recent audio Chain-of-Thought modeling, to improve explainability and attribute discrimination.
• Structured, citation-aware captioning: Generating SpeakerCards with explicit field-to-evidence linking, perhaps via LLMs equipped with tool use or retrieval-augmented generation.
• Further exploration of schema-agnostic and schema-enforced regimes: To understand their impact on robustness, generalization, and security (e.g., adversarial state leakage).

Conclusion

SpeakerCard-1M systematically advances the state of evidence-grounded speaker verification by releasing a large-scale, bilingual, tool-extracted and schema-validated corpus of speaker cards. Joint audio–text training achieves only minor degradation in traditional SV metrics while enabling powerful cross-modal and attribute-level evaluation. AC-Verify exposes consistent deficits in pitch-discrimination for Audio LLMs, with the dual encoder surpassing all others. The resource will inform further research into interpretable, robust, and verifiable speaker verification and holds promise for a new generation of evidence-traceable, attribute-discriminative foundation models for speech.