Audio Private Attribute Profiling

• Audio private attribute profiling is the process by which privacy-sensitive traits such as speaker identity, demographics, emotional state, and context are inferred from audio representations.
• Leakage mechanisms involve the reuse of features like MFCCs and spectrograms, where even after speech content is masked, vital paralinguistic cues remain exposed.
• Defensive approaches include privacy-preserving feature extraction and adaptive task-specific methodologies, evaluated using metrics like the Speaker Information Leakage Index to balance utility and privacy.

Searching arXiv for the papers on arXiv and closely related work to ground the article in current literature. arXiv search: "audio private attribute profiling voice privacy attribute-based perspective HearSay FeatureSense" Audio private attribute profiling denotes the intentional or unintentional inference of privacy-sensitive attributes about a person or environment from captured audio beyond the explicit sensing task. In recent work, the phenomenon is treated as a multi-dimensional privacy risk spanning privacy-preserving audio classification, always-on sensing, commercial voice assistants, voice anonymization, and Audio LLMs (ALLMs): acoustic representations can expose speech content, speaker identity, demographics, health status, emotional state, location, activities, habits, and socio-economic attributes, including settings in which semantic content is removed and inference proceeds from voiceprints alone (Chhaglani et al., 2024, Wang et al., 7 Jan 2026, Chhaglani et al., 30 May 2025).

1. Conceptual scope and attribute space

The literature treats audio as a rich biometric and contextual signal. One line of work enumerates private or sensitive attributes inferable from audio as identity and biometrics; demographics and social attributes such as gender, age, and geographical origin or accent; health status including respiratory problems, sleep apnea, Parkinson’s disease, intoxication, level of sleepiness, and breathing patterns; emotional and mental state such as stress, mood, anger, sadness, and joy; location and environment such as home versus office or public place, room type, and proximity to televisions, traffic, or appliances; activities and lifestyle; personal preferences and habits; and other contextual cues such as who else is nearby, group gender composition, and number of people (Chhaglani et al., 2024).

A recurrent conceptual split distinguishes three privacy dimensions: content privacy, speaker privacy, and context privacy. Content privacy concerns speech intelligibility and explicit semantic content. Speaker privacy concerns identity, demographics, and vocal traits. Context privacy concerns location, environment, co-located people, and activities. A central claim in this literature is that privacy-preserving audio systems cannot be defined solely by the removal of intelligible words, because speaker and context attributes remain inferable even when content is suppressed (Chhaglani et al., 2024).

Benchmark design has made this attribute space explicit. HearSay evaluates eight private attributes—Age, Gender, Health Status, Weight / BMI category, Accent, Education level, Income level, and Social Stratum—and defines profiling as inference from voiceprints alone, using acoustic and paralinguistic properties such as pitch, timbre, resonance, rhythm, prosody, and vocal quality rather than semantic cues (Wang et al., 7 Jan 2026). AP$^2$ broadens the scope to twelve attributes—AGE, GEN, ACC, HEA, HAB, PER, SOP, SOS, INC, EDU, OCC, and MAR—covering bio-demographic, health, psychological, behavioral, socio-economic, and life-status variables (Wang et al., 14 Jul 2025). In deployed commercial ecosystems, profiling labels exposed by data-access tools include relationship status, household income, education, homeownership, parenting, and ad-interest categories such as Fashion, Electronics, and Video Entertainment (Khezresmaeilzadeh et al., 2024).

2. Leakage mechanisms and threat models

The basic leakage mechanism is representational reuse. Conventional front-end features such as MFCCs, STFT-based spectrograms, and Mel spectrograms encapsulate detailed information about the audio source and can be repurposed to infer human speech and speaker identity. In speech and speaker pipelines, these representations retain formant structure, linked to speech content, and fundamental frequency, linked to speaker traits; in acoustic scene classification they encode detailed background noise that reveals environment type and possibly location or household patterns; in health monitoring they preserve cues from cough, breathing, and vocal production that correlate with respiratory diseases and Parkinson’s disease (Chhaglani et al., 2024).

The threat models described across the literature are broad. Application providers and cloud services may upload, log, and reuse spectrograms or MFCCs for secondary analyses. Third-party data analysts or advertisers may run later-stage inference on stored feature vectors. Eavesdroppers or attackers who gain access to features or model inputs may train their own profiling models. ALLM evaluations add a black-box API attacker who submits audio and prompts and extracts sensitive attributes zero-shot; commercial assistant audits add a first-party platform adversary that uses voice interactions to assign and update persistent profile labels (Wang et al., 7 Jan 2026, Khezresmaeilzadeh et al., 2024).

A useful formal lens models audio private attribute profiling as attribute inference from a representation $Z=f(X)$ extracted from waveform $X$. In that view, a representation is privacy-leaky when it retains high information about a hidden attribute $A$, such as speaker identity, gender, age, location, or emotional state. The same literature associates such leakage with model inversion or reconstruction, re-identification across datasets, and secondary-use profiling. This suggests that privacy risk arises not only from raw audio retention but from any intermediate representation rich enough to preserve speaker- or context-specific structure.

Work on always-on sensing sharpens this distinction by showing that masking speech content is insufficient. Age, gender, and ethnicity remain inferable after low-pass filtering, subsampling, spectral downsampling, and speech-segment removal, because pitch, formants, spectral envelope, MFCCs, timbre, and combinations of apparently weaker features still encode speaker-specific information (Chhaglani et al., 30 May 2025). Low-frequency speech studies make a related point from the verbal-privacy side: reducing bandwidth can obstruct lexical recovery, yet nonverbal attributes such as gender and other paralinguistic cues are not the main target of protection and may remain accessible (Liu et al., 2024).

3. Feature design and profiling-oriented methodologies

One major methodological response treats privacy as a front-end feature design problem. A representative pipeline for ESC-50 uses a sliding window of 500 ms with 100 ms overlap, silence removal with a top decibel threshold of 20 dB, and extraction of Zero Crossing Rate, Harmonic-to-Noise Ratio, Spectral Contrast, peak amplitude, RMS, Energy, Spectral Roll-off, Spectral Flatness, Spectral Bandwidth, and Spectral Centroid. The feature set explicitly excludes fundamental frequency and formants because they are strongly tied to speaker pitch, vocal tract resonances, speech content, and identity; MFCCs and related speech-specific representations are likewise excluded from the privacy-preserving design rationale (Chhaglani et al., 2024).

These descriptors are hand-crafted, low-to-mid-level summaries of texture, energy distribution, and noise-versus-tone structure rather than rich time-frequency encodings. The intended effect is to preserve “what type of sound is this?” while discarding the fine-grained harmonic and temporal detail needed for speaker recognition, speech reconstruction, or detailed voiceprint profiling. On ESC-50, a Random Forest trained on such feature vectors reaches a test accuracy of 92.23%, with spectral contrast, ZCR, and HNR identified as the most important features by Gini importance (Chhaglani et al., 2024).

FeatureSense systematizes this direction at the library level. It provides a curated open-source feature set integrated at the feature-extraction layer of the audio stack and introduces a three-part privacy evaluation framework: speech leakage, speaker attribute leakage, and feature-level leakage analysis. The library groups features into time-domain, spectral, phonetic or linguistic, statistical, perceptual, high-level time-frequency, voice-specific, and derived categories, while explicitly excluding MFCCs, formants, spectral envelope, LPC/LPCC, filterbank energies, $F_0$, timbre, and Chroma features from the default privacy-aware set (Chhaglani et al., 30 May 2025).

FeatureSense also introduces an adaptive task-specific feature selection algorithm that optimizes the privacy-utility-cost trade-off. For each feature $i$, utility $u_i$ is derived from task importance, privacy leakage $p_i$ from age, gender, and ethnicity classifiers, and cost $t_i$ from extraction latency. Feature selection is then formulated as

$$\max_x \sum_{i=1}^{n} \left(\alpha u_i - (1-\alpha)p_i\right)x_i \quad \text{s.t.} \quad \sum_{i=1}^{n} t_i x_i \le T,\;\; x_i \in \{0,1\},$$

where $Z=f(X)$0 controls the privacy-utility emphasis and $Z=f(X)$1 bounds latency (Chhaglani et al., 30 May 2025).

4. Empirical evidence of profiling capability

The strongest direct evidence comes from benchmarked ALLMs. HearSay evaluates 13 models and defines three metrics: Inference Accuracy Rate (IAR), Answer Refusal Rate (ARR), and Blind Bias Rate (BBR). IAR is the fraction of correct, non-refusal profiling decisions,

$Z=f(X)$2

and ARR is the fraction of refusals,

$Z=f(X)$3

Across open-source models, Gender reaches an average IAR of 92.89% with ARR approximately 0.62%; several models exceed 95%. Multi-class Age reaches 50.32% for MiniCPM-o-2.6, Accent reaches 54.05% for Gemini-2.5-Pro, Education reaches 58.25% for Qwen3-Omni-Flash, Income reaches 61.19% for Qwen3-Omni-Flash, and Social Stratum reaches 47.16% for Gemini-2.5-Pro. With-Audio performance is consistently above random guessing, while Transcribed-Text is often below both With-Audio and random, indicating that profiling power derives from audio rather than semantics in the benchmark design (Wang et al., 7 Jan 2026).

HearSay also shows that current safety mechanisms are uneven. Open-source models exhibit near-zero ARR for almost all attributes. Closed-source models refuse some overtly sensitive queries, but Gender and Age often remain answerable with high accuracy. Prompt-level defense increases ARR substantially for several attributes, yet Gender ARR remains low for many models. Chain-of-Thought reasoning can amplify risk in capable models: for Qwen3-Omni-Flash, Education rises by 7.0% IAR, Income by 10.8%, and Accent by 22.1%, described as a 4.5x improvement over baseline for Accent (Wang et al., 7 Jan 2026).

Attribute-based privacy analysis reaches similar conclusions even after anonymization. Using four attributes—Gender, Age, Accent, and Profession—one study trains ECAPA-TDNN-based embeddings and MLP classifiers and reports utterance-level accuracies on original speech of 0.99 for Gender, 0.76 for Age, 0.64 for Accent, and 0.53 for Profession. At speaker level, 38.9% of speakers are unique under ground-truth attribute profiles, 65.3% have anonymity set size $Z=f(X)$4, and the median $Z=f(X)$5 is 2. The anonymity set is defined as

$Z=f(X)$6

Standard anonymization reduces direct signal-level linkability, but inferred attributes still remain non-trivially predictive. For anonymized speech, Gender accuracy remains 0.80 under McAdams and 0.67 under NAC, while Age remains 0.56 under McAdams and 0.65 under NAC. Exact-match re-identification attacks using attribute profiles yield error rates well below the degenerate baseline in both original and anonymized settings, showing that attribute inference errors do not eliminate singling-out risk (Rahman et al., 19 Mar 2026).

MLLM agent studies extend the phenomenon beyond direct ALLM prompting. AP$Z=f(X)$7 introduces two benchmark subsets—AP$Z=f(X)$8-Com and AP$Z=f(X)$9-TV—and the Gifts framework, a hybrid ALM–LLM agent system comprising Guidance, Inference, Forensics, scruTinization, and conSolidation. On AP$X$0-Com, Gifts achieves an average score of 86.7, exceeding ALM-only and LLM-only baselines by 9.8–40.7 percentage points and 15.5–23.5 points, respectively. In a human study, humans average 67.3, a simple ALM+LLM agent averages 86.4, and Gifts averages 89.8 while using 8.2 minutes rather than 34.7 minutes total (Wang et al., 14 Jul 2025).

5. Deployed systems and real-world profiling practices

Commercial voice assistants provide evidence that profiling is not confined to academic benchmarks. A 20-month audit of Google Assistant, Amazon Alexa, and Apple Siri performs 1171 experiments and 24530 training queries using fresh accounts, factory-reset devices, new phone numbers, and VPNs. Profiling is operationalized through labels exposed in My Ad Center, Amazon advertising exports, and DSAR tools (Khezresmaeilzadeh et al., 2024).

Google Assistant performs demographic profiling via voice queries, but voice is slower and less accurate than web. Across tested labels, voice yields 48% accuracy and 18.0 ± 4.1 days to convergence, whereas web yields 62.86% accuracy and 2.2 ± 2.0 days. Label-specific voice results include 70% for Married, 30% for Single, 50% for Advanced Degree, and 80% for Renters, with several other personas remaining at 0% or “Not Enough Info.” Google also prepopulates labels without interaction: among 200 fresh accounts left unused for one week, 195 are labeled Homeowners, 195 Not Parents, and 194 Moderately High Income (Khezresmaeilzadeh et al., 2024).

Amazon exhibits a different profiling logic. General informational voice queries produce 0% accuracy for the seven tested interest personas, but transactional command queries such as adding items to cart or lists produce 100% accuracy across Fashion, Video Entertainment, Beauty and Personal Care, Electronics, Toys and Games, Pet Supplies, and Books and Magazines, with a mean time to label assignment of 7.7 ± 1.3 days. Voice and web are effectively equivalent for these command-driven interest labels. Siri, by contrast, shows no evidence of observable profiling, either directly in Apple DSAR outputs or indirectly through linked Google accounts in the reported experiments (Khezresmaeilzadeh et al., 2024).

These platform studies concern profiling from interaction content rather than from acoustic voiceprints. A plausible implication is that they delineate only one layer of deployed audio privacy risk. The benchmark literature on ALLMs and voice privacy suggests that if such platforms retain or process richer acoustic representations, then first-party profiling could in principle extend beyond semantic query content to speaker and paralinguistic attributes.

6. Evaluation regimes, defenses, and unresolved problems

A central methodological development is the move from narrow verbal-privacy metrics to broader leakage evaluation. Earlier evaluation practice often emphasized Word Error Rate, Phoneme Error Rate, human transcription, or ASR APIs on raw or masked audio. More recent work argues that these metrics capture only speech intelligibility. Speaker identity, gender, age, location, environment, emotion, health status, and socio-economic attributes require separate evaluation protocols (Chhaglani et al., 2024).

FeatureSense formalizes this broadening through the Speaker Information Leakage Index (SILI),

$X$1

where $X$2 is post-protection classification accuracy for attribute $X$3, $X$4 is the raw-audio baseline, and $X$5 is the attribute weight. It complements this with WER, PER, and eSTOI aggregated into CSLI, and with feature-level correlation, mutual information, Random Forest importance, and permutation importance. On Common Voice three-class tasks, FeatureSense reduces Gender accuracy to 30.0%, Ethnicity to 31.0%, and Age to 40.0%, while prior privacy techniques remain roughly in the 77–95% range. The paper states that the system outperforms existing privacy techniques by 60.6% in preserving user specific privacy (Chhaglani et al., 30 May 2025).

Verbal-privacy work on low-frequency speech adds another evaluation axis. Low-pass filtering and downsampling with an order-8 Chebyshev type-I filter drive Whisper WER above 97.5% for VCTK at 300–800 Hz and above 98% for Pop-glass at 300–1250 Hz. Human listeners recognize almost no words at 800 Hz and only fragmentary content at 1250 Hz, while Voice Activity Detection remains acceptable at 800 Hz for clean single-speaker audio and at 2000 Hz for mingling audio. The paper therefore treats low-frequency capture as promising for verbal privacy, yet explicitly notes that it is not a complete privacy solution and does not prevent profiling of nonverbal attributes (Liu et al., 2024).

Defensive proposals now span feature design, model behavior, and data transformation. Feature-level defenses emphasize on-device extraction of low-risk representations, data minimization, and task-specific feature selection (Chhaglani et al., 2024, Chhaglani et al., 30 May 2025). Model-level defenses for ALLMs include privacy alignment, safety training, encoder-level de-identification, and prompt-level refusals, although the reported refusal behavior is inconsistent across attributes (Wang et al., 7 Jan 2026). AP$X$6 evaluates two additional defenses against MLLM agents: In-Context Unlearning, which reduces Gifts from 86.7 to 58.3 on AP$X$7-Com and to 56.1 on AP$X$8-TV, and anti-eavesdropping jamming, which reduces Gifts to 46.2 on AP$X$9-Com and 43.8 on AP$A$0-TV (Wang et al., 14 Jul 2025).

The unresolved problems are consistent across the literature. Formal guarantees are largely absent: there is no differential privacy mechanism, no information-theoretic bound in most systems, and no cryptographic anonymity guarantee for the feature-design approaches (Chhaglani et al., 2024, Chhaglani et al., 30 May 2025). Residual leakage persists even in coarse or anonymized representations; contextual privacy remains underexplored; multilingual and cross-domain generalization are incomplete; and stronger adversaries—fine-tuned ASR, adaptive attribute classifiers, jailbreak prompting, or probabilistic profile matching—are expected to perform better than the baseline attackers currently reported (Liu et al., 2024, Rahman et al., 19 Mar 2026). This suggests that audio private attribute profiling is best understood not as a solved subproblem of speech privacy, but as a continuing systems-level challenge involving representation design, model alignment, deployment policy, and attribute-aware auditing.