Bystander Privacy Fine-Tuning (BPFT)

• BPFT is a framework that defines and addresses bystander privacy by training models to selectively refuse irrelevant content.
• It employs selective fine-tuning with synthetic multi-speaker data to balance main-speaker comprehension and robust privacy protection.
• Empirical results demonstrate that BPFT improves bystander refusal rates without compromising main-speaker accuracy in both audio and mixed reality contexts.

Bystander Privacy Fine-Tuning (BPFT) refers to a set of emerging methodologies and design paradigms aimed at enabling machine learning systems—particularly those operating in mixed reality (MR) and large audio LLM (LALM) settings—to minimize information disclosure about “bystanders”: individuals exposed to system sensors who are neither the system’s primary user nor explicit application participants. BPFT encompasses both practical fine-tuning pipelines for foundation models and broader socio-technical frameworks for balancing data utility, usability, and privacy. The central innovation of BPFT is to teach models to selectively refuse or sanitize content related to bystanders, without compromising legitimate comprehension or utility for intended users or subjects (Corbett et al., 2023, Zhan et al., 6 Dec 2025).

1. Formalization of Bystander Privacy

The Bystander Privacy Problem (BPP) is defined as the discrepancy between the level of privacy protection demanded by individuals incidentally present in the model’s capture radius and the actual protection afforded by the system’s sensor stack and software. Bystanders are formally identified as any person detected by the device’s sensors who is neither (i) the primary wearer (“user”), nor (ii) a pre-consented participant (“subject”) (Corbett et al., 2023).

In LALM contexts, the BPFT paradigm operationalizes bystander privacy as the model’s capacity to distinguish and exclude information about voices or utterances not attributable to the intended main speaker. The SH-Bench benchmark establishes query-response metrics probing whether a model can robustly identify, suppress, or refuse information requests about bystander speech (Zhan et al., 6 Dec 2025).

2. BPFT Methodologies in Audio LLMs

The state-of-the-art BPFT methodology is instantiated in selective hearing–oriented fine-tuning pipelines targeting foundation models for audio understanding (Zhan et al., 6 Dec 2025). The core training pipeline involves:

• Data Preparation: Construction of multi-speaker audio mixtures by inserting attenuated bystander speech (–10 dB, 20–50 s segments) into dense main-speaker utterances (2–3 min, ≥70 % activity), with convolutional room reverberation to simulate realistic meeting acoustics.
• Annotation: Each mixture is paired with multiple-choice and open-ended questions about main-speaker or bystander content, with explicit instruction annotation: (a) answer as usual, or (b) refuse if the query concerns bystander content.
• Fine-Tuning Strategy: Off-the-shelf audio encoders are frozen; a LoRA (Low-Rank Adaptation) adapter (rank 32) is applied to the LLM backbone’s transformer weights. Training seeks to minimize a cross-entropy sum of (i) main-speaker answer accuracy and (ii) bystander refusal (maximizing “I don’t know” responses) under selective prompts.
• Loss Function:

$$\mathcal{L} = \mathcal{L}_{\text{MS}} + \lambda_{\text{ref}} \mathcal{L}_{\text{refusal}}$$

with $\lambda_{\text{ref}} = 1.0$, partitioning batches into main and bystander questions and targeting high bystander refusal under selective prompts.

• Model Examples: Qwen-2.5-Omni 7B, Step-Audio-2-mini baseline models.

BPFT thus enforces conditional privacy behaviors learned from synthetic mixtures and paired instructions, enabling the trained model to switch between full-comprehension and privacy-protective refusal modes (Zhan et al., 6 Dec 2025).

3. BPFT in Mixed Reality: Taxonomy of Approaches

In mixed reality and vision-based systems, BPFT is conceptualized as a goal state rather than as a settled algorithm. Prevailing methods are categorized into:

• Explicit Schemes: Out-of-band user or bystander interventions, e.g., privacy gestures, manual consent registration, opt-in/out hardware tokens or beacons (Corbett et al., 2023).
• Implicit Schemes: On-device inference systems treating individuals as bystanders or subjects based on contextual factors, such as gaze-based heuristics (assign “subject” status to individuals intersecting the user’s gaze vector, blur or excise others), spatial bounding-box centering, or classifier-driven sanitization (Corbett et al., 2023).

No system of formal equations for BPFT in MR currently exists; functional implementations rely on pre-trained classifiers and deterministic rules mapped to real-time sensor streams.

4. Selective Efficacy Benchmarking and Empirical Outcomes

Empirical evaluation of BPFT is conducted using the SH-Bench framework, deploying paired general and selective hearing queries and introducing the Selective Efficacy (SE) metric:

$$\mathrm{SE} = \frac{4}{\dfrac{1}{\mathrm{Acc}_{\text{gen,main}}} + \dfrac{1}{\mathrm{Acc}_{\text{sel,main}}} + \dfrac{1}{\mathrm{Acc}_{\text{gen,bys}}} + \dfrac{1}{\mathrm{Acc}_{\text{sel,bys}}}}$$

SE jointly measures:

• Main-speaker accuracy in general and selective modes,
• Bystander comprehension and refusal rates.

Key findings:

Model	Gen Main	Sel Main	Gen Bys	Sel Bys	SE (%)
Step-Audio-2-mini	94.2	93.7	54.7	31.5	56.1
Step-Audio-2-mini + BPFT	97.4	94.3	81.0	96.1	91.7
Qwen-2.5-Omni 7B	96.0	95.5	48.2	47.6	63.9
Qwen-2.5-Omni 7B + BPFT	93.3	92.7	82.0	93.8	90.2
Gemini 2.5 Pro	97.3	97.0	65.5	59.2	75.8

• BPFT augments bystander refusal accuracy to 93–96 % in selective mode, raising SE by up to 15.9 pp (to 91.7 %) over the best pre-BPFT baseline.
• BPFT does not cause catastrophic forgetting of main-speaker comprehension (>94 % preserved).
• Open-ended refusal rates rise from ≈ 45 % pre-BPFT to 92–96 % post-BPFT, showing robust generalization of privacy instructions (Zhan et al., 6 Dec 2025).

Ablations demonstrate that speaker-identity cues and explicit “I don’t know” tokens are significant for reliable bystander refusal, but BPFT-induced behaviors are at least partially robust in their absence.

5. Technical and Usability Trade-offs

BPFT is intrinsically a trade-off among:

• Usability: Maintenance of seamless user experience, free of disruptive prompts or computational lag.
• Bystander Protection: Assurance that incidental or non-consenting individuals are reliably sanitized or excluded (audio: refusal, vision: blurring/excision).
• Availability of Legitimate Data: Avoidance of over-sanitization that would compromise target subject data utility or authorized operations.
• Consent Mechanisms: Accommodation for explicit bystander opt-in/out or consent signaling, whether through technological or social means (Corbett et al., 2023).

Explicit approaches optimize bystander control and consent granularity but impose interactional or computational costs. Implicit approaches, while scalable and continuous, may misclassify ambiguous actors and rely on inference context.

6. Limitations and Future Research Directions

Current BPFT practice is subject to several limitations:

• Training regimes are dominated by synthetic data, with performance drops observed in real-world audio conditions (notably, a ∼5 percentage point generalization gap for bystander questions).
• Present pipelines address only two-speaker mixtures; extension to dynamic, multi-party conversational settings is an open challenge.
• In selective mode, absence of explicit “IDK” response tokens impairs refusal rates in some models, indicating instruction internalization is not universal.

Research priorities outlined include:

1. Deployment of on-device, ultra-low-latency privacy filters for empirical comfort and efficacy testing.
2. Socio-technical studies on evolution of privacy norms in heavily instrumented environments.
3. Elaboration of richer actor taxonomies beyond binary subject/bystander classes.
4. Adaptation to multi-modal, visually grounded models integrating scene and speech cues.
5. Continual and prompt-based BPFT to incrementally update proprietary LALMs without full retraining.

Further, educational intervention to clarify privacy affordances and risks for both users and bystanders is regarded as critical for real-system adoption (Corbett et al., 2023, Zhan et al., 6 Dec 2025).