X-Sensitive Dataset: Definitions & Benchmarks

• X-Sensitive datasets are specialized benchmarks designed to probe and model risks related to sensitive content, privacy, and modality shifts.
• They are constructed using manual and synthetic annotation, rigorous calibration, and fine-structured label spaces to ensure precise sensitivity measurement.
• Applications include enhancing safety pipelines, auditing privacy, improving fairness in recommendations, and supporting spatiotemporal reasoning in diverse domains.

An X-Sensitive dataset, as formalized by multiple recent works, refers to a benchmark or resource whose construction and evaluation protocols are specifically oriented toward detecting, modeling, or evaluating sensitivity in its broadest sense: either the sensitive content of the data itself (e.g., personal data, offensive or harmful language, sensitive topics) or the sensitivity of the task or environment to certain variables (e.g., imaging modality, spatiotemporality, user well-being). Such datasets are foundational for developing and evaluating models with explicit robustness, safety, privacy, and alignment constraints, and their precise meaning is domain-dependent. Representative instantiations include large-scale PII-rich corpora for LLM memorization studies, social-media datasets annotated for fine-grained sensitive content categories, benchmarks for sensitivity-aware recommendations, and resources requiring fine-tuned sensitivity to spatiotemporal information.

1. Definitions Across Domains

The term “X-Sensitive Dataset” is context-dependent and serves as an umbrella for datasets constructed to probe, benchmark, or mitigate sensitivity risks. Key instantiations include:

• Sensitive Content Detection: Datasets constructed to enable supervised or semi-supervised learning of models that classify, localize, or filter potentially harmful, offensive, or regulated content—such as profanity, explicit language, self-harm references, hate speech, and spam in social data (Antypas et al., 2024).
• Sensitive Data Memorization: Large-scale corpora specifically engineered to contain synthetic or real sensitive data—personally identifiable information (PII), quasi-identifiers, or private attributes—to measure memorization and leakage risks in LLMs (e.g., “PANORAMA” (Selvam et al., 18 May 2025)).
• Modality/Condition Sensitivity: Datasets highlighting a domain shift or sensitivity of classical CV algorithms to domain-specific variables, like the difference in keypoint repeatability, feature matching, or motion estimation between visible and X-ray spectra (Chekanov et al., 2019).
• Recommendation Sensitivity: Resources jointly encoding user–item preferences and content warnings across a large taxonomy of harmful or sensitive topics, supporting the evaluation of user-level avoidance or exposure in algorithmic content curation (Kovacs et al., 8 Sep 2025).
• Spatiotemporal Sensitivity: QA or reasoning benchmarks where the core challenge is to resolve fine-grained, time- or location-dependent facts and relationships (e.g., Point of Interest trajectories and scheduling) (2505.10928).

2. Construction Methodologies

The construction pipeline varies by the axis of sensitivity:

• Manual and semi-automated annotation: For social content moderation (e.g., X-Sensitive (Antypas et al., 2024)), domain experts and crowd workers annotate large sets of seed data according to detailed taxonomies. Multi-stage keyword expansion and manual filtering enforce category validity and diversity.
• Synthetic profile simulation and generative pipelines: In PII benchmarks like PANORAMA (Selvam et al., 18 May 2025), a large population of internally consistent, multi-attribute synthetic human profiles (names, birth dates, jobs, medical conditions, etc.) is generated using constrained sampling algorithms. Downstream content (social posts, reviews, marketplace ads) is produced programmatically or with LLM assistance, ensuring high PII density and context-appropriate embedding.
• Calibration and ground-truth protocols: In the imaging domain, highly controlled acquisition pipelines (precise turntables, paired imaging modalities, full calibration metadata) guarantee unbiased evaluation of sensitivity to domain shifts in image analysis (Chekanov et al., 2019).
• Alignment of user–item–warning matrices: For content warnings in recommender datasets, large-scale repositories (MovieLens, AO3) are aligned at the item level with independently sourced, community-labeled warning taxonomies, yielding high-dimensional binary label vectors per item (e.g., 137 warning types for ML-DDD) (Kovacs et al., 8 Sep 2025).
• Spatiotemporal fact mining and manual validation: Geographic trajectory + POI benchmarks rely on large-scale mining (e.g., of vehicle GPS logs), careful coordinate normalization and de-duplication, and multiple stages of manual review to align spatial events with natural-language descriptions and questions (2505.10928).

3. Taxonomies and Label Spaces

The core of an X-Sensitive dataset is an explicit, fine-structured label space:

Domain	Example Label Categories	Label Cardinality	Key Features
Social Moderation	Profanity, Conflictual Language, Sexually Explicit, Drug-Related, Self-Harm, Spam (Antypas et al., 2024)	6	Multi-label, span annotation
PII/Privacy	PII string types: name, address, phone, medical, salary, quasi-identifiers (Selvam et al., 18 May 2025)	>20 attributes	Cross-context, cross-genre
Recommendation	137 warnings (ML-DDD), 36 warnings (AO3): violence, self-harm, discrimination, etc. (Kovacs et al., 8 Sep 2025)	36–137	Item-aligned binary vectors
Imaging/CV	Not label-driven, but modality acts as sensitive axis (visible/X-ray) (Chekanov et al., 2019)	2 (domains)	Per-frame, per-modality
Spatiotemporal QA	POI category (major–sub–fine), temporal relations (2505.10928)	19–959	Multi-granularity, temporal

Taxonomy definition impacts the scope, annotation strategy, and downstream model task.

4. Evaluation Protocols and Metrics

Evaluation strategies are tightly coupled to sensitivity objectives.

Sensitive Content (Social Moderation)

• Macro-averaged Precision, Recall, F1: Reported per-label and overall (macro-F1 up to 85.6 binary, 69.8 multi-label for fine-tuned 8B LLMs) (Antypas et al., 2024).
• Test-Set Keyword Disentanglement: Half of test instances exclude seed keywords to measure generalization.
• Span-based support annotation: Workers highlight evidence supporting each label.

Memorization/Privacy

• PII Memorization Rate: Fraction of training-set PII extractable by prefix completion.
• Soft Match Rate: Proportion of completions with ROUGE-L F1 exceeding a threshold (e.g., τ=0.8).
• k× Data Repetition Protocol: Manipulates training exposure to quantify memorization amplification (Selvam et al., 18 May 2025).

User-Targeted Recommendation

• Sensitivity-Weighted Precision@k, Recall@k: Weigh recommendations in top-k lists by their warning labels (Kovacs et al., 8 Sep 2025).
• Warning Amplification@k: Measures if exposure to sensitive content is amplified or suppressed relative to user history.

Imaging/Perception

• Mean Absolute Error (MAE), Root Mean Squared Error (RMSE): Quantify angle estimation error per sequence (Chekanov et al., 2019).
• Keypoint Repeatability, Descriptor Precision & Recall: Assess robustness of vision pipelines to domain shift.

Spatiotemporal QA

• HR@k, NDCG@k, BLEU: For top-k answer retrieval and sequence generation (2505.10928).

5. Empirical Findings and Limitations

Distinct X-Sensitive datasets support specific empirical claims:

• Improved Model Robustness: Fine-tuning LLMs on multi-label sensitive-content sets (X-Sensitive/(Antypas et al., 2024)) yields binary F1 gains up to ≈10–15% over off-the-shelf APIs, especially on profanity and explicit categories; performance remains lower (≈50–62 F1) for sparse or subtle types (Drug, Self-Harm, Conflictual).
• Memorization Escalation: PANORAMA (Selvam et al., 18 May 2025) shows memorization rates increase sharply with training repetition (Soft Match Rate: 8.8% at 1×, 51.2% at 25×). Certain contexts (marketplace ads: 4%→68%) are exceptionally vulnerable; brief or slang-heavy contexts resist extraction.
• Domain Shifts in Vision: Classical pipelines (SIFT/FLANN/RANSAC) on XV-CM maintain low MAE (≈0.2°) in visible; degrade in X-ray (≈0.5°), confirming sensitivity of feature detectors to projection modality (Chekanov et al., 2019).
• Personalization Risk: ML-DDD/AO3 datasets enable quantification of recommendation risks; e.g., mean ratings for items with “blood/gore” warnings (2.96) are lower than for non-flagged items (3.15), reflecting user discomfort (Kovacs et al., 8 Sep 2025).
• Spatiotemporal Reasoning Gaps: LLMs remain far from human-level on POI-QA, with HR@10 ≤ 0.41 vs. 0.56 for humans on simplest tasks (2505.10928).

Limitations are generally domain-specific: language/cultural focus (Korean, English-only), small sample sizes for rare labels, lack of multi-modality, and subjectivity in sensitivity annotation.

6. Applications and Broader Impact

X-Sensitive datasets are leveraged for:

• Safety and Moderation: Enabling LLM-based safety pipelines, filter-moderation, and classifier-in-the-loop safe response selection in high-stakes contexts (chatbots, open-domain Q&A) (Antypas et al., 2024, Lee et al., 2023).
• Privacy Auditing and Defense: Supporting the quantitative benchmarking of privacy defenses (differential privacy, deduplication, dememorization) and risk modeling for LLMs with synthetic but realistic PII (Selvam et al., 18 May 2025).
• Algorithmic Fairness and Alignment: Providing concrete metrics and training resources for models to avoid exposing users to unwanted sensitive material or amplifying exposure in recommender settings (Kovacs et al., 8 Sep 2025).
• Cross-Modal or Domain Generalization: Studying the transferability and robustness of classical and neural algorithms under domain, modality, or context shifts (Chekanov et al., 2019).
• Spatiotemporal Decision Support: Benchmarking advanced QA and reasoning systems on tasks that integrate both spatial and temporal context for applications in logistics and urban computing (2505.10928).

A notable implication is that X-Sensitive benchmarks offer a means to quantify trade-offs between utility and risk, supporting principled development of models and systems that satisfy complex ethical, privacy, and regulatory requirements.