PIIBench: A Unified Multi-Source Benchmark Corpus for Personally Identifiable Information Detection

Abstract: We present PIIBench, a unified benchmark corpus for Personally Identifiable Information (PII) detection in natural language text. Existing resources for PII detection are fragmented across domain-specific corpora with mutually incompatible annotation schemes, preventing systematic comparison of detection systems. We consolidate ten publicly available datasets spanning synthetic PII corpora, multilingual Named Entity Recognition (NER) benchmarks, and financial domain annotated text, yielding a corpus of 2,369,883 annotated sequences and 3.35 million entity mentions across 48 canonical PII entity types. We develop a principled normalization pipeline that maps 80+ source-specific label variants to a standardized BIO tagging scheme, applies frequency-based suppression of near absent entity types, and produces stratified 80/10/10 train/validation/test splits preserving source distribution. To establish baseline difficulty, we evaluate eight published systems spanning rule-based engines (Microsoft Presidio), general purpose NER models (spaCy, BERT-base NER, XLM-RoBERTa NER, SpanMarker mBERT, SpanMarker BERT), a PII-specific model (Piiranha DeBERTa), and a financial NER specialist (XtremeDistil FiNER). All systems achieve span-level F1 below 0.14, with the best system (Presidio, F1=0.1385) still producing zero recall on most entity types. These results directly quantify the domain-silo problem and demonstrate that PIIBench presents a substantially harder and more comprehensive evaluation challenge than any existing single source PII dataset. The dataset construction pipeline and benchmark evaluation code are publicly available at https://github.com/pritesh-2711/pii-bench.

• The paper introduces PIIBench, a unified benchmark that consolidates ten datasets to rigorously evaluate PII detection systems across domains.
• It employs a multi-stage normalization pipeline that standardizes over 80 source-specific annotation variants into a unified BIO tagging scheme.
• Baseline evaluations reveal severe domain-silo effects, with current models achieving F1 scores below 0.14 on diverse PII types.

PIIBench: A Unified Multi-Source Benchmark and Its Impact on PII Detection

Motivation and Context

Personally Identifiable Information (PII) detection is foundational for compliance with regulations such as GDPR, HIPAA, and CCPA. Despite the proliferation of Named Entity Recognition (NER) datasets and domain-specific PII resources, there has been a critical absence of a comprehensive, standardized benchmark that enables rigorous, cross-domain evaluation of PII detection systems. Existing resources are fragmented by annotation schemes and domain silos, which precludes direct system comparison and limits progress in unified solution development. "PIIBench: A Unified Multi-Source Benchmark Corpus for Personally Identifiable Information Detection" (2604.15776) addresses this gap by constructing a large-scale, multi-source corpus with standardized annotation, addressing both coverage and label normalization challenges.

Corpus Construction and Normalization

PIIBench consolidates ten publicly available datasets, encompassing synthetic PII corpora, general and multilingual NER resources, and financial-domain annotated text. This results in a corpus of 2,369,883 annotated sequences with over 133 million tokens and approximately 3.35 million entity mentions spanning 48 canonical PII entity types. Crucially, the authors design a multi-stage normalization pipeline that resolves 80+ source-specific annotation label variants to a standardized BIO tagging scheme, effectively harmonizing incompatible label taxonomies from diverse sources.

Curating the corpus involves three critical steps:

1. Source capping: To ensure financial regulatory filings (especially from the FINER-139 dataset) do not dominate, records from these sources are capped during dataset splits.
2. Rare entity suppression: Entity types with fewer than 500 mentions are filtered out, targeting markup artifacts and non-PII categories.
3. Stratified splits: An 80/10/10 train/validation/test split is applied independently to each source, followed by interleaving, preserving the proportional diversity of entity types and source domains across all partitions.

The resulting distribution is purposefully skewed towards real-world prevalence, e.g., FINANCIAL_ENTITY (29.1%), LOC (14.1%), and PERSON (12.1%), reflecting the diversity found in actual text encountered in compliance and data privacy contexts.

Baseline Evaluation: Quantifying Domain Silos

Eight baseline detection systems spanning rule-based (Microsoft Presidio), general-purpose NER (spaCy, BERT-base-NER, XLM-RoBERTa NER, SpanMarker mBERT/BERT), a PII-specific transformer (Piiranha DeBERTa), and a financial-domain specialist (XtremeDistil FiNER) were evaluated on a stratified subset of the test split (1,398 records, 3,383 entity mentions). Evaluation is strictly span-level with exact match constraints on both boundary and entity type, using "seqeval" metrics.

Key findings:

• All systems yield F1 below 0.14. Highest is Presidio (F1=0.1385), leveraging regex for classical PII types but providing zero recall on most entity classes.
• Domain-specific generalization failure is severe. For example, XtremeDistil FiNER achieves the highest span-level precision (0.6990) but near-zero recall (0.0213) due to its financial entity focus; it fails comprehensively for other PII types.
• General NER models cover only universally-shared classes. CoNLL/ WikiANN/ FewNERD-based systems detect LOC and PERSON but are blind to critical identifiers like CREDIT_CARD, IBAN, API_KEY, and USERNAME, which are only present in synthetic or finance-specific corpora.
• PII-tailored systems do not generalize. Piiranha DeBERTa is trained on a single synthetic corpus, leading to weak performance on the composite benchmark.

The results robustly quantify the domain-silo phenomenon: each system achieves coverage only for entity classes that dominate its original training data, with negligible or zero recall for types outside that distribution.

Theoretical and Practical Implications

The construction and empirical evaluation using PIIBench illuminate key limitations in the current state of PII detection:

• Evaluation practices based on single-source datasets are misleadingly optimistic. Performance reported on synthetic or legacy NER datasets does not transfer to multi-domain settings, leading to inflated expectations of model generalization.
• Regulatory and real-world requirements demand broad recall across heterogeneous entity types. The inability of current systems to detect even simple credential types or financial identifiers—unless explicitly included in training—demonstrates the risk of relying on domain-specific models for data privacy in production.
• Label normalization is non-trivial and error-prone. The manual consolidation of source labels into canonical types is necessary but must be performed conservatively to prevent semantic drift or noise.

From a practical standpoint, PIIBench enables robust, reproducible benchmarking and highlights the necessity for training strategies that incorporate heterogeneous supervised data and domain adaptation techniques. Furthermore, its open pipeline and canonical mapping facilitate community-driven extension and refinement of the benchmark.

Future Directions

The work outlines several avenues for extension:

• Cross-source generalization: Leave-one-domain-out experiments can quantify transferability and identify bottleneck domains.
• Multilingual extension: Non-English splits from sources like ai4privacy and gretelai-finance will be incorporated for cross-lingual PII redaction research.
• Domain expansion: Inclusion of medical, legal, and social media corpora is critical for heightened applicability in domains with nuanced privacy requirements.
• Annotation auditing: Analysis of inter-annotator agreement and synthetic data noise will define the benchmark’s quality floor.

These directions will further solidify PIIBench as the core evaluation ecosystem for PII detection models.

Conclusion

PIIBench introduces a rigorously normalized, large-scale, open benchmark encompassing realistic cross-domain PII annotation, solving the comparison and coverage limitations that have impeded progress in the field. By unifying fragmentary resources and empirically demonstrating the acute limitations of current systems, PIIBench provides both a challenge and a standard for researchers and practitioners. Benchmark results establish robust lower bounds and frame the development of future multi-domain PII detection models—ushering in an era where direct, comprehensive evaluation is possible and necessary for privacy-centric NLP deployment (2604.15776).