RedactionBench

Abstract: LLMs are increasingly applied to sensitive domains that require redaction of personally identifiable information (PII). While redacting PII is a data cleaning prerequisite, existing benchmarks conflate extraction mechanics with privacy semantics. A public phone number is not equivalent to a phone number in a medical record. Whether information constitutes a violation depends heavily on who holds it, why, and in what context, fundamentally differentiating redaction from simple entity recognition. Grounded in contextual integrity, we introduce RedactionBench, a manually annotated benchmark comprising 200 diverse documents across 11 domains, mostly seeded from real-world sources. We also introduce R-Score, a novel character-level metric that treats semantically similar redactions equally and nullifies shallow formatting choices, such as varying masking styles for phone numbers. Evaluations across Named Entity Recognition models, entity extraction Small LLMs, and frontier models equipped with agentic tools demonstrate that contextual redaction remains an unsolved problem. A human evaluation with over 80 users on RedactionBench reveals a stark dichotomy in privacy perceptions. Annotators show consensus with target labels for mandatory redactions (89.4 percent) and safe text preservations (94.1 percent), but fail to agree on contextual redactions (47.7 percent). This variance demonstrates the subjective nature of contextual privacy and motivates R-Score, which decouples contextual ambiguity from strict precision. We compare 35 models across families and report their performance in redacting PII. Finally, we release RedactionBench to establish a baseline for future privacy-preserving systems, hoping to inspire efficient model design and standardized evaluations.

• The paper introduces RedactionBench, a context-sensitive benchmark using a ternary labeling scheme on real-world documents to evaluate PII redaction systems.
• It presents the R-Score, a character-level metric that blends Intersection-over-Union and strict scoring to assess redaction precision and boundary sensitivity.
• Comparative evaluations show that frontier generative LLMs with tool-calling abilities outperform traditional classifiers in managing context-aware privacy redaction.

RedactionBench: Establishing Contextual Integrity for PII Redaction

Motivation and Benchmark Construction

The propagation of LLMs across privacy-sensitive domains has amplified the necessity for precise, robust redaction systems capable of masking PII at character granularity, preserving semantic integrity. Existing benchmarks focus on extraction mechanics but fail to encode contextual privacy, leading to label taxonomies loosely coupled to actual risk. The RedactionBench framework introduces a unified, contextual integrity–grounded benchmark for privacy-preserving text redaction, comprising 200 manually annotated documents spanning 11 domains sourced predominantly from real-world artifacts. The core design adopts a ternary label scheme: mandatory (unconditional unsafe), contextual (context-dependent ambiguity), and gaps (benign). Segmentations disambiguate overlapping or composite entities, leveraging combinators to map coherent entity boundaries.

Figure 1: RedactionBench segmentations illustrate mandatory, contextual, and combinator spans enabling entity-aware privacy evaluation.

Compared to synthetic benchmarks (e.g., Nemotron-PII, Ai4Privacy, Gretel-PII), RedactionBench is distinguished by its cross-domain document realism and extrinsically validated privacy criteria grounded in contextual integrity, as reflected by extensive real document length diversity.

Figure 2: RedactionBench overcomes length and structure limitations of prior benchmarks, promoting generalization.

R-Score: Context-Sensitive Redaction Evaluation Metric

Conventional NER evaluation relies on strict Precision/Recall/F1 token- or span-matching paradigms, exhibiting high sensitivity to boundary fuzziness and over-penalizing minor deviations. RedactionBench introduces R-Score—a character-level, entity-centric metric blending Intersection-over-Union with strict scoring. R-Score provides:

• Invariance to entity length, penalizing imprecise boundaries steeply,
• Partial credit for partial redaction overlap,
• Coverage-driven mandatory entity performance, decoupled from contextual ambiguity.

The R-Score is formalized as:

$$\textrm{R-Score} = \frac{\sum_{i} n_i}{\sum_{i} d_i}$$

where $n_i$ and $d_i$ accumulate per-entity coverage and penalty terms, supporting combinator-aware span fusion and false-positive gap weighting.

Comparative Model Evaluation

Thirty-five models across token-classifiers, span-based extractors (GLiNER), generative LLMs, and special-purpose agentic LLM baselines were evaluated against RedactionBench. Smaller BERT-backed domain-adapted classifiers outperform most entity-extraction SLMs (best SLM R-Score = 0.45, BERT-based = 0.58, GLiNER = 0.47). Frontier LLMs employing agentic tool-calling mechanisms exhibit highest mean R-Score (0.71), with Privacy Filter (OpenAI) matching BERT performance (0.58).

Figure 3: Mean R-Score vs parameter count demonstrates frontier generative LLM dominance—Pareto frontier highlights cost-efficiency tradeoffs.

Token-prediction models are constrained by fixed vocabularies and lack dynamism; span models (GLiNER) leverage label flexibility but saturate on entity density; generative LLMs surpass human benchmarks on context-aware edit tools. Sliding-window generative NER approaches (e.g., B2NER) are limited by context length and merge penalties.

Human Baseline and Privacy Perception

A user study with 85 expert annotators on RedactionBench revealed substantial dichotomy between consensus for mandatory redactions (89.4% agreement) and contextual entities (47.7% agreement), confirming the subjectivity and diversity of privacy perception. Aggregate human R-Score was 0.77, with frontier LLMs (e.g., GPT-5.4, Claude Opus) exceeding or matching this threshold. Annotator disagreement for contextual labels is statistically $1.8\times$ that of mandatory, indicating unit-type concentrated variance rather than labeling execution noise.

Figure 4: Target-match rates quantify unit-type agreement—mandatory and gap units yield consensus, contextual redactions diverge.

Figure 5: Study tool interface enables context-rich redaction window annotation with granularity control.

Edit Trajectory and Metric Smoothness

Edit trajectories corroborate that R-Score is smooth and attractive: incremental annotation reliably improves scores, with no observed saturation thresholds where participant disagreement dominates. Probability of R-Score improvement after edits correlates positively with starting score.

Figure 6: Probability that R-Score increases after edits, stratified by starting R-Score, confirms metric sensitivity.

Figure 7: Window-level score distribution dynamics demonstrate edit trajectory convergence toward optimal redactions.

Limitations and Implications

RedactionBench currently evaluates in isolation; contextual shifts (e.g., query-aware or conversation-conditioned privacy) remain unaddressed. The characterization of synthetic entity fidelity, re-identifiability, and domain expert inclusion for structured formats offers fertile ground for future research.

Practically, RedactionBench establishes clear cost-performance tradeoffs for edge-deployable vs agentic LLM solutions. Theoretically, it advances privacy modeling beyond taxonomy-based heuristics toward contextually rigorous evaluation, demonstrating that frontier generative LLMs equipped with tool-calling harnesses can optimize privacy directives more effectively than both specialized classifiers and human annotators.

Future Directions

RedactionBench's context-integrity-driven approach suggests several directions:

• Query-aware contextual redaction,
• Fidelity analysis for synthetic samples,
• Integration of expert annotation for structured domains,
• Increasing efficiency and interpretability of frontier LLMs for real-time privacy-on-the-edge.

Conclusion

RedactionBench establishes a systematic, context-sensitive baseline for PII redaction, validates contextual integrity as an evaluative principle, and benchmarks the practical upper bound of current privacy redaction systems. The work confirms that agentic LLMs currently dominate strict and contextual privacy tasks but also highlights inherent subjectivity in privacy perception. The benchmark is poised to become foundational for standardizing privacy-preserving text redaction research and guiding future advances in model design, evaluation, and policy alignment.