RedactionBench: Privacy Redaction Benchmark

• RedactionBench is a benchmark for privacy-preserving text redaction that defines redaction as a context-sensitive character-level masking problem, distinguishing between mandatory, contextual, and safe text.
• It comprises 200 manually annotated documents across 11 domains with 53,286 annotations, using the novel R-Score metric to evaluate precise, context-aware redaction.
• The benchmark evaluates diverse model families—including token-level, span-level, and generative approaches—highlighting challenges in balancing redaction coverage and over-redaction.

Searching arXiv for the benchmark and closely related work. RedactionBench is a benchmark for privacy-preserving text redaction that treats redaction as a context-sensitive character-level masking problem rather than as a closed-taxonomy named entity recognition task. It is grounded in Contextual Integrity (CI), comprises 200 manually annotated documents across 11 domains, includes 53,286 annotations, and pairs its corpus with R-Score, a character-level metric intended to reward coverage of mandatory redactions, penalize unnecessary masking, and reduce sensitivity to contextual ambiguity and shallow formatting differences in the rendered mask (Brynjólfsson et al., 17 Jun 2026).

1. Conceptual basis

RedactionBench was introduced from the premise that privacy is not an intrinsic property of a string alone. The same surface form may be benign in one setting and privacy-sensitive in another, depending on document type, intended disclosure, and social norms of information flow. The benchmark therefore distinguishes redaction from generic PII extraction: a phone number in a public directory is not equivalent to a phone number in a medical record, and a detail that is public in a quarterly report may become sensitive when the document is marked “DO NOT DISTRIBUTE - INTERNAL MEMO” (Brynjólfsson et al., 17 Jun 2026).

Within this framing, the benchmark defines redaction as “the precise, localized masking of targeted entities to remove sensitive information while preserving overall semantic utility.” That definition makes boundary placement and preservation of safe text central. Redaction is therefore treated as a character-level decision problem: a system must identify unsafe content, avoid masking safe content, place boundaries correctly, and preserve utility by redacting only the necessary characters.

The conceptual foundation is explicitly Nissenbaum’s Contextual Integrity. RedactionBench operationalizes CI through a three-way distinction between Mandatory entities, Contextual entities, and Gaps / safe text. Mandatory entities are always unsafe to share. Contextual entities are ambiguous under the available context and may be unsafe or safe depending on additional information. Gaps are text that should remain visible. This division is not only an annotation device; it is also the basis for the benchmark’s metric design and for its treatment of human disagreement.

A plausible implication is that RedactionBench is less a taxonomy of entity types than a benchmark of privacy judgments under constrained contextual evidence. That distinguishes it from extraction-centric resources that ask whether a span is a name, email, or account number, but not whether it should be masked in the present document.

2. Corpus design and annotation scheme

RedactionBench contains 200 documents spanning 11 taxonomic categories, divided into 7 unstructured categories and 4 structured categories (Brynjólfsson et al., 17 Jun 2026).

Category	Count
Academic	16
Emails	16
Financial	18
Government	16
Legal	18
Medical	16
Operations	15
Code	16
Files	36
Logs	17
Terminal	16

The corpus is reported as balanced between 101 real documents and 99 synthetic documents, while the paper also states that a majority is seeded from real-world sources. Real documents were drawn especially from publicly available materials such as government websites, academic resources, and company-provided forms. Synthetic generation was used when realistic public examples were difficult to obtain or lacked sufficient sensitive content. The benchmark emphasizes document realism and length diversity; many sources were PDFs extracted primarily with pdfminer, with OlmOCR, InfinityParser, and PaddleOCR used for more difficult cases, and the longest documents were capped at 10 pages (Brynjólfsson et al., 17 Jun 2026).

The benchmark’s document genres are deliberately heterogeneous. Examples include academic probation appeals, transcripts, FAFSA forms, recommendation letters, bank alerts, password resets, phishing lures, travel itineraries, 1040 forms, bank statements, wire transfer requests, W-9s, Bitcoin wallet exports, birth certificates, voter registration cards, police reports, visas, leases, offer letters, immigration forms, prenuptial agreements, discharge summaries, prescriptions, psychotherapy notes, radiology reports, pay stubs, boarding passes, shipping labels, resignation letters, .env files, AWS config JSON, JWT token files, CloudTrail logs, Kubernetes sessions, SSH configs, wallet exports, CI logs, and source code or config files. This breadth makes the benchmark simultaneously a natural-language and machine-readable redaction resource.

RedactionBench uses three label classes: Mandatory spans, Contextual spans, and Implicit gaps between them. A distinctive design decision is that contextual labels are overloaded to include structural and combination characters used to connect subspans into larger entities. This supports equivalent segmentations. For example, an IPv4 address such as 127.0.0.1 can be decomposed into multiple digit groups separated by periods, and a name such as John M. Doe can be treated either as separate components or jointly. The benchmark therefore decomposes entities into semantically meaningful subspans and supplies combination rules that allow equivalent segmentations to score equally.

This annotation logic is intended to prevent naive entity priors from dominating evaluation. The paper’s example is John Hancock Center: the substring John Hancock has the surface form of a person name, but in that phrase the full span denotes a venue. A system that redacts only John Hancock may look plausible under NER assumptions while deviating from the benchmark’s target labels (Brynjólfsson et al., 17 Jun 2026).

All documents were hand-annotated, and the paper states that LMs were used as a warm-start checkpoint via semantic pattern matching for documents with hundreds of entities. At the same time, the paper does not provide a conventional inter-annotator agreement statistic for the core benchmark annotations, and it does not describe a multiple-gold-annotator plus adjudication pipeline in the main annotation section. That omission is a material caveat for interpreting the gold labels.

3. Formal task and R-Score

The benchmark evaluates character-level redaction. Let $P$ be the set of characters redacted by a user or model, $\mathbf{M}$ the set of all mandatory spans, $R$ a partition of $\mathbf{M}$ into mandatory entities, $\mathbf{C}$ the set of all contextual spans, and $Y$ a partition of $\mathbf{C}$ into contextual entities. Each span contributes a local score $(n_i,d_i)$, aggregated at the entity level and pooled globally. The overall metric is (Brynjólfsson et al., 17 Jun 2026):

$\textrm{R-Score}=\frac{\sum_i n_i}{\sum_i d_i}$

For a mandatory entity $r \in R$, the contribution is the mean coverage of its constituent spans:

$\mathbf{M}$0

Mandatory entities therefore receive continuous partial credit through overlap, while each entity has denominator $\mathbf{M}$1, making entity contributions length-invariant.

Contextual entities are handled differently. The active subset is defined as:

$\mathbf{M}$2

If $\mathbf{M}$3 is empty, the contextual entity contributes nothing. Otherwise:

$\mathbf{M}$4

Under the benchmark’s standard setting, contextual entities therefore act mainly as residual penalties. Clean handling avoids harm, but contextual entities do not ordinarily add positive score. The exception is documents with no mandatory labels or all contextual labels; in that case contextual spans behave like optional mandatory labels so that the document does not admit only zero-valued scoring.

False positives are defined as contiguous redacted character sets that do not intersect any mandatory or contextual support. They contribute:

$\mathbf{M}$5

This doubles the denominator when a prediction redacts an entire gap of at least three characters between entities, directly penalizing over-redaction across boundaries.

The benchmark also introduces hand-annotated combinators and prediction-dependent grouping rules so that semantically equivalent segmentations count equally. The paper references Algorithm 1 for combinator structure construction and Algorithm 2 for prediction-dependent contextual selection and entity fusion. Grouping uses graph-based heuristics over punctuation, slashes, matched delimiters, and whitespace bridges. The stated purpose is to make redaction scoring invariant to shallow formatting choices such as masking a phone number as (***) ***-**** versus **************, provided the same underlying original characters are suppressed.

The paper positions R-Score as lying “between IoU and strict F1.” Relative to standard precision, recall, and F1, it is intended to solve five specific problems: semantically equivalent redactions, formatting-invariant masking, continuous credit for partial overlap, contextual ambiguity, and over-redaction across entity boundaries. This suggests that R-Score is not merely a tolerance-adjusted overlap measure, but a redaction-specific metric whose denominators encode a benchmark policy about mandatory disclosure, contextual uncertainty, and safe-text preservation.

4. Evaluation protocol and model families

RedactionBench evaluates 35 public model entries, corresponding to 34 unique checkpoints, organized into three model families (Brynjólfsson et al., 17 Jun 2026).

The first family is token-level models, including BIO-style token classifiers such as DeBERTa, DistilBERT, RoBERTa, ModernBERT, and the OpenAI Privacy Filter. For these models, any non-O BIO label is mapped to redact, and entity groupings are applied in postprocessing.

The second family is span-level models, represented by the GLiNER family. The paper states that best performance was obtained by using the subset of labels corresponding to each model’s own training inventory rather than arbitrary open-vocabulary prompts, with a single global score threshold selected by aggregate optimization over evaluation performance. Reported thresholds include nvidia/gliner-PII at 0.6531 and gretel-gliner-bi-large-v1.0 at 0.1019.

The third family is generative models, including extraction-oriented SLMs or generative NER systems such as B2NER, NuExtract, UniNER, Distil-PII, and OpenPipe, together with frontier LLMs equipped with forced tool use: Claude Opus 4.6, GPT-5.4, Qwen3.5-397B-A17B, and GLM-5.1. For B2NER, the system is prompted with fixed label lists and evaluated using fixed-size sliding windows whose overlapping predictions are merged. For UniNER, the paper reports one prompt per label per chunk over a 91-label set.

A notable empirical protocol detail is that frontier models were not evaluated via free-form text generation. The paper reports that “disguising the task as a file-editing tool call yields stronger results than free-form querying,” so all frontier systems were evaluated with forced tool use. Claude Opus used a forced MultiEdit file-editing tool, GPT-5.4 used a forced predict_entities tool with marked spans, and Qwen3.5 together with GLM-5.1 used a forced bash tool generating chained sed substitutions. Prompts instructed the models to redact for public release, use the narrowest substrings possible, and make exactly one tool call.

Evaluation reports mean, median ($\mathbf{M}$6), and 20th percentile ($\mathbf{M}$7) across domains. Frontier closed models were run through APIs or Together AI, while local models, including the largest B2NER system, were run on a single NVIDIA H100 80GB.

5. Main results

The benchmark’s headline empirical conclusion is that contextual redaction remains unsolved, although frontier agentic LLMs substantially outperform other model families (Brynjólfsson et al., 17 Jun 2026). Across the full 200-document benchmark, the highest overall mean R-Scores are 0.71 for claude-opus-4-6, 0.66 for gpt-5.4, 0.59 for Qwen/Qwen3.5-397B-A17B, and 0.56 for zai-org/GLM-5.1. Among non-frontier systems, the best reported scores are 0.58 for openai/privacy-filter, 0.47 for gretelai/gretel-gliner-bi-large-v1.0, 0.46 for OpenMed/OpenMed-PII-SuperClinical-Large-434M-v1, and 0.45 for B2NER internlm2-20b. Low-performing systems include Distil-PII-gemma-3-270m-it at 0.09, OpenPipe/PII-Redact-General at 0.11, and Distil-PII-Llama-3.2-1B-Instruct at 0.18.

The benchmark repeatedly emphasizes that extraction-oriented SLMs finetuned for entity extraction or PII redaction are often weaker than smaller encoder models. In the introduction’s summary comparison, the best SLM reaches 0.45, the best BERT-based model 0.58, the best GLiNER model 0.47, and the OpenAI Privacy Filter 0.58. This indicates that strong redaction performance does not follow directly from extraction specialization.

Domain-wise, frontier models are strongest on unstructured categories, with mean scores reported around 0.80–0.85 for academic documents, up to 0.88 in financial documents, up to 0.91 in operations, and roughly 0.66–0.84 in medical documents depending on model. Structured domains remain harder, but frontier systems still dominate relative to the rest. For the OpenAI Privacy Filter, reported means include 0.62 on Code, 0.47 on Files, 0.23 on Logs, and 0.40 on Terminal.

The benchmark’s error picture is organized less as a formal taxonomy than through the properties of R-Score. Models lose performance through context-dependent mistakes, over-redaction of safe context, imprecise treatment of structured secrets, and poor span fidelity in generative extraction outputs. The paper explicitly notes that humans mainly lose points by redacting imprecisely and redacting more than necessary, and the same pattern plausibly extends to many model errors because false-positive penalties and boundary-sensitive character coverage are central to the metric.

6. Human evaluation, ambiguity, and benchmark interpretation

A major contribution of RedactionBench is its explicit study of human privacy judgments under the same contextual conditions it uses for model evaluation (Brynjólfsson et al., 17 Jun 2026). The user study involved 85 participants, all United States residents aged 25–65, each with at least a Bachelor’s degree, recruited on Prolific. The average pay rate was \$13.08/hour, and the median completion time was 27:32. Participants completed a tutorial, saw windows from up to 11 documents, could inspect surrounding greyed-out context, and could search the web if unsure.

The study used only the 7 unstructured categories, because structured categories often require domain expertise. Sampling was 8 documents from each of the 7 unstructured categories, for 56 documents total; each document was viewed by 16 users; each user was assigned 10–11 documents; windows were limited to roughly 25 lines; and the human-versus-model comparison used windows with starting R-Score = 0. Participants made only binary redact / do-not-redact decisions and were not shown the benchmark’s ternary labels.

The central finding is a sharp asymmetry across label classes. Mandatory entities were redacted by users 89.4% of the time, Contextual entities were redacted 47.7% of the time, and Gaps / safe text were left unredacted 94.1% of the time. The paper describes this as a “stark dichotomy”: strong consensus on mandatory redactions, strong consensus on safe preservation, and near-even disagreement on contextual entities.

The disagreement table sharpens that point. For mandatory entities, the paper reports $\mathbf{M}$8, $\mathbf{M}$9, $R$0 with interval $R$1, and $R$2. For contextual entities, it reports $R$3, $R$4, $R$5 with interval $R$6, and $R$7. For gaps, it reports $R$8, $R$9, $\mathbf{M}$0 with interval $\mathbf{M}$1, and $\mathbf{M}$2. The paper further notes that contextual disagreement is 1.80× that of mandatory entities and 3.22× that of gaps, while also cautioning that low per-tier $\mathbf{M}$3 for mandatory and gap units is influenced by the prevalence paradox, making pairwise disagreement $\mathbf{M}$4 more interpretable.

Mode-pooled human redactions on zero-start windows yield an overall mean R-Score = 0.77. Category means on the user-study split are 0.85 for Academic, 0.82 for Email, 0.78 for Financial, 0.70 for Government, 0.82 for Legal, 0.64 for Medical, 0.75 for Operations, and 0.77 overall across unstructured documents. On the 48-document user-study subset, frontier models are slightly above or at this human aggregate baseline: gpt-5.4 reaches 0.81, Qwen3.5-397B-A17B 0.80, claude-opus-4-6 0.79, GLM-5.1 0.77, and the Human aggregated baseline is 0.77.

These results justify the benchmark’s metric design. Mandatory redaction and safe preservation appear benchmarkable with relatively strong confidence, whereas contextual redaction is intrinsically subjective under the available evidence. R-Score’s asymmetric handling of contextual entities is therefore not an implementation detail but a direct response to measured human disagreement.

7. Position in the benchmark landscape and limitations

RedactionBench occupies a distinct place among recent privacy- and safety-related benchmarks. It is not a generic PII detection benchmark, not a red-teaming benchmark, and not a document-edit propagation benchmark, although all three neighboring areas are relevant for comparison (Brynjólfsson et al., 17 Jun 2026).

The closest neighboring benchmark in subject matter is “REDACT: A Systematically Controlled Multilingual Benchmark for Personal Information Detection” (Vats et al., 18 Jun 2026). REDACT evaluates personal information detection in unstructured text with 13,427 records, 324,078 entity annotations, 51 entity types, 25 languages, and 9 scripts, and it is highly relevant to the detection stage of redaction. However, REDACT is a detection benchmark rather than an end-to-end redaction benchmark: it evaluates whether systems identify PII spans and types, not whether they produce privacy-preserving character-level masking outputs or resolve contextual ambiguity in the sense used by RedactionBench.

A second related resource is “EditPropBench: Measuring Factual Edit Propagation in Scientific Manuscripts” (Kruthof, 3 May 2026). EditPropBench is not a privacy benchmark, but its distinction between direct targets, required downstream updates, and protected text is conceptually adjacent to redaction because redaction also has cascade structure. This suggests a possible document-level extension for future redaction benchmarks, though RedactionBench itself does not provide a formal treatment of co-reference or re-identifiability.

A third similarly named but substantively different resource is “RedBench: A Universal Dataset for Comprehensive Red Teaming of LLMs” (Dang et al., 7 Jan 2026). Despite the name overlap, RedBench is about LLM red teaming, attack prompts, refusal prompts, and safety evaluation under a 22-risk, 19-domain taxonomy; it is not a benchmark for privacy redaction.

Within its own scope, RedactionBench also states several limitations. Documents are evaluated in isolation, without user queries, conversation history, or system prompts. The benchmark does not yet handle query-aware redaction, offers no formal treatment of co-reference or re-identifiability, excludes structured documents from the user study because they require expertise, and does not provide a formal analysis of the fidelity or irregularity rate of synthetic entities. The paper further states that the benchmark is publicly released, but in the provided excerpt no explicit repository or leaderboard URL is given.

Taken together, these features make RedactionBench a benchmark of privacy semantics with character-level execution constraints. Its distinctive contribution is to separate what is unambiguously unsafe, what is contextually ambiguous, and what is safe to preserve, and then to score models in a way that reflects that separation. The resulting picture is technically specific: frontier tool-using LLMs currently perform best, smaller encoder models remain strong baselines, many extraction-oriented SLMs underperform relative to those baselines, and contextual privacy remains unresolved both computationally and socially (Brynjólfsson et al., 17 Jun 2026).