PRvL: PII Redaction via Language Models

• PRvL is an open-source suite of fine-tuned language models designed for precise and scalable PII redaction in unstructured text.
• It leverages diverse model families—dense LLMs, small LMs, and mixture-of-experts—to compare redaction accuracy, leakage, and computational cost.
• The system supports customizable, secure deployment with strategies like instruction tuning and adapter training to optimize performance and privacy.

PRvL, expanded in the report as “PII Redaction via LLMs,” is an open-source suite of fine-tuned models and evaluation tools for general-purpose PII Redaction. It is framed as a comprehensive analysis of LLMs as privacy-preserving PII Redaction systems, with evaluation spanning redaction performance, semantic preservation, PII leakage, latency, and computational cost. The suite is built entirely on open-source LLMs, supports multiple inference settings, is designed to be easily customized for different domains, and is fully operable within secure, self-managed environments, so that data owners can perform redactions without relying on third-party services or exposing sensitive content beyond their own infrastructure (Garza et al., 7 Aug 2025).

1. Scope, motivation, and system boundaries

Redacting Personally Identifiable Information from unstructured text is presented as critical for ensuring data privacy in regulated domains. Earlier approaches are described as relying on rule-based systems and domain-specific Named Entity Recognition models, with the limitation that these methods fail to generalize across formats and contexts. Recent advances in LLMs are positioned as a promising alternative because they perform strongly on tasks requiring contextual language understanding, including the redaction of PII in free-form text (Garza et al., 7 Aug 2025).

Within that problem setting, PRvL studies how architectural and training choices affect PII redaction. The report explicitly compares dense LLMs, small LLMs, Mixture-of-Experts systems, long-range models, structured state models, and a BERT-NER baseline. A central boundary condition is that no additional domain pretraining was performed: each pre-trained LLM is adapted via parameter-efficient fine-tuning or instruction-tuning, and the only “domain” signal comes from the PII-annotated corpus AI4Privacy used in downstream adaptation. This places the emphasis on adaptation strategy rather than on corpus-scale continued pretraining.

A plausible implication is that PRvL is intended less as a single-model claim than as a controlled framework for comparing redaction configurations under deployment-relevant constraints such as self-hosting, latency, and leakage.

2. Model families and adaptation procedures

The evaluated model families include dense decoder-only LLMs, small LMs, Mixture-of-Experts models, long-range models, structured state models, and a token-classification baseline. Open-source architectures are deployed via Docker on NVIDIA RTX 6000 GPUs with 48 GB memory (Garza et al., 7 Aug 2025).

Family	Models	Notes
Dense LLMs	LLaMA 3.1-8B; GPT-4	decoder-only
Small LMs	LLaMA 3.2-3B; T5	3B and 11B
Mixture-of-Experts	Mixtral	sparse activation
Long-Range Models	DeepSeek-Q1; OpenAI-o3	extended or long context
Structured State Models	FalconMamba	sub-quadratic sequence modeling
NER baseline	BERT-NER	token classification

The fine-tuning process uses supervised sequence-to-sequence training for generative models and token-classification for BERT-NER. Training data are parallel pairs $(x,y)$ in which $x$ is original text with PII and $y$ replaces each annotated span $(s_j,e_j,t_j)$ by a placeholder \<TYPE\>. For generative models, the objective is cross-entropy:

$$\mathcal{L}(\theta) \;=\; -\frac{1}{N}\sum_{i=1}^N\sum_{t=1}^{|y^{(i)}|}\log p_\theta\bigl(y^{(i)}_t\mid y^{(i)}_{<t},\,x^{(i)}\bigr)\,.$$

For token-classification, the objective is binary cross-entropy per token:

$$\mathcal{L}(\theta) \;=\; -\frac{1}{M}\sum_{m=1}^{M}\bigl[y_m\log p_m+(1-y_m)\log(1-p_m)\bigr],$$

where $y_m\in\{0,1\}$ indicates a PII token.

Adapter training is performed with LoRA modules injected into attention layers at down-rank $r=8$, updating much less than $1\%$ of weights. Optimization uses AdamW with linear warmup over $5\%$ of steps, cosine decay, and early stopping on validation loss. Instruction-tuning uses the same hidden base plus LoRA and is trained on a small set of in-context examples per instruction template. The report’s comparative results later show that instruction tuning exceeds full fine-tuning on accuracy, type fidelity, and privacy, which suggests that PRvL treats prompt-conditioned adaptation as a first-class redaction regime rather than a secondary convenience (Garza et al., 7 Aug 2025).

3. PII formalization and evaluation methodology

PRvL uses a standardized set of approximately 30 label types. Examples listed in the report include [GIVENNAME1], [LASTNAME1…3], [EMAIL], [TEL], [IP], [SOCIALNUMBER], [DRIVERLICENSE], [STREET], [CITY], [COUNTRY], [POSTCODE], [GEOCOORD], [DATE], and [TIME] (Garza et al., 7 Aug 2025). This taxonomy operationalizes redaction as structured replacement rather than simple deletion, which is important for preserving document semantics and downstream usability.

Redaction performance is evaluated in two settings. In Span-Correct, only span match matters. In Label-Exact, both span and type must match. The report also evaluates semantic preservation using BERTScore in its F1 variant and cosine similarity of sentence embeddings. This combination separates redaction success from the preservation of non-PII content, which is especially relevant for systems intended to produce usable sanitized text rather than merely detect sensitive spans.

For privacy leakage, PRvL defines the SPriV score. Let $x$0 be generated tokens, and let $x$1 be the indices of true PII in the reference. Then

$x$2

and

$x$3

where $x$4 means perfect masking. This makes leakage an output-level quantity tied directly to unmasked ground-truth PII rather than only to classification error. A plausible implication is that SPriV is particularly useful when generative models produce fluent rewrites whose token-level errors may not be well summarized by standard extraction metrics alone (Garza et al., 7 Aug 2025).

4. Datasets, inference modes, and systems characteristics

The experimental setup uses AI4Privacy-300K and AI4Privacy-500K. AI4Privacy-300K contains 300,000 synthetic sentences in English, Spanish, and Italian with rich, contextually embedded PII spans. AI4Privacy-500K contains 500,000 similar English examples. Each example is annotated with span indices $x$5 and type $x$6, and the held-out test set contains 1,000 examples per language (Garza et al., 7 Aug 2025).

PRvL supports several inference modes. Vanilla generation uses direct decoding with standard or nucleus sampling. Retrieval-Augmented Generation retrieves top-$x$7 exemplar redactions, typically with $x$8, using a dense retriever, concatenates them to the input, and decodes. The deployment layer supports both batch and streaming, token-by-token pipelines, all self-hosted. This design connects model evaluation directly to operational requirements such as latency and infrastructure control.

Latency and throughput are reported for 150-token generation on RTX 6000 (48 GB):

Model	Latency	Throughput
T5 (FT)	3000 ms	50 tok/s
LLaMA 3.2-3B (FT)	1667 ms	90 tok/s
LLaMA 3.1-8B (FT)	1648 ms	91 tok/s
DeepSeek-Q1 (FT)	1456 ms	102 tok/s
Mixtral (FT)	5000 ms	30 tok/s
FalconMamba (SSM)	2500 ms	60 tok/s
OpenAI-o3 (API)	1546 ms	97 tok/s
GPT-4 (API)	7895 ms	19 tok/s

These figures matter because the PRvL analysis is not restricted to redaction quality; it explicitly compares quality against latency and computational cost. This suggests an evaluation philosophy in which privacy-preserving redaction is treated as a systems problem as much as a modeling problem.

5. Empirical findings and comparative behavior

On the English AI4Privacy-300K test set under Span-Correct evaluation, the baseline BERT-NER achieves Acc 0.986 / P 0.907 / R 0.982. Among fine-tuned generative systems, LLaMA 3.1-8B records 0.986 / 0.915 / 0.969, T5 records 0.883 / 0.727 / 0.830, DeepSeek-Q1 records 0.993 / 0.963 / 0.978, and Mixtral records 0.988 / 0.940 / 0.957. Under instruction tuning, LLaMA 3.1-8B IT reaches 0.992 / 0.975 / 0.962, and DeepSeek-Q1 IT reaches 0.994 / 0.973 / 0.981. RAG yields modest gains in recall but lower precision; the example reported is LLaMA 3.1-8B RAG at 0.930 / 0.827 / 0.717 (Garza et al., 7 Aug 2025).

Under Label-Exact evaluation, DeepSeek-Q1 FT records Acc 0.992 / P 0.925 / R 0.953 / Mislabels 3033, while DeepSeek-Q1 IT records Acc 0.994 / P 0.945 / R 0.960 / Mislabels 3047. The report also states that instruction tuning reduces type errors by approximately 10–15%. Sequence-level evaluation shows that T5 (FT) attains the best ROUGE-1/2/L at 0.940/0.857/0.934, while DeepSeek-Q1 (FT) and DeepSeek-Q1 (IT) attain the best BLEU at 0.906 and 0.908, respectively. The lowest SPriV is achieved by DeepSeek-Q1 FT and IT at 0.002, described as almost no leakage.

Several comparative conclusions are explicit. Instruction tuning is better than full fine-tuning on accuracy, type fidelity, and privacy. Smaller models such as DeepSeek-Q1 7B and LLaMA 3B can rival 8–56B models when instruction-tuned. RAG improves structure but risks over-mask drift and slight leakage. These results directly counter a common assumption that parameter count alone determines redaction quality. They also show that retrieval support is not uniformly beneficial: structural gains may coexist with increased leakage, reported as SPriV around 0.02–0.03 for RAG variants.

6. Deployment patterns, customization, and privacy-risk controls

PRvL’s deployment guidance is domain-specific. For high-accuracy and low-latency settings such as chatbots, the report recommends DeepSeek-Q1 (IT) or LLaMA 3.1-8B (FT). For edge or cost-sensitive settings, it recommends LLaMA 3.2-3B (IT). For documents longer than 32K tokens, it recommends long-range models such as DeepSeek-Q1 or structured state models such as FalconMamba with chunked inference (Garza et al., 7 Aug 2025).

The self-managed integration path is explicitly containerized. PRvL provides Docker images for each model and a simple REST API with the interface “POST raw text → JSON with masked text & PII span metadata.” It can optionally attach a vector database such as Faiss for a RAG exemplar index. This operational profile is consistent with the suite’s stated emphasis on secure, self-managed environments and compliance-sensitive deployment.

Customization of new PII categories is described as a four-step procedure: extend the taxonomy with a new label NEWTYPE; generate approximately 1,000 synthetic examples containing NEWTYPE and annotate spans; instruction-tune LoRA adapters on mixed original and new data; and deploy updated adapter weights without modifying the base model. This procedure suggests that the redaction stack is designed for incremental adaptation at the adapter layer rather than for repeated end-to-end model replacement.

Privacy and risk assessment in PRvL includes SPriV as a direct measure of unmasked ground-truth PII and membership inference probes on model outputs. The mitigation strategies listed are data curation to remove high-risk examples from pretraining or fine-tuning corpora, differential privacy on adapter training, inference-time filtering that disallows generation of raw PII patterns via a blocklist, and secure enclaves or encrypted compute for self-hosted inference. The release includes open-source code, model weights, and evaluation scripts at https://anonymous.4open.science/r/PRvL-C1BF, with Apache 2.0 for code and permissive licensing for model weights. These release choices reinforce the system’s orientation toward reproducibility and controlled deployment rather than dependence on external API-only infrastructure (Garza et al., 7 Aug 2025).