Compact Sanitization Models

• Compact sanitization models are algorithmic frameworks that precisely remove or obfuscate sensitive data via minimal, context-aware interventions, ensuring strong privacy and high utility.
• They leverage information theory and combinatorial optimization to establish quantifiable privacy thresholds while preserving the structural, frequency, and sequential integrity of the original data.
• Used in document redaction, string obfuscation, and LLM sanitization, these models enable compliance with privacy regulations by balancing minimal modifications with robust data protection.

Compact sanitization models are algorithmic frameworks designed to remove or obfuscate sensitive content from textual, sequential, or model-based data in a manner that guarantees strong privacy while preserving maximal utility, structure, or knowledge relevant to non-sensitive patterns. Unlike coarse redaction or overgeneralized suppression, these models explicitly target minimal, contextually defined interventions that are provably necessary and sufficient for privacy, often leveraging information theory, combinatorial optimization, or low-rank adaptations. They have been formalized for document sanitization, string pattern obfuscation, and model knowledge editing, providing a foundation for privacy compliance and operational efficiency across domains where sensitive data must be exchanged or released.

1. Formal Definitions and Fundamental Principles

Compact sanitization models are grounded in precise formalizations that quantify both privacy risk and utility preservation. The C-sanitized model for document redaction operates on the basis of information content (IC) and pointwise mutual information (PMI) between terms and sensitive entities. Let $D$ be the input document, $C = \{c_1, \ldots, c_k\}$ the set of sensitive entities, $K$ the domain knowledge (corpus or taxonomy), with $I(t) = -\log p(t)$ the IC of a term $t$ and $$\operatorname{PMI}(c;T) = \log [p(c \land T) / (p(c)p(T))]$$ for (multi-)sets $T\subseteq D$.

An $(\alpha, C)$-sanitized output $D^\prime$ is defined by the constraint:

$$\forall c \in C, \forall t \in D^\prime, ~ \operatorname{PMI}(c;t) \le \frac{I(c)}{\alpha}$$

and likewise for any term set $T \subseteq D'$. $\alpha \ge 1$ is a tunable parameter trading off privacy versus utility. This formalism ensures that $D^\prime$ may not disclose sensitive concepts beyond a quantifiable threshold, conferring explicit a-priori privacy guarantees (Sanchez et al., 2014).

In combinatorial string sanitization, let $S \in \Sigma^n$ be the original string, $P_s \subset \Sigma^k$ the set of sensitive patterns, and $P_{ns} = \Sigma^k \setminus P_s$ the non-sensitive patterns of length $k$. The minimal-length and minimal-edit-distance settings require constructing a string $S'$ such that:

• No $p \in P_s$ appears in $S'$
• For all $p \in P_{ns}$, frequency is preserved: $freq_{S'}(p) = freq_S(p)$
• Total or partial order of non-sensitive patterns is preserved
• In minimal-edit-distance sanitization, also $\min \operatorname{ED}(S, S')$ (Bernardini et al., 2019)

For model sanitization, particularly with LLMs, compactness is measured in terms of targeted, low-overhead factual forgetting. LoRA-based sanitization learns low-rank corrections $\Delta W = AB$ (with $$A \in \mathbb{R}^{d \times r}, B \in \mathbb{R}^{r \times k}, r \ll d, k$$) injected only into multi-layer perceptron (MLP) sublayers, causing LLMs to emit a harmless fallback (e.g., “I don’t know”) for queries about forbidden facts, with minimal effect on general performance (Ishibashi et al., 2023).

2. Algorithmic Approaches for Compact Sanitization

C-sanitized document sanitization follows a linear-time procedure:

1. Initialize $D' \leftarrow D$.
2. For each term $t$ in $D$:
   • Compute $\operatorname{PMI}(c; t)$ for all $c \in C$. If any $\operatorname{PMI}(c;t) > I(c)/\alpha$, mark $t$ risky.
   • For risky $t$, identify the least-general taxonomic ancestor $g(t)$ such that all $\operatorname{PMI}(c;g(t)) \le I(c)/\alpha$.
   • Replace $t$ with $g(t)$. If no safe generalization exists, redact $t$.
3. Repeat until all risky terms are addressed; return $D'$.

String sanitization employs distinct algorithms:

• TFS-ALGO: Produces the shortest output preserving total order and frequency of all non-sensitive $k$-grams, inserting separators (#) to mask sensitive spans while greedily merging overlapping safe substrings.
• PFS-ALGO: Relaxes order preservation to partial order, using de Bruijn graph path covers over $k-1$-gram overlaps for further compactness.
• MCSR-ALGO: Replaces or deletes separators with alphabet symbols to avoid leaking sensitive pattern positions and to prevent the emergence of spurious or implausible substrings, solved as a multiple-choice knapsack instance.

For LLMs, LoRA-based knowledge sanitization fine-tunes only small-rank adapters in MLP layers. Training uses a curated mix of forbidden QA pairs (whose true answers are replaced with a fixed, harmless phrase) and retained pairs, optimizing cross-entropy loss over this union. The resultant low-rank adapters are compact (typically $\ll 0.1\%$ of model parameters), inserted at inference for on-demand sanitization (Ishibashi et al., 2023).

3. Complexity, Compactness, and Utility Considerations

The defining property of compact sanitization models is that they minimize transformations while imposing mathematically certified privacy constraints.

• C-sanitized: Algorithmic time is $O(n k h)$, where $n$ is the count of unique terms, $k = |C|$, and $h$ the maximum taxonomy height; only risky terms are transformed, and the most specific generalization is always preferred, ensuring that $>90\%$ of terms remain intact in practice.
• String algorithms:
  • TFS-ALGO: $O(k n)$, provably outputs the minimal-length string consistent with order and counts.
  • PFS-ALGO: $O(n)$ additional merge/composition phase; outputs are up to $30\%$ shorter than TFS while honoring partial order.
  • MCSR-ALGO: NP-hard in $k$, practical for moderate problem sizes using standard dynamic programming or approximation.
  • ETFS-ALGO: $O(k|\Sigma| n^2)$ (quadratic), optimal for minimal-edit-distance sanitization but less scalable.
• LLM LoRA sanitization: Adapter parameter overhead is minimal (e.g., for LLaMA-7B, $\sim 3.8$M parameters or $0.055\%$ of total), with no measurable increase in inference cost and negligible effect on model perplexity or general accuracy.

Empirical analyses confirm high utility retention: sanitized documents or models retain most task-relevant information/functionality, with the majority of edits limited to localized, provably "risky" data regions or weights. As the privacy parameter $\alpha$ increases, recall (privacy) rises and output utility may decrease, offering a strictly controlled trade-off.

4. Comparative Evaluation and Empirical Performance

Compact sanitization models significantly outperform ad hoc and non-semantic baselines in producing minimal, utility-preserving sanitized outputs with explicit privacy guarantees.

For document redaction, C-sanitized evaluation against manually redacted Wikipedia articles demonstrates:

• At $\alpha=1.0$: Recall very low (4–8.6%), precision maximal (100%), and near-maximal utility ($\sim$97%).
• At $\alpha=2.0$: Recall increases dramatically (80–96%), F1 improves ($\sim$83–88%), utility reduced to 61–85%.
• Term replacement with generalizations recovers 20–50 percentage points of utility over outright redaction.
• Only 20–30% of terms are typically affected, affirming compactness (Sanchez et al., 2014).

In string pattern sanitization, TFS and PFS deliver optimal outputs (length and order/frequency preservation) in linear time; ETFS provides optimal edit distance results but at quadratic cost (Bernardini et al., 2019).

For LLMs, LoRA-based sanitization drives target forgetting accuracy (model outputs forbidden facts) from 74.0% to 7.0% in LLaMA, while model retention accuracy (non-sensitive questions) is preserved (49.9% to 49.8%). General-task performance alters by at most 1 percentage point, establishing the targeted and minimal impact of the approach. Alternative knowledge editing baselines, such as negative gradient or ROME, yield either catastrophic utility losses or insufficient factual forgetting (Ishibashi et al., 2023).

5. Relationship to Other Privacy Models

Compact sanitization is distinct from population-level statistical anonymization (k-anonymity, ℓ-diversity, differential privacy) and heuristic pattern suppression:

• k-Anonymity/ℓ-Diversity: Designed for structured relational data; lacks semantic context awareness; often entails global attribute generalization, which is not minimal for text or sequence data.
• Differential Privacy: Provides strong aggregate guarantees by adding calibrated noise; unsuitable for direct application on free text, high semantic distortion.
• Ad-hoc redaction/sanitization: No a-priori guarantee, heavy manual curation, risk of over- or under-redacting.

Compact models such as C-sanitized enforce a mathematically bounded information leakage per term or factual string, explicitly adapting to semantic, sequential, or model settings. The information-theoretic and combinatorial guarantees enable compliance with legal frameworks (e.g., HIPAA, EU Directive 95/46/EC) and operational policies (Sanchez et al., 2014, Bernardini et al., 2019).

6. Limitations and Directions for Future Research

Notable constraints and open research directions include:

• Group-term and multi-pattern inference: Existing models often overlook complex multi-term indirect disclosures; fine-grained multi-pattern PMI control is computationally hard.
• Probability estimation: IC/PMI estimation using Web frequencies is subject to lexical ambiguity; improvements require word sense disambiguation or search engine result fusion.
• Knowledge base coverage: Limited domain taxonomies may force excessive redaction; integration of richer ontologies is needed.
• Hyperparameter calibration: Automated selection of disclosure parameter $\alpha$ or, in LLM contexts, forbidden knowledge sets remains challenging.
• Model sanitization: Current LoRA-based approaches are limited to facts that can be exhaustively enumerated; open-ended sensitivity or analogical inferences in deep models represent ongoing challenges.

Promising extensions involve hierarchical multi-term masking, probabilistic utility-privacy visualization tools, interactive noise addition for real-time privacy, and adversarial prompt detection in LLMs (Sanchez et al., 2014, Ishibashi et al., 2023). For algorithmic string sanitization, improvements in approximate regular expression matching and practical heuristics for large-scale knapsack formulations are ongoing areas of research (Bernardini et al., 2019).

7. Practical Implications and Compliance

Compact sanitization models operationalize privacy in sensitive text, sequential, and model-based data releases. By providing a-priori leakage bounds, explicit trade-off control, and minimal perturbation, these methods facilitate release workflows that conform with statutory de-identification clauses and regulatory guidelines. For example, minimal generalization of “AIDS” to “viral disease” or of “Los Angeles County” to “administrative region” can satisfy Safe Harbor requirements by algorithmic construction, without manual intervention (Sanchez et al., 2014).

In model deployments, LoRA-adapted sanitization offers organizations a compact, low-cost, and robust mitigation against data extraction and leakage threats, while maintaining pretrained model behavior for all non-targeted knowledge (Ishibashi et al., 2023). In string and genomic data contexts, provably minimal and plausibility-preserving pattern surgery ensures that sanitized outputs remain fit for downstream analytic purposes (Bernardini et al., 2019).