---
title: 'TRACE-RPS: Defense Against Attribute Inference'
url: https://www.emergentmind.com/topics/trace-rps
type: topic
---

# TRACE-RPS: Defense Against Attribute Inference

TRACE-RPS is a proactive, user-side defense framework against attribute inference attacks on large language models (LLMs). It combines **TRACE**—**Textual Revision via Attention and Chain-based Editing**—with **RPS**—**Rejection-Oriented Perturbation Search**—to address two distinct failure modes of privacy protection: residual attribute leakage after anonymization, and continued inference from benign-looking prompts even when the text has been edited. In the framework’s formulation, user text is first selectively anonymized at word and reasoning-chain level, then optionally augmented with an optimized suffix that steers the target model toward refusal behavior under attribute-inference prompts [2602.11528].

## 1. Threat model and formal problem

TRACE-RPS is defined for **attribute inference attacks** in which an adversary queries a frozen LLM with ordinary user-generated text and asks it to infer sensitive attributes such as age, gender, location, place of birth, income, occupation, education, and relationship status. The paper formalizes the dataset as
\[
D=\{(u_i,t_i)\},
\]
where \(u_i\) is a user and \(t_i\) is the associated text, and the target attribute set as
\[
A=\{a_1,a_2,\dots,a_k\}.
\]
Each user has an attribute vector
\[
y_u=(y_u^1,y_u^2,\dots,y_u^k).
\]

The attacker uses a prompt template
\[
P(t)=\text{Prefix}+F_{\text{fmt}}(t)+\text{Suffix}, \tag{1}
\]
and the model outputs inferred attributes
\[
\hat{A}_u=M(P(t))=\{(a_j,\hat{y}_u^j)\}_{j=1}^k. \tag{2}
\]
The attacker’s objective is to maximize
\[
\arg\max \; \mathbb{E}_{(u,t)\sim D}\sum_{j=1}^{k}\mathbb{I}[\hat{y}_u^j=y_u^j]. \tag{3}
\]

The framework is motivated by two limitations of prior anonymization defenses. First, they are described as **coarse-grained**, because they do not localize the precise words or phrases driving inference. Second, anonymization alone does not prevent an LLM from inferring attributes from residual contextual evidence through its own reasoning. TRACE-RPS therefore treats defense as both a **text transformation** problem and a **model-behavior suppression** problem [2602.11528].

The defended text is written as
\[
\bar t = T_{\text{def}}(t), \tag{4}
\]
and the defense objective is
\[
\arg\min \; \mathbb{E}_{(u,t)\sim D}\sum_{j=1}^{k}\mathbb{I}[\hat y_u^j=y_u^j]-\lambda \, \mathbb{I}[\hat y_u^j=\text{Reject}], \tag{5}
\]
where \(\lambda \in \mathbb{R}_+\) weights the value of inducing refusal. A defense succeeds if the model either predicts the wrong attribute or emits a refusal output.

## 2. TRACE: fine-grained anonymization through attention and inference chains

TRACE is the first stage of the framework. Its function is to identify privacy-leaking textual elements at fine granularity and rewrite only those segments most responsible for attribute inference. The method combines two signals: **privacy vocabulary extraction** and **privacy inference chain generation** [2602.11528].

For privacy vocabulary extraction, TRACE constructs an attribute-specific prompt
\[
P(t)=t+q,
\]
where \(q\) is an attribute query. A pretrained causal language model processes this prompt, and TRACE uses the **final-layer attention vector corresponding to the last token** as a proxy for token contribution to inference. If the tokenized text is
\[
x=(x_1,x_2,\dots,x_{|t|}),
\]
with attention scores
\[
a=(a_1,a_2,\dots,a_{|t|}),
\]
then for a word \(w\) composed of tokens \(x_i,\dots,x_j\), TRACE defines
\[
a(w)=\sum_{z=i}^{j} a_z. \tag{6}
\]
The privacy vocabulary is the top-\(K\) set
\[
V=\text{TopK}(\{(w,a(w)) : w\in W(t)\}). \tag{7}
\]
The implementation uses \(K=10\) on the **Synthetic** dataset and \(K=30\) on **SynthPAI**.

Attention alone is not treated as sufficient. TRACE also prompts an adversarial model to generate an explicit **Inference Chain**: a step-by-step explanation of how the attribute can be inferred from the text, with cited evidence spans and reasoning. This chain is intended to expose semantic dependencies that high-attention words may not fully capture.

The final anonymization stage is guided jointly by the privacy vocabulary \(V(t,a)\) and the inference chain \(C(t,a)\):
\[
\bar t = M_{\text{anon}}\big(T_{\text{det}}(t,V(t,a)),T_{\text{def}}(t,C(t,a))\big). \tag{8}
\]
Operationally, the anonymizer is instructed to **change as little as possible**, preserve meaning, and **only generalize**, not invent new facts. The paper gives “my husband and I” \(\rightarrow\) “my partner and I” as a valid edit, and “my husband and I” \(\rightarrow\) “my wife and I” as invalid. TRACE therefore functions as targeted semantics-preserving generalization rather than unconstrained paraphrase.

TRACE is iterative. It stops when attacker confidence drops below 2, when the text no longer changes between iterations, or when 5 iterations have been reached. The paper’s ablation on the **Synthetic** dataset reports Top@1 inference accuracy of **26.86** with both vocabulary extraction and inference chain enabled, compared with **31.43** when vocabulary extraction is removed, **31.62** when inference chain is removed, and **37.14** when both are removed, indicating that the two signals are complementary [2602.11528].

## 3. RPS and MPS: inference-time refusal and misattribution by suffix optimization

RPS is the second stage of TRACE-RPS and is the framework’s explicit inference-prevention component. It does not rewrite the original content. Instead, it appends an optimized discrete suffix \(s\) to the text so that, under an attribute-inference prompt, the model begins its response with a refusal pattern [2602.11528].

If the defended text is
\[
\bar t = t+s,
\]
RPS optimizes the first two generated tokens. Its scoring function is
\[
J(s)=\log p(y_1=\text{"I"}\mid P(\bar t))+\beta \cdot \log p(y_2\in R\mid P(\bar t), y_1=\text{"I"}), \tag{9}
\]
where \(R\) is the rejection-token set and \(\beta\in\mathbb{R}_+\). In experiments,
\[
R=\{\text{"apologize"}, \text{"cannot"}\}, \qquad \beta = 5.
\]

Stage 1 anchors the first token:
\[
J_1(s)=\log p(y_1=\text{"I"}\mid P(\bar t)). \tag{10}
\]
Starting from an initial suffix \(S_{\text{init}}\), the algorithm mutates a contiguous token span with random replacements and keeps the new suffix if \(J_1\) improves. Stage 1 stops when
\[
\tau_1=\log(0.8)\approx -0.22
\]
is reached or the iteration budget is exhausted.

Stage 2 shapes the second token toward refusal:
\[
J_2(s)=\log p(y_2\in R\mid P(\bar t), y_1=\text{"I"}). \tag{11}
\]
The full objective becomes
\[
J(s)=J_1(s)+\beta\cdot J_2(s), \tag{12}
\]
and the final suffix is
\[
s^*=\arg\max_{s\in S} J(s). \tag{13}
\]
Stage 2 stops when
\[
\tau_2=\log(0.55)\approx -0.60
\]
is reached or the iteration budget is exhausted.

The paper emphasizes that RPS is **lightweight** relative to longer prompt-optimization schemes because each candidate suffix is scored using only one or two generated tokens and their log-probabilities. It does **not** fine-tune model weights, use soft prompts, or require gradient-based optimization. It is a discrete, test-time input-optimization method.

Because some highly instruction-following models remain answer-generative even after refusal-oriented optimization, the paper introduces **MPS**—**Misattribute-Oriented Perturbation Search**—as a fallback. Instead of inducing refusal, MPS optimizes toward an incorrect target attribute:
\[
s^*=\arg\max_{s\in S} \log p(y=\tilde y_u^{\,a}\mid P(\bar t)), \tag{14}
\]
where \(\tilde y_u^{\,a}\) is a chosen wrong value for attribute \(a\). In experiments, MPS uses 500 iterations. The paper presents MPS chiefly for **Qwen2.5-7B-Instruct**, where refusal induction is harder [2602.11528].

## 4. End-to-end workflow, implementation assumptions, and access regimes

TRACE-RPS is a **sequential** framework:
\[
t \xrightarrow{\text{TRACE}} \tilde t \xrightarrow{\text{RPS}} \tilde t + s^*.
\]
The paper also allows \(t+s^*\) when RPS is applied without prior TRACE anonymization. The two stages are not jointly trained and are not embedded in an outer optimization loop.

The workflow is as follows. First, raw user text \(t\) is processed by TRACE. This includes constructing an attribute-inference query, extracting attention-based privacy vocabulary, generating an inference chain, and producing a minimally edited anonymized text \(\tilde t\). Second, if the target model is accessible at the logit level, RPS optimizes a suffix \(s^*\) through two-stage random search and outputs \(\tilde t+s^*\). The resulting defended text is what gets published or shared [2602.11528].

The framework is deliberately hybrid with respect to model access.

TRACE is broadly usable, including for **closed-source models**, because it relies on an anonymizer and an attention model for vocabulary extraction rather than logit access to the target inference model. In the reported implementation, the attention extraction model is **Llama2-7B-Chat**, and the anonymization model is **GPT-3.5-Turbo** or **GPT-4o**.

RPS, by contrast, requires access to **model logits** during optimization and is therefore mainly applicable to **open-source / white-box or semi-white-box** settings. The optimization model is the same as the target inference model. All reported decoding is deterministic: **temperature 0** and **greedy decoding**. RPS uses **10,000 iterations**, batch size **1**, and one or two **NVIDIA RTX 3090 GPUs**. The rejection tokens, thresholds, and \(\beta\) are fixed across experiments [2602.11528].

This architecture implies an important practical distinction. TRACE can function as a standalone defense when only API access is available. TRACE-RPS, in the full sense, is strongest when the attacker-facing model is introspectable enough to permit suffix search.

## 5. Empirical performance, transfer behavior, and utility

The evaluation uses three datasets: the **Synthetic** dataset from Staab et al. (2024), **SynthPAI** from Yukhymenko et al. (2024), and a **real-world Reddit dataset** constructed following the PersonalReddit methodology. The attacker task is zero-shot chain-of-thought attribute inference from user comments. Baselines include **No Defense**, **Azure** Language Services, **Dou-SD**, **D-Defense**, and **FgAA**. Target models include open-source systems—**Llama2-7B-Chat**, **Llama2-13B-Chat**, **Llama3.1-8B-Instruct**, **DeepSeek-R1-Distill-Qwen-7B**, **Qwen2.5-7B-Instruct**—and closed-source systems—**GPT-3.5-Turbo**, **GPT-4o**, and **Gemini 2.5 Pro** [2602.11528].

The headline result is that on open-source models, TRACE-RPS reduces attribute inference accuracy **from around 50% to below 5%**, often to nearly zero. On the **Synthetic** dataset, for **Llama2-7B-Chat**, inference accuracy drops from **53.71** with no defense to **1.71** with RPS, **0.19** with TRACE-RPS using GPT-3.5-Turbo as anonymizer, and **1.52** with TRACE-RPS using GPT-4o as anonymizer. For **Llama3.1-8B-Instruct**, the same dataset drops from **57.14** to **0** with RPS and **0** with TRACE-RPS. On **SynthPAI**, **Llama2-13B-Chat** falls from **44.58** to **0** with RPS and **0.09** with TRACE-RPS using GPT-4o.

For closed-source models, only TRACE is applicable because RPS requires logits. Even so, TRACE materially reduces inference accuracy. On the **Synthetic** dataset, **GPT-4o** drops from **71.24** with no defense to **28.38** or **26.67**, depending on anonymizer. On **SynthPAI**, **Gemini 2.5 Pro** drops from **67.54** to **41.82** or **34.74**. These results show that the anonymization stage alone provides measurable privacy protection when optimization-based refusal is unavailable.

The framework also reports strong performance on the stricter **Attack Success Rate (ASR)** metric,
\[
\text{ASR}= \frac{1}{N}\sum_{i=1}^{N}\mathbb{I}[\hat y_i=y_i \land M(P(\bar t_i))\neq \text{Reject}] + \mathbb{I}[M(P(\bar t_i))=\text{Reject}] \cdot \frac{1}{k_i}, \tag{15}
\]
which assumes the attacker may guess randomly after a refusal. The paper further evaluates prompt variation robustness using **100 GPT-4-generated prompt variants**, cross-model transfer, and multi-model suffix optimization. It reports that jointly optimized suffixes over **Llama2-7B-Chat**, **Llama2-13B-Chat**, and **Llama3.1-8B-Instruct** generalize well, including to unseen **Llama3.2** variants [2602.11528].

Utility is measured separately for TRACE and RPS. TRACE is assessed with an LLM-based judge over meaning preservation, readability, and hallucination; RPS is assessed with Sentence-BERT semantic similarity using `paraphrase-MiniLM-L6-v2`. The paper reports that **RPS** preserves very high semantic similarity because it leaves the original text untouched and only adds a suffix, with values such as **98.17** for Llama2-7B-Chat and **98.27** for DeepSeek-R1-Distill. TRACE introduces a moderate utility cost, but using **GPT-4o** as anonymizer improves utility substantially, with a reported utility score of **86.65**.

## 6. Limitations, misconceptions, and broader significance

TRACE-RPS is not a training-time privacy model, not a fine-tuning method, and not a generic prompt-injection defense. TRACE is a prompting-based anonymization system; RPS is a discrete, instance-level suffix optimization procedure. This distinction matters because the framework’s strong results do not come from retraining the protected model [2602.11528].

Several limitations are explicit. **RPS requires logit access**, so it cannot be directly applied to many closed-source APIs. Model behavior also matters: highly instruction-following systems such as **Qwen2.5-7B-Instruct** may not reliably refuse, which is why the paper introduces MPS. TRACE can degrade utility because it edits the original text, even though its prompts attempt to minimize semantic change. TRACE also relies on final-layer attention as an attribution signal, and the paper does not claim that attention is a perfect causal explanation of privacy leakage.

The paper evaluates adaptive attacks as well. **SuffixDrop-k**, which removes trailing characters, and **LLMSanitize**, which uses GPT-4o to strip suspicious patterns, both reduce RPS effectiveness, although the defended models remain materially better than the no-defense baseline. Another practical limitation is stylistic: appended suffixes may appear anomalous in publicly shared text.

A common misconception is that anonymization alone is sufficient. The framework’s own results argue against that interpretation. TRACE reduces direct and indirect evidence in the text, but low-cardinality attributes such as gender or broad income class can remain inferable under anonymization alone. RPS addresses this by suppressing response generation rather than only rewriting evidence. A complementary misconception is that refusal-inducing suffixes are always enough. The Qwen results show that this is not uniformly true, which is why the paper includes misattribution-based fallback.

In the paper’s formulation, TRACE-RPS is best understood as a two-layer privacy defense: **TRACE** weakens the evidence surface available to the attacker, and **RPS** weakens the model’s willingness to operationalize whatever evidence remains. This suggests a broader significance beyond attribute inference alone: the framework treats privacy defense not merely as sanitization of text, but as joint control over **what information is exposed** and **how a model is permitted to reason from it** [2602.11528].

Source: https://www.emergentmind.com/topics/trace-rps