DP-RFT: Differentially Private Reinforcement Fine-Tuning

• DP-RFT is a framework that fine-tunes large language models using reinforcement learning and DP-noised reward signals to rigorously enforce differential privacy.
• It integrates PPO and DP-SGD techniques to balance model utility with strong privacy guarantees during the fine-tuning process.
• Empirical results demonstrate that DP-RFT narrows performance gaps with non-private methods, achieving notable fidelity gains under strict privacy budgets.

Differentially Private Reinforcement Fine-Tuning (DP-RFT) encompasses a family of techniques for fine-tuning LLMs and related neural architectures under rigorous differential privacy (DP) constraints, typically in reinforcement learning (RL) or RL from human feedback (RLHF) settings. DP-RFT addresses the challenge of maximizing model utility—such as the quality of synthetic text or alignment with preferences—while ensuring the formal privacy of sensitive training data. Recent work has unified the field by introducing DP-protected signal pathways, refined privacy accounting via composition, reinforcement-based or reward-centric optimization objectives, and comprehensive empirical and theoretical analysis (Xu et al., 20 Feb 2026, Cho et al., 23 Mar 2026, Khadangi et al., 30 Jul 2025).

1. Formal Problem Setting

The primary setting for DP-RFT involves a sensitive private corpus $\mathcal{D}_{\mathrm{priv}} = \{d_1, \dots, d_n\}$, over which direct access to individual examples is forbidden (the “eyes-off” constraint). The aim is to learn a generative model (typically an LLM parameterized by $\theta$) such that, for prompts $p \sim P_\mathrm{prompt}$, the synthetic distribution $x \sim \pi_\theta(\cdot|p)$ closely matches $\mathcal{D}_{\mathrm{priv}}$ in statistical properties relevant for downstream tasks, while the entire mechanism

$M: \mathcal{D}_{\mathrm{priv}} \mapsto \theta$

satisfies $(\epsilon,\delta)$-differential privacy for the data. The learning criterion is to maximize the expected quality of synthetic samples, as measured by a DP-noised reward function $r(x)$:

$$R(\theta) = \mathbb{E}_{p \sim P_\mathrm{prompt},\, x \sim \pi_\theta(\cdot|p)}[r(x)].$$

The general approach replaces direct training on private instances with RL fine-tuning guided by DP-protected reward signals, effectively decoupling privacy from most of the model’s learning dynamics (Xu et al., 20 Feb 2026, Cho et al., 23 Mar 2026).

2. DP-Protected Reward Signal Construction

DP-RFT implementations rely on privatized reward signals that enable on-policy RL updates while ensuring DP for the underlying dataset. A common approach uses a nearest-neighbor voting mechanism: for a candidate $x$, each $\theta$0 computes a similarity (e.g., cosine distance in embedding space), clipped to $\theta$1, producing $\theta$2. Summing over the corpus gives $\theta$3, which is then perturbed with Gaussian noise:

$\theta$4

The $\theta$5-sensitivity is $\theta$6. Each privatized reward query corresponds to $\theta$7-DP. To obtain the total privacy budget, one applies advanced composition or moments accountant techniques, yielding an overall $\theta$8 after $\theta$9 queries (Xu et al., 20 Feb 2026).

To discourage reward hacking, an auxiliary reward $p \sim P_\mathrm{prompt}$0 ensures adherence to prompts, e.g., via LLM judgments, and the final scalar reward is:

$p \sim P_\mathrm{prompt}$1

where $p \sim P_\mathrm{prompt}$2.

Other DP-RFT frameworks target RLHF: privacy is imposed only on reward learning, typically through DP-SGD on a linear reward head trained to fit human preferences (pairwise comparison data). The policy itself is obtained by post-processing the private reward. Since policy derivation does not access raw data, the overall process remains $p \sim P_\mathrm{prompt}$3-DP (Cho et al., 23 Mar 2026).

3. Reinforcement Learning and Policy Optimization

DP-RFT typically leverages policy-gradient-based RL, with Proximal Policy Optimization (PPO) as a canonical update rule. For each batch:

• Prompts $p \sim P_\mathrm{prompt}$4 are sampled,
• On-policy synthetic outputs $p \sim P_\mathrm{prompt}$5 are generated,
• DP-noised rewards $p \sim P_\mathrm{prompt}$6 are computed,
• Discounted returns $p \sim P_\mathrm{prompt}$7 and advantages $p \sim P_\mathrm{prompt}$8 (with $p \sim P_\mathrm{prompt}$9 a value network) are estimated.

The PPO surrogate loss is:

$x \sim \pi_\theta(\cdot|p)$0

where $x \sim \pi_\theta(\cdot|p)$1 and $x \sim \pi_\theta(\cdot|p)$2 scales the KL penalty. The total loss sums the PPO objective and value-function loss. After each PPO batch, privacy spend is updated, and the noise multiplier $x \sim \pi_\theta(\cdot|p)$3 is chosen so the composition over all batches respects $x \sim \pi_\theta(\cdot|p)$4.

In RLHF settings with decoupled reward modeling (Cho et al., 23 Mar 2026), DP-SGD trains a linear reward model $x \sim \pi_\theta(\cdot|p)$5, and the final policy is obtained by Boltzmann reweighting over samples from a reference policy $x \sim \pi_\theta(\cdot|p)$6, e.g.:

$x \sim \pi_\theta(\cdot|p)$7

Sampling or top-$x \sim \pi_\theta(\cdot|p)$8 selection over proposals (post-processing) does not incur further privacy cost.

4. Theoretical Privacy Guarantees and Utility Bounds

DP-RFT frameworks are supported by rigorous privacy accounting and utility guarantees. Privacy composition across $x \sim \pi_\theta(\cdot|p)$9 batches with the Gaussian mechanism follows advanced composition or moments accountant approaches (Abadi et al. 2016; Balle & Wang 2018), ensuring cumulative $\mathcal{D}_{\mathrm{priv}}$0-DP. In decoupled reward modeling (Cho et al., 23 Mar 2026), the only source of privacy loss is reward learning; since the policy is a function of the reward model, post-processing preserves the privacy guarantee.

Sample complexity and utility bounds have been established for DP-RFT. With linear reward realizability and a fixed feature map, the suboptimality gap $\mathcal{D}_{\mathrm{priv}}$1 for a policy $\mathcal{D}_{\mathrm{priv}}$2 derived from a DP-SGD-trained reward model decomposes as:

$\mathcal{D}_{\mathrm{priv}}$3

where $\mathcal{D}_{\mathrm{priv}}$4 is feature dimension, $\mathcal{D}_{\mathrm{priv}}$5 sample size, $\mathcal{D}_{\mathrm{priv}}$6 feature norm bound, $\mathcal{D}_{\mathrm{priv}}$7 regularization temperature, and $\mathcal{D}_{\mathrm{priv}}$8 the per-sample privacy budget. The first (non-private) term dominates when $\mathcal{D}_{\mathrm{priv}}$9; otherwise, the privacy-induced term prevails. Lower bounds demonstrate near-optimality in intermediate regimes (Cho et al., 23 Mar 2026).

5. Empirical Evaluation and Performance Benchmarks

Empirical results demonstrate significant improvements in fidelity and utility for synthetic data generation and RLHF alignment under strict privacy budgets.

For synthetic text generation (Xu et al., 20 Feb 2026), DP-RFT closes most of the gap to DP-FT (DP-SGD fine-tuning, which requires direct data access) while strictly enforcing the privacy boundary. The table below summarizes next-token prediction accuracy (↑ better) on PubMed using GPT-2, under several privacy budgets ($M: \mathcal{D}_{\mathrm{priv}} \mapsto \theta$0):

Method	$M: \mathcal{D}_{\mathrm{priv}} \mapsto \theta$1	$M: \mathcal{D}_{\mathrm{priv}} \mapsto \theta$2	$M: \mathcal{D}_{\mathrm{priv}} \mapsto \theta$3	$M: \mathcal{D}_{\mathrm{priv}} \mapsto \theta$4
QWEN	33.04	33.04	33.04	33.04
Aug-PE	33.91	34.10	33.57	34.55
DP-RFT	38.31	37.37	37.19	36.75
DP-FT	40.33	42.33	42.06	41.64

On BERT Small, DP-RFT closes $M: \mathcal{D}_{\mathrm{priv}} \mapsto \theta$550% of the gap between Aug-PE and DP-FT at $M: \mathcal{D}_{\mathrm{priv}} \mapsto \theta$6. Fidelity gains are confirmed by increases in embedding similarity metrics and reductions in Fréchet Inception Distance, as well as downstream accuracy improvements of 4–35% over Aug-PE under tight budgets.

For RLHF (Cho et al., 23 Mar 2026), DP-RFT with a private reward model outperforms DP-DPO and DP-RLHF in suboptimality gap and alignment. On Anthropic HH-RLHF with Gemma-2B-IT, DP-RFT achieves reward accuracy of 58.9–59.7% (vs. 51.9–53.0% for baselines) across $M: \mathcal{D}_{\mathrm{priv}} \mapsto \theta$7.

Reinforcement-learned privacy dial control further improves sample efficiency in DP-SGD-based fine-tuning, decreasing perplexity by 5.4% on average and reducing gradient-update requirements by 71% relative to strong baselines (Khadangi et al., 30 Jul 2025). Privacy attacks (membership inference, canary extraction) show no increase in leakage relative to baselines.

6. Trade-offs, Limitations, and Practical Considerations

DP-RFT exposes important trade-offs:

• Privacy–Utility Trade-off: As $M: \mathcal{D}_{\mathrm{priv}} \mapsto \theta$8 tightens, model quality decreases. DP-RFT methods degrade more gracefully than non-interactive alternatives (e.g., Aug-PE), retaining 85% of baseline performance at $M: \mathcal{D}_{\mathrm{priv}} \mapsto \theta$9 (Xu et al., 20 Feb 2026).
• Reward Hacking Mitigation: Use of auxiliary prompt adherence rewards and diverse prompts is essential to prevent degenerate exploitation of embedding similarities.
• Computational Cost: DP-RFT requires a single training phase (e.g., ~40 GPU-hours plus 1 GPU-hour inference for DP-RFT vs. 100 GPU-hours for Aug-PE references (Xu et al., 20 Feb 2026)).
• Expressivity vs. Overfitting: Maximizing DP-noised similarity reward alone can cause overfitting to noise or collateral statistical artifacts.
• Policy Drift and Coverage: Guarantees often require the policy to remain close to the reference $(\epsilon,\delta)$0 (Cho et al., 23 Mar 2026).
• Model Architecture Constraints: Decoupled reward modeling typically restricts learning to linear (or adapter) heads; full LLM backbone tuning complicates DP sensitivity accounting.

7. Algorithmic Workflow and Pseudocode

A representative high-level workflow for DP-RFT in synthetic data generation (Xu et al., 20 Feb 2026):

1. Initialize LLM policy parameters $(\epsilon,\delta)$1 and value network $(\epsilon,\delta)$2.
2. For each training round $(\epsilon,\delta)$3:
   • Sample a batch of prompts $(\epsilon,\delta)$4.
   • Generate on-policy samples $(\epsilon,\delta)$5 via $(\epsilon,\delta)$6.
   • For each $(\epsilon,\delta)$7:
     • Compute clipped similarities to each $(\epsilon,\delta)$8: $(\epsilon,\delta)$9.
     • Sum and add Gaussian noise: $r(x)$0.
     • Normalize to compute $r(x)$1; calculate $r(x)$2.
     • Set scalar reward $r(x)$3 via prompt thresholding.
   • Compute advantages and perform PPO updates on $r(x)$4 to minimize PPO+value losses.
3. After $r(x)$5 batches, return the trained $r(x)$6.

The decoupled reward-modeling RLHF algorithm (Cho et al., 23 Mar 2026) trains a DP-protected reward head using DP-SGD on preference data, then derives the final policy via post-processing (Gibbs reweighting or best-of-$r(x)$7 selection) with no further DP cost.

For DP-SGD fine-tuning, adaptive privacy dial control can be achieved via closed-loop RL, with a Soft Actor-Critic hyper-policy controlling per-parameter clipping and noise (Khadangi et al., 30 Jul 2025).

• DP-RFT: "DP-RFT: Learning to Generate Synthetic Text via Differentially Private Reinforcement Fine-Tuning" (Xu et al., 20 Feb 2026)
• Decoupled reward-modeling RLHF: "Privacy-Preserving Reinforcement Learning from Human Feedback via Decoupled Reward Modeling" (Cho et al., 23 Mar 2026)
• RLDP: "Efficient Differentially Private Fine-Tuning of LLMs via Reinforcement Learning" (Khadangi et al., 30 Jul 2025)