RecPS: Privacy Risk Scoring for Recommender Systems

• The paper introduces RecPS, a framework that quantifies privacy risk at both interaction and user levels using a DP hypothesis-testing approach.
• RecPS leverages an adapted likelihood-ratio attack, RecLiRA, to distinguish sensitive interactions with superior AUC performance compared to prior methods like MINER.
• The framework enables targeted unlearning by ranking user interactions to balance privacy risk reduction and recommendation utility in practical deployments.

Searching arXiv for the exact RecPS paper and closely related recommender-system privacy/MIA work to ground the article. RecPS is a privacy risk scoring framework for recommender systems that quantifies how sensitive each user-item interaction, and each user, is to membership inference attacks (MIAs). It was introduced as a black-box auditing method for recommender models trained on sensitive interaction data, with the explicit goal of supporting privacy-aware model development, user opt-out guidance, and targeted unlearning. The framework is built around an interaction-level MIA called RecLiRA, adapted from the LiRA likelihood-ratio attack, and it defines an $\epsilon$-like privacy score grounded in the hypothesis-testing interpretation of differential privacy (DP) (He et al., 24 Jul 2025).

1. Motivation and problem setting

RecPS addresses a specific gap in recommender-system privacy analysis: although recommender systems depend on highly sensitive user-item interactions, and although users increasingly exercise the right to opt out under regimes such as GDPR and CCPA, there has been no formal sample-level method for deciding which interactions are more privacy-sensitive than others (He et al., 24 Jul 2025). In deployed systems, protection often relies on controlled access rather than strong DP guarantees, because DP training can reduce recommendation utility through added noise and gradient clipping.

The framework is motivated by the observation that privacy leakage in recommender systems is not uniform. Some interactions are more vulnerable to membership inference than others, and this heterogeneity matters both operationally and normatively. Prior recommender-system MIA work mainly measured user-level leakage, while existing interaction-level MIAs were not sufficiently strong to support fine-grained risk scoring. RecPS is designed to fill that gap by assigning privacy risk scores at two levels: the individual interaction $(u,i)$ and the user $u$.

The threat model is black-box. The adversary can query the online recommender API with user-item pairs and obtain scores or probabilities for “interaction” versus “no interaction,” but does not observe the training process, parameters, or gradients. Under this model, successful inference of whether an interaction or a user’s interaction set was present in the training data reveals sensitive historical preferences and may facilitate further inferences such as data reconstruction (He et al., 24 Jul 2025).

2. Differential-privacy interpretation and score definition

The central conceptual move in RecPS is to derive an empirical privacy score from the hypothesis-testing interpretation of DP. For adjacent datasets $D_0$ and $D_1$ differing by exactly one record, an algorithm $M$ satisfies $(\epsilon,\delta)$-DP if

$$\Pr(M(D_0) \in \mathcal{O}) \le e^{\epsilon}\Pr(M(D_1) \in \mathcal{O}) + \delta.$$

Under the corresponding hypothesis-testing view, a per-record audit yields

$$\ln\!\left(\frac{\mathrm{TPR}}{\mathrm{FPR}}\right) \le \epsilon,$$

where $\mathrm{TPR}$ and $(u,i)$0 are measured by an attack that attempts to distinguish whether the record was IN or OUT of the training set (He et al., 24 Jul 2025).

RecPS uses this relation to define an interaction-level $(u,i)$1-like sensitivity bound for a non-DP recommender model $(u,i)$2. For a known interaction $(u,i)$3 and adjacent datasets $(u,i)$4, the paper defines

$(u,i)$5

A sufficiently strong interaction-level MIA then yields an empirical lower bound

$(u,i)$6

As the attack quality improves, $(u,i)$7. In this formulation, RecPS does not claim to provide a formal DP guarantee for a non-DP-trained recommender; rather, it supplies a DP-grounded privacy score interpretable as sample-specific sensitivity (He et al., 24 Jul 2025). This distinction is crucial. A common misconception is to treat high RecPS scores as equivalent to certified DP parameters; the framework instead uses DP auditing logic to construct theoretically meaningful empirical risk scores.

3. RecLiRA: the interaction-level membership inference engine

The interaction-level score depends on RecLiRA, an attack adapted from LiRA to recommender models that output probabilities. For a record $(u,i)$8, the recommender returns

$(u,i)$9

interpreted as the probability of “interaction,” with confidence vector $u$0. RecLiRA compresses this output to the confidence-separation statistic

$u$1

and then applies the logit transform

$u$2

Empirically, $u$3 is approximately Gaussian for OUT samples and is higher for IN samples (He et al., 24 Jul 2025).

In its offline adaptation, RecLiRA estimates only the OUT distribution of $u$4 and performs a one-sided hypothesis test. For each shadow model $u$5, it computes

$u$6

where $u$7 is a Gaussian fit to OUT samples’ $u$8. Shadow models are trained by 50% Bernoulli subsampling of the training set, using the same architecture and training procedure as the target model. For stable low-FPR estimation, the framework recommends $u$9 shadow models and at least $D_0$0 OUT samples to estimate $D_0$1 (He et al., 24 Jul 2025).

Threshold selection is interaction-specific. For a given $D_0$2, candidate thresholds are drawn from $D_0$3 values corresponding to OUT-shadow models. Each threshold $D_0$4 induces predicted labels $D_0$5, from which TPR and FPR are computed. The interaction-level score is then

$D_0$6

over valid thresholds with $D_0$7. This procedure is designed to maximize discrimination for each interaction rather than imposing a global threshold across all records (He et al., 24 Jul 2025).

This design has two important implications. First, it exploits recommender-specific confidence signals rather than generic classifier outputs. Second, its quality as a scoring mechanism depends directly on the separation between IN and OUT distributions; the paper’s empirical results show that this separation is substantially stronger than in the MINER baseline.

4. User-level composition, scoring pipeline, and computational characteristics

RecPS extends interaction-level scores to the user level by invoking a DP-composition-style argument. For a user $D_0$8 with interaction set $D_0$9, the paper considers adjacent datasets that differ by all of $D_1$0’s interactions and, under independence approximations, derives

$D_1$1

Because this sum would systematically inflate scores for heavy users, the practical user-level score reported by RecPS is the normalized average

$D_1$2

The paper states that this preserves ranking of sensitive users while avoiding penalizing those with more interactions (He et al., 24 Jul 2025). This suggests that RecPS prioritizes comparative risk ordering over strict user-level composition tightness.

The scoring pipeline has offline and online phases. Offline preparation trains the target architecture, trains shadow models on Bernoulli half-samples, and fits the Gaussian OUT distribution from pooled $D_1$3 values. Online scoring evaluates each interaction $D_1$4 across all shadow models, computes $D_1$5, $D_1$6, and $D_1$7, searches thresholds maximizing $D_1$8, and finally averages the resulting interaction scores to obtain $D_1$9.

The reported complexity is dominated offline by shadow-model training, with

$M$0

where $M$1 is the cost of training one model and $M$2 is inference cost. Online complexity per interaction is

$M$3

from evaluating $M$4 and $M$5 across $M$6 models and scanning thresholds (He et al., 24 Jul 2025). The computational burden is therefore front-loaded into offline preparation and amortized across many subsequent scoring requests.

5. Datasets, models, and empirical performance

The empirical study evaluates RecPS on three benchmark datasets and two standard recommender architectures: Neural Collaborative Filtering (NCF) and LightGCN. Users with fewer than 20 interactions are removed during preprocessing, and training follows a chronological split per user, with the last interaction used for test and the second-to-last for validation. The remaining interactions form the training set. The negative sampling ratio is $M$7, optimization uses SGD with learning rate $M$8, batch size $M$9, up to $(\epsilon,\delta)$0 epochs, and early stopping after $(\epsilon,\delta)$1 epochs without improvement. Experiments were run on $(\epsilon,\delta)$2 NVIDIA RTX 2080 Ti GPUs (He et al., 24 Jul 2025).

Dataset	Scale	Preprocessing
MovieLens-1M	6,040 users, 3,706 items, 1,000,209 interactions	Retain users with $(\epsilon,\delta)$3 interactions
Amazon Digital Music	840,372 users, 456,992 items, 1,584,082 interactions	Retain users with $(\epsilon,\delta)$4 interactions
Amazon Beauty	1,210,271 users, 249,274 items, 2,023,070 interactions	Retain users with $(\epsilon,\delta)$5 interactions

At the interaction level, RecLiRA substantially outperforms MINER. On MovieLens-1M, RecLiRA reaches AUC $(\epsilon,\delta)$6 with LightGCN and $(\epsilon,\delta)$7 with NCF, compared with MINER’s $(\epsilon,\delta)$8 and $(\epsilon,\delta)$9. At $$\Pr(M(D_0) \in \mathcal{O}) \le e^{\epsilon}\Pr(M(D_1) \in \mathcal{O}) + \delta.$$0, RecLiRA attains $$\Pr(M(D_0) \in \mathcal{O}) \le e^{\epsilon}\Pr(M(D_1) \in \mathcal{O}) + \delta.$$1 for LightGCN and $$\Pr(M(D_0) \in \mathcal{O}) \le e^{\epsilon}\Pr(M(D_1) \in \mathcal{O}) + \delta.$$2 for NCF, while MINER attains $$\Pr(M(D_0) \in \mathcal{O}) \le e^{\epsilon}\Pr(M(D_1) \in \mathcal{O}) + \delta.$$3 and $$\Pr(M(D_0) \in \mathcal{O}) \le e^{\epsilon}\Pr(M(D_1) \in \mathcal{O}) + \delta.$$4. On Amazon Digital Music, RecLiRA reaches AUC $$\Pr(M(D_0) \in \mathcal{O}) \le e^{\epsilon}\Pr(M(D_1) \in \mathcal{O}) + \delta.$$5 with LightGCN and $$\Pr(M(D_0) \in \mathcal{O}) \le e^{\epsilon}\Pr(M(D_1) \in \mathcal{O}) + \delta.$$6 with NCF, compared with MINER’s $$\Pr(M(D_0) \in \mathcal{O}) \le e^{\epsilon}\Pr(M(D_1) \in \mathcal{O}) + \delta.$$7 and $$\Pr(M(D_0) \in \mathcal{O}) \le e^{\epsilon}\Pr(M(D_1) \in \mathcal{O}) + \delta.$$8; at $$\Pr(M(D_0) \in \mathcal{O}) \le e^{\epsilon}\Pr(M(D_1) \in \mathcal{O}) + \delta.$$9, RecLiRA attains $$\ln\!\left(\frac{\mathrm{TPR}}{\mathrm{FPR}}\right) \le \epsilon,$$0 and $$\ln\!\left(\frac{\mathrm{TPR}}{\mathrm{FPR}}\right) \le \epsilon,$$1, while MINER attains $$\ln\!\left(\frac{\mathrm{TPR}}{\mathrm{FPR}}\right) \le \epsilon,$$2 and $$\ln\!\left(\frac{\mathrm{TPR}}{\mathrm{FPR}}\right) \le \epsilon,$$3. On Amazon Beauty, RecLiRA reaches AUC $$\ln\!\left(\frac{\mathrm{TPR}}{\mathrm{FPR}}\right) \le \epsilon,$$4 with LightGCN and $$\ln\!\left(\frac{\mathrm{TPR}}{\mathrm{FPR}}\right) \le \epsilon,$$5 with NCF, compared with MINER’s $$\ln\!\left(\frac{\mathrm{TPR}}{\mathrm{FPR}}\right) \le \epsilon,$$6 and $$\ln\!\left(\frac{\mathrm{TPR}}{\mathrm{FPR}}\right) \le \epsilon,$$7 (He et al., 24 Jul 2025).

At the user level, reported RecPS scores $$\ln\!\left(\frac{\mathrm{TPR}}{\mathrm{FPR}}\right) \le \epsilon,$$8 lie roughly in $$\ln\!\left(\frac{\mathrm{TPR}}{\mathrm{FPR}}\right) \le \epsilon,$$9 across datasets. LightGCN tends to produce slightly higher scores than NCF, and the distributions stratify users: the top $\mathrm{TPR}$0–$\mathrm{TPR}$1 have distinctly higher scores, while the bottom $\mathrm{TPR}$2–$\mathrm{TPR}$3 have lower scores (He et al., 24 Jul 2025). The paper associates this pattern with stronger recommendation utility, though this remains an empirical correlation rather than a formal theorem.

6. Privacy-aware unlearning, practical uses, and limitations

A major application of RecPS is targeted unlearning. Instead of removing all data from users designated as sensitive, the framework ranks users by $\mathrm{TPR}$4, then ranks each selected user’s interactions by $\mathrm{TPR}$5, and removes only the top $\mathrm{TPR}$6 of those interactions until the user’s score falls below a threshold $\mathrm{TPR}$7. The paper suggests choosing $\mathrm{TPR}$8 as the minimum score among the top-$\mathrm{TPR}$9 sensitive users (He et al., 24 Jul 2025).

The reported utility trade-off is markedly different for user-level versus interaction-level removal. On Amazon Digital Music, removing all interactions of the top $(u,i)$00 sensitive users causes an approximately $(u,i)$01 drop in LightGCN HR@100 and an approximately $(u,i)$02 drop in NCF HR@100. By contrast, removing only the top $(u,i)$03 sensitive interactions for those users demotes $(u,i)$04 of the top-$(u,i)$05 users below $(u,i)$06 while reducing LightGCN HR@100 by approximately $(u,i)$07 and NCF HR@100 by approximately $(u,i)$08 (He et al., 24 Jul 2025). Score-guided removal therefore outperforms random removal in both utility preservation and score reduction for targeted users.

The framework also measures the “privacy onion effect,” namely the fact that scores depend on the full dataset and can change after removals. After removing the top-$(u,i)$09 sensitive users, remaining users’ score changes $(u,i)$10 lie mostly in $(u,i)$11, and about $(u,i)$12–$(u,i)$13 of users see slight increases. Interaction-level removal weakens this effect: the fraction of users whose scores increase drops by approximately $(u,i)$14–$(u,i)$15 (He et al., 24 Jul 2025).

Several limitations are explicit. RecLiRA currently targets recommender models with probability outputs, such as NCF and LightGCN, and does not directly apply to ranking-only models such as matrix factorization or ALS without probabilities. The method is expensive offline because stable low-FPR estimation benefits from at least $(u,i)$16 shadow models. The Gaussian approximation for the OUT distribution may be miscalibrated if fitting is poor, and score magnitudes depend on model class, training stability, and dataset characteristics. The threat model is controlled-access black-box auditing; white-box attacks or richer side information could yield stronger inferences than those measured by RecPS (He et al., 24 Jul 2025).

These caveats delimit the framework’s interpretation. RecPS is best understood not as a certified privacy mechanism, but as a theoretically motivated auditing and prioritization tool for privacy-aware recommender-system operations. Its core contribution is to make privacy risk legible at the granularity of individual interactions and users, thereby enabling opt-out guidance and targeted unlearning strategies that preserve substantially more recommendation utility than blanket deletion policies (He et al., 24 Jul 2025).