Personalization Salience Score (PSS)

• Personalization Salience Score (PSS) is a metric that quantifies the degree to which a model output is tailored to an individual compared to global trends.
• It decomposes the personalization effect into components like user-specific loss, accuracy gain, uniqueness, and subjective factors, enabling performance tradeoffs.
• PSS is applied across domains from recommender systems to visual saliency, grounding evaluations in privacy constraints and adaptive model tuning.

A Personalization Salience Score (PSS) formally quantifies the degree to which a predictive system, signal, or output is tailored to an individual user, as opposed to reflecting general or community-level patterns. Across theoretical, experimental, and practical domains, PSS is invoked to measure, decompose, and tune the tension between personalization and global generalization, to audit user-centricity and privacy, and to ground the evaluation of personal models in domains such as recommendation, sentiment classification, highlight prediction, and visual saliency. Methodologies for computing PSS vary by domain, but universally include explicit mechanisms for isolating the personal component of a prediction from generic and crowd baselines, typically under privacy or data-sharing constraints.

1. Formal Definitions and General Frameworks

Several domain-general and domain-specific instantiations of the Personalization Salience Score are present in the literature. In "Sometimes You Want to Go Where Everybody Knows Your Name" (Brasher et al., 2018), PSS is defined for per-user models as a convex combination of loss on user-specific data $X_i$ and a global held-out dataset $D$:

$$\mathrm{PSS}_i = \alpha \cdot L(X_i, M_i) + (1 - \alpha) \cdot L(D, M_i)$$

where $L$ denotes a loss function (e.g., cross-entropy), $M_i$ is a model personalized for user $i$, and $\alpha \in [0, 1]$ interpolates between pure personalization ($\alpha=1$) and pure generalization ($\alpha=0$).

In more general ML evaluation contexts, "How Personal is Machine Learning Personalization?" (Greene et al., 2019) expands the framework to a convex or weighted sum over objective and subjective personalization components, each normalized to [0,1] scale:

$$\mathrm{PSS} = \sum_{i} w_i \cdot \mathrm{PSS}_{d_i},\qquad \sum_i w_i = 1,\quad w_i \ge 0$$

where $D$0 range over components such as personal data usage, uniqueness, accuracy gain, user control/consent, explanation alignment, and moral stakes.

Across recent work in social highlighting and saliency, PSS is typically operationalized as an "own-versus-other" gap: the difference in predictive quality (e.g., average precision, AP) between a model conditioned on an individual's history and the same model conditioned on another matched user's history (Nakayashiki et al., 8 Jun 2026, Nakayashiki et al., 9 Jun 2026).

2. Component-Wise Decomposition and Domains

Personalization Salience decomposition varies by application:

• User-Specific Loss vs. Global Loss: In personalization under privacy constraints, PSS balances per-user fit and global generalization via $D$1 ("personalization weight"). It enables tradeoffs: high $D$2 overfits to local data for rare users, low $D$3 ignores the user entirely (Brasher et al., 2018).
• Feature-Based and Behavioral Decomposition: In general ML settings, PSS can combine:
  • Personal Data Usage ($D$4): Ratio of explicit personal features to all features.
  • Uniqueness ($D$5): Fraction of users sharing the same output.
  • Accuracy Gain ($D$6): Normalized improvement over a non-personalized baseline.
  • Self-Determination ($D$7): Fraction of features with user opt-in.
  • Right-Reasons ($D$8): User's subjective evaluation of explanation alignment.
  • Moral Importance ($D$9): Task's contextual or moral stakes (Greene et al., 2019).
• Salience, Crowd, and Personal Residual: In text highlighting, PSS is often modeled as a linear combination:

$$\mathrm{PSS}_i = \alpha \cdot L(X_i, M_i) + (1 - \alpha) \cdot L(D, M_i)$$0

with $$\mathrm{PSS}_i = \alpha \cdot L(X_i, M_i) + (1 - \alpha) \cdot L(D, M_i)$$1 = generic salience (sentence-to-doc centroid), $$\mathrm{PSS}_i = \alpha \cdot L(X_i, M_i) + (1 - \alpha) \cdot L(D, M_i)$$2 = crowd salience (co-reader mark frequency), $$\mathrm{PSS}_i = \alpha \cdot L(X_i, M_i) + (1 - \alpha) \cdot L(D, M_i)$$3 = personal salience (cosine to user's historical embedding profile) (Nakayashiki et al., 8 Jun 2026). Here, $$\mathrm{PSS}_i = \alpha \cdot L(X_i, M_i) + (1 - \alpha) \cdot L(D, M_i)$$4 is generally far smaller than $$\mathrm{PSS}_i = \alpha \cdot L(X_i, M_i) + (1 - \alpha) \cdot L(D, M_i)$$5; the personal residual is much weaker for span salience than for selection among candidates.

• Saliency in Visual Domains: In visual attention, model architectures condition on observer label (e.g., age group) to obtain personalized saliency maps; PSS is defined as the difference between personalized and population map scores under metrics such as AUC, NSS, or CC (Yu et al., 2017).

3. Methodologies for Estimation and Evaluation

Methodology for PSS estimation is domain-dependent but shares common structure:

• Loss-Based Measurement: For models trained per-user, losses on user-local data and on global data are tracked separately (Brasher et al., 2018).
• Component Aggregation: For general ML, component scores are individually computed and combined using stakeholder-weighted or context-specific weights (Greene et al., 2019). Examples are provided for recommender systems, with clear operationalizations for each component.
• Own-Versus-Other Gap: In highlight prediction, the "identity control" paradigm creates pairs matching users and co-readers, quantifies average precision on true vs. control profile, and uses the mean gap as the PSS for salience or selection. Strict leakage prevention is critical: user histories for profile construction must exclude the target document, and the comparison must be method-matched (Nakayashiki et al., 8 Jun 2026, Nakayashiki et al., 9 Jun 2026).
• Experimental Ensembling/Selection: In sentiment, separate models for each user are evaluated individually and as ensembles; the break-even $$\mathrm{PSS}_i = \alpha \cdot L(X_i, M_i) + (1 - \alpha) \cdot L(D, M_i)$$6 threshold empirically partitions the regime where per-user models outperform ensemble models (Brasher et al., 2018).
• Visual Domains: For image saliency, PSS is expressed as per-group improvement over population baselines in standard metrics (AUC, NSS, etc.), and composite metrics can be set as weighted sums (Yu et al., 2017).

4. Empirical Results and Observed Tensions

Empirical findings across multiple domains converge on several themes:

• Privacy-Constrained Personalization: Strict no-centralization constraints are respected by computing only on-device losses and aggregating only model/proxy metrics (Brasher et al., 2018). Federated and differentially private extensions are straightforward.
• Highlighting/Social Domains: Salience (which span is highlighted) is dominated by crowd signals; own-vs-other AP gaps are small (PSS_salience ≈ +0.017 AP) and not productively exploitable. By contrast, selection-level personalization (which highlighted span belongs to whom) exhibits a larger personal signal (PSS_selection ≈ +0.13–0.14 AP), mostly topic-driven but robust to peer control (Nakayashiki et al., 8 Jun 2026, Nakayashiki et al., 9 Jun 2026).
• Tradeoff Calibration: In user modeling, empirical $$\mathrm{PSS}_i = \alpha \cdot L(X_i, M_i) + (1 - \alpha) \cdot L(D, M_i)$$7 values (e.g., $$\mathrm{PSS}_i = \alpha \cdot L(X_i, M_i) + (1 - \alpha) \cdot L(D, M_i)$$80.91 with $$\mathrm{PSS}_i = \alpha \cdot L(X_i, M_i) + (1 - \alpha) \cdot L(D, M_i)$$9 users) make explicit the threshold at which user-specific data must dominate to justify model isolation (Brasher et al., 2018).
• Visual Saliency: Personalized saliency networks show measurable and significant PSS in AUC and related metrics for observer groups. The personalization is achieved via explicit observer labels; scores are reported as $L$0-metrics over group-agnostic population maps (Yu et al., 2017).

5. Limitations, Biases, and Methodological Safeguards

Several challenges and pitfalls are identified in the literature:

• Overstated Personalization via Leakage: History-conditioned profiles must be leakage-free; otherwise, AP gains of +0.07 to +0.15 can be spuriously attributed to personalization (Nakayashiki et al., 8 Jun 2026).
• Small Crowd Artifacts: Limiting crowd baselines to ≤5 co-readers can understate the shared component and inflate the apparent personal signal by ≈0.055 AP (Nakayashiki et al., 8 Jun 2026).
• Feature Engineering and Component Weights: Sensitivity of PSS to feature inclusion/exclusion, accuracy metric choice, scoring function granularity, and subjective weight setting can affect interpretability and robustness (Greene et al., 2019).
• Subjective Component Bias: Overweighting subjective dimensions (e.g., self-determination, moral importance) can privilege vocal users or domains with higher user engagement (Greene et al., 2019).
• Generalizability and Interpretability: Most documented PSS schemas require explicit stakeholder or domain-driven selection of weights and metrics, limiting purely automatic deployment (Greene et al., 2019).

6. Practical Recommendations and Future Directions

Authors recommend several pragmatic and research trajectories:

• Adaptive/Automatic Scheduling of $L$1: Data-driven adjustment per-user based on data availability or model confidence is encouraged, conceptualized from a Bayesian MAP standpoint (Brasher et al., 2018).
• Aggregation over Subpopulations: For salience, aggregating over user clusters or subpopulations is advocated instead of pushing deeper into per-user salience personalization (Nakayashiki et al., 9 Jun 2026).
• Component Validation and Audits: Empirical user studies, regulatory audits (GDPR, opt-in/opt-out), and field experiments are advised for validating both objective and subjective PSS components (Greene et al., 2019).
• Differential Privacy and Federated Learning: PSS frameworks naturally align with privacy-preserving learning via summary/statistical aggregation and no raw data movement (Brasher et al., 2018).
• Further Metric Extensions: Proposed additional axes include serendipity, transparency, and evolutionary alignment with user narrative identity (Greene et al., 2019).
• Continual/On-Device Learning: Integration of PSS into continual learning frameworks and deeper scrutiny of catastrophic forgetting, particularly in decentralized/low-data regimes, is identified as a future research area (Brasher et al., 2018).

7. Comparison of PSS Instantiations Across Domains

A summary comparison of notable formalizations and their domains:

Domain	PSS Formalism	Main Empirical Finding
Per-User Modeling (Brasher et al., 2018)	$L$2	Empirical $L$3 quantifies break-even point for personalization vs. generalization; strict privacy supported
ML Personalization (Greene et al., 2019)	Weighted sum of 6 components	Bridges technical and humanistic personalization; operationalized for compliance/fairness
Text Highlighting (Nakayashiki et al., 8 Jun 2026)	AP(own profile) – AP(other)	Crowd dominates salience; identity signals emerge in selection, not salience
Visual Saliency (Yu et al., 2017)	$L$4 in (AUC, NSS, etc.)	Observer labels enable measurable, group-wise PSS; explicit improvement over population

Distinct methodologies share fundamental traits: quantitative separation of personal, crowd, and generic structure contributions, with explicit audit of identity signal and safeguards against confounds. PSS thus functions as both a quality metric and a practical tool for diagnosis and system steering in personalization-sensitive systems.