A Production-Ready RL Framework for Personalized Utility Tuning with Pareto Sweeping in Pinterest Recommender Systems

Abstract: Large-scale recommenders encode multi-objective trade-offs by combining multiple predicted outcomes into a single utility score. Although this utility layer can be updated independently of the ranker, weight tuning remains largely manual, globally applied, slow to adapt to changing environments and business needs, and hard to govern as priorities shift. We propose PRL-PUTS, a Production-ready, ranker independent RL framework for Personalized Utility-weight Tuning with Pareto Sweeping. We cast utility tuning as a one-step, value-based RL problem: given request context, an agent selects a utility-weight vector that re-weights ranker predictions to maximize request-level engagement rewards. To visualize performance across the trade-off spectrum and allow decision makers to update the deployed operating policy instantly, we adopt an inference-time Pareto frontier sweeping via a scalarization parameter, producing a family of policies and an empirical Pareto frontier used as a governance artifact for operating policy selection. PRL-PUTS runs in parallel with ranking inference without adding serving latency. We validate PRL-PUTS with offline analysis using unbiased exploration logs and online experiments on Pinterest Homefeed where PRL-PUTS showed significant increases in engagement compared to baseline such as +0.13\% increase in successful session, a core metric for user engagement.

• The paper introduces PRL-PUTS, a production-ready contextual-bandit layer that selects request-level weights for Repin and P2P objectives without changing candidate generation or ranker inference.
• The framework sweeps 25 scalarization settings across 49 discrete actions to produce auditable Pareto policies, achieving up to 2.26% Repin lift, 0.30% P2P lift, and 0.13% Successful Sessions improvement online.
• The results show personalization drives the gains: PRL-PUTS outperformed a matched static-weight baseline, while offline predictions closely tracked online outcomes with correlations of 0.999 for Repin and 0.986 for P2P impressions.

Overview

PRL-PUTS (Production-ready RL framework for Personalized Utility-weight Tuning with Pareto Sweeping) is a control-layer framework deployed on Pinterest Homefeed that replaces manually tuned, globally static utility weights with a request-level reinforcement learning policy. The system addresses a persistent operational problem in large-scale recommenders: multi-task rankers produce per-objective predictions that are aggregated into a single ranking score via a linear utility function whose weights are typically selected through ad hoc offline analysis and repeated A/B tests, then frozen as global constants for extended periods. This process is slow (weeks to months from ideation to launch), non-personalized, and prone to staleness as the ranker and traffic distribution evolve.

The key architectural decision is ranker independence: the RL agent does not modify item scoring or candidate generation. Instead, given request context, it selects a utility-weight vector applied only at the aggregation step over the existing per-objective predictions of a fixed multi-task ranker. The framework runs in parallel with ranking inference and adds no measurable serving latency.

Problem formulation

The authors cast utility tuning as a one-step MDP (contextual bandit) at the request level, with discount factor $\gamma = 0$ since actions affect only the current request and only immediate engagement is optimized. Each logged interaction is a tuple $(s, a, \mathbf{r})$, where $s$ comprises serving-time features (user profile signals and embeddings, a sequence of the last $N$ user actions encoded via a Transformer, and context features such as device type and surface), $a$ is a utility-weight action, and $\mathbf{r} = [r^{repin}, r^{p2p}]$ is a vector of request-level rewards.

Two design choices constrain the problem deliberately:

• Restricted action space: only the two heads with the largest contribution to the production utility score are controlled — Repin (saves) and P2P impressions — which together account for roughly 90% of the utility score contribution. Each weight is discretized into $K=7$ linearly spaced values (Repin weights in $[10, 200]$, P2P impression weights in $[1, 30]$, versus production defaults of 91.6 and 9.1), yielding $|\mathcal{A}| = 49$ actions. The paper concedes this reduces expressivity but bounds exploration risk, keeps the control surface reviewable, and ensures adequate support in logged data for stable offline evaluation.
• Clipped binary rewards: because more than 80% of requests receive at most one engagement event, counts are clipped to $(s, a, \mathbf{r})$0 to stabilize learning. The authors explicitly note this discards intensity information beyond the first event, treating two and two hundred engagements identically, which may limit optimization of the upper tail.

Method

Two-head value model

Since the setting is one-step, objective-specific action-value functions reduce to conditional expected rewards: $(s, a, \mathbf{r})$1 and analogously for P2P impressions. The model encodes categorical features via embedding tables, sequential features via a Transformer followed by average pooling, concatenates these with a user embedding, and passes them through an MLP state module. Actions are min–max normalized within their candidate ranges and embedded by an MLP; state and action representations are concatenated into a shared three-layer MLP backbone (batch norm, linear projection, ReLU) with two sigmoid-bounded output heads, allowing value estimates to be interpreted as conditional success probabilities. Training minimizes summed MSE across both heads; binary cross-entropy was tried and did not improve offline or online performance.

Actions are encoded as inputs rather than enumerated output neurons, keeping the architecture extensible to larger or continuous action sets, though no such extension is evaluated here.

Inference-time Pareto sweeping

Rather than fixing a scalarization during training (which would require retraining whenever priorities shift), PRL-PUTS learns objective-specific value functions once and exposes a scalarization parameter $(s, a, \mathbf{r})$2 at inference time. The policy selects the action maximizing $(s, a, \mathbf{r})$3. Sweeping $(s, a, \mathbf{r})$4 over a grid of 25 values induces a family of deterministic policies from a single trained model; evaluating each offline yields paired lift estimates relative to production, from which non-dominated policies form an empirical Pareto frontier. Changing the operating policy in production requires only a configuration change to $(s, a, \mathbf{r})$5 — no retraining — enabling fast updates and rollback. This separation of learning from selection is the central governance mechanism of the paper.

Production integration

PRL-PUTS is implemented as a decoupled layer alongside the existing Homefeed serving stack. The agent infers a request-level weight vector from context only, running in parallel with per-item ranker inference; once predictions are available, a blending layer applies the selected weights. There is no measurable end-to-end latency overhead, and the system falls back to static production weights on timeout, missing features, or inference failure. Ranker refreshes do not require changes to PRL-PUTS as long as the controlled heads retain consistent semantics; retraining the controller is needed only when its contract materially changes or when substantial distribution shift invalidates learned mappings.

Training data comes from a capped 1.25% of live traffic under uniform random exploration over the 49-action set, with propensities logged. Policies are trained on 14 days of logs and evaluated on a disjoint 7-day hold-out using Reward@HIT, a rejection-style off-policy estimator that averages observed rewards only over requests where the target policy's predicted action matches the logged action. Under uniform exploration, inverse-propensity weights cancel in the self-normalized estimator. The authors choose this estimator for simplicity and auditability, leaving more sample-efficient OPE methods open.

Experimental results

Online experiments ran for two weeks with each arm receiving 1% of Homefeed traffic, comparing three operating policies (Repin-leaning, balanced/knee, P2P-leaning) against the production baseline.

Global operating policies show the expected trade-off structure, with strong offline-to-online agreement:

Policy	$(s, a, \mathbf{r})$6	Repin (online)	P2P (online)	SS (online)
Repin-leaning	0.17	+2.26%	−0.21%	−0.09%
Balanced	0.21	+1.35%	−0.04%	−0.02%
P2P-leaning	0.24	+0.66%	+0.30%	+0.13%

Pearson correlation between offline-predicted and online-observed lifts is 0.999 for Repin and 0.986 for P2P impressions, indicating the offline Pareto frontier is decision-useful prior to online testing. Successful Sessions, a composite engagement metric, improved by +0.13% under the P2P-leaning policy.

Cohort-conditioned policies vary $(s, a, \mathbf{r})$7 across three user cohorts (CORE: saved on ≥4 of last 28 days; CASUAL: active ≥4 days; REST). CASUAL and REST cohorts respond to $(s, a, \mathbf{r})$8 as expected, with up to +7.62% online Repin lift for CASUAL users under the Repin-leaning configuration. CORE users exhibit a constrained trade-off: offline evaluation identified only one operating point non-degrading on both objectives, so a single $(s, a, \mathbf{r})$9 was reused across variants for that cohort. Offline analysis shows up to 10% Repin lift for CASUAL and up to 5% for REST users, while CORE offers few mutually improving operating points — a notable heterogeneity finding that itself justifies personalization.

Personalization versus global retuning: to disentangle whether gains come from contextual selection or merely a better global weight vector, the authors constructed a static baseline matching the traffic-level average weights produced by PRL-PUTS (e.g., raising the P2P weight from 9.1 to the learned average of 11.83). The matched static variant was neutral-to-negative online (−0.24% Repin, +0.07% P2P, +0.02% SS), whereas PRL-PUTS achieved +0.12%, +0.21%, and +0.11% respectively. This is the paper's strongest causal claim: improvements are attributable to context-dependent weight selection rather than a globally shifted weight vector.

A feature-group isolation ablation found user information the strongest standalone signal, sequence features highly consistent with it, and context-only features substantially insufficient for accurate value estimation.

Limitations and open questions

The paper is explicit about its constraints. The one-step formulation optimizes immediate request-level outcomes and does not capture longer-horizon effects such as retention or future engagement; extending to delayed credit assignment would require trajectory logging, longer attribution windows, and bootstrapping-based learning, with increased sensitivity to confounding and non-stationarity. Only two objectives are controlled, and a single scalarization knob cannot govern three or more goals — the authors suggest constrained selection or multiple auditable preference parameters as directions. The discrete action space ($s$0 per head) improves safety and OPE stability but limits expressivity; continuous actions would raise new exploration and off-policy evaluation challenges. The clipped binary reward discards engagement-intensity signal, and the hit-based OPE estimator is less sample-efficient than alternatives such as doubly robust methods, which remain unexplored here. Finally, the cohort-level conditioning is coarse; finer-grained personalization within cohorts is not evaluated.

Conclusion

PRL-PUTS demonstrates that utility-weight tuning can be reframed as a one-step, value-based RL problem solved atop a fixed multi-task ranker, with inference-time Pareto sweeping converting a single trained model into a family of governable operating policies. Deployed on Pinterest Homefeed with negligible latency overhead, it delivers controllable Repin–P2P trade-offs (+2.26% Repin at the Repin-leaning extreme, +0.13% Successful Sessions at the P2P-leaning extreme), near-perfect offline-to-online correlation, and evidence that gains stem from personalization rather than global reweighting. The main open questions left by the work are scaling to more than two objectives without losing auditability, moving to continuous action spaces with trustworthy counterfactual validation, and incorporating longer-horizon objectives under production safeguards.