A2PO: Asymmetric Advantage Policy Optimization

• A2PO is a policy optimization method that computes separate business and semantic advantages and fuses them asymmetrically to safeguard business goals.
• It standardizes heterogeneous reward streams within rollout groups, using a consistency gate and magnitude balancing to stabilize updates.
• Empirical results show that avoiding naive reward fusion prevents performance collapse and enhances recommendation metrics in practice.

Asymmetric Advantage Policy Optimization (A2PO) is the policy-optimization component of the S-GRec framework for semantic-aware generative recommendation. In that setting, a lightweight autoregressive generator is trained from two heterogeneous supervision sources: a platform-facing business reward, such as eCPM, CTR, or GMV, and an offline semantic reward produced by a LLM-based Personalized Semantic Judge (PSJ). A2PO is defined as a business-anchored policy optimization rule that computes separate business and semantic advantages and injects semantic guidance only when it is directionally consistent with the business objective, thereby preventing semantic supervision from overriding platform goals (Jiang et al., 11 Feb 2026).

1. Problem setting and design rationale

A2PO is motivated by a specific tension in generative recommendation. Behavioral logs provide weak supervision about why a user prefers an item, while LLM-derived semantic judgments can supply richer signals about profile relevance, future-interest relevance, novelty, and contextual relevance. The same paper argues, however, that “semantic signals can conflict with platform business objectives,” because an LLM judge models user-intent semantics rather than the platform’s calibrated revenue objective. An item can therefore be semantically appealing or novel while still being suboptimal for eCPM, CTR calibration, monetization, advertiser constraints, or related business targets (Jiang et al., 11 Feb 2026).

This conflict is not only semantic but also statistical. Business rewards are platform-calibrated, whereas semantic rewards are “judge-dependent” and heterogeneous in scale and distribution. For that reason, the paper explicitly rejects naive reward-level fusion of the form

$r = r_{\mathrm{biz}} + \alpha r_{\mathrm{sem}},$

stating that such fusion is brittle, sensitive to the hand-tuned coefficient $\alpha$, and prone to instability when one signal interferes with the other. A2PO is introduced precisely as an alternative to raw reward mixing: it standardizes the two reward streams separately within each rollout group, computes objective-specific advantages, and fuses them asymmetrically so that semantic guidance cannot reverse the direction implied by business reward (Jiang et al., 11 Feb 2026).

This design means that A2PO is not presented as a generic reward mixing trick. It is described as a business-anchored policy optimization rule whose asymmetry is structural: business advantage is always active, whereas semantic advantage is auxiliary, selectively admitted, and explicitly bounded.

2. Formal setting and reward construction

The optimized policy is the autoregressive generative recommender

$$\pi_{\theta}(y \mid x) = \prod_{t=1}^{T} \pi_{\theta}\!\left(y_t \mid x, y_{<t}\right),$$

where $x$ is the unified user or request input, $y=(y_1,\ldots,y_T)$ is the generated Semantic ID sequence, and $a=\mathrm{Map}(y)$ is the corresponding item or ad. For each input $x$, the system samples a rollout group of $G$ candidate sequences,

$\mathcal{Y}(x)=\{y^{1},y^{2},\ldots,y^{G}\}.$

Each candidate receives two scalar rewards: a business reward $r_{\mathrm{biz}}^{(i)}$ and a semantic reward $\alpha$0 (Jiang et al., 11 Feb 2026).

The business reward depends on the experimental setting. In the public Amazon experiments, because monetary signals are unavailable, it is instantiated as a ranking-based proxy from the ground-truth next item. In the online advertising system, the paper states that $\alpha$1 corresponds to eCPM. The semantic reward is produced by PSJ as a holistic semantic score,

$\alpha$2

PSJ itself has a two-stage construction. It first predicts an aspect vector

$\alpha$3

covering profile relevance, future-interest relevance, novelty, and contextual relevance. It then predicts user-conditional aggregation weights

$\alpha$4

and forms the scalar holistic score

$\alpha$5

A2PO therefore does not optimize directly on aspect-level evidence. It consumes a single scalar semantic reward, but that scalar is already an aggregated, user-conditional semantic preference score (Jiang et al., 11 Feb 2026).

This arrangement has an important methodological implication. PSJ serves as a stabilization mechanism before A2PO ever sees the semantic signal: aspect decomposition makes the judgment interpretable, and pairwise-trained user-conditional aggregation is meant to make the scalar semantic reward more stable and aligned with comparative preference.

3. Asymmetric fused advantage and clipped policy objective

The paper does not specify an explicit learned value function, a critic regression loss, or a GAE recursion for A2PO. Instead, it states that A2PO follows group-based policy optimization and uses “group-relative (standardized) advantages computed within $\alpha$6 to reduce reward scale sensitivity and stabilize updates.” The most faithful reconstruction is therefore that $\alpha$7 and $\alpha$8 are obtained by within-group standardization of $\alpha$9 and $$\pi_{\theta}(y \mid x) = \prod_{t=1}^{T} \pi_{\theta}\!\left(y_t \mid x, y_{<t}\right),$$0, respectively (Jiang et al., 11 Feb 2026).

The fused asymmetric advantage is defined as

$$\pi_{\theta}(y \mid x) = \prod_{t=1}^{T} \pi_{\theta}\!\left(y_t \mid x, y_{<t}\right),$$1

with adaptive semantic coefficient

$$\pi_{\theta}(y \mid x) = \prod_{t=1}^{T} \pi_{\theta}\!\left(y_t \mid x, y_{<t}\right),$$2

The first factor is a hard consistency gate. If business and semantic advantages disagree in sign, then $$\pi_{\theta}(y \mid x) = \prod_{t=1}^{T} \pi_{\theta}\!\left(y_t \mid x, y_{<t}\right),$$3 and the semantic term is removed entirely. The second factor performs magnitude balancing. Even when signs agree, semantic influence is attenuated if the two advantages have very different magnitudes. The paper emphasizes the resulting bound

$$\pi_{\theta}(y \mid x) = \prod_{t=1}^{T} \pi_{\theta}\!\left(y_t \mid x, y_{<t}\right),$$4

whenever $$\pi_{\theta}(y \mid x) = \prod_{t=1}^{T} \pi_{\theta}\!\left(y_t \mid x, y_{<t}\right),$$5. This is the operational meaning of “anchoring optimization on business rewards”: semantic contribution can reinforce business advantage, but it cannot exceed it in effective magnitude (Jiang et al., 11 Feb 2026).

The final objective is a clipped likelihood-ratio objective in PPO/GRPO style:

$$\pi_{\theta}(y \mid x) = \prod_{t=1}^{T} \pi_{\theta}\!\left(y_t \mid x, y_{<t}\right),$$6

Several negative statements are explicit and significant. The paper does not specify a stop-gradient operator, an explicit value-function network $$\pi_{\theta}(y \mid x) = \prod_{t=1}^{T} \pi_{\theta}\!\left(y_t \mid x, y_{<t}\right),$$7, a token-level reward decomposition, a discount factor $$\pi_{\theta}(y \mid x) = \prod_{t=1}^{T} \pi_{\theta}\!\left(y_t \mid x, y_{<t}\right),$$8, or off-policy correction beyond the PPO-style importance ratio. The rewards appear to be sequence-level candidate rewards assigned to each generated SID sequence, and the policy gradient is applied to the whole sequence through the likelihood ratio (Jiang et al., 11 Feb 2026).

4. Integration with PSJ and the S-GRec training pipeline

A2PO is one component in a larger semantic-aware recommendation pipeline. The generator is initialized by supervised pretraining; the paper states that the recommender backbone is Qwen2.5-1.5B with RQ-VAE tokenization and is “initialized from the MiniOneRec SFT checkpoint.” Separately, PSJ is trained offline in two stages: aspect-level scorer training from human-verified pointwise labels, followed by user-conditional preference aggregation via pairwise preference learning. Once PSJ is trained, it serves as an offline reward provider during RL-style fine-tuning of the generator (Jiang et al., 11 Feb 2026).

During this fine-tuning stage, a rollout group of candidate sequences is sampled for each context $$\pi_{\theta}(y \mid x) = \prod_{t=1}^{T} \pi_{\theta}\!\left(y_t \mid x, y_{<t}\right),$$9. Business reward is computed broadly, while PSJ supplies semantic reward on only a subset of instances. The deployment section states that the Business Reward Model scores 100% of the data, while PSJ evaluates only 5%. The updated lightweight generator is then periodically deployed, and critically there is no real-time LLM inference at serving time (Jiang et al., 11 Feb 2026).

This sparse-query regime is central to the system design. The paper states that sparse semantic querying with $x$0 is sufficient to reach 99.1% and 99.6% of full performance on Office and Industrial, respectively. It also reports that querying PSJ on only 5% of instances reduces semantic supervision cost by roughly 20×. A plausible implication is that A2PO is designed not only for objective alignment but also for cost-efficient distillation of expensive semantic supervision into the lightweight generator.

5. Empirical behavior, ablations, and observed conflict

A2PO is motivated by the empirical claim that conflict between business and semantic directions is common rather than exceptional. The paper reports the empirical directional consistency rate between $x$1 and $x$2 as only 45.4% on Amazon benchmarks and 49.8% in the online advertising system. This means that roughly half of the time, the two directions disagree, which is the strongest experimental support for the asymmetry design (Jiang et al., 11 Feb 2026).

The ablation study directly contrasts A2PO with simpler fusion rules.

Variant	Office HR@10 / NDCG@10	Industrial HR@10 / NDCG@10
MiniOneRec	0.1634 / 0.1242	0.1586 / 0.1167
Reward-Sum	0.0904 / 0.0761	0.0585 / 0.0543
Adv-Sum	0.1515 / 0.1223	0.1348 / 0.1020
w/o Gate	0.1691 / 0.1302	0.1591 / 0.1169
w/o Magnitude	0.1652 / 0.1294	0.1624 / 0.1189
S-GRec / full A2PO	0.1689 / 0.1308	0.1632 / 0.1202

The most dramatic result is the collapse of Reward-Sum, which the paper describes as “catastrophic collapse.” Raw reward-level summation reduces Office HR@10 from 0.1634 to 0.0904 and Industrial HR@10 from 0.1586 to 0.0585. Advantage-level standardization without asymmetry, denoted Adv-Sum, recovers much of the damage but still underperforms the business-only baseline. Removing either submodule from full A2PO also hurts: gate removal is especially damaging on Industrial NDCG@10, whereas removing magnitude control also lowers final performance. The authors interpret these results as evidence that both the consistency gate and the magnitude-balancing term matter (Jiang et al., 11 Feb 2026).

At end-to-end level, the full S-GRec system improves HR@10 and NDCG@10 by 2.9% and 3.0% on Industrial, and by 3.4% and 5.3% on Office, relative to MiniOneRec. In online A/B testing on WeChat Channels advertising traffic, the deployed system yields +1.19% GMV, +1.55% GMV-Normal, +1.16% CTR, and −2.02% dislike rate versus the production baseline. The paper presents these results as evidence that semantic supervision can be exploited without sacrificing business goals, which is precisely the role assigned to A2PO (Jiang et al., 11 Feb 2026).

6. Terminological ambiguity and relation to adjacent methods

The acronym “A2PO” is not stable across the recent literature. In the S-GRec paper, A2PO denotes Asymmetric Advantage Policy Optimization (Jiang et al., 11 Feb 2026). In a distinct offline RL paper, A2PO denotes Advantage-Aware Policy Optimization, a TD3+BC-style method that uses a CVAE conditioned on critic-based advantage to disentangle mixed-quality behavior policies in D4RL (Qing et al., 2024). In cooperative MARL, A2PO denotes Agent-by-agent Policy Optimization, a sequential multi-agent trust-region method in which agents are updated one by one with PreOPC-corrected advantages (Wang et al., 2023). These are separate algorithms with different objectives, problem settings, and mathematical structures.

This ambiguity makes taxonomy important. The S-GRec version is neither an offline RL behavior-modeling method nor a multi-agent ordered update scheme. Its asymmetry is not based on separate positive-versus-negative token handling, as in ASPO’s advantage-sign-conditioned importance-sampling rule (Wang et al., 7 Oct 2025), nor on an architectural large-actor/small-critic asymmetry, as in AsyPPO’s mini-critic ensemble (Liu et al., 2 Oct 2025). It is most accurately described as a sequence-level, group-based, PPO/GRPO-style on-policy optimization method with dual reward streams and asymmetric fused advantages (Jiang et al., 11 Feb 2026).

A persistent misconception is therefore that “A2PO” names a single algorithmic family. The evidence in the literature suggests instead that the acronym has been reused for at least three unrelated formulations, of which the S-GRec method is specifically the business-anchored semantic-reward fusion mechanism introduced for industrial generative recommendation (Jiang et al., 11 Feb 2026).