Reinforced Strategy Optimization

• RSO is a family of feedback-driven methods that decompose decision-making into explicit high-level strategies optimized via reinforcement learning.
• It spans varied applications such as solver-aware modeling in SAGE, hierarchical planning in conversational recommenders, Q-learning-based feature selection, and iterative gene-panel optimization.
• Common limitations include dependence on externally defined structures, reward model quality, and the need for warm-start strategies for effective convergence.

Reinforced Strategy Optimization (RSO) is a non-canonical term used in several arXiv literatures for procedures that optimize high-level decision strategies under explicit feedback, but the acronym does not identify a single standardized algorithm. In contemporary usage, it denotes: a strategy-aware post-training pipeline within SAGE for automated optimization modeling; a hierarchical Planner-plus-Actor framework for conversational recommender systems; a Q-learning augmentation of Bee Swarm Optimization for feature selection; an iterative reinforcement-learning scheme for gene-panel selection summarized as an RSO approach within RiGPS; a distinct acronym for Statistical Rejection Sampling Optimization in offline preference optimization; and a broader reinforcement-learning program for strategic recommendations (Zhao et al., 4 May 2026, Zhao, 30 Sep 2025, Basak et al., 2021, Zhang et al., 2024, Liu et al., 2023, Theocharous et al., 2020).

1. Terminological scope and naming

The literature uses the same acronym for several unrelated research programs. Some are explicitly about “strategy” in the sense of macro-level action planning, while others apply reinforcement-learning or rejection-sampling ideas to search, selection, or alignment problems.

Usage of RSO	Domain	Core mechanism
SAGE	Optimization modeling with reasoning LLMs	Multi-strategy dataset + Segment-Weighted GRPO
Planner + Network-of-Experts	Conversational recommender systems	Macro strategy planning + micro adaptation
Reinforced Swarm Optimization	Feature selection	Bee Swarm Optimization + Q-learning
RiGPS summary as RSO	Gene panel selection	Pre-filtering + knowledge injection + actor-critic
Statistical Rejection Sampling Optimization	Preference optimization	Rejection sampling from an estimated optimal policy
Strategic recommendations	Sequential recommendation systems	MDPs, off-policy evaluation, safe deployment

A common misconception is to treat RSO as a single method with a shared mathematical core. The cited papers instead use the acronym for distinct constructions with different state spaces, objectives, reward definitions, and optimization dynamics. In some cases the relation is conceptual rather than terminological: the strategic recommendations literature formalizes sequential intervention policies with reinforcement learning, whereas the offline preference-optimization paper uses “RSO” for rejection sampling rather than reinforced strategy planning (Liu et al., 2023, Theocharous et al., 2020).

2. Strategy-aware optimization modeling in SAGE

In “Strategy-Aware Optimization Modeling with Reasoning LLMs,” RSO is the post-training mechanism inside SAGE that teaches a reasoning LLM not only to generate correct optimization code but also to “explicitly choose and commit to high-level modeling strategies” such as “flow-based vs. assignment-based,” with solver efficiency treated as part of the optimization target (Zhao et al., 4 May 2026).

The pipeline has two major components. First, SAGE constructs a multi-strategy, solver-verified corpus. For each operations-research problem $q_i$, a “strategy teacher” proposes $K$ distinct modeling paradigms,

$S(q_i)=\{s_i^{(1)},\dots,s_i^{(K)}\}.$

A second LLM then generates a proof-style trace $r_i^{(k)}$ and solver-executable Gurobi Python code $c_i^{(k)}$ for each strategy. Executions that crash, return infeasible or unbounded when a solution is expected, or yield an objective deviating from ground truth are discarded. An LLM-as-Judge then removes semantically redundant strategies, producing

$$\mathcal{D}_{SFT}=\{(q_i,r_i^{(k)},c_i^{(k)})\mid k\in K_i\}.$$

This dataset is used for supervised fine-tuning so that the student model learns “strategy-conditioned traces and code” before reinforcement learning refines strategy choice and solver-efficiency trade-offs.

Second, SAGE applies Segment-Weighted GRPO, a critic-free policy-gradient variant that imposes a three-segment reasoning template—<strategy>, <modeling>, <check>—followed by code. The return is a composite solver-informed reward,

$$R(y)=R_{\text{format}}(y)+R_{\text{outcome}}(y)+R_{\text{eff}}(y).$$

Here $R_{\text{format}}\in[0,1]$ counts correctly emitted tags from $$\{\texttt{<think>},\texttt{<strategy>},\texttt{<modeling>},\texttt{<check>},\texttt{code}\}$$, each worth $0.2$; $K$0 is $K$1 only when the code runs and matches ground truth; and $K$2 applies only when $K$3, with

$K$4

where $K$5 is a solver-reported metric such as simplex iterations for LP or a weighted combination of branch-and-bound nodes and optimality gap for MILP. Credit assignment is then biased toward early paradigm decisions through segment weights satisfying

$K$6

with examples $K$7.

Empirically, SAGE improves average pass@1 from $K$8 to $K$9 over the strongest open-source baseline across eight benchmarks. In the complex subset it yields a $S(q_i)=\{s_i^{(1)},\dots,s_i^{(K)}\}.$0 absolute gain in pass@1. Under Pass@16, correct-formulation diversity measured by distinct LP components increases by $S(q_i)=\{s_i^{(1)},\dots,s_i^{(K)}\}.$1–$S(q_i)=\{s_i^{(1)},\dots,s_i^{(K)}\}.$2 over baselines. At $S(q_i)=\{s_i^{(1)},\dots,s_i^{(K)}\}.$3 problem size, SAGE formulations use $S(q_i)=\{s_i^{(1)},\dots,s_i^{(K)}\}.$4 fewer constraints and exhibit $S(q_i)=\{s_i^{(1)},\dots,s_i^{(K)}\}.$5–$S(q_i)=\{s_i^{(1)},\dots,s_i^{(K)}\}.$6 lower solve times. Ablations report that RL improves average accuracy by $S(q_i)=\{s_i^{(1)},\dots,s_i^{(K)}\}.$7, the three-segment template contributes $S(q_i)=\{s_i^{(1)},\dots,s_i^{(K)}\}.$8, segment weighting adds $S(q_i)=\{s_i^{(1)},\dots,s_i^{(K)}\}.$9, and including the efficiency reward yields $r_i^{(k)}$0 on complex benchmarks. These results operationalize RSO as explicit strategy planning coupled to solver-verified execution and solver-aware reward shaping (Zhao et al., 4 May 2026).

3. Hierarchical RSO in conversational recommender systems

In “Reinforced Strategy Optimization for Conversational Recommender Systems via Network-of-Experts,” RSO is a hierarchical framework that decomposes response generation into “macro-level strategy planning and micro-level adaptation.” A Planner Expert selects a macro strategy $r_i^{(k)}$1, while a Network-of-Experts performs preference reasoning, factual retrieval, and response realization (Zhao, 30 Sep 2025).

The method is cast as an MDP with dialogue state $r_i^{(k)}$2, macro action space $r_i^{(k)}$3, and a two-level policy

$r_i^{(k)}$4

optimized for

$r_i^{(k)}$5

The Planner implements $r_i^{(k)}$6, where $r_i^{(k)}$7, using a lightweight LLM such as a “RoBERTa-based” model that outputs a softmax over the candidate strategy set. Training proceeds in two stages: supervised fine-tuning with

$r_i^{(k)}$8

followed by entropy-regularized policy gradient,

$r_i^{(k)}$9

Micro-level adaptation is distributed across four experts. The Preference Reasoner produces $c_i^{(k)}$0, the Fact Retriever returns $c_i^{(k)}$1, the Actor generates $c_i^{(k)}$2, and a fixed Rewarder Expert returns $c_i^{(k)}$3. There is “no learned gating network”; instead, the Actor prompt concatenates $c_i^{(k)}$4, the preference summary, retrieved facts, and dialogue history. Turn-level rewards are supplied by an LLM judge as $c_i^{(k)}$5, normalized to $c_i^{(k)}$6, and averaged over $c_i^{(k)}$7 judge samples. The algorithm samples strategies rather than taking argmax, uses an entropy bonus to prevent premature collapse, employs $c_i^{(k)}$8, and terminates early if $c_i^{(k)}$9.

Experiments use Inspired and ReDial, with evaluation on Conversation Success Rate, Watching Intention, Persuasiveness, Credibility, Diversity, and recommendation metrics including Recall@1, Recall@5, and Rec-SR. RSO achieves the highest WI, PRS, Cred, and Conv-SR, with competitive Distinct-2 on both datasets. It reports up to $$\mathcal{D}_{SFT}=\{(q_i,r_i^{(k)},c_i^{(k)})\mid k\in K_i\}.$$0 relative Rec-SR improvement on Inspired and $$\mathcal{D}_{SFT}=\{(q_i,r_i^{(k)},c_i^{(k)})\mid k\in K_i\}.$$1 on ReDial over UniCRS. Ablations show that removing the Fact Retriever lowers Credibility by more than $$\mathcal{D}_{SFT}=\{(q_i,r_i^{(k)},c_i^{(k)})\mid k\in K_i\}.$$2 points, removing the Planner reduces Conv-SR from $$\mathcal{D}_{SFT}=\{(q_i,r_i^{(k)},c_i^{(k)})\mid k\in K_i\}.$$3, and removing the Preference Reasoner lowers Conv-SR and Rec-SR by $$\mathcal{D}_{SFT}=\{(q_i,r_i^{(k)},c_i^{(k)})\mid k\in K_i\}.$$4. Strategy-distribution analysis shows that SFT alone concentrates on “safe” strategies such as Credibility and Opinion Inquiry, whereas entropy-regularized RL spreads mass to under-used strategies such as Similarity and Rephrase Preference. The reported limitations are dependence on an external knowledge graph and pre-defined strategy set $$\mathcal{D}_{SFT}=\{(q_i,r_i^{(k)},c_i^{(k)})\mid k\in K_i\}.$$5, possible bias and higher compute cost from the LLM-judge reward model, and the fact that the micro-experts are not learned end-to-end (Zhao, 30 Sep 2025).

4. Reinforced swarm optimization for feature selection

In “RSO: A Novel Reinforced Swarm Optimization Algorithm for Feature Selection,” RSO denotes a wrapper-based feature-selection algorithm that embeds a Q-learning-style reinforcement learner inside Bee Swarm Optimization. Each bee retains the local-search dynamics of BSO but also maintains a Q-table over $$\mathcal{D}_{SFT}=\{(q_i,r_i^{(k)},c_i^{(k)})\mid k\in K_i\}.$$6 pairs so that successful feature-inclusion or feature-exclusion moves are rewarded and unsuccessful ones penalized (Basak et al., 2021).

The state space is the binary feature-subset vector $$\mathcal{D}_{SFT}=\{(q_i,r_i^{(k)},c_i^{(k)})\mid k\in K_i\}.$$7, where a $$\mathcal{D}_{SFT}=\{(q_i,r_i^{(k)},c_i^{(k)})\mid k\in K_i\}.$$8 indicates that a feature is included. The action space consists of single-bit flips or small bit-block swaps. Rewards are defined after evaluating the candidate subset $$\mathcal{D}_{SFT}=\{(q_i,r_i^{(k)},c_i^{(k)})\mid k\in K_i\}.$$9 with a K-NN classifier; the reward structure encourages improved accuracy and penalizes larger subsets, with an additional positive term when subset size is reduced. Action selection uses an $$R(y)=R_{\text{format}}(y)+R_{\text{outcome}}(y)+R_{\text{eff}}(y).$$0-greedy policy, and Q-values are updated through

$$R(y)=R_{\text{format}}(y)+R_{\text{outcome}}(y)+R_{\text{eff}}(y).$$1

The algorithm initializes a population of $$R(y)=R_{\text{format}}(y)+R_{\text{outcome}}(y)+R_{\text{eff}}(y).$$2 feature subsets, repeatedly chooses actions for each bee, evaluates $$R(y)=R_{\text{format}}(y)+R_{\text{outcome}}(y)+R_{\text{eff}}(y).$$3 by 5-fold KNN, updates Q-values, and either accepts $$R(y)=R_{\text{format}}(y)+R_{\text{outcome}}(y)+R_{\text{eff}}(y).$$4, rejects it, or reinitializes the bee when a “ChanceMax” threshold of failures is exceeded. Exploration and exploitation are governed by the $$R(y)=R_{\text{format}}(y)+R_{\text{outcome}}(y)+R_{\text{eff}}(y).$$5-schedule, the “Flip” parameter, and ChanceMax.

The paper reports experiments on 25 UCI datasets containing balanced and imbalanced data. The abstract states that the proposed model outperforms BSO in 22 out of 25 instances $$R(y)=R_{\text{format}}(y)+R_{\text{outcome}}(y)+R_{\text{eff}}(y).$$6 and performs best among all compared methods in 19 out of 25 cases $$R(y)=R_{\text{format}}(y)+R_{\text{outcome}}(y)+R_{\text{eff}}(y).$$7. The detailed summary further reports that, on average, RSO used $$R(y)=R_{\text{format}}(y)+R_{\text{outcome}}(y)+R_{\text{eff}}(y).$$8–$$R(y)=R_{\text{format}}(y)+R_{\text{outcome}}(y)+R_{\text{eff}}(y).$$9 fewer features than BSO and reduced runtime by $R_{\text{format}}\in[0,1]$0–$R_{\text{format}}\in[0,1]$1, while convergence plots show RSO “climbing more steadily and never stalling in local optima,” in contrast to BSO’s early flatlining. In this usage, RSO is not about symbolic reasoning or dialogue strategy; it is a reinforced local-search scheme for adaptive subset search (Basak et al., 2021).

5. Iterative gene-panel selection and pre-filtering synergy

In “Enhanced Gene Selection in Single-Cell Genomics: Pre-Filtering Synergy and Reinforced Optimization,” the reinforcement-based procedure is summarized as an RSO approach within the RiGPS framework. The task is to select a compact gene panel $R_{\text{format}}\in[0,1]$2 that maximizes a clustering-quality metric $R_{\text{format}}\in[0,1]$3 after pre-filtering (Zhang et al., 2024).

Formally,

$R_{\text{format}}\in[0,1]$4

optionally subject to $R_{\text{format}}\in[0,1]$5. The RL formulation uses one binary decision agent per candidate gene in $R_{\text{format}}\in[0,1]$6. At iteration $R_{\text{format}}\in[0,1]$7, the currently selected genes $R_{\text{format}}\in[0,1]$8 are summarized through descriptive statistics, flattened, and passed through a shared autoencoder to produce the latent state $R_{\text{format}}\in[0,1]$9. Each agent chooses $$\{\texttt{<think>},\texttt{<strategy>},\texttt{<modeling>},\texttt{<check>},\texttt{code}\}$$0, producing $$\{\texttt{<think>},\texttt{<strategy>},\texttt{<modeling>},\texttt{<check>},\texttt{code}\}$$1. The reward balances “spatial separability” and compactness:

$$\{\texttt{<think>},\texttt{<strategy>},\texttt{<modeling>},\texttt{<check>},\texttt{code}\}$$2

where $$\{\texttt{<think>},\texttt{<strategy>},\texttt{<modeling>},\texttt{<check>},\texttt{code}\}$$3 is a normalized mutual information term and $$\{\texttt{<think>},\texttt{<strategy>},\texttt{<modeling>},\texttt{<check>},\texttt{code}\}$$4 rewards smaller panels. Optimization uses an actor-critic scheme with prioritized replay, with critic loss

$$\{\texttt{<think>},\texttt{<strategy>},\texttt{<modeling>},\texttt{<check>},\texttt{code}\}$$5

and policy-gradient actor updates based on $$\{\texttt{<think>},\texttt{<strategy>},\texttt{<modeling>},\texttt{<check>},\texttt{code}\}$$6.

RiGPS begins with a pre-filtering stage in which $$\{\texttt{<think>},\texttt{<strategy>},\texttt{<modeling>},\texttt{<check>},\texttt{code}\}$$7 basic filter methods yield per-gene scores, each method is weighted by downstream clustering NMI, and genes with meta-score $$\{\texttt{<think>},\texttt{<strategy>},\texttt{<modeling>},\texttt{<check>},\texttt{code}\}$$8 form $$\{\texttt{<think>},\texttt{<strategy>},\texttt{<modeling>},\texttt{<check>},\texttt{code}\}$$9. A subsequent “knowledge injection phase” seeds each agent’s replay buffer with experiences derived from existing gene-selection algorithms. Reinforced iterations then alternate exploration and optimization until convergence. The summary notes that the actor-critic scheme with experience replay is known to converge to a local optimum under standard RL assumptions, but “no formal proof is given in the paper.”

Empirically, across 25 datasets RiGPS achieves the highest NMI on 21 out of 25 datasets and ranks in the top-3 on all. Ablations report that removing RL lowers NMI by 5–15 points, removing knowledge injection lowers NMI by 3–8 points, and removing pre-filtering lowers NMI by 4–10 points. Efficiency analysis indicates that parameter count and training time scale linearly with cell count. On the Puram dataset, t-SNE and heatmaps indicate better separated clusters and more distinct cell-type patterns. In convergence comparison, RiGPS reaches final NMI in about 20 iterations, whereas geneBasis takes more than 50 and plateaus lower. This usage of RSO is therefore an iterative combinatorial optimization scheme over gene subsets rather than a dialogue or symbolic-planning method (Zhang et al., 2024).

6. Preference optimization and strategic recommendation usages

The acronym RSO also denotes “Statistical Rejection Sampling Optimization” in offline preference optimization. In “Statistical Rejection Sampling Improves Preference Optimization,” the goal is to overcome the off-policy sampling limitations of SLiC and DPO by approximating preference pairs drawn from the KL-regularized optimum

$0.2$0

A pairwise reward-ranking model $0.2$1 is first learned, and rejection sampling is then performed from the estimated optimum $0.2$2, with acceptance probability

$0.2$3

Accepted samples are paired and labeled by $0.2$4, after which either a logistic-norm loss or a hinge-norm loss is optimized. On Reddit TL;DR, RSO achieves $0.2$5 Proxy, $0.2$6 Gold, and $0.2$7 AutoSxS in the sigmoid-norm variant, and $0.2$8, $0.2$9, and $K$00 in the hinge-norm variant, exceeding the reported DPO and SLiC baselines. On AnthropicHH, the sigmoid-norm variant reports $K$01 Proxy, $K$02 Gold, and $K$03 AutoSxS, again surpassing the direct baselines. The paper identifies three limitations: extra reward-model inference cost, sensitivity to reward-model quality, and the fact that only one round of RSO is studied (Liu et al., 2023).

A broader reinforcement-learning conception of RSO appears in “Reinforcement Learning for Strategic Recommendations,” where the sequential recommendation problem is formalized as a finite-horizon MDP

$K$04

with return

$K$05

and policy objective $K$06. The framework covers actor-critic updates, fitted Q-iteration, high-confidence off-policy evaluation through trajectory-wise and per-decision importance sampling, safe policy improvement with confidence bounds, and an online master loop called “Daedalus.” It further addresses non-stationarity via time-series forecasting of OPE estimates, bootstrapping from passive data through a probabilistic suffix tree and DS-PSRL, resource-constrained multi-user systems via column generation, large action spaces through action embeddings and “PG-RA,” and dynamic action sets via LAICA. Reported empirical claims include up to $K$07 higher normalized reward in POI experiments for DS-PSRL, $K$08–$K$09 faster convergence for PG-RA on large action spaces, and statistically significant lift in lifetime engagements across industrial deployments. In this usage, RSO is best understood as strategic sequential decision-making under safety, uncertainty, and deployment constraints rather than as a single named update rule (Theocharous et al., 2020).

7. Recurrent design principles, limitations, and points of clarification

Taken together, these works suggest that the stable content of “RSO” lies less in a shared optimizer than in a recurring design pattern: explicit strategy or action decomposition, task-specific reward engineering, and a learning mechanism that allocates credit to decisions that are otherwise implicit. In SAGE, the decomposition is the enforced <strategy>, <modeling>, <check> template with segment-weighted GRPO; in conversational recommendation it is the split between a Planner and a Network-of-Experts; in RiGPS it is the combination of pre-filtering, knowledge injection, and actor-critic refinement; and in reinforced swarm optimization it is the conversion of stochastic bee moves into remembered state-action decisions (Zhao et al., 4 May 2026, Zhao, 30 Sep 2025, Zhang et al., 2024, Basak et al., 2021).

A second recurrent pattern is warm-starting or prior injection. SAGE begins from supervised fine-tuning on $K$10; the CRS planner is warmed up with annotated macro strategies; RiGPS injects experiences from existing gene-selection methods into replay buffers; and the strategic-recommendation framework bootstraps active control from passive-data models. This suggests that, across domains, the reinforced component is typically not asked to discover syntax, representation, or plausible priors from scratch (Zhao et al., 4 May 2026, Zhao, 30 Sep 2025, Zhang et al., 2024, Theocharous et al., 2020).

The principal limitations are also recurrent. Several variants depend on externally specified structures: the CRS framework requires a pre-defined strategy set $K$11 and an external knowledge graph, SAGE depends on solver execution and solver-reported metrics, and strategic recommendations require reliable logged data for safe off-policy evaluation. Reward quality is another shared bottleneck: the CRS Rewarder Expert may introduce bias and higher compute cost, and the preference-optimization RSO explicitly states that a poor reward model produces noisy preference pairs. Formal guarantees are uneven: RiGPS notes that no formal proof is given in the paper, while the strategic-recommendation framework places unusual emphasis on confidence-bounded deployment and safe policy improvement (Zhao, 30 Sep 2025, Liu et al., 2023, Zhang et al., 2024, Theocharous et al., 2020).

Accordingly, cross-domain claims about “RSO” should be interpreted with care. The acronym names a family of reinforcement- or feedback-driven optimization strategies, but the underlying objects of optimization differ sharply: solver-efficient formulations, dialogue strategies, feature subsets, gene panels, preference pairs, or long-term intervention policies. The technical significance of any particular RSO result therefore depends on the specific state representation, reward design, and deployment regime of the corresponding paper rather than on the acronym alone.