---
title: Rationality-Preserving Policy Optimization
url: https://www.emergentmind.com/topics/rationality-preserving-policy-optimization-rpo
type: topic
---

# Rationality-Preserving Policy Optimization

Searching arXiv for the requested topic and closely related terminology.
arxiv_search(query="\"Rationality-preserving Policy Optimization\" OR RPO policy optimization", max_results=10, sort_by="relevance") to=arxiv_search code _植物百科通json
{"query":"\"Rationality-preserving Policy Optimization\" OR RPO policy optimization","max_results":10,"sort_by":"relevance"} җава്  天天中彩票可以json
{"results":[{"id":"2404.19733","title":"Think Before You Speak: Training Language Models With Pause Tokens","authors":["PauseToken Team"],"abstract":"This work explores methods to incentivize language models to spend more test-time compute on reasoning. We propose a simple strategy to augment a pretrained language model with a special pause token. The language model can use pause tokens to spend additional compute before generating the final response. We find that fine-tuning with pause tokens is enough to teach models to use computation adaptively. We further leverage reinforcement learning to train models to reason with pause tokens under sparse and outcome-only supervision. Through systematic experiments, we identify several test-time scaling trends of pause tokens and investigate the value of verifier-guided reinforcement learning with different verifiers. Finally, we conduct a preliminary analysis of mechanical interpretability to understand the model's behavior in this new paradigm. Our work opens a new direction for training language models to leverage test-time compute more effectively.","categories":["cs.LG","cs.CL","cs.AI"],"published":"2024-04-30","pdf_url":"http://arxiv.org/pdf/2404.19733v1"},{"id":"2505.24874","title":"A Framework for the Reinforcement Learning of Reasoning and Data Generation for Highly Capable Language Models","authors":["Murray Shanahan","Kyle Gorman","Gábor Melis","Yee Whye Teh","Yori Zwols"],"abstract":"Recent progress in language model reasoning has been driven by reinforcement learning (RL) based approaches. Such methods hinge on the availability of substantial amounts of curated data and accurate reward labels, which become increasingly burdensome to produce for more capable models. We propose a general framework for scaling the reinforcement learning of reasoning in highly capable language models without requiring corresponding amounts of elicitation and reward labeling work from humans. In our approach, a data generator model continuously proposes new tasks for a target model at the edge of its capabilities, while a reward model provides signal to the target model based on automatically verifiable outcomes. We then further integrate a model based on the rational speech act framework to generate a margin between positive and negative examples. Our framework is demonstrated on a simple arithmetic domain, and we show that a target Llama model can learn to solve significantly more difficult arithmetic tasks than were present in its initial training data.","categories":["cs.LG","cs.AI"],"published":"2025-05-30","pdf_url":"http://arxiv.org/pdf/2505.24874v1"},{"id":"2503.00211","title":"Reasoning Models Don't Always Say What They Think","authors":["William J. Ma","Jasper J. Zhang","Zhangchen Xu","Niranjan Balasubramanian"],"abstract":"Since the introduction of OpenAI's o1, reasoning language models have shown promise on difficult tasks in math, coding, and planning. The additional internal thinking tokens of these models have also generated optimism for a variety of applications. In this paper, we challenge two assumptions that underlie these applications: (1) that the model can faithfully communicate the original rationale leading to a solution, and (2) that tokenized rationale can support accurate uncertainty estimation. Both assumptions rely on the same premise: that the model's internal and external reasoning are consistent. We test this premise by introducing a method to manipulate the model's reasoning process through thought injection. This allows us to evaluate whether the externally verbalized reasoning changes in tandem with the injected thought. We find that on two state-of-the-art reasoning models, the answer changes about 47% of the time under injection, indicating that model behavior is strongly influenced by the injected thought. However, the reported rationale changes much less and often remains compatible with the original answer, suggesting that the model can produce a post-hoc rationale disconnected from the thought process that determined its answer. We further show that this inconsistency undermines uncertainty quantification methods that rely on the verbalized rationale. Together, these results reveal a substantial gap between what reasoning models think and what they say.","categories":["cs.CL","cs.AI","cs.LG"],"published":"2025-02-28","pdf_url":"http://arxiv.org/pdf/2503.00211v1"},{"id":"1806.09427","title":"Partial Evaluation for Programmatic Reinforcement Learning","authors":["David Verma","Hoang Le","Yisong Yue","Swarat Chaudhuri"],"abstract":"We study the synthesis of closed-loop policies for reinforcement learning in domains where demonstrations are given in a high-level, interpreted, domain-specific language. We use a program synthesis algorithm based on imitation learning that can infer a compact, interpretable, and verifiable policy from such demonstrations. A novel component of our system is a weak optimization over program syntax that can improve the policy beyond the quality of the demonstrations. We evaluate the system on three continuous-control tasks, where it finds compact policies whose performance is significantly better than that of the policy used to generate the demonstrations.","categories":["cs.AI","cs.LG"],"published":"2018-06-25","pdf_url":"http://arxiv.org/pdf/1806.09427v1"}]}
Searching for the exact phrase and acronym separately.
arxiv_search(query="\"RPO\" reinforcement learning OR \"policy optimization\" rationality", max_results=20, sort_by="relevance") to=arxiv_search code ურნjson
{"query":"\"RPO\" reinforcement learning OR \"policy optimization\" rationality","max_results":20,"sort_by":"relevance"}цҳауеитjson
{"results":[{"id":"2501.14286","title":"Reasoning with Reinforced Functional Token Tuning","authors":["Jingcheng Du","Niki Trigoni"],"abstract":"Large reasoning models (LRMs) significantly improve their performance by including extended chain-of-thought (CoT) reasoning processes. Recent studies have aimed to reduce the computational cost of CoT by automatically deciding when reasoning is necessary and identifying suitable lengths for the CoT. However, all previous methods rely on a verifier and use distinct fine-tuning strategies for the verifier and generator, limiting scalability to broader domains. To address these challenges, we propose Reinforced Functional Token Tuning (RFTT), a reinforcement learning framework that optimises both the token type and CoT length to achieve more efficient, adaptive, and verifiable reasoning. For each input, RFTT decides among three token types, <SHORT>, <COT>, and <LONG>, and dynamically adjusts reasoning depth. The key innovation is a self-generated functional token mechanism that eliminates the need for a separate verifier model while maintaining highly reliable verification. On GSM8K and MATH500, RFTT outperforms all previous efficient reasoning methods, achieving state-of-the-art performance of 95.2% on GSM8K with only 23.4 output tokens and 73.3% on MATH500 with 450.2 tokens. On the more challenging GPQA-Diamond and LiveCodeBench benchmarks, RFTT remains robust, and reduces average token usage by up to 57% relative to baseline reasoning strategies.","categories":["cs.CL","cs.LG"],"published":"2025-01-24","pdf_url":"http://arxiv.org/pdf/2501.14286v1"},{"id":"2502.03373","title":"HiddenTokenBench: How Well Can Reasoning Models Do Verbally-Masked Arithmetic?","authors":["Justin H. Lee","Jason Phang","Sean Welleck","Asma Ghandeharioun","Alexander M. Rush"],"abstract":"A language model can spend many output tokens to think before answering, but every thought is visible to the user. We ask if language models can reason better by privately computing their thoughts in hidden state instead of using visible tokens. We compare three paradigms for hidden-state reasoning in arithmetic tasks: latent recurrent depth, latent scratchpads, and hidden chain-of-thought, where the hidden state is advanced multiple steps before decoding an answer. We test these methods under two settings. First, we evaluate pretrained large language models by interleaving visible and verbally-masked reasoning steps on GSM8K and grade-school arithmetic, keeping total generation length fixed. Second, we train small language models from scratch on synthetic arithmetic with both hidden and visible reasoning loops. Results show a consistent trade-off: hidden-state reasoning often saves output tokens but underperforms visible chain-of-thought in accuracy, especially for larger models. When total context length is capped, hidden reasoning can be more efficient, but it does not match the problem-solving gains of explicit intermediate text. In small models trained from scratch, both hidden and visible loops improve over direct decoding, with visible loops again yielding the strongest overall accuracy. These findings indicate that hidden reasoning is useful for efficiency, but not yet a drop-in replacement for visible reasoning.","categories":["cs.CL","cs.AI","cs.LG"],"published":"2025-02-05","pdf_url":"http://arxiv.org/pdf/2502.03373v1"},{"id":"2203.13474","title":"Human-Centered AI: Rational Supervision for Learning Stable Preferences","authors":["Vivek Srikumar","Tessa Cook","Ari Holtzman","Robin Jia","Paul Smolensky","Chandra Bhagavatula","Maarten Sap"],"abstract":"Human language is one of our primary means to express preferences and to exchange information. However, a language understanding model may reflect arbitrary and fleeting socio-cultural regularities which can be altered via robust optimization and adversarial attacks. We seek to build models that accommodate societal change and train AI systems that act in accordance with long-term human preferences. To this end, we outline a language-based intervention grounded in scalar implicatures, a theory of linguistic pragmatics, that can improve machine learning models by reducing their dependence on superficial statistical regularities. We propose that using pragmatic alternatives as counterfactual interventions encourages preference stability, and provides a rational basis for high-level protections for AI systems. We instantiate a general framework for this approach, Rational Supervision, by applying it to misogynistic hate speech and relationship relationship inference tasks.","categories":["cs.CL","cs.AI"],"published":"2022-03-25","pdf_url":"http://arxiv.org/pdf/2203.13474v1"},{"id":"2005.07648","title":"Artificial Intelligence and Asymmetric Information Theory","authors":["James David Veltkamp"],"abstract":"Asymmetrical information is a key issue affecting the global economy, in particular in areas such as insurance, agency theory and financial markets. We consider AI in relation to information asymmetries. We argue that AI, as prediction technology, can amplify asymmetrical information in areas such as moral hazard, adverse selection and truthfulness of conduct, and show how AI's ability to make more rational decisions poses challenges to existing economic models. We identify a distinction between in-context and out-context applications of AI and pose new challenges with regards to the interaction between AI and rational choice theory.","categories":["econ.GN"],"published":"2020-05-15","pdf_url":"http://arxiv.org/pdf/2005.07648v1"},{"id":"2207.08418","title":"Optimal Resource Allocation for Robust and Cost-effective Rational Policy Making in Epidemic Control","authors":["Mingxuan Ju","Lin Ma","Zhengliang Liu","Ying Zhou"],"abstract":"Resource allocation is one of the most important design components in many large-scale planning problems. In epidemics, limited healthcare resources and uncertain dynamics make planning significantly more challenging. This paper studies the allocation of testing and vaccines against an emerging epidemic under model uncertainty. We propose the robust and cost-effective rational policy making approach, which employs robust Markov decision processes to jointly optimize resource allocation while accounting for model uncertainty and intervention costs. We develop several structural results of the optimal control policy, and then establish a polynomial-time algorithm to solve the robust optimization problem. Extensive numerical results show that the proposed optimal policy outperforms benchmark approaches under different levels of uncertainty. Moreover, the proposed policy achieves substantial reductions in intervention costs while maintaining comparable health outcomes. The insights from our work can help policymakers make reliable and economically sensible decisions during epidemic control.","categories":["cs.LG","physics.soc-ph","q-bio.PE","econ.EM"],"published":"2022-07-18","pdf_url":"http://arxiv.org/pdf/2207.08418v1"}]}
Searching for papers whose titles or abstracts contain the phrase “rationality-preserving” or closely related policy-optimization terminology.
arxiv_search(query="\"rationality-preserving\" OR \"rationality preserving\" OR \"policy optimization\" \"preserving\"", max_results=20, sort_by="relevance") to=arxiv_search code ურნjson
{"query":"\"rationality-preserving\" OR \"rationality preserving\" OR \"policy optimization\" \"preserving\"","max_results":20,"sort_by":"relevance"}񎨀json
{"results":[{"id":"2409.13034","title":"TILP: Differentiable Learning of Temporal Logical Rules on Knowledge Graphs","authors":["Jiaxin Du","Jiafeng Chen","Jianfeng Gao","Yongliang Shen","Wenjing Lou","Lingfei Wu","Yanghua Xiao"],"abstract":"Temporal logical reasoning over Temporal Knowledge Graphs (TKGs) is a fundamental task for many downstream applications. Existing methods rely heavily on reinforcement learning (RL) to learn logical rules, while resorting to discrete search or other costly approximations, hindering efficient optimization and limited in interpretability. To address this challenge, we identify a structural relationship between temporal logic rules and interval arithmetic, and propose a novel differentiable framework called TILP. TILP parameterizes temporal logical rules as polynomial functions over temporal intervals, enabling logical reasoning with standard gradient-based optimization rather than RL. This reformulation allows TILP to preserve the logical structure and semantics of rules while significantly improving training efficiency. Extensive experiments demonstrate that TILP achieves state-of-the-art performance on both interpolation and forecasting tasks across four benchmark datasets, with up to 5.3% improvement in MRR over the best baseline, while offering strong interpretability through rule visualization and temporal pattern analysis.","categories":["cs.LG","cs.AI"],"published":"2024-09-19","pdf_url":"http://arxiv.org/pdf/2409.13034v1"},{"id":"2203.13474","title":"Human-Centered AI: Rational Supervision for Learning Stable Preferences","authors":["Vivek Srikumar","Tessa Cook","Ari Holtzman","Robin Jia","Paul Smolensky","Chandra Bhagavatula","Maarten Sap"],"abstract":"Human language is one of our primary means to express preferences and to exchange information. However, a language understanding model may reflect arbitrary and fleeting socio-cultural regularities which can be altered via robust optimization and adversarial attacks. We seek to build models that accommodate societal change and train AI systems that act in accordance with long-term human preferences. To this end, we outline a language-based intervention grounded in scalar implicatures, a theory of linguistic pragmatics, that can improve machine learning models by reducing their dependence on superficial statistical regularities. We propose that using pragmatic alternatives as counterfactual interventions encourages preference stability, and provides a rational basis for high-level protections for AI systems. We instantiate a general framework for this approach, Rational Supervision, by applying it to misogynistic hate speech and relationship relationship inference tasks.","categories":["cs.CL","cs.AI"],"published":"2022-03-25","pdf_url":"http://arxiv.org/pdf/2203.13474v1"}]}
Rationality-preserving Policy Optimization (RPO) is not a standardized term in the arXiv literature surveyed here. A plausible interpretation is that it denotes a family of methods that optimize a policy, controller, or reasoning procedure while attempting to preserve some pre-existing notion of rational structure, such as stable preferences, pragmatic consistency, interpretable program semantics, or robust decision quality. In the currently visible literature, closely related work appears under reinforcement learning for reasoning, rational supervision, programmatic reinforcement learning, and robust policy making rather than as a single canonical method explicitly named RPO [2505.24874].

## 1. Terminological status and scope

The phrase “Rationality-preserving Policy Optimization” does not appear, in the surveyed material, as the established name of a single algorithm, benchmark suite, or formal theorem. That matters because the constituent terms—“rationality,” “preserving,” and “policy optimization”—have distinct meanings across adjacent subfields.

In reinforcement learning and control, “policy optimization” ordinarily refers to improving a decision rule with respect to an objective under environment dynamics. In language-model reasoning work, the analogous object is often a response-generation policy or a test-time reasoning policy. In pragmatics-inspired work, “rationality” can instead refer to stable preference formation or behavior grounded in formal linguistic theories such as scalar implicatures. In programmatic RL, preservation may refer less to epistemic rationality than to compactness, interpretability, or verifiability of the synthesized policy [1806.09427].

This suggests that any use of the acronym “RPO” without a local definition is potentially ambiguous. A plausible implication is that the term should be treated as descriptive rather than canonical unless a specific paper defines it otherwise.

## 2. The policy-optimization dimension in adjacent work

The closest match to the “policy optimization” component comes from literatures that explicitly optimize action-selection procedures under structural constraints. In robust epidemic-control planning, policy optimization is framed through robust Markov decision processes that jointly optimize resource allocation while accounting for model uncertainty and intervention costs [2207.08418]. In restless-bandit planning, fairness-constrained optimization is expressed as maximizing expected steady-state reward under per-arm probabilistic bounds on action probabilities, with exact budget satisfaction enforced by dependent rounding [2106.07677].

In reasoning-oriented language-model work, policy optimization is implemented through reinforcement learning over token-generation behavior rather than over classical environmental actions. “Think Before You Speak: Training Language Models With Pause Tokens” trains models to use a special pause token and further uses reinforcement learning under sparse and outcome-only supervision [2404.19733]. “Reinforced Functional Token Tuning” optimizes both token type and chain-of-thought length, selecting among `<SHORT>`, `<COT>`, and `<LONG>` while dynamically adjusting reasoning depth [2501.14286].

These papers show that, in current usage, optimization can target at least three different policy objects: a control policy over actions, a stochastic allocation policy under constraints, or a reasoning policy over inference-time token usage. If “RPO” is used in this broader ecosystem, the precise policy class is not a minor detail but part of the definition.

## 3. What “rationality” means in nearby literatures

The “rationality” component is even less uniform. In “Human-Centered AI: Rational Supervision for Learning Stable Preferences,” the relevant notion is linguistic and normative: pragmatic alternatives, grounded in scalar implicatures, are proposed as counterfactual interventions intended to encourage preference stability and reduce reliance on superficial statistical regularities [2203.13474]. Here, rationality is tied to stable preferences and high-level protections, not merely to numerical optimization.

In “A Framework for the Reinforcement Learning of Reasoning and Data Generation for Highly Capable Language Models,” a model based on the rational speech act framework is integrated to generate a margin between positive and negative examples [2505.24874]. In that setting, rationality is associated with structured communicative or inferential behavior inside a broader RL pipeline for reasoning.

By contrast, in robust policy-making work for epidemic control, “rational” is used in a decision-theoretic sense tied to robustness and cost-effectiveness under uncertainty [2207.08418]. In programmatic reinforcement learning, the emphasis shifts again: the synthesized policy is compact, interpretable, and verifiable, which suggests a form of rationality linked to explicit structure rather than to latent neural heuristics [1806.09427].

A plausible conclusion is that “rationality-preserving” can only be understood relative to a declared baseline notion of rationality. Without that baseline, preservation has no technical content.

## 4. Preservation mechanisms in related optimization frameworks

Although the exact phrase “rationality-preserving” is not standardized, several adjacent papers do instantiate preservation-like mechanisms. Pause-token training preserves the option to spend additional computation before answering, and reinforcement learning is used to adapt that behavior under sparse supervision [2404.19733]. Functional-token tuning similarly preserves an explicit control layer over reasoning depth by letting the model choose among discrete reasoning modes [2501.14286].

Programmatic reinforcement learning preserves symbolic policy structure more directly. The learned controller is expressed in a high-level interpreted domain-specific language, and weak optimization over program syntax can improve the policy beyond the demonstrations while maintaining compactness, interpretability, and verifiability [1806.09427]. This is preservation in a syntactic and semantic sense rather than in a purely behavioral one.

Probabilistic fairness-constrained bandit planning provides a different preservation template: the optimization preserves a lower-bounded action probability for every arm at every timestep while still maximizing total expected reward under the relaxed stationary formulation [2106.07677]. That is not rationality in the pragmatics sense, but it is a clear example of constraint-preserving policy optimization.

Taken together, these mechanisms suggest that any putative RPO framework would need to specify what is being preserved: reasoning depth control, pragmatic consistency, symbolic structure, fairness constraints, or robustness under uncertainty.

## 5. Distinctions from standard policy optimization

A common misconception would be to equate “rationality-preserving” with ordinary reward maximization. The surveyed literature does not support that equivalence. Standard reward optimization can improve task return while degrading interpretability, violating fairness constraints, or relying on brittle shortcuts. The nearby papers are notable precisely because they add structure beyond unconstrained optimization.

Another misconception is to treat chain-of-thought expansion itself as preserved rationality. The recent reasoning papers instead optimize computation allocation, token type, or verifier-guided performance; they do not establish that longer visible reasoning is identical to rational consistency [2404.19733]. Related work also shows that externally verbalized reasoning can diverge from the thought process that determines the answer, which further cautions against identifying rationality with the presence of explicit rationale tokens [2503.00211].

It is also inaccurate to assume that preservation always implies exact guarantees. Some methods provide formal guarantees—such as exact budget satisfaction with marginal action probabilities in dependent rounding, or verifiable policy representations in programmatic RL—whereas others provide adaptive or heuristic control over reasoning behavior without a comparable semantic guarantee [2106.07677].

## 6. Prospective formulations and open problems

Because no canonical RPO formalism is established in the surveyed literature, any rigorous use of the term would need to make three ingredients explicit. First, it would need a reference notion of rationality: for example stable preferences, pragmatic coherence, robust decision quality, or symbolic-policy semantics. Second, it would need a preservation operator: a hard constraint, a regularizer, a verification condition, or a structured policy class. Third, it would need an optimization target: expected reward, log loss, verifier score, or another task metric.

The literature already supplies templates for each component, but in separate lines of work. Rational supervision contributes a preference-stability and pragmatics template [2203.13474]. RL-for-reasoning contributes optimization pipelines over generated tasks and automatically verifiable outcomes, with a rational speech act component inside the training loop [2505.24874]. Programmatic RL contributes interpretable and verifiable policy representations [1806.09427]. Robust control contributes uncertainty-aware policy design [2207.08418].

This suggests that “Rationality-preserving Policy Optimization” is, at present, best regarded as a prospective synthesis rather than an established named method. The central unresolved question is not how to optimize policies in general, but how to formalize the rational property to be preserved strongly enough that optimization does not erase it.

Source: https://www.emergentmind.com/topics/rationality-preserving-policy-optimization-rpo