Meta-PO: Meta-Level Optimization in Diverse Domains

• Meta-PO is a naming pattern for meta-level control applied to varied optimization tasks, with its meaning tailored to specific domains.
• It encompasses approaches ranging from uncertainty-weighted latent fusion in partial observability to explicit meta-plan and meta-policy optimization in LLM agents and reinforcement learning.
• Practical frameworks like POEM, the H-learner, and equilibrium-preserving reallocations illustrate measurable performance improvements across tasks such as few-shot learning and fair division.

Meta-PO is an overloaded term in recent arXiv literature rather than a single standardized framework. In different research areas it denotes meta-learning under partial observability, potential-outcome-centered causal meta-learners, meta-policy or meta-plan optimization in reinforcement learning and LLM agents, and a market-equilibrium route to preserving Pareto optimality in fair division (Jelley et al., 2023, Liang et al., 16 Jun 2025, Hiraoka et al., 2020, Xiong et al., 4 Mar 2025, Kim et al., 28 Apr 2025, Lin et al., 8 Jan 2025). The technical object named by the term therefore changes with domain: sometimes it is a probabilistic latent representation, sometimes a planner that emits reusable strategic guidance, sometimes a policy-level optimizer, and sometimes an equilibrium invariant used to maintain efficiency during fairness-improving reallocations.

1. Terminological scope and domain-specific meanings

The same label appears in several distinct literatures, each attaching it to a different optimization target and formal apparatus.

Domain	Meaning	Representative source
Few-shot learning	Meta-learning under partial observability via POEM	(Jelley et al., 2023)
LLM agents	Meta Plan Optimization with feedback-optimized meta plans	(Xiong et al., 4 Mar 2025)
Causal inference	Potential-outcome-centered indirect meta-learners	(Liang et al., 16 Jun 2025)
Meta-RL	Meta-Model-Based Meta-Policy Optimization	(Hiraoka et al., 2020)
Multi-agent planning	Meta-policy-guided MCTS in POSGs	(Schwartz et al., 2023)
LLM alignment	Meta Policy Optimization with evolving reward prompts	(Kim et al., 28 Apr 2025)
Fair division	Equilibrium-preserving route to Pareto optimality	(Lin et al., 8 Jan 2025)

These usages are not interchangeable. In the few-shot setting, the central problem is representation learning from partial views. In causal inference, the term refers to a family of estimators built around separate potential-outcome models. In agentic planning, the emphasis shifts to explicit guidance, search priors, or reward-specification adaptation. In chore allocation, the label is attached to a meta-level principle: preserve equilibrium and therefore Pareto optimality while modifying the allocation to improve fairness.

This suggests that “Meta-PO” is best understood as a naming pattern for meta-level control around a base optimization problem, not as a unified formal theory.

2. Meta-learning under partial observability: POEM

In "Contrastive Meta-Learning for Partially Observable Few-Shot Learning" (Jelley et al., 2023), Meta-PO denotes meta-learning under partial observability. The setting consists of episodic few-shot tasks in which each support item is observed through multiple partial views, and useful features may appear in only some views. The paper’s central claim is that standard contrastive and embedding-based meta-learners break down when views are partially disjoint, because distance-based objectives penalize informative but view-specific components and thereby encourage their suppression.

The proposed remedy is Partial Observation Experts Modelling (POEM). Each view $x_v^m$ of support item $m$ maps to a conditional factor $\phi(z \mid x_v^m)$ over a shared latent representation $z$, with component-wise uncertainty encoded through learned precisions. The support posterior is defined by a product-of-experts: $$p(\mathbf{z}^{m}|\mathbf{X}^{m})=\frac{p(\mathbf{z}^{m})\prod_{v=1}^{V^{m}}\phi(\mathbf{z}^{m}|\mathbf{x}_v^{m})}{\int d\mathbf{z}^{\prime}\ p(\mathbf{z}^{\prime})\prod_{v=1}^{V^{m}}\phi(\mathbf{z}^{\prime}|\mathbf{x}_v^{m})}.$$ Query matching is performed through the marginal predictive distribution

$$p(\mathbf{x}^*|\mathbf{X}^{m})=p(\mathbf{x}^*) \frac{\lambda(\mathbf{x}^*, \mathbf{X}^{m})}{\lambda^{\prime}(\mathbf{X}^{m})},$$

and training uses a contrastive log-softmax objective over support items: $$\mathcal{L}=-\sum_t \left[\log \frac{\lambda(\mathbf{x}^*, \mathbf{X}^{m^*})}{\lambda^{\prime}(\mathbf{X}^{m^*})} - \log \sum_{m}\frac{\lambda(\mathbf{x}^*, \mathbf{X}^m)}{\lambda^{\prime}(\mathbf{X}^m)}\right].$$

The implementation is deliberately close to standard few-shot pipelines. Each view is encoded by a ResNet-18 pre-trained on ImageNet at $126\times126$, then passed through two 3-layer MLP heads that output a $512$-dimensional mean $\mu_v$ and precision $m$0. Viewpoint coordinates are concatenated to the features before these heads. Under diagonal Gaussian assumptions, aggregation is analytic and component-wise: $m$1 Missing views require no special masking; the product simply ranges over the available factors, so uninformed components are handled by high variance or low precision.

The paper evaluates POEM on an adapted Meta-Dataset benchmark under strong SimCLR-style augmentations with stronger random cropping such that the entire support set for a sample contains at most $m$2 of the original image content, with ways varying between $m$3 and $m$4. On the Partially Observed Meta-Dataset, POEM is best on $m$5 subsets: Aircraft $m$6, Birds $m$7, Flowers $m$8, Fungi $m$9, and Omniglot $\phi(z \mid x_v^m)$0; Textures is the exception, where Finetune achieves $\phi(z \mid x_v^m)$1 and POEM $\phi(z \mid x_v^m)$2. On standard Meta-Dataset, the learned precisions’ variance becomes negligible, and with all precisions fixed to $\phi(z \mid x_v^m)$3, POEM reduces to Prototypical Networks up to scaling factors. The same formalism is also used for environment representation learning in MiniGrid $\phi(z \mid x_v^m)$4 Simple Crossing, where POEM achieves $\phi(z \mid x_v^m)$5 environment recognition versus $\phi(z \mid x_v^m)$6 for ProtoNet and $\phi(z \mid x_v^m)$7 for a GRU-based recurrent baseline.

The significance of this line of work lies in its replacement of global embedding consistency by uncertainty-weighted latent fusion. Under partial observability, consistency is no longer defined by small Euclidean distance between paired views, but by overlap after marginalizing uncertainty.

3. Meta plans as explicit guidance for LLM agents

In "MPO: Boosting LLM Agents with Meta Plan Optimization" (Xiong et al., 4 Mar 2025), the core object is not a latent representation but a meta plan: high-level, abstract, natural-language guidance that is decoupled from environment-specific entities and trajectories. The target setting is interactive planning, where an agent must produce interleaved thoughts and actions while responding to environment observations in ALFWorld and ScienceWorld.

The formalization separates the base agent $\phi(z \mid x_v^m)$8 from a meta planner $\phi(z \mid x_v^m)$9. Without explicit guidance, an agent trajectory $z$0 has likelihood

$z$1

With a meta plan $z$2 inserted into the prompt,

$z$3

Meta plan quality is estimated by Monte Carlo rollouts: $z$4

The optimization pipeline has three stages. First, seed meta plans are collected by summarizing golden trajectories with GPT-4o into standardized "Step 1, Step 2, …" plans, followed by supervised fine-tuning of the meta planner. Second, the SFT-initialized planner samples $z$5 candidate meta plans per task at temperature $z$6, and each plan is scored using $z$7 agent rollouts. Third, contrastive plan pairs $z$8 are used in Direct Preference Optimization against the SFT reference model. The base agent is frozen throughout; only the meta planner is optimized. At inference time, a single deterministic meta plan is generated at temperature $z$9 and inserted into the task instruction, which the paper reports as the best insertion point. Insertion into the thought process degrades performance, and insertion into observations is second-best but still inferior.

The empirical results emphasize plug-and-play transfer across agents. For GPT-4o, average reward moves from $$p(\mathbf{z}^{m}|\mathbf{X}^{m})=\frac{p(\mathbf{z}^{m})\prod_{v=1}^{V^{m}}\phi(\mathbf{z}^{m}|\mathbf{x}_v^{m})}{\int d\mathbf{z}^{\prime}\ p(\mathbf{z}^{\prime})\prod_{v=1}^{V^{m}}\phi(\mathbf{z}^{\prime}|\mathbf{x}_v^{m})}.$$0 to $$p(\mathbf{z}^{m}|\mathbf{X}^{m})=\frac{p(\mathbf{z}^{m})\prod_{v=1}^{V^{m}}\phi(\mathbf{z}^{m}|\mathbf{x}_v^{m})}{\int d\mathbf{z}^{\prime}\ p(\mathbf{z}^{\prime})\prod_{v=1}^{V^{m}}\phi(\mathbf{z}^{\prime}|\mathbf{x}_v^{m})}.$$1 on seen/unseen ScienceWorld and from $$p(\mathbf{z}^{m}|\mathbf{X}^{m})=\frac{p(\mathbf{z}^{m})\prod_{v=1}^{V^{m}}\phi(\mathbf{z}^{m}|\mathbf{x}_v^{m})}{\int d\mathbf{z}^{\prime}\ p(\mathbf{z}^{\prime})\prod_{v=1}^{V^{m}}\phi(\mathbf{z}^{\prime}|\mathbf{x}_v^{m})}.$$2 to $$p(\mathbf{z}^{m}|\mathbf{X}^{m})=\frac{p(\mathbf{z}^{m})\prod_{v=1}^{V^{m}}\phi(\mathbf{z}^{m}|\mathbf{x}_v^{m})}{\int d\mathbf{z}^{\prime}\ p(\mathbf{z}^{\prime})\prod_{v=1}^{V^{m}}\phi(\mathbf{z}^{\prime}|\mathbf{x}_v^{m})}.$$3 on seen/unseen ALFWorld. For Llama-3.1-8B-Instruct, the average moves from $$p(\mathbf{z}^{m}|\mathbf{X}^{m})=\frac{p(\mathbf{z}^{m})\prod_{v=1}^{V^{m}}\phi(\mathbf{z}^{m}|\mathbf{x}_v^{m})}{\int d\mathbf{z}^{\prime}\ p(\mathbf{z}^{\prime})\prod_{v=1}^{V^{m}}\phi(\mathbf{z}^{\prime}|\mathbf{x}_v^{m})}.$$4 and $$p(\mathbf{z}^{m}|\mathbf{X}^{m})=\frac{p(\mathbf{z}^{m})\prod_{v=1}^{V^{m}}\phi(\mathbf{z}^{m}|\mathbf{x}_v^{m})}{\int d\mathbf{z}^{\prime}\ p(\mathbf{z}^{\prime})\prod_{v=1}^{V^{m}}\phi(\mathbf{z}^{\prime}|\mathbf{x}_v^{m})}.$$5 to $$p(\mathbf{z}^{m}|\mathbf{X}^{m})=\frac{p(\mathbf{z}^{m})\prod_{v=1}^{V^{m}}\phi(\mathbf{z}^{m}|\mathbf{x}_v^{m})}{\int d\mathbf{z}^{\prime}\ p(\mathbf{z}^{\prime})\prod_{v=1}^{V^{m}}\phi(\mathbf{z}^{\prime}|\mathbf{x}_v^{m})}.$$6 and $$p(\mathbf{z}^{m}|\mathbf{X}^{m})=\frac{p(\mathbf{z}^{m})\prod_{v=1}^{V^{m}}\phi(\mathbf{z}^{m}|\mathbf{x}_v^{m})}{\int d\mathbf{z}^{\prime}\ p(\mathbf{z}^{\prime})\prod_{v=1}^{V^{m}}\phi(\mathbf{z}^{\prime}|\mathbf{x}_v^{m})}.$$7, reported as up to $$p(\mathbf{z}^{m}|\mathbf{X}^{m})=\frac{p(\mathbf{z}^{m})\prod_{v=1}^{V^{m}}\phi(\mathbf{z}^{m}|\mathbf{x}_v^{m})}{\int d\mathbf{z}^{\prime}\ p(\mathbf{z}^{\prime})\prod_{v=1}^{V^{m}}\phi(\mathbf{z}^{\prime}|\mathbf{x}_v^{m})}.$$8 improvement over base. The framework also improves trained agents: Llama-3.1-8B-Instruct + ETO rises from $$p(\mathbf{z}^{m}|\mathbf{X}^{m})=\frac{p(\mathbf{z}^{m})\prod_{v=1}^{V^{m}}\phi(\mathbf{z}^{m}|\mathbf{x}_v^{m})}{\int d\mathbf{z}^{\prime}\ p(\mathbf{z}^{\prime})\prod_{v=1}^{V^{m}}\phi(\mathbf{z}^{\prime}|\mathbf{x}_v^{m})}.$$9 and $$p(\mathbf{x}^*|\mathbf{X}^{m})=p(\mathbf{x}^*) \frac{\lambda(\mathbf{x}^*, \mathbf{X}^{m})}{\lambda^{\prime}(\mathbf{X}^{m})},$$0 to $$p(\mathbf{x}^*|\mathbf{X}^{m})=p(\mathbf{x}^*) \frac{\lambda(\mathbf{x}^*, \mathbf{X}^{m})}{\lambda^{\prime}(\mathbf{X}^{m})},$$1 and $$p(\mathbf{x}^*|\mathbf{X}^{m})=p(\mathbf{x}^*) \frac{\lambda(\mathbf{x}^*, \mathbf{X}^{m})}{\lambda^{\prime}(\mathbf{X}^{m})},$$2. A case study in ALFWorld contrasts an SFT-only plan that repeatedly directs the agent to a wrong "sidetable" with an MPO-optimized plan that instead instructs abstract search for pillow locations, which reduces loops and invalid subplans.

The framework’s role in the broader Meta-PO landscape is to externalize planning structure. Rather than retraining the agent or relying on fragile implicit reasoning, it learns reusable strategic guidance and optimizes that guidance from environment feedback.

4. Potential-outcome-centered Meta-PO in causal inference

In "Hybrid Meta-learners for Estimating Heterogeneous Treatment Effects" (Liang et al., 16 Jun 2025), Meta-PO denotes the family of indirect meta-learners that estimate conditional average treatment effects by first fitting potential outcome functions. The data are observational: $$p(\mathbf{x}^*|\mathbf{X}^{m})=p(\mathbf{x}^*) \frac{\lambda(\mathbf{x}^*, \mathbf{X}^{m})}{\lambda^{\prime}(\mathbf{X}^{m})},$$3 with binary treatment $$p(\mathbf{x}^*|\mathbf{X}^{m})=p(\mathbf{x}^*) \frac{\lambda(\mathbf{x}^*, \mathbf{X}^{m})}{\lambda^{\prime}(\mathbf{X}^{m})},$$4, potential outcomes $$p(\mathbf{x}^*|\mathbf{X}^{m})=p(\mathbf{x}^*) \frac{\lambda(\mathbf{x}^*, \mathbf{X}^{m})}{\lambda^{\prime}(\mathbf{X}^{m})},$$5, potential outcome functions $$p(\mathbf{x}^*|\mathbf{X}^{m})=p(\mathbf{x}^*) \frac{\lambda(\mathbf{x}^*, \mathbf{X}^{m})}{\lambda^{\prime}(\mathbf{X}^{m})},$$6, propensity score $$p(\mathbf{x}^*|\mathbf{X}^{m})=p(\mathbf{x}^*) \frac{\lambda(\mathbf{x}^*, \mathbf{X}^{m})}{\lambda^{\prime}(\mathbf{X}^{m})},$$7, and CATE $$p(\mathbf{x}^*|\mathbf{X}^{m})=p(\mathbf{x}^*) \frac{\lambda(\mathbf{x}^*, \mathbf{X}^{m})}{\lambda^{\prime}(\mathbf{X}^{m})},$$8. Identification relies on Consistency, Unconfoundedness, and Positivity.

Indirect learners regularize the potential outcomes and then subtract: $$p(\mathbf{x}^*|\mathbf{X}^{m})=p(\mathbf{x}^*) \frac{\lambda(\mathbf{x}^*, \mathbf{X}^{m})}{\lambda^{\prime}(\mathbf{X}^{m})},$$9 with analogous constructions for the S-learner and TARNet. Their generic loss is

$$\mathcal{L}=-\sum_t \left[\log \frac{\lambda(\mathbf{x}^*, \mathbf{X}^{m^*})}{\lambda^{\prime}(\mathbf{X}^{m^*})} - \log \sum_{m}\frac{\lambda(\mathbf{x}^*, \mathbf{X}^m)}{\lambda^{\prime}(\mathbf{X}^m)}\right].$$0

Direct learners instead construct pseudo-outcomes $$\mathcal{L}=-\sum_t \left[\log \frac{\lambda(\mathbf{x}^*, \mathbf{X}^{m^*})}{\lambda^{\prime}(\mathbf{X}^{m^*})} - \log \sum_{m}\frac{\lambda(\mathbf{x}^*, \mathbf{X}^m)}{\lambda^{\prime}(\mathbf{X}^m)}\right].$$1 satisfying $$\mathcal{L}=-\sum_t \left[\log \frac{\lambda(\mathbf{x}^*, \mathbf{X}^{m^*})}{\lambda^{\prime}(\mathbf{X}^{m^*})} - \log \sum_{m}\frac{\lambda(\mathbf{x}^*, \mathbf{X}^m)}{\lambda^{\prime}(\mathbf{X}^m)}\right].$$2, then fit $$\mathcal{L}=-\sum_t \left[\log \frac{\lambda(\mathbf{x}^*, \mathbf{X}^{m^*})}{\lambda^{\prime}(\mathbf{X}^{m^*})} - \log \sum_{m}\frac{\lambda(\mathbf{x}^*, \mathbf{X}^m)}{\lambda^{\prime}(\mathbf{X}^m)}\right].$$3 directly. The paper lists IPW, X-learner, and DR-learner pseudo-outcomes, including

$$\mathcal{L}=-\sum_t \left[\log \frac{\lambda(\mathbf{x}^*, \mathbf{X}^{m^*})}{\lambda^{\prime}(\mathbf{X}^{m^*})} - \log \sum_{m}\frac{\lambda(\mathbf{x}^*, \mathbf{X}^m)}{\lambda^{\prime}(\mathbf{X}^m)}\right].$$4

for the X-learner and

$$\mathcal{L}=-\sum_t \left[\log \frac{\lambda(\mathbf{x}^*, \mathbf{X}^{m^*})}{\lambda^{\prime}(\mathbf{X}^{m^*})} - \log \sum_{m}\frac{\lambda(\mathbf{x}^*, \mathbf{X}^m)}{\lambda^{\prime}(\mathbf{X}^m)}\right].$$5

for the DR-learner.

The paper’s central contribution is the H-learner, which interpolates between indirect and direct regularization. It learns two intermediate functions $$\mathcal{L}=-\sum_t \left[\log \frac{\lambda(\mathbf{x}^*, \mathbf{X}^{m^*})}{\lambda^{\prime}(\mathbf{X}^{m^*})} - \log \sum_{m}\frac{\lambda(\mathbf{x}^*, \mathbf{X}^m)}{\lambda^{\prime}(\mathbf{X}^m)}\right].$$6 and estimates

$$\mathcal{L}=-\sum_t \left[\log \frac{\lambda(\mathbf{x}^*, \mathbf{X}^{m^*})}{\lambda^{\prime}(\mathbf{X}^{m^*})} - \log \sum_{m}\frac{\lambda(\mathbf{x}^*, \mathbf{X}^m)}{\lambda^{\prime}(\mathbf{X}^m)}\right].$$7

by minimizing

$$\mathcal{L}=-\sum_t \left[\log \frac{\lambda(\mathbf{x}^*, \mathbf{X}^{m^*})}{\lambda^{\prime}(\mathbf{X}^{m^*})} - \log \sum_{m}\frac{\lambda(\mathbf{x}^*, \mathbf{X}^m)}{\lambda^{\prime}(\mathbf{X}^m)}\right].$$8

When $$\mathcal{L}=-\sum_t \left[\log \frac{\lambda(\mathbf{x}^*, \mathbf{X}^{m^*})}{\lambda^{\prime}(\mathbf{X}^{m^*})} - \log \sum_{m}\frac{\lambda(\mathbf{x}^*, \mathbf{X}^m)}{\lambda^{\prime}(\mathbf{X}^m)}\right].$$9, the method reduces to indirect Meta-PO training; when $126\times126$0, it reduces to direct $126\times126$1-regularization. The motivation is a bias-variance tradeoff. Independent regularization of $126\times126$2 and $126\times126$3 can generate “regularization-induced confounding,” producing spurious heterogeneity in $126\times126$4 even when the true effect is simple, while direct pseudo-outcomes can have high variance under imbalance or weak overlap.

Empirically, the paper reports that the optimal $126\times126$5 lies strictly between $126\times126$6 and $126\times126$7 across semi-synthetic setups that vary potential-outcome similarity, treated proportion, and confounding strength. On IHDP, test $126\times126$8 is $126\times126$9 for T-learner, $512$0 for TARNet, $512$1 for X-learner, $512$2 for DR-learner, and $512$3 for H-learner (X), which is the best reported result. On ACIC 2016, the corresponding numbers are $512$4, $512$5, $512$6, $512$7, and $512$8, again with H-learner (X) best. Structural regularizers that shrink $512$9 toward $\mu_v$0, such as TARNet-WR, OffsetNet, and FlexTENet, underperform TARNet and H-learner.

Within this usage, Meta-PO is therefore a specific inductive bias: regularize through the potential outcomes first, then recover treatment effects by differencing. The H-learner is a response to the fact that this bias is dataset-dependent rather than uniformly optimal.

5. Meta-policy optimization in reinforcement learning and partially observable planning

In reinforcement learning and multi-agent planning, the relevant object is a meta-policy rather than a potential-outcome model or explicit natural-language plan. Two representative instances are Meta-Model-Based Meta-Policy Optimization (M3PO) (Hiraoka et al., 2020) and Partially Observable Type-based Meta Monte-Carlo Planning (POTMMCP) (Schwartz et al., 2023).

M3PO treats meta-RL as a POMDP whose hidden state includes task identity and observation. The agent acts with a history-conditioned meta-policy $\mu_v$1, where $\mu_v$2 is a truncated interaction history of length $\mu_v$3. A learned meta-model

$\mu_v$4

is represented as an ensemble of $\mu_v$5 diagonal Gaussians and trained by maximum likelihood on real transitions. Synthetic data are generated by short-horizon branched rollouts from real histories, yielding $\mu_v$6, and the policy is updated off-policy with a soft-value KL objective. The theoretical analysis extends MBPO-style guarantees to meta-RL by reducing the POMDP to an MDP over histories and bounding the discrepancy between real and model returns in terms of one-step model error $\mu_v$7 and maximal policy divergence $\mu_v$8. The paper’s corollary states that $\mu_v$9 branched rollouts reduce sensitivity to model error compared with full-model rollouts, and its ablations report that $m$00 is best empirically. On six MuJoCo meta-RL benchmarks—Halfcheetah-fwd-bwd, Halfcheetah-pier, Ant-fwd-bwd, Ant-crippled-leg, Walker2D-randomparams, and Humanoid-direc—M3PO outperforms PEARL and L2A in short-term sample efficiency, while the hybrid-buffer variant M3PO-h anneals the model-generated fraction $m$01 to reduce model bias and improve long-term performance.

POTMMCP addresses a different problem: online planning against unobserved other-agent types in partially observable stochastic games. The planner maintains a particle belief over tuples

$m$02

combining environment state, the other agents’ joint policy, and their joint history. Its meta-policy $m$03 is computed from an empirical game over policy types using a softmax over payoff entries: $m$04 This meta-policy supplies both a prior for PUCT action selection and a value estimate for leaf-node evaluation. The search tree stores particle approximations to the belief at each history node, and action selection uses

$m$05

with a prior term that mixes the meta-policy prior and uniform noise through $m$06, ensuring positive prior probability for all actions. The paper proves convergence in probability to an $m$07-optimal value function under mild conditions and reports empirical scalability to problems with up to $m$08 states and $m$09 observations. With a $m$10 s/step budget, the observed maximum search depth is approximately $m$11 for POTMMCP versus approximately $m$12 for the baseline.

These two methods illustrate a shared but domain-specific use of meta-policy structure. In M3PO, the meta-policy is the adaptive control law over history, trained with model-based rollouts. In POTMMCP, the meta-policy is a search prior over candidate planner policies conditioned on beliefs over opponent types.

6. Meta Policy Optimization for evolving reward models in LLM alignment

In "Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Models" (Kim et al., 28 Apr 2025), MPO is an alignment framework in which a meta-reward model dynamically rewrites the reward model’s prompt during reinforcement learning. The stated motivation is that fixed reward prompts encourage reward hacking and require brittle, labor-intensive prompt engineering.

The framework contains three components: a policy $m$13, a reward model $m$14 parameterized by a rubric prompt $m$15, and a meta-reward model $m$16 that produces a refined rubric prompt $m$17 from the current rubric and recent training context. The KL-regularized objective is

$m$18

Inner-loop policy updates use PPO, while the outer loop periodically updates the rubric: $m$19 The paper formalizes this as an evolving partition of state space, where observation categories become finer over time and observation-based rewards $m$20 track a more discriminating rubric.

Operationally, every $m$21 PPO batches the system runs three text-level meta steps: Meta-Analysis, Meta-Refinement, and Meta-Merging. These diagnose reward hacking under the current rubric, generate stricter instance-level rubrics, and merge them into a new prompt. Neither the reward model nor the meta-reward model is trained; both operate in inference mode. The policy is Qwen2-1.5B-Instruct, and the reward/meta-reward models are Qwen2.5-32B-Instruct-AWQ or Qwen2.5-72B-Instruct-AWQ served via SGLang. Reported schedules are $m$22 for essay writing, $m$23 for BillSum summarization, $m$24 for Scruples-Anecdotes, and $m$25 for MATH, with $m$26 rubric lanes for math.

The main empirical evidence concerns the suppression of reward hacking and the competitiveness of automatically evolved rubrics. In essay writing, Elo from $m$27 GPT-4o pairwise comparisons reaches $m$28 for MPO $m$29, compared with $m$30 for PPO with the fixed "iter0" $m$31B prompt and $m$32 for the base LLM. A $m$33B static-prompt PPO run fails through reward hacking by producing title-only essays that still score $m$34–$m$35, whereas MPO eliminates this failure. On BillSum, MPO $m$36 achieves ROUGE-1 $m$37, ROUGE-2 $m$38, ROUGE-L $m$39, ROUGE-Lsum $m$40, and Elo $m$41, improving over PPO $m$42 and the base model. On Scruples-Anecdotes, test accuracy rises from $m$43 for the base model and $m$44 for PPO $m$45 to $m$46 for MPO $m$47. On MATH, exact answer match improves from $m$48 to $m$49 under fixed-prompt PPO and $m$50 under MPO.

This usage of MPO differs from the meta-plan framework above. There, the meta-level object is strategic guidance injected into the agent prompt; here, it is the reward specification itself, which evolves in response to the policy’s current failure modes.

7. Meta-PO as Pareto-optimality preservation in fair division of chores

In "Approximately EFX and PO Allocations for Bivalued Chores" (Lin et al., 8 Jan 2025), Meta-PO refers to a design principle for chore allocation: preserve market equilibrium, and therefore Pareto optimality, while performing local reallocations that strengthen fairness. The setting has agents $m$51, indivisible items $m$52, additive costs, and bi-valued costs $m$53 with $m$54.

The fairness notions are cost-based. EF1 requires that for every pair of agents $m$55, either $m$56 or there exists $m$57 such that

$m$58

EFX requires the same inequality for all $m$59 with $m$60, and $m$61-EFX relaxes it to

$m$62

Pareto optimality means there is no alternative allocation that weakly lowers all agents’ costs and strictly lowers at least one.

The market formalism uses payments $m$63, pain-per-buck ratios $m$64, and minimum-pain-per-buck sets $m$65. If an allocation $m$66 satisfies $m$67 for all agents, then it is an equilibrium and therefore Pareto optimal by the First Welfare Theorem. Because cost and payment coincide up to the agent-specific factor on MPB items, payment-based EF1 or EFX implies the corresponding cost-based notion.

The algorithm begins from an integral pEF1 equilibrium $m$68 with payments in $m$69, computable in $m$70 time. The main structural split is between $m$71, where $m$72, and $m$73, where $m$74. If the allocation is not already $m$75-pEFX, there exists a threshold $m$76 such that agents in $m$77 have total payments in $m$78, agents in $m$79 have total payments in $m$80, and only cross-group envy remains. The local move then exchanges the envying $m$81 agent’s unique high-payment chore for the $m$82 agent’s low-payment bundle. MPB feasibility is preserved, so equilibrium and Pareto optimality are preserved at every step.

The paper proves two results. First, for any $m$83-instance, there is a polynomial-time algorithm that returns an allocation that is $m$84-EFX with

$m$85

and Pareto optimal. Second, for the special case $m$86, there is a polynomial-time algorithm that computes an exact EFX and PO allocation. The reallocation phase requires at most $m$87 iterations and $m$88 time after the initial equilibrium is found. This improves on a prior $m$89-EFX and PO guarantee for bi-valued chores obtained via rounding a fractional earning-restricted equilibrium.

In this literature, Meta-PO is therefore not a learning algorithm at all. It is a meta-level efficiency invariant: fairness can be strengthened through local exchanges as long as all reallocations remain MPB-feasible, because equilibrium preservation automatically preserves Pareto optimality.