Order-Invariant Reinforcement Learning

• Order-invariant RL is a framework that exploits redundant order structures in states, rewards, or policies to focus on essential features and improve learning efficiency.
• Techniques include permutation invariant set encoders, equivariant policy networks, and order-randomized training that mitigate the burden of learning irrelevant orderings.
• Empirical studies in tasks like food scavenging and resource allocation demonstrate significant reductions in training epochs and sample complexity with these methods.

Order-invariant reinforcement learning denotes a heterogeneous body of work in which reinforcement learning exploits the irrelevance of some ordering, relabeling, or ranking. In one major line, the state is treated as a set of exchangeable objects, and the policy or representation is required to be invariant or equivariant under permutations of object order (Mern et al., 2020). In other lines, “order” refers to ordinal reward structure rather than cardinal reward magnitude (Zap et al., 2019), to distributions over topological orderings in causal structure learning (Yang et al., 2022), or to partial orders over policies induced by information dominance (Zhang et al., 5 Aug 2025). The literature is therefore unified less by a single architecture than by a recurring principle: if order is extraneous to control, encoding that fact can reduce redundancy, improve sample efficiency, and alter what is learned.

1. Exchangeability and permutation invariance over object sets

The most direct interpretation of order-invariant reinforcement learning arises when an RL state is naturally a set of objects but is presented to a neural policy as an ordered concatenation. In that setting, two semantically identical states can differ only because identical-type objects were listed in a different order. “Exchangeable Input Representations for Reinforcement Learning” formalizes this problem by defining permutation-invariant functions over states represented as sets of objects,

$f(S)=f(\hat S)\ \forall\ \hat S\in \Pi,$

where $S$ is a state defined by a set of objects and $\Pi$ is the set of all permutations on $S$ (Mern et al., 2020).

That work introduces an attention-based set encoder for object-centric RL. If the full state $S_i$ contains $m$ object vectors $\mathbf s_i^{(j)}\in\mathbb R^n$, these are arranged into $\mathbf A_i\in\mathbb R^{m\times n}$. Two shared subnetworks are then applied independently to every object: an abstraction network $\pi_{\text{abstraction}}$, producing object embeddings $\mathbf Z_i\in\mathbb R^{m\times k}$, and a filter network $S$0, producing scalar weights $S$1. After softmax normalization, the final abstract representation is

$S$2

Because the same networks are applied to every object and the aggregation is a sum, the representation is permutation invariant (Mern et al., 2020).

The paper explicitly connects this construction to the Deep Sets characterization

$S$3

and rewrites its attention mechanism in that form. It also extends the single-set encoder to partial exchangeability by using one invariant attention mechanism per object class, then concatenating the resulting class summaries. This supports multiple object classes and variable numbers of objects without fixed-size padding (Mern et al., 2020).

A central theoretical intuition is that an ordered representation unnecessarily distinguishes $S$4 permutations of the same exchangeable configuration. With $S$5 objects whose values are drawn from $S$6 unique possibilities, the ordered input space is counted as

$S$7

whereas the invariant abstract space is

$S$8

so the ratio is $S$9 (Mern et al., 2020). The 2019 extended abstract had already made the same search-space argument and positioned attention-based permutation invariance as the remedy for naive ordered inputs (Mern et al., 2019).

Empirically, the full paper reports that PPO with the naive ordered representation fails to solve the food scavenger problem within 1,000 training epochs for more than two targets, whereas the invariant attention abstraction solves tasks up to five targets. In the convoy protection task, naive PPO is completely unable to learn over 10,000 training epochs and 20 million sample time steps, while the invariant abstraction learns a successful policy in 3,000 epochs and 6 million time steps (Mern et al., 2020). The earlier extended abstract reported the same qualitative pattern: baseline policies could not achieve optimal performance for tasks with more than two food particles, while the invariant policy learned an optimal policy for all tested cases up to five food particles, and in convoy protection the invariant policy approached optimal behavior after approximately 2,500 epochs whereas the baseline showed no improvement after 10,000 epochs (Mern et al., 2019).

These results established the canonical object-set view of order-invariant RL: when object identity is irrelevant and only object attributes matter, the policy should not have to learn permutation symmetry from data.

2. Permutation-equivariant policies in sequential resource allocation

A second, more structured line studies order invariance in sequential allocation problems over interchangeable entities. “Efficient Reinforcement Learning in Resource Allocation Problems Through Permutation Invariant Multi-task Learning” formulates a family of MDPs

$\Pi$0

where each task $\Pi$1 contains a subset of entities $\Pi$2, the state is

$\Pi$3

and the action is a simplex-valued allocation

$\Pi$4

(Cai et al., 2021).

The paper defines a “Permutation Invariant Policy Network” by

$\Pi$5

As noted in the technical synthesis accompanying the paper, this is mathematically a permutation equivariance condition: reordering the entities in the state reorders the allocation vector in the same way (Cai et al., 2021). In this setting, the symmetry is not only representational. The paper states that if the transition kernels can be factored into independent and identical transition kernels across entities, then the optimal policy is indeed permutation invariant (Cai et al., 2021).

Its main contribution is to turn this symmetry into a multi-task sample-efficiency mechanism. Task similarity is measured by the Bellman-difference operator

$\Pi$6

Under a uniform mismatch bound $\Pi$7, the paper derives a finite-sample improvement from approximately $\Pi$8 for single-task learning to

$\Pi$9

when learning over $S$0 related tasks (Cai et al., 2021). When the global universe has $S$1 entities and each task uses $S$2 entities, the number of subset tasks is

$S$3

which makes the task-family growth combinatorial (Cai et al., 2021).

The practical architecture is a shared-parameter per-entity scorer followed by a softmax over entities. In portfolio optimization, each instrument’s flattened price-history input is passed through a common RNN with 25 hidden units and tanh activation, concatenated with the latest allocation fraction, and then passed through a common dense layer to produce a score. In meta federated learning, the same permutation-invariant policy network architecture is used with an RNN of 10 hidden units (Cai et al., 2021). The training procedure also introduces prioritized task sampling,

$S$4

which acts as a task-level analogue of prioritized experience replay (Cai et al., 2021).

The empirical message is that permutation symmetry is not merely an architectural prior. In the portfolio domain, multi-task learning outperforms single-task learning, 30-task MTL outperforms 5-task MTL, and prioritized MTL outperforms uniform MTL. In meta federated learning, multi-task learning discovers nonuniform client-weighting strategies that improve convergence and final accuracy over both uniform averaging and single-task RL (Cai et al., 2021). This suggests that, in exchangeable-entity domains, order invariance can be leveraged simultaneously at the levels of representation, policy class, and task construction.

3. Order-randomized autoregressive reinforcement learning for combinatorial optimization

A more recent formulation appears in black-box combinatorial optimization, where the “order” in question is the autoregressive generation order of decision variables. “Black-Box Combinatorial Optimization with Order-Invariant Reinforcement Learning” studies the discrete optimization problem

$S$5

and casts candidate-solution construction as an episodic MDP whose terminal reward is the objective value $S$6 (Goudet et al., 2 Oct 2025).

For a permutation $S$7 of variable indices, the generator uses the autoregressive factorization

$S$8

The architecture does not enforce exact permutation invariance analytically. Instead, it trains the model under random generation orders and random training orders. The paper is explicit that the model is not analytically permutation-invariant by construction; it is trained to behave robustly across many orderings and not to rely on a single arbitrary factorization (Goudet et al., 2 Oct 2025).

The policy update is based on a GRPO-style, rank-based advantage computed within the current population: $S$9 with default utility

$S_i$0

This yields a scale-invariant, critic-free advantage signal (Goudet et al., 2 Oct 2025). The trust-region mechanism is a KL-regularized PPO surrogate rather than clipped PPO, because generation and training may use different masks and orders (Goudet et al., 2 Oct 2025).

The order-randomized version introduces a sampled generation order $S_i$1 and an independently sampled training order $S_i$2. Its central surrogate objective averages over both: $S_i$3 (Goudet et al., 2 Oct 2025). The paper interprets random order training as a form of information-preserving dropout: at any conditional prediction, some future variables are masked out, but the full candidate solution still underlies the joint model (Goudet et al., 2 Oct 2025).

The empirical ablation isolates four variants: fixed generation and fixed training order, fixed generation and random training order, random generation with training on the same order, and random generation with random training order. On NK with $S_i$4 and $S_i$5, $S_i$6-RL-EDA is the best variant (Goudet et al., 2 Oct 2025). On the full benchmark, it ranks $S_i$7 on all six QUBO instance types at $S_i$8, ranks $S_i$9 on all four NK ruggedness settings at $m$0, and is markedly weaker only on the hardest categorical NK3 setting with $m$1 (Goudet et al., 2 Oct 2025). The paper therefore treats order randomization as both a diversity mechanism and a regularizer against overfitting to arbitrary generation order.

This formulation broadens the meaning of order invariance. The goal is no longer to make the policy invariant to object permutation in its input, but to prevent the generative search distribution from collapsing onto one arbitrary variable order.

4. Order as rank, topological sequence, or policy partial order

Not all work labeled or adjacent to order-invariant RL concerns permutation symmetry of observations. A distinct strand concerns ordinal structure in the reward signal itself. “Deep Ordinal Reinforcement Learning” replaces scalar rewards by ordered reward tiers

$m$2

represents state-action quality by a distribution vector

$m$3

and compares actions using the measure of statistical superiority

$m$4

rather than expected reward magnitude (Zap et al., 2019). The deep extension uses an array of $m$5 neural networks, one per action, each producing an $m$6-dimensional ordinal distribution (Zap et al., 2019). This is order-based RL in the sense of rank-preserving rewards, not in the sense of permutation-invariant state encoders.

A different use of “order” appears in “Reinforcement Causal Structure Learning on Order Graph.” There, RL is performed over a layered graph whose states are subsets of already ordered variables and whose actions add one new variable, so that an episode constructs a complete topological order. The method defines

$m$7

and learns a posterior-like transition model over orderings by DQN on this order graph (Yang et al., 2022). The paper is explicit that this is not permutation invariance in the representation-theoretic sense. It is reinforcement learning over many possible orderings rather than commitment to one fixed order (Yang et al., 2022).

“Reinforcement Learning in MDPs with Information-Ordered Policies” uses yet another meaning. It defines a partial order $m$8 when data collected under $m$9 can be used to estimate the performance of $\mathbf s_i^{(j)}\in\mathbb R^n$0. In the sample-path version,

$\mathbf s_i^{(j)}\in\mathbb R^n$1

and the resulting epoch-based policy-elimination algorithm attains a regret bound

$\mathbf s_i^{(j)}\in\mathbb R^n$2

where $\mathbf s_i^{(j)}\in\mathbb R^n$3 is the width of the partial order (Zhang et al., 5 Aug 2025). The paper emphasizes that this bound is independent of the state and action space sizes (Zhang et al., 5 Aug 2025). Here again, the subject is not order invariance of inputs, but an order-theoretic structure over policies.

Taken together, these works show that “order” in RL can refer to at least three mathematically distinct objects: reward rankings, topological variable orderings, and partial orders over policies. None is interchangeable with permutation invariance over sets.

5. Adjacent symmetry-based RL that is not order invariance

A large adjacent literature studies invariance in RL without targeting order invariance specifically. “Invariant Transform Experience Replay” is exemplary. It models multi-goal robotic tasks as a multi-goal MDP

$\mathbf s_i^{(j)}\in\mathbb R^n$4

and defines a symmetry as a bijection

$\mathbf s_i^{(j)}\in\mathbb R^n$5

where $\mathbf s_i^{(j)}\in\mathbb R^n$6 is the space of feasible trajectories (Lin et al., 2019). Its two instantiations are Kaleidoscope Experience Replay, which uses geometric reflection symmetry, and Goal-augmented Experience Replay, which uses threshold-induced equivalence classes of goals. In the Fetch tasks, the paper reports 13×, 3×, and 5× speedups in pushing, sliding, and pick-and-place, respectively, and 46/50 successful trials on a real Baxter robot (Lin et al., 2019). The paper is explicit that this is trajectory-space symmetry invariance, not permutation or order invariance of observations or policy architecture (Lin et al., 2019).

“Invariant Policy Optimization” studies domain invariance rather than order invariance. It seeks a representation $\mathbf s_i^{(j)}\in\mathbb R^n$7 and policy $\mathbf s_i^{(j)}\in\mathbb R^n$8 such that the same predictor is simultaneously optimal across all training domains: $\mathbf s_i^{(j)}\in\mathbb R^n$9 The practical algorithm uses one policy per domain, an averaged policy, and best-response PPO updates (Sonar et al., 2020). Its relevance to order-invariant RL is therefore indirect: it is an invariant RL method, but its symmetry axis is inter-domain stability rather than permutation of exchangeable inputs (Sonar et al., 2020).

“Reinforcement Learning Using known Invariances” provides a more general group-theoretic template. It studies episodic MDPs invariant under a known group $\mathbf A_i\in\mathbb R^{m\times n}$0 acting on states and actions, defines invariant kernels

$\mathbf A_i\in\mathbb R^{m\times n}$1

and shows that symmetry-aware kernelized optimistic value iteration improves information-gain and covering-number complexity terms by factors depending on $\mathbf A_i\in\mathbb R^{m\times n}$2 (Cioba et al., 5 Nov 2025). The paper explicitly notes that this framework applies naturally to order-invariant RL by taking $\mathbf A_i\in\mathbb R^{m\times n}$3 to be a permutation group $\mathbf A_i\in\mathbb R^{m\times n}$4 acting on state and action representations (Cioba et al., 5 Nov 2025). This suggests a route from generic symmetry-aware RL theory to exact permutation invariance, although the reported experiments are geometric rather than permutation-based.

Finally, “Task-Induced Representational Invariances Depend on Learning Objective in Deep RL” studies invariance that emerges from the optimization objective itself. In navigation domains, DQN learns representations invariant to MDP homomorphism symmetries, while PPO learns representations invariant to action symmetries (Halvagal et al., 1 Jun 2026). The paper presents this as structural symmetry, not literal permutation invariance over exchangeable sets (Halvagal et al., 1 Jun 2026). The result is important because it shows that invariance in RL is not monolithic: different objectives compress different equivalence classes.

6. Conceptual boundaries and persistent misconceptions

The literature therefore supports a sharp distinction between at least five notions that are often conflated. First, there is permutation invariance over exchangeable observations, where a state is a set and the output should be unchanged by reordering its elements; this is the sense targeted by attention-based set encoders and sequential allocation policies (Mern et al., 2020). Second, there is permutation equivariance, where reordering object inputs should reorder object-indexed actions in the same way; the resource-allocation literature uses this formulation explicitly, even when it is named “Permutation Invariant Policy Network” (Cai et al., 2021). Third, there is ordinal RL, where only the order of reward categories is retained (Zap et al., 2019). Fourth, there are order-structured search problems, such as topological ordering in causal discovery and information orders over policy classes (Yang et al., 2022, Zhang et al., 5 Aug 2025). Fifth, there are broader symmetry-based RL methods that exploit geometry, group actions, or domain invariance without addressing input order directly (Lin et al., 2019, Sonar et al., 2020).

A recurring misconception is that any “invariant RL” paper is an order-invariant RL paper. The available evidence does not support that conflation. Invariant replay augmentation over reflected robot trajectories, invariant policy learning across training domains, and MDP-homomorphism invariance in learned representations are all technically meaningful forms of invariance, but they do not solve the same problem as exchangeable-set encoding (Lin et al., 2019, Sonar et al., 2020, Halvagal et al., 1 Jun 2026).

Another recurrent boundary concerns exact versus induced invariance. The attention-based set encoders and permutation-equivariant resource-allocation policies hard-code symmetry into the architecture (Mern et al., 2020, Cai et al., 2021). By contrast, the order-randomized autoregressive EDA is not exactly invariant by construction; it encourages robustness across orderings by randomizing generation order and training order inside a KL-regularized RL objective (Goudet et al., 2 Oct 2025). This suggests that the field contains both algebraic symmetry enforcement and stochastic symmetry induction.

The outstanding technical issues are correspondingly varied. The exchangeable-input representation of Mern, Sadigh, and Kochenderfer does not model pairwise or higher-order object interactions in its attention score, since object importance is determined without considering other objects present (Mern et al., 2020). The resource-allocation framework assumes exact or near symmetry across tasks and incurs an $\mathbf A_i\in\mathbb R^{m\times n}$5 penalty when that assumption is violated (Cai et al., 2021). The kernel-based group-invariance theory is formulated for exact known symmetries and leaves partial or approximate symmetries to future work (Cioba et al., 5 Nov 2025). The black-box combinatorial optimizer encourages rather than guarantees order invariance and remains weaker on the hardest NK3 setting with $\mathbf A_i\in\mathbb R^{m\times n}$6 (Goudet et al., 2 Oct 2025).

This suggests that order-invariant reinforcement learning is best understood not as a single method class but as a cluster of strategies for quotienting out irrelevant order structure. In the strictest sense, it refers to RL on exchangeable sets with permutation-invariant or permutation-equivariant policies. In a broader sense, it includes ordinal returns, search over order spaces, and policy classes ordered by counterfactual identifiability. The technical commonality is the same: exploiting an equivalence relation that would otherwise force the learner to spend samples distinguishing states, actions, or trajectories that should not matter for control.