---
title: Order-Invariant Reinforcement Learning
url: https://www.emergentmind.com/topics/order-invariant-reinforcement-learning
type: topic
---

# Order-Invariant Reinforcement Learning

Order-invariant reinforcement learning denotes a heterogeneous body of work in which reinforcement learning exploits the irrelevance of some ordering, relabeling, or ranking. In one major line, the state is treated as a set of exchangeable objects, and the policy or representation is required to be invariant or equivariant under permutations of object order [2003.09022]. In other lines, “order” refers to ordinal reward structure rather than cardinal reward magnitude [1905.02005], to distributions over topological orderings in causal structure learning [2211.12151], or to partial orders over policies induced by information dominance [2508.03904]. The literature is therefore unified less by a single architecture than by a recurring principle: if order is extraneous to control, encoding that fact can reduce redundancy, improve sample efficiency, and alter what is learned.

## 1. Exchangeability and permutation invariance over object sets

The most direct interpretation of order-invariant reinforcement learning arises when an RL state is naturally a set of objects but is presented to a neural policy as an ordered concatenation. In that setting, two semantically identical states can differ only because identical-type objects were listed in a different order. “Exchangeable Input Representations for Reinforcement Learning” formalizes this problem by defining permutation-invariant functions over states represented as sets of objects,
\[
f(S)=f(\hat S)\ \forall\ \hat S\in \Pi,
\]
where \(S\) is a state defined by a set of objects and \(\Pi\) is the set of all permutations on \(S\) [2003.09022].

That work introduces an attention-based set encoder for object-centric RL. If the full state \(S_i\) contains \(m\) object vectors \(\mathbf s_i^{(j)}\in\mathbb R^n\), these are arranged into \(\mathbf A_i\in\mathbb R^{m\times n}\). Two shared subnetworks are then applied independently to every object: an abstraction network \(\pi_{\text{abstraction}}\), producing object embeddings \(\mathbf Z_i\in\mathbb R^{m\times k}\), and a filter network \(\pi_{\text{filter}}\), producing scalar weights \(\mathbf y_i\in\mathbb R^{m\times 1}\). After softmax normalization, the final abstract representation is
\[
\mathbf s_i^*=\sum_{j=1}^m z_i^{(j)}w_i^{(j)}.
\]
Because the same networks are applied to every object and the aggregation is a sum, the representation is permutation invariant [2003.09022].

The paper explicitly connects this construction to the Deep Sets characterization
\[
f(X)=\rho\Big(\sum_{x\in\mathcal X}\phi(x)\Big),
\]
and rewrites its attention mechanism in that form. It also extends the single-set encoder to partial exchangeability by using one invariant attention mechanism per object class, then concatenating the resulting class summaries. This supports multiple object classes and variable numbers of objects without fixed-size padding [2003.09022].

A central theoretical intuition is that an ordered representation unnecessarily distinguishes \(m!\) permutations of the same exchangeable configuration. With \(m\) objects whose values are drawn from \(n\) unique possibilities, the ordered input space is counted as
\[
|\mathcal S|=\frac{n!}{(n-m)!},
\]
whereas the invariant abstract space is
\[
|\hat{\mathcal S}|=\frac{n!}{m!(n-m)!},
\]
so the ratio is \(m!\) [2003.09022]. The 2019 extended abstract had already made the same search-space argument and positioned attention-based permutation invariance as the remedy for naive ordered inputs [1905.02698].

Empirically, the full paper reports that PPO with the naive ordered representation fails to solve the food scavenger problem within 1,000 training epochs for more than two targets, whereas the invariant attention abstraction solves tasks up to five targets. In the convoy protection task, naive PPO is completely unable to learn over 10,000 training epochs and 20 million sample time steps, while the invariant abstraction learns a successful policy in 3,000 epochs and 6 million time steps [2003.09022]. The earlier extended abstract reported the same qualitative pattern: baseline policies could not achieve optimal performance for tasks with more than two food particles, while the invariant policy learned an optimal policy for all tested cases up to five food particles, and in convoy protection the invariant policy approached optimal behavior after approximately 2,500 epochs whereas the baseline showed no improvement after 10,000 epochs [1905.02698].

These results established the canonical object-set view of order-invariant RL: when object identity is irrelevant and only object attributes matter, the policy should not have to learn permutation symmetry from data.

## 2. Permutation-equivariant policies in sequential resource allocation

A second, more structured line studies order invariance in sequential allocation problems over interchangeable entities. “Efficient Reinforcement Learning in Resource Allocation Problems Through Permutation Invariant Multi-task Learning” formulates a family of MDPs
\[
\mathcal M_t=\langle \mathcal X,\mathcal A,R_t,P_t,\gamma\rangle,
\]
where each task \(t\) contains a subset of entities \(\mathcal U_t\subseteq \mathcal U\), the state is
\[
x_n=(x_{i,n})_{i\in\mathcal U_t},
\]
and the action is a simplex-valued allocation
\[
a_n=(a_{i,n})_{i\in\mathcal U_t},\qquad 0\le a_{i,n}\le 1,\qquad \sum_{i\in\mathcal U_t}a_{i,n}=1
\]
[2102.09361].

The paper defines a “Permutation Invariant Policy Network” by
\[
\pi_\theta(\sigma(x))=\sigma(\pi_\theta(x)) \quad\text{for any permutation }\sigma.
\]
As noted in the technical synthesis accompanying the paper, this is mathematically a permutation equivariance condition: reordering the entities in the state reorders the allocation vector in the same way [2102.09361]. In this setting, the symmetry is not only representational. The paper states that if the transition kernels can be factored into independent and identical transition kernels across entities, then the optimal policy is indeed permutation invariant [2102.09361].

Its main contribution is to turn this symmetry into a multi-task sample-efficiency mechanism. Task similarity is measured by the Bellman-difference operator
\[
\mathcal D_t^\pi V=\mathcal T_t^\pi V-\mathcal T^\pi V.
\]
Under a uniform mismatch bound \(\epsilon\), the paper derives a finite-sample improvement from approximately \(O(1/\sqrt N)\) for single-task learning to
\[
O(1/\sqrt{NT})+O(\epsilon)+c
\]
when learning over \(T\) related tasks [2102.09361]. When the global universe has \(M\) entities and each task uses \(m\) entities, the number of subset tasks is
\[
T={M\choose m}\ge \left(\frac{M}{m}\right)^m,
\]
which makes the task-family growth combinatorial [2102.09361].

The practical architecture is a shared-parameter per-entity scorer followed by a softmax over entities. In portfolio optimization, each instrument’s flattened price-history input is passed through a common RNN with 25 hidden units and tanh activation, concatenated with the latest allocation fraction, and then passed through a common dense layer to produce a score. In meta federated learning, the same permutation-invariant policy network architecture is used with an RNN of 10 hidden units [2102.09361]. The training procedure also introduces prioritized task sampling,
\[
p_t=\frac{s_t^\alpha}{\sum_{t'}s_{t'}^\alpha},\qquad
w_t=\frac{1}{(Tp_t)^\beta},
\]
which acts as a task-level analogue of prioritized experience replay [2102.09361].

The empirical message is that permutation symmetry is not merely an architectural prior. In the portfolio domain, multi-task learning outperforms single-task learning, 30-task MTL outperforms 5-task MTL, and prioritized MTL outperforms uniform MTL. In meta federated learning, multi-task learning discovers nonuniform client-weighting strategies that improve convergence and final accuracy over both uniform averaging and single-task RL [2102.09361]. This suggests that, in exchangeable-entity domains, order invariance can be leveraged simultaneously at the levels of representation, policy class, and task construction.

## 3. Order-randomized autoregressive reinforcement learning for combinatorial optimization

A more recent formulation appears in black-box combinatorial optimization, where the “order” in question is the autoregressive generation order of decision variables. “Black-Box Combinatorial Optimization with Order-Invariant Reinforcement Learning” studies the discrete optimization problem
\[
\mathcal X=\mathcal X_1\times\cdots\times\mathcal X_n,\qquad \max_{x\in\mathcal X} f(x),
\]
and casts candidate-solution construction as an episodic MDP whose terminal reward is the objective value \(f(x)\) [2510.01824].

For a permutation \(\sigma\) of variable indices, the generator uses the autoregressive factorization
\[
\pi_\theta(x\mid \sigma)=\prod_{k=1}^n \pi_\theta(x_{\sigma_k}\mid x_{\sigma_{<k},\sigma).
\]
The architecture does not enforce exact permutation invariance analytically. Instead, it trains the model under random generation orders and random training orders. The paper is explicit that the model is not analytically permutation-invariant by construction; it is trained to behave robustly across many orderings and not to rely on a single arbitrary factorization [2510.01824].

The policy update is based on a GRPO-style, rank-based advantage computed within the current population:
\[
A_{\Gamma_\lambda^t}(x)=U\left(\frac{\mathrm{rk}(x,\Gamma_\lambda^t,f)}{\lambda-1}\right),
\qquad
\mathrm{rk}(x,\Gamma,f)=|\{x'\in\Gamma:f(x')>f(x)\}|,
\]
with default utility
\[
U(u)=1-2u.
\]
This yields a scale-invariant, critic-free advantage signal [2510.01824]. The trust-region mechanism is a KL-regularized PPO surrogate rather than clipped PPO, because generation and training may use different masks and orders [2510.01824].

The order-randomized version introduces a sampled generation order \(\sigma\sim \xi(\sigma)\) and an independently sampled training order \(\sigma'\sim \xi(\sigma'|\sigma)\). Its central surrogate objective averages over both:
\[
L(\theta)
=
\mathbb E_{\sigma,\sigma'}
\mathbb E_{\pi_{\theta^t}(x\mid \sigma)}
\sum_{k=1}^n
\left[
\frac{\pi_\theta(x_k\mid \sigma'(x)_{<k})}
{\pi_{\theta^t}(x_k\mid \sigma(x)_{<k})}
A^{\pi_{\theta^t}(\sigma(x)_{<k},x_k)}
-
\beta D_{\mathrm{KL}}\!\left(
\pi_{\theta^t}(\cdot\mid \sigma(x)_{<k})
\;\|\;
\pi_\theta(\cdot\mid \sigma'(x)_{<k})
\right)
\right]
\]
[2510.01824]. The paper interprets random order training as a form of information-preserving dropout: at any conditional prediction, some future variables are masked out, but the full candidate solution still underlies the joint model [2510.01824].

The empirical ablation isolates four variants: fixed generation and fixed training order, fixed generation and random training order, random generation with training on the same order, and random generation with random training order. On NK with \(N=256\) and \(K=4\), \((\sigma,\sigma')\)-RL-EDA is the best variant [2510.01824]. On the full benchmark, it ranks \(1/505\) on all six QUBO instance types at \(n=256\), ranks \(1/505\) on all four NK ruggedness settings at \(n=256\), and is markedly weaker only on the hardest categorical NK3 setting with \(K=8\) [2510.01824]. The paper therefore treats order randomization as both a diversity mechanism and a regularizer against overfitting to arbitrary generation order.

This formulation broadens the meaning of order invariance. The goal is no longer to make the policy invariant to object permutation in its input, but to prevent the generative search distribution from collapsing onto one arbitrary variable order.

## 4. Order as rank, topological sequence, or policy partial order

Not all work labeled or adjacent to order-invariant RL concerns permutation symmetry of observations. A distinct strand concerns ordinal structure in the reward signal itself. “Deep Ordinal Reinforcement Learning” replaces scalar rewards by ordered reward tiers
\[
r_o\in\{1,\dots,n\},
\]
represents state-action quality by a distribution vector
\[
D(s,a)=
\begin{bmatrix}
d_1(s,a)\ \dots\ d_n(s,a)
\end{bmatrix},
\]
and compares actions using the measure of statistical superiority
\[
\mathds P(a\succ a')
=
\sum_{o=1}^n
p_o(s,a)\left(p_{o^<}(s,a')+\frac{1}{2}p_o(s,a')\right)
\]
rather than expected reward magnitude [1905.02005]. The deep extension uses an array of \(k\) neural networks, one per action, each producing an \(n\)-dimensional ordinal distribution [1905.02005]. This is order-based RL in the sense of rank-preserving rewards, not in the sense of permutation-invariant state encoders.

A different use of “order” appears in “Reinforcement Causal Structure Learning on Order Graph.” There, RL is performed over a layered graph whose states are subsets of already ordered variables and whose actions add one new variable, so that an episode constructs a complete topological order. The method defines
\[
Q(s,a)=R(a)\sum_{a'\in\mathcal A_{s'}}Q(s',a')
\]
and learns a posterior-like transition model over orderings by DQN on this order graph [2211.12151]. The paper is explicit that this is not permutation invariance in the representation-theoretic sense. It is reinforcement learning over many possible orderings rather than commitment to one fixed order [2211.12151].

“Reinforcement Learning in MDPs with Information-Ordered Policies” uses yet another meaning. It defines a partial order \(\pi_{\theta'}\preceq \pi_\theta\) when data collected under \(\pi_\theta\) can be used to estimate the performance of \(\pi_{\theta'}\). In the sample-path version,
\[
\pi_{\theta'}\preceq \pi_\theta
\quad\Longleftrightarrow\quad
G_{\theta'}(\mathcal H_{\theta'}^T)\in \mathcal H_\theta^T
\quad\text{for all }T\in\mathbb N^+,
\]
and the resulting epoch-based policy-elimination algorithm attains a regret bound
\[
O(\sqrt{w \log(|\Theta|) T}),
\]
where \(w\) is the width of the partial order [2508.03904]. The paper emphasizes that this bound is independent of the state and action space sizes [2508.03904]. Here again, the subject is not order invariance of inputs, but an order-theoretic structure over policies.

Taken together, these works show that “order” in RL can refer to at least three mathematically distinct objects: reward rankings, topological variable orderings, and partial orders over policies. None is interchangeable with permutation invariance over sets.

## 5. Adjacent symmetry-based RL that is not order invariance

A large adjacent literature studies invariance in RL without targeting order invariance specifically. “Invariant Transform Experience Replay” is exemplary. It models multi-goal robotic tasks as a multi-goal MDP
\[
\langle \mathcal S,\mathcal A,\mathcal G,T,R,p,\gamma\rangle
\]
and defines a symmetry as a bijection
\[
\sigma:\overline\Gamma\to\overline\Gamma
\quad\text{such that}\quad
\sigma(\Gamma)=\Gamma,
\]
where \(\Gamma\) is the space of feasible trajectories [1909.10707]. Its two instantiations are Kaleidoscope Experience Replay, which uses geometric reflection symmetry, and Goal-augmented Experience Replay, which uses threshold-induced equivalence classes of goals. In the Fetch tasks, the paper reports 13×, 3×, and 5× speedups in pushing, sliding, and pick-and-place, respectively, and 46/50 successful trials on a real Baxter robot [1909.10707]. The paper is explicit that this is trajectory-space symmetry invariance, not permutation or order invariance of observations or policy architecture [1909.10707].

“Invariant Policy Optimization” studies domain invariance rather than order invariance. It seeks a representation \(\Phi\) and policy \(\pi\) such that the same predictor is simultaneously optimal across all training domains:
\[
\max_{\Phi,\pi}\sum_{d=1}^{n_d}R^d(\pi\circ\Phi)
\quad\text{s.t.}\quad
\pi\in \arg\max_{\bar\pi}R^d(\bar\pi\circ\Phi),\ \forall d.
\]
The practical algorithm uses one policy per domain, an averaged policy, and best-response PPO updates [2006.01096]. Its relevance to order-invariant RL is therefore indirect: it is an invariant RL method, but its symmetry axis is inter-domain stability rather than permutation of exchangeable inputs [2006.01096].

“Reinforcement Learning Using known Invariances” provides a more general group-theoretic template. It studies episodic MDPs invariant under a known group \(G\) acting on states and actions, defines invariant kernels
\[
k_G(z,z')=\frac{1}{|G|}\sum_{g\in G}k(g(z),z'),
\]
and shows that symmetry-aware kernelized optimistic value iteration improves information-gain and covering-number complexity terms by factors depending on \(|G|\) [2511.03473]. The paper explicitly notes that this framework applies naturally to order-invariant RL by taking \(G\) to be a permutation group \(S_n\) acting on state and action representations [2511.03473]. This suggests a route from generic symmetry-aware RL theory to exact permutation invariance, although the reported experiments are geometric rather than permutation-based.

Finally, “Task-Induced Representational Invariances Depend on Learning Objective in Deep RL” studies invariance that emerges from the optimization objective itself. In navigation domains, DQN learns representations invariant to MDP homomorphism symmetries, while PPO learns representations invariant to action symmetries [2606.01868]. The paper presents this as structural symmetry, not literal permutation invariance over exchangeable sets [2606.01868]. The result is important because it shows that invariance in RL is not monolithic: different objectives compress different equivalence classes.

## 6. Conceptual boundaries and persistent misconceptions

The literature therefore supports a sharp distinction between at least five notions that are often conflated. First, there is **permutation invariance over exchangeable observations**, where a state is a set and the output should be unchanged by reordering its elements; this is the sense targeted by attention-based set encoders and sequential allocation policies [2003.09022]. Second, there is **permutation equivariance**, where reordering object inputs should reorder object-indexed actions in the same way; the resource-allocation literature uses this formulation explicitly, even when it is named “Permutation Invariant Policy Network” [2102.09361]. Third, there is **ordinal RL**, where only the order of reward categories is retained [1905.02005]. Fourth, there are **order-structured search problems**, such as topological ordering in causal discovery and information orders over policy classes [2211.12151][2508.03904]. Fifth, there are **broader symmetry-based RL methods** that exploit geometry, group actions, or domain invariance without addressing input order directly [1909.10707][2006.01096].

A recurring misconception is that any “invariant RL” paper is an order-invariant RL paper. The available evidence does not support that conflation. Invariant replay augmentation over reflected robot trajectories, invariant policy learning across training domains, and MDP-homomorphism invariance in learned representations are all technically meaningful forms of invariance, but they do not solve the same problem as exchangeable-set encoding [1909.10707][2006.01096][2606.01868].

Another recurrent boundary concerns exact versus induced invariance. The attention-based set encoders and permutation-equivariant resource-allocation policies hard-code symmetry into the architecture [2003.09022][2102.09361]. By contrast, the order-randomized autoregressive EDA is not exactly invariant by construction; it encourages robustness across orderings by randomizing generation order and training order inside a KL-regularized RL objective [2510.01824]. This suggests that the field contains both algebraic symmetry enforcement and stochastic symmetry induction.

The outstanding technical issues are correspondingly varied. The exchangeable-input representation of Mern, Sadigh, and Kochenderfer does not model pairwise or higher-order object interactions in its attention score, since object importance is determined without considering other objects present [2003.09022]. The resource-allocation framework assumes exact or near symmetry across tasks and incurs an \(O(\epsilon)\) penalty when that assumption is violated [2102.09361]. The kernel-based group-invariance theory is formulated for exact known symmetries and leaves partial or approximate symmetries to future work [2511.03473]. The black-box combinatorial optimizer encourages rather than guarantees order invariance and remains weaker on the hardest NK3 setting with \(K=8\) [2510.01824].

This suggests that order-invariant reinforcement learning is best understood not as a single method class but as a cluster of strategies for quotienting out irrelevant order structure. In the strictest sense, it refers to RL on exchangeable sets with permutation-invariant or permutation-equivariant policies. In a broader sense, it includes ordinal returns, search over order spaces, and policy classes ordered by counterfactual identifiability. The technical commonality is the same: exploiting an equivalence relation that would otherwise force the learner to spend samples distinguishing states, actions, or trajectories that should not matter for control.

Source: https://www.emergentmind.com/topics/order-invariant-reinforcement-learning