Inverse Permutation Learning

• Inverse permutation learning is defined as recovering a canonical ordering from permuted data by predicting the inverse permutation or constructing a permutation matrix.
• It employs differentiable relaxations, such as the Sinkhorn operator, to convert discrete permutation constraints into continuous, trainable representations.
• The topic also examines transformer expressivity and architectural limitations, providing insights into scalability and efficiency in sequence recovery tasks.

Searching arXiv for papers directly relevant to inverse permutation learning, including the provided IDs, so the article can be grounded in the current literature. Could you share the arXiv search tool output or enable the search tool? I need it to verify and ground the article in current arXiv papers as requested. Inverse permutation learning studies how to recover a canonical ordering from permuted data. In one formulation, a set $X=\{x_1,\ldots,x_n\}$ is scrambled by an unknown permutation $\pi$ into $X_\pi=[x_{\pi(1)},\ldots,x_{\pi(n)}]$, and the objective is either to predict $\pi^{-1}$ directly or to construct a permutation matrix $P$ that restores the original order. In another formulation, the input explicitly includes both a permutation and a permuted sequence, $(\pi,\pi\cdot x)$, and the task is to recover the original sequence $x$. The topic therefore lies at the interface of permutation-invariant representation learning, differentiable relaxations of discrete combinatorics, and transformer expressivity under architectural constraints (Zhang et al., 2018, Alur et al., 28 Sep 2025).

1. Formal problem statements

In the set-based formulation, the inverse-permutation learning problem is defined by an unknown permutation $\pi$ on $\{1,\ldots,n\}$ applied to a set $X=\{x_1,\ldots,x_n\}$, producing

$\pi$0

The target model $\pi$1, given the scrambled set $\pi$2, either predicts the inverse permutation $\pi$3 directly or produces a permutation matrix $\pi$4 such that

$\pi$5

thereby recovering a canonical ordering such as the sorted order (Zhang et al., 2018).

A permutation matrix in this formulation satisfies three constraints: $\pi$6, $\pi$7 for each row $\pi$8, and $\pi$9 for each column $X_\pi=[x_{\pi(1)},\ldots,x_{\pi(n)}]$0. Because these constraints are discrete, the matrix is relaxed to a real matrix $X_\pi=[x_{\pi(1)},\ldots,x_{\pi(n)}]$1 constrained to be doubly-stochastic, that is, an element of the Birkhoff polytope (Zhang et al., 2018).

The sequence-based formulation makes the permutation explicit. Let $X_\pi=[x_{\pi(1)},\ldots,x_{\pi(n)}]$2, and define the permuted sequence by

$X_\pi=[x_{\pi(1)},\ldots,x_{\pi(n)}]$3

The task is: input the pair $X_\pi=[x_{\pi(1)},\ldots,x_{\pi(n)}]$4, output $X_\pi=[x_{\pi(1)},\ldots,x_{\pi(n)}]$5. Equivalently, if $X_\pi=[x_{\pi(1)},\ldots,x_{\pi(n)}]$6 is the permutation matrix with $X_\pi=[x_{\pi(1)},\ldots,x_{\pi(n)}]$7 iff $X_\pi=[x_{\pi(1)},\ldots,x_{\pi(n)}]$8, and if $X_\pi=[x_{\pi(1)},\ldots,x_{\pi(n)}]$9, then the goal is to compute

$\pi^{-1}$0

Since $\pi^{-1}$1 is a permutation matrix, $\pi^{-1}$2, so the desired output is $\pi^{-1}$3 (Alur et al., 28 Sep 2025).

These two formulations differ in whether the permutation itself is observed, but they coincide on the central objective: undo the action of a permutation and recover an ordered representation. This suggests a useful conceptual split between latent-permutation inversion in set models and explicit-permutation inversion in sequence models.

2. Differentiable permutation recovery

The central construction in "Learning Representations of Sets through Optimized Permutations" is the Permutation-Optimisation module, which learns how to permute a set end-to-end before downstream processing (Zhang et al., 2018). The module begins with a pairwise ordering function

$\pi^{-1}$4

where $\pi^{-1}$5 is a small MLP applied to the concatenation $\pi^{-1}$6. By construction, $\pi^{-1}$7 is antisymmetric, so $\pi^{-1}$8. The full matrix is then normalized to unit Frobenius norm,

$\pi^{-1}$9

Under this convention, $P$0 indicates that $P$1 should go after $P$2, while $P$3 indicates that $P$4 should go before $P$5 (Zhang et al., 2018).

To enforce approximate row- and column-sum constraints differentiably, the method applies the Sinkhorn operator. Starting from

$P$6

it performs $P$7 rounds of alternating column and row normalization: $P$8 The final output

$P$9

is an approximately doubly-stochastic matrix. Since exponentiation and normalization are differentiable, gradients can propagate from a downstream loss into $(\pi,\pi\cdot x)$0 and therefore into the parameters of the ordering network (Zhang et al., 2018).

The model does not only predict $(\pi,\pi\cdot x)$1 in one shot. It introduces an unnormalized permutation matrix $(\pi,\pi\cdot x)$2 and optimizes it by gradient descent on a total cost $(\pi,\pi\cdot x)$3. At each inner iteration $(\pi,\pi\cdot x)$4, it computes

1. $(\pi,\pi\cdot x)$5,
2. $(\pi,\pi\cdot x)$6 evaluated at $(\pi,\pi\cdot x)$7,
3. $(\pi,\pi\cdot x)$8.

After $(\pi,\pi\cdot x)$9 steps, the module outputs $x$0. Because the inner loop is unrolled, the entire optimization remains end-to-end differentiable (Zhang et al., 2018).

The total cost is

$x$1

Its interpretation is explicitly pairwise: if $x$2 places $x$3 on average before $x$4, the term in brackets is positive, so the model pays $x$5; if $x$6 places $x$7 after $x$8, the bracket is negative, so it pays $x$9 (Zhang et al., 2018).

3. Supervision regimes and reported empirical behavior

The 2018 formulation distinguishes explicit and implicit supervision. Under explicit supervision, when a target permutation $\pi$0 is known, the optimization objective is the negative log-likelihood of the ground-truth assignment,

$\pi$1

Under implicit supervision, there is no direct permutation target; the model receives only a downstream task loss $\pi$2, such as classification or reconstruction, and back-propagates that loss through the Sinkhorn-based permutation module into the ordering function $\pi$3 (Zhang et al., 2018).

The reported experiments span four distinct inverse-permutation tasks. They are notable because the same module is used both when the canonical order is known and when permutation recovery is only an intermediate latent computation.

Task	Setup	Reported result
Number sorting	Sets of $\pi$4 real numbers; sorted order as supervision	PO-U sorts up to $\pi$5 perfectly and generalizes across intervals $\pi$6, $\pi$7, $\pi$8
Image-mosaic reassembly	Images split into $\pi$9 tiles, shuffled; reconstruction by MSE	On CIFAR10 $\{1,\ldots,n\}$0: LinAssign MSE $\{1,\ldots,n\}$1 vs. PO-LA $\{1,\ldots,n\}$2; PO-LA achieves highest exact reassembly accuracy
Implicit classification from mosaics	Shuffled tiles classified via a frozen ResNet-18	On CIFAR10 $\{1,\ldots,n\}$3: random tiles $\{1,\ldots,n\}$4, LinAssign $\{1,\ldots,n\}$5, PO-U $\{1,\ldots,n\}$6, PO-LA $\{1,\ldots,n\}$7, ground-truth max $\{1,\ldots,n\}$8
Visual question answering	Variable-size set of $\{1,\ldots,n\}$9–$X=\{x_1,\ldots,x_n\}$0 object proposals plus question; BAN baseline	On VQA v2 val: BAN $X=\{x_1,\ldots,x_n\}$1 $X=\{x_1,\ldots,x_n\}$2 BAN+PO-U $X=\{x_1,\ldots,x_n\}$3 overall; “Number” improves from $X=\{x_1,\ldots,x_n\}$4 to $X=\{x_1,\ldots,x_n\}$5

In number sorting, the loss is the mean-squared error between network output and the ground-truth sorted sequence, together with Hungarian post-processing. In image-mosaic reassembly, the module uses a grid-version in which row- and column-costs are combined. In the mosaic-classification setting, the permutation module is trained implicitly by the classification cross-entropy while the classifier is a frozen ResNet-18. In VQA, PO-U is inserted per attention glimpse to permute object features into a sequence fed to an LSTM, with BAN as the baseline architecture (Zhang et al., 2018).

The paper summarizes these experiments as state-of-the-art results on number sorting, image mosaics, classification from image mosaics, and visual question answering (Zhang et al., 2018).

4. Complexity, sensitivities, and extension points

The computational bottleneck in the Permutation-Optimisation module is the evaluation of $X=\{x_1,\ldots,x_n\}$6, which requires a matrix multiplication of two $X=\{x_1,\ldots,x_n\}$7 matrices and therefore has $X=\{x_1,\ldots,x_n\}$8 cost. With $X=\{x_1,\ldots,x_n\}$9 inner optimization steps, the overall cost per forward pass is $\pi$00. Empirically, $\pi$01–$\pi$02 suffices (Zhang et al., 2018).

This cubic complexity leads directly to the stated scalability limit: scalability beyond $\pi$03 is challenging due to cubic cost. The method is also sensitive to the Sinkhorn temperature $\pi$04 and the number of Sinkhorn iterations $\pi$05, both of which affect convergence and sharpness (Zhang et al., 2018).

A further limitation concerns ambiguity in absolute positions. Two identical or blank elements can cause ambiguity in absolute positions; the paper states that this is mitigated by the linear-assignment initialization PO-LA. This is important because inverse permutation learning is not only a question of relative ordering but also of whether the data itself determines a unique canonical arrangement (Zhang et al., 2018).

The extension points identified in the same work remain narrowly technical: faster graph-partition or divide-and-conquer schemes to reduce complexity, alternative total-cost functions or convex relaxations, and variational hard-permutation relaxations (Gumbel–Sinkhorn) (Zhang et al., 2018). A plausible implication is that the main open problems are less about the existence of differentiable relaxations and more about their algorithmic scaling, conditioning, and identifiability under weak supervision.

5. Transformer expressivity and impossibility results

A later technical note shifts the focus from differentiable permutation modules to the expressive capacity of transformer architectures. Its formal task fixes $\pi$06, takes input $\pi$07, and asks whether a decoder-only transformer can recover the original sequence $\pi$08 for all permutations and all inputs (Alur et al., 28 Sep 2025). The transformer is modeled as a function

$\pi$09

with input residual stream $\pi$10, where $\pi$11, and the model is said to solve inverse permutation learning if some contiguous block of the final residual stream recovers $\pi$12 for all $\pi$13 and $\pi$14 (Alur et al., 28 Sep 2025).

The central theorem is an impossibility result for decoder-only transformers with causal masks: for any choice of decoder-only, causal-mask transformer parameters and any nontrivial permutation matrix $\pi$15, there exists an input $\pi$16 such that the transformer's output does not contain the canonical inverse-permuted matrix $\pi$17 in any block of its final residual stream (Alur et al., 28 Sep 2025).

The proof outline is based on a causal-mask invariance lemma: in a causal-mask attention layer, the representation at output position $\pi$18 depends only on input positions $\pi$19. Therefore, if a nontrivial permutation moves an element from a later input position $\pi$20 to an earlier output location $\pi$21, information cannot flow from token $\pi$22 back to position $\pi$23. The architectural obstruction is thus independent of training dynamics or sample complexity; it concerns expressive capacity (Alur et al., 28 Sep 2025).

The same note gives two constructive alternatives under which inverse permutation learning is feasible. First, if the causal mask is removed, there exist parameters for a 2-layer causal-mask-free transformer such that the final residual stream contains the block $\pi$24 exactly. Second, in a causal decoder, augmenting the input $\pi$25 with scratch-token rows $\pi$26 permits a 2-layer construction that outputs $\pi$27 in the final residual stream (Alur et al., 28 Sep 2025).

The scratch-token construction is framed mechanistically. The extra rows act as a temporary buffer in later positions. Layer 1 copies the permuted data $\pi$28 forward into the scratch area, exploiting the allowed causal direction from past to future; Layer 2 copies from the scratch area back to the original data positions in inverse order, thereby reassembling the canonical form. The note conjectures that this may suggest an alternative mechanism by which chain-of-thought prompting, or more generally intermediate “thinking” tokens, can enable reasoning in LLMs even when the tokens encode no meaningful semantic information (Alur et al., 28 Sep 2025).

The paper explicitly argues that inverse permutation learning models a natural robustness property across long-context retrieval, multiple choice QA, and in-context learning (Alur et al., 28 Sep 2025). This suggests that inverse permutation learning is not merely an abstract sorting problem: it also functions as a diagnostic for whether an architecture can undo arbitrary reorderings of information under its own information-flow constraints.

6. Conceptual boundaries and terminological disambiguation

The phrase “inverse permutation” can refer to mathematically distinct objects, and inverse permutation learning should not be conflated with algebraic inversion of permutation polynomials. A separate line of work constructs permutation trinomials over $\pi$29 and gives explicit polynomial formulas for their compositional inverses. One recent example constructs four new classes of permutation trinomials over the cubic extension of a finite field with even characteristic and explicitly provides the compositional inverse of each class in polynomial form (Hasan et al., 4 May 2025).

In that algebraic setting, the object being inverted is a polynomial bijection on $\pi$30, not a learned permutation acting on a set or sequence. The methods are correspondingly different: the paper uses the local method, Frobenius conjugation, resultant computations, and Dickson-matrix formulas for $\pi$31-linear maps (Hasan et al., 4 May 2025). For example, when $\pi$32 with $\pi$33, inversion is reduced to inverting the $\pi$34-linear map $\pi$35; in other cases, one partitions $\pi$36 into fibers and glues a piecewise inverse (Hasan et al., 4 May 2025).

The shared vocabulary is therefore potentially misleading. Inverse permutation learning concerns recovery of canonical orderings from permuted observations, often under differentiability or architectural constraints. Compositional inverses of permutation trinomials concern explicit inverses of finite-field permutations. The commonality is the inversion of a bijection; the divergence is between statistical learning and algebraic closed-form inversion.

Taken together, the literature identifies two complementary dimensions of the topic. One dimension is algorithmic: how to learn or relax permutations so that inverse ordering can be recovered end-to-end from sets or structured observations (Zhang et al., 2018). The other is architectural: which model classes can, in principle, realize inverse permutation learning, and which cannot, due to constraints such as the causal attention mask (Alur et al., 28 Sep 2025). Under that synthesis, inverse permutation learning serves simultaneously as an optimization problem, a representation-learning primitive, and a probe of the computational limits of sequence models.