---
title: Permutation Language Modeling (PLM)
url: https://www.emergentmind.com/topics/permutation-language-modeling-plm
type: topic
---

# Permutation Language Modeling (PLM)

Permutation Language Modeling (PLM) is a pre-training paradigm that generalizes traditional language modeling by replacing a fixed or masked token prediction objective with prediction under random or adaptive permutations of the input or target sequence. Unlike conventional masked language modeling (MLM), where models learn to recover masked tokens given their surrounding context, or left-to-right (L2R) autoregressive language modeling, which predicts tokens using only the left context, PLM exposes the model to a stochastic or structured variety of prediction factors. By altering which tokens are considered context or targets according to sampled permutations, PLM enables each token to be conditioned on diverse, possibly bidirectional, contextual information, improving models' capacity to capture both short- and long-range dependencies in language, vision, and multimodal tasks.

## 1. Formal Objectives and Mathematical Foundations

The canonical PLM framework defines the training objective in terms of permutations $z$ of a token sequence $x_1, \ldots, x_T$. For a randomly drawn permutation $z = (z_1, \dots, z_T)$ and a context cut-point $c$, the PLM objective aims to maximize the log-likelihood of the "future" tokens in permutation order, conditioned on the "past":

$$
L_{PLM}(\theta) = \mathbb{E}_{z \sim Z_T} \Big[ \sum_{t=c+1}^T \log p_\theta(x_{z_t} \mid x_{z_{<t}}) \Big]
$$
[2004.09297][2210.12339][1909.10924].

This formulation contrasts with MLM, where the model predicts masked tokens that are replaced with a [MASK] token and are conditionally independent given the visible context. In PLM, the autoregressive conditioning structure is over a random permutation, so that the model must successively predict each target token in the permutation, conditioned on its permuted history. For multi-modal or generative settings, such as in P$^3$LM, the permutation is over target tokens and the joint likelihood aggregates across all possible or sampled permutations during training [2210.12339].

Notable PLM variants include:

- **Position Prediction** (e.g., PERT): Shuffle a subset of tokens and train the model to predict the original position for each permuted token. The loss is a cross-entropy over possible positions [2203.06906].
- **Target Prediction** (e.g., XLNet/MPNet): For each permutation, predict tokens in the permuted order, potentially preceded by a context of visible tokens, and optimize the likelihood over both permutation and prediction order [2004.09297].

Some architectures, such as MPNet, further incorporate explicit position compensation to address position information discrepancies present in vanilla PLM [2004.09297].

## 2. Permutation Strategies and Sampling Schemes

PLM implementations differ in the scope and manner in which permutations are applied:

- **Global Permutations**: Full random permutations over all sequence positions, as in XLNet's original formulation or P$^3$LM. Computation is restricted by sampling only a few permutations per example for tractability [2004.09297][2210.12339][2207.06966].
- **Partial Permutations**: Permute only a subset of the sequence (e.g., 15%), using N-gram or whole-word heuristics to identify candidates for shuffling; the remainder is left in canonical order [2203.06906].
- **Adaptive Permutations**: Learn a set of attention masks or orderings conditioned on the input or model state, such as HAAP's Implicit Permutation Neurons (IPN), which dynamically compute permutation-dependent attention masks via learned linear transformations of query and order embeddings [2405.09125].

Permutation selection impacts both training efficiency and convergence. Random permutations increase training data diversity but can introduce training instability ("fit oscillations"), while adaptive or structured permutations can preserve stability and ensure coverage of informative conditional dependencies [2405.09125][2207.06966].

## 3. Model Architectures and Training Procedures

PLM is agnostic to underlying model architecture but must flexibly support variable context masking and conditional inference. Key approaches include:

- **Transformer Encoders**: Used in encoder-only formulations, such as XLNet and PERT, where bidirectional or partial context attention is realized by configuring attention masks according to the current permutation [2203.06906][2004.09297].
- **Two-Stream Attention**: XLNet and derivatives utilize a two-stream architecture: a content stream encodes observed context, and a query stream computes conditional predictions with appropriate masks to prevent target-leakage along permuted orders [2004.09297].
- **Encoder–Decoder**: In generative tasks, P$^3$LM introduces a standard encoder–decoder backbone but uses a multi-stream decoder with order-aware self-attention and "prophet streams" to simultaneously predict multiple future tokens under permutation, with each stream governed by a separate attention mask [2210.12339].
- **Multimodal Extensions**: For vision-language (e.g., scene text recognition), PARSeq and HAAP replace or augment random permutations with ViT-based image encoders, and merge permutation-masked language modeling with cross-modal attention decoders [2207.06966][2405.09125].

Training is typically conducted using Adam or AdamW optimizers, large batch sizes, and is often parallelized across available permutations within each batch. Unlike standard MLM, PLM models do not rely on artificial [MASK] tokens (except in MPNet's "mask block" for position compensation [2004.09297]), thereby reducing the pretrain–finetune divergence in token and position distributions.

## 4. Integration with Masking, Position, and Hybrid Objectives

PLM-based pre-training is often combined or contrasted with traditional masking strategies:

- **Masking Integration**: PERT, for example, applies whole-word and N-gram masking heuristics to identify candidate shuffle groups, leveraging insights from MacBERT's masking strategies [2203.06906].
- **Hybrid Architectures**: MPNet combines permutation prediction with explicit mask token insertion to maintain full positional exposure, thus addressing the position information gap between pre-training and downstream fine-tuning absent in plain XLNet [2004.09297].
- **Bidirectionality and Cloze-style Decoding**: PLM enables bidirectional context use in both encoding and decoding by permuting the order in which target tokens are predicted (as in PARSeq's unified AR, non-AR, and cloze refinement modes) [2207.06966].

PLM-based models can be further enhanced by tuning permutation granularity (token-level > N-gram > word-level), jointly training with both masking and permutation, or dynamically harvesting pre-training checkpoints based on downstream task needs [2203.06906].

## 5. Empirical Findings and Downstream Task Impact

Extensive empirical evaluations demonstrate that PLM-induced pre-training yields gains on certain classes of tasks:

- **Span Extraction and Tagging**: PERT achieves superior results on tasks closely aligned with its permutation prediction objective—e.g., machine reading comprehension, named entity recognition, and word order recovery—often outperforming or matching masked-LM baselines (e.g., +1.5 F1 on CMRC, +0.3–0.5 F1 on NER) [2203.06906].
- **Sequence Generation**: P$^3$LM achieves state-of-the-art summarization and question generation performance, outperforming ProphetNet on the GLGE benchmark by +0.9 in average score, and benefiting particularly from multi-stream permutation-based prediction [2210.12339].
- **Scene Text Recognition (STR)**: PLM models such as PARSeq and HAAP achieve or exceed state-of-the-art STR accuracy (91.9–96.0%), with the ability to flexibly unify AR, NAR, and bidirectional inference within one architecture, as well as reduced FLOPS and lower latency in adaptive permutation regimes [2207.06966][2405.09125].
- **Spoken Language Understanding**: BERT-PLM, applied to phoneme posterior inputs, results in 10–15% relative error reduction in SLU benchmarks, especially helpful in low-resource or small fine-tuning datasets [1909.10924].

However, PLM's effectiveness can vary by task. For example, PERT underperforms MLM-based models on classification tasks sensitive to global word order and PERT's classification accuracy can be ~1 point lower than strong MLM baselines [2203.06906]. Empirical analysis indicates benefits saturate with a small number of sampled permutations (K=6 for PARSeq [2207.06966], K=2 for HAAP/IPN [2405.09125]).

## 6. Limitations, Variants, and Open Problems

Key challenges and limitations include:

- **Training–Inference Mismatch**: Decoding at inference is almost always performed in L2R order, even when the model is trained on a mixture of permuted orderings (P$^3$LM, XLNet). It is unclear whether optimized ordering at inference can further improve generation [2210.12339].
- **Computational Overhead**: Sampling multiple permutations per example increases training cost, though adaptive or learned permutation strategies such as IPN can halve FLOPS and improve training stability [2405.09125].
- **Position Discrepancy**: Without explicit position compensation, as in MPNet, the model may suffer from partial exposure to position information during pre-training compared to fine-tuning, which can impair adaptation to downstream tasks reliant on positional accuracy [2004.09297].
- **Permutation Distribution**: The optimal choice of permutation distribution remains open; plausibly, syntax- or content-informed permutations could yield larger gains than uniform random sampling [2210.12339].

Active directions for research involve hybridizing PLM with masking objectives, task-adaptive permutation schedules, order regularization, dynamic checkpointing tailored to downstream requirements, and theoretical characterization of expressivity and inductive biases introduced by permutation-based factorization [2203.06906][2210.12339].

## 7. Cross-Modal and Non-Textual Extensions

Recent advances demonstrate PLM's relevance beyond pure text:

- **Vision-Context STR**: In scene text recognition, internal PLM-based decoders enable robust character sequence modeling, unify context-free and context-aware inference, and provide bidirectional refinement in a single weight-shared transformer (PARSeq, HAAP) [2207.06966][2405.09125].
- **Speech**: BERT-PLM extends PLM to phoneme posterior sequences, supporting regression objectives and full bidirectional context over acoustic-probability vectors, yielding substantial downstream SLU improvements [1909.10924].
- **Cross-modal Attention**: Hierarchical attention mechanisms can be holistically integrated with permutation-induced masking (e.g., CHA in HAAP), enabling tight coupling of context, position, and cross-modal features without costly iterative refinement [2405.09125].

This suggests PLM is a general strategy applicable to diverse modalities and architectures, capable of improving robustness to context fragmentation and providing a uniform approach to bidirectional dependency modeling.

---

**References**

- [2203.06906] PERT: Pre-training BERT with Permuted Language Model
- [2405.09125] HAAP: Vision-context Hierarchical Attention Autoregressive with Adaptive Permutation for Scene Text Recognition
- [1909.10924] Understanding Semantics from Speech Through Pre-training
- [2207.06966] Scene Text Recognition with Permuted Autoregressive Sequence Models
- [2004.09297] MPNet: Masked and Permuted Pre-training for Language Understanding
- [2210.12339] P$^3$LM: Probabilistically Permuted Prophet Language Modeling for Generative Pre-Training

Source: https://www.emergentmind.com/topics/permutation-language-modeling-plm