Permutation Language Modeling (PLM)

• Permutation language modeling is a pre-training paradigm that replaces fixed token prediction with random or adaptive token orderings to provide rich bidirectional context.
• Methodologies include global, partial, and adaptive permutation strategies implemented in Transformer architectures with two-stream attention and position compensation.
• Empirical studies show PLM enhances tasks like text comprehension, sequence generation, and scene text recognition, while challenges include training–inference mismatches and computational overhead.

Permutation Language Modeling (PLM) is a pre-training paradigm that generalizes traditional language modeling by replacing a fixed or masked token prediction objective with prediction under random or adaptive permutations of the input or target sequence. Unlike conventional masked language modeling (MLM), where models learn to recover masked tokens given their surrounding context, or left-to-right (L2R) autoregressive language modeling, which predicts tokens using only the left context, PLM exposes the model to a stochastic or structured variety of prediction factors. By altering which tokens are considered context or targets according to sampled permutations, PLM enables each token to be conditioned on diverse, possibly bidirectional, contextual information, improving models' capacity to capture both short- and long-range dependencies in language, vision, and multimodal tasks.

1. Formal Objectives and Mathematical Foundations

The canonical PLM framework defines the training objective in terms of permutations $z$ of a token sequence $x_1, \ldots, x_T$. For a randomly drawn permutation $z = (z_1, \dots, z_T)$ and a context cut-point $c$, the PLM objective aims to maximize the log-likelihood of the "future" tokens in permutation order, conditioned on the "past":

$$L_{PLM}(\theta) = \mathbb{E}_{z \sim Z_T} \Big[ \sum_{t=c+1}^T \log p_\theta(x_{z_t} \mid x_{z_{<t}}) \Big]$$

(Song et al., 2020, Bao et al., 2022, Wang et al., 2019).

This formulation contrasts with MLM, where the model predicts masked tokens that are replaced with a [MASK] token and are conditionally independent given the visible context. In PLM, the autoregressive conditioning structure is over a random permutation, so that the model must successively predict each target token in the permutation, conditioned on its permuted history. For multi-modal or generative settings, such as in P$^3$LM, the permutation is over target tokens and the joint likelihood aggregates across all possible or sampled permutations during training (Bao et al., 2022).

Notable PLM variants include:

• Position Prediction (e.g., PERT): Shuffle a subset of tokens and train the model to predict the original position for each permuted token. The loss is a cross-entropy over possible positions (Cui et al., 2022).
• Target Prediction (e.g., XLNet/MPNet): For each permutation, predict tokens in the permuted order, potentially preceded by a context of visible tokens, and optimize the likelihood over both permutation and prediction order (Song et al., 2020).

Some architectures, such as MPNet, further incorporate explicit position compensation to address position information discrepancies present in vanilla PLM (Song et al., 2020).

2. Permutation Strategies and Sampling Schemes

PLM implementations differ in the scope and manner in which permutations are applied:

• Global Permutations: Full random permutations over all sequence positions, as in XLNet's original formulation or P$^3$LM. Computation is restricted by sampling only a few permutations per example for tractability (Song et al., 2020, Bao et al., 2022, Bautista et al., 2022).
• Partial Permutations: Permute only a subset of the sequence (e.g., 15%), using N-gram or whole-word heuristics to identify candidates for shuffling; the remainder is left in canonical order (Cui et al., 2022).
• Adaptive Permutations: Learn a set of attention masks or orderings conditioned on the input or model state, such as HAAP's Implicit Permutation Neurons (IPN), which dynamically compute permutation-dependent attention masks via learned linear transformations of query and order embeddings (Chen et al., 2024).

Permutation selection impacts both training efficiency and convergence. Random permutations increase training data diversity but can introduce training instability ("fit oscillations"), while adaptive or structured permutations can preserve stability and ensure coverage of informative conditional dependencies (Chen et al., 2024, Bautista et al., 2022).

3. Model Architectures and Training Procedures

PLM is agnostic to underlying model architecture but must flexibly support variable context masking and conditional inference. Key approaches include:

• Transformer Encoders: Used in encoder-only formulations, such as XLNet and PERT, where bidirectional or partial context attention is realized by configuring attention masks according to the current permutation (Cui et al., 2022, Song et al., 2020).
• Two-Stream Attention: XLNet and derivatives utilize a two-stream architecture: a content stream encodes observed context, and a query stream computes conditional predictions with appropriate masks to prevent target-leakage along permuted orders (Song et al., 2020).
• Encoder–Decoder: In generative tasks, P$^3$LM introduces a standard encoder–decoder backbone but uses a multi-stream decoder with order-aware self-attention and "prophet streams" to simultaneously predict multiple future tokens under permutation, with each stream governed by a separate attention mask (Bao et al., 2022).
• Multimodal Extensions: For vision-language (e.g., scene text recognition), PARSeq and HAAP replace or augment random permutations with ViT-based image encoders, and merge permutation-masked language modeling with cross-modal attention decoders (Bautista et al., 2022, Chen et al., 2024).

Training is typically conducted using Adam or AdamW optimizers, large batch sizes, and is often parallelized across available permutations within each batch. Unlike standard MLM, PLM models do not rely on artificial [MASK] tokens (except in MPNet's "mask block" for position compensation (Song et al., 2020)), thereby reducing the pretrain–finetune divergence in token and position distributions.

4. Integration with Masking, Position, and Hybrid Objectives

PLM-based pre-training is often combined or contrasted with traditional masking strategies:

• Masking Integration: PERT, for example, applies whole-word and N-gram masking heuristics to identify candidate shuffle groups, leveraging insights from MacBERT's masking strategies (Cui et al., 2022).
• Hybrid Architectures: MPNet combines permutation prediction with explicit mask token insertion to maintain full positional exposure, thus addressing the position information gap between pre-training and downstream fine-tuning absent in plain XLNet (Song et al., 2020).
• Bidirectionality and Cloze-style Decoding: PLM enables bidirectional context use in both encoding and decoding by permuting the order in which target tokens are predicted (as in PARSeq's unified AR, non-AR, and cloze refinement modes) (Bautista et al., 2022).

PLM-based models can be further enhanced by tuning permutation granularity (token-level > N-gram > word-level), jointly training with both masking and permutation, or dynamically harvesting pre-training checkpoints based on downstream task needs (Cui et al., 2022).

5. Empirical Findings and Downstream Task Impact

Extensive empirical evaluations demonstrate that PLM-induced pre-training yields gains on certain classes of tasks:

• Span Extraction and Tagging: PERT achieves superior results on tasks closely aligned with its permutation prediction objective—e.g., machine reading comprehension, named entity recognition, and word order recovery—often outperforming or matching masked-LM baselines (e.g., +1.5 F1 on CMRC, +0.3–0.5 F1 on NER) (Cui et al., 2022).
• Sequence Generation: P$^3$LM achieves state-of-the-art summarization and question generation performance, outperforming ProphetNet on the GLGE benchmark by +0.9 in average score, and benefiting particularly from multi-stream permutation-based prediction (Bao et al., 2022).
• Scene Text Recognition (STR): PLM models such as PARSeq and HAAP achieve or exceed state-of-the-art STR accuracy (91.9–96.0%), with the ability to flexibly unify AR, NAR, and bidirectional inference within one architecture, as well as reduced FLOPS and lower latency in adaptive permutation regimes (Bautista et al., 2022, Chen et al., 2024).
• Spoken Language Understanding: BERT-PLM, applied to phoneme posterior inputs, results in 10–15% relative error reduction in SLU benchmarks, especially helpful in low-resource or small fine-tuning datasets (Wang et al., 2019).

However, PLM's effectiveness can vary by task. For example, PERT underperforms MLM-based models on classification tasks sensitive to global word order and PERT's classification accuracy can be ~1 point lower than strong MLM baselines (Cui et al., 2022). Empirical analysis indicates benefits saturate with a small number of sampled permutations (K=6 for PARSeq (Bautista et al., 2022), K=2 for HAAP/IPN (Chen et al., 2024)).

6. Limitations, Variants, and Open Problems

Key challenges and limitations include:

• Training–Inference Mismatch: Decoding at inference is almost always performed in L2R order, even when the model is trained on a mixture of permuted orderings (P$^3$LM, XLNet). It is unclear whether optimized ordering at inference can further improve generation (Bao et al., 2022).
• Computational Overhead: Sampling multiple permutations per example increases training cost, though adaptive or learned permutation strategies such as IPN can halve FLOPS and improve training stability (Chen et al., 2024).
• Position Discrepancy: Without explicit position compensation, as in MPNet, the model may suffer from partial exposure to position information during pre-training compared to fine-tuning, which can impair adaptation to downstream tasks reliant on positional accuracy (Song et al., 2020).
• Permutation Distribution: The optimal choice of permutation distribution remains open; plausibly, syntax- or content-informed permutations could yield larger gains than uniform random sampling (Bao et al., 2022).

Active directions for research involve hybridizing PLM with masking objectives, task-adaptive permutation schedules, order regularization, dynamic checkpointing tailored to downstream requirements, and theoretical characterization of expressivity and inductive biases introduced by permutation-based factorization (Cui et al., 2022, Bao et al., 2022).

7. Cross-Modal and Non-Textual Extensions

Recent advances demonstrate PLM's relevance beyond pure text:

• Vision-Context STR: In scene text recognition, internal PLM-based decoders enable robust character sequence modeling, unify context-free and context-aware inference, and provide bidirectional refinement in a single weight-shared transformer (PARSeq, HAAP) (Bautista et al., 2022, Chen et al., 2024).
• Speech: BERT-PLM extends PLM to phoneme posterior sequences, supporting regression objectives and full bidirectional context over acoustic-probability vectors, yielding substantial downstream SLU improvements (Wang et al., 2019).
• Cross-modal Attention: Hierarchical attention mechanisms can be holistically integrated with permutation-induced masking (e.g., CHA in HAAP), enabling tight coupling of context, position, and cross-modal features without costly iterative refinement (Chen et al., 2024).

This suggests PLM is a general strategy applicable to diverse modalities and architectures, capable of improving robustness to context fragmentation and providing a uniform approach to bidirectional dependency modeling.

---

References

• (Cui et al., 2022) PERT: Pre-training BERT with Permuted LLM
• (Chen et al., 2024) HAAP: Vision-context Hierarchical Attention Autoregressive with Adaptive Permutation for Scene Text Recognition
• (Wang et al., 2019) Understanding Semantics from Speech Through Pre-training
• (Bautista et al., 2022) Scene Text Recognition with Permuted Autoregressive Sequence Models
• (Song et al., 2020) MPNet: Masked and Permuted Pre-training for Language Understanding
• (Bao et al., 2022) P$x_1, \ldots, x_T$0LM: Probabilistically Permuted Prophet Language Modeling for Generative Pre-Training