Semantic Reordering: Principles & Applications

• Semantic Reordering (SR) is a framework that rearranges linguistic units based on semantic relevance to enhance coherence and model performance.
• SR techniques span word ordering, sentence selection, intermediate representation reordering, and even patch sequencing in computational pathology.
• Applications of SR in summarization, semantic parsing, machine translation, and pathology demonstrate measurable improvements in fluency, interpretability, and downstream task metrics.

Semantic Reordering (SR) denotes a family of operations in which the order of linguistic or structured units is changed so that the resulting sequence better matches semantic salience, discourse coherence, monotonic alignments, or learned semantic neighborhoods. In the cited literature, SR is instantiated at multiple granularities: word ordering or linearization, sentence ordering, sentence selection and reordering for abstractive summarization, reordering of intermediate meaning representations in semantic parsing, syntactic preordering for paraphrase generation and phrase-based machine translation, and sequence reordering of patch embeddings in whole slide image (WSI) classification (Schmaltz et al., 2016, Logeswaran et al., 2016, Salemi et al., 2021, Cazzaro et al., 2022, Yang et al., 2024). The term therefore does not name a single algorithm; it names a problem class and a design principle in which ordering is treated as a learnable, semantically consequential variable.

1. Scope and representational units

Across the literature, SR is applied to different units and with different target criteria. In some settings, the input is an unordered collection that must be linearized into a coherent sequence. In others, the input is already ordered, but the model deliberately perturbs or reconstructs order during pre-training or decoding. In still others, SR is used to expose latent alignments, reduce effective propagation distance, or improve interpretability.

Unit being reordered	Representative mechanism	Typical objective
Words or phrases	Beam search over permutations; language-model scoring	Recover fluent word order
Sentences	Set-to-sequence decoding; text-to-marker generation; coherence graphs	Recover coherent document order
Summary sentences	Semantic sentence selection plus modified sentence reordering	Form pseudo-summaries for pre-training
Meaning representations	Monotonic translation plus reordering; SSQL; separable permutations	Improve compositional generalization
WSI patch embeddings	Sequence reordering; semantic clustering and reversible permutation	Improve global modeling and interpretability

For word ordering, the task is also called linearization and consists of recovering the order of a sentence from an unordered collection of words or phrases (Schmaltz et al., 2016). For sentence ordering, the task is to organize a shuffled set of sentences into a coherent text, a setting used to evaluate understanding of causal and temporal relations (Chowdhury et al., 2021). In abstractive summarization, ARMAN uses salient sentence selection and modified sentence reordering to create pseudo-summaries during pre-training (Salemi et al., 2021). In semantic parsing, TPOL explicitly decomposes parsing into monotonic translation and a reordering step, while other work models reordering as a latent structured permutation or uses an intermediate representation such as SSQL (Cazzaro et al., 2022, Wang et al., 2021, Li et al., 2023). In computational pathology, sequence reordering reorganizes WSI patches so that semantically similar or distributionally informative instances become easier to process with state space models (Yang et al., 2024, Gan et al., 30 Aug 2025).

2. Core formulations and algorithmic patterns

A central SR pattern is to score candidate units by semantic relevance and then reorder only the subset judged most informative. ARMAN formalizes this through Top Sentence Selection (TSS), where each sentence $s_i$ is scored against the document minus $s_i$ using a FastText-based BERTScore variant $F1_{FT}$. The paper defines

$$R_{FT} =\frac{1}{|x|} \sum_{x_i \in x} \max_{\hat{x}_j \in \hat{x}} x_i^\top \hat{x}_j,$$

$$P_{FT} =\frac{1}{|\hat{x}|} \sum_{\hat{x}_j \in \hat{x}} \max_{x_i \in x} x_i^\top \hat{x}_j,$$

$$F1_{FT} = 2 \frac{P_{FT} \cdot R_{FT}}{P_{FT} + R_{FT}},$$

where $x_i$ and $\hat{x}_j$ are FastText embedding vectors; the top $m$ sentences under this score become the pseudo-summary (Salemi et al., 2021).

A second pattern is search over permutations under a sequence model. In word ordering without syntax, the score of a partial hypothesis $y$ is

$s_i$0

with a future-cost estimate

$s_i$1

and beam search is used because exhaustive search over all permutations is intractable (Schmaltz et al., 2016). This formulation treats SR as a decoding problem rather than as a syntactic derivation.

A third pattern makes the permutation itself a structured latent variable. In “Structured Reordering for Modeling Latent Alignments in Sequence Transduction,” the alignment is decomposed as $s_i$2, where $s_i$3 is a permutation matrix and $s_i$4 is a monotonic alignment after reordering. The model restricts $s_i$5 to separable permutations and computes the expected permutation matrix

$s_i$6

by dynamic programming (Wang et al., 2021). “Compositional Generalisation with Structured Reordering and Fertility Layers” uses the same reordering step after a fertility expansion, again computing an expected permutation matrix and composing the two structural operations end-to-end (Lindemann et al., 2022).

A fourth pattern reorders by learned semantic labels. In SemaMIL, each patch embedding $s_i$7 is mapped by

$s_i$8

then

$s_i$9

and the sequence is reordered by sorting the semantic labels,

$F1_{FT}$0

After processing, outputs are restored by the inverse permutation $F1_{FT}$1 (Gan et al., 30 Aug 2025). This makes SR a reversible permutation layer within the model pipeline.

3. Summarization and paraphrase generation

In abstractive summarization, SR is used as a pre-training signal that teaches both content selection and order recovery. ARMAN is a Transformer-based encoder-decoder model pre-trained with three novel objectives. Its TSS-Modified Sentence Reordering (MSR) objective uses two regimes: in 90% of pre-training cases, salient sentences are masked for reconstruction, and in the remaining 10%, all sentences in the document are shuffled. The model must then reorder the sentences and keep only the top 30% most important sentences, based on the semantic score, in their original order (Salemi et al., 2021). This objective is explicitly designed to teach what content to select and how to order it.

The reported results are unusually broad for a pre-training scheme centered on SR. On six downstream Persian summarization tasks, ARMAN achieves state-of-the-art performance on all six tasks measured by ROUGE and BERTScore, and ARMAN(MSR) consistently outperformed prior models such as PEGASUS, ParsBERT, and mT5. On PN-Summary, for example, PEGASUS scores $F1_{FT}$2, ARMAN(SS) scores $F1_{FT}$3, and ARMAN(MSR) scores $F1_{FT}$4 for ROUGE-1/2/L. Human evaluation with 10 raters on 30 summaries ranked 50.33% of ARMAN(MSR) summaries best, compared to 38.33% for ARMAN(SS-80) and 31% for PEGASUS, with $F1_{FT}$5 under a Student t-test on ranking scores (Salemi et al., 2021). The same paper reports that ARMAN(MSR) summaries were more abstractive, with lower coverage and density than PEGASUS, and delivered higher F1-target and Precision-source factual consistency scores.

Document-level paraphrase generation also makes sentence reordering a first-class operation. CoRPG builds a coherence relationship graph $F1_{FT}$6 over sentences using ALBERT’s Sentence Order Prediction (SOP) probability $F1_{FT}$7, thresholded at $F1_{FT}$8, and applies a Graph GRU to derive coherence-aware sentence representations (Lin et al., 2021). Because the input is presented as an unordered set of sentences and no positional embeddings are used, the decoder must reconstruct a coherent order using semantic and coherence-aware information. CoRPG outperforms several strong baseline models on BERTScore and diversity scores, and human evaluation shows more diversity and semantic preservation. The detailed results report self-TER of 60.00, COH of 79.21, BERTScore of 70.52, and mean human scores of 4.12 for diversity, 3.96 for relevancy, and 3.86 for coherence; removing the Graph GRU drops COH to 57.28 (Lin et al., 2021).

Controlled paraphrase generation uses preordering more explicitly. “Neural Syntactic Preordering for Controlled Paraphrase Generation” introduces a two-stage framework, Sow and Reap. Sow produces sentence-specific syntactic rearrangements, and Reap injects reordering embeddings into the encoder output,

$F1_{FT}$9

so that the decoder attends to source words in a particular order (Goyal et al., 2020). The automatic evaluation shows Sow-Reap attains Oracle BLEU 30.9, ROUGE-L 61.7, % Rejected 15.9, self-BLEU 38.0, and self-WER 57.9, compared with a regular Transformer at Oracle BLEU 32.8, ROUGE-L 63.3, % Rejected 12.7, self-BLEU 50.7, and self-WER 35.4 (Goyal et al., 2020). The intended trade-off is structurally diverse paraphrases without major loss in quality.

4. Word ordering, sentence ordering, and coherence modeling

Early neural and language-model work on ordering established that strong ordering performance does not require explicit syntax. “Word Ordering Without Syntax” shows that a simple n-gram LM with a heuristic surpasses the syntax-based model ZGen by almost 5 BLEU points on the Penn Treebank test set, and that an LSTM LM exceeds the syntax-based model by 11.5 BLEU points on the Words+BNPs task. The final reported BLEU scores are 49.4 for ZGen, 54.3 for NGram-64, 60.9 for LSTM-64, 63.2 for LSTM-512, and 65.8 for LSTM-512+gw on Words+BNPs (Schmaltz et al., 2016). The paper further reports that language-model outputs achieve even higher unlabeled attachment scores than syntax-based linearizers after parsing, indicating syntactically plausible orderings.

Sentence ordering work extends the same question to discourse structure. “Sentence Ordering and Coherence Modeling using Recurrent Neural Networks” proposes an end-to-end unsupervised set-to-sequence model with a sentence encoder, an order-invariant set encoder, and a pointer-network decoder. It strongly outperforms prior methods in order discrimination and in ordering abstracts from scientific articles, reaching mean accuracies of 0.944 on Accidents and 0.997 on Earthquakes, and 58.06% accuracy with $$R_{FT} =\frac{1}{|x|} \sum_{x_i \in x} \max_{\hat{x}_j \in \hat{x}} x_i^\top \hat{x}_j,$$0 on ACL abstracts (Logeswaran et al., 2016). The paper also reports that sentence representations learned from ordering capture high-level logical structure.

A later formulation treats sentence ordering as conditional text-to-marker generation. Reorder-BART takes a set of shuffled sentences with sentence-specific markers as input and generates a sequence of position markers; the sentences are then reordered according to those markers (Chowdhury et al., 2021). Re-BART achieves the state-of-the-art performance across 7 datasets in Perfect Match Ratio (PMR) and Kendall’s tau, and zero-shot evaluation shows that the model generalizes well across other datasets. The paper attributes part of this to BART’s denoising pre-training, which includes a sentence permutation objective (Chowdhury et al., 2021).

The relation between architecture and order sensitivity is more contested at the word level. “Assessing the Ability of Self-Attention Networks to Learn Word Order” introduces a word reordering detection task in which one word is moved and the model must identify the original and inserted positions. When trained directly for WRD, SAN is inferior to RNN and DiSAN, with Insert/Original/Both scores of 73.2/66.0/60.1 for SAN, compared with 78.4/73.4/68.2 for RNN and 79.6/70.1/68.0 for DiSAN. But when encoders are pre-trained on machine translation and then probed, SAN exceeds RNN in order awareness, reaching WRD scores of 41.6 on En→De and 32.8 on En→Ja, compared with 33.9 and 29.0 for RNN (Yang et al., 2019). The paper’s explicit conclusion is that learning objectives matter more than architecture in downstream tasks such as machine translation.

The task dependence of order sensitivity is reinforced by “Word Order’s Impacts: Insights from Reordering and Generation Analysis.” On order reconstruction, the score drop from best to worst disruption is 19% for RTP, 13% for CS, 27% for BF, and 97% for Loop. In continuing generation, the changes are -13% for RTP, +0.1% for CS, -35% for BF, and -26% for Loop (Zhao et al., 2024). The paper concludes that ChatGPT takes word order information into account during inference, thereby challenging the hypothesis that models do not rely on word order, but it states that the results cannot support or negate the redundancy relations between word order and lexical semantics (Zhao et al., 2024).

5. Semantic parsing and machine translation

In semantic parsing, SR is often used to isolate the combinatorial difficulty of output order from the lexical difficulty of translation. TPOL (“Translate First Reorder Later”) decomposes parsing into a Translator and a Reorderer. The Translator maps natural language $$R_{FT} =\frac{1}{|x|} \sum_{x_i \in x} \max_{\hat{x}_j \in \hat{x}} x_i^\top \hat{x}_j,$$1 to a monotonic meaning representation $$R_{FT} =\frac{1}{|x|} \sum_{x_i \in x} \max_{\hat{x}_j \in \hat{x}} x_i^\top \hat{x}_j,$$2, and the Reorderer maps $$R_{FT} =\frac{1}{|x|} \sum_{x_i \in x} \max_{\hat{x}_j \in \hat{x}} x_i^\top \hat{x}_j,$$3 to the target meaning representation $$R_{FT} =\frac{1}{|x|} \sum_{x_i \in x} \max_{\hat{x}_j \in \hat{x}} x_i^\top \hat{x}_j,$$4 (Cazzaro et al., 2022). The key observation is that many alignments between natural language and meaning representations are monotonic, and that non-monotonic cases can often be handled by permuting the target to induce monotonicity. On the compositional query split of GEOALIGNED in English, TPOL attains 87.8% exact match, compared with 69.4 for mBART, 74.6 for SPANBASED, and 84.1 for LEAR; with automatic alignments, TPOL still reaches 79% on the English Query split, compared with 43.2% for REMOTO and 51.8% for SPANBASED (Cazzaro et al., 2022).

T5-SR uses the term SR for a seq2seq-oriented decoding strategy in text-to-SQL. It introduces SSQL, an intermediate representation that retains semantic content while deferring schema-specific JOIN construction to heuristic post-processing, and combines beam search with a score re-estimator through

$$R_{FT} =\frac{1}{|x|} \sum_{x_i \in x} \max_{\hat{x}_j \in \hat{x}} x_i^\top \hat{x}_j,$$5

with $$R_{FT} =\frac{1}{|x|} \sum_{x_i \in x} \max_{\hat{x}_j \in \hat{x}} x_i^\top \hat{x}_j,$$6 by default (Li et al., 2023). On Spider, T5-SR-3b reaches QM/EM of 77.2/79.9 on dev and 72.4/75.2 on test, compared with 71.5/74.4 and 68.0/70.1 for T5-3b, and 75.5/79.3 and 71.9/75.1 for T5-PICARD (Li et al., 2023). The reported SSQL representation also reduces target sequence length from 17.5 to 13.6 tokens.

Two related lines of work make the permutation structure itself explicit. “Structured Reordering for Modeling Latent Alignments in Sequence Transduction” introduces a reorder-first align-later framework with a neural reordering module over separable permutations and exact marginal inference by dynamic programming (Wang et al., 2021). “Compositional Generalisation with Structured Reordering and Fertility Layers” composes a fertility step with a structured reordering step, again using expected alignments and permutations (Lindemann et al., 2022). On GeoQuery length split, the latter reports 68.8% accuracy for the full model, compared with 19.3% for BART-base, 24.3% for a seq2seq baseline, 41.4% for Neural Span, and 63.6% for Neural QCFG; on ATIS length split it reports 35.4% versus 19.0% for BART-base (Lindemann et al., 2022). On EN-JA length split, Hard-ReMoto reaches BLEU 28.7, compared with 25.3 for seq2seq, 26.5 for SSNT, and 27.1 for a local reordering model (Wang et al., 2021).

In machine translation, preordering and explicit reordering have a longer history. “Reordering rules for English-Hindi SMT” treats reordering as a preprocessing stage in phrase-based SMT, applying syntactic transformations to parsed English sentences so that their order more closely matches Hindi syntax. On the reported setup, BLEU improves from 21.55 for the baseline and 22.19 for limited reordering to 24.47 for the proposed approach; NIST improves from 5.72 to 5.88, while mWER decreases from 68.08 to 64.71 and mPER from 45.54 to 43.89 (Patel et al., 2016). Phrase extraction analysis shows that distinct 4-word phrases increase from 103,851 to 141,535.

“Continuous Space Reordering Models for Phrase-based MT” replaces count-based lexicalized reordering and operation sequence models with feed-forward neural models over source-target operation histories (Durrani et al., 2018). The paper reports BLEU improvements of up to 0.6 for German→English and 0.5 for English→German over the baseline phrase-based systems, and improvements over models trained with POS tags or word clusters. At the same time, when explicit reordering triggers are injected into encoder-decoder NMT, the experiments do not yield improvements over baseline NMT systems (Durrani et al., 2018). This contrast recurs elsewhere: explicit SR is often helpful when it addresses sparse alignments or compositionality, but not uniformly beneficial when the downstream model already internalizes long-range reordering.

6. Sequence reordering in computational pathology

In computational pathology, SR is adapted from linguistic sequencing to long-sequence modeling over patch embeddings. MambaMIL introduces Sequence Reordering Mamba (SR-Mamba), which processes both the original and a reordered sequence of instances in parallel, using Mamba blocks in both branches and aggregating the outputs with gating and residual connections (Yang et al., 2024). The motivation is that WSI patches are spatially scattered, positive patches are rare, and default scanning order may obscure global contextual relationships. The reported ablation over seven survival datasets gives mean scores of 0.662 for Mamba, 0.665 for Bi-Mamba, and 0.680 for SR-Mamba with ResNet-50 features (Yang et al., 2024). The paper also states that SR-Mamba-based MambaMIL is more robust to overfitting than Transformer-based TransMIL.

SemaMIL makes the semantic nature of the permutation more explicit. It introduces Semantic Reordering (SR), which clusters and arranges semantically similar patches in sequence through a reversible permutation, together with a Semantic-guided Retrieval State Space Module (SRSM) (Gan et al., 30 Aug 2025). In the ablation study, adding SR alone improves EBRAINS ACC from 0.722 to 0.740 and BRACS ACC from 0.561 to 0.584; the full SR+SRSM system reaches EBRAINS AUC/ACC of 0.984/0.751 and BRACS AUC/ACC of 0.861/0.603 (Gan et al., 30 Aug 2025). The same paper reports that SemaMIL is more efficient than all attention/transformer and prior SSM baselines, with 0.464M parameters and 0.248B FLOPs. Because the permutation is reversible, outputs can be mapped back to original spatial positions, a property used to support interpretability (Gan et al., 30 Aug 2025).

These pathology models extend SR beyond language in a narrow sense, but not beyond sequence semantics. Their core claim is that arbitrary order can destroy meaningful structure and that semantically informed order can shorten interaction paths, improve global modeling, and make outputs easier to map back to biologically meaningful regions (Yang et al., 2024, Gan et al., 30 Aug 2025). This suggests that SR is best understood as an inductive bias for sequence organization rather than as a domain-specific NLP trick.

7. Evaluation regimes, recurring findings, and controversies

The evaluation of SR is heterogeneous because the task family is heterogeneous. Word and phrase ordering use BLEU, sometimes together with syntactic plausibility analyses such as UAS (Schmaltz et al., 2016). Sentence ordering uses PMR, Kendall’s tau, accuracy, and order discrimination accuracy (Chowdhury et al., 2021, Logeswaran et al., 2016). Summarization uses ROUGE, BERTScore, human rankings, abstractiveness measures such as coverage and density, and factual consistency indicators such as F1-target and Precision-source (Salemi et al., 2021). Document-level paraphrase work adds diversity measures such as self-TER and self-WER, coherence measures such as COH and COH-p, and human judgments of diversity, relevancy, and coherence (Lin et al., 2021). Semantic parsing reports exact match, QM, EM, and split-specific compositional generalization results (Cazzaro et al., 2022, Li et al., 2023). WSI classification reports AUC, ACC, C-Index, FLOPs, and parameter counts (Yang et al., 2024, Gan et al., 30 Aug 2025).

Several recurring findings cut across these settings. First, explicit syntax is not uniformly required for strong reordering performance: surface-level n-gram and LSTM LLMs can outperform syntax-based word-ordering systems (Schmaltz et al., 2016). Second, architecture alone does not determine order sensitivity: SANs look weaker than RNNs on a direct word reordering detection objective, yet retain more word-order information than RNNs when trained for machine translation (Yang et al., 2019). Third, reordering is strongly task-dependent: order disruption causes only small drops on some datasets but very large drops on BF and especially Loop, where the reported worst-case reconstruction degradation reaches 97% (Zhao et al., 2024). Fourth, explicit SR can improve sparse or structured transduction settings such as phrase-based SMT and compositional semantic parsing, yet the same kind of explicit reordering signal may fail to improve or may even hurt conventional NMT (Patel et al., 2016, Durrani et al., 2018).

The major misconception addressed in this literature is the claim that order is either redundant with lexical semantics or largely ignored by modern models. The available evidence is more specific. Some tasks can be handled with limited reliance on order, some objectives cause models to internalize order more strongly than others, and some domains require explicit reordering modules for reliable generalization (Yang et al., 2019, Zhao et al., 2024, Cazzaro et al., 2022). A plausible implication is that SR should be viewed neither as a universally necessary architectural component nor as a dispensable preprocessing detail. In the studies reviewed here, it functions as a controllable mechanism for exposing salience, enforcing coherence, exploiting monotonicity, or organizing long sequences so that downstream models can learn the right dependencies.