Word-Replacement Synthesis Procedures

• Word-replacement-based synthesis procedures are systematic methods that generate new sequences by substituting words or symbols using probabilistic, heuristic, or model-informed rules.
• They incorporate diverse techniques such as conditional substitution, rule-based corruption, alignment-based and recompression methods to address challenges across text, code, and materials domains.
• Empirical evaluations show these procedures enhance performance in applications like grammatical error correction, code-switching augmentation, and controlled material synthesis with significant quality improvements.

Word-Replacement-Based Synthesis Procedures

Word-replacement-based synthesis procedures comprise a class of algorithmic and model-based strategies for systematically generating new sequences (textual or symbolic) by substituting words or symbols, often with the goal of improving quality, simulating errors, constructing data augmentation sets, or modeling physical/chemical processes. These procedures span a spectrum from rule-driven transformations in formal language theory and chemical synthesis to sophisticated, model-preference–aware word substitution engines in natural language processing. They are unified by the central mechanism of explicit replacement operations, typically governed by probabilistic, heuristic, or score-based selection criteria.

1. Fundamental Methodologies

At the core of word-replacement-based synthesis is the explicit application of replacement operations at the level of individual elements (words, tokens, or letters), which may be guided by static rules, probabilistic distributions, or learned models.

• Conditional Substitution Distribution: In smart word substitution, the model parameterized by θ defines a conditional distribution $p_\theta(x_n = \tilde{w}_k | X)$ over candidate substitutes at a target position $n$ (Liu et al., 9 Feb 2025).
• Probabilistic Rule-Based Corruption: In grammatical error correction data synthesis, each token is independently corrupted (deleted, inserted, or replaced) with user-specified probability $E_n$, and the relative ratios of error types $(p_m, p_u, p_r)$ are strictly controlled (Yang et al., 2019).
• Alignment and Dictionary-Based Lexical Replacement: For code-switched data augmentation, replacements are made based on bilingual dictionaries, random word alignments, or learned CS-point predictors, each targeting particular classes of switch points in source-target aligned sentence pairs (Hamed et al., 2022).
• Semantic/Phonemic Paraphasia Substitution: In aphasia transcript synthesis, procedural routines implement independent per-token probabilities for replacement with semantic or phonemic neighbors, modulated by severity-scaled parameters (Pittman et al., 28 Oct 2025).
• Symbolic Replacement in Word Equations: The recompression technique for word equations systematically applies pairwise replacement of adjacent symbols (possibly variables), producing a straight-line grammar that encodes all solutions (Jeż, 2012).
• Atomic Site Replacement in Materials Synthesis: In molten salt A-site replacement for MAX phases, specific atom sites in a starting structure are replaced under controlled reaction conditions, achieved by redox exchanges and monitored by stoichiometric ratios (Ding et al., 2019).

These methodologies support algorithmic, statistical, or physically-constrained control of both the position and type of replacement, facilitating a wide array of downstream applications.

2. Loss Functions, Score Metrics, and Statistical Selection

Model-driven or probabilistic word-replacement frameworks increasingly rely on auxiliary score functions and carefully crafted loss objectives to ensure that replacements align with intended semantic or functional outcomes.

• Sentence Quality Scoring with BARTScore: Each candidate substitution yields a new sequence $\tilde{X}_k$, whose quality is quantified by the log-likelihood sum

$$M(\tilde{X}_k) = \sum_{i=1}^N \log p_{\hat{\theta}}(\tilde{x}_i | \tilde{x}_{<i}, X)$$

where $\hat{\theta}$ are BART parameters fixed during training (Liu et al., 9 Feb 2025).

• Empirical P-value for Candidate Superiority: Statistical hypothesis testing is performed by sampling $K_s$ candidate substitutions and computing

$$p_{\tilde{X}_1} = \frac{1}{K_s-1} \sum_{k=2}^{K_s} I[M(\tilde{X}_k) > M(\tilde{X}_1)]$$

to assess whether a given candidate is significantly better than random draws under the model (Liu et al., 9 Feb 2025).

• Preference-Aware Loss Functions: Margin-ranking (MR) loss and average-score (AS) loss are combined:

$$L_{\textrm{MR}} = \sum_{1 \leq j < k \leq K} \max(0, s_j - s_k + \lambda_{jk}), \qquad L_{\textrm{AS}} = -\sum_{k=1}^K \textrm{softmax}(s_k) \cdot M(\tilde{X}_k)$$

with total loss $n$0, enforcing that preference in model scores follows quality ordering (Liu et al., 9 Feb 2025).

• Error Rate and Type Control: Error synthesis procedures for GEC enforce both target error rate $n$1 and relative ratios by direct sampling and post-hoc filtering (Yang et al., 2019), e.g., after generating synthetic pairs, errors are filtered to respect desired distributional constraints.

The integration of such scoring and selection procedures is critical for aligning system outputs with subjective or domain-specific quality criteria in the absence of reliable human-annotated labels.

3. Algorithmic Workflows and Pseudocode Schematics

The operationalization of word-replacement-based synthesis consistently involves algorithmic patterns of candidate generation, ranking or selection, and mutation of the underlying sequence or structure.

High-Level Algorithmic Paradigm (Word Substitution with Model Ranking) (Liu et al., 9 Feb 2025):

1. Input sequence $n$2, model $n$3, scorer $n$4, candidate count $n$5, sampling size $n$6, significance level $n$7, margin $n$8, weighting $n$9.
2. For selected positions $E_n$0:
   • Sample $E_n$1 candidate substitutions; score via $E_n$2; rank candidates.
   • Compute model logits; evaluate loss; update $E_n$3.
3. At inference:
   • Sample $E_n$4 reference candidates; compute $E_n$5 and $E_n$6-value.
   • Replace only if $E_n$7 and $E_n$8.

Procedural Error Synthesis (GEC) (Yang et al., 2019):

• For each token, with probability $E_n$9, corrupt by delete/insert/replace, with probabilities $(p_m, p_u, p_r)$0 respectively.
• Replacement candidates for tokens (words or punctuation) are sampled uniformly from the vocabulary $(p_m, p_u, p_r)$1 or from a punctuational alphabet $(p_m, p_u, p_r)$2.

Aphasia Transcript Paraphasia (Procedural Pseudocode) (Pittman et al., 28 Oct 2025):

$(p_m, p_u, p_r)$7

This generalized workflow underpins diverse applications, with task-specific modifications to candidate pools, replacement types, and contextual or structural constraints.

4. Applications Across Domains

Word-replacement-based synthesis has broad applicability, with instantiations across text NLP, formal language theory, code-switching generation, error modeling, and materials science.

• Lexical Substitution & Sentence Enhancement: Optimizing word choice to improve sentence-level fluency and clarity using masked models and model-based rankers (Liu et al., 9 Feb 2025).
• Code-Switching Data Augmentation: Synthesizing code-switched sentences for low-resource ASR, MT, and ST by lexical replacement at predicted or randomly aligned switch points, leading to significant improvements in downstream task performance (Hamed et al., 2022).
• Grammatical Error Simulation: Generating controlled, realistic error-annotated corpora by systematic deletion, insertion, and replacement, thus facilitating data-efficient GEC training (Yang et al., 2019).
• Aphasia Transcript Generation: Simulating varying degrees of linguistic impairment by paraphasia-replacement and word dropping, with both procedural and LLM-based approaches compared for their impact on transcript metrics (Pittman et al., 28 Oct 2025).
• Symbolic Solution Construction in Equational Systems: The recompression (pair-replacement) algorithm efficiently solves word equations, producing compact descriptions (SLPs) of all possible solutions (Jeż, 2012).
• Atomic Site Substitution in Materials Synthesis: A-site replacement reactions (e.g., Nb₂AlC + 3CuI → Nb₂CuC + AlI₃ + 2Cu) systematically achieve atom-level substitution in MAX phases, verifiable by structural and compositional analysis (Ding et al., 2019).

Tables such as the following summarize representative application modes:

Domain	Replacement Type	Selection Mechanism
NLP - Lexical Substitution	Word for word	Model-based ranking (BARTScore)
Code-Switch Augmentation	Token, segment	Alignment, predictor, dictionary
GEC Synthesis	Word (delete/insert/replace)	Probabilistic, parameterized
Aphasia Simulation	Semantic/phonemic paraphasia	Per-token severity scaling
Word Equations	Symbol pairs	Structural recompression
Materials Synthesis	Atomic species at layer sites	Redox, stoichiometric control

5. Quantitative Results and Empirical Evaluation

The impact of word-replacement-based synthesis procedures is established through both intrinsic and extrinsic metrics, spanning alignment with quality scores, human evaluation, and downstream performance.

• Lexical Substitution (Liu et al., 9 Feb 2025):
  • On the SWS test set, combined MR+AS loss yields median CS (alignment) = 0.983 vs. BERT-SWS/BART-SWS ≈0.93, ABR = 0.90 vs. BERT-SWS ≈0.84, p-value significance rate 0.917 vs. GPT-4o ≈ 0.563.
  • The method outperforms standard MLMs and prompted LLMs on alignment and matches or exceeds on absolute sentence quality.
• Code-Switched Data Augmentation (Hamed et al., 2022):
  • Predictive n–n alignment achieves best reduction in LM perplexity (PPL: 415.1 → 273.4), ASR WER improvement (34.7% → 32.9%), MT BLEU gain (+4.0–5.1), ST BLEU gain (+2.1–2.2).
  • Human-rated "naturalness": Predictive n–n (18.7% of sentences 4–5), substantially higher than dictionary (2.7%).
• GEC Error Synthesis (Yang et al., 2019):
  • Optimal performance at $(p_m, p_u, p_r)$3 error rate; controlling types (e.g. $(p_m, p_u, p_r)$4=1:1:1) improves per-error category F₀.₅.
  • Final system scores F₀.₅=59.79 on CoNLL-2014, rivaling large-scale benchmarks with much less data.
• Aphasia Simulation (Pittman et al., 28 Oct 2025):
  • Procedural method yields linear decline in NDW and word count; Mistral LLM best matches human-like trends; Llama 3.1 overestimates lexical diversity.
• Physical Synthesis - MAX Phases (Ding et al., 2019):
  • XRD, SEM–EDS, and STEM confirm successful replacement; cleavage energy reduction (e.g. Nb₂CuC $(p_m, p_u, p_r)$5 J/m² vs. Nb₂AlC $(p_m, p_u, p_r)$6) supports improved MXene exfoliation potential.

These results collectively demonstrate the effectiveness of principled replacement schemes both for controlled data generation and for practical performance improvement in diverse fields.

6. Extensions, Generalizations, and Limitations

Word-replacement-based synthesis admits a variety of extensions but is also constrained by practical and theoretical factors:

• Adaptive Model-Based Learning: Incorporation of auxiliary scorers obviates reliance on human annotation, allowing fully automatic yet preference-aware substitution procedures (Liu et al., 9 Feb 2025).
• Function Synthesis from Logic Specifications: In logic for word transductions (ℒ𝕋), regular synthesis constructions produce deterministic two-way transducers realizing input-output relations automata-theoretically (Dartois et al., 2017).
• Parameter Optimization and Filtering: Empirical control via filtering (back-translation in GEC) ensures synthetic data respects desired error distributions, avoiding performance degradation from over-representation of unbalanced patterns (Yang et al., 2019).
• LLM-Based vs. Procedural Synthesis: LLM-based approaches yield higher-fidelity, context-sensitive errors, while procedural methods provide strict control and closed-form predictability (Pittman et al., 28 Oct 2025).
• Material Synthesis Scope: Molten salt A-site replacement extends to other late transition metals (Au, Ir, Zn), but partial substitution is sometimes observed due to kinetic or phase-diagram constraints (Ding et al., 2019).
• Complexity and Feasibility: For word equations, recompression techniques provide PSPACE algorithms and exponential solution bounds (Jeż, 2012); in logic-based word transductions, synthesis cost is non-elementary in full generality (Dartois et al., 2017).

Limitations often derive from the combinatorial explosiveness of unconstrained substitution, the challenge of balancing naturalness with control, and in non-ML contexts, phase stability and reaction pathway uncertainty.

---

In summary, word-replacement-based synthesis procedures constitute a foundational toolkit for the algorithmic generation, transformation, and augmentation of textual, symbolic, and material structures. Their diversity and flexibility are matched by rigorous formal underpinnings and empirically validated impact across a variety of research frontiers.