Pragmatics-Based Perturbations in NLP

• Pragmatics-based perturbations are targeted modifications that use linguistic principles like implicature and context sensitivity to enhance model robustness and interpretability.
• They extend formal models, such as the Rational Speech Act framework, by incorporating adaptive components to address listener knowledge disparities.
• Empirical evaluations in referential games demonstrate that these perturbations improve accuracy by 8-12 points while reducing training time significantly.

Pragmatics-based perturbations are targeted modifications that leverage principles of linguistic pragmatics—such as implicature, context sensitivity, and listener adaptation—to probe or improve communicative robustness and interpretability in language processing agents. These perturbations are central both to theoretical cognitive modeling (e.g., Rational Speech Act models), to the design of adaptive artificial speakers and listeners in referential games, and to empirical evaluations of LLMs under minimal but pragmatically meaningful changes to their input.

1. Foundations in Pragmatic Modeling

Contemporary research roots pragmatics-based perturbations in formal models of communication, particularly the Rational Speech Act (RSA) framework and its variants. The standard RSA model specifies:

• $S_0(u \mid o)$: a literal speaker generating utterances $u$ for object $o$ without pragmatic reasoning.
• $L_1(o \mid u) \propto S_0(u \mid o)P(o)$: a literal listener interpreting $u$, often via Bayes’ rule.
• $$S_1(u \mid o^*) \propto \exp[\alpha \log p_i - C(u_i)]$$: a pragmatic speaker simulating the likelihood a listener will select the intended referent $o^*$, with $C(u_i)$ a cost function and $p_i = L_1(o^* \mid u_i)$.

Crucially, these models capture the recursive reasoning inherent to pragmatic competence: speakers produce utterances not solely based on truth-conditional semantics, but also on anticipated listener interpretation, accounting for ambiguities and contextual disparities (Bao et al., 2022, Cohn-Gordon et al., 2018).

2. Disparity-Driven Perturbations and the Pragmatic Rational Speaker (PRS)

Disparities between agents—including knowledge limitations, perceptual blind spots, or word-concept mismatches—strongly motivate the development of perturbations that target pragmatic accommodation. The Pragmatic Rational Speaker (PRS) framework extends RSA by introducing a lightweight, adaptive working-memory component to accommodate listener-specific disparities:

$$S_{PRS}(u \mid o, d) \propto S_1(u \mid o) \cdot \exp[\beta f(d, u)]$$

Here, $f(d, u)$ is a learned adjustment, implemented as a single-layer MLP over BERT sentence embeddings of $u$ and parameterized by $\delta$. The score $q_i = \text{MLP}_\delta(E_i)$ modulates the ranking of utterances, down-weighting those unsuitable for the listener’s disparity and up-weighting accessible alternatives. Only this adjustment layer is updated during adaptation—enabling rapid, targeted perturbation for new listener types without retraining the underlying RSA machinery (Bao et al., 2022).

3. Empirical Realization in Referential Games

Empirical studies operationalize pragmatics-based perturbations through controlled referential games. In these settings:

• The speaker must communicate a target (image or concept) to a listener who may possess a knowledge disparity, such as only recognizing hypernyms (“food” vs. “pizza”) or suffering from limited visual detection (e.g., inability to recognize animal entities).
• The PRS adjusts its captions using the disparity-aware perturbation, yielding higher referential-game accuracy (an improvement of 8–12 points over non-adaptive RSA speakers) and efficient adaptation (about 40–50% less training time than retraining all model components).

Table 1 summarizes observed adaptation effects:

Model	Accuracy Gain vs. S₁	Vocabulary Adjustment
PRS	8–12 points	Hypernyms ↑, Hyponyms ↓
Word-level PRS	Nearly perfect	Restricts inaccessible

Vocabulary shift plots reveal that, under knowledge disparity, PRS sharply decreases use of listener-inaccessible words in favor of their hypernyms; under perceptual disparity, it avoids inaccessible categories entirely (Bao et al., 2022).

4. Incremental Pragmatic Perturbations and Sensitivity

Incremental models (e.g., the Incremental Iterated Response, or IR, model) formalize how perturbations at the word level propagate through communicative reasoning. In these frameworks:

• Partial utterances $c = [u_1, \ldots, u_k]$ condition both production and interpretation.
• Perturbations—such as modifying, inserting, or deleting words—can dramatically affect the distribution over intended referents at each incremental step.
• Sensitivity to perturbation is quantitatively assessed, e.g., tracking KL-divergence

$$D_{KL}\big(L_1(\cdot \mid c, u) \| L_1(\cdot \mid c, \text{alt})\big)$$

between posteriors induced by alternative next words.

Concretely, inserting an attribute early (e.g., “red” in “red dress”) can commit the ongoing utterance to a more constrained set of completions, or cause a listener's referent probabilities to spike for alternatives supporting the new evidence. In adversarial or robustness testing tasks, perturbation generation algorithms may greedily maximize probability for a distractor referent by incremental word selection (Cohn-Gordon et al., 2018).

5. Benchmarking LLM Robustness under Pragmatic Perturbations

Recent benchmarks such as PragWorld systematically assess the fragility of LLMs under minimal but pragmatically salient perturbations. The PragWorld suite applies seven classes of minimal edits—including negation, variable substitution, quantity and quantifier changes, variable swaps, logical connective changes, and injection of inconsistent data—to conversational QA contexts.

Empirical results highlight:

• Robust accuracy: State-of-the-art models exhibit only 47–68% accuracy on perturbed data (robust accuracy: correct on both original and all altered variants).
• Perturbation-dependent sensitivity: Negation and quantity change perturbations yield the most pronounced drops in performance—exceeding 30 percentage points for some LLMs.
• Layer-wise attribution: Analyses using direct-effect patching and MLP zero-out ablation reveal that specific transformer layers encode either “useful” (supporting robust reasoning) or “harmful” (injecting spurious correlations) signals. Logical connective perturbations are particularly susceptible to harmful layers.
• Remediation: Layer-regularization via useful layer amplification and harmful layer suppression drastically improve model robustness, raising accuracy by up to 28.6 percentage points in small models (Vashistha et al., 17 Nov 2025).

6. Concrete Examples and Application Scenarios

Illustrative examples clarify the range and intent of pragmatics-based perturbations:

Knowledge Disparity

• RSA speaker: “There is a pizza on the table.”
• Listener with hypernymic knowledge: fails to interpret “pizza.”
• PRS (with perturbation): adapts to “There is food on the table,” ensuring comprehension.

Perceptual Disparity

• RSA speaker: “Look at the cat on the rug.”
• Listener lacks animal detection: yields no match.
• PRS: substitutes with accessible descriptors, e.g., “Look at the bench by the tree.”

Benchmark Perturbations (PragWorld)

• Negation swap: “He didn’t.” → “He did.”
• Variable substitution: “I left watermelons” → “I left oranges.”
• Logical connective change: “apples and oranges” → “apples or oranges.”

Such perturbations not only probe model robustness but also foreground the context-dependency and listener-centricity that define pragmatic competence.

7. Significance, Challenges, and Measurement

Pragmatics-based perturbations serve as both a diagnostic tool and a driver for progress in adaptive communication. Measurement approaches include:

• Referential-game win rate under disparity types.
• BLEU-4 for naturalness in utterance production.
• Efficiency in adaptation (training time for working memory layers).
• Robustness metrics on perturbed benchmarks (original versus perturbed versus robust accuracy).
• Analysis of vocabulary or attribute use shifts.

A central insight is that purely literal models or globally-trained caption generators fail systematically for non-canonical or resource-limited “listeners.” Lightweight, targeted perturbation modules—whether as MLP adjustment layers or intermediate layer regularization—substantially bridge this gap while retaining interpretability (as evidenced by transparent word-frequency shifts and layer-wise effect analyses) (Bao et al., 2022, Vashistha et al., 17 Nov 2025).

A plausible implication is that broad adoption of pragmatics-based perturbation strategies will be essential for robust, context-aware, interactive NLP systems—whether in grounded referential contexts, conversational QA, or adversarial robustness testing.