---
title: Pragmatics-Based Perturbations in NLP
url: https://www.emergentmind.com/topics/pragmatics-based-perturbations
type: topic
---

# Pragmatics-Based Perturbations in NLP

Pragmatics-based perturbations are targeted modifications that leverage principles of linguistic pragmatics—such as implicature, context sensitivity, and listener adaptation—to probe or improve communicative robustness and interpretability in language processing agents. These perturbations are central both to theoretical cognitive modeling (e.g., Rational Speech Act models), to the design of adaptive artificial speakers and listeners in referential games, and to empirical evaluations of language models under minimal but pragmatically meaningful changes to their input.

## 1. Foundations in Pragmatic Modeling

Contemporary research roots pragmatics-based perturbations in formal models of communication, particularly the Rational Speech Act (RSA) framework and its variants. The standard RSA model specifies:

- $S_0(u \mid o)$: a literal speaker generating utterances $u$ for object $o$ without pragmatic reasoning.
- $L_1(o \mid u) \propto S_0(u \mid o)P(o)$: a literal listener interpreting $u$, often via Bayes’ rule.
- $S_1(u \mid o^*) \propto \exp[\alpha \log p_i - C(u_i)]$: a pragmatic speaker simulating the likelihood a listener will select the intended referent $o^*$, with $C(u_i)$ a cost function and $p_i = L_1(o^* \mid u_i)$.

Crucially, these models capture the recursive reasoning inherent to pragmatic competence: speakers produce utterances not solely based on truth-conditional semantics, but also on anticipated listener interpretation, accounting for ambiguities and contextual disparities [2203.13685][1810.00367].

## 2. Disparity-Driven Perturbations and the Pragmatic Rational Speaker (PRS)

Disparities between agents—including knowledge limitations, perceptual blind spots, or word-concept mismatches—strongly motivate the development of perturbations that target pragmatic accommodation. The Pragmatic Rational Speaker (PRS) framework extends RSA by introducing a lightweight, adaptive working-memory component to accommodate listener-specific disparities:

$$
S_{PRS}(u \mid o, d) \propto S_1(u \mid o) \cdot \exp[\beta f(d, u)]
$$

Here, $f(d, u)$ is a learned adjustment, implemented as a single-layer MLP over BERT sentence embeddings of $u$ and parameterized by $\delta$. The score $q_i = \text{MLP}_\delta(E_i)$ modulates the ranking of utterances, down-weighting those unsuitable for the listener’s disparity and up-weighting accessible alternatives. Only this adjustment layer is updated during adaptation—enabling rapid, targeted perturbation for new listener types without retraining the underlying RSA machinery [2203.13685].

## 3. Empirical Realization in Referential Games

Empirical studies operationalize pragmatics-based perturbations through controlled referential games. In these settings:

- The speaker must communicate a target (image or concept) to a listener who may possess a knowledge disparity, such as only recognizing hypernyms (“food” vs. “pizza”) or suffering from limited visual detection (e.g., inability to recognize animal entities).
- The PRS adjusts its captions using the disparity-aware perturbation, yielding higher referential-game accuracy (an improvement of 8–12 points over non-adaptive RSA speakers) and efficient adaptation (about 40–50% less training time than retraining all model components).

Table 1 summarizes observed adaptation effects:

| Model         | Accuracy Gain vs. S₁ | Vocabulary Adjustment      |
|---------------|---------------------|---------------------------|
| PRS           | 8–12 points         | Hypernyms ↑, Hyponyms ↓   |
| Word-level PRS| Nearly perfect      | Restricts inaccessible    |

Vocabulary shift plots reveal that, under knowledge disparity, PRS sharply decreases use of listener-inaccessible words in favor of their hypernyms; under perceptual disparity, it avoids inaccessible categories entirely [2203.13685].

## 4. Incremental Pragmatic Perturbations and Sensitivity

Incremental models (e.g., the Incremental Iterated Response, or IR, model) formalize how perturbations at the word level propagate through communicative reasoning. In these frameworks:

- Partial utterances $c = [u_1, \ldots, u_k]$ condition both production and interpretation.
- Perturbations—such as modifying, inserting, or deleting words—can dramatically affect the distribution over intended referents at each incremental step.
- Sensitivity to perturbation is quantitatively assessed, e.g., tracking KL-divergence
  $$
  D_{KL}\big(L_1(\cdot \mid c, u) \| L_1(\cdot \mid c, \text{alt})\big)
  $$
  between posteriors induced by alternative next words.

Concretely, inserting an attribute early (e.g., “red” in “red dress”) can commit the ongoing utterance to a more constrained set of completions, or cause a listener's referent probabilities to spike for alternatives supporting the new evidence. In adversarial or robustness testing tasks, perturbation generation algorithms may greedily maximize probability for a distractor referent by incremental word selection [1810.00367].

## 5. Benchmarking LLM Robustness under Pragmatic Perturbations

Recent benchmarks such as PragWorld systematically assess the fragility of large language models (LLMs) under minimal but pragmatically salient perturbations. The PragWorld suite applies seven classes of minimal edits—including negation, variable substitution, quantity and quantifier changes, variable swaps, logical connective changes, and injection of inconsistent data—to conversational QA contexts.

Empirical results highlight:

- **Robust accuracy**: State-of-the-art models exhibit only 47–68% accuracy on perturbed data (robust accuracy: correct on both original and all altered variants).
- **Perturbation-dependent sensitivity**: Negation and quantity change perturbations yield the most pronounced drops in performance—exceeding 30 percentage points for some LLMs.
- **Layer-wise attribution**: Analyses using direct-effect patching and MLP zero-out ablation reveal that specific transformer layers encode either “useful” (supporting robust reasoning) or “harmful” (injecting spurious correlations) signals. Logical connective perturbations are particularly susceptible to harmful layers.
- **Remediation**: Layer-regularization via useful layer amplification and harmful layer suppression drastically improve model robustness, raising accuracy by up to 28.6 percentage points in small models [2511.13021].

## 6. Concrete Examples and Application Scenarios

Illustrative examples clarify the range and intent of pragmatics-based perturbations:

### Knowledge Disparity
- RSA speaker: “There is a pizza on the table.”
- Listener with hypernymic knowledge: fails to interpret “pizza.”
- PRS (with perturbation): adapts to “There is food on the table,” ensuring comprehension.

### Perceptual Disparity
- RSA speaker: “Look at the cat on the rug.”
- Listener lacks animal detection: yields no match.
- PRS: substitutes with accessible descriptors, e.g., “Look at the bench by the tree.”

### Benchmark Perturbations (PragWorld)
- **Negation swap**: “He didn’t.” → “He did.”
- **Variable substitution**: “I left watermelons” → “I left oranges.”
- **Logical connective change**: “apples and oranges” → “apples or oranges.”

Such perturbations not only probe model robustness but also foreground the context-dependency and listener-centricity that define pragmatic competence.

## 7. Significance, Challenges, and Measurement

Pragmatics-based perturbations serve as both a diagnostic tool and a driver for progress in adaptive communication. Measurement approaches include:

- Referential-game win rate under disparity types.
- BLEU-4 for naturalness in utterance production.
- Efficiency in adaptation (training time for working memory layers).
- Robustness metrics on perturbed benchmarks (original versus perturbed versus robust accuracy).
- Analysis of vocabulary or attribute use shifts.

A central insight is that purely literal models or globally-trained caption generators fail systematically for non-canonical or resource-limited “listeners.” Lightweight, targeted perturbation modules—whether as MLP adjustment layers or intermediate layer regularization—substantially bridge this gap while retaining interpretability (as evidenced by transparent word-frequency shifts and layer-wise effect analyses) [2203.13685][2511.13021].

A plausible implication is that broad adoption of pragmatics-based perturbation strategies will be essential for robust, context-aware, interactive NLP systems—whether in grounded referential contexts, conversational QA, or adversarial robustness testing.

Source: https://www.emergentmind.com/topics/pragmatics-based-perturbations