---
title: 'WUGNECTIVES: A Benchmark for Discourse Inference'
url: https://www.emergentmind.com/topics/wugnectives
type: topic
---

# WUGNECTIVES: A Benchmark for Discourse Inference

WUGNECTIVES is primarily a benchmark for testing whether discourse connectives can inform language models about the world independently of prior factual knowledge. It reverses the usual discourse task: rather than predicting which connective best links two known arguments, it asks whether a connective linking unknown entities allows a model to infer properties of those entities. To isolate connective meaning as the critical signal, the benchmark replaces familiar referents with novel entities such as **Wugs**, **Daxes**, **Wugfest**, and **Wugsville** [2510.09556].

## 1. Conceptual basis

WUGNECTIVES studies the inverse problem of connective prediction. In the standard formulation, a model uses world knowledge about two arguments to choose a connective such as *because* or *however*. WUGNECTIVES instead asks what a model can infer about the arguments given the connective. The benchmark therefore targets the functional meaning of discourse connectives rather than stored encyclopedic knowledge about real entities [2510.09556].

The framework is grounded in Penn Discourse Treebank senses. It covers **Expansion.Instantiation**, **Contingency.Cause**, **Comparison.Concession**, and **Temporal.Asynchronous**. Within this design, concessive connectives are especially important because they require denial or cancellation of a causal expectation raised by one argument using the other. A representative example is: *“Although I hate leafy vegetables, I prefer wugs to daxes.”* In this case, the concession blocks the naive causal inference that the preferred item should instantiate the disliked property [2510.09556].

The use of nonce entities is methodologically central. Bare plurals, events, and locations are constructed so that the model has no entrenched prior knowledge to exploit. This makes the benchmark a controlled probe of whether connective semantics alone can support inference. A plausible implication is that WUGNECTIVES belongs to the broader family of “wug” evaluations, but it relocates the wug paradigm from morphology to discourse-semantic inference [2510.09556].

## 2. Dataset architecture and connective taxonomy

The benchmark comprises **8,880 stimuli** created by embedding a base set of utterances in **12 prompt templates**. The paper states a base set of **740 utterances** embedded across **12 templates** for **8,880 total stimuli**, and it organizes the stimuli around three inference families: **Instantiation**, **Preference**, and **Temporal** [2510.09556].

The connective inventory spans **41 unique connective forms** across **7 third-level senses**. The reported sense inventory is as follows.

| PDTB sense | Count | Representative connectives |
|---|---:|---|
| Expansion.Instantiation.Arg2-as-instance | 100 | for example, for instance, in particular, specifically, such as |
| Contingency.Cause.Reason | 160 | as, because, for, since |
| Contingency.Cause.Result | 240 | as a result, so, therefore, thus |
| Comparison.Concession.Arg1-as-denier | 98 | although, as much as, even though, though |
| Comparison.Concession.Arg2-as-denier | 302 | although, but, even though, however, yet, nevertheless, though, while, despite that |
| Temporal.Asynchronous.Precedence | 110 | afterwards, before, even before, finally, hence, later, next, so, subsequently, then, consequently |
| Temporal.Asynchronous.Succession | 130 | after, as soon as, once, previously, thereafter, eventually, earlier, since |

The nonce inventory is carefully balanced. Bare plurals include **Wugs, Daxes, Feps, Geks, Blickets**; events include **Wugfest, Daxday, Fepfestival, Gextravaganza, Blicketbash**; locations include **Wugsville, Daxburgh, Fepopolis, Gektopia, Blicketland**. Pairwise use of nonce entities avoids single-entity co-occurrence biases, and nonce pairs are counterbalanced across stimuli so a bias toward any surface form yields chance performance. Temporal stimuli vary neutral event verbs such as **happened**, **took place**, and **occurred** to reduce superficial cues [2510.09556].

Preference stimuli manipulate both property type and polarity. The benchmark uses **10 binary-style properties**: five for bare plurals and five for locations. Polarity is manipulated by swapping **love** and **hate**, so that the same connective class can license either positive entailments or their negations. Concessive and causal connectives are also used in fronted and non-fronted positions and in either the preference clause or the property clause, making clause-order variation part of the test rather than an uncontrolled artifact [2510.09556].

## 3. Task formulation, evaluated models, and core results

Each stimulus contains a premise and an explicit question. Models answer **Yes/No** for Instantiation and Preference, and one of the two event names for Temporal. The primary metric is accuracy, with chance at **50%** for all tasks:

$$
\mathrm{Acc} = \frac{1}{N}\sum_{i=1}^{N}\mathbb{1}[\hat{y}_i = y_i].
$$

The paper reports **95% confidence intervals across prompt templates** to capture sensitivity to wording [2510.09556].

The evaluation covers **17 open-source decoder-only LMs**: **Llama 3.1 8B** in base and instruct form; **Qwen 2.5** at **0.5B, 1.5B, 3B, 7B, 14B**, each in base and instruct variants; **OLMo-2** at **1B** and **7B**, again in base and instruct variants; and **Qwen2.5-DS 14B**, a reasoning-tuned model produced by fine-tuning **Qwen-2.5-14B** on reasoning traces generated by the larger **DeepSeek R1** model [2510.09556].

For base and instruction-tuned models, responses are extracted from token probabilities using **minicons**, ranking surface-form variants of **Yes** and **No**, or the two event names, by conditional probability. For **Qwen2.5-DS**, the model is prompted to produce a rationale followed by a final answer in **\(\boxed{\cdot}\)** format, with an algorithmic post-processor used when the boxed answer is absent. Decoding for the reasoning model uses **temperature 0.6** [2510.09556].

The principal empirical findings are sharply differentiated by connective type. All models systematically struggle on **Comparison.Concession**. By contrast, **Contingency.Cause** is considerably easier: **thirteen of the seventeen models exceed chance by at least 5 percentage points for both Contingency.Cause senses**, and **Qwen2.5-DS** exceeds **75% accuracy** for both **Reason** and **Result**. **Instantiation** is generally near chance except for larger **Qwen** models at **\(\ge 7\)B**, with **Qwen2.5-DS** reaching **82% accuracy**. The reasoning-tuned model is the strongest overall across senses except concession [2510.09556].

The regression analysis is equally specific. For the **Qwen 14B** variants, reasoning-oriented tuning yields a positive effect relative to both base and instruct training: **\(\beta_{\text{reasoning vs base}} = 0.06, p < .001\)** and **\(\beta_{\text{reasoning vs instruct}} = 0.08, p < .001\)**. By contrast, there is **no clear global effect of scale**, with **\(\beta_{\text{params}} = 0.008, p = .10\)**, and **instruction-tuning** shows scant improvements over base, with **\(\beta_{\text{instruct}} = -0.004, p = .826\)**. Connective frequency in the **3T-token Dolma corpus** is not correlated with per-connective accuracy; the reported \(r^2\) values are non-significant for all models [2510.09556].

## 4. Diagnostic interpretation and recurrent failure modes

The most prominent failure mode is concessive semantics. Models often behave as if concession were ordinary contrast or even causal reinforcement, failing to recognize that connectives such as **although** and **even though** cancel a straightforward causal entailment. This pattern holds for both **Arg1-as-denier** and **Arg2-as-denier** senses and is exacerbated by fronted concessive clauses. The benchmark therefore isolates a pragmatic weakness that is not reducible to entity familiarity, since the entities are novel by construction [2510.09556].

Temporal performance requires more careful interpretation. The paper introduces two diagnostic heuristics: **Choose-first**, which selects the first event mentioned, and **Choose-recent**, which selects the most recent event in linear order. On **Precedence**, **Choose-first** attains **92.5%** accuracy and **Choose-recent** **7.5%**; on **Succession**, the same heuristics yield **56.4%** and **43.6%** respectively. This shows that much of the Precedence subset can be solved by a superficial linear-order heuristic because most Precedence connectives are not frontable, whereas Succession is harder to game [2510.09556].

Two common misunderstandings are therefore not supported by the results. First, strong performance on some temporal items does not by itself demonstrate robust discourse reasoning, because shallow positional strategies already perform well on part of the benchmark. Second, larger parameter counts and generic instruction tuning are not sufficient explanations of success, since the reported mixed-effects analyses do not show a clear global scale effect and show scant improvements for instruction tuning [2510.09556].

The benchmark also states several limits of interpretation. Even with nonce entities, the stimuli include lexical material such as **prefer**, **love**, and **hate**, so fully isolating connective impact is intractable. Novel words are also not entirely tokenization-free, because subtokens already exist in model vocabularies. Finally, the comparison between reasoning models and base or instruction-tuned models is not perfectly symmetric, because the former generate rationales while the latter are scored through constrained probability extraction [2510.09556].

## 5. Alternative use of the label in Word Usage Graph research

A distinct line of work associates “WUGNECTIVES” with **Word Usage Graphs (WUGs)** rather than discourse connectives. One paper explicitly states that it **does not use the term “WUGNECTIVES”**, but also states that its main contribution is exactly what the term suggests: generating concise, human-readable cluster labels that serve as sense definitions for clusters in WUGs [2403.18024].

In that setting, a **Word Usage Graph** is a weighted, undirected graph built for a single target word, whose nodes are individual usage instances in context and whose edge weights reflect semantic proximity from human annotations of contextual similarity. After annotation, nodes are clustered automatically, and each cluster roughly corresponds to a word sense or usage type. A **Diachronic WUG (DWUG)** contains usages from multiple time periods, allowing clusters to be tracked over time [2403.18024].

The labeling pipeline is called **DefGen**. It fine-tunes **mT0-xl (3.7B parameters)**, a multilingual variant of **Flan-T5**, to produce contextualized dictionary-like definitions from usage sentences plus a simple instruction prompt. Definitions are generated per usage, embedded with multilingual **SBERT** using **distiluse-base-multilingual-cased-v1**, and aggregated by selecting the definition closest to the cluster centroid in cosine space. Only clusters with at least **three usage instances** are labeled [2403.18024].

Human evaluation uses a **guess-the-cluster-by-definition** task. On the **English DWUG**, **DefGen** achieves **69.17% accuracy**, compared with **21.67%** for **Lesk** and **50.00%** for **GlossReader**. On the **German DWUG**, **DefGen** reaches **57.89%** accuracy against **53.68%** for **GlossReader**. On **Russian RuDSI** with English definitions, **DefGen** attains **71.79%** compared with **64.10%** for **GlossReader**. The paper reports **no significance tests or confidence intervals**. In this usage, “WUGNECTIVES” denotes interpretable sense labels rather than a discourse benchmark [2403.18024].

## 6. Place within the broader wug-testing tradition

WUGNECTIVES is part of a wider methodological tradition in which models are evaluated on generalization to novel forms or entities under controlled conditions. In speech, one precursor models unsupervised learning of an identity-based pattern—partial CV reduplication—from raw continuous data using **ciwGAN**, and introduces an explicit wug-testing protocol for CNNs trained on speech. In that setting, latent manipulation with **\(c = [0, 7.25]\)** and **\(z90 = 7\)** yields **33/100** correctly reduplicated **[s]-initial** forms, whereas a bare **WaveGAN** reaches only **1/100** or **4/100** under analogous strong manipulations. The paper explicitly connects these findings to WUGNECTIVES by describing the approach as a principled acoustic wug-testing harness for identity constraints in continuous speech [2009.06110].

In morphological inflection, another line of work shows that a standard character-level Transformer almost completely fails under wug-test-like conditions when training and test lemmata do not overlap. Its proposed remedy is substring-based hallucination that injects copying bias through **2-, 3-, and 4-gram** dummy stems. The reported **+hall-2k-substr** results include **68.50** for **Finnish**, **62.62** for **German**, **69.26** for **Russian**, and **91.94** for **Spanish**, outperforming earlier augmentation methods in most languages [2104.06483].

Large language models have also been examined with nonce-based morphological tests. A multilingual study of **gpt-3.5-turbo-0613** in **English, German, Tamil, and Turkish** reports that ChatGPT massively underperforms purpose-built systems, especially in English, and identifies recurrent error types such as **real word bias** and **productivity amplification**. A later multilingual pluralization study in **Catalan, English, Greek, and Spanish** concludes that model accuracy is **“resource-sensitive but language-blind”**: accuracy patterns align more closely with **community size and data availability** than with **structural complexity**, even though the models can reach human-like accuracy once **BERT** is excluded [2310.15113] [2510.12463].

This suggests a shared methodological logic across otherwise different tasks. Novel items, leakage control, and carefully constrained prompts are used to separate genuine generalization from memorization or lexical familiarity. WUGNECTIVES extends that logic from morphology and speech to discourse connectives and entity inference.

## 7. Limitations and prospective directions

Several limitations are already explicit in the current literature. For the discourse benchmark, the stimuli remain **English-only**, the dataset is **evaluation-only**, and the paper does not define **train/dev/test splits**. The benchmark is therefore best understood as a diagnostic probe rather than a full training resource. The authors also note that generalization to other languages and genres remains open, and that stronger causal mechanism analysis, such as mechanistic probing, is left for future work [2510.09556].

The Word Usage Graph line has a different set of constraints. It relies on contextualized definition datasets that are available for **English**, **Norwegian**, and **Russian**, while **German** is handled by zero-shot English generation. Future work proposed there includes **multilingual joint fine-tuning**, extension to **Chinese, Latin, Spanish, and Swedish WUGs**, and strategies for labeling small clusters with **\(\le 2\)** usages [2403.18024].

The speech and morphology literature points to additional open problems. The reduplication study models only one simple **CV-prefixing partial reduplication** in a **single-speaker, single-language** setting and relies on manual acoustic evaluation. The pluralization study emphasizes that **fusion** and **informativity** are global grammatical measures and not morphology-specific indices, so a tighter complexity analysis would require measures tailored to the inflectional domain under study [2009.06110] [2510.12463].

Taken together, these limitations indicate that “WUGNECTIVES” is not a single settled object but a family resemblance term spanning at least three research agendas: discourse-connective inference over novel entities, interpretable labeling of Word Usage Graph clusters, and wug-style generalization tests in speech and morphology. The common thread is controlled novelty: the systems are evaluated on inputs designed to suppress rote recall and foreground the structural or functional cues that the benchmark intends to measure.

Source: https://www.emergentmind.com/topics/wugnectives