WUGNECTIVES: A Benchmark for Discourse Inference

Updated 4 July 2026

WUGNECTIVES is a benchmark that evaluates if language models can infer properties of novel entities using only discourse connectives.
It employs controlled nonce entities and diverse prompt templates to isolate connective semantics, revealing distinct model strengths in causal connectives and weaknesses in concessive inference.
The framework tests various language models across temporal, instantiation, and preference tasks, providing diagnostic insights into inference strategies and generalization without factual biases.

WUGNECTIVES is primarily a benchmark for testing whether discourse connectives can inform LLMs about the world independently of prior factual knowledge. It reverses the usual discourse task: rather than predicting which connective best links two known arguments, it asks whether a connective linking unknown entities allows a model to infer properties of those entities. To isolate connective meaning as the critical signal, the benchmark replaces familiar referents with novel entities such as Wugs, Daxes, Wugfest, and Wugsville (Brubaker et al., 10 Oct 2025).

1. Conceptual basis

WUGNECTIVES studies the inverse problem of connective prediction. In the standard formulation, a model uses world knowledge about two arguments to choose a connective such as because or however. WUGNECTIVES instead asks what a model can infer about the arguments given the connective. The benchmark therefore targets the functional meaning of discourse connectives rather than stored encyclopedic knowledge about real entities (Brubaker et al., 10 Oct 2025).

The framework is grounded in Penn Discourse Treebank senses. It covers Expansion.Instantiation, Contingency.Cause, Comparison.Concession, and Temporal.Asynchronous. Within this design, concessive connectives are especially important because they require denial or cancellation of a causal expectation raised by one argument using the other. A representative example is: “Although I hate leafy vegetables, I prefer wugs to daxes.” In this case, the concession blocks the naive causal inference that the preferred item should instantiate the disliked property (Brubaker et al., 10 Oct 2025).

The use of nonce entities is methodologically central. Bare plurals, events, and locations are constructed so that the model has no entrenched prior knowledge to exploit. This makes the benchmark a controlled probe of whether connective semantics alone can support inference. A plausible implication is that WUGNECTIVES belongs to the broader family of “wug” evaluations, but it relocates the wug paradigm from morphology to discourse-semantic inference (Brubaker et al., 10 Oct 2025).

2. Dataset architecture and connective taxonomy

The benchmark comprises 8,880 stimuli created by embedding a base set of utterances in 12 prompt templates. The paper states a base set of 740 utterances embedded across 12 templates for 8,880 total stimuli, and it organizes the stimuli around three inference families: Instantiation, Preference, and Temporal (Brubaker et al., 10 Oct 2025).

The connective inventory spans 41 unique connective forms across 7 third-level senses. The reported sense inventory is as follows.

PDTB sense	Count	Representative connectives
Expansion.Instantiation.Arg2-as-instance	100	for example, for instance, in particular, specifically, such as
Contingency.Cause.Reason	160	as, because, for, since
Contingency.Cause.Result	240	as a result, so, therefore, thus
Comparison.Concession.Arg1-as-denier	98	although, as much as, even though, though
Comparison.Concession.Arg2-as-denier	302	although, but, even though, however, yet, nevertheless, though, while, despite that
Temporal.Asynchronous.Precedence	110	afterwards, before, even before, finally, hence, later, next, so, subsequently, then, consequently
Temporal.Asynchronous.Succession	130	after, as soon as, once, previously, thereafter, eventually, earlier, since

The nonce inventory is carefully balanced. Bare plurals include Wugs, Daxes, Feps, Geks, Blickets; events include Wugfest, Daxday, Fepfestival, Gextravaganza, Blicketbash; locations include Wugsville, Daxburgh, Fepopolis, Gektopia, Blicketland. Pairwise use of nonce entities avoids single-entity co-occurrence biases, and nonce pairs are counterbalanced across stimuli so a bias toward any surface form yields chance performance. Temporal stimuli vary neutral event verbs such as happened, took place, and occurred to reduce superficial cues (Brubaker et al., 10 Oct 2025).

Preference stimuli manipulate both property type and polarity. The benchmark uses 10 binary-style properties: five for bare plurals and five for locations. Polarity is manipulated by swapping love and hate, so that the same connective class can license either positive entailments or their negations. Concessive and causal connectives are also used in fronted and non-fronted positions and in either the preference clause or the property clause, making clause-order variation part of the test rather than an uncontrolled artifact (Brubaker et al., 10 Oct 2025).

3. Task formulation, evaluated models, and core results

Each stimulus contains a premise and an explicit question. Models answer Yes/No for Instantiation and Preference, and one of the two event names for Temporal. The primary metric is accuracy, with chance at 50% for all tasks:

$\mathrm{Acc} = \frac{1}{N}\sum_{i=1}^{N}\mathbb{1}[\hat{y}_i = y_i].$

The paper reports 95% confidence intervals across prompt templates to capture sensitivity to wording (Brubaker et al., 10 Oct 2025).

The evaluation covers 17 open-source decoder-only LMs: Llama 3.1 8B in base and instruct form; Qwen 2.5 at 0.5B, 1.5B, 3B, 7B, 14B, each in base and instruct variants; OLMo-2 at 1B and 7B, again in base and instruct variants; and Qwen2.5-DS 14B, a reasoning-tuned model produced by fine-tuning Qwen-2.5-14B on reasoning traces generated by the larger DeepSeek R1 model (Brubaker et al., 10 Oct 2025).

For base and instruction-tuned models, responses are extracted from token probabilities using minicons, ranking surface-form variants of Yes and No, or the two event names, by conditional probability. For Qwen2.5-DS, the model is prompted to produce a rationale followed by a final answer in $\boxed{\cdot}$ format, with an algorithmic post-processor used when the boxed answer is absent. Decoding for the reasoning model uses temperature 0.6 (Brubaker et al., 10 Oct 2025).

The principal empirical findings are sharply differentiated by connective type. All models systematically struggle on Comparison.Concession. By contrast, Contingency.Cause is considerably easier: thirteen of the seventeen models exceed chance by at least 5 percentage points for both Contingency.Cause senses, and Qwen2.5-DS exceeds 75% accuracy for both Reason and Result. Instantiation is generally near chance except for larger Qwen models at $\ge 7$ B, with Qwen2.5-DS reaching 82% accuracy. The reasoning-tuned model is the strongest overall across senses except concession (Brubaker et al., 10 Oct 2025).

The regression analysis is equally specific. For the Qwen 14B variants, reasoning-oriented tuning yields a positive effect relative to both base and instruct training: $\beta_{\text{reasoning vs base}} = 0.06, p < .001$ and $\beta_{\text{reasoning vs instruct}} = 0.08, p < .001$ . By contrast, there is no clear global effect of scale, with $\beta_{\text{params}} = 0.008, p = .10$ , and instruction-tuning shows scant improvements over base, with $\beta_{\text{instruct}} = -0.004, p = .826$ . Connective frequency in the 3T-token Dolma corpus is not correlated with per-connective accuracy; the reported $r^2$ values are non-significant for all models (Brubaker et al., 10 Oct 2025).

4. Diagnostic interpretation and recurrent failure modes

The most prominent failure mode is concessive semantics. Models often behave as if concession were ordinary contrast or even causal reinforcement, failing to recognize that connectives such as although and even though cancel a straightforward causal entailment. This pattern holds for both Arg1-as-denier and Arg2-as-denier senses and is exacerbated by fronted concessive clauses. The benchmark therefore isolates a pragmatic weakness that is not reducible to entity familiarity, since the entities are novel by construction (Brubaker et al., 10 Oct 2025).

Temporal performance requires more careful interpretation. The paper introduces two diagnostic heuristics: Choose-first, which selects the first event mentioned, and Choose-recent, which selects the most recent event in linear order. On Precedence, Choose-first attains 92.5% accuracy and Choose-recent 7.5%; on Succession, the same heuristics yield 56.4% and 43.6% respectively. This shows that much of the Precedence subset can be solved by a superficial linear-order heuristic because most Precedence connectives are not frontable, whereas Succession is harder to game (Brubaker et al., 10 Oct 2025).

Two common misunderstandings are therefore not supported by the results. First, strong performance on some temporal items does not by itself demonstrate robust discourse reasoning, because shallow positional strategies already perform well on part of the benchmark. Second, larger parameter counts and generic instruction tuning are not sufficient explanations of success, since the reported mixed-effects analyses do not show a clear global scale effect and show scant improvements for instruction tuning (Brubaker et al., 10 Oct 2025).

The benchmark also states several limits of interpretation. Even with nonce entities, the stimuli include lexical material such as prefer, love, and hate, so fully isolating connective impact is intractable. Novel words are also not entirely tokenization-free, because subtokens already exist in model vocabularies. Finally, the comparison between reasoning models and base or instruction-tuned models is not perfectly symmetric, because the former generate rationales while the latter are scored through constrained probability extraction (Brubaker et al., 10 Oct 2025).

5. Alternative use of the label in Word Usage Graph research

A distinct line of work associates “WUGNECTIVES” with Word Usage Graphs (WUGs) rather than discourse connectives. One paper explicitly states that it does not use the term “WUGNECTIVES”, but also states that its main contribution is exactly what the term suggests: generating concise, human-readable cluster labels that serve as sense definitions for clusters in WUGs (Fedorova et al., 2024).

In that setting, a Word Usage Graph is a weighted, undirected graph built for a single target word, whose nodes are individual usage instances in context and whose edge weights reflect semantic proximity from human annotations of contextual similarity. After annotation, nodes are clustered automatically, and each cluster roughly corresponds to a word sense or usage type. A Diachronic WUG (DWUG) contains usages from multiple time periods, allowing clusters to be tracked over time (Fedorova et al., 2024).

The labeling pipeline is called DefGen. It fine-tunes mT0-xl (3.7B parameters), a multilingual variant of Flan-T5, to produce contextualized dictionary-like definitions from usage sentences plus a simple instruction prompt. Definitions are generated per usage, embedded with multilingual SBERT using distiluse-base-multilingual-cased-v1, and aggregated by selecting the definition closest to the cluster centroid in cosine space. Only clusters with at least three usage instances are labeled (Fedorova et al., 2024).

Human evaluation uses a guess-the-cluster-by-definition task. On the English DWUG, DefGen achieves 69.17% accuracy, compared with 21.67% for Lesk and 50.00% for GlossReader. On the German DWUG, DefGen reaches 57.89% accuracy against 53.68% for GlossReader. On Russian RuDSI with English definitions, DefGen attains 71.79% compared with 64.10% for GlossReader. The paper reports no significance tests or confidence intervals. In this usage, “WUGNECTIVES” denotes interpretable sense labels rather than a discourse benchmark (Fedorova et al., 2024).

6. Place within the broader wug-testing tradition

WUGNECTIVES is part of a wider methodological tradition in which models are evaluated on generalization to novel forms or entities under controlled conditions. In speech, one precursor models unsupervised learning of an identity-based pattern—partial CV reduplication—from raw continuous data using ciwGAN, and introduces an explicit wug-testing protocol for CNNs trained on speech. In that setting, latent manipulation with $c = [0, 7.25]$ and $z90 = 7$ yields 33/100 correctly reduplicated [s]-initial forms, whereas a bare WaveGAN reaches only 1/100 or 4/100 under analogous strong manipulations. The paper explicitly connects these findings to WUGNECTIVES by describing the approach as a principled acoustic wug-testing harness for identity constraints in continuous speech (Beguš, 2020).

In morphological inflection, another line of work shows that a standard character-level Transformer almost completely fails under wug-test-like conditions when training and test lemmata do not overlap. Its proposed remedy is substring-based hallucination that injects copying bias through 2-, 3-, and 4-gram dummy stems. The reported +hall-2k-substr results include 68.50 for Finnish, 62.62 for German, 69.26 for Russian, and 91.94 for Spanish, outperforming earlier augmentation methods in most languages (Liu et al., 2021).

LLMs have also been examined with nonce-based morphological tests. A multilingual study of gpt-3.5-turbo-0613 in English, German, Tamil, and Turkish reports that ChatGPT massively underperforms purpose-built systems, especially in English, and identifies recurrent error types such as real word bias and productivity amplification. A later multilingual pluralization study in Catalan, English, Greek, and Spanish concludes that model accuracy is “resource-sensitive but language-blind”: accuracy patterns align more closely with community size and data availability than with structural complexity, even though the models can reach human-like accuracy once BERT is excluded (Weissweiler et al., 2023, Pantelidou et al., 14 Oct 2025).

This suggests a shared methodological logic across otherwise different tasks. Novel items, leakage control, and carefully constrained prompts are used to separate genuine generalization from memorization or lexical familiarity. WUGNECTIVES extends that logic from morphology and speech to discourse connectives and entity inference.

7. Limitations and prospective directions

Several limitations are already explicit in the current literature. For the discourse benchmark, the stimuli remain English-only, the dataset is evaluation-only, and the paper does not define train/dev/test splits. The benchmark is therefore best understood as a diagnostic probe rather than a full training resource. The authors also note that generalization to other languages and genres remains open, and that stronger causal mechanism analysis, such as mechanistic probing, is left for future work (Brubaker et al., 10 Oct 2025).

The Word Usage Graph line has a different set of constraints. It relies on contextualized definition datasets that are available for English, Norwegian, and Russian, while German is handled by zero-shot English generation. Future work proposed there includes multilingual joint fine-tuning, extension to Chinese, Latin, Spanish, and Swedish WUGs, and strategies for labeling small clusters with $\boxed{\cdot}$ 0 usages (Fedorova et al., 2024).

The speech and morphology literature points to additional open problems. The reduplication study models only one simple CV-prefixing partial reduplication in a single-speaker, single-language setting and relies on manual acoustic evaluation. The pluralization study emphasizes that fusion and informativity are global grammatical measures and not morphology-specific indices, so a tighter complexity analysis would require measures tailored to the inflectional domain under study (Beguš, 2020, Pantelidou et al., 14 Oct 2025).

Taken together, these limitations indicate that “WUGNECTIVES” is not a single settled object but a family resemblance term spanning at least three research agendas: discourse-connective inference over novel entities, interpretable labeling of Word Usage Graph clusters, and wug-style generalization tests in speech and morphology. The common thread is controlled novelty: the systems are evaluated on inputs designed to suppress rote recall and foreground the structural or functional cues that the benchmark intends to measure.