CoBRA: Adversarial Response Assessment

• CoBRA is a computational framework that quantifies strategic language use in adversarial settings using metrics like BaT, PaT, and NRBaT.
• It operationalizes dialogue turns by decomposing commitment types and norm violations, applying its methodology to the CHARM corpus of courtroom cross-examinations.
• The framework reveals that while LLMs align well on surface pragmatic features, they struggle with core strategic assessments in high-stakes, non-cooperative exchanges.

CoBRA, short for Cooperation-Breach Response Assessment, is a theoretical and computational framework for quantifying the strategic effects of discourse moves in non-cooperative, high-stakes adversarial dialogue, especially where ordinary cooperative pragmatic assumptions are inadequate. It was introduced to address a gap in the evaluation of language use and LLMs: much existing work on pragmatics assumes cooperativity, whereas settings such as courtroom cross-examination are structured by evasion, damage control, implicature, and other strategically motivated departures from cooperative exchange. CoBRA operationalizes these phenomena through three interpretable turn-level metrics—Benefit at Turn (BaT), Penalty at Turn (PaT), and Normalized Relative Benefit at Turn (NRBaT)—and demonstrates them on CHARM, an annotated corpus of real courtroom cross-examinations (Zheng et al., 1 Jun 2025).

1. Conceptual scope and intellectual basis

CoBRA is designed to measure the strategic, often non-cooperative, use of language in adversarial discourse. Its motivation is the observation that traditional pragmatic accounts centered on Gricean cooperativity do not adequately characterize exchanges in which interlocutors have opposed goals and may benefit from being evasive, misleading, or selectively informative. The framework therefore adapts ideas from both Gricean pragmatics and game-theoretic pragmatics, specifically drawing on Message Exchange (ME) Games, to model how a turn may help or hurt a participant’s strategic position relative to the current Question Under Discussion (QUD) (Zheng et al., 1 Jun 2025).

The framework is intended to do four things. First, it decomposes dialogue turns into interpretable dimensions of strategic gain and strategic cost. Second, it provides objective annotation targets for non-cooperative discourse rather than relying only on holistic judgments of who “won” a turn. Third, it supports comparative evaluation of LLMs on pragmatic understanding outside cooperative settings. Fourth, it offers a formalism for tracking how local discourse moves accumulate into a trajectory of strategic standing over the course of a dialogue (Zheng et al., 1 Jun 2025).

A central premise is that strategic discourse should not be reduced to surface violations of conversational norms. A response may be relevant, fluent, and apparently truthful while still being strategically weak; conversely, it may mitigate immediate damage without producing a genuine strategic gain. This suggests a distinction between surface pragmatic conformity and adversarial strategic effectiveness, and CoBRA makes that distinction explicit through separate commitment, penalty, and cumulative-trajectory components.

2. Formal metrics and commitment structure

CoBRA defines a turn through a commitment value function and three derived metrics. Each response is first assigned a commitment type relative to the current QUD:

• Beneficial commitment: score $+1$
• Neutral commitment: score $+0.5$
• No commitment: score $-0.5$
• Detrimental commitment: score $-1$

Formally, the commitment function is:

$$f_c(C_i) = \begin{cases} 1 & \text{if } C_i=\text{beneficial (green!20)} \ 0.5 & \text{if } C_i=\text{neutral (yellow!20)} \ -0.5 & \text{if } C_i=\text{no commitment (gray!25)} \ -1 & \text{if } C_i=\text{detrimental (red!20)} \end{cases}$$

On top of this commitment labeling, CoBRA defines Benefit at Turn (BaT), which measures immediate strategic gain at turn $i$. BaT incorporates both the commitment type and penalties associated with maxim violations of relevance, manner, and quality:

$$\text{BaT}_i = \begin{cases} f_c(C_i) & \text{if } C_{i} \in \{\text{green!20}, \text{yellow!20}\} \ f_c(C_i) \times (\text{Rel}_i + \text{Man}_i + \text{Qual}_i) & \text{if } C_{i} = \text{red!20} \ 0 & \text{otherwise} \end{cases}$$

Here, $\text{Rel}_i$, $\text{Man}_i$, and $\text{Qual}_i$ are ratings for relevance, manner/clarity, and quality/truthfulness. In this scheme, clear and relevant beneficial commitments receive the highest benefit, while detrimental commitments are further discounted by norm violations (Zheng et al., 1 Jun 2025).

The second metric, Penalty at Turn (PaT), captures loss from detrimental or non-committal moves and adds a consistency penalty when a turn conflicts with earlier commitments:

$+0.5$0

The term $+0.5$1 penalizes inconsistency with prior commitments. The cumulative $+0.5$2 term means that contradiction can discount earlier gains without completely erasing them (Zheng et al., 1 Jun 2025).

The third metric, Normalized Relative Benefit at Turn (NRBaT), is a cumulative normalized measure of net strategic standing:

$+0.5$3

Because it uses $+0.5$4-score normalization, NRBaT is intended to support comparison across dialogues and annotators. It measures not just local turn success but the evolving strategic trajectory of the interaction (Zheng et al., 1 Jun 2025).

3. CHARM: corpus design and annotation

CoBRA is grounded empirically in CHARM, the Corpus of High-stakes Adversarial Response Maneuvers. CHARM consists of real-world adversarial dialogue drawn from actual U.S. legal trials, including cross-examinations from cases such as the West Memphis Three, O.J. Simpson, and Enron. The corpus is explicitly not synthetic or gamified; it is intended to capture discourse in which non-cooperation is normal rather than exceptional (Zheng et al., 1 Jun 2025).

The full collected material contains approximately 4,452 total turns and approximately 3,325 Q/A pairs. An annotated subset contains approximately 800 turns across four trials. Each annotated item includes the question, the response, and background context. Responses are labeled by trained linguists and journalists along several dimensions: commitment type, violations of Gricean maxims of relevance, clarity, and quality, consistency with prior statements, local turn outcome, and reasoning behind that outcome (Zheng et al., 1 Jun 2025).

CHARM serves two roles simultaneously. At the annotation level, it instantiates CoBRA’s theoretical categories on real adversarial dialogue. At the evaluation level, it provides a benchmark for testing whether LLMs can recover the same strategic structure from discourse context. Because the data come from courtroom cross-examination, the corpus emphasizes settings in which the strategic value of an answer depends on institutional roles, prior commitments, and the adversarial goals of the participants.

A plausible implication is that CHARM functions less as a generic dialogue corpus than as a stress test for pragmatic competence under conflict. That interpretation follows from its design: the annotation scheme prioritizes strategic commitment, consistency, and norm violation rather than only semantic correctness or local entailment.

4. Experimental use for LLM evaluation

The CoBRA framework is used to evaluate LLMs in a zero-shot setting. Models receive the same instructions as human annotators, with temperature set to 0.1, and are asked to judge commitment type, maxim violations, and the strategic effects needed to compute BaT, PaT, and NRBaT (Zheng et al., 1 Jun 2025).

The evaluated model set includes state-of-the-art LLMs such as GPT-4o, Gemini-2.5, Qwen, and Llama3, in both reasoning and non-reasoning variants and across multiple sizes. Performance is compared to human annotation using Spearman’s $+0.5$5, Cohen’s $+0.5$6, and AUC, rather than only raw accuracy. This choice reflects the fact that the task is structured and graded, not merely categorical (Zheng et al., 1 Jun 2025).

The reported results distinguish sharply between different aspects of pragmatic competence. LLMs show high agreement on surface maxim violations, with $+0.5$7–$+0.5$8 for relevance, manner, and quality violations. By contrast, they perform much worse on the strategically central dimensions: commitment type has low $+0.5$9, BaT correlates with human annotation at $-0.5$0, PaT at $-0.5$1, and NRBaT at $-0.5$2 (Zheng et al., 1 Jun 2025).

The framework is also compared with outcome-based evaluation. CoBRA’s cumulative metric NRBaT shows inter-annotator $-0.5$3, whereas unstructured outcome judgments show $-0.5$4. This indicates that the decomposed CoBRA formulation is substantially more stable across annotators than direct judgments of who won a turn (Zheng et al., 1 Jun 2025).

These findings support a narrow but important conclusion: current LLMs can often detect surface pragmatic irregularities while failing to model the adversarial strategic consequences of those irregularities.

5. Main empirical findings on model behavior

The empirical picture that emerges from CoBRA is not that LLMs lack all pragmatic competence, but that their competence is uneven and strongly biased toward cooperative or surface-level interpretation. The paper reports that LLMs often misread hedging and deflection as beneficial or neutral, even when those moves are strategically costly in adversarial context. In other words, they can detect that a response is off-topic or unclear, yet still fail to represent how that response affects the speaker’s standing under cross-examination (Zheng et al., 1 Jun 2025).

Model size is associated with better performance. Larger models align better with human annotation on commitment types and all three CoBRA metrics, with effect sizes of up to $-0.5$5 for BaT. The reported gains are statistically significant for BaT and commitment labeling, and less so for NRBaT, whose cumulative structure makes it a more difficult target (Zheng et al., 1 Jun 2025).

A more counterintuitive result concerns explicit reasoning. Adding chain-of-thought or reasoning-style prompting does not improve performance overall and can degrade it, especially on PaT, where the mean change is reported as $-0.5$6. The paper attributes this to overcomplication, confusion between loss minimization and genuine strategic benefit, and occasionally contradictory rationales. By the same report, the only area where reasoning offers a slight benefit is maxim-violation detection, and that effect is described as minor (Zheng et al., 1 Jun 2025).

This suggests a specific failure mode: reasoning-augmented models may elaborate plausible explanations without stabilizing the latent representation of adversarial intent. A plausible implication is that standard “reasoning” scaffolds can amplify interpretive verbosity while weakening strategic calibration in settings where discourse is inherently conflictual.

6. Significance, limitations, and future directions

CoBRA’s principal significance lies in shifting evaluation from cooperative pragmatics to strategic, non-cooperative language use. The framework argues that performance on Gricean maxims is insufficient as a proxy for pragmatic competence in adversarial settings. A model may recognize relevance or clarity failures yet still fail to infer the strategic structure of a witness’s answer, the cumulative effect of contradictions, or the difference between tactical evasion and genuine advantage (Zheng et al., 1 Jun 2025).

The framework also positions itself against outcome-only assessment. Because BaT, PaT, and NRBaT are decomposable and tied to explicit discourse properties, they are presented as more objective and less susceptible to annotator bias than holistic judgments of who is “winning.” This does not eliminate subjectivity entirely, but the reported agreement statistics indicate that the structured formulation is considerably more stable (Zheng et al., 1 Jun 2025).

The paper identifies several limitations. The annotated CHARM data cover only a subset of the collected material. The domain is legal and specifically courtroom-based, so generalization to other adversarial discourse types—such as political debate or negotiation—remains untested. The framework also does not yet model downstream strategic reasoning over longer horizons, that is, how speakers anticipate future moves rather than only evaluating the local and cumulative effects of prior turns (Zheng et al., 1 Jun 2025).

Proposed future directions include extending CoBRA and CHARM to additional domains, enlarging the annotated data, incorporating more legal expertise in annotation, modeling longer-horizon strategic planning, and using CoBRA-like diagnostics for alignment and safety research on manipulative or adversarial language behavior in LLMs. This suggests that CoBRA is best understood not only as an evaluation toolkit but also as a formal proposal for how pragmatic competence should be operationalized when cooperation cannot be assumed (Zheng et al., 1 Jun 2025).