ContraGen: Enterprise Contradiction Benchmark

• ContraGen is a multi-agent, contradiction-aware framework that rigorously evaluates enterprise RAG systems in realistic document settings.
• It systematically generates and injects both self- and pairwise contradictions into policy-style documents, enhancing contradiction detection.
• The hybrid mining pipeline paired with a retrieval-verifiability layer offers robust insights into retrieval gaps and reasoning errors.

ContraGen is a multi-agent, contradiction-aware benchmark framework for enterprise contradiction detection in Retrieval-Augmented Generation (RAG) systems. It is designed to stress-test RAG in enterprise environments by generating realistic enterprise-style documents, systematically embedding both self-contradictions within a document and pairwise contradictions across documents, mining those contradictions with a hybrid automated pipeline, and validating them with human annotators. Its central objective is to model the contradiction patterns that arise in long, policy-like corpora such as contracts, financial filings, compliance reports, HR manuals, and policy documents, where unresolved conflicts can undermine compliance, governance, accountability, and risk management (Mantravadi et al., 3 Oct 2025).

1. Problem setting and enterprise scope

ContraGen is motivated by a specific failure mode of enterprise RAG: retrieved evidence can be contradictory even when each source is locally plausible. In the problem formulation, a typical enterprise RAG system answers questions or drafts content using retrieved documents such as policies, contracts, compliance reports, financial filings, or HR manuals. Those corpora often contain outdated versus updated versions of policies, department-specific rules that conflict with global policies, misaligned procedures or inconsistent KPIs, and conflicting interpretations of regulations. When a system retrieves such evidence without recognizing or resolving the conflict, the LLM may merge conflicting facts, produce ambiguous or hedged answers, or hallucinate procedures and standards (Mantravadi et al., 3 Oct 2025).

The enterprise setting is important because the consequences are explicitly framed as high-risk. ContraGen situates contradictions as a threat to compliance, governance, accountability, and risk management rather than as an abstract natural language inference problem. Existing contradiction benchmarks are presented as inadequate for this setting because they are mostly sentence-level NLI, lack enterprise document structure such as sections and cross-references, rarely include cross-document contradictions, restrict labels to contradiction versus entailment versus neutral, and do not support systematic controllable injection of enterprise-relevant contradiction types (Mantravadi et al., 3 Oct 2025).

The synthetic corpus is anchored in a fictional multinational aerospace company described as “a large aerospace company designing commercial aircraft and defense systems, with major manufacturing sites in Wichita, Toulouse, and Nagoya, and operating under strict international export controls.” The corpus spans six high-level domains with five subdomains each: HR and Personnel Management, Engineering and Development, Sales and Marketing, Finance and Operations, IT and Technical Support, and Legal and Compliance. The reported total is 84 synthetic documents (Mantravadi et al., 3 Oct 2025).

2. Multi-agent architecture and end-to-end workflow

ContraGen is implemented as a multi-agent system orchestrated via a central controller, or orchestrator, with agents communicating over Pyro4 RPC. The three main agents are the Contradiction-Aware Content Generation Agent, the Contradiction Mining Agent, and the Retrieval Verifiability Agent. The orchestrator manages task scheduling and agent communication and maintains metadata covering documents, contradictions, detection results, human labels, and retrieval-verifiability tags (Mantravadi et al., 3 Oct 2025).

The workflow begins with metadata and base document generation. A fictional enterprise profile and a domain-subdomain taxonomy are defined, and metadata is generated for title, topic, date, department, location, document type, and authority level. The Content Generation Agent then produces multi-paragraph enterprise documents from that metadata. Contradiction injection follows: the generator selects important factual sentences and injects self-contradictions within a document and pairwise contradictions across documents, subject to fluency constraints. The resulting corpus is then processed by the Contradiction Mining Agent, which applies semantic filtering, NLI classification, LLM-based judgment, and a confidence-weighted hybrid score to identify high-confidence contradictory pairs. Human annotators inspect candidate pairs and surrounding context, verify or correct labels, and add missed injected contradictions. Finally, the Retrieval Verifiability Agent labels each validated contradiction as either retrieval-verifiable or retrieval-resistant (Mantravadi et al., 3 Oct 2025).

The output is a gold-standard dataset of enterprise documents with labeled self- and pairwise contradictions, including contradiction type, location, and verifiability. This decomposition is also presented as a broader design pattern for trustworthy LLM pipelines: generation, detection, and retrieval classification are separated into specialized roles with distinct failure modes and explicit handoff points (Mantravadi et al., 3 Oct 2025).

3. Synthetic enterprise documents and controlled contradiction injection

ContraGen’s document synthesis is metadata-driven and explicitly enterprise-styled. Given the company description and a selected domain-subdomain pair, the Content Generator first creates metadata fields for title, topic, date, department, location, document type, and authority level. It then produces a 4–6-paragraph document structured with sections, headings, numbered lists, and bullets, written in assertive, policy-oriented language. The generation prompt requires at least one clear, factual, assertive sentence early in the document to serve as a contradiction target, along with dates and time intervals near the given date, specific personnel titles and contact information, sub-locations within locations, varied tone, and rich process descriptions. Each document ends with NEW PEOPLE META DATA and NEW DOCUMENT META DATA (Mantravadi et al., 3 Oct 2025).

Contradiction injection is systematic and controllable. For self-contradictions, ContraGen first identifies one sentence that is clear, factual, assertive, important to the document’s main topic, and specific. A second prompt then generates one or two sentences that directly contradict that target sentence, are “hard,” maintain style and tone, and instantiate one of the benchmark’s contradiction types. A third prompt blends the contradiction into the document while preserving coherence and without explicit discourse markers such as “however.” For pairwise contradictions, the same process begins from a source document $d_1$, but the contradictory statement is embedded into a distinct document $d_2$, which must not contain the original target sentence from $d_1$ (Mantravadi et al., 3 Oct 2025).

Fluency is controlled with GPT-2 perplexity. Documents are rejected if the contradicted version exceeds an absolute perplexity cap of 22.0, and relative perplexity increase is also bounded. Self-contradictions must satisfy a relative increase of at most 5.5%, while pairwise contradictions must satisfy at most 7.5%. In the corpus-wide injection strategy, HR, IT, and Legal documents each contain one self-contradiction, whereas in Engineering, Finance, and Sales every second document contradicts the preceding one. Human evaluation on a 5-point Likert scale reported Fluency 4.2, Specificity 4.7, Coherence 4.5, and Legitimacy 3.5. Annotators detected contradictions in only 54% of cases, which the authors interpret as evidence that the injected contradictions are realistic and often subtle (Mantravadi et al., 3 Oct 2025).

4. Contradiction taxonomy and mining pipeline

ContraGen formalizes six contradiction types common in enterprise workflows. These types are not mutually exclusive; a single contradiction can be counted in multiple categories, such as temporal plus specificity (Mantravadi et al., 3 Oct 2025).

Type	Definition	Typical spans
Temporal	Conflicting dates, times, deadlines, or phases	Dates, time intervals, phase schedules
Numerical	Conflicting numeric values: amounts, percentages, counts	Numbers in text, tables, KPIs
Authority	Conflicting sources or issuers	“Issued by…”, role or department attributions
Process	Incompatible procedures, channels, or sequences of steps	Process descriptions, workflow steps, submission methods
Policy Reversal	Direct normative reversal	Policy clauses, compliance obligations
Specificity	Broader or narrower scope that creates conflict	Scope clauses such as regions, roles, conditions

Illustrative examples include “Starts Jan 15” versus “Starts end of Q1” for Temporal, “$12M surplus” versus “$5M deficit” for Numerical, “Issued by Compliance Office” versus “Issued by Strategy Unit” for Authority, “Submit via HR portal” versus “Submit through admins” for Process, “Remote work mandatory” versus “Remote work not permitted” for Policy Reversal, and “Applies globally” versus “Applies only to APAC” for Specificity. The injected contradiction distribution is reported as 88 self-contradictions and 89 pairwise contradictions, broken down as Temporal 25/25, Numerical 7/6, Specificity 16/24, Policy Reversal 26/14, Authority 4/9, and Process 10/11 (Mantravadi et al., 3 Oct 2025).

The Contradiction Mining Agent is designed to avoid exhaustive $\mathcal{O}(n^2)$ sentence-pair comparison. It first applies top-$k$ semantic filtering using msmarco-distilbert-base-v3 sentence embeddings, with $k = 5$, a similarity threshold $\theta_s$, and filters excluding very short sentences, pure numbers, and bullet-point fragments, followed by deduplication via text hashing. Candidate pairs are then passed to facebook/bart-large-mnli, which outputs contradiction, neutral, or entailment labels and a confidence score. Pairs labeled contradiction or pairs with $p_{\text{NLI}} \leq 0.7$ are sent to GPT-4o, which returns a binary contradiction label, justification text, and a confidence score. NLI and LLM outputs are then combined with a confidence-weighted hybrid score, and a pair is classified as contradictory when the hybrid score exceeds $\tau = 0.5$ (Mantravadi et al., 3 Oct 2025).

Human-in-the-loop validation is a constitutive part of the benchmark rather than a post hoc audit. Annotators label candidate pairs as contradiction or non-contradiction after reading surrounding context, and injected contradictions not detected by any automated method are manually added. For self-contradictions ($d_2$0), reported inter-annotator agreement is 95.0%, with Cohen’s $d_2$1 and Krippendorff’s $d_2$2. For pairwise contradictions ($d_2$3), agreement is 96.7%, with Cohen’s $d_2$4 and Krippendorff’s $d_2$5. Cases with under 90% agreement are adjudicated by a subject-matter expert. The final gold-standard dataset contains 141 self-contradiction pairs and 426 pairwise contradiction pairs, each annotated with text spans, human label, and predictions from NLI, LLM judge, and hybrid models (Mantravadi et al., 3 Oct 2025).

5. Retrieval verifiability and empirical performance

ContraGen is not presented as a full end-to-end RAG pipeline implementation. Instead, it is positioned as a contradiction-aware evaluation layer on top of RAG systems. Synthetic enterprise documents are intended to be indexed in a retrieval system such as a vector store or BM25, and the corpus is designed so that enterprise queries can retrieve documents with known contradictory statements. Example query forms include “When can new employees start working remotely?”, “Who must approve export control changes?”, and “How many phases must prototypes pass before launch?” (Mantravadi et al., 3 Oct 2025).

Its distinctive diagnostic layer is the Retrieval Verifiability Agent. For each validated contradiction pair $d_2$6, the agent assigns one of two labels. A contradiction is retrieval-verifiable when reliable external evidence exists that can resolve which statement is correct; in that case, failure is attributed primarily to retrieval gaps. A contradiction is retrieval-resistant when external evidence is insufficient and resolution requires reasoning beyond retrieval or normative judgment; failures here are attributed to reasoning gaps or misalignment. This separation is meant to support future evaluation of contradiction detection, contradiction resolution, and output consistency without collapsing all RAG failures into a single error type (Mantravadi et al., 3 Oct 2025).

The contradiction detection experiments evaluate three methods against human labels using Accuracy, Precision, Recall, and F1: an NLI baseline (facebook/bart-large-mnli), a GPT-4o LLM Judge, and the confidence-weighted Hybrid method. For self-contradictions, NLI reports 41.1% accuracy, 40.0% precision, 76.7% recall, and 52.6% F1; the LLM Judge reports 82.3%, 79.7%, 78.3%, and 79.0%; and the Hybrid reports 90.1%, 92.6%, 83.3%, and 87.7%. For pairwise contradictions, NLI reports 29.3% accuracy, 12.9% precision, 63.1% recall, and 21.4% F1; the LLM Judge reports 72.1%, 34.1%, 89.2%, and 49.4%; and the Hybrid reports 87.8%, 57.8%, 73.8%, and 64.9% (Mantravadi et al., 3 Oct 2025).

These results support a central empirical claim of the framework: self-contradictions are relatively easier to detect, whereas cross-document contradictions are substantially more challenging. The paper attributes this difference to the locality of intra-document contradictions versus the entity resolution, varying framing, and distinct contexts required for cross-document reasoning. In this setting, NLI alone over-flags contradictions, GPT-4o achieves very high recall but low precision on pairwise cases, and the Hybrid method offers the most robust balance (Mantravadi et al., 3 Oct 2025).

6. Significance, limitations, and position within adjacent research

ContraGen is intended to support more trustworthy and accountable enterprise RAG by providing realistic stress tests, controlled contradiction injection, hybrid mining with human oversight, and a retrieval-verifiability layer that separates retrieval issues from reasoning issues. This suggests a benchmarking regime in which candidate RAG stacks can be evaluated before deployment, contradiction detectors can be inserted as guardrails for human review, and prompting strategies can be tested for whether they surface contradictions rather than merge them into an apparently coherent answer (Mantravadi et al., 3 Oct 2025).

Its limitations are explicit. All documents are synthetic, even if designed to be realistic, and the domain coverage remains tied to a single fictional aerospace company context. The multi-agent generation, mining, and annotation pipeline is computationally and financially expensive. Ground truth depends heavily on human annotators and experts. Thresholds such as perplexity caps, similarity scores, and decision cutoffs are chosen heuristically, and no extensive hyperparameter search is reported, which may affect reproducibility and generalization. Proposed future directions include mixed synthetic-real benchmarks, more advanced cross-document reasoning, systematic ablations of thresholds and model choices, direct RAG evaluation on the corpus, and deployment-ready guardrails with redaction, access control, and auditing (Mantravadi et al., 3 Oct 2025).

Within the broader literature, ContraGen occupies a distinct position. It is not a contrastive generation method in the sense of image-generation work such as “ContraGAN: Contrastive Learning for Conditional Image Generation” (Kang et al., 2020), nor a sequence-level contrastive text generation framework such as “CoNT: Contrastive Neural Text Generation” (An et al., 2022). Its contribution is instead to contradiction-aware benchmark generation and evaluation for enterprise RAG. That distinction matters because the term “ContraGen” could otherwise be misread as referring to contrastive generation; in this usage, it denotes contradiction detection in enterprise document corpora and the infrastructure required to evaluate it rigorously (Mantravadi et al., 3 Oct 2025).