NyayChat: Benchmark for Indian Consumer Redressal
- NyayChat is a curated dataset of 303 multi-turn dialogues that simulate ideal grievance redressal interactions in Indian consumer law.
- The dataset is constructed from Indian legal judgments and refined by law students and legal experts to ensure accurate procedural guidance.
- Evaluated using HAB metrics and reference-based scores, NyayChat serves as a benchmark for dialogue management and a scaffold for prompt design in legal AI.
Searching arXiv for the cited paper and closely related Indian legal AI work to ground the article with current references. Searching arXiv for "Grahak-Nyay Consumer Grievance Redressal through LLMs" and related NyayaAI / TathyaNyaya / NyayaAnumana papers. NyayChat is a multi-turn conversational dataset introduced within "Grahak-Nyay: Consumer Grievance Redressal through LLMs" as part of a consumer-law assistance framework for India. It is not a separate model, agent, or runtime subsystem. Rather, it is a curated corpus of user–chatbot interactions intended to represent “ideal” grievance-redressal dialogues, to inform prompt and flow design, and to benchmark end-to-end chatbot behavior in realistic Indian consumer-law scenarios (Ganatra et al., 7 Jul 2025).
1. Definition and placement within Grahak-Nyay
Within Grahak-Nyay, NyayChat is explicitly a dataset of conversations rather than a retriever, generator, or orchestration component. The paper characterizes it as a collection of annotated conversations between users and the chatbot on “various issues and complaints,” and Section 2.4 further describes “303 simulated conversations meticulously crafted by a team of legal experts,” each mirroring a real-world interaction between a user and the chatbot (Ganatra et al., 7 Jul 2025).
This distinction is important because Grahak-Nyay itself is a prompt-engineered Retrieval-Augmented Generation system. Its operational pipeline is described as: user query, query rewriting by an LLM, retrieval from a knowledge base comprising GeneralQA, SectoralQA, SyntheticQA, and Judgments, and response generation by Llama-3.1-8B-Instruct conditioned on the system prompt, retrieved chunks, and chat history. NyayChat sits around this pipeline rather than inside it: it is not retrieved at runtime, and the paper does not describe fine-tuning the generation model on it (Ganatra et al., 7 Jul 2025).
The dataset therefore serves three functions. First, it is a benchmark for conversation-level evaluation, especially under the proposed HAB metrics of Helpfulness, Accuracy, and Brevity. Second, it acts as a design scaffold for “good” consumer-law conversations, shaping prompt structure and turn-taking logic. Third, it grounds evaluation in the procedural realities of Indian consumer grievance redressal rather than in isolated single-turn question answering (Ganatra et al., 7 Jul 2025).
A common misconception is to treat NyayChat as the chatbot itself. The paper’s description does not support that reading. NyayChat is best understood as the conversational benchmark and exemplar corpus surrounding the deployed Grahak-Nyay chatbot, not as a standalone conversational engine (Ganatra et al., 7 Jul 2025).
2. Corpus construction, annotation, and statistics
The paper reports NyayChat in two slightly different ways. The abstract and contribution summary refer to “300 annotated conversations,” while the main body and Table 1 report 303 conversations. The detailed table gives the more specific dataset statistics, and these establish NyayChat as a long-dialogue corpus with substantial contextual depth (Ganatra et al., 7 Jul 2025).
| Statistic | Value |
|---|---|
| Total Conversations | 303 |
| Average Turns per Conversation | 32.01 |
| Average Tokens per Conversation | 3475.26 |
NyayChat was created from a case-driven synthetic workflow. The starting point was a database of 1,200 District Consumer Disputes Redressal Commission judgments in India. Cases were randomly sampled from this pool, and law students were instructed to “reimagine each situation as if they just encountered their grievances and were looking for an immediate resolution.” These synthetic chats were then reviewed by two legal experts to ensure legal and procedural correctness and to make the interactions “ideal” from the standpoint of consumer assistance (Ganatra et al., 7 Jul 2025).
The annotation scheme is therefore not a fine-grained per-turn tagging system. The paper does not itemize intent labels, dialogue-act labels, or a turn-level ontology. Instead, “annotation” refers to the drafting of synthetic but realistic conversation scripts, their grounding in actual judgments, and their expert review. For a subset of 65 conversations, the authors also produced reference responses for evaluation with BLEU, ROUGE, BERTScore, METEOR, and HAB-based assessments (Ganatra et al., 7 Jul 2025).
The paper does not report inter-annotator agreement statistics such as Cohen’s kappa. It also does not describe tokenization or normalization pipelines for NyayChat beyond reporting average token counts. What is specified clearly is the legal provenance of the source scenarios, the two-stage drafting-and-review process, and the use of the resulting conversations as benchmark material (Ganatra et al., 7 Jul 2025).
3. Legal scope and dialogue structure
NyayChat is tailored to Indian consumer law and grievance redressal. Section 2.4 states that its conversations cover sectors such as “e-commerce, medical negligence, railways, airlines, and more,” while the broader system prompt and knowledge base span “Airlines, Automobile, Banking, E-Commerce, Education, Electricity, Food Safety, Insurance, Real-Estate, Technology, Telecommunications, and more” (Ganatra et al., 7 Jul 2025).
The conversations capture a broad range of user intents. These include filing or escalating a complaint, understanding jurisdiction and limitation, drafting documents such as legal notices or affidavits, and seeking explanations of legal remedies and forums. The procedural orientation is central: NyayChat dialogues are not merely doctrinal Q&A exchanges but multi-step interactions in which the chatbot gathers facts and then advises on escalation pathways, documentation, and forum selection (Ganatra et al., 7 Jul 2025).
The paper indicates a consistent conversation flow. The chatbot is instructed to greet the user, ask one question at a time, collect facts, time of incident, opposing party details, and desired relief, warn about limitation if the grievance is older than two years, and then offer soft remedies and/or legal remedies. Conversations often include references to the National Consumer Helpline, the e-Daakhil portal, and consumer courts. Non-consumer-law queries are filtered out by prompt design (Ganatra et al., 7 Jul 2025).
This structure gives NyayChat a strongly procedural character. It captures both substantive legal issues, such as “deficiency in service,” and operational questions, such as where to file, when limitation applies, what documents are needed, and whether escalation should proceed through helplines, ombudsman-style channels, or consumer commissions. In that sense, NyayChat models the process of grievance redressal rather than only the final legal answer (Ganatra et al., 7 Jul 2025).
4. Evaluation, HAB metrics, and LLM-as-judge
NyayChat is central to the evaluation methodology of Grahak-Nyay. The paper introduces HAB metrics: Helpfulness, Accuracy, and Brevity. Helpfulness concerns whether the chatbot understands and resolves the user’s issue with actionable and relevant guidance. Accuracy concerns factual and legal correctness, including concrete details such as URLs and phone numbers. Brevity concerns concision and avoidance of unnecessary verbosity or excessive questioning. HAB is operationalized as three separate scalar scores, typically on a 1–5 Likert scale; the paper does not define a single aggregate formula such as (Ganatra et al., 7 Jul 2025).
Human evaluation is performed by legal experts, who rate conversations on each HAB dimension. These judgments are used to compare Grahak-Nyay against ChatGPT-4.0, Claude-3.5, Llama-3.1-405b-128k, and Llama-3.1-8b-128k. The reported outcome is that Grahak-Nyay outperforms the other systems in Helpfulness and Brevity and is competitive in Accuracy. The qualitative comparisons attribute this to domain grounding, current procedural information, and a more disciplined conversational flow (Ganatra et al., 7 Jul 2025).
NyayChat also serves as a testbed for automated evaluation. For 75 conversations with binary human HAB judgments, multiple evaluator models were prompted to score conversations. Table-level results report point-biserial correlations and Spearman’s between LLM scores and human judgments. The strongest reported correlations are for gpt-4o-mini on Helpfulness, with and , and the paper selects gpt-4o-mini as the primary automatic evaluator; Llama-3.1-70B is the best open-source evaluator in this comparison (Ganatra et al., 7 Jul 2025).
On the 65-conversation subset with reference responses, the system is evaluated with standard reference-based metrics and LLM-scored HAB values:
| Metric | Score |
|---|---|
| ROUGE-1 | 66.9 |
| ROUGE-2 | 41.1 |
| ROUGE-L | 33.2 |
| BERTScore | 90.9 |
| METEOR | 41.9 |
| BLEU | 37.4 |
| Helpfulness (LLM, 1–5) | 4.65 |
| Accuracy (LLM, 1–5) | 3.61 |
| Brevity (LLM, 1–5) | 3.12 |
These results indicate that, on NyayChat-style conversations, Grahak-Nyay aligns reasonably well with expert references and is judged especially strong on Helpfulness, while Accuracy and Brevity remain less saturated. The paper also makes clear that NyayChat is not used to evaluate the retriever in isolation; retriever-oriented RAG evaluation is conducted with GeneralQA, SectoralQA, and SyntheticQA. Even so, NyayChat indirectly probes retrieval quality because Accuracy scoring explicitly compares responses against retrieved context in multi-turn interactions (Ganatra et al., 7 Jul 2025).
5. Relation to companion datasets and the wider Indian legal AI landscape
Grahak-Nyay introduces several data artifacts with distinct roles. GeneralQA contains 53 question–answer pairs on general consumer law. SectoralQA contains 889 sector-specific question–answer pairs, though the appendix reports 835. SyntheticQA contains approximately 4,734 automatically generated question–answer pairs for RAG evaluation. Judgments contains 570 annotated and summarized Indian consumer court judgments. NyayChat differs from all of these by focusing on simulated multi-turn dialogue rather than knowledge chunks or long-form case documents (Ganatra et al., 7 Jul 2025).
| Artifact | Primary role in Grahak-Nyay |
|---|---|
| GeneralQA | General consumer-law knowledge base |
| SectoralQA | Sector-specific knowledge base |
| SyntheticQA | RAG evaluation dataset |
| Judgments | Retrieved legal decisions and summaries |
| NyayChat | Multi-turn conversational benchmark |
The distinction is methodological. GeneralQA, SectoralQA, SyntheticQA, and Judgments are used as retrieval content in the RAG pipeline. NyayChat is not. The paper does not report experiments that mix NyayChat with those datasets for training or retrieval; instead, the resources are complementary, with the QA sets and judgments grounding factual retrieval and NyayChat assessing end-to-end conversational performance (Ganatra et al., 7 Jul 2025).
Within the broader Indian legal AI literature, NyayChat occupies a different niche from several adjacent efforts. "NyayaAI: An AI-Powered Legal Assistant Using Multi-Agent Architecture and Retrieval-Augmented Generation" presents a multi-agent legal assistant grounded in a curated Indian legal knowledge base, with reported domain-classification precision of 70\%, RAG retrieval precision of 74\%, and overall response accuracy of 72\% (Deepanshu et al., 11 May 2026). "TathyaNyaya and FactLegalLlama: Advancing Factual Judgment Prediction and Explanation in the Indian Legal Context" introduces a fact-centric judgment-prediction-and-explanation dataset of roughly 16,000 judgments and a specialized explanatory model built on LLaMa-3-8B (Nigam et al., 7 Apr 2025). "NyayaAnumana & INLegalLlama: The Largest Indian Legal Judgment Prediction Dataset and Specialized LLM for Enhanced Decision Analysis" contributes 7,02,945 preprocessed Indian cases for judgment prediction and a domain-specialized LLaMa-2-based legal model (Nigam et al., 2024).
Compared with those resources, NyayChat is distinctive in emphasizing consumer grievance redressal, procedural guidance, and long multi-turn interactions. It is therefore closer to an evaluation corpus for legal dialogue management than to datasets built for judgment prediction, fact extraction, or generic legal research assistance. This suggests a complementary role in the Indian legal AI ecosystem: NyayChat addresses conversational and procedural assistance where other datasets chiefly address predictive or document-centric tasks (Ganatra et al., 7 Jul 2025).
6. Limitations, release status, and significance
Several limitations are explicit or strongly indicated in the paper. NyayChat contains 303 conversations, each deep but relatively few in number compared with large general-domain dialogue corpora. The conversations are synthetic, even though they are grounded in actual DCDRC judgments. Real users may be less systematic, more emotionally variable, and more prone to incomplete or ambiguous narration than the expert-shaped interactions in the dataset. Coverage is also bounded by the underlying pool of 1,200 judgments and by the final 303-conversation subset, which may under-represent niche sectors or regional variation. In addition, the drafting process encodes “ideal” interaction patterns defined by law students and expert reviewers rather than organically occurring user behavior (Ganatra et al., 7 Jul 2025).
NyayChat is also English-focused in the current work. The paper presents multilingual extension as a natural direction, given the linguistic diversity of Indian legal practice. Another clear boundary is methodological: Grahak-Nyay does not fine-tune Llama-3.1-8B-Instruct on NyayChat, so the dataset functions as benchmark and design scaffold rather than as supervised training data in the reported system (Ganatra et al., 7 Jul 2025).
The release status is stated succinctly: “Code and datasets will be released.” This includes NyayChat together with GeneralQA, SectoralQA, SyntheticQA, and Judgments. The paper does not specify a repository URL or license in the text provided, so reproducibility is announced but not fully operationalized there (Ganatra et al., 7 Jul 2025).
The significance of NyayChat lies in the problem formulation it makes concrete. It shifts attention from isolated legal QA and judgment prediction toward multi-turn grievance-redressal dialogue anchored in Indian consumer law. It also pairs that shift with a domain-specific evaluation framework, HAB, and with experiments on LLM-based judging. Because many legal NLP resources focus on case retrieval, summarization, or prediction, NyayChat extends the empirical basis of the field toward procedural assistance and consumer access to justice. A plausible implication is that it can serve as a foundational benchmark for future work on dialogue management, legal assistance evaluation, and consumer-facing RAG systems in the Indian context (Ganatra et al., 7 Jul 2025).
