Nyay-Darpan: AI Judicial Assistance

• Nyay-Darpan is an AI-based judicial framework for consumer law in India that integrates a dual-module system for case-file summarization and similar-case retrieval.
• The system utilizes advanced LLM techniques in a two-step pipeline to generate coherent material summaries and enhance quasi-judicial decision-making.
• It employs evaluation metrics like ROUGE scores and precision@5, while addressing challenges such as sector misclassification and input quality issues.

Nyay-Darpan is an AI-based judicial assistance framework specifically designed for the consumer law domain in India. Addressing a critical gap in the application of AI to consumer law, Nyay-Darpan comprises a dual-module system: it generates structured material summaries of consumer case files and retrieves the most semantically similar past judgments to support quasi-judicial decision-making. The system operationalizes legal logic by reflecting the core facts and disputes of consumer cases, thereby enhancing efficiency and reproducibility in legal reasoning and case management (Bhattacharyya et al., 8 Jul 2025).

1. System Architecture

Nyay-Darpan implements a bifurcated framework with two interacting components: a case-file summarization module and a similar-case retrieval module.

1.1 Case-File Summarization Module

This module processes concatenated “complaint” and “written statement” documents after pre-processing steps such as whitespace normalization and basic OCR corrections. LLMs including Llama 3.1 8B, DeepSeek R1 Distill Llama 8B, Ministral 8B, and Qwen 2.5 7B are utilized. Summarization proceeds via a two-step pipeline:

• Salient-Part Extraction: A part-wise prompting paradigm divides the case into six discrete components: Overview, Sector, Issues, Evidence by Complainant, Evidence by Opposing Party, Reliefs. Each is extracted sequentially.
• Abstractive Assembly: Extracted parts are concatenated to generate a cohesive “material summary.” The part-wise strategy demonstrates improved semantic coherence and information fidelity, outperforming monolithic prompts under both reference-based and human evaluation protocols.

1.2 Similar-Case Retrieval Module

• Sector Classification: Implementation of a Chain-of-Thought (CoT) prompting mechanism to determine the sector code (e.g., Banking and Financial Services 101) for each case.
• Embedding-Based Retrieval: Dense embeddings for both query and candidate cases are computed using all-MiniLM-L6-v2; cosine similarity is used for initial ranking.
• Lexical Retrieval: BM25 scoring quantifies term overlap relevance.
• Hybrid Scoring: The final retrieval employs a weighted sum of semantic and lexical similarity scores:

$$s_\text{hyb}(q,c) = \lambda\,\cos(\mathbf{h}_q,\mathbf{h}_c) + (1-\lambda)\,\frac{\mathit{BM25}(q,c)}{\max_{c'}\mathit{BM25}(q,c')}$$

with $\lambda = 0.5$. The top-5 candidates sharing a sector are returned.

2. Formal Objectives and Quantitative Functions

2.1 Summarization Loss

The summarization objective is to minimize the negative log-likelihood of the target summary tokens conditioned on the case input and system prompt:

$$\mathcal{L}_\mathrm{sum} = -\sum_{t=1}^{T} \log P(y_t \mid y_{<t}, x)$$

where $x$ denotes the processed case text plus prompt and $y_{1\ldots T}$ represents the gold summary token sequence.

2.2 Retrieval Scoring and Evaluation

• Cosine Similarity: Quantifies semantic relatedness in embedding space.
• Precision@5: Fraction of the top-5 retrieved cases judged relevant by legal domain experts.

3. Dataset Resources

3.1 Consumer Case Files and Material Summaries (CCFMS)

• 152 expert-curated case files across 23 consumer-protection sectors (e-commerce, banking, insurance, telecom, etc.).
• Each file includes legal expert-authored material summaries and annotations for six mandatory components, with five “gold-standard” most similar cases per query.

3.2 Consumer Case Judgement Database

• 570 case judgments with validated sector codes, spanning the same 23 sectors.
• Forms the retrieval backend for similarity computations.

3.3 Annotation Protocols and Pre-Processing

• Summaries authored and independently verified by legal practitioners, under detailed expert guidelines.
• Text preprocessing includes non-UTF8 character removal and sentence segmentation.
• Inference settings: temperature 0.7, top_p 0.95, top_k 50, with token limits per section (e.g., 512 for Overview).

4. Evaluation Metrics and Empirical Performance

4.1 Summarization Quality

• Reference-Based Metrics: ROUGE-1 (54.01), ROUGE-2 (24.34), ROUGE-L (23.80), BLEU-1 (37.28), BERTScore (97.18) for the Llama 3.1 8B + Partwise + SR configuration.
• Reference-Free (LLM) Metrics: Eight evaluation dimensions using GPT-4O-Mini (overview accuracy, oversimplification, etc.).
• Human Evaluations: Likert (1–5) and binary scoring by practitioners; average Likert scores > 4.0 and binary metrics > 0.67.

Strong Spearman correlation between human and LLM-based metrics (from ρ = 0.43 for Issues Accuracy to ρ = 0.86 for Sector Relevance) demonstrates alignment of evaluation modalities.

4.2 Retrieval Performance

• Embedding only: Precision@5 ~ 0.55–0.76
• BM25 only: Precision@5 ~ 0.56–0.79
• Hybrid scoring: Precision@5 ~ 0.44–0.65, with DeepSeek 8B + CoT giving the highest F1.
• Aggregate retrieval accuracy: Approximately 75%.

4.3 Error Taxonomy

• Sector misclassification is a principal error source, notably in cases involving confusion between “Banking” and “Insurance.”
• Llama 3.1 8B demonstrates strong fact extraction but is less effective on reasoning tasks compared to CoT-tuned models.
• Hybrid retrieval achieves a favorable balance between semantic and lexical similarity.

5. Limitations and Open Challenges

• Input Dependence: Performance is adversely affected by incomplete or low-quality scans.
• Content Hallucination: Structured LLM-generated summaries risk omitting domain-specific legal nuances.
• Sector and Jurisdictional Variability: Reasoning protocols differ among Indian consumer commissions.
• Evaluator Reliability: Legal practitioner evaluations were performed by a single annotator, limiting inter-annotator validity.

6. Prospects for Future Work and Deployment

• Technical Enhancements: Integration of clustering mechanisms (e.g., K-means over embeddings) for increased retrieval diversity; dataset augmentation with multi-rater annotations and inter-annotator agreement computation; expert-in-the-loop prompt refinement; retrieval-augmented generation for joint summarization and retrieval workflows.
• System Deployment: Embedding Nyay-Darpan within court case-management systems to automate summary-generation and precedent retrieval pipelines, with editable outputs for practitioner validation.
• Ethics and Compliance: Mandatory labeling of AI-generated summaries and required human verification pre-citation; deployment restricted to public documents or established confidentiality protocols for sealed cases.

A plausible implication is that Nyay-Darpan substantially lowers entry barriers for advanced AI research in Indian consumer law by releasing both its codebase and the CCFMS dataset, thereby facilitating reproducibility and extension in this underexplored legal subfield (Bhattacharyya et al., 8 Jul 2025).