From Query to Counsel: Structured Reasoning with a Multi-Agent Framework and Dataset for Legal Consultation

Abstract: Legal consultation question answering (Legal CQA) presents unique challenges compared to traditional legal QA tasks, including the scarcity of high-quality training data, complex task composition, and strong contextual dependencies. To address these, we construct JurisCQAD, a large-scale dataset of over 43,000 real-world Chinese legal queries annotated with expert-validated positive and negative responses, and design a structured task decomposition that converts each query into a legal element graph integrating entities, events, intents, and legal issues. We further propose JurisMA, a modular multi-agent framework supporting dynamic routing, statutory grounding, and stylistic optimization. Combined with the element graph, the framework enables strong context-aware reasoning, effectively capturing dependencies across legal facts, norms, and procedural logic. Trained on JurisCQAD and evaluated on a refined LawBench, our system significantly outperforms both general-purpose and legal-domain LLMs across multiple lexical and semantic metrics, demonstrating the benefits of interpretable decomposition and modular collaboration in Legal CQA.

• The paper introduces a modular multi-agent architecture (JurisMA) that decomposes complex legal queries into structured reasoning steps using semantic graphs.
• The paper demonstrates significant performance gains through contrastive supervision with DPO on the novel, expert-annotated JurisCQAD dataset for Chinese law.
• The paper validates its approach with strong cross-lingual generalization and superior human evaluations, underscoring practical benefits in transparent legal consulting.

Structured Legal Reasoning via Modular Multi-Agent Collaboration: An Analysis of JurisMA and JurisCQAD

Introduction and Motivation

Legal consultation question answering (Legal CQA) presents fundamental challenges distinct from conventional legal QA, primarily due to domain intricacies such as task composition, contextual dependence, and the scarcity of high-fidelity, scenario-based data. The paper "From Query to Counsel: Structured Reasoning with a Multi-Agent Framework and Dataset for Legal Consultation" (2604.10470) addresses these issues by proposing a modular multi-agent architecture—JurisMA—and releasing a large-scale, expert-annotated dataset—JurisCQAD—specifically tailored to Chinese legal consultation.

Figure 1: An illustrative example of legal consultation task decomposition, highlighting key challenges, limitations of prior approaches.

This figure contextualizes the limitations of classical approaches, which either rely on unsupervised domain pretraining or pipeline retrieval augmentation, both failing to address the nuanced, context-rich requirements of real-world legal queries (see Figure 1).

Methodological Advances: JurisMA Framework

The JurisMA system is architected to decompose complex legal consultations into a sequence of structured reasoning steps, instantiated as an element graph comprising nodes for entities, events, user intents, and legal issues.

Figure 2: Overview of JurisMA, a multi-agent framework that parses legal queries into element graphs, refines drafts via agent collaboration, and outputs a final legal opinion with supporting statutes.

Semantic Graph Construction

A dedicated Element Agent first extracts key legal elements and relationships from the user query, encoding the result as an interpretable JSON-based graph. This structured representation is serialized and concatenated with the original query to serve as rich, context-grounded input for downstream decision processes.

Multi-Agent Iterative Optimization

JurisMA's core comprises several specialized agents coordinated by a centralized Manager Agent:

• Draft Agent: Generates initial response leveraging the element graph.
• FormatCheck Agent: Targets linguistic and organizational clarity through focused revision suggestions.
• LawSearch Agent: Incorporates statutory grounding by retrieving relevant legal provisions.
• ContentCheck Agent: Finalizes the output, ensuring professional fluency and dual-section formatting: advisory opinion and legal basis.

By adopting dynamic agent routing and iterative revision rather than a monolithic pipeline, JurisMA supports controllable, modular optimization and explicit legal reasoning traceability.

Training with Contrastive Supervision and DPO

The underlying LLM (Qwen2.5 series) is trained using Direct Preference Optimization (DPO) on the JurisCQAD dataset, which comprises over 43,000 real Chinese legal consultation queries, each annotated with both expert-validated positive and adversarially constructed negative responses. This triplet-based supervision facilitates sharp discrimination between nuanced legal judgments.

Dataset: JurisCQAD

JurisCQAD embodies several critical advances:

• Sourced from authentic, post-2021 Civil Code legal consultation data.
• Each instance is represented as a (query, positive response, negative response) triplet, with negative samples adversarially generated and human-verified to contain subtle legal or factual flaws.
• Covers major legal topoi: contract law, torts, family law, labor, and more.

This dataset structure supports robust supervised learning for open-ended, generative Legal CQA—addressing limitations in both size and annotation quality present in prior resources (e.g., LawGPT, LawBench).

Experimental Results

JurisMA, fine-tuned on JurisCQAD, is evaluated against both generalist LLMs (GPT-4o, Qwen3-14B) and leading legal-specific LLMs (ChatLaw-33B, Hanfei, Lawyer-LLaMA, etc.), as well as agent-based/retrieval baselines (e.g., ReAct, MMEP). All models are assessed in a unified, zero-shot setting on a corrected version of LawBench.

Figure 3: Rouge-L and BertScore comparison before and after DPO across Qwen2.5 models (3B/7B/14B).

Notable findings include:

• JurisMA achieves the highest scores across principal text generation metrics (Rouge, Bleu-2/N, BertScore, LLMScore) and is statistically superior to all baselines (see Table 2 of the paper).
• Fine-tuning with JurisCQAD via DPO yields absolute performance gains (e.g., +4.47 Rouge-L, +2.51 BertScore on Qwen2.5-3B), while fine-tuning with LawGPT data results in pronounced regression.
• Human evaluation confirms JurisMA’s advantage on legal soundness, explicit reasoning, and completeness.
• Ablation demonstrates the necessity of all core modules: removal of the legal element graph, manager agent, or iterative revision induces substantial performance declines (up to –11.18 Rouge-L, –5.49 BertScore).

Qualitative Analysis and Generalization

A representative case study Figure 4 involving a time-sensitive background check query regarding pre-2011 drunk driving demonstrates that JurisMA alone delivers the correct conclusion with statutory citation and invocation of non-retroactivity, outperforming both GPT-4o and LawGPT by a meaningful margin.

Figure 4: Case study comparing model-generated responses to a time-sensitive legal query. The example illustrates differences in factual interpretation, statutory grounding, and reasoning structure.

Further, JurisMA exhibits strong cross-lingual and cross-jurisdictional generalization, outperforming both general and domain-specific LLMs on LegalBench-RuleQA (English), LawBench Reading Comprehension (paragraph-level), and KoBLEX (Korean legal QA).

Theoretical and Practical Implications

The JurisMA framework illustrates that explicit task decomposition, structured representational grounding, and modular agent cooperation confer measurable benefits for complex legal CQA tasks. The use of semantic graphs aligns with cognitive/planning theories of legal reasoning, while iterative agent-based revision approaches the organizational rigor of professional consultancy. JurisCQAD’s design philosophy and annotation protocol set a new paradigm for high-value contrastive supervision in specialty QA.

Practically, this architecture increases transparency, controllability, and legal compliance—all essential for mission-critical deployments. The modular design facilitates adaptation to new jurisdictions, languages, or legal subdomains.

Limitations and Future Directions

The modular, multi-agent design introduces additional inference latency. The coverage of rare legal circumstances, though broad, can benefit from targeted supplementation. For full deployment in high-stakes contexts, continued model supervision via real-world feedback and longitudinal tracking of statutory updates will be required.

Future work should emphasize:

• Real-time efficiency optimization for the multi-agent loop.
• Expansion of JurisCQAD to low-frequency or emergent legal issues.
• Extension to multilingual, multi-jurisdictional scenarios with transformer-based cross-system adaptation.
• Integration with retrieval-augmented verifiable pipelines for up-to-date statutory coverage.

Conclusion

The combination of the JurisMA multi-agent architecture and the JurisCQAD dataset yields significant advances in the domain of Legal CQA, as evidenced by substantial gains over both general and legal-specific LLMs in comprehensive benchmarks. The approach’s interpretability, modularity, and empirical effectiveness position it as a compelling template for future AI systems requiring transparent and accountable domain reasoning in law and other high-stakes applications.