Measuring the Benefits of State-of-the-Art Reasoning Models for Complex Legal Annotations

Evaluate the extent to which state-of-the-art reasoning models improve performance on complex legal annotation tasks that require legal reasoning, relative to existing approaches and baselines, given the current lack of adequate assessment.

Background

The paper reports that general-purpose models struggle with complex legal annotation tasks, while fine-tuned, domain-specific models can perform near human level. It suggests that newer reasoning-focused models might do better but notes this has not been sufficiently evaluated.

A rigorous evaluation of such models on complex legal reasoning annotations would clarify their practical utility and guide model selection and training strategies for legal NLP.