Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
51 tokens/sec
GPT-4o
60 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
8 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain (2301.13126v3)

Published 30 Jan 2023 in cs.CL, cs.AI, and cs.LG

Abstract: Lately, propelled by the phenomenal advances around the transformer architecture, the legal NLP field has enjoyed spectacular growth. To measure progress, well curated and challenging benchmarks are crucial. However, most benchmarks are English only and in legal NLP specifically there is no multilingual benchmark available yet. Additionally, many benchmarks are saturated, with the best models clearly outperforming the best humans and achieving near perfect scores. We survey the legal NLP literature and select 11 datasets covering 24 languages, creating LEXTREME. To provide a fair comparison, we propose two aggregate scores, one based on the datasets and one on the languages. The best baseline (XLM-R large) achieves both a dataset aggregate score a language aggregate score of 61.3. This indicates that LEXTREME is still very challenging and leaves ample room for improvement. To make it easy for researchers and practitioners to use, we release LEXTREME on huggingface together with all the code required to evaluate models and a public Weights and Biases project with all the runs.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (98)
  1. Aralegal-bert: A pretrained language model for arabic legal text. In NLLP.
  2. Proceedings of the Natural Legal Language Processing Workshop 2022. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates (Hybrid).
  3. Named entity recognition, linking and generation for greek legislation. In JURIX, volume 313 of Frontiers in Artificial Intelligence and Applications, pages 1–10. IOS Press.
  4. European Union language resources in Sketch Engine. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 2799–2803, Portorož, Slovenia. European Language Resources Association (ELRA).
  5. gabert — an irish language model. In International Conference on Language Resources and Evaluation.
  6. Longformer: The long-document transformer. CoRR, abs/2004.05150.
  7. On the dangers of stochastic parrots: Can language models be too big? In FAccT ’21: 2021 ACM Conference on Fairness, Accountability, and Transparency, Virtual Event / Toronto, Canada, March 3-10, 2021, pages 610–623. ACM.
  8. Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: on meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 5185–5198. Association for Computational Linguistics.
  9. Language Models are Few-Shot Learners. arXiv:2005.14165 [cs]. ArXiv: 2005.14165.
  10. Multilegalsbd: A multilingual legal sentence boundary detection dataset. In Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, ICAIL ’23, page 42–51, New York, NY, USA. Association for Computing Machinery.
  11. Ilias Chalkidis. 2023. Chatgpt may pass the bar exam soon, but has a long way to go for the lexglue benchmark. CoRR, abs/2304.12202.
  12. MultiEURLEX - a multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6974–6996, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  13. LEGAL-BERT: The muppets straight out of law school. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2898–2904, Online. Association for Computational Linguistics.
  14. Paragraph-level rationale extraction through regularization: A case study on European court of human rights cases. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 226–241, Online. Association for Computational Linguistics.
  15. LeXFiles and LegalLAMA: Facilitating English multinational legal language model development. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15513–15535, Toronto, Canada. Association for Computational Linguistics.
  16. LexGLUE: A benchmark dataset for legal language understanding in English. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4310–4330, Dublin, Ireland. Association for Computational Linguistics.
  17. FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing. arXiv:2203.07228 [cs]. ArXiv: 2203.07228.
  18. German’s next language model. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6788–6796, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  19. PaLM: Scaling Language Modeling with Pathways. arXiv:2204.02311 [cs]. ArXiv: 2204.02311.
  20. Resolving Legalese: A Multilingual Exploration of Negation Scope Resolution in Legal Documents.
  21. Victor Hugo Ciurlino. 2021. Bertbr: a pretrained language model for law texts. Master’s thesis, Universidade de Brasília.
  22. European Commission. 2005. A new framework strategy for multilingualism.
  23. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online. Association for Computational Linguistics.
  24. Spanish datasets for sensitive entity detection in the legal domain. In Proceedings of the Thirteenth International Conference on Language Resources and Evaluation (LREC’22), Marseille, France, june. European Language Resource Association (ELRA). Dataset URL: https://tinyurl.com/mv65cp66.
  25. BERTIN: efficient pre-training of a spanish language model using perplexity sampling. Proces. del Leng. Natural, 68:13–23.
  26. Bertje: A dutch bert model. ArXiv, abs/1912.09582.
  27. RobBERT: a Dutch RoBERTa-based Language Model. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3255–3265, Online. Association for Computational Linguistics.
  28. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pages 248–255. IEEE Computer Society.
  29. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  30. JuriBERT: A masked-language model adaptation for French legal text. In Proceedings of the Natural Legal Language Processing Workshop 2021, pages 95–101, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  31. A corpus for multilingual analysis of online terms of service. In Proceedings of the Natural Legal Language Processing Workshop 2021, pages 1–8. Dataset URL: http://claudette.eui.eu/corpora/.
  32. The birth of Romanian BERT. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4324–4328, Online. Association for Computational Linguistics.
  33. Candida Maria Greco and Andrea Tagarelli. 2023. Bringing order into the realm of transformer-based language models for artificial intelligence and law. Artificial Intelligence and Law, To be published.
  34. Legalbench: Prototyping a collaborative benchmark for legal reasoning. CoRR, abs/2209.06120.
  35. Spanish legalese language model and corpora. ArXiv, abs/2110.12201.
  36. MarIA: Spanish language models. ArXiv, abs/2107.07253.
  37. DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing. CoRR, pages 1–17.
  38. Deberta: Decoding-enhanced bert with disentangled attention. ArXiv, abs/2006.03654.
  39. Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset. ArXiv:2207.00220 [cs].
  40. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
  41. XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In ICML, volume 119 of Proceedings of Machine Learning Research, pages 4411–4421. PMLR.
  42. LegalRelectra: Mixed-domain Language Modeling for Long-range Legal Text Comprehension. ArXiv:2212.08204 [cs].
  43. Dane: A named entity resource for danish. In International Conference on Language Resources and Evaluation.
  44. A multi-task benchmark for korean legal language understanding and judgement prediction. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
  45. Efficient long-text understanding with short-text models. Transactions of the Association for Computational Linguistics, 11:284–299.
  46. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651–3657, Florence, Italy. Association for Computational Linguistics.
  47. Natural Language Processing in the Legal Domain.
  48. Reduced, reused and recycled: The life of a dataset in machine learning research. In NeurIPS Datasets and Benchmarks.
  49. Greek-bert: The greeks visiting sesame street. In 11th Hellenic Conference on Artificial Intelligence, SETN 2020, page 110–117, New York, NY, USA. Association for Computing Machinery.
  50. Predicting brazilian court decisions. PeerJ Computer Science, 8:e904. Dataset URL: https://github.com/proflage/predicting-brazilian-court-decisions.
  51. Holistic Evaluation of Language Models. ArXiv:2211.09110 [cs].
  52. Daniele Licari and Giovanni Comandé. 2022. ITALIAN-LEGAL-BERT: A pre-trained transformer language model for italian law. In EKAW (Companion), volume 3256 of CEUR Workshop Proceedings. CEUR-WS.org.
  53. RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR, abs/2111.09543(1).
  54. Lener-br: a dataset for named entity recognition in brazilian legal text. In International Conference on Computational Processing of the Portuguese Language, pages 313–323. Springer. Dataset URL: https://huggingface.co/datasets/lener_br.
  55. Playing with words at the national library of sweden - making a swedish bert. ArXiv, abs/2007.01658.
  56. CamemBERT: a tasty French language model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7203–7219, Online. Association for Computational Linguistics.
  57. jurBERT: A Romanian BERT model for legal judgement prediction. In Proceedings of the Natural Legal Language Processing Workshop 2021, pages 86–94, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  58. B.W. Matthews. 1975. Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochimica et Biophysica Acta (BBA) - Protein Structure, 405(2):442–451.
  59. Dávid Márk Nemeskey. 2020. Natural Language Processing Methods for Language Modeling. Ph.D. thesis, Eötvös Loránd University.
  60. Swiss-judgment-prediction: A multilingual legal judgment prediction benchmark. In Proceedings of the Natural Legal Language Processing Workshop 2021, pages 19–35, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  61. Joel Niklaus and Daniele Giofre. 2023. Can we pretrain a SotA legal language model on a budget from scratch? In Proceedings of The Fourth Workshop on Simple and Efficient Natural Language Processing (SustaiNLP), pages 158–182, Toronto, Canada (Hybrid). Association for Computational Linguistics.
  62. Automatic Anonymization of Swiss Federal Supreme Court Rulings. ArXiv:2310.04632 [cs].
  63. Multilegalpile: A 689gb multilingual legal corpus.
  64. An empirical study on cross-X transfer for legal judgment prediction. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 32–46, Online only. Association for Computational Linguistics.
  65. Named entity recognition in the Romanian legal domain. In Proceedings of the Natural Legal Language Processing Workshop 2021, pages 9–18, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  66. Multi-granular legal topic classification on Greek legislation. In Proceedings of the Natural Legal Language Processing Workshop 2021, pages 63–75, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  67. Transfer learning in biomedical natural language processing: An evaluation of BERT and elmo on ten benchmarking datasets. In BioNLP@ACL, pages 58–65. Association for Computational Linguistics.
  68. Slovakbert: Slovak masked language model. CoRR, abs/2109.15254.
  69. AI and the everything in the whole wide world benchmark. In NeurIPS Datasets and Benchmarks.
  70. Evaluating protein transfer learning with TAPE. In NeurIPS, pages 9686–9698.
  71. Scale: Scaling up the complexity for advanced language model evaluation.
  72. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108.
  73. ClassActionPrediction: A challenging benchmark for legal judgment prediction of class action cases in the US. In Proceedings of the Natural Legal Language Processing Workshop 2022, pages 31–46, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  74. Multi-lexsum: Real-world summaries of civil rights lawsuits at multiple granularities. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
  75. Czert – Czech BERT-like model for language representation. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 1326–1338, Held Online. INCOMA Ltd.
  76. Bertimbau: Pretrained bert models for brazilian portuguese. In Intelligent Systems: 9th Brazilian Conference, BRACIS 2020, Rio Grande, Brazil, October 20–23, 2020, Proceedings, Part I, page 403–417, Berlin, Heidelberg. Springer-Verlag.
  77. Andrea Tagarelli and Andrea Simeri. 2022. Lamberta: Law article mining based on bert architecture for the italian civil code. In ICRDL.
  78. EstBERT: A pretrained language-specific BERT for Estonian. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pages 11–19, Reykjavik, Iceland (Online). Linköping University Electronic Press, Sweden.
  79. Shavrina Tatiana and Malykh Valentin. 2021. How not to lie with a benchmark: Rearranging NLP leaderboards. In I (Still) Can’t Believe It’s Not Better Workshop at NeurIPS 2021, volume abs/2112.01342.
  80. Rena Torres Cacoullos. 2020. Code-switching strategies: Prosody and syntax. Frontiers in Psychology, 11.
  81. LLaMA: Open and Efficient Foundation Language Models.
  82. SUPERB-SG: enhanced speech processing universal performance benchmark for semantic and generative capabilities. In ACL (1), pages 8479–8492. Association for Computational Linguistics.
  83. Dimitrios Tsarapatsanis and Nikolaos Aletras. 2021. On the ethical limits of natural language processing on legal text. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3590–3599, Online. Association for Computational Linguistics.
  84. A multilingual approach to identify and classify exceptional measures against covid-19. In Proceedings of the Natural Legal Language Processing Workshop 2021, pages 46–62. Dataset URL: https://tinyurl.com/ycysvtbm.
  85. Design and implementation of german legal decision corpora. In Proceedings of the 13th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART,, pages 515–521. SciTePress.
  86. Thirty years of artificial intelligence and law: the third decade. Artificial Intelligence and Law, 30:561–591.
  87. Multilingual is not enough: Bert for finnish. ArXiv, abs/1912.07076.
  88. Superglue: A stickier benchmark for general-purpose language understanding systems. In NeurIPS, pages 3261–3275.
  89. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR (Poster). OpenReview.net.
  90. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red Hook, NY, USA. Curran Associates Inc.
  91. Lawformer: A pre-trained language model for chinese legal long documents. AI Open, 2:79–84.
  92. CLUE: A Chinese language understanding evaluation benchmark. In COLING, pages 4762–4772, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  93. mT5: A massively multilingual pre-trained text-to-text transformer. arXiv:2010.11934 [cs]. ArXiv: 2010.11934.
  94. SUPERB: speech processing universal performance benchmark. In Interspeech, pages 1194–1198. ISCA.
  95. Ying Yin and Ivan Habernal. 2022. Privacy-preserving models for legal natural language processing. In Proceedings of the Natural Legal Language Processing Workshop 2022, pages 172–183, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  96. CBLUE: A Chinese biomedical language understanding evaluation benchmark. In ACL, pages 7888–7915, Dublin, Ireland. Association for Computational Linguistics.
  97. When does pretraining help? assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law, ICAIL ’21, page 159–168, New York, NY, USA. Association for Computing Machinery.
  98. How does NLP benefit legal system: A summary of legal artificial intelligence. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5218–5230, Online. Association for Computational Linguistics.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Joel Niklaus (21 papers)
  2. Veton Matoshi (3 papers)
  3. Pooja Rani (20 papers)
  4. Andrea Galassi (9 papers)
  5. Matthias Stürmer (13 papers)
  6. Ilias Chalkidis (40 papers)
Citations (43)
View on Semantic Scholar