Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processing (2101.03289v5)

Abstract: We introduce Trankit, a light-weight Transformer-based Toolkit for multilingual NLP. It provides a trainable pipeline for fundamental NLP tasks over 100 languages, and 90 pretrained pipelines for 56 languages. Built on a state-of-the-art pretrained LLM, Trankit significantly outperforms prior multilingual NLP pipelines over sentence segmentation, part-of-speech tagging, morphological feature tagging, and dependency parsing while maintaining competitive performance for tokenization, multi-word token expansion, and lemmatization over 90 Universal Dependencies treebanks. Despite the use of a large pretrained transformer, our toolkit is still efficient in memory usage and speed. This is achieved by our novel plug-and-play mechanism with Adapters where a multilingual pretrained transformer is shared across pipelines for different languages. Our toolkit along with pretrained models and code are publicly available at: https://github.com/nlp-uoregon/trankit. A demo website for our toolkit is also available at: http://nlp.uoregon.edu/trankit. Finally, we create a demo video for Trankit at: https://youtu.be/q0KGP3zGjGc.

Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processing

The paper presents Trankit, a Transformer-based toolkit designed for multilingual NLP. It offers a trainable pipeline capable of executing fundamental NLP tasks over 100 languages and comes with 90 pretrained pipelines for 56 languages. Trankit boasts improved performance on sentence segmentation, part-of-speech (POS) tagging, morphological feature tagging, and dependency parsing compared to existing multilingual NLP toolkits. Furthermore, it maintains competitive results in tokenization, multi-word token expansion, and lemmatization over 90 Universal Dependencies treebanks.

Key Components and Architecture

Trankit capitalizes on the state-of-the-art multilingual pretrained transformer, XLM-Roberta, and introduces a novel plug-and-play mechanism with Adapters. By effectively sharing a single multilingual pretrained transformer across different pipelines, Trankit achieves memory efficiency and speed. Adapters serve as small networks inserted throughout the transformer layers, facilitating language-specific feature extraction while mitigating extensive memory consumption. This methodological choice enables simultaneous loading of different pipeline components for multilingual tasks, regardless of language.

The architecture includes several key components:

1. Joint Token and Sentence Splitter: Utilizing a wordpiece-based method, this component processes input sequences to determine token, multi-word token, and sentence boundaries.
2. Joint Model for POS Tagging, Morphological Tagging, and Dependency Parsing: This model jointly executes multiple tasks at the sentence level, enhancing performance by reducing error propagation.
3. Named Entity Recognizer (NER): Implementing a sequence labeling architecture with a Conditional Random Field, this module achieves competitive results on public datasets.
4. Adapters: Crucially, adapters allow each language and component to maintain task-specific features without necessitating multiple versions of the core transformer model.

Results and Implications

Trankit demonstrates substantial improvements over current state-of-the-art toolkits like Stanza, particularly in sentence segmentation and dependency parsing. Table 1 from the paper illustrates its performance across 90 Universal Dependencies treebanks, with significant enhancements in POS and morphological tagging. Additionally, Trankit achieves competitive NER performance across diverse languages, further validating the effectiveness of leveraging pretrained transformers in multilingual settings.

The implications of Trankit's development are multifaceted. Practically, it offers a resource-efficient solution for multilingual NLP, enabling users to execute comprehensive language processing tasks without exorbitant memory demands. Theoretically, it reaffirms the potential of integrating adapter-based architectures within large-scale pretrained models, suggesting avenues for future research in optimizing NLP pipelines.

Future Directions

The paper suggests potential enhancements to Trankit, including exploring alternative pretrained transformers such as mBERT and XLM-Roberta_large. Furthermore, expanding the NER capabilities for additional languages and incorporating more NLP tasks stand as future objectives.

In conclusion, Trankit serves as a significant contribution within the field of multilingual NLP, advancing performance benchmarks across a wide array of tasks while remaining resource-conscious. It provides a robust framework for tackling language diversity with efficiency and accuracy, and its architecture could inspire further innovations in adapter-based transformer applications.