Towards Better UD Parsing: Deep Contextualized Word Embeddings, Ensemble, and Treebank Concatenation (1807.03121v3)

Abstract: This paper describes our system (HIT-SCIR) submitted to the CoNLL 2018 shared task on Multilingual Parsing from Raw Text to Universal Dependencies. We base our submission on Stanford's winning system for the CoNLL 2017 shared task and make two effective extensions: 1) incorporating deep contextualized word embeddings into both the part of speech tagger and parser; 2) ensembling parsers trained with different initialization. We also explore different ways of concatenating treebanks for further improvements. Experimental results on the development data show the effectiveness of our methods. In the final evaluation, our system was ranked first according to LAS (75.84%) and outperformed the other systems by a large margin.

• The paper’s main contribution is integrating deep contextualized word embeddings, ensemble methods, and treebank concatenation to enhance multilingual UD parsing.
• It employs diverse parser ensembles that average predictions to achieve robust parsing performance, notably reaching an average LAS of 75.84%.
• The study demonstrates innovative treebank concatenation techniques to combine datasets, improving parsing in low-resource languages despite annotation variances.

Towards Better UD Parsing: Deep Contextualized Word Embeddings, Ensemble, and Treebank Concatenation

The paper under review presents a robust system, HIT-SCIR, which was submitted to the CoNLL 2018 shared task on Multilingual Parsing from Raw Text to Universal Dependencies. This system builds upon the established success of the Stanford system that triumphed in the CoNLL 2017 shared task, introducing crucial refinements that elevate its performance in parsing multilingual text data. The modifications revolve around the integration of deep contextualized word embeddings, parser ensemble methodologies, and creative treebank concatenation strategies.

Methodological Enhancements

1. Deep Contextualized Word Embeddings: The integration of ELMo embeddings, as pioneered by Peters et al. (2018), enhances both part-of-speech tagging (POS) and dependency parsing. ELMo creates context-rich word representations by training on large-scale raw data using a bidirectional LSTM. The paper emphasizes that these embeddings were incorporated as static representations in their system without parameter tuning, simplifying the processing pipeline while delivering substantial improvements in both syntactic and semantic tasks.
2. Parser Ensemble: The practice of training multiple parsers with varied initializations and averaging their outputs (softmaxed scores) is employed to enhance parsing robustness. This ensemble approach capitalizes on recent findings (Reimers and Gurevych, 2017; Liu et al., 2018) demonstrating that varying initialization leads to diversity in predictions, which, when combined, produce a more accurate and generalizable model.
3. Treebank Concatenation: The shared task provided multiple treebanks for some languages, differing by domain or family. Cross-domain and cross-lingual treebank concatenation leverages complementary datasets to bolster parsing accuracy, notably across low-resource languages. Although treebank concatenation shows promising results, the authors acknowledge that vocabulary and annotation divergences may sometimes limit performance gains.

Experimentation and Results

Experimentation was rigorous, with a detailed analysis of each enhancement's impact on POS tagging and dependency parsing across multiple development data sets. The system achieved an averaged LAS (Labeled Attachment Score) of 75.84% on the official test set, positioning it at the top with a significant performance margin over other submissions. The careful orchestration of ELMo, ensemble parsing, and data strategy demonstrated through quantitative analyses underscores this system's effectiveness.

Implications and Future Directions

The results underscore the utility of contextual embeddings and hybrid parsing strategies in advancing natural language understanding across diverse languages. This work opens avenues for further exploration into:

• Adaptive integration of ELMo or other contextual embeddings in various parsing scenarios including non-traditional text domains.
• Advanced ensembling techniques that might dynamically adapt the contribution of individual models based on contextual relevance.
• Automated selection and combination methods for treebank concatenation to mitigate the manual trial-and-error process currently necessary.

Conclusion

This paper’s technical richness and the strategic deployment of recent advancements highlight its contributions to the parsing community, providing insights valuable for enhancing both practical applications and our theoretical understanding of multilingual parsing. Future developments may build upon this foundational work, exploring more nuanced applications of contextual embeddings and optimization strategies in dynamic multilingual environments.