Portuguese Word Embeddings: Evaluating on Word Analogies and Natural Language Tasks (1708.06025v1)

Abstract: Word embeddings have been found to provide meaningful representations for words in an efficient way; therefore, they have become common in Natural Language Processing sys- tems. In this paper, we evaluated different word embedding models trained on a large Portuguese corpus, including both Brazilian and European variants. We trained 31 word embedding models using FastText, GloVe, Wang2Vec and Word2Vec. We evaluated them intrinsically on syntactic and semantic analogies and extrinsically on POS tagging and sentence semantic similarity tasks. The obtained results suggest that word analogies are not appropriate for word embedding evaluation; task-specific evaluations appear to be a better option.

Portuguese Word Embeddings: Evaluation and Implications

In the paper "Portuguese Word Embeddings: Evaluating on Word Analogies and Natural Language Tasks," Hartmann et al. explore the efficacy of various word embedding models in processing the Portuguese language, covering both Brazilian (PT-BR) and European (PT-EU) variants. The paper stands out for its comprehensive evaluation involving both intrinsic methods, like word analogies, and extrinsic NLP tasks, such as Part-of-Speech (POS) tagging and semantic similarity.

The research evaluates 31 models across four embedding techniques: FastText, GloVe, Wang2Vec, and Word2Vec, targeting their strength in capturing syntactic and semantic nuances. The embeddings were trained on a robust corpus, amalgamating various genres from multiple sources, ensuring the data reflects linguistic diversity.

Key Findings

• Intrinsic vs. Extrinsic Evaluations: The paper reveals a divergence between intrinsic evaluations, where GloVe excelled, and extrinsic evaluations like POS tagging and semantic similarity, where Wang2Vec showed superior performance. This indicates intrinsic methods, such as word analogies, might not reliably predict downstream task performance.
• Model Performance: FastText demonstrated prowess in syntactic properties likely due to its morphological approach, yet struggled in downstream tasks when compared to Wang2Vec, which excelled in capturing structural syntactic properties.
• Dimensionality Impact: As expected, larger dimensionalities generally improved performance across tasks, though Word2Vec showed an unusual drop at higher dimensions for the POS task, suggesting that extremely large vectors might not always enhance performance.

Implications and Future Research

The findings prompt several insights for future work in word embeddings and NLP:

1. Task-Specific Evaluation: The results bolster the view that embedding evaluations should be closely tied to the specific tasks they aim to improve, rather than relying solely on general benchmarks like word analogies.
2. Corpus Composition: The combination of Brazilian and European Portuguese texts suggests that corpus size and diversity might outweigh the potential disadvantages of mixed dialects. Future embeddings can potentially harness cross-dialectal corpora for wider applicability without compromising performance.
3. Fine-Tuning and Optimization: Exploring alternative tokenization, normalization, or even lemmatization strategies may yield improvements in model training, especially for linguistically rich languages like Portuguese.

As a theoretical implication, the work underscores the complexity of embedding evaluations, highlighting the dependency of model choice on specific linguistic tasks rather than general performance metrics. Practically, researchers training NLP models for Portuguese should consider a contextual appraisal based on their target use rather than relying purely on conventional intrinsic evaluations.

By demystifying the alignment (or lack thereof) between intrinsic evaluations and task-specific performance, this paper clarifies pathways for designing more effective Portuguese language processing systems. Future developments in AI for NLP should accordingly adjust evaluation metrics to better reflect model application potential in diverse linguistic contexts.