Roget's Thesaurus and Semantic Similarity (1204.0245v1)

Abstract: We have implemented a system that measures semantic similarity using a computerized 1987 Roget's Thesaurus, and evaluated it by performing a few typical tests. We compare the results of these tests with those produced by WordNet-based similarity measures. One of the benchmarks is Miller and Charles' list of 30 noun pairs to which human judges had assigned similarity measures. We correlate these measures with those computed by several NLP systems. The 30 pairs can be traced back to Rubenstein and Goodenough's 65 pairs, which we have also studied. Our Roget's-based system gets correlations of .878 for the smaller and .818 for the larger list of noun pairs; this is quite close to the .885 that Resnik obtained when he employed humans to replicate the Miller and Charles experiment. We further evaluate our measure by using Roget's and WordNet to answer 80 TOEFL, 50 ESL and 300 Reader's Digest questions: the correct synonym must be selected amongst a group of four words. Our system gets 78.75%, 82.00% and 74.33% of the questions respectively.

• The paper demonstrates that a Roget’s Thesaurus-based approach attains a correlation of 0.878 with human judgments on semantic similarity benchmarks.
• It achieves success rates of 78.75% for TOEFL and 82.00% for ESL tests, significantly outperforming WordNet-based measures.
• The study validates Roget’s comprehensive taxonomy as a robust alternative for semantic analysis in NLP and suggests promising avenues for future research.

Roget’s Thesaurus and Semantic Similarity in NLP

The paper "Roget’s Thesaurus and Semantic Similarity" authored by Mario Jarmasz and Stan Szpakowicz, presents a computational approach to measure semantic similarity through the implementation of the 1987 computerized version of Roget's Thesaurus. The research aims to evaluate the effectiveness of this system in measuring semantic similarity, comparing its performance against traditional WordNet-based methods.

Semantic similarity, the extent to which two words can be interchanged without altering the meaning of an expression, forms a critical concept in NLP. The paper embarks on this exploration by building upon Miller and Charles’ existing semantic similarity framework and correlates the results with benchmark human judgments from the Miller and Charles list, as well as the broader Rubenstein and Goodenough’s 65 noun pairs.

Key Findings

1. Correlation with Human Judgment: The Roget-based system achieves a correlation of .878 with human evaluations on the Miller and Charles dataset, which is a laudable performance, closely paralleling the .885 accuracy when human assessments were solely employed. For Rubenstein and Goodenough’s data, the system attains a correlation of .818, reaffirming its efficacy.
2. Answering Standardized Tests: The system's capacity was further validated via multiple-choice synonym tests contained within various English proficiency settings (TOEFL, ESL, and Reader's Digest). The results show success rates of 78.75% for TOEFL and 82.00% for ESL questions, considerably outperforming WordNet-based measures, which lagged with maxima not exceeding 37%.

Implications and Analysis

This paper asserts the viability of Roget’s Thesaurus as an alternative to the frequently utilized WordNet lexicon for measuring semantic similarity. Specifically, it addresses certain limitations inherent in WordNet, such as the reliance on fixed semantic relations and the inability to address conceptual connections across different parts of speech. Roget’s, with its comprehensive taxonomy, allows for the exploration of such cross-categorical semantic links, enhancing its applicability in real-world language processing tasks.

The paper also emphasizes Roget's systematic concept hierarchy, which mitigates some deficiencies in WordNet's edge-counting methodology. WordNet's edge-based distance metrics assume uniform distances between nodes, which can distort semantic correlation. However, Roget’s more uniform and exhaustive structure provides a more reliable basis for edge counting, thus delivering strong results in standard tests.

Future Directions

The conclusions drawn pose significant implications for future NLP applications and lexicon development. Given the robustness of a Roget’s-based similarity metric, future research should contemplate the integration of its taxonomy with machine learning approaches to further automate and enhance semantic analysis. Additionally, expanding the scope of the current method to accommodate a broader array of language constructs could bolster its utility in dynamic and complex linguistic environments. Emerging AI models and algorithms might also leverage these insights to augment contextual understanding where comprehensive similarity assessments are crucial.

In summary, this paper contributes valuable discourse around semantic similarity measurement using traditional lexicons in contemporary computational contexts. By affirming Roget’s Thesaurus as a viable tool for semantic analysis, it opens new avenues for exploration in the enhancement of linguistic algorithms and semantic comprehension in AI systems.