Large-Scale Analysis of the Accuracy of the Journal Classification Systems of Web of Science and Scopus (1511.00735v2)

Abstract: Journal classification systems play an important role in bibliometric analyses. The two most important bibliographic databases, Web of Science and Scopus, each provide a journal classification system. However, no study has systematically investigated the accuracy of these classification systems. To examine and compare the accuracy of journal classification systems, we define two criteria on the basis of direct citation relations between journals and categories. We use Criterion I to select journals that have weak connections with their assigned categories, and we use Criterion II to identify journals that are not assigned to categories with which they have strong connections. If a journal satisfies either of the two criteria, we conclude that its assignment to categories may be questionable. Accordingly, we identify all journals with questionable classifications in Web of Science and Scopus. Furthermore, we perform a more in-depth analysis for the field of Library and Information Science to assess whether our proposed criteria are appropriate and whether they yield meaningful results. It turns out that according to our citation-based criteria Web of Science performs significantly better than Scopus in terms of the accuracy of its journal classification system.

• The paper conducts a large-scale analysis evaluating the accuracy of journal classification systems in Web of Science and Scopus using novel citation-based criteria.
• Key results indicate Web of Science demonstrates superior classification accuracy, particularly in identifying misclassified journals, with Scopus flagging over double the percentage of journals meeting the primary misclassification criterion.
• The study recommends more stringent, citation-guided classification by databases like Scopus, calls for transparency in methods, and suggests future research directions using text-based analysis or publication-level systems.

Analysis of WoS and Scopus Journal Classification Systems

This paper conducts a comprehensive evaluation of the journal classification systems used by two prominent bibliographic databases: Web of Science (WoS) and Scopus. The paper addresses the lack of systematic analysis regarding the accuracy of these classification systems, which is crucial for bibliometric research and scholarly communications. The researchers developed two criteria grounded in citation analysis to investigate the alignment between a journal's classification and its citation relations, providing insights into the respective accuracies of the WoS and Scopus systems.

The key findings of the paper reveal that WoS demonstrates superior performance when compared to Scopus, particularly in terms of Criterion I, which flags journals with weak connections to their assigned categories. Statistical analysis across varying thresholds (𝛼 = 0.05, 0.1, and 0.2) consistently indicates that Scopus misclassifies a significantly higher number of journals than WoS. Specifically, the percentage of journals and journal-category assignments that satisfy Criterion I is over double for Scopus compared to WoS.

The results of Criterion II, which identifies journals strongly related to categories to which they are not assigned, show a more favorable performance for both databases. However, WoS still performs slightly better than Scopus. This indicates that both databases generally categorize journals correctly when strong citation links are present but suggests a greater leniency or perhaps an over-assignment of categories in Scopus.

By combining both criteria, the paper exposes journals with the most questionable classifications. When a journal meets both Criterion I and II conditions, it signals even greater misclassification concern. The analysis identifies only one such journal in WoS, while Scopus identifies thirty-two, further evidencing WoS's more accurate classification framework.

The implications of these findings are notable. They advocate for a more stringent application of journal-category assignments, particularly in Scopus, possibly guided by citation analysis to improve accountability. Additionally, the discussion opens a field for future research applications, such as using text-based and expert-based approaches for better assessment in fields with limited citation relationships or the evaluation of newer publications.

The paper calls attention to the necessity for transparency in the methods utilized by both WoS and Scopus to construct their classification systems, emphasizing that accurate classifications are crucial not only for bibliometricians but also for researchers relying on comprehensive, field-related literature indicators.

The limitations of this research, as identified by the authors, revolve around the singular focus on citation-based analysis and the inherent biases due to database coverage differences. Future research could consider addressing these by employing more holistic methodologies incorporating text analysis or by explicitly considering database-specific biases. Furthermore, understanding the effect of varied category sizes and their definitions could provide more inclusive insights into classification dynamics.

Ultimately, as the landscape of scientific publication shifts with the rise of multidisciplinary journals, this paper underscores the importance of evolving classification systems, perhaps pivoting towards more sophisticated publication-level systems to enhance the precision of bibliometric analyses.