Where there's a will there's a way: ChatGPT is used more for science in countries where it is prohibited (2406.11583v4)

Abstract: Regulating AI is a key societal challenge, but which regulation methods are effective is unclear. This study measures the effectiveness of restricting AI services geographically, focusing on ChatGPT. OpenAI restricts ChatGPT access in several countries, including China and Russia. If restrictions are effective, ChatGPT use should be minimal in these countries. We measured use with a classifier based on distinctive word usage found in early versions of ChatGPT, e.g. "delve." We trained the classifier on pre- and post-ChatGPT "polished" abstracts and found it outperformed GPTZero and ZeroGPT on validation sets, including papers with self-reported AI use. Applying the classifier to preprints from Arxiv, BioRxiv, and MedRxiv showed ChatGPT was used in about 12.6% of preprints by August 2023, with 7.7% higher usage in restricted countries. The gap appeared before China's first major legal LLM became widely available. To test the possibility that, due to high demand, use in restricted countries would have been even higher without restrictions, we compared Asian countries with high expected demand (where English is not an official language) and found that use was higher in those with restrictions. ChatGPT use was correlated with higher views and downloads, but not citations or journal placement. Overall, restricting ChatGPT geographically has proven ineffective in science and possibly other domains, likely due to widespread workarounds.

• The paper introduces a novel classifier that detects ChatGPT-modified abstracts across major research repositories.
• The paper finds that ChatGPT usage in restricted countries like China reaches 22.29%, significantly higher than in non-restricted areas.
• The paper suggests that geographical restrictions on AI are ineffective, prompting a re-evaluation of current regulatory frameworks.

Geographical Restrictions Ineffective in Limiting ChatGPT Use in Scientific Research

The paper by Honglin Bao, Mengyi Sun, and Misha Teplitskiy, titled "Where there's a will there's a way: ChatGPT is used more for science in countries where it is prohibited," presents a rigorous analysis of the effectiveness of geographical restrictions on ChatGPT usage in the scientific domain. This research, which builds on OpenAI's decision to restrict access to ChatGPT in specific countries like China and Russia, leverages a bespoke classifier to detect ChatGPT-modified scientific abstracts across a substantial dataset.

Methodology

The researchers developed a classifier using distinct word usage patterns that are characteristic of ChatGPT-generated text, such as the overrepresentation of words like "delve." The classifier was trained on pre- and post-ChatGPT-polished abstracts from Arxiv, BioRxiv, and MedRxiv datasets and validated against existing detectors like GPTZero and ZeroGPT. Notably, the authors demonstrated that their classifier markedly outperformed these off-the-shelf tools in identifying AI-modified abstracts across various scientific corpora.

Key Findings

The paper revealed that by August 2023, approximately 12.6% of scientific preprints utilized ChatGPT, with notably higher usage in countries where it is legally restricted. Specifically, in China, the usage rate of ChatGPT in scientific abstracts was around 22.29%, compared to 11.07% in countries with legal access. This discrepancy was observed even before the widespread availability of China's domestic LLM, Ernie Bot. Furthermore, a comparison among Asian countries with high expected demand for ChatGPT (where English is not the official language) indicated higher usage in those with restrictions, which suggests that regulatory efforts may have been circumvented through various workarounds.

Implications

The results challenge the effectiveness of geographical restrictions intended to limit the use of AI tools like ChatGPT. Despite considerable regulatory efforts, ChatGPT adoption in restricted countries surpassed that in non-restricted countries. This suggests that scientific communities in these countries have effectively employed workarounds to bypass restrictions, such as acquiring approved accounts, proxy servers, and other means.

The empirical findings hold significant implications for policymakers. The ineffectiveness of geographical restrictions signals a need to rethink regulatory strategies concerning advanced AI tools. Regulations based on geographical locations may not only be easy to circumvent but also could inadvertently promote the development of secondary markets for workarounds.

Academic Outcomes

The paper also investigated the impact of ChatGPT usage on various academic metrics, such as citations, journal placement, and attention metrics within the BioRxiv corpus. The results indicated that ChatGPT use correlated with increased attention measures – about two percentiles higher views of abstracts, PDFs, and online full texts – but did not significantly affect citations or journal impact factors. This suggests that while ChatGPT may enhance the visibility and dissemination of scientific work, it does not necessarily translate to higher scholarly impact.

Future Directions

The paper opens several avenues for future research. Empirical studies on the specific types of workarounds used to bypass geographical restrictions and their prevalence could provide deeper insights into the dynamics of AI tool adoption under restrictive regimes. Furthermore, examining the granularity of AI tools' use in various research tasks beyond abstract writing could offer a more comprehensive understanding of their impact on scientific productivity and integrity.

Overall, the findings prompt a re-evaluation of current regulatory frameworks governing AI tools, highlighting the necessity for more nuanced and robust approaches that consider the multifaceted nature of technology adoption and compliance. The rapid proliferation of workarounds underscores the challenges of enforcing geographically-based restrictions in the digital age, necessitating innovative solutions that balance access with oversight.