CodeQA: A Question Answering Dataset for Source Code Comprehension

Abstract: We propose CodeQA, a free-form question answering dataset for the purpose of source code comprehension: given a code snippet and a question, a textual answer is required to be generated. CodeQA contains a Java dataset with 119,778 question-answer pairs and a Python dataset with 70,085 question-answer pairs. To obtain natural and faithful questions and answers, we implement syntactic rules and semantic analysis to transform code comments into question-answer pairs. We present the construction process and conduct systematic analysis of our dataset. Experiment results achieved by several neural baselines on our dataset are shown and discussed. While research on question-answering and machine reading comprehension develops rapidly, few prior work has drawn attention to code question answering. This new dataset can serve as a useful research benchmark for source code comprehension.

• The paper introduces CodeQA, a large-scale dataset for source code comprehension using comment-derived QA pairs from GitHub repositories.
• It employs a multi-step methodology with semantic role labeling and templated transformations to generate diverse questions covering functionality, purpose, property, and workflow.
• Baseline models including CodeBERT are evaluated using metrics like BLEU, ROUGE, METEOR, Exact Match, and F1, underscoring improvements in automated code maintenance and bug detection.

CodeQA: A Question Answering Dataset for Source Code Comprehension

Introduction to CodeQA

The paper presents CodeQA, a large-scale dataset specifically designed to aid in the comprehension of source code by leveraging question-answer (QA) pairs. The primary objective is to bridge the gap between traditional text processing in QA systems and the intricacies of source code understanding. CodeQA provides a rich dataset containing free-form QA pairs derived from code comments across Java and Python repositories from GitHub. The dataset encompasses over 119,778 pairs in Java and 70,085 pairs in Python, aiming to facilitate advancements in QA tasks within software development contexts.

Dataset Construction

The construction of CodeQA is a multi-step process starting with the selection of well-documented code repositories from GitHub. Comments within these repositories serve as the basis for generating QA pairs, utilizing syntactic rules and semantic analysis to transform comments into questions and answers. The resulting QA pairs are designed to capture varied information, ranging in granularity from method-level to variable-level details. This process involves filtering comments, identifying potential answers through semantic role labeling, and applying templated transformations to generate diverse question types including Wh-questions and Yes/No questions.

Dataset Analysis and Categorization

CodeQA's distinctive feature is its diversity and the free-form nature of the QA pairs. The dataset is categorized into four primary types of comprehension tasks: functionality, purpose, property, and workflow. These categorizations reflect the multifaceted nature of code comments, enabling an extensive exploration of a code snippet's characteristics. Analysis reveals that a significant portion of dataset pairs deals with functionality, while others focus on purpose and properties intrinsic to code operations.

Baseline Models and Performance

The paper evaluates several baseline neural models including Seq2seq, Dual Encoder, Transformer, and CodeBERT. CodeBERT, a pre-trained bimodal model for both natural language and programming languages, shows superior performance across various metrics. The evaluation is based on automatic metrics such as BLEU, ROUGE, METEOR, Exact Match, and F1, complemented by human evaluation for fluency and correctness.

Implications and Future Work

CodeQA promises significant implications in enhancing software engineering tasks such as bug detection, specification inference, and automated code maintenance. It advocates for structured benchmarking in the development of models capable of accurately interpreting and interacting with code in a manner akin to traditional natural language. Future work includes expanding question diversity and improving both the quality of QA pairs and the models that leverage CodeQA.

Conclusion

CodeQA is a pioneering step towards systematic understanding and processing of source code through free-form question answering. The dataset's richness and diversity offer fertile ground for advancing both machine comprehension of code and the utility of QA systems within the coding landscape. By providing robust baselines and comprehensive evaluations, CodeQA lays the groundwork for future explorations into more sophisticated QA models and their applications in real-world coding environments.