An Empirical Validation of Cognitive Complexity as a Measure of Source Code Understandability (2007.12520v1)

Abstract: Background: Developers spend a lot of their time on understanding source code. Static code analysis tools can draw attention to code that is difficult for developers to understand. However, most of the findings are based on non-validated metrics, which can lead to confusion and code, that is hard to understand, not being identified. Aims: In this work, we validate a metric called Cognitive Complexity which was explicitly designed to measure code understandability and which is already widely used due to its integration in well-known static code analysis tools. Method: We conducted a systematic literature search to obtain data sets from studies which measured code understandability. This way we obtained about 24,000 understandability evaluations of 427 code snippets. We calculated the correlations of these measurements with the corresponding metric values and statistically summarized the correlation coefficients through a meta-analysis. Results: Cognitive Complexity positively correlates with comprehension time and subjective ratings of understandability. The metric showed mixed results for the correlation with the correctness of comprehension tasks and with physiological measures. Conclusions: It is the first validated and solely code-based metric which is able to reflect at least some aspects of code understandability. Moreover, due to its methodology, this work shows that code understanding is currently measured in many different ways, which we also do not know how they are related. This makes it difficult to compare the results of individual studies as well as to develop a metric that measures code understanding in all its facets.

Empirical Validation of Cognitive Complexity in Source Code Understandability

The paper "An Empirical Validation of Cognitive Complexity as a Measure of Source Code Understandability," authored by Marvin Muñoz Barón, Marvin Wyrich, and Stefan Wagner, presents a meticulous investigation aimed at validating Cognitive Complexity as an effective measure for source code understandability. This metric, widely integrated into static code analysis tools, is designed explicitly to identify difficult-to-understand code sections.

Critical Analysis and Findings

The authors conducted a systematic literature search and meta-analysis involving approximately 24,000 evaluations of 427 code snippets from various studies. The aim was to ascertain the correlation between Cognitive Complexity and several understandability proxy variables including comprehension time, correctness of understanding, subjective ratings, and physiological measures.

Key findings include:

• Comprehension Time: Cognitive Complexity shows a strong positive correlation with the time taken to understand code snippets. This implies that higher complexity values generally indicate more challenging code, in terms of time spent on comprehension tasks. The effect size was found to be 0.54, implying that it could be a reliable indicator of source code complexity, much above other existing metrics.
• Correctness: The metric displayed mixed results concerning the correctness of comprehension tasks. This moderate negative correlation suggests that while Cognitive Complexity may be capturing certain aspects of difficulty, its relation to accuracy in understanding or modifying code is unclear. This indicates potential discrepancies between perceived complexity and tangible understanding.
• Subjective Ratings: Cognitive Complexity demonstrated a negative correlation with perceived understandability ratings. Developers tend to rate code snippets with higher Cognitive Complexity values as less understandable, supporting its validity in gauging perceived complexity in code structure.
• Physiological Measures: The metric showed inconsistent correlation with physiological responses measured during program comprehension, such as brain activity. This may highlight the intricate nature of cognitive load during code comprehension and reflect limitations in current physiological measurement techniques or sample sizes.

These insights collectively affirm Cognitive Complexity's utility as a tool to flag potentially complex code segments that may burden comprehension. However, discrepancies suggest it should be used in conjunction with other assessment measures to reliably gauge all facets of understandability.

Implications for Software Engineering

The validated correlation of Cognitive Complexity with comprehension time and subjective understandability underscores its potential in real-time code analysis scenarios. Developers can leverage this tool to enhance code quality proactively, optimize refactoring efforts, and facilitate better collaboration via improved code readability assessments.

Directions for Future Research

The validation of Cognitive Complexity opens avenues for further investigation into defining threshold values that indicate unduly complex code segments. Moreover, future work should focus on establishing a more standardized correlation framework between various understandability measures, including physiological metrics, to refine the tool's accuracy and applicability across diverse programming paradigms.

Conclusion

This validation paper represents a substantial contribution to empirical software engineering, demonstrating Cognitive Complexity’s significance as a metric capable of capturing key dimensions of code understandability. While promising, further refinement and contextual integration are necessary to optimize its effectiveness in guiding software development practices.