Long-Range Modeling of Source Code Files with eWASH: Extended Window Access by Syntax Hierarchy

Abstract: Statistical language modeling and translation with transformers have found many successful applications in program understanding and generation tasks, setting high benchmarks for tools in modern software development environments. The finite context window of these neural models means, however, that they will be unable to leverage the entire relevant context of large files and packages for any given task. While there are many efforts to extend the context window, we introduce an architecture-independent approach for leveraging the syntactic hierarchies of source code for incorporating entire file-level context into a fixed-length window. Using concrete syntax trees of each source file we extract syntactic hierarchies and integrate them into context window by selectively removing from view more specific, less relevant scopes for a given task. We evaluate this approach on code generation tasks and joint translation of natural language and source code in Python programming language, achieving a new state-of-the-art in code completion and summarization for Python in the CodeXGLUE benchmark. We also introduce new CodeXGLUE benchmarks for user-experience-motivated tasks: code completion with normalized literals, method body completion/code summarization conditioned on file-level context.

• The paper introduces eWASH, enhancing transformer models by integrating concrete syntax trees to manage extended context in large source code files.
• It employs a syntactic hierarchy to filter irrelevant code segments, thereby improving the efficiency of code generation and summarization.
• eWASH achieves state-of-the-art performance on CodeXGLUE benchmarks, marking a significant advancement in program understanding and development.

The paper "Long-Range Modeling of Source Code Files with eWASH: Extended Window Access by Syntax Hierarchy" addresses the challenge of leveraging extensive context in source code files using transformer-based models, which typically have limited context windows. This limitation can hinder the ability to fully utilize relevant information, especially in handling large files and software packages.

To overcome these constraints, the authors introduce eWASH, an architecture-independent approach that integrates syntactic hierarchies of source code into the context window. The method involves using concrete syntax trees to extract these hierarchies, thereby allowing the model to selectively ignore less relevant scopes while maintaining a relevant overarching context.

The effectiveness of eWASH is demonstrated through its application to code generation tasks, as well as joint translation of natural language and source code in Python. The results include achieving state-of-the-art performance in code completion and summarization tasks on the CodeXGLUE benchmark. Moreover, the paper introduces new benchmarks for tasks like code completion with normalized literals and method body completion/code summarization conditioned on the entire file-level context.

Overall, this work represents a significant advancement in program understanding and generation tasks, making it particularly relevant for modern software development environments where managing extensive codebase context is crucial.