ChangeScribe: Heuristic Commit Messaging

• ChangeScribe is a heuristic-based tool that condenses raw code diffs into structured commit message templates, addressing the semantic gap between code changes and human-readable summaries.
• It employs rule-based logic to extract key code comments, identifiers, and summarized changes, thereby enhancing template clarity and model interpretability.
• Evaluations demonstrated significant metric improvements (BLEU-Norm, METEOR, ROUGE-L) and superior developer ratings, validating its practical effectiveness.

ChangeScribe is a heuristic-based software tool for improving automated commit message generation in version control systems by condensing code changes into concise, structured templates. Its central design addresses two chronic challenges in software engineering: the semantic gap between low-level code diffs and developer-readable commit messages, and the inefficiencies of existing datasets and input representations for LLMs tasked with summarizing code changes.

1. Methodological Foundations

ChangeScribe operates by parsing raw code diffs and translating them into a standardized template consisting of three core components:

• Summarized Code Changes: The tool identifies and categorizes modifications at hierarchical levels—repository, package, file, class, and method—then produces a readable summary across these levels. It specifically groups change types such as behavioral modifications (structural alterations to classes) and method-level inline changes.
• Elicited Comments: Using regular expressions and rule-based logic, ChangeScribe extracts all inline, block, and JavaDoc comments present in the diff. Comments are further categorized (e.g., license, TODO, general, JavaDoc) so both rationale and developer intent are captured and represented explicitly.
• Emphasized Code Identifiers: The tool parses and highlights all semantically significant code identifiers—method names, class names, variables—favoring CamelCase conventions prevalent in Java code. These are surfaced in the template to ensure contextual salience during commit message generation.

ChangeScribe achieves condensation by removing redundant detail, de-noising raw diffs, and contextualizing only the most relevant modifications, facilitating more effective downstream processing by LLMs.

2. Tool Functionality and Template Composition

ChangeScribe’s template creation involves applying a series of heuristic rules designed to maximize readability and model interpretability. It detects change categories and produces structured text with a header specifying repository and modification type (e.g., “Behavior Modification” or “Method Inline Changes”).

Table: ChangeScribe Template Structure

Component	Extraction Method	Role in Generation
Summarized Code Changes	Heuristic grouping	Core change context for generation
Elicited Comments	Regexp + rule-based logic	Captures developer rationale/intent
Emphasized Identifiers	CamelCase tokenization	Preserves semantic entity information

Each component is critical in forming a prompt that balances informativeness, brevity, and fidelity to developer conventions.

3. Integration with CodeLlama-7B

For commit message synthesis, ChangeScribe’s templates are paired with CodeLlama-7B—a large pretrained LLM—using a supervised sequence-to-sequence fine-tuning strategy. The process is mathematically formalized with an auto-regressive token generation probability:

$$P(Y \mid \mathcal{P}) = \prod_{k=1}^{n} P(y_k \mid \mathcal{P}, Y_{1:k-1})$$

where $\mathcal{P}$ is the ChangeScribe prompt and $Y$ is the commit message. The model is trained to minimize supervised cross-entropy loss:

$$\mathcal{L}_{SFT} = - \sum_{k=1}^n \log P(y_k \mid \mathcal{P}, Y_{1:k-1})$$

This framework enables CodeLlama-7B to map condensed representations to fluent, context-aware commit messages.

4. Evaluation and Comparative Performance

ChangeScribe’s efficacy was established using standard metrics:

• BLEU-Norm: Measures average modified n-gram overlap with case insensitivity.
• METEOR: Evaluates unigram-level precision/recall, accounting for non-consecutive matches.
• ROUGE-L: Scores longest common subsequence similarity.

Empirical results on a widely used dataset revealed that ChangeScribe templates, combined with fine-tuning, outperformed six baseline systems, with average metric improvements of 51.7% (BLEU-Norm), 78.7% (METEOR), and 62.5% (ROUGE-L). This demonstrates substantial gains in the relevance and readability of generated commit messages.

5. Ablation Study and Human Evaluation

Ablation experiments, removing components from the template (comments, identifiers, concise change summaries) or substituting raw diffs, consistently produced inferior results. Each element was shown to be integral to template quality.

Human evaluation involving experienced Java developers corroborated the metric findings: messages generated from condensed templates were rated statistically and semantically superior, with higher alignment to developer intent and lower incidence of irrelevant or “vague” descriptions.

6. Practical Significance and Implementation Insights

ChangeScribe’s approach enables a scalable, model-friendly bridge between raw code modifications and natural language explanations for software maintenance. The templating strategy reduces cognitive load on both models and developers, avoids dilution of key semantic details, and systematizes comment and identifier extraction. This highly structured condensation leads to improved commit message synthesis and offers a generalizable blueprint for pairing heterogeneous code change representations with pre-trained LLMs.

7. Limitations and Plausible Implications

The efficiency of ChangeScribe is contingent on the precision of its heuristic rules and the extent to which extracted comments and identifiers represent true developer intent. A plausible implication is that the approach could be extended to additional programming languages and repository structures, subject to comparable annotation and semantic conventions. The paper does not report on handling multi-language repositories or interfacing with code review systems, suggesting future avenues for context-aware condensation and ontology-driven change summarization.

In conclusion, ChangeScribe represents a significant methodological advance in commit message generation workflows by providing an automated, structurally concise, and semantically rich intermediate representation for code changes, demonstrably improving both objective metrics and subjective developer evaluations (Kuang et al., 19 Sep 2025).