Commit Message Generation: Trends & Techniques

• Commit Message Generation (CMG) is the task of automatically translating code diffs into concise, descriptive messages that capture both what changed and why.
• Techniques have evolved from template-based and retrieval methods to advanced sequence-to-sequence and LLM-driven models that bridge the semantic gap between code and natural language.
• Current research stresses rich context representations, curated datasets, and improved evaluation metrics to better reflect real developer utility and editing effort.

Commit Message Generation (CMG) is the task of automatically producing a natural-language commit message from a code change, typically represented as a diff. In the dominant formulation, CMG is a code-to-text problem in which a system maps a code diff to a short descriptive sentence, often expected to summarize both what changed and, when possible, why it changed. The field has evolved from template-based and retrieval-based systems to sequence-to-sequence models, pretrained code LLMs, retrieval-augmented generation, graph- and AST-aware methods, issue- and history-aware conditioning, and, more recently, LLM-driven generation and optimization. Across this progression, three recurrent themes have defined the literature: the semantic gap between code and natural language, the difficulty of obtaining high-quality and legally reusable datasets, and the instability of evaluation when overlap-based metrics are treated as direct proxies for developer utility (Schall et al., 2024, Jung, 2021).

1. Conceptual scope and problem formulation

CMG is usually framed as a direct mapping from code diff to message, sometimes written as $f(\mathrm{diff}) \rightarrow \mathrm{message}$, and commonly implemented as a sequence-to-sequence problem in which a diff token sequence $x_1,\ldots,x_n$ is translated into a message token sequence $y_1,\ldots,y_m$ (Xiong et al., 23 Jul 2025, Schall et al., 2024). This framing emphasizes the summary function of commit messages, but the literature repeatedly notes that high-quality messages are not merely lexical condensations of changed lines. They are expected to encode action, scope, and often rationale, which is why several works distinguish “What” from “Why,” or use dimensions such as Rationality, Comprehensiveness, Conciseness, and Expressiveness (Li et al., 15 Mar 2025, Xue et al., 2024).

A central technical obstacle is the semantic gap between source code and natural language. Early and mid-stage CMG work characterized this as a limitation of flat token-sequence modeling: static embeddings ignore context, raw diffs omit structure, and line-level edits often fail to expose intent. CoreGen therefore argued for contextualized code representations rather than static code token embeddings (Nie et al., 2020). ATOM explicitly incorporated the abstract syntax tree and hybridized generation with retrieval, arguing that AST is important for capturing code semantics and that retrieval can mitigate the code–NL semantic gap that pure generation models struggle to bridge (Liu et al., 2019). COMET made the same point in graph form, representing changes through a delta graph derived from code property graphs so that syntax, control flow, and data dependencies become part of the CMG input (Mandli et al., 2024).

The problem has also been generalized beyond standard diff-to-message translation. One line of work treats commit repair and commit message generation jointly, modeling the conditional distribution $P(C,F\mid B)$ over buggy code $B$, fixed code $F$, and commit message $C$, and showing that a joint model can outperform cascaded repair-then-generate systems on CMG (Bai et al., 2021). Another line argues that generation is sometimes the wrong abstraction altogether: Commit Message Completion (CMC) conditions on a human-written prefix, while Commit Message Optimization (CMO) starts from a human-written message and iteratively improves it rather than regenerating from scratch (Eliseeva et al., 2023, Li et al., 16 Jan 2025).

2. Datasets, benchmarks, and corpus construction

Dataset construction has become a major research topic because results are highly sensitive to duplication, message quality, licensing, language coverage, and split strategy. An influential evaluation study found that most public CMG datasets were Java-only, that duplication could inflate retrieval-based performance, and that commit-, timestamp-, and project-based splits could change model rankings by a large margin (Tao et al., 2021). CommitBench subsequently systematized dataset curation around license-aware and privacy-conscious principles, arguing that poor commit selection, duplicates, privacy issues, and missing licenses can produce unusable models and skewed evaluations (Schall et al., 2024).

Dataset	Scale	Distinctive properties
MCMD	2.25M commits	Five languages; commit, timestamp, and project splits
CommitBench	1,664,590 commits	71,676 projects; MIT-licensed sources; privacy scrubbing
CommitChronicle	10,699,767 commits	20 languages; repository-based split; author history retained
ApacheCM	234,799 commits	50 Apache projects; 10,000-commit test set
CommitSuite	63,533 commits	CCS-compliant; AST-level changes; “what/why” annotations

MCMD was explicitly designed as a multi-programming-language commit message dataset, balancing 450,000 commits per language across Java, C#, C++, Python, and JavaScript, and exposing three split regimes: random by commit, chronological by timestamp, and repository-disjoint by project (Tao et al., 2021). CommitBench is larger and more aggressively curated: it contains 1,664,590 commits from 71,676 unique projects and 38,578 organizations across six languages, uses only MIT-licensed repositories, removes bots, revert commits, binary and file-mode changes, and deduplicates by identical diff content (Schall et al., 2024). CommitChronicle extends scale and temporal realism still further, with 10,699,767 commits across 20 programming languages, repository-based 80/10/10 splits, author-overlap removal between train and validation/test, and retained history suitable for personalization and completion studies (Eliseeva et al., 2023).

ApacheCM and CommitSuite represent newer benchmark design choices. ApacheCM was built from the top 50 Apache-hosted GitHub repositories and, after filtering, contains 234,799 commits; it underpins contextual retrieval work such as C3Gen and CoRaCMG (Xiong et al., 23 Jul 2025, Xiong et al., 22 Sep 2025). CommitSuite instead centers CCS-formatted commits, assembling 63,533 Conventional Commits-compliant samples from 243 candidate repositories, later curated to 172 repositories after filtering and outlier removal, and enriching them with AST-level code changes and LLM-assisted annotations of “what” and “why” (Wan et al., 4 May 2026).

Several studies also show that benchmark quality depends on message quality, not only corpus size. One empirical study cleaned the widely used MCMD test set in three stages—language filtering, Bi-LSTM-based “What/Why” filtering, and dual-evaluator manual selection—ending with 500 curated samples per language, because many ground-truth messages in the raw test set lacked essential “What” and “Why” information (Xue et al., 2024). This supports a broader conclusion already visible in CommitBench and CommitSuite: CMG benchmarks increasingly encode corpus governance, semantic filtering, and evaluation intent, rather than simply aggregating mined diff–message pairs (Schall et al., 2024, Wan et al., 4 May 2026).

3. Modeling paradigms and code-change representations

CMG research can be organized into template-based, retrieval-based, generation-based, hybrid, and optimization-based paradigms, but the more consequential distinction is often the representation of code change. Early sequence-to-sequence systems treated unified diffs as text. CommitBERT retained this text-to-text formulation but improved the encoder input by isolating only changed lines and separating additions from deletions as $[CLS]$ Add $[SEP]$ Del $[SEP]$, while also initializing the encoder from CodeBERT and then from CodeSearchNet Code-to-NL weights. On its 345K-pair six-language dataset, this preprocessing and initialization consistently improved BLEU-4 across Python, PHP, JavaScript, Java, Go, and Ruby (Jung, 2021).

A second family emphasizes contextualized representations. CoreGen introduced a two-stage framework: Stage I learns contextualized code representations through commit-type-aware self-supervised objectives, and Stage II fine-tunes them for downstream CMG. On the benchmark used in that study, CoreGen achieved BLEU-4 21.06 versus 16.43 for NNGen, a gain described as at least 28.18% over the best baseline (Nie et al., 2020). COMET extended the representation shift from token context to graph context by building a delta graph from the union of added, deleted, and one-hop contextual subgraphs, then linearizing that representation for a transformer generator. In the reported experiments, COMET outperformed state-of-the-art techniques in bleu-norm and meteor while remaining comparable in ROUGE-L (Mandli et al., 2024).

A third direction explicitly models structural program information. ATOM incorporated AST into commit change representation and added hybrid ranking over retrieved and generated candidate messages. Its abstract reports evaluation on a dataset crawled from 56 popular Java repositories and a 30.72% increase over state-of-the-art models in BLEU-4 (Liu et al., 2019). Joint learning work went further by coupling code repair with CMG: rather than derive messages from a predicted fix in a cascaded way, the joint model decodes fixed code and commit message together and uses changes-aware dynamic attention over buggy and fixed code representations. On the monolingual Java setting, the joint model reached BLEU-4 11.48 and ROUGE-L 10.62, outperforming cascaded CMG variants, and multilingual training further improved CMG in all five languages studied (Bai et al., 2021).

These methods collectively show that CMG progress has depended less on a single universal architecture than on progressively richer representations of change: added/deleted token separation, contextualized code embeddings, AST structure, code property graphs, and explicit modeling of the before/after semantics of edits (Jung, 2021, Nie et al., 2020, Mandli et al., 2024, Bai et al., 2021).

4. Retrieval, external context, and repository history

Retrieval has remained central to CMG because commit messages exhibit recurring local patterns, project-specific terminology, and stylistic regularities. RACE formalized retrieval-augmented CMG as an end-to-end model with a retrieval module and a generation module linked by an exemplar guider. The guider computes a learned similarity score $x_1,\ldots,x_n$0 and uses it to control how strongly generation should rely on the retrieved exemplar. Across Java, C, C++, Python, and JavaScript, RACE outperformed all baselines on BLEU, METEOR, ROUGE-L, and CIDEr (Shi et al., 2022).

Subsequent work replaced task-specific retrieval with broader RAG-style frameworks. REACT combined BM25 with CodeT5+-based dense retrieval, selected a top-1 exemplar, and integrated it either into PLM fine-tuning or LLM in-context learning. Its reported gains were large: BLEU improved by up to 55% for CodeT5 and by 102% for Llama 3, while hybrid retrieval outperformed either sparse or dense retrieval alone (Zhang et al., 2024). CoRaCMG, built on ApacheCM, restricted retrieval to historical commits from the same repository and used retrieved diff–message pairs to transfer project-specific terminology and writing style. With a single retrieved example, GPT-4o improved from BLEU 9.12 to 17.24 and DeepSeek-R1 from 9.87 to 17.42; gains plateaued beyond roughly three examples (Xiong et al., 22 Sep 2025).

Another line of work broadens retrieval targets beyond historical commit pairs. C3Gen argues that diff-only input is insufficient because repository-wide usage context is missing. It constructs code structure graphs with tree-sitter, identifies modified entities from the diff, retrieves functions or classes that invoke or instantiate them, and concatenates this “Relevant Code Context” with the diff. On human evaluation, C3Gen consistently improved Completeness by about 5% on average across GPT-4o, GPT-4.1, DeepSeek V3, and DeepSeek R1, even though similarity-based automatic metrics were mixed and often statistically insignificant (Xiong et al., 23 Jul 2025). ExGroFi extends external context in a different direction by linking commits to issues, extracting actual and expected states from issue text, and using that rationale-bearing representation in a grounding stage before CMG fine-tuning. Across six strong backbones, ExGroFi-enhanced models improved BLEU, ROUGE-L, METEOR, and CIDEr relative to their original forms, with particularly large gains for encoder–decoder and hybrid models (Wang et al., 2023).

Repository history is another contextual source. CommitChronicle reframed CMG as commit message completion and added prior commit messages from the same author and repository as conditioning context. In the generation setting, history raised B-Norm from 15.3 to 16.9 for the best model; in completion, longer prefixes produced even larger gains, with RACE reaching B-Norm 27.28 when 50% of the message was supplied as a prefix together with history (Eliseeva et al., 2023). Taken together, retrieval in CMG no longer means only nearest-neighbor reuse: it now includes exemplar selection, repository-scoped code context, issue rationale, and author history.

5. Evaluation, metric instability, and the move toward user-centered assessment

Evaluation has been one of the most contested aspects of CMG. A systematic study of existing models showed that different BLEU implementations—B-Moses, B-Norm, and B-CC—produce materially different rankings, and that split strategy can reverse conclusions about which models generalize best (Tao et al., 2021). In that study, B-Norm correlated best with human judgments, with Spearman’s $x_1,\ldots,x_n$1 and Kendall’s $x_1,\ldots,x_n$2, and was therefore recommended for CMG reporting (Tao et al., 2021).

Later work questioned BLEU more fundamentally. “Evaluating Commit Message Generation: To BLEU Or Not To BLEU?” argued that BLEU-4 is ill-suited to short, information-dense, code-flavored messages because higher-order n-gram precision collapses on short texts, morphology and synonymy are ignored, and brevity penalty can punish idiomatic concision. That study proposed Log-MNEXT, a METEOR-derived metric with case folding, punctuation removal, semantic matching, and a modified fragmentation penalty, and reported a Spearman correlation of 0.831 with human judgments—higher than BLEU, ROUGE, METEOR, or METEOR-NEXT (Dey et al., 2022).

An even stronger critique emerged from realistic online/offline evaluation. JetBrains reported that in deployed CMG systems the most useful online signal is not similarity to a reference message, but how much a user edits the suggestion before committing. Using character-level Levenshtein edit distance $x_1,\ldots,x_n$3 between generated message $x_1,\ldots,x_n$4 and final committed message $x_1,\ldots,x_n$5 as the online metric, the study found that edit distance had the highest Spearman correlation with online behavior at 0.74, whereas BLEU and METEOR showed low or non-significant correlation. This directly contradicted earlier assumptions that similarity metrics aligned with real user preference (Tsvetkov et al., 2024).

Recent benchmarks therefore move toward manual or reference-free evaluation. One empirical LLM study introduced practitioner-oriented manual metrics—Accuracy, Integrity, Applicability, and Readability—and found GPT-3.5 best overall after the test set had been cleaned to retain messages with clear “What” and “Why” content (Xue et al., 2024). CommitSuite formalized a reference-free framework with five binary metrics—Rationality, Comprehensiveness, Non-redundancy, Authenticity, and Logicality—and reported Cohen’s $x_1,\ldots,x_n$6 between the LLM evaluator and human judgments (Wan et al., 4 May 2026). A plausible implication is that CMG evaluation is moving from reference overlap toward semantic adequacy, user effort, and workflow-grounded acceptance.

6. LLMs, optimization, and emerging directions

The arrival of LLMs changed both the performance frontier and the problem definition. A preliminary study comparing ChatGPT and Llama 2 with classic CMG baselines found mixed automatic-metric results, but human evaluators selected LLM-generated messages as best in 78% of 366 samples; only 13.1% of the time was the original human-written message selected as best (Zhang et al., 2024). A broader empirical study subsequently showed that, on a cleaned multilingual benchmark, GPT-3.5 achieved average METEOR 19.10, BLEU 26.79, and ROUGE-L 18.34, outperforming prior CMG approaches overall and scoring highest on Accuracy, Integrity, Applicability, and Readability under manual assessment (Xue et al., 2024).

At the same time, several papers argue that generation from scratch is not always the most faithful formulation. Commit Message Optimization (CMO) begins with the human-written message and iteratively improves it using curated software contexts, a retrieval-based evaluator, and fine-tuned LLM evaluators. One 2025 study reported that CMO generated commit messages that were significantly more Rational, Comprehensive, and Expressive while outperforming state-of-the-art CMG methods and human messages 88.2%–95.4% of the time on three core dimensions (Li et al., 16 Jan 2025). A related paper framed the same idea as “Consider What Humans Consider,” reported improvements over OMG, CMC, and human-written messages in the range of 40.3% to 78.4% depending on metric and baseline, and introduced CMO-blank and CMO-OMG variants for the cases where no human draft exists or where an existing CMG output should be post-edited (Li et al., 15 Mar 2025).

This reframing interacts with structured benchmarks and repository conventions. CommitSuite ties CMG to the Conventional Commits Specification, providing CCS type labels, AST-level code changes, and “what/why” annotations, while also showing that AST context significantly improves “what” descriptions in CMG ablations (Wan et al., 4 May 2026). CommitChronicle, by contrast, suggests that in some settings completion is preferable to generation because a user-written prefix already supplies style and intention, allowing the model to focus on completion rather than invention (Eliseeva et al., 2023).

The emerging picture is therefore heterogeneous rather than convergent. Diff-to-message generation remains the canonical task, but practical systems now include retrieval-augmented generation, repository-scoped contextualization, issue-aware grounding, history-aware completion, and optimization of human drafts. This suggests that CMG is increasingly treated not as a single model architecture problem, but as a family of developer-assistance tasks centered on commit-time documentation quality, project conventions, and measurable editing effort (Zhang et al., 2024, Li et al., 15 Mar 2025, Tsvetkov et al., 2024).