Automated Release Note Generation

• Automated release note generation is a process that uses NLP, ML, and LLMs to automatically convert version-control artifacts, like commits and pull requests, into structured documentation.
• It employs both extractive and abstractive summarization techniques, including graph-based and pointer-generator models, to accurately capture key software changes.
• Advanced systems integrate metadata, classification, and hierarchical modeling to enhance clarity, coverage, and compliance in software release notes.

Automated release note generation refers to the application of NLP, ML, and, more recently, LLMs to produce software release notes directly from version-control artifacts such as commit messages, pull request (PR) titles and bodies, and code diffs. The field addresses a recognized bottleneck in modern software engineering: practitioners consistently underproduce or neglect release notes due to the labor required, despite their importance for traceability, compliance, and user communication. Automated systems aim to close this documentation gap by leveraging structured and unstructured source code metadata and recent advances in summarization algorithms.

1. Core Task Definition and Problem Formulation

The automated release note generation task can be formally cast as a multi-step text processing and natural language generation pipeline, operating over the source artifact sequence $S_\text{seq}=(S_1, \ldots, S_n)$ (e.g., sequential commit messages, PR titles), or, in more advanced settings, over commit trees or fine-grained code diffs. The system's objective is to produce a target release note $RN_\text{gen}$ that matches as closely as possible the human reference $RN_\text{ref}$. Under the commonly adopted extractive formulation, this reduces to optimal selection and ranking of source sentences, while abstractive approaches model $RN_\text{gen}$ directly as a conditional natural language generation task. More recent formulations further require categorization of entries (e.g., feature, fix, docs) and grouping under structured section headings.

Mathematically, extractive summarization seeks a scoring function $f(S_i)$ to rank and select the top $m$ sentences (often $m = |RN_\text{ref}|$), whereas abstractive approaches learn a sequence-to-sequence function $f: X \rightarrow Y$ that directly maps the input representation to well-formed, canonical release note sentences.

2. Input Sources and Preprocessing Pipelines

Automated systems vary in their requirements for input granularity and preprocessing rigor. The field has evolved from heuristic pipelines centered on commit messages to systems able to integrate multiple sources:

• Commit-level extraction: All commits between two tags are collected; message first lines are extracted (often empirically the most descriptive).
• Pull request aggregation: Systematic collection of PR titles and bodies, with deduplication logic for squash/rebase merges.
• Commit tree structure: Tree-structured representations, encoding merge and branch hierarchy via ASCII-formatted trees.
• Code diff mining: Line-level or file-level diffs, capturing added/modified/removed code lines (+/–) and change type.
• Normalization and Filtering: Cleaning steps are critical: removal of empty or trivial entries, HTML/XML sanitization, stripping of issue references (e.g., "#123"), URLs, signatures, and markdown artifacts. Trivial commits (“merge branch”, “update .gitattributes”) are filtered.

Advanced systems (notably SmartNote (Daneshyan et al., 23 May 2025)) supplement these with project and release metadata (semantic release type, author counts, project domain classification) and impose per-commit grouping, significance scoring, or minimum-significance thresholds for inclusion.

3. Architectures and Methodologies

Three methodological paradigms predominate:

3.1 Extractive Graph-Based Summarization

Classic approaches such as TextRank construct a similarity-weighted, undirected graph $G=(V,E,W)$ over input sentences. Nodes correspond to sentences; edge weights are computed by normalized lexical overlap:

$$w_{ij} = \text{Sim}(S_i, S_j) = \frac{|t \in S_i \cap S_j|}{\log|S_i| + \log|S_j|}$$

A PageRank-style algorithm propagates sentence centrality via

$$S_c(S_i) = (1-d) + d \cdot \sum_{S_j \in \text{In}(S_i)} \frac{w_{ji}}{\sum_{S_k \in \text{Out}(S_j)} w_{jk}} S_c(S_j)$$

However, this approach is limited by its reliance on raw lexical overlap, neglecting semantic similarity.

3.2 Embedding-Augmented Extractive Summarization

To address semantic blindness, enhanced variants replace bag-of-words overlap by cosine similarity of GloVe-embedded sentence vectors:

$$w_{ij} = \cos(v_i, v_j) = \frac{v_i \cdot v_j}{\|v_i\|\|v_j\|}$$

where $v_i$ is the averaged GloVe vector for (non-stopword) tokens in $S_i$. The resulting semantic graph enables more meaningfully ranked summaries. Empirical results indicate this substantially improves ROUGE-1/2/L scores and human preference over classic LSA and unaugmented TextRank (Nath et al., 2022).

3.3 Abstractive and LLM-Centric Generation

Recent work shifts toward sequence-to-sequence neural models and LLM prompting:

• Pointer-generator networks (DeepRelease (Jiang et al., 2022)): Inputs PR title, body, and commit messages, concatenated with “[sep]” tokens. An encoder-decoder LSTM with Bahdanau attention, augmented by a pointer-generator mechanism for copying out-of-vocabulary tokens, generates concise entry summaries. Training minimizes the negative log-likelihood over reference tokens. Output is further classified into change categories by a FastText classifier, and entries are grouped under standard headings. Performance exceeds heuristic baselines by +6.2% ROUGE-2 F1 and +22% macro-F1 for classification.
• LLM-driven pipelines (SmartNote (Daneshyan et al., 23 May 2025)): Commit and PR information, code diffs, and project metadata are summarized and prioritized via two XGBoost classifiers (for change-type and significance) and passed—after grouping and scoring—to a specialized LLM (gpt-4o) with tailored prompts. The pipeline achieves commit coverage of 81%, clarity and organization scores above all leading baselines, and robust applicability across real-world projects.

4. Datasets and Benchmarks

Evolution in scale and reproducibility has been driven by recent benchmark initiatives:

• ReleaseEval (Meng et al., 4 Nov 2025): 94,987 release notes, commit trees, and code diffs from 3,369 GitHub repositories (license-audited for reusability), across six languages. Supports three task granularities: commit2sum (commit messages), tree2sum (commit tree + messages), diff2sum (full code diffs). All filtering and segmenting scripts are open-sourced, supporting systematic comparison and future task evolution.
• Domain-specific resources: DeepRelease curated 46,656 annotated entries with PR linkage and manual gold-standard categories (from 400 repos, eight languages). The TextRank+GloVe dataset comprises 1,213 gold note instances (Java, Python, PHP).

Quality assurance is backed by expert manual annotation (Fleiss κ = 0.85 in ReleaseEval), and datasets expose common distributional properties: high prevalence of empty or trivial notes, and domain-specific sectioning.

5. Evaluation Protocols and Empirical Results

Evaluation employs both automated and human-centric metrics:

• Automated: ROUGE-N (precision, recall, F1 of n-gram overlap), ROUGE-L (Longest Common Subsequence), BLEU-4, and METEOR. Information coverage and organization are further captured by entropy over heading distribution, commit mention rates, and entity density metrics. Example: SmartNote achieves coverage of 81% (auto), info entropy 1.59, and ARI 33.06 (lower is better) (Daneshyan et al., 23 May 2025).
• Human: Likert-scale ratings (1–5 or 1–7) along criteria of completeness, clarity, conciseness, and organization, with studies blinded to generator identity. Inter-annotator agreement is high (κ ≥ 0.82 in ReleaseEval).

Recent results underscore the superior performance of semantically aware and LLM-powered methods:

• On human ratings, SmartNote topped completeness and organization (4.00, 4.10/5), clarity (4.06/5), and outperformed DeepRelease, hand-crafted notes, and Conventional Changelog.
• DeepRelease outperformed lead-commit and PR-title extractive baselines in both ROUGE and classification F1 (0.642 vs 0.379 and 0.604, resp.).
• In ReleaseEval, fine-tuned LLMs (Mistral-8B, LLaMA3.1-8B) yielded BLEU-4/ROUGE-L scores up to 42.77/53.14 for tree2sum, but struggled with diff2sum (~44.8), demonstrating abstraction limitations in summarizing low-level code changes.

System	Completeness	Clarity	Organization	Conciseness	Commit Coverage (%)
SmartNote	4.00	4.06	4.10	3.35	81
DeepRelease	3.39	2.97	3.42	3.03	41
Original RNs	3.71	3.81	3.52	3.68	31
ConvChangelog	2.74	2.71	2.61	2.52	13

6. Limitations, Failure Modes, and Open Challenges

Despite clear quantitative progress, several limitations persist:

• Extractive systems (e.g., TextRank+GloVe) cannot paraphrase or abstract, performing poorly when human reference notes are highly generalized but underlying commits are granular; sentence-count matching may misalign output brevity.
• Abstractive systems (e.g., DeepRelease) may overcompress details or misclassify entries, especially when PR or commit messages are terse or ambiguous. FastText classifiers depend on keyword distribution and can muddle documentation vs. bug categories.
• LLM approaches (SmartNote) incur inference cost and exhibit classifier drift for low-quality commit messages; coverage is high but sometimes at the expense of conciseness (longer outputs, default MST threshold). Implementation cost depends on LLM inference pricing; current estimates are ~$0.90 USD/release.
• Structural modeling: ReleaseEval shows LLMs excel when given tree-structured commit input (tree2sum) but degrade on raw diffs, indicating persistent challenges in compressing large, semantically noisy artifacts.
• Applicability and personalization: Most prior tools are sensitive to project structure or workflow (PR-centric, commit-convention reliant). SmartNote addresses this by supporting domain/audience-sensitive output, commit significance, and workflow agnosticism, but LLM personalization beyond four coarse domains remains an open research direction.

7. Trends and Future Directions

Emerging research directions include:

• Abstractive summarization with LLM fine-tuning: Pointer-generator and transformer-based models promise further alignment of generated notes with human-authored RNs, especially when trained on large, high-quality datasets derived from ReleaseEval (Meng et al., 4 Nov 2025).
• Hierarchical and graph-based modeling: Explicit encoding of commit tree dependencies or pre-filtering of high-impact code hunks via retrieval-augmented architecture (e.g., GNNs, hierarchical transformers) is proposed to distill complex development histories into salient summary bullets.
• Automated classification and structuring: Future systems aim for automated section generation, precise change-type tagging, and structured note composition to meet both technical and business user requirements.
• Continuous benchmark evolution and licensing: Ensuring openness, reproducibility, and sync with upstream repository changes (e.g., snapshot archiving) is recognized as a necessity, as highlighted in ReleaseEval.
• Interactive and integrated workflows: Integration with CI pipelines, interactive editing, and user-in-the-loop summarization (parameter tuning, feedback incorporation) represent practical extensions.

A plausible implication is that further advances in LLM alignment, multi-modal context modeling, artifact-aware pretraining, and practical interface design will increasingly automate and personalize software release note generation, raising both coverage and communicative quality to meet the needs of diverse audiences in modern software development.