LLMCup: Automatic Comment Update Framework

• LLMCup is a framework for automatic comment updating that generates candidate comments from multiple prompt strategies and selects the best one with CupRank.
• It employs a structured update-then-rank pipeline, using k-shot prompting and code–comment co-evolution data to ensure minimal edits and consistency.
• Evaluation shows significant accuracy improvements over prior methods, demonstrating better alignment of comments with actual code modifications.

Searching arXiv for the specified LLMCup paper and closely related work to ground the article. LLMCup is a framework for automatic comment updating in evolving software, introduced to address the common case in which method implementations are revised while associated natural-language comments remain outdated or inconsistent (Ge et al., 11 Jul 2025). The task takes an old method implementation $C_{\mathrm{old}}$, an updated implementation $C_{\mathrm{new}}$, and an old comment $com_{\mathrm{old}}$, and generates an updated comment $com_{\mathrm{new}}$ that is consistent with the code change, preserves valid prior information, and follows a minimal-edits principle. LLMCup operationalizes this as an update-then-rank pipeline: multiple prompt strategies are used to generate diverse candidate comments with a LLM, and a learned ranking model, CupRank, selects the final output by scoring alignment between code changes and comment changes (Ge et al., 11 Jul 2025).

1. Task formulation and design objective

Automatic comment updating is defined as producing a revised comment that best aligns with the change from $C_{\mathrm{old}}$ to $C_{\mathrm{new}}$ while preserving still-correct content from $com_{\mathrm{old}}$ (Ge et al., 11 Jul 2025). The formulation emphasizes three properties: consistency with the revised code, minimal editing of unaffected content, and fluency.

In LLMCup, this objective is expressed through candidate generation followed by ranking. If $\{com_{\mathrm{new}}^{(j)}\}$ denotes the candidate set produced under distinct prompting strategies, the selected output is

$$\hat{com}_{\mathrm{new}}=\arg\max_j s\!\big(C_{\mathrm{old}}, C_{\mathrm{new}}, com_{\mathrm{old}}, com_{\mathrm{new}}^{(j)}\big).$$

This formulation is notable because it does not assume that a single prompting strategy is uniformly optimal across update cases. Instead, it treats prompt variability as a source of candidate diversity and delegates final selection to a learned scorer. A plausible implication is that LLMCup reframes comment updating from a single-sequence generation problem into a structured retrieval-and-ranking problem over generated hypotheses.

2. Prior methods and motivating failure modes

LLMCup is positioned against two earlier approaches: CUP and HebCup (Ge et al., 11 Jul 2025). CUP is a neural sequence-to-sequence model tailored for comment updating; it encodes old code, new code, and the old comment, and decodes the updated comment using task-specific tokenization and co-attention. HebCup is a heuristic rule-based method derived from CUP’s failure cases and applies simple token-level edit heuristics directly on comments to reflect code changes.

The reported motivation for LLMCup is that both prior methods exhibit systematic failure modes. They frequently miss crucial code changes, such as a change in return type, and often generate outputs that remain overly similar to the old comment, leaving inconsistencies unresolved (Ge et al., 11 Jul 2025). They also struggle with complex updates that cannot be inferred from superficial code edits, and they exhibit weaker natural-language quality, including awkward phrasing and grammatical issues.

The paper’s JetBrains/android example illustrates these issues. A method is modified to return a boolean under several conditions, while the original comment states that it “returns the list of their names.” The developer’s own revised comment still omits the boolean return type; CUP and HebCup likewise continue to state “returns the list of their names.” By contrast, an LLM-generated update explicitly mentions the boolean return conditions (Ge et al., 11 Jul 2025). This example is used to motivate the claim that LLMs can better capture the semantics of code modification, but that prompt effectiveness varies across cases and models. LLMCup’s core response to this variability is to generate under multiple prompt strategies and then rank.

3. End-to-end workflow and prompting strategy

The LLMCup workflow has two phases: building a ranker through data augmentation, and inference through generation plus ranking (Ge et al., 11 Jul 2025). The available code–comment co-evolution data contains only positives, namely ground-truth updated comments, so LLMCup augments training data with negatives produced by open LLMs under multiple prompt strategies. For each original sample $$(C_{\mathrm{old}}, com_{\mathrm{old}}, C_{\mathrm{new}}, com_{\mathrm{new}}^{gt})$$, the framework removes the ground-truth update, prompts LLMs with the remaining triple, and labels any generated updated comment that matches $C_{\mathrm{new}}$0 as positive; all others are negatives. Duplicates are removed, and samples that fail to produce any negative are discarded. Semantically similar negatives are intentionally retained, because the ranking objective benefits from hard negatives.

After augmentation, each sample group contains one positive and several negatives for the same code/comment context. LLMCup converts each sample into two edit-token sequences, one representing code change and the other comment change, and trains CupRank with a listwise loss. At inference time, for a new instance $C_{\mathrm{new}}$1, the framework retrieves $C_{\mathrm{new}}$2 similar examples by new-code similarity using a SentenceBERT encoder, constructs prompts with different shot sizes, generates candidate comments with the base LLM, scores each candidate with CupRank, and returns the top-scoring one (Ge et al., 11 Jul 2025).

The prompt strategy is based on $C_{\mathrm{new}}$3-shot prompting with $C_{\mathrm{new}}$4. Prompts contain a system-role instruction specifying the task, optional demonstrations retrieved from the training corpus by semantic similarity of $C_{\mathrm{new}}$5, the old method code, the new method code, the old comment, and an answering instruction. The answering instruction imposes two explicit rules: the fewer changes, the better, and typos in the original comment should be fixed (Ge et al., 11 Jul 2025). Demonstration retrieval by semantic similarity is reported to be more effective than random example selection. Unless otherwise specified, all LLMs are decoded at temperature $C_{\mathrm{new}}$6, and up to five demonstrations are used because of context-window constraints.

4. Edit-token representation and CupRank

CupRank is a dual-encoder ranking model designed to score alignment between a code change and a candidate comment change (Ge et al., 11 Jul 2025). Its input construction begins by tokenizing heterogeneous inputs uniformly with CodeBERT, including subword splitting and special tokens such as <s> and </s>. Old and new versions are concatenated separately for code and comment, and token-level diffs are obtained with Python difflib to make implicit modifications explicit.

For each token $C_{\mathrm{new}}$7, LLMCup defines an edit operation $C_{\mathrm{new}}$8 and a token origin $C_{\mathrm{new}}$9. The corresponding edit token is

$com_{\mathrm{old}}$0

Its embedding concatenates a CodeBERT token embedding, a one-hot edit-operation encoding, and a scalar origin flag:

$com_{\mathrm{old}}$1

CupRank then applies two cross-attentions to model bi-directional relationships between code-change sequence $com_{\mathrm{old}}$2 and comment-change sequence $com_{\mathrm{old}}$3:

$com_{\mathrm{old}}$4

These are followed by residual addition,

$com_{\mathrm{old}}$5

and a 2-layer Transformer encoder that produces contextual representations $com_{\mathrm{old}}$6 and $com_{\mathrm{old}}$7. Max-pooling aggregates token-level representations into fixed-size vectors $com_{\mathrm{old}}$8 and $com_{\mathrm{old}}$9, which are projected into a shared space $com_{\mathrm{new}}$0 and $com_{\mathrm{new}}$1. The ranking score is the cosine similarity

$com_{\mathrm{new}}$2

Training is listwise. For one positive sample $com_{\mathrm{new}}$3 and $com_{\mathrm{new}}$4 negatives $com_{\mathrm{new}}$5, LLMCup defines

$com_{\mathrm{new}}$6

with temperature $com_{\mathrm{new}}$7, and optimizes

$com_{\mathrm{new}}$8

Ablation results reported for this design attribute measurable contributions to the edit-operation and origin features, the temperature term in the listwise loss, and especially the representation-learning layer based on cross-attention plus Transformer refinement (Ge et al., 11 Jul 2025).

5. Experimental setting and quantitative results

The evaluation uses a refined version of the CUP dataset built from 1,500 Java repositories (Ge et al., 11 Jul 2025). After HebCup’s filtering of trivial revisions and the removal of validation–test duplicates, the final split contains 80,591 training samples, 8,827 validation samples, and 8,125 testing samples. Samples are also partitioned by update type, including Code-Ind versus Non-Code-Ind and Single-Token, Single-Sub-Token, and Multi-Tokens categories.

The evaluated LLMs are GPT-4o and six open-source models: Code Llama-7B, DeepSeek-Coder-v2-16B, Qwen2.5-Coder-7B, Mistral-7B, Llama3-8B, and Gemma-7B (Ge et al., 11 Jul 2025). GPT-4o is excluded from the augmentation stage for budget reasons. CupRank is trained with learning rate $com_{\mathrm{new}}$9, batch size $C_{\mathrm{old}}$0, CodeBERT embeddings from codebert-base, fixed seeds, and checkpoint selection by lowest validation loss.

The reported metrics are Accuracy, AED, RED, BLEU-4, METEOR, F1 based on ROUGE-L, and SentenceBERT similarity (Ge et al., 11 Jul 2025). Accuracy is token-level exact match after normalization. RED is defined as the ratio between the edit distance from updated comment to ground truth and the edit distance from the old comment to ground truth. SentenceBERT similarity uses cosine similarity from stsb-roberta-large.

Using GPT-4o as the base LLM and generating four candidates from 0-, 1-, 3-, and 5-shot prompting, LLMCup achieves Accuracy $C_{\mathrm{old}}$1, compared with CUP at $C_{\mathrm{old}}$2 and HebCup at $C_{\mathrm{old}}$3 (Ge et al., 11 Jul 2025). The corresponding relative improvements are reported as $C_{\mathrm{old}}$4 over CUP and $C_{\mathrm{old}}$5 over HebCup. BLEU-4 reaches $C_{\mathrm{old}}$6, METEOR $C_{\mathrm{old}}$7, F1 $C_{\mathrm{old}}$8, and SentenceBERT similarity $C_{\mathrm{old}}$9, with improvements of $C_{\mathrm{new}}$0 in BLEU-4, $C_{\mathrm{new}}$1 in METEOR, $C_{\mathrm{new}}$2 in F1, and $C_{\mathrm{new}}$3 in SentenceBERT similarity over the two baselines. AED and RED are higher for LLMCup than for CUP and HebCup, which the paper attributes to more verbose or semantically rich edits even when lexical and semantic quality improve.

Against ranking baselines on GPT-4o-generated candidates, CupRank reaches Accuracy $C_{\mathrm{new}}$4, outperforming Random at $C_{\mathrm{new}}$5, Self-Ranking via GPT-4o at $C_{\mathrm{new}}$6, RankNet at $C_{\mathrm{new}}$7, and the best single-shot prompting baseline, 5-shot, at $C_{\mathrm{new}}$8 (Ge et al., 11 Jul 2025). Across seven LLMs, LLMCup improves over the best single-shot prompting baseline by Accuracy $C_{\mathrm{new}}$9 to $com_{\mathrm{old}}$0, reduces AED by $com_{\mathrm{old}}$1 to $com_{\mathrm{old}}$2, reduces RED by $com_{\mathrm{old}}$3 to $com_{\mathrm{old}}$4, and improves BLEU-4, METEOR, F1, and SentenceBERT similarity as well. Because GPT-4o outputs were excluded from augmentation, the reported gains on GPT-4o suggest cross-LLM robustness.

A separate temperature study on 1,000 test cases shows that higher temperatures degrade Accuracy for most models (Ge et al., 11 Jul 2025). GPT-4o declines from $com_{\mathrm{old}}$5 at $com_{\mathrm{old}}$6 to $com_{\mathrm{old}}$7 at $com_{\mathrm{old}}$8; CodeLlama declines from $com_{\mathrm{old}}$9 to $\{com_{\mathrm{new}}^{(j)}\}$0; similar downward trends are reported for Llama3, DeepSeek-Coder-v2, and Qwen2.5-Coder, while Mistral shows a slight uptick at $\{com_{\mathrm{new}}^{(j)}\}$1 before declining and Gemma remains relatively stable. The paper concludes that low temperatures, approximately $\{com_{\mathrm{new}}^{(j)}\}$2 to $\{com_{\mathrm{new}}^{(j)}\}$3, are preferable for comment updating.

6. Qualitative behavior, deployment considerations, and limitations

Performance is not uniform across update types (Ge et al., 11 Jul 2025). LLMCup performs best on code-aligned simple edits, achieving Accuracy $\{com_{\mathrm{new}}^{(j)}\}$4 on Code-Ind Single-Token cases, corresponding to $\{com_{\mathrm{new}}^{(j)}\}$5, and $\{com_{\mathrm{new}}^{(j)}\}$6 on Code-Ind Single-Sub-Token cases, corresponding to $\{com_{\mathrm{new}}^{(j)}\}$7. The hardest category is Non-Code-Ind Multi-Tokens, where Accuracy drops to $\{com_{\mathrm{new}}^{(j)}\}$8, or $\{com_{\mathrm{new}}^{(j)}\}$9. These cases are characterized as comment changes that are not directly anchored in code edits and that require broader reasoning about design or intent. Even so, LLMCup surpasses CUP and HebCup in absolute correct counts across all categories, with reported gains ranging from $$\hat{com}_{\mathrm{new}}=\arg\max_j s\!\big(C_{\mathrm{old}}, C_{\mathrm{new}}, com_{\mathrm{old}}, com_{\mathrm{new}}^{(j)}\big).$$0 to $$\hat{com}_{\mathrm{new}}=\arg\max_j s\!\big(C_{\mathrm{old}}, C_{\mathrm{new}}, com_{\mathrm{old}}, com_{\mathrm{new}}^{(j)}\big).$$1 depending on the category.

The JetBrains/android ranking example clarifies CupRank’s intended behavior. Candidate comments that explicitly describe boolean return behavior receive higher similarity scores because the code-change sequence includes insertions such as return true and return false, whereas candidates that retain outdated “returns names” language align poorly with the code-change representation (Ge et al., 11 Jul 2025). This suggests that CupRank is primarily sensitive to semantic compatibility between explicit code edits and proposed comment edits, rather than to surface fluency alone.

A user study involved seven graduate students with an average of five years in Java, evaluating 100 instances in which LLMCup’s output differed from the ground truth (Ge et al., 11 Jul 2025). On five-point Likert scales, LLMCup scored higher than human-written updates on consistency ($$\hat{com}_{\mathrm{new}}=\arg\max_j s\!\big(C_{\mathrm{old}}, C_{\mathrm{new}}, com_{\mathrm{old}}, com_{\mathrm{new}}^{(j)}\big).$$2 versus $$\hat{com}_{\mathrm{new}}=\arg\max_j s\!\big(C_{\mathrm{old}}, C_{\mathrm{new}}, com_{\mathrm{old}}, com_{\mathrm{new}}^{(j)}\big).$$3), naturalness ($$\hat{com}_{\mathrm{new}}=\arg\max_j s\!\big(C_{\mathrm{old}}, C_{\mathrm{new}}, com_{\mathrm{old}}, com_{\mathrm{new}}^{(j)}\big).$$4 versus $$\hat{com}_{\mathrm{new}}=\arg\max_j s\!\big(C_{\mathrm{old}}, C_{\mathrm{new}}, com_{\mathrm{old}}, com_{\mathrm{new}}^{(j)}\big).$$5), and helpfulness ($$\hat{com}_{\mathrm{new}}=\arg\max_j s\!\big(C_{\mathrm{old}}, C_{\mathrm{new}}, com_{\mathrm{old}}, com_{\mathrm{new}}^{(j)}\big).$$6 versus $$\hat{com}_{\mathrm{new}}=\arg\max_j s\!\big(C_{\mathrm{old}}, C_{\mathrm{new}}, com_{\mathrm{old}}, com_{\mathrm{new}}^{(j)}\big).$$7). The paper interprets this as evidence that automatic metrics can miss qualitative properties valued by developers. Inter-rater agreement statistics are not reported.

For practical use, the paper recommends multiple prompt strategies, retrieval of demonstrations similar to the updated code via SentenceBERT, low decoding temperature, and prompts that include old code, new code, and old comment together with rules enforcing minimal edits and typo correction (Ge et al., 11 Jul 2025). It further recommends integration into CI pipelines so that, when pull requests modify commented methods, the system generates candidates and presents the top-ranked suggestion and top-$$\hat{com}_{\mathrm{new}}=\arg\max_j s\!\big(C_{\mathrm{old}}, C_{\mathrm{new}}, com_{\mathrm{old}}, com_{\mathrm{new}}^{(j)}\big).$$8 alternatives for human review. Additional safeguards include contradiction checks, style and grammar linting, and routing complex non-code-indicative updates to human reviewers or augmenting prompts with design context such as commit messages.

The reported limitations are equally explicit. Evaluation is confined to Java and open-source repositories; performance on proprietary code or other programming languages is not established (Ge et al., 11 Jul 2025). Only 0-, 1-, 3-, and 5-shot prompting is studied. GPT-4o is excluded from augmentation for budget reasons. Higher-temperature sampling degrades performance. Non-code-indicative multi-token updates remain difficult, and LLMCup can be verbose, as reflected in AED and RED. Within those bounds, the framework is presented as a specialized combination of LLM-based candidate generation and learned ranking for maintaining consistency between comments and evolving code.