GoodPoint-ICLR: Constructive Feedback Dataset

• GoodPoint-ICLR is a structured corpus of ICLR submissions paired with full review threads and author responses aimed at producing targeted and valid feedback.
• The dataset operationalizes constructive feedback using binary labels for validity and author actionability derived from author responses.
• It underpins the GOODPOINT training recipe by leveraging supervised fine-tuning and direct preference optimization on an 8B-parameter LLM to enhance feedback generation.

Searching arXiv for the cited paper and closely related uses of “GoodPoint-ICLR” to ground the article. GoodPoint-ICLR is a corpus of ICLR submissions paired with full review threads and author responses, introduced to study constructive feedback generation in an explicitly author-centric way. In "GoodPoint: Learning Constructive Scientific Paper Feedback from Author Responses" (Mun et al., 13 Apr 2026), constructive feedback is defined as targeted, actionable feedback that helps authors improve both their research and its presentation, and its effectiveness is operationalized along two binary axes: validity and author action. The dataset serves both as a supervision source and as an evaluation substrate for training LLMs to generate feedback that authors acknowledge as valid and worth acting on.

1. Problem setting and conceptual framing

The GoodPoint line of work argues that LLMs should be used to augment and empower researchers rather than to automate research without human oversight (Mun et al., 13 Apr 2026). Within that framing, the central task is constructive feedback generation: producing feedback for scientific papers that is not merely fluent or review-like, but specifically useful from the author’s perspective.

The paper formalizes usefulness through two author-response-derived dimensions. Validity asks whether the author agrees with or rebuts a feedback point. Author actionability asks whether the author commits to “will revise” or “defer to future work.” This shifts the target away from surface similarity to reviewer prose and toward signals grounded in rebuttal behavior. A plausible implication is that GoodPoint-ICLR treats author responses as weak supervision for feedback quality, rather than using acceptance outcomes or reviewer scores as the primary signal.

The dataset is tied to a broader training recipe, GOODPOINT, built on Qwen3-8B. In that system, the corpus provides positive examples for supervised fine-tuning and preference pairs for direct preference optimization, making GoodPoint-ICLR the empirical foundation of the overall method (Mun et al., 13 Apr 2026).

2. Corpus construction

GoodPoint-ICLR is assembled from ICLR submissions spanning 2020–2026, with each paper linked to its entire review thread and author rebuttals on OpenReview (Mun et al., 13 Apr 2026). Source years 2020–2023 were obtained via the Re$^2$ dataset; for 2024–2025, paper titles from OpenReview were matched to arXiv metadata; and 2026 submissions were ingested directly from PDF uploads on OpenReview. All papers were converted to Markdown via the Marker tool.

After filtering non-English and incomplete threads, the final corpus comprises 18,936 papers, of which 14,517 were rejected and 4,419 accepted. Each paper yields an average of 5.66 self-contained feedback units. The paper also reports that the parsed corpus contains approximately 107,000 feedback-response pairs (Mun et al., 13 Apr 2026).

The resulting object is not simply a collection of review texts. It is a structured corpus in which each feedback unit is intended to be self-contained and aligned with a corresponding author response. This design makes the dataset suitable for both discriminative evaluation and generative training.

3. Parsing, annotation, and success labels

Each review thread is parsed by GPT-4.1 into atomic feedback units $f_i$ and paired with a corresponding author response $r_i$ (Mun et al., 13 Apr 2026). Two binary labels are assigned to each $(f_i,r_i)$ pair:

• Validity $V(f_i,r_i)\in\{0,1\}$: whether the author agrees or rebuts the point.
• Actionability $A(f_i,r_i)\in\{0,1\}$: whether the author commits to “will revise” or “defer to future work.”

A feedback unit is deemed successful iff both labels are positive:

$$1_{\text{success}}(f_i,r_i)= \begin{cases} 1 & \text{if } V(f_i,r_i)=1 \wedge A(f_i,r_i)=1\ 0 & \text{otherwise.} \end{cases}$$

Two human annotators independently verified a stratified sample of 111 units. For Validity, observed agreement is 0.874, PABAK is 0.747, and Cohen’s $\kappa$ is 0.000. For Author Action, observed agreement is 0.919, PABAK is 0.837, and Cohen’s $\kappa$ is 0.271 (Mun et al., 13 Apr 2026). These figures indicate that the paper reports agreement using multiple statistics rather than relying on a single coefficient.

On the full corpus, the joint distribution over $(V,A)$ labels is as follows:

Label combination	Share
Valid & Actionable	33.2%
Valid & Not Actionable	18.5%
Invalid & Actionable	5.8%
Invalid & Not Actionable	42.5%

The temporal partitioning is also explicit. ICLR 2020–2025 are partitioned into training, development, and test sets, while all 2026 papers (4,419) are held out as a temporally separate test set to guard against domain leakage (Mun et al., 13 Apr 2026).

4. Role in the GOODPOINT training recipe

GOODPOINT applies a two-stage alignment procedure on top of Qwen3-8B, an 8B-parameter open-weight LLM (Mun et al., 13 Apr 2026). In the first stage, supervised fine-tuning uses only successful units with $f_i$0 from 6,000 papers in 2020–2025. Feedback units from each reviewer are concatenated into a single target string per paper to capture cohesive style. The objective is standard next-token cross-entropy,

$f_i$1

The reported SFT hyperparameters are: base model Qwen3-8B, sequence length 30,000 tokens, batch size 32 with micro-batch 1, learning rate $f_i$2, and BF16, ZeRO-3, FlashAttention2 (Mun et al., 13 Apr 2026).

In the second stage, the model is aligned with direct preference optimization. Two preference-pair datasets are used. The first contains Valid vs. Invalid pairs drawn from 9,000 papers, where $f_i$3 is valid feedback and $f_i$4 invalid feedback. The second contains synthetic corruption pairs, where $f_i$5 is human-written valid-and-actionable feedback and $f_i$6 is a corrupted rewrite violating exactly one quality dimension: generic, vague, inaccurate, non-essential, or unsupportive. These corruptions are produced by GPT-5 mini and verified by an LLM-judge (Mun et al., 13 Apr 2026).

The DPO objective is

$f_i$7

with $f_i$8 as the KL-penalty weight. An auxiliary NLL term on SFT data is added with weight 0.2 to prevent drift, and training halts after 50 DPO steps to avoid overfitting (Mun et al., 13 Apr 2026).

This makes GoodPoint-ICLR more than a benchmark dataset: it supplies both the positive supervision and the preference structure needed for alignment.

5. Evaluation protocol and empirical results

Automatic evaluation is conducted on a held-out benchmark of 1,198 ICLR papers, comprising 600 from 2020–2025 and 598 from 2026 (Mun et al., 13 Apr 2026). All LLMs generate up to five feedback units per paper, randomly subsampled. Two complementary automatic metrics are used.

The first metric is Author-Response Prediction, reported as Predicted Success Rate. A separate Qwen3-8B evaluator is trained to predict $f_i$9 and $r_i$0 given a paper and a feedback unit. Combined Success Rate is the fraction of generated units with $r_i$1, and the paper also reports Validity Only and Action Only pass rates. Confidence intervals use $r_i$2 bootstrap samples at the 95% level (Mun et al., 13 Apr 2026).

Under this metric, the reported combined success rates are 37.9% for Gemini-3-flash, 45.8% for GPT-5.2, 1.8% for Llama3.1-8B-Instruct, 8.0% for base Qwen3-8B, 9.2% for GOODPOINT-SFT, and 14.7% for GOODPOINT-DPO. The abstract summarizes this as an 83.7% improvement in predicted success rate over the base model for a GoodPoint-trained Qwen3-8B (Mun et al., 13 Apr 2026).

The second metric is Human Consensus-Based Feedback Matching. The consensus set $r_i$3 consists of essential feedback units raised by at least two human reviewers and also judged successful, i.e. $r_i$4. Precision, recall, and $r_i$5 are defined in the usual way from the subset $r_i$6 of model-generated units matching $r_i$7. Matches are first filtered by cosine similarity, with thresholds 0.55 for human-human and 0.45 for human-LLM comparisons, and then classified by GPT-5.2 (Mun et al., 13 Apr 2026).

On this metric, GOODPOINT-SFT attains precision 0.138, compared with 0.128 for Gemini-3-flash and 0.130 for GPT-5.2, while GOODPOINT-DPO attains precision 0.093, recall 0.107, and $r_i$8 0.087. All confidence-interval half-widths in this table are reported as < 0.001. The paper states that GOODPOINT-SFT not only closes the gap to much larger proprietary models but surpasses them in precision, demonstrating selectivity in surfacing high-value critiques (Mun et al., 13 Apr 2026).

6. Human study, interpretation, and related nomenclature

To test whether the automatic metrics correspond to real researcher utility, the paper reports an expert human study with 13 academic authors, specifically PhD students with 2–6 years of experience in NLP, ML, or HCI (Mun et al., 13 Apr 2026). Participants rated feedback on their own manuscripts from ACL, EMNLP, ICLR, and ICML. For each of three models—Gemini-3-flash, GOODPOINT-DPO, and Qwen3-8B—they assessed five random feedback units along four dimensions: Validity (binary agree/disagree), Actionability (six-way choice), Specificity (1–5 Likert), and Helpfulness (1–5 Likert).

The reported results are:

• Gemini-3-flash: Validity Rate 72.3%, Action Rate 56.9%, Specificity 4.42 (0.86), Helpfulness 3.40 (1.40).
• GOODPOINT-DPO: Validity Rate 58.1%, Action Rate 40.3%, Specificity 3.50 (1.16), Helpfulness 2.77 (1.29).
• Qwen3-8B: Validity Rate 41.5%, Action Rate 32.3%, Specificity 2.89 (1.28), Helpfulness 2.25 (1.24).

Superscripts in the original table denote statistically significant pairwise differences, using $r_i$9 + Fisher’s exact for binary variables and Kruskal–Wallis + Mann–Whitney U for ordinal variables, with $(f_i,r_i)$0 (Mun et al., 13 Apr 2026). The paper concludes that across all four criteria, GOODPOINT-DPO significantly outperforms its base model and narrows the gap to Gemini-3-flash, supporting the claim that the automatic author-response and consensus-matching metrics translate into author-perceived usefulness.

A recurrent source of confusion is the name itself. GoodPoint-ICLR in (Mun et al., 13 Apr 2026) refers to the ICLR review-thread corpus used for constructive scientific feedback generation. It is distinct from the earlier computer-vision method "GoodPoint: unsupervised learning of keypoint detection and description" (Belikov et al., 2020), which studies keypoint detectors and descriptors, and also distinct from the publication-effort metric presented in "ICLR Points: How Many ICLR Publications Is One Paper in Each Area?" (Luo, 20 Mar 2025), where a secondary exposition labels the metric “GoodPoint-ICLR” or “ICLR points.” Within the scientific-feedback literature, however, GoodPoint-ICLR denotes the author-response-grounded dataset introduced alongside GOODPOINT (Mun et al., 13 Apr 2026).