---
title: 'GoodPoint-ICLR: Constructive Feedback Dataset'
url: https://www.emergentmind.com/topics/goodpoint-iclr
type: topic
---

# GoodPoint-ICLR: Constructive Feedback Dataset

Searching arXiv for the cited paper and closely related uses of “GoodPoint-ICLR” to ground the article.
GoodPoint-ICLR is a corpus of ICLR submissions paired with full review threads and author responses, introduced to study constructive feedback generation in an explicitly author-centric way. In "GoodPoint: Learning Constructive Scientific Paper Feedback from Author Responses" [2604.11924], constructive feedback is defined as targeted, actionable feedback that helps authors improve both their research and its presentation, and its effectiveness is operationalized along two binary axes: validity and author action. The dataset serves both as a supervision source and as an evaluation substrate for training LLMs to generate feedback that authors acknowledge as valid and worth acting on.

## 1. Problem setting and conceptual framing

The GoodPoint line of work argues that LLMs should be used to augment and empower researchers rather than to automate research without human oversight [2604.11924]. Within that framing, the central task is constructive feedback generation: producing feedback for scientific papers that is not merely fluent or review-like, but specifically useful from the author’s perspective.

The paper formalizes usefulness through two author-response-derived dimensions. **Validity** asks whether the author agrees with or rebuts a feedback point. **Author actionability** asks whether the author commits to “will revise” or “defer to future work.” This shifts the target away from surface similarity to reviewer prose and toward signals grounded in rebuttal behavior. A plausible implication is that GoodPoint-ICLR treats author responses as weak supervision for feedback quality, rather than using acceptance outcomes or reviewer scores as the primary signal.

The dataset is tied to a broader training recipe, GOODPOINT, built on Qwen3-8B. In that system, the corpus provides positive examples for supervised fine-tuning and preference pairs for direct preference optimization, making GoodPoint-ICLR the empirical foundation of the overall method [2604.11924].

## 2. Corpus construction

GoodPoint-ICLR is assembled from ICLR submissions spanning 2020–2026, with each paper linked to its entire review thread and author rebuttals on OpenReview [2604.11924]. Source years 2020–2023 were obtained via the Re\(^2\) dataset; for 2024–2025, paper titles from OpenReview were matched to arXiv metadata; and 2026 submissions were ingested directly from PDF uploads on OpenReview. All papers were converted to Markdown via the Marker tool.

After filtering non-English and incomplete threads, the final corpus comprises **18,936 papers**, of which **14,517** were rejected and **4,419** accepted. Each paper yields an average of **5.66 self-contained feedback units**. The paper also reports that the parsed corpus contains approximately **107,000** feedback-response pairs [2604.11924].

The resulting object is not simply a collection of review texts. It is a structured corpus in which each feedback unit is intended to be self-contained and aligned with a corresponding author response. This design makes the dataset suitable for both discriminative evaluation and generative training.

## 3. Parsing, annotation, and success labels

Each review thread is parsed by GPT-4.1 into atomic feedback units $f_i$ and paired with a corresponding author response $r_i$ [2604.11924]. Two binary labels are assigned to each $(f_i,r_i)$ pair:

- **Validity** $V(f_i,r_i)\in\{0,1\}$: whether the author agrees or rebuts the point.
- **Actionability** $A(f_i,r_i)\in\{0,1\}$: whether the author commits to “will revise” or “defer to future work.”

A feedback unit is deemed successful iff both labels are positive:

$$
1_{\text{success}}(f_i,r_i)=
\begin{cases}
1 & \text{if } V(f_i,r_i)=1 \wedge A(f_i,r_i)=1\\
0 & \text{otherwise.}
\end{cases}
$$

Two human annotators independently verified a stratified sample of **111** units. For **Validity**, observed agreement is **0.874**, PABAK is **0.747**, and Cohen’s $\kappa$ is **0.000**. For **Author Action**, observed agreement is **0.919**, PABAK is **0.837**, and Cohen’s $\kappa$ is **0.271** [2604.11924]. These figures indicate that the paper reports agreement using multiple statistics rather than relying on a single coefficient.

On the full corpus, the joint distribution over $(V,A)$ labels is as follows:

| Label combination | Share |
|---|---:|
| Valid & Actionable | 33.2% |
| Valid & Not Actionable | 18.5% |
| Invalid & Actionable | 5.8% |
| Invalid & Not Actionable | 42.5% |

The temporal partitioning is also explicit. ICLR 2020–2025 are partitioned into training, development, and test sets, while all **2026 papers (4,419)** are held out as a temporally separate test set to guard against domain leakage [2604.11924].

## 4. Role in the GOODPOINT training recipe

GOODPOINT applies a two-stage alignment procedure on top of **Qwen3-8B**, an **8B-parameter open-weight LLM** [2604.11924]. In the first stage, supervised fine-tuning uses only successful units with $V=1 \wedge A=1$ from **6,000 papers** in 2020–2025. Feedback units from each reviewer are concatenated into a single target string per paper to capture cohesive style. The objective is standard next-token cross-entropy,

$$
\mathcal{L}_{\text{SFT}}(\theta)=-\mathbb{E}_{(x,F^*)}\log p_\theta(F^*|x).
$$

The reported SFT hyperparameters are: base model **Qwen3-8B**, sequence length **30,000 tokens**, batch size **32** with micro-batch **1**, learning rate **$5\times 10^{-6}$**, and **BF16, ZeRO-3, FlashAttention2** [2604.11924].

In the second stage, the model is aligned with direct preference optimization. Two preference-pair datasets are used. The first contains **Valid vs. Invalid** pairs drawn from **9,000 papers**, where $y^+$ is valid feedback and $y^-$ invalid feedback. The second contains **synthetic corruption pairs**, where $y^+$ is human-written valid-and-actionable feedback and $y^-$ is a corrupted rewrite violating exactly one quality dimension: **generic, vague, inaccurate, non-essential, or unsupportive**. These corruptions are produced by **GPT-5 mini** and verified by an **LLM-judge** [2604.11924].

The DPO objective is

$$
\mathcal{L}_{\text{DPO}}(\theta)=
-\mathbb{E}_{(x,y^+,y^-)}
\log \sigma\!\Bigl(
\tfrac{1}{\tau}
\bigl(\log \pi_\theta(y^+|x)-\log \pi_\theta(y^-|x)\bigr)
\Bigr)
+\beta\,\mathrm{KL}\!\bigl(\pi_\theta(\cdot|x)\Vert \pi_{\mathrm{ref}}(\cdot|x)\bigr),
$$

with $\beta=0.1$ as the KL-penalty weight. An auxiliary NLL term on SFT data is added with weight **0.2** to prevent drift, and training halts after **50 DPO steps** to avoid overfitting [2604.11924].

This makes GoodPoint-ICLR more than a benchmark dataset: it supplies both the positive supervision and the preference structure needed for alignment.

## 5. Evaluation protocol and empirical results

Automatic evaluation is conducted on a held-out benchmark of **1,198 ICLR papers**, comprising **600** from 2020–2025 and **598** from 2026 [2604.11924]. All LLMs generate up to **five** feedback units per paper, randomly subsampled. Two complementary automatic metrics are used.

The first metric is **Author-Response Prediction**, reported as **Predicted Success Rate**. A separate Qwen3-8B evaluator is trained to predict $\hat{V}$ and $\hat{A}$ given a paper and a feedback unit. Combined Success Rate is the fraction of generated units with $\hat{V}=1 \wedge \hat{A}=1$, and the paper also reports **Validity Only** and **Action Only** pass rates. Confidence intervals use **$B=1{,}000$ bootstrap samples** at the **95%** level [2604.11924].

Under this metric, the reported combined success rates are **37.9%** for Gemini-3-flash, **45.8%** for GPT-5.2, **1.8%** for Llama3.1-8B-Instruct, **8.0%** for base Qwen3-8B, **9.2%** for GOODPOINT-SFT, and **14.7%** for GOODPOINT-DPO. The abstract summarizes this as an **83.7%** improvement in predicted success rate over the base model for a GoodPoint-trained Qwen3-8B [2604.11924].

The second metric is **Human Consensus-Based Feedback Matching**. The consensus set $C$ consists of essential feedback units raised by at least **two** human reviewers and also judged successful, i.e. $V=1 \wedge A=1$. Precision, recall, and $F_1$ are defined in the usual way from the subset $M$ of model-generated units matching $C$. Matches are first filtered by cosine similarity, with thresholds **0.55** for human-human and **0.45** for human-LLM comparisons, and then classified by GPT-5.2 [2604.11924].

On this metric, GOODPOINT-SFT attains **precision 0.138**, compared with **0.128** for Gemini-3-flash and **0.130** for GPT-5.2, while GOODPOINT-DPO attains **precision 0.093**, **recall 0.107**, and **$F_1$ 0.087**. All confidence-interval half-widths in this table are reported as **< 0.001**. The paper states that GOODPOINT-SFT not only closes the gap to much larger proprietary models but surpasses them in precision, demonstrating selectivity in surfacing high-value critiques [2604.11924].

## 6. Human study, interpretation, and related nomenclature

To test whether the automatic metrics correspond to real researcher utility, the paper reports an expert human study with **13 academic authors**, specifically PhD students with **2–6 years of experience** in **NLP, ML, or HCI** [2604.11924]. Participants rated feedback on their own manuscripts from **ACL, EMNLP, ICLR,** and **ICML**. For each of three models—**Gemini-3-flash, GOODPOINT-DPO, and Qwen3-8B**—they assessed **five random feedback units** along four dimensions: **Validity** (binary agree/disagree), **Actionability** (six-way choice), **Specificity** (1–5 Likert), and **Helpfulness** (1–5 Likert).

The reported results are:

- **Gemini-3-flash**: Validity Rate **72.3%**, Action Rate **56.9%**, Specificity **4.42 (0.86)**, Helpfulness **3.40 (1.40)**.
- **GOODPOINT-DPO**: Validity Rate **58.1%**, Action Rate **40.3%**, Specificity **3.50 (1.16)**, Helpfulness **2.77 (1.29)**.
- **Qwen3-8B**: Validity Rate **41.5%**, Action Rate **32.3%**, Specificity **2.89 (1.28)**, Helpfulness **2.25 (1.24)**.

Superscripts in the original table denote statistically significant pairwise differences, using **$\chi^2$ + Fisher’s exact** for binary variables and **Kruskal–Wallis + Mann–Whitney U** for ordinal variables, with **$p<0.05$** [2604.11924]. The paper concludes that across all four criteria, GOODPOINT-DPO significantly outperforms its base model and narrows the gap to Gemini-3-flash, supporting the claim that the automatic author-response and consensus-matching metrics translate into author-perceived usefulness.

A recurrent source of confusion is the name itself. **GoodPoint-ICLR** in [2604.11924] refers to the ICLR review-thread corpus used for constructive scientific feedback generation. It is distinct from the earlier computer-vision method **"GoodPoint: unsupervised learning of keypoint detection and description"** [2006.01030], which studies keypoint detectors and descriptors, and also distinct from the publication-effort metric presented in **"ICLR Points: How Many ICLR Publications Is One Paper in Each Area?"** [2503.16623], where a secondary exposition labels the metric “GoodPoint-ICLR” or “ICLR points.” Within the scientific-feedback literature, however, GoodPoint-ICLR denotes the author-response-grounded dataset introduced alongside GOODPOINT [2604.11924].

Source: https://www.emergentmind.com/topics/goodpoint-iclr