- The paper proposes establishing a formal track in ML conferences for refutations and critiques to address misleading and flawed research.
- It outlines tailored review criteria that ensure methodological rigor, reproducibility, and transparent evaluation of corrective submissions.
- The proposal aims to foster a self-correcting research ecosystem by linking critical responses directly to the affected work.
Advocating for a "Refutations and Critiques" Track in ML Conferences
This position paper (2506.19882) advocates for the establishment of a dedicated "Refutations and Critiques" (R{content}C) Track within ML conferences. The central argument is that current ML conference ecosystems lack official mechanisms for systematically correcting misleading, incorrect, flawed, or potentially fraudulent research, which can be accepted due to the fallibility of peer review. The authors posit that an R{content}C Track would provide a reputable platform to critically challenge prior research, fostering a dynamic, self-correcting research ecosystem.
Problem Statement: Flaws in the ML Research Ecosystem
The authors identify several critical issues within the ML research landscape:
- Acceptance of Flawed Research: ML conferences often accept flawed research due to the fallibility of peer review. This can lead to the proliferation of false information, incentivizing less rigorous research, diverting resources from more rigorous work, and undermining public trust. Examples cited include contested papers in privacy, post-training of LLMs, parameter-free optimization, and neuroscience-inspired AI.
- Lack of Corrective Mechanisms: ML conferences lack official processes for rectifying the scientific record when published papers are later found to be flawed. This includes the absence of editorial boards with the power to remove flawed papers, mechanisms for attaching corrigenda or errata, and means to compel authors to withdraw their work.
- Impracticality of Reforming Peer Review: Meaningfully overhauling peer review to prevent the publication of flawed research faces obstacles such as inadequate incentives for reviewers, insufficient time for rigorous scrutiny, and a growing shortage of qualified reviewers.
- Undesirable Outcomes: The lack of formal recourse mechanisms can lead to the persistence and propagation of errors, delayed exposure of problematic research, low visibility of public comments on platforms like OpenReview, and reliance on informal channels like blogs and social media, which lack scrutiny by independent experts.
Proposed Solution: The "Refutations and Critiques" Track
To address these issues, the authors propose establishing a dedicated R{content}C Track within ML conferences. This track would:
- Provide a high-profile, reputable, and rigorously peer-reviewed platform for research that identifies, analyzes, and corrects misleading, incorrect, or potentially fraudulent claims in ML publications.
- Encourage researchers to submit responses and criticisms of others' and their own research.
- Integrate authors of critiqued papers into the peer review process to ensure fairness and quality.
- Cultivate a dynamic and robustly self-correcting research ecosystem.
The authors emphasize that the name "Refutations and Critiques" is intended to encourage a broader scope of engagement with prior work beyond just reproducibility assessments, including critiques of assumptions, methodologies, and interpretations.
Justification for a Standalone Track
The authors argue that a standalone R{content}C Track is merited for several reasons:
- Reduce Reviewer Indifference and Opposition: Critical responses can be harder to get through the peer review process due to reviewer indifference or affiliation with the prior work being criticized.
- Create Bespoke Reviewing Standards: Responses and critiques have unique aspects that are best served via unique standards of evaluation.
- Link Accepted Papers to Prior Work: A response or critique should be explicitly linked to the publication being critiqued, similar to corrigenda and errata.
The authors draw an analogy to the NeurIPS Datasets and Benchmarks Track, which was created to address the inadequate incorporation of datasets and benchmarks into the research process.
Guiding Principles for Evaluation of R{content}C Submissions
The success of the R{content}C Track will necessitate tailored review criteria, distinct from those for regular research papers. The authors propose the following principles to guide the evaluation of R{content}C submissions:
- Correct, Rigorous, and Meticulous: Submissions must demonstrate methodological soundness, reproducibility, and transparent analysis.
- Substantive: Submissions must address substantive aspects of the original work and have meaningful implications for the broader research community.
- Constructive: Submissions must aim to correct and improve the scientific record and guide future research, maintaining a professional tone and focusing on the research rather than the researchers.
- Significant: Submissions should be judged based on the influence of the criticized work and how significantly the submissions change the field's understanding.
Addressing Alternative Viewpoints
The authors acknowledge and address alternative perspectives regarding the utility and implementation of an R{content}C Track, including concerns about the sufficiency of existing mechanisms, the potential for frivolous or adversarial submissions, and the possibility of flawed refutations. They argue that the guiding principles and review process can mitigate these risks.
Conclusion
The authors conclude that the establishment of an R{content}C Track would add professionalism and visibility to the process of refuting prior work, streamline the scientific process, incentivize higher-quality papers, and pave the way for a more open scientific discussion. They emphasize the importance of producing a clean scientific record in a field like machine learning, where mistakes can have significant societal risks.