Formal human evaluation and difficulty calibration of P-ARC

Establish formal human solve-rate and solve-time measurements for the P-ARC benchmark, and determine its difficulty relative to ARC-AGI-1 and ARC-AGI-2 rather than relying on a qualitative estimate.

Background

P-ARC is a held-out benchmark of 50 hand-crafted ARC-style tasks intended to evaluate abstract rule competence on previously unseen tasks. During construction, candidate tasks were shown to team members other than their authors, and tasks were retained when at least one reviewer produced the correct output. This feasibility procedure was not designed as a formal study: multiple attempts were permitted, solve times were not recorded, and the resulting assessment of P-ARC’s difficulty relative to ARC-AGI-1 and ARC-AGI-2 remains qualitative.

A formal human evaluation would provide quantitative solve-rate and solve-time measurements and would allow the benchmark’s difficulty positioning to be empirically calibrated. The authors explicitly leave this study for future work, making the issue an unresolved methodological question rather than a result established by the paper.

References

Solve times were not recorded, so we report no solve-rate or solve-time statistics; a formal human study is left to future work, and our positioning of P-ARC between ARC-AGI-1 and ARC-AGI-2 in difficulty remains a qualitative estimate.

— PotARCin: Multi-Dimensional Evaluation of Skill Acquisition in Abstract Reasoning Tasks  (2609.27288 - Beger et al., 23 Sep 2026) in Appendix, Section “P-ARC Construction and Calibration,” subsection “Feasibility check”