Dice Question Streamline Icon: https://streamlinehq.com

Objective, algorithmic classification of natural-language rules

Develop an objective, algorithmic method to classify natural-language rules for ConceptARC tasks into the categories correct-intended, correct-unintended, and incorrect, reducing reliance on subjective human judgment.

Information Square Streamline Icon: https://streamlinehq.com

Background

The authors categorize rules into correct-intended, correct-unintended, and incorrect, but this process currently relies on manual evaluation and consensus among annotators, which introduces subjectivity.

They report that attempts to automate such evaluations using LLMs were not sufficiently successful and explicitly note the absence of an objective algorithmic method for this classification.

References

Our classification of human- and machine-generated rules was done manually, and involved some subjectivity; we do not know of any objective or algorithmic means to usefully classify these natural-language rules into our various categories.

— Do AI Models Perform Human-like Abstract Reasoning Across Modalities? (2510.02125 - Beger et al., 2 Oct 2025) in Section: Limitations