Towards Learning to Reason: Comparing LLMs with Neuro-Symbolic on Arithmetic Relations in Abstract Reasoning (2412.05586v1)

Abstract: This work compares LLMs and neuro-symbolic approaches in solving Raven's progressive matrices (RPM), a visual abstract reasoning test that involves the understanding of mathematical rules such as progression or arithmetic addition. Providing the visual attributes directly as textual prompts, which assumes an oracle visual perception module, allows us to measure the model's abstract reasoning capability in isolation. Despite providing such compositionally structured representations from the oracle visual perception and advanced prompting techniques, both GPT-4 and Llama-3 70B cannot achieve perfect accuracy on the center constellation of the I-RAVEN dataset. Our analysis reveals that the root cause lies in the LLM's weakness in understanding and executing arithmetic rules. As a potential remedy, we analyze the Abductive Rule Learner with Context-awareness (ARLC), a neuro-symbolic approach that learns to reason with vector-symbolic architectures (VSAs). Here, concepts are represented with distributed vectors s.t. dot products between encoded vectors define a similarity kernel, and simple element-wise operations on the vectors perform addition/subtraction on the encoded values. We find that ARLC achieves almost perfect accuracy on the center constellation of I-RAVEN, demonstrating a high fidelity in arithmetic rules. To stress the length generalization capabilities of the models, we extend the RPM tests to larger matrices (3x10 instead of typical 3x3) and larger dynamic ranges of the attribute values (from 10 up to 1000). We find that the LLM's accuracy of solving arithmetic rules drops to sub-10%, especially as the dynamic range expands, while ARLC can maintain a high accuracy due to emulating symbolic computations on top of properly distributed representations. Our code is available at https://github.com/IBM/raven-large-language-models.

• The paper compares the performance of Large Language Models (LLMs) like GPT-4 and Llama-3 with a neuro-symbolic model (ARLC) on abstract reasoning tasks, finding ARLC significantly outperforms LLMs, especially on tasks involving arithmetic relations.
• While LLMs show limited abstract arithmetic reasoning capabilities, especially with larger or higher-range inputs where accuracy drops below 10%, the ARLC model maintains robust accuracy even on expanded, out-of-distribution tasks.
• The findings suggest that neuro-symbolic approaches like ARLC offer better scalability and out-of-distribution generalization for abstract arithmetic reasoning than current LLMs, pointing towards potential in developing more robust AI reasoning systems.

Comparative Analysis of LLMs and Neuro-Symbolic Approaches in Abstract Reasoning

The paper "Towards Learning to Reason: Comparing LLMs with Neuro-Symbolic on Arithmetic Relations in Abstract Reasoning" presents a paper that evaluates the performance of LLMs against neuro-symbolic approaches in the context of solving visual abstract reasoning tasks, particularly Raven's Progressive Matrices (RPM). This analysis is motivated by understanding how these methodologies can achieve or potentially mimic human-like reasoning abilities through computational processes.

The authors focus on contrasting the performance of two prominent LLMs, GPT-4 and Llama-3 70B, against a neuro-symbolic model, the Abductive Rule Learner with Context-awareness (ARLC), which leverages vector-symbolic architectures (VSAs). The paper considers RPM tasks as an exemplary test case for abstract reasoning capabilities due to their requirement for understanding and applying arithmetic and pattern recognition rules.

Key Results and Findings

Performance and Accuracy:

1. LLMs Performance: The LLMs, specifically GPT-4 and Llama-3 70B, demonstrate limited abstract reasoning capabilities when evaluated on the I-RAVEN dataset. GPT-4 achieves an accuracy of 93.2%, while Llama-3 70B reaches 85.0% under the structured settings. These models show significant limitations in executing arithmetic rules despite possessing adequate performance in handling simple constant or progressive rules.
2. ARLC Accuracy: The ARLC's performance is notably superior, with an accuracy of 98.4% on I-RAVEN, indicating its efficacy in solving tasks that predominantly involve arithmetic reasoning. ARLC's architecture, founded on vector-symbolic architectures, promotes high dimensional encoding of attributes that preserve similarity through operations resembling arithmetic computations, yielding enhanced performance.
3. I-RAVEN-X Analysis: Extending beyond typical 3x3 matrices, the authors introduce I-RAVEN-X featuring expanded grid sizes (e.g., 3x10) and value ranges (10 to 1000). It is observed that LLMs show a notable drop in accuracy, especially with larger grids and ranges in the arithmetic rule, with accuracy plunging below 10% in high dynamic ranges. Conversely, ARLC maintains robust accuracy, evidencing its adaptability and scalability for complex configurations without requiring retraining.

Theoretical and Practical Implications

The findings highlight critical insights about the intrinsic strategy of existing LLMs versus neuro-symbolic approaches. While LLMs exhibit promising performance in disentangled, static prompts, their arithmetic execution in dynamic scopes is flawed, possibly due to inherent relational reasoning biases that focus on implicit associative patterns rather than explicit rule extraction and application.

This paper positions ARLC, with its neuro-symbolic foundations, as a viable alternative that performs symbolic computations with high accuracy, maintaining performance even with expanded problem dimensions. The method not only promises higher accuracy but also demonstrates OOD (out-of-distribution) generalization capabilities, relevant in the advancement and application of artificial intelligence systems in tasks requiring robust reasoning mechanisms.

Speculation on Future Directions

Future developments could potentially examine hybrid systems integrating the adaptability of LLMs with the structured precision of neuro-symbolic architectures like ARLC. Innovations in neuro-symbolic reasoning could lead to more practical AI systems capable of translating complex visual perception into logical reasoning, allowing applications in diverse domains such as automated problem-solving, cognitive robotics, and educational technology.

Moreover, extensive investigations into the scalability of LLMs, integrating structured decompositions over semantic tensor spaces akin to VSAs, could enhance their performance on reasoning tasks. Bridging these methodologies could result in models with comprehensive abstract reasoning capabilities essential for next-generation AI systems.

In conclusion, the paper provides valuable insights into the strengths and limitations of these approaches in abstract reasoning, emphasizing the need for further exploration and integration of cognitive and symbolic reasoning paradigms in the AI research landscape.