When No Paths Lead to Rome: Benchmarking Systematic Neural Relational Reasoning

Abstract: Designing models that can learn to reason in a systematic way is an important and long-standing challenge. In recent years, a wide range of solutions have been proposed for the specific case of systematic relational reasoning, including Neuro-Symbolic approaches, variants of the Transformer architecture, and specialised Graph Neural Networks. However, existing benchmarks for systematic relational reasoning focus on an overly simplified setting, based on the assumption that reasoning can be reduced to composing relational paths. In fact, this assumption is hard-baked into the architecture of several recent models, leading to approaches that can perform well on existing benchmarks but are difficult to generalise to other settings. To support further progress in the field of systematic relational reasoning with neural networks, we introduce NoRA, a new benchmark which adds several levels of difficulty and requires models to go beyond path-based reasoning.

• The paper introduces NoRA, a benchmark that challenges path-based neural relational reasoning with complex, ambiguous facts and multi-relational entity pairs.
• It employs systematic metrics such as reasoning depth, backtrack load, and off-path edge count to quantify the complexity of inference tasks.
• Empirical results reveal that state-of-the-art models, including edge transformers and LLMs, significantly struggle with off-path reasoning, indicating a need for novel architectures.

Benchmarking Systematic Neural Relational Reasoning Beyond Path-Based Inference

Introduction

The paper "When No Paths Lead to Rome: Benchmarking Systematic Neural Relational Reasoning" (2510.23532) introduces NoRA, a benchmark designed to rigorously evaluate neural models' capacity for systematic relational reasoning in settings that transcend the limitations of path-based inference. Existing benchmarks, such as CLUTRR, predominantly focus on scenarios where relational reasoning can be reduced to composing relations along a single path in a knowledge graph. This has led to the development of models with strong inductive biases toward path-based reasoning, which struggle to generalize to more complex relational structures. NoRA is constructed to break these inductive biases by introducing richer relational structures, ambiguous facts, and multi-relational entity pairs, thereby requiring models to perform reasoning that cannot be captured by simple path composition.

Problem Formulation and Dataset Construction

NoRA formalizes relational reasoning as the task of predicting all relationships entailed between two entities in a story, given a set of world rules encoded in Answer Set Programming (ASP). Stories consist of unary, binary, and ambiguous facts, the latter expressed as cardinality constraints (e.g., "entity $a$ is the father of either $b$ or $c$"). The reasoning process involves generating all possible refinements of a story by resolving ambiguities, applying definite rules to infer new facts, and enforcing constraints to eliminate inconsistent refinements. The ground truth for each query is the intersection of relationships that hold across all valid answer sets.

NoRA's data generation process leverages Clingo to sample stories and compute answer sets, ensuring that each instance is non-trivial and requires inference beyond explicit facts. The benchmark introduces several metrics to quantify problem difficulty:

• Reasoning Depth: The maximal number of inference steps required to derive the answer.
• Reasoning Width: The number of distinct derivations/proofs across all refinements.
• Backtrack Load (BL): The ratio of inference steps to the number of entities involved, capturing non-path reasoning complexity.
• Off-Path Edge Count (OPEC): The number of edges in a derivation not on any direct path between the source and target entities.

These metrics enable the construction of systematic train/test splits, with test sets specifically designed to be out-of-distribution along one or more axes (e.g., higher reasoning depth, width, BL, or OPEC than seen during training).

Analysis of Model Performance

The empirical evaluation covers state-of-the-art models for systematic relational reasoning, including Edge Transformers (ETs), Relation-Aware Transformers (RAT), Epistemic GNNs (EpiGNN), NBFNet, and R-GCN. Models are trained on both ambiguous and non-ambiguous datasets, and evaluated on in-distribution and out-of-distribution test splits.

Figure 1: Analysis of the performance of ETs on various splits of the dataset.

Edge Transformers consistently outperform other architectures on in-distribution and most out-of-distribution splits, but all models exhibit a marked decline in accuracy on test sets requiring high OPEC or BL, indicating a fundamental limitation in handling off-path reasoning. Notably, GNN-based models (EpiGNN, NBFNet, R-GCN) perform poorly on BL and OPEC splits, reflecting their strong alignment with path-based reasoning. The margin-based loss is shown to be unsuitable for multi-label settings, with binary cross-entropy yielding better results.

Figure 2: Breakdown of the performance of edge transformers on Test-D, controlling for BL and reasoning width.

When controlling for BL and reasoning width, ET performance drops sharply as reasoning depth increases, especially for multi-edge variants, underscoring the need for more expressive input representations and architectures capable of compositional generalization.

Expanded Benchmarking: NoRA v1.1 and HetioNet

To address the challenge of generating sufficiently hard problem instances, the authors introduce NoRA v1.1, which uses recursive subgraph expansion to systematically construct examples with higher OPEC, reasoning depth, and BL. This guarantees that every test instance can be solved by compositional generalization from training examples.

Figure 3: Results for the expanded version of NoRA (v1.1) that uses recursive subgraph expansion to generate harder splits along the axes: (a) OPEC, (b) Reasoning Depth (c) BL.

The inability of current models to handle off-path reasoning remains robust across variations in data generation, as evidenced by consistent performance drops on high-OPEC and high-BL splits in NoRA v1.1.

Additionally, the HetioNet dataset, based on biomedical entities and relations, demonstrates that the observed limitations are not specific to family-relationship domains. While EpiGNN outperforms ETs on HetioNet's OPEC split, the general trend of poor performance on off-path reasoning persists, suggesting that architectural advances are needed to capture more complex relational regularities.

Ambiguity and Shortcut Reasoning

NoRA incorporates ambiguous facts to reflect real-world uncertainty, requiring models to reason over multiple possible worlds and enforce constraints. Surprisingly, ambiguity does not significantly degrade model performance, which is attributed to the presence of shortcuts that allow models to bypass explicit reasoning about ambiguity. Hard ambiguous instances—where ambiguity must be resolved to answer the query—are rare, but when present, model accuracy drops substantially.

Figure 4: Train-A (with ambiguity).

Figure 5: Illustration of the correlation between OPEC and BL using box plots of BL distributions for various OPEC values. The top panel shows data generated with ambiguous facts, and the bottom panel shows data generated without ambiguous facts.

Large Reasoning Models and Rule Recovery

Large Reasoning Models (LRMs), such as OpenAI's o3 and o4-mini, are evaluated on NoRA with explicit access to world rules. While LRMs achieve near-perfect accuracy on path-based queries (OPEC=0), their performance degrades sharply on off-path queries (OPEC=3), even when the required rules are provided. This suggests that current LLMs, despite their rule-learning capabilities, are not equipped for systematic off-path relational reasoning.

Figure 6: Performance of OpenAI's o3 model on Query Completion and Rule Recovery Tasks. Results separated according to OPEC and the reasoning depth of examples.

Practical and Theoretical Implications

The results demonstrate that systematic relational reasoning in neural models is fundamentally limited by architectural inductive biases toward path-based inference. Off-path reasoning, multi-relational entity pairs, and ambiguity present significant challenges that are not addressed by current state-of-the-art models or LLMs. The findings suggest that progress in neural relational reasoning will require fundamentally new architectures capable of compositional generalization and constraint-based reasoning.

The introduction of NoRA, NoRA v1.1, and HetioNet provides a rigorous framework for evaluating such models, with difficulty metrics enabling fine-grained analysis and systematic generalization testing. The benchmark's realistic world rules and ambiguous facts make it suitable for both neural and symbolic approaches, as well as for evaluating LLMs' reasoning capabilities.

Figure 7: Distribution of predicates/relationships in the NoRA-1.1 training set.

Conclusion

NoRA sets a new standard for benchmarking systematic neural relational reasoning, exposing the limitations of path-based models and highlighting the need for architectures that can handle off-path inference, ambiguity, and multi-relational reasoning. The empirical results indicate that neither current neural architectures nor LLMs are capable of robust systematic generalization in these settings. Future research should focus on developing models with explicit mechanisms for compositional rule induction, constraint satisfaction, and reasoning over multiple possible worlds. The benchmark and datasets introduced in this work will be instrumental in driving such advances.