TreeGPT: Pure TreeFFN Encoder-Decoder Architecture for Structured Reasoning Without Attention Mechanisms (2509.05550v2)

Abstract: We present TreeGPT, an attention-free neural architecture that explores the potential of pure TreeFFN encoder-decoder design for structured reasoning tasks. Unlike traditional transformer approaches that rely on attention mechanisms, TreeGPT employs bidirectional TreeFFN components that process sequences through adjacent connections in parallel, aiming to achieve computational efficiency while maintaining reasoning capabilities. Our approach centers on a TreeFFN Encoder-Decoder mechanism: $$\text{Encoder TreeFFN (L} \rightarrow \text{R)} + \text{Decoder TreeFFN (R} \leftarrow \text{L)} \rightarrow \text{Parallel Processing}$$ where the encoder processes left-to-right dependencies while the decoder handles right-to-left patterns, both using simple neighbor-to-neighbor connections. This design eliminates attention computation while maintaining sequence modeling capabilities. We evaluate our approach on the ARC Prize 2025 dataset, where TreeGPT achieves 99\% validation accuracy using 3.16M parameters. The model converges within 1500 training steps and demonstrates 100\% token-level accuracy on selected evaluation samples. Our preliminary results suggest that for certain structured reasoning tasks, specialized TreeFFN architectures may offer advantages over attention-based approaches. While these findings are encouraging, we acknowledge that further investigation across diverse tasks and datasets would be valuable to establish the broader applicability of attention-free designs.

• The paper introduces TreeGPT, a neural architecture that eliminates attention mechanisms in favor of a bidirectional TreeFFN design.
• The architecture processes sequences in parallel using adjacent connections, achieving 99% validation accuracy on challenging visual reasoning tasks with only 3.16M parameters.
• Experimental results indicate TreeGPT outperforms state-of-the-art models while significantly reducing computational complexity, setting a new direction for structured reasoning in AI.

TreeGPT: Pure TreeFFN Encoder-Decoder Architecture for Structured Reasoning Without Attention Mechanisms

Introduction

The presented paper introduces TreeGPT, a neural architecture designed to tackle structured reasoning tasks without relying on standard attention mechanisms. This is accomplished through a bidirectional pure Tree Feed-Forward Network (TreeFFN) design that facilitates computational efficiency while retaining robust reasoning capabilities. TreeGPT's framework addresses the limitations of traditional transformer architectures, which often struggle with computational inefficiencies and insufficient information propagation when dealing with hierarchical structures, such as Abstract Syntax Trees (ASTs).

Methodology

Pure TreeFFN Encoder-Decoder Architecture

TreeGPT's core innovation lies in abandoning attention mechanisms in favor of a bidirectional TreeFFN setup that processes sequences in parallel through straightforward adjacent connections, thereby greatly reducing computational complexity. The encoder and decoder simultaneously handle left-to-right and right-to-left dependencies, respectively.

• Encoder: Processes sequences with incremental left-to-right adjacency.
• Decoder: Handles sequences in a reverse, right-to-left adjacency manner.
• Parallel Processing: Executes both encoder and decoder operations on entire sequences simultaneously, entirely eliminating the need for attention calculations.

  Figure 1: TreeGPT Pure TreeFFN Encoder-Decoder Architecture. The model processes sequences through bidirectional TreeFFN components: encoder processes left-to-right dependencies while decoder handles right-to-left generation, both operating in parallel with adjacent connections only. No attention mechanisms are employed.

Component Design and Processing

The TreeFFN component allows various enhancements and adjustments:

• Edge Projections: Transform edge features via learned projections, enabling efficient information flow.
• Gated Aggregation: Adaptive gating for information flow control.
• Residual Connections: Optionally maintain gradient flow to prevent overfitting.

The processing pipeline ensures that both adjacent connections and the resulting outputs leverage parallel efficiency, as outlined in the pipeline flow algorithm provided, illustrating the sequential execution of both TreeFFN components.

Figure 2: Component Flow Diagram illustrating the TreeFFN processing pipeline with detailed visualization of the global parent-child aggregation mechanism.

Experimental Results

Performance on ARC-AGI-2 Dataset

TreeGPT's performance was assessed on the ARC-AGI-2 dataset, renowned for its challenging visual reasoning tasks. Compared to existing models, TreeGPT achieves remarkable accuracy with optimized parameter usage:

• Parameter Efficiency: TreeGPT uses only 3.16 million parameters, achieving 99% validation accuracy within 1500 training steps.
• Component Effectiveness: Ablation studies underscore the critical role of edge projections.

  Figure 3: ARC-AGI Visual Reasoning Task Examples. The figure shows three representative task types: (top) pattern tiling where models must extend geometric patterns, (middle) rectangle filling where enclosed areas must be filled with specific colors, and (bottom) color transformation where consistent color mapping rules must be applied. Each task requires understanding abstract rules from input grids and generating correct outputs.

Comparison with State-of-the-Art

The benchmark comparison displays TreeGPT's competitive edge over existing models:

• Small and LLMs: TreeGPT outpunched models like DeepSeek-R1 and Grok-4, with better performance and far fewer parameters.
• Program Synthesis Methods: Demonstrated superiority against tailored program synthesis techniques like SOAR and the Greenblatt Method.

Discussion

Implications and Theoretical Insights

TreeGPT's empirical results highlight its potential for cutting down computational loads while improving accuracy in structured reasoning. The ability to process hierarchical data without dependency on traditional attention opens pathway adjustments for efficient architecture alternatives in other domains.

Future Perspectives

Moving forward, TreeGPT's framework may expand to accommodate multimodal inputs, undefined tree structures, and scaled implementations suitable for broader AI reasoning systems. Further exploration of fusion techniques between attentionless TreeFFNs and other architectural paradigms stands to drive advancements in the efficiency and general application of structured reasoning AI.

Conclusion

TreeGPT successfully introduces an attention-free paradigm suitable for structured reasoning tasks, offering computational efficiency and robust performance while capitalizing on TreeFFNs' inherent advantages. Although preliminary and focused on specific datasets, the results suggest valuable insights for future AI systems development aimed at hierarchical data processing. Future research will extend TreeGPT's framework and evaluate its full potential to redefine structured reasoning paradigms in AI.