SymbolicAI: A framework for logic-based approaches combining generative models and solvers (2402.00854v4)

Abstract: We introduce SymbolicAI, a versatile and modular framework employing a logic-based approach to concept learning and flow management in generative processes. SymbolicAI enables the seamless integration of generative models with a diverse range of solvers by treating LLMs as semantic parsers that execute tasks based on both natural and formal language instructions, thus bridging the gap between symbolic reasoning and generative AI. We leverage probabilistic programming principles to tackle complex tasks, and utilize differentiable and classical programming paradigms with their respective strengths. The framework introduces a set of polymorphic, compositional, and self-referential operations for multi-modal data that connects multi-step generative processes and aligns their outputs with user objectives in complex workflows. As a result, we can transition between the capabilities of various foundation models with in-context learning capabilities and specialized, fine-tuned models or solvers proficient in addressing specific problems. Through these operations based on in-context learning our framework enables the creation and evaluation of explainable computational graphs. Finally, we introduce a quality measure and its empirical score for evaluating these computational graphs, and propose a benchmark that compares various state-of-the-art LLMs across a set of complex workflows. We refer to the empirical score as the "Vector Embedding for Relational Trajectory Evaluation through Cross-similarity", or VERTEX score for short. The framework codebase and benchmark are linked below.

• The paper introduces a NeSy framework that combines LLMs' zero- and few-shot learning with solver integration to dynamically construct and evaluate computational graphs.
• It proposes a novel VERTEX score, inspired by the Fréchet distance, to benchmark multi-step generative processes and complex reasoning tasks.
• The framework democratizes AI by fusing rule-based symbolic methods with sub-symbolic learning, paving the way for adaptive and autonomous systems.

Introduction

The field of AI has traditionally been divided into two approaches: symbolic AI, which focuses on rule-based systems, and sub-symbolic AI, reliant on neural networks and statistical methods. Bridging these is the domain of Neuro-Symbolic (NeSy) AI, which seeks to combine the strengths of both approaches to unlock new capabilities. The presented work on SymbolicAI introduces a NeSy framework that leverages LLMs within the generative AI landscape. This framework distinguishes itself by incorporating solvers for various domains—ranging from formal language engines to theorem provers—into the generative process, thereby extending the operational range of LLMs and enhancing their utility in complex workflows.

Framework Design

At the core of SymbolicAI is a logic-based philosophy for managing data streams, enabling the dynamic construction and evaluation of computational graphs. SymbolicAI capitalizes on the generative models' zero- and few-shot learning capabilities to model these graphs and uses a varied assembly of solvers for specialized problem-solving. To facilitate this, SymbolicAI defines symbols and expressions capable of morphing into complex structures via polymorphic operations. These operations, along with instruction-based models for fine-tuning LLMs, overarching benchmarking, and a novel quality measure – the VERTEX score, solidify the framework’s utility and robustness. The integration emphasizes the utilization of language as a vehicle for computation, offering a structured path for mapping real-world issues through language inference techniques.

Evaluation Protocol

The proposed VERTEX score tackles the challenge of evaluating multi-step generative processes. This measure is essential for assessing computational graphs and is derived using principles similar to the Fréchet distance in generative models. It captures the trajectory of distributions associated with generative nodes and evaluates them concerning a reference distribution. SymbolicAI claims to facilitate advanced learning protocols and the development of self-referential agents by creating an apparatus for LLMs to manage task execution autonomously. In an essential departure from conventional methods, the framework proposes a computational process devoid of rich-context limitations, enabling long-horizon planning and complex reasoning for machine-based learning models.

Contributions and Implications

SymbolicAI contributes notably to the field by providing a tangible framework that seamlessly integrates solvers with generative models. It operationalizes the construction of computational graphs capable of executing intricate tasks by leveraging LLMs as semantic parsers. Moreover, the framework advocates open-source AI development, which aligns with the broader push for democratizing AI technology.

Conclusion

SymbolicAI has the potential to serve as a foundation for versatile applications through its innovative melding of symbolic and sub-symbolic AI elements. Its introduction of a quality measure, combined with a significant benchmark, poses a substantial step forward in systematizing the evaluation of NeSy processes. Such a framework paves the road to more adaptive, efficient, and autonomous AI systems by enhancing the ability to learn and reason across various contexts and domains.