Querying Databases with Function Calling

Abstract: The capabilities of LLMs are rapidly accelerating largely thanks to their integration with external tools. Querying databases is among the most effective of these integrations, enabling LLMs to access private or continually updating data. While Function Calling is the most common method for interfacing external tools to LLMs, its application to database querying as a tool has been underexplored. We propose a tool definition for database querying that unifies accessing data with search queries, filters, or a combination both, as well as transforming results with aggregation and groupby operators. To evaluate its effectiveness, we conduct a study with 8 LLMs spanning 5 model families. We present a novel pipeline adapting the Gorilla LLM framework to create synthetic database schemas and queries. We primarily evaluate the models with the Exact Match of predicted and ground truth query APIs. Among the models tested, Claude 3.5 Sonnet achieves the highest performance with an Exact Match score of 74.3%, followed by GPT-4o mini at 73.7%, and GPT-4o at 71.8%. We further breakdown these results per API component utilized and across synthetic use cases. We find that LLMs are highly effective at utilizing operators on boolean properties, but struggle with text property filters. Across use cases we find robust results with the higher performing models such as GPT-4o, but significant performance variance across use cases from lower performing models. We additionally conduct ablation studies exploring the impact of parallel tool calling, adding a rationale as an argument of the tool call, using a separate tool per database collection, and tool calling with structured outputs. Our findings demonstrate the effectiveness of enabling LLMs to query databases with Function Calling. We have open-sourced our experimental code and results at github.com/weaviate/gorilla.

• The paper introduces DBGorilla, a benchmark for assessing function calling that translates natural language queries into structured database operations.
• It compares eight LLMs using Exact Match, AST alignment, and preference rankings across simple to complex query tasks.
• Ablation studies reveal that while the tool definition is robust, LLMs struggle with text property filters, highlighting areas for improvement.

Querying Databases with Function Calling

This paper (2502.00032) introduces DBGorilla, a benchmark for evaluating LLMs in querying databases using Function Calling. It presents a tool definition that unifies data access with search queries and result transformations, evaluating the effectiveness of this approach across multiple LLMs. The study also includes ablation studies to assess the impact of various experimental factors on performance.

DBGorilla Benchmark

The DBGorilla benchmark adapts the Gorilla LLM framework to create synthetic search database schemas and queries. It consists of five use cases, each with three related collections and four properties per collection. This benchmark tests the ability of LLMs to translate natural language commands into query APIs, covering search queries, property filters, aggregations, and grouping operations.

Experimental Setup

The experimental setup involves comparing eight LLMs from five model families, including Claude 3.5 Sonnet, GPT-4o, GPT-4o mini, Gemini 1.5 Pro, Gemini 2.0 Flash experimental, Command R+, Command R7B, and Llama 3.1 8B Instruct. The primary evaluation metric is Exact Match, which assesses if the predicted query from the LLM is identical to the ground truth query. The study also reports Abstract Syntax Tree (AST) alignment scores and LLM-as-Judge preference rankings.

Figure 1: DBGorilla Leaderboard results (last updated January 1st, 2025). The Exact Match and AST Score columns report the respective averages across all tested queries. Query scores are further separated into categories of "Simple", "Moderate", and "Complex" according to how many arguments are used in the ground truth function call with 1, 2, and 3 or more, respectively. Collection routing reports the percentage the predicted query is routed to the correct database collection.

Results and Analysis

Claude 3.5 Sonnet achieves the highest performance with an Exact Match score of 74.3%, followed by GPT-4o mini at 73.7%, GPT-4o at 71.8%, and Gemini 1.5 Pro at 70.2%. The results are broken down by API component, revealing that LLMs are highly effective at utilizing operators on boolean-valued properties but struggle with text property filters. The study also visualizes performance across synthetic use cases, showing robust results with higher-performing models like GPT-4o, but significant performance variance with lower-performing models.

Figure 2: An illustration of a natural language command, How many menu items are priced under 20?, translated to Function Calling arguments for database querying.

Ablation Studies

The paper includes ablation studies exploring the impact of parallel tool calling, adding a rationale as an argument of the tool call, using a separate tool per database collection, and tool calling with structured outputs. The results show minimal performance variance across these ablation experiments with GPT-4o, indicating that the tool definition is robust across different configurations.

Figure 3: An illustration of the Function Calling loop. Beginning with the user's input prompt, the LLM then enters a loop where it can either choose to call one or multiple functions, or return a response to the user. If a function is called, the function is executed, the response is sent back to the LLM, and the Function Calling loop continues.

Implications and Future Directions

The findings suggest that Function Calling is an effective interface for enabling LLMs to query databases. The DBGorilla benchmark provides a valuable resource for future research in this area, with potential expansions to include more complex schemas, diverse property distributions, and explicit relationships between collections. Future work could also explore the integration of other tools, such as web search and data visualization, to create more sophisticated Compound AI Systems. Techniques such as Reflexion prompting [15] and DSPy Assertions could be used to refine queries based on validation or user feedback.

Conclusion

This paper demonstrates the effectiveness of Function Calling for enabling natural language database access. The DBGorilla benchmark and the comprehensive evaluation of multiple LLMs provide valuable insights into the capabilities and limitations of current models in translating natural language to structured database operations. The research contributes to the development of more intuitive and efficient interfaces for querying databases with LLMs.