StructuredRAG: JSON Response Formatting with Large Language Models (2408.11061v1)

Abstract: The ability of LLMs to generate structured outputs, such as JSON, is crucial for their use in Compound AI Systems. However, evaluating and improving this capability remains challenging. In this work, we introduce StructuredRAG, a benchmark of six tasks designed to assess LLMs' proficiency in following response format instructions. We evaluate two state-of-the-art LLMs, Gemini 1.5 Pro and Llama 3 8B-instruct with 4-bit quantization using two distinct prompting strategies. We introduce these prompting strategies as f-String and Follow the Format (FF) prompting. Across 24 experiments, we find an average success rate of 82.55%. We further find a high variance in performance across tasks, models, and prompting strategies with success rates ranging from 0 to 100%. We find that Llama 3 8B-instruct often performs competitively with Gemini 1.5 Pro. We observe that task complexity significantly influences performance, with tasks involving lists or composite object outputs proving more challenging. Our findings highlight the need for further research into improving the reliability and consistency of structured output generation in LLMs. We have open-sourced our experimental code and results at github.com/weaviate/structured-rag.

• The paper introduces StructuredRAG, a benchmark assessing LLMs' JSON response generation with an average success rate of 82.55% across diverse tasks.
• Methodologies leverage innovative f-String and Follow the Format prompting techniques to compare performance between Gemini 1.5 Pro and Llama 3 8B-instruct.
• The study highlights challenges in formatting complex JSON structures and calls for further research to improve LLM integration in compound AI systems.

StructuredRAG: JSON Response Formatting with LLMs

The paper, "StructuredRAG: JSON Response Formatting with LLMs," presents a significant contribution to the domain of AI systems that demand structured outputs from LLMs. This research addresses the evaluation of LLMs' capabilities in structured output generation, specifically JSON format, which is essential for integrating LLMs into Compound AI Systems.

Core Contributions

The paper introduces StructuredRAG, a benchmark comprising six tasks aimed at assessing LLMs' abilities to follow JSON response format instructions through Zero-Shot Learning. The authors evaluate two state-of-the-art models, Gemini 1.5 Pro and Llama 3 8B-instruct, using novel prompting strategies: f-String and Follow the Format (FF) prompting.

Key results include an average success rate of 82.55% across 24 experiments, highlighting the variability in LLM performance. Notably, simpler tasks exhibit higher success rates, while tasks involving more complex structures, such as lists or composite objects, prove challenging. The research underscores the necessity for further exploration to enhance reliability in structured output generation.

Methodological Insights

The researchers utilize a structured and rigorous methodological approach:

• Benchmark Design: StructuredRAG is crafted to test various outputs, including strings, integers, and boolean values, alongside composite structures. This comprehensive testing matrix elucidates LLM performance across diverse JSON compositions.
• Prompting Strategies: The paper explores f-String and FF prompting, finding no significant consensus on which method offers superior success. However, the clear variance in outcomes across tasks suggests an intricate relationship between prompting strategy and task complexity.
• Model Comparison: The comparative analysis between Gemini 1.5 Pro and Llama 3 8B-instruct indicates that although Gemini 1.5 Pro generally outperforms Llama 3 8B-instruct, the latter shows competitive capabilities in specific tasks.

Numerical Results

• Performance Metrics: Gemini 1.5 Pro achieves an average success rate of 93.4%, notably higher than Llama 3 8B-instruct's 71.7%. Task complexity impacts success rates, with outputs involving lists and composite objects seeing reduced performance.
• Variability and Failures: The performance variability is significant, with some tasks encountering success rates as low as 0%. Models excel in different tasks, with Gemini 1.5 Pro demonstrating higher consistency.

Implications and Future Directions

The implications of this research are multifaceted:

• Theoretical Impact: By highlighting the innate challenges in structured output generation, the paper signifies the complexity of JSON formatting for LLMs, revealing an area ripe for algorithmic enhancement.
• Practical Applications: The findings suggest potential strategies for improving LLM integration into Compound AI Systems, primarily through refined prompting tactics and structured decoding.
• OPRO Optimization: The introduction of OPRO optimization demonstrates a 100% success rate in specific tasks, indicating promising avenues for prompt optimization techniques.

Conclusion

"StructuredRAG: JSON Response Formatting with LLMs" contributes critical insights into the capabilities and limitations of LLMs in generating structured outputs. This research sets the foundation for future exploration in optimizing LLM responses for integration into complex AI systems. The paper reveals substantial opportunities for improvement through advanced methods such as ensembling, retry mechanisms, and chain-of-thought prompting, warranting further investigation into response formatting without resorting to structured decoding methods.