RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture (2401.08406v3)

Abstract: There are two common ways in which developers are incorporating proprietary and domain-specific data when building applications of LLMs: Retrieval-Augmented Generation (RAG) and Fine-Tuning. RAG augments the prompt with the external data, while fine-Tuning incorporates the additional knowledge into the model itself. However, the pros and cons of both approaches are not well understood. In this paper, we propose a pipeline for fine-tuning and RAG, and present the tradeoffs of both for multiple popular LLMs, including Llama2-13B, GPT-3.5, and GPT-4. Our pipeline consists of multiple stages, including extracting information from PDFs, generating questions and answers, using them for fine-tuning, and leveraging GPT-4 for evaluating the results. We propose metrics to assess the performance of different stages of the RAG and fine-Tuning pipeline. We conduct an in-depth study on an agricultural dataset. Agriculture as an industry has not seen much penetration of AI, and we study a potentially disruptive application - what if we could provide location-specific insights to a farmer? Our results show the effectiveness of our dataset generation pipeline in capturing geographic-specific knowledge, and the quantitative and qualitative benefits of RAG and fine-tuning. We see an accuracy increase of over 6 p.p. when fine-tuning the model and this is cumulative with RAG, which increases accuracy by 5 p.p. further. In one particular experiment, we also demonstrate that the fine-tuned model leverages information from across geographies to answer specific questions, increasing answer similarity from 47% to 72%. Overall, the results point to how systems built using LLMs can be adapted to respond and incorporate knowledge across a dimension that is critical for a specific industry, paving the way for further applications of LLMs in other industrial domains.

Overview of RAG and Fine-Tuning

The utilization of LLMs such as GPT-4 in domain-specific applications involves two primary methods: Retrieval-Augmented Generation (RAG) and fine-tuning. RAG enriches the prompt given to the model with external data, while fine-tuning incorporates additional knowledge directly into the model. This paper outlines a pipeline that combines these methods to improve LLMs' performance in industry-specific contexts, with a particular focus on agriculture.

The Pipeline and Evaluation Metrics

The researchers propose a pipeline that includes stages such as extracting information from PDFs, generating questions and answers, and refining the data for model fine-tuning. The performance of RAG and fine-tuning is assessed using meticulously crafted metrics. These metrics gauge the relevance and quality of questions and answers, as well as the models' ability to incorporate spatially scoped knowledge vital for the examined industry domain.

Fine-Tuning Versus RAG

The paper provides an extensive comparison of the results achieved through RAG and fine-tuning. It highlights the merits and trade-offs of each approach. While RAG is especially advantageous for its low initial costs and improved accuracy by providing contextual relevance, fine-tuning is commended for precise outputs tailored to specific domain knowledge. However, the high initial cost of fine-tuning is acknowledged.

Applications and Future Directions

By illustrating the qualitative and quantitative benefits of both RAG and fine-tuning for different models including GPT-4, the paper paves the way for further applications of LLMs in various industrial segments. It suggests future explorations into how to enhance the structured extraction of document information for the development of knowledgeable AI systems and the potential for multimodal fine-tuning with visuals and text.

The paper concludes that while both RAG and fine-tuning have their specific applications, combining both methods could lead to significant enhancements in LLMs for datasets specific to particular industries. This advancement could extend beyond just agriculture to any domain where specialized knowledge is paramount.