Free to play: UN Trade and Development's experience with developing its own open-source Retrieval Augmented Generation Large Language Model application (2407.16896v1)

Abstract: Generative AI, and in particular LLMs, have exploded in popularity and attention since the release to the public of ChatGPT's Generative Pre-trained Transformer (GPT)-3.5 model in November of 2022. Due to the power of these general purpose models and their ability to communicate in natural language, they can be useful in a range of domains, including the work of official statistics and international organizations. However, with such a novel and seemingly complex technology, it can feel as if generative AI is something that happens to an organization, something that can be talked about but not understood, that can be commented on but not contributed to. Additionally, the costs of adoption and operation of proprietary solutions can be both uncertain and high, a barrier for often cost-constrained international organizations. In the face of these challenges, United Nations Trade and Development (UNCTAD), through its Global Crisis Response Group (GCRG), has explored and developed its own open-source Retrieval Augmented Generation (RAG) LLM application. RAG makes LLMs aware of and more useful for the organization's domain and work. Developing in-house solutions comes with pros and cons, with pros including cost, flexibility, and fostering institutional knowledge. Cons include time and skill investments and gaps and application polish and power. The three libraries developed to produce the app, nlp_pipeline for document processing and statistical analysis, local_rag_LLM for running a local RAG LLM, and streamlit_rag for the user interface, are publicly available on PyPI and GitHub with Dockerfiles. A fourth library, local_LLM_finetune, is also available for fine-tuning existing LLMs which can then be used in the application.

Overview of UNCTAD's Open-Source RAG LLM Application Development

The paper "Free to play: UN Trade and Development's experience with developing its own open-source RAG LLM application" explores the development of an autonomous Retrieval Augmented Generation (RAG) LLM application by UN Trade and Development (UNCTAD). This initiative emerged as a solution to the growing need for cost-effective, flexible AI solutions within international organizations.

Background and Motivation

The advent of OpenAI's ChatGPT ushered in widespread interest in generative AI, with LLMs demonstrating their utility across various domains, including official statistics and international agencies. Despite their potential, the proprietary nature and high operational costs of existing AI models present significant barriers for organizations like UNCTAD. In response, UNCTAD, through its Global Crisis Response Group (GCRG), embarked on an initiative to create an open-source LLM application tailored to its specific needs. This decision was driven by considerations of cost, flexibility, and the importance of developing institutional knowledge.

Technical Implementation

UNCTAD's solution involves the creation of a RAG LLM application, which addresses LLMs' intrinsic limitations related to their training data. LLMs often lack current or domain-specific information due to the static nature of their training. RAG enhances LLMs by incorporating a retrieval step that queries a vector database to supply the model with relevant, up-to-date information. UNCTAD's approach leverages several open-source technologies:

• Document Processing and Embedding: Utilizing the nlp_pipeline library for text conversion and metadata enrichment, making it compatible with the embedding process.
• Vector Database: Implementing pgvector with PostgreSQL for vector storage and retrieval.
• User Interface: Deploying streamlit_rag for a streamlined user experience.
• Fine-Tuning Capabilities: Employing the local_LLM_finetune library for domain-specific model adjustment, enhancing the LLM's applicability to UNCTAD's corpus.

These components collectively facilitate an adaptable, user-oriented application capable of handling a variety of queries within the constraints of UNCTAD's domain.

Advantages of the In-House Development Approach

1. Cost Efficiency: Developing an in-house solution primarily involves staff salaries and infrastructure costs, avoiding unpredictable pricing models associated with proprietary services.
2. Flexibility and Control: The open-source approach provides significant flexibility in adapting the application to specific organizational needs without dependency on external providers, avoiding potential limitations from commercial offerings.
3. Institutional Knowledge and Capacity Building: Building the application internally fosters technical expertise, enhancing the organization’s ability to innovate and operate independently of third-party technologies.
4. Data Privacy: The solution ensures data is confined to organizationally controlled environments, mitigating risks associated with external data handling and breaches.

Challenges and Constraints

Despite the evident advantages, in-house development presents challenges, including the requirement for skilled personnel and the potential scalability issues that proprietary solutions typically alleviate. Additionally, open-source models may lag behind in performance compared to frontier proprietary LLMs.

Implications and Future Directions

The successful deployment of UNCTAD's application underscores the feasibility of open-source solutions for international organizations, promoting a model of innovation that aligns with institutional constraints. Future enhancements may involve further optimizing the RAG pipeline, fine-tuning models with more sophisticated datasets, and exploring hybrid models that integrate both proprietary and open-source components to balance performance and cost.

Conclusion

UNCTAD's experience provides valuable insights into how international organizations can leverage open-source AI technologies to meet their unique requirements without incurring substantial costs. The approach detailed in this paper not only exemplifies a pragmatic pathway for adopting AI but also enriches the broader open-source ecosystem, potentially inspiring similar initiatives among national statistics offices and beyond. Through sharing its journey, UNCTAD contributes to a growing body of knowledge, facilitating broader access to advanced AI technologies within the global development community.