Agentic AI Systems Applied to tasks in Financial Services: Modeling and model risk management crews (2502.05439v2)

Abstract: The advent of LLMs has ushered in a new era of agentic systems, where artificial intelligence programs exhibit remarkable autonomous decision-making capabilities across diverse domains. This paper explores agentic system workflows in the financial services industry. In particular, we build agentic crews with human-in-the-loop module that can effectively collaborate to perform complex modeling and model risk management (MRM) tasks. The modeling crew consists of a judge agent and multiple agents who perform specific tasks such as exploratory data analysis, feature engineering, model selection/hyperparameter tuning, model training, model evaluation, and writing documentation. The MRM crew consists of a judge agent along with specialized agents who perform tasks such as checking compliance of modeling documentation, model replication, conceptual soundness, analysis of outcomes, and writing documentation. We demonstrate the effectiveness and robustness of modeling and MRM crews by presenting a series of numerical examples applied to credit card fraud detection, credit card approval, and portfolio credit risk modeling datasets.

• The paper introduces a framework using two agentic AI "crews" (modeling and MRM), powered by LLMs and memory, to perform complex financial tasks collaboratively.
• Experiments across credit card fraud detection, approval, and portfolio risk show the system achieves competitive performance and manages data intricacies effectively.
• Implementing such systems offers significant implications for finance by streamlining workflows, enhancing model accuracy, ensuring regulatory compliance, and providing scalable solutions.

Agentic AI Systems in Financial Services: Modeling and Model Risk Management

The paper "Agentic AI Systems Applied to Tasks in Financial Services: Modeling and Model Risk Management Crews" explores the application of agentic AI systems, empowered by LLMs, within the financial services sector. The research focuses on harnessing the autonomous decision-making capabilities of such systems through "agentic crews" designed to collaboratively undertake complex modeling and model risk management (MRM) tasks.

Overview and Structure

The paper delineates a comprehensive framework consisting of two separate crews: the modeling crew and the MRM crew. Each crew is composed of a manager and specialized agents, each agent programmed to carry out specific functions within the broader task. The modeling crew engages in data processing and model development, including exploratory data analysis, feature engineering, model selection, hyperparameter tuning, and model training and evaluation. Meanwhile, the MRM crew ensures compliance and reliability through documentation verification, model replication, conceptual soundness assessment, and outcome analysis.

Methodology and Experimentation

A central highlight of the paper is the detailed implementation process of these crews, articulated through agentic workflows. The collaborative nature of these crews is underpinned by a memory module, allowing agents to store and retrieve information efficiently, facilitating a seamless flow of data and decisions across tasks. The framework's robustness is demonstrated by applying it to three distinct use cases related to credit card fraud detection, credit card approval, and portfolio credit risk modeling, all executed via an agent-based system initialized with GPT-3.5 Turbo as the LLM. Each use case illustrates the system's capacity to manage complexities inherent in financial data processing and model development.

The experiments showcase the system's effectiveness in achieving respectable performance metrics: for instance, the credit card fraud detection task yielded an accuracy and F1-score surpassing popular solutions on platforms like Kaggle. The system demonstrated substantial competence in handling data imbalances and feature intricacies across different financial datasets.

Results and Implications

The endeavor to simulate human decision-making processes in financial modeling and risk management using LLM-based multi-agent systems is pivotal. Notably, the research underscores the system's capacity to maintain model reliability and compliance by reenacting industry-standard procedures and evaluating model robustness through rigorous stress tests involving shifted and adversarial inputs.

The paper's findings have significant implications for the financial sector, where compliance and risk mitigation are paramount. Implementing agentic systems could streamline workflows, enhance model accuracy, and ensure stringent adherence to regulatory demands. Moreover, such system architectures promise scalable solutions capable of simultaneously managing multiple financial products or portfolios, thus improving the efficiency of financial services operations.

Future Directions

The research identifies potential avenues for enhancing agentic system capabilities, such as developing self-improving agents that can evolve and adapt beyond their initial roles. Additionally, future work could explore creating generalized frameworks for dynamically assembling agent crews tailored to specific financial tasks. These improvements could result in more responsive and agile systems that can handle a broader spectrum of financial services scenarios.

Conclusion

The integration of agentic AI systems within financial services, as investigated in this paper, represents a meaningful advancement in leveraging LLMs for complex decision-making tasks. By systematically addressing both modeling and MRM workflows, the research provides a substantive groundwork for applying autonomous AI capabilities in regulated and dynamic environments such as finance. The nuanced handling of intricate tasks, compliance checks, and risk assessments contributes to the growing landscape of AI applications in this critical industry domain.