MASCA: LLM based-Multi Agents System for Credit Assessment

Abstract: Recent advancements in financial problem-solving have leveraged LLMs and agent-based systems, with a primary focus on trading and financial modeling. However, credit assessment remains an underexplored challenge, traditionally dependent on rule-based methods and statistical models. In this paper, we introduce MASCA, an LLM-driven multi-agent system designed to enhance credit evaluation by mirroring real-world decision-making processes. The framework employs a layered architecture where specialized LLM-based agents collaboratively tackle sub-tasks. Additionally, we integrate contrastive learning for risk and reward assessment to optimize decision-making. We further present a signaling game theory perspective on hierarchical multi-agent systems, offering theoretical insights into their structure and interactions. Our paper also includes a detailed bias analysis in credit assessment, addressing fairness concerns. Experimental results demonstrate that MASCA outperforms baseline approaches, highlighting the effectiveness of hierarchical LLM-based multi-agent systems in financial applications, particularly in credit scoring.

• The paper introduces MASCA, a multi-agent framework driven by LLMs that enhances credit assessment accuracy and fairness through contrastive learning.
• The methodology employs specialized agents in a hierarchical design for data ingestion, risk evaluation, and reward estimation, outperforming traditional models.
• Experimental results on the German Credit Dataset demonstrate superior performance with 83.33% recall and 73.33% F1 score compared to zero-shot and single-agent approaches.

MASCA: LLM-Based Multi-Agent System for Credit Assessment

The paper introduces MASCA, a novel multi-agent framework driven by LLMs, aimed at enhancing credit assessment processes. It integrates specialized agents to break down complex credit evaluation tasks and employs contrastive learning to evaluate risk and reward, all within a signaling game theoretic framework for effective decision-making.

Introduction

MASCA is proposed to address the limitations of traditional credit assessment methods, which often rely heavily on historical data and lack transparency. The paper identifies significant challenges such as bias and inflexibility in existing models and posits that LLMs, with their ability to process diverse data sources, can significantly improve the accuracy and fairness of credit decisions. The multi-agent system structure mirrors real-world credit assessment teams, enhancing the adaptability and efficacy of credit evaluations.

Figure Description

Figure 1: MASCA: The multi-agent framework for credit assessment.

Methodology

MASCA's hierarchical design involves a series of specialized agents, each performing distinct roles:

1. Data Ingestion and Contextualization Layer

• Data Analyst: Aggregates and standardizes raw applicant data.
• Contextualizer: Constructs detailed user personas, integrating behavioral insights.
• Feature Engineer: Computes financial ratios and enhances data with derived features.

2. Multidimensional Assessment Layer

This layer consists of distinct teams focusing on risk and reward assessments:

• Risk Modeler: Evaluates credit history for potential risks.
• Income & Stability Analyst: Analyzes income consistency and financial stability.
• Debt Analyst: Assesses existing debt and loan specifics.
• Reward Modeler: Estimates potential benefits from loan approval.

3. Strategic Optimization Layer

Utilizes risk-reward ratios and scenario simulations to balance potential risks against expected rewards, supporting strategic decision-making.

Experimental Setup

Experiments were conducted using the German Credit Dataset to evaluate MASCA's performance across various configurations. The results demonstrated significant improvements over traditional zero-shot and single-agent baselines, particularly in handling complex reasoning tasks.

Performance Results

Experimental results highlight MASCA's superior accuracy and reliability compared to baseline models. The multi-agent framework achieved a recall of 83.33% and an F1 score of 73.33%, outperforming single-agent methods and demonstrating the advantages of task-specific agent deployment.

Bias Analysis

(Figure 2 & Figure 3)

Figure 2: Gender Bias Analysis.

Figure 3: Confusion Matrix for experiments in bias section.

Credit assessments revealed gender- and race-specific biases, underscoring the importance of fairness-focused evaluation techniques. MASCA showed variations in approval rates across demographic groups, prompting further investigation into mitigating these biases.

Signaling Game Theory Integration

Signaling game theory is applied to capture strategic interactions among agents, enhancing decision-making by optimizing information exchange and belief updating processes. This approach enables agents to adjust their behaviors dynamically, improving overall decision resilience and accuracy.

Conclusion

MASCA represents a significant step toward more adaptive, transparent, and fair credit assessment systems. By incorporating LLM-driven multi-agent architectures, it addresses key challenges in traditional credit scoring methods. Future research should explore broader datasets and model architectures to further validate these findings.

Limitations and Future Work

While the initial results are promising, MASCA's current evaluation is limited to certain datasets and LLM configurations. Further exploration with diverse datasets, as well as a deeper focus on bias mitigation and interpretability, will be essential for refining the framework and enhancing its applicability in real-world settings.