ClusterLLM: Large Language Models as a Guide for Text Clustering (2305.14871v2)

Abstract: We introduce ClusterLLM, a novel text clustering framework that leverages feedback from an instruction-tuned LLM, such as ChatGPT. Compared with traditional unsupervised methods that builds upon "small" embedders, ClusterLLM exhibits two intriguing advantages: (1) it enjoys the emergent capability of LLM even if its embeddings are inaccessible; and (2) it understands the user's preference on clustering through textual instruction and/or a few annotated data. First, we prompt ChatGPT for insights on clustering perspective by constructing hard triplet questions <does A better correspond to B than C>, where A, B and C are similar data points that belong to different clusters according to small embedder. We empirically show that this strategy is both effective for fine-tuning small embedder and cost-efficient to query ChatGPT. Second, we prompt ChatGPT for helps on clustering granularity by carefully designed pairwise questions <do A and B belong to the same category>, and tune the granularity from cluster hierarchies that is the most consistent with the ChatGPT answers. Extensive experiments on 14 datasets show that ClusterLLM consistently improves clustering quality, at an average cost of ~$0.6 per dataset. The code will be available at https://github.com/zhang-yu-wei/ClusterLLM.

• The paper demonstrates that leveraging LLMs as guides significantly improves clustering accuracy and normalized mutual information over traditional methods.
• The methodology employs a two-phase process with entropy-based triplet sampling and pairwise queries to fine-tune clustering based on user preferences.
• The approach offers a cost-effective solution, achieving enhanced performance across 14 datasets at an average cost of about $0.6 per dataset.

Overview of "ClusterLLM": LLMs as a Guide for Text Clustering

The paper "ClusterLLM" introduces a novel framework for enhancing text clustering by utilizing the capabilities of LLMs, such as ChatGPT. In contrast to conventional methods reliant on static, smaller embedders, ClusterLLM leverages the nuanced understanding of LLMs. This framework capitalizes on two principal advantages: the emergent capabilities of LLMs, even when embeddings are inaccessible, and the accommodation of user preferences in clustering via textual instructions or minimal annotations.

ClusterLLM employs a two-phase methodology designed for refining clustering perspectives and granularity effectively. The first stage involves using LLMs to perform triplet tasks aimed at understanding user-preferred clustering perspectives. By prompting LLMs with hard triplet questions like <does A better correspond to B than C>, where A is the anchor, ClusterLLM tunes smaller embedders to align with user-specified criteria such as topic, intent, or emotion. The process is enhanced through an entropy-based sampling strategy, which identifies the most informative triplets by examining the entropy of cluster assignments, thereby optimizing the embedder's fine-tuning.

The second stage addresses the determination of clustering granularity. It involves constructing hierarchical cluster structures and then leveraging pairwise questions posed to LLMs—<do A and B belong to the same category>—to ascertain the optimal level of granularity that aligns with LLM predictions and user expectations. This approach ensures that clustering granularity is as consistent as possible with LLM interpretations, thus bridging the gap between high-level language capabilities and clustering tasks.

The authors conducted extensive evaluations across 14 datasets that encompass tasks like intent discovery, topic mining, and emotion detection. ClusterLLM demonstrated consistent improvements in clustering quality with an average expenditure of approximately $0.6 per dataset, indicating a cost-effective solution. Furthermore, an analysis of sampling strategies indicated that the entropy-based triplet selection was more effective than random sampling, highlighting the robustness of the proposed sampling method.

Key Results and Implications

1. Clustering Accuracy and Performance: Across all tested datasets, ClusterLLM notably enhanced clustering accuracy and normalized mutual information (NMI), surpassing both traditional deep clustering and self-supervised baselines.
2. Cost-Effectiveness: The methodology is cost-efficient, with costs calculated using the GPT-3.5-turbo model, suggesting scalability for widespread application without significant economic constraints.
3. Iterative Improvement: The framework allows for iterative enhancement, indicating potential for even further performance gains through repeated applications of the method.
4. Generalization to Different Granularities: The pairwise task-driven granularity estimation demonstrates adaptability across different domains and clustering levels, supporting diverse applications.

Theoretical and Practical Implications

In theoretical terms, the integration of LLMs as a guiding mechanism in text clustering expands the functional versatility of these models beyond mere text generation and understanding tasks. Practically, this framework paves the way for user-interactive clustering solutions in scenarios where capturing user preferences is crucial, such as market research or personalized content curation. Future developments could enhance scalability and efficiency, for instance, by further optimizing model-free clustering methodologies or integrating stronger active learning strategies to further minimize reliance on pre-trained embedders.

Future Directions in AI

The promising results from using LLM-guided clustering indicate that similar strategies could be beneficial in other areas where user preferences need integration with large-scale automated processes. Future research might focus on exploring LLMs for more dynamic applications, including real-time data interactions or more complex multi-modal clustering scenarios, potentially taking advantage of advancements in LLM's contextual and cognitive abilities.

In conclusion, "ClusterLLM" represents a significant step in advancing text clustering methodologies by harnessing the emergent capabilities of LLMs, offering a sophisticated, cost-effective mechanism that significantly improves clustering quality with minimal demand for direct data access—a promising development for both academia and industry applications.