Logistic Regression makes small LLMs strong and explainable "tens-of-shot" classifiers

Abstract: For simple classification tasks, we show that users can benefit from the advantages of using small, local, generative LLMs instead of large commercial models without a trade-off in performance or introducing extra labelling costs. These advantages, including those around privacy, availability, cost, and explainability, are important both in commercial applications and in the broader democratisation of AI. Through experiments on 17 sentence classification tasks (2-4 classes), we show that penalised logistic regression on the embeddings from a small LLM equals (and usually betters) the performance of a large LLM in the "tens-of-shot" regime. This requires no more labelled instances than are needed to validate the performance of the large LLM. Finally, we extract stable and sensible explanations for classification decisions.

• The paper demonstrates that applying penalized logistic regression on small LLM embeddings achieves performance parity with larger models in tens-of-shot classification tasks.
• The methodology offers enhanced interpretability, yielding clear decision boundaries and stable, explainable weights for classification decisions.
• Experiments across 17 sentence classification tasks validate that small, cost-effective LLMs can serve as robust and privacy-conscious classifiers.

The query "Logistic Regression makes small LLMs strong and explainable 'tens-of-shot' classifiers" refers to a specific approach discussed in a paper (2408.03414). This paper investigates the effectiveness of using small, local generative LLMs for simple classification tasks, particularly in the "tens-of-shot" learning regime, where the model is provided with tens of examples to learn from.

Key Findings and Contributions:

1. Performance Parity with Large LLMs: The research demonstrates that penalized logistic regression, when applied to the embeddings generated by small LLMs, can match or even surpass the performance of larger LLMs for classification tasks. This finding is significant because it suggests that smaller models, which are often more cost-effective and easier to deploy, can be viable alternatives to their larger counterparts.
2. Advantages Beyond Performance: Utilizing small LLMs provides several benefits, including enhanced privacy, reduced costs, improved availability, and better explainability. These are critical factors for many applications, particularly where data sensitivity and compliance with privacy regulations are paramount.
3. Explainability: One of the standout features of this approach is the ability to extract stable and sensible explanations for classification decisions. This is achieved through the logistic regression framework, which inherently offers interpretable weights and decision boundaries, thereby making the classification process more transparent.
4. Practical Evaluation: The claims are backed by experiments on 17 different sentence classification tasks, each involving 2-4 classes. This extensive evaluation reinforces the robustness and generalizability of the proposed method across various domains.

This approach aligns with broader trends in AI and machine learning that emphasize the union of model performance and interpretability. The use of logistic regression on embeddings from small LLMs thus emerges as a powerful technique for practitioners aiming to deploy efficient, explainable classifiers with modest dataset sizes.