Verbalized Machine Learning: Revisiting Machine Learning with Language Models (2406.04344v2)

Abstract: Motivated by the progress made by LLMs, we introduce the framework of verbalized machine learning (VML). In contrast to conventional ML models that are typically optimized over a continuous parameter space, VML constrains the parameter space to be human-interpretable natural language. Such a constraint leads to a new perspective of function approximation, where an LLM with a text prompt can be viewed as a function parameterized by the text prompt. Guided by this perspective, we revisit classical ML problems, such as regression and classification, and find that these problems can be solved by an LLM-parameterized learner and optimizer. The major advantages of VML include (1) easy encoding of inductive bias: prior knowledge about the problem and hypothesis class can be encoded in natural language and fed into the LLM-parameterized learner; (2) automatic model class selection: the optimizer can automatically select a model class based on data and verbalized prior knowledge, and it can update the model class during training; and (3) interpretable learner updates: the LLM-parameterized optimizer can provide explanations for why an update is performed. We empirically verify the effectiveness of VML, and hope that VML can serve as a stepping stone to stronger interpretability.

• The paper introduces a verbalized machine learning (VML) framework that employs natural language prompts to define model parameters for enhanced interpretability.
• It demonstrates how integrating large language models with classical ML tasks enables dynamic model selection and effective encoding of inductive biases.
• Experimental results across regression and classification tasks confirm VML’s ability to refine model performance and improve transparency in critical applications.

Insights into Verbalized Machine Learning: Revisiting Machine Learning with LLMs

The paper entitled "Verbalized Machine Learning: Revisiting Machine Learning with LLMs" delineates the framework of verbalized machine learning (VML), a novel concept that integrates LLMs into classical machine learning tasks through human-interpretable natural language prompts. The authors, Tim Z. Xiao, Robert Bamler, Bernhard Schölkopf, and Weiyang Liu, provide a compelling argument for leveraging natural language as the medium to specify and optimize model parameters, thereby enhancing interpretability and integrating prior knowledge into machine learning models.

Abstract and Motivation

The motivation for VML stems from the substantial progress made by LLMs in solving complex problems. Unlike conventional models optimized over a continuous parameter space, VML confines the parameter space to human-understandable language. This paradigm shift positions an LLM with a text prompt as a function approximator, controlled by language-based model parameters.

Framework and Methodology

VML redefines classical problems such as regression and classification, accommodating an LLM-parameterized learner and optimizer. This approach proffers several advantages:

1. Encoding Inductive Bias: Natural language easily encapsulates prior knowledge, allowing the input of inductive biases directly into the LLM.
2. Automatic Model Selection: The optimizer LLM can dynamically select and modify the model class during training.
3. Interpretable Updates: Each adjustment made by the optimizer is explainable in human language, enhancing transparency and trustworthiness.

The paper outlines an iterative training process wherein the optimizer LLM refines the text prompt (representing the model parameters) based on the training data. Optimization ensues by sampling from the distribution defined by the LLM's temperature setting, enabling a stochastic perspective akin to Bayesian inference.

Experimental Analysis and Results

The authors validate VML through several classical machine learning tasks, including linear, polynomial, and sinusoidal regressions, as well as classifications of two blobs and two circles.

• Linear Regression: VML accurately captures the linear relationship in the data, with the optimizer incrementally refining the scaling factor and bias terms.
• Polynomial Regression: The model transitions from an erroneous linear assumption to accurately identifying and fitting a quadratic relationship.
• Sinusoidal Regression: With prior knowledge about periodicity, VML significantly ameliorates the capture and extrapolation of the sine wave pattern compared to a neural network's performance.
• Classification Tasks: The models benefit from both inductive biases and dynamic rule generation, leading to accurate classifications of data points in 2-D space.

Furthermore, a distinct advantage of VML is discerned in medical image classification using X-ray images. When provided with prior domain-specific knowledge, such as features indicative of pneumonia, VML yields a simpler, more intuitive model with fewer false positives and negatives compared to a model without prior.

Theoretical and Practical Implications

VML introduces a promising paradigm for enhanced interpretability in machine learning models. By leveraging LLMs' capabilities, VML aligns closely with the objectives of explainable AI (XAI), promoting models whose decision-making processes are transparent and accessible to human scrutiny. This approach is particularly beneficial for domains such as healthcare, where interpretability is paramount.

Moreover, the framework hints at future applications where programs and data converge, resonating with the von Neumann architecture. This unification could see LLMs becoming versatile problem solvers, orchestrating both data and model instructions within a single context.

Conclusion and Future Directions

While the VML framework showcases immense potential, certain limitations require attention. Training variance remains substantial, partly attributable to LLM inference stochasticity and prompt design. Additionally, numerical errors in LLMs during function evaluations pose challenges, necessitating improvements in numerical handling capabilities. Finally, context window constraints limit high-dimensional and large-batch processing, an area warranting exploration for scalability.

Future work should aim at refining optimization strategies, mitigating numerical errors, and expanding the applicability of VML to high-dimensional data. The trajectory outlined by this paper suggests a transformative avenue for machine learning, where AI models are not only proficient but also comprehensible and trustworthy.

The paper, through its robust empirical studies and theoretical insights, offers a pivotal stepping stone towards integrating natural language within the core processes of machine learning, heralding a more interpretable and human-aligned AI future.