LM4OPT: Unveiling the Potential of Large Language Models in Formulating Mathematical Optimization Problems (2403.01342v1)
Abstract: In the rapidly evolving field of natural language processing, the translation of linguistic descriptions into mathematical formulation of optimization problems presents a formidable challenge, demanding intricate understanding and processing capabilities from LLMs. This study compares prominent LLMs, including GPT-3.5, GPT-4, and Llama-2-7b, in zero-shot and one-shot settings for this task. Our findings show GPT-4's superior performance, particularly in the one-shot scenario. A central part of this research is the introduction of `LM4OPT,' a progressive fine-tuning framework for Llama-2-7b that utilizes noisy embeddings and specialized datasets. However, this research highlights a notable gap in the contextual understanding capabilities of smaller models such as Llama-2-7b compared to larger counterparts, especially in processing lengthy and complex input contexts. Our empirical investigation, utilizing the NL4Opt dataset, unveils that GPT-4 surpasses the baseline performance established by previous research, achieving an F1-score of 0.63, solely based on the problem description in natural language, and without relying on any additional named entity information. GPT-3.5 follows closely, both outperforming the fine-tuned Llama-2-7b. These findings not only benchmark the current capabilities of LLMs in a novel application area but also lay the groundwork for future improvements in mathematical formulation of optimization problems from natural language input.
- OptiMUS: Optimization Modeling Using mip Solvers and large language models. ArXiv, abs/2310.06116.
- Language Models are Few-Shot Learners. ArXiv, abs/2005.14165.
- Training Deep Nets with Sublinear Memory Cost. ArXiv, abs/1604.06174.
- Training Verifiers to Solve Math Word Problems. ArXiv, abs/2110.14168.
- Ner4Opt: Named Entity Recognition for Optimization Modelling from Natural Language. In Integration of AI and OR Techniques in Constraint Programming.
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In North American Chapter of the Association for Computational Linguistics.
- LoRA: Low-Rank Adaptation of Large Language Models. ArXiv, abs/2106.09685.
- NEFTune: Noisy Embeddings Improve Instruction Finetuning. ArXiv, abs/2310.05914.
- Karmarkar, N. 1984. A new polynomial-time algorithm for linear programming. Combinatorica, 4: 373–395.
- Green algorithms: quantifying the carbon footprint of computation. Advanced science, 8(12): 2100707.
- Domain Adaptation with Pre-trained Transformers for Query-Focused Abstractive Text Summarization. Computational Linguistics, 48: 279–320.
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In Annual Meeting of the Association for Computational Linguistics.
- Large Language Models for Supply Chain Optimization. ArXiv, abs/2307.03875.
- Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning. ArXiv, abs/2205.05638.
- Decoupled Weight Decay Regularization. In International Conference on Learning Representations.
- Nash, J. C. 2000. The (Dantzig) simplex method for linear programming. Comput. Sci. Eng., 2: 29–31.
- OpenAI. 2023. GPT-4 Technical Report. ArXiv, abs/2303.08774.
- NL4Opt Competition: Formulating Optimization Problems Based on Their Natural Language Descriptions. ArXiv, abs/2303.08233.
- Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. In Annual Meeting of the Association for Computational Linguistics.
- Llama 2: Open Foundation and Fine-Tuned Chat Models. ArXiv, abs/2307.09288.
- Holy Grail 2.0: From Natural Language to Constraint Models. ArXiv, abs/2308.01589.
- Attention is All you Need. In Neural Information Processing Systems.
- Large Language Models as Optimizers. ArXiv, abs/2309.03409.
- Tasnim Ahmed (18 papers)
- Salimur Choudhury (11 papers)