Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning (2402.17453v5)

Published 27 Feb 2024 in cs.LG

Abstract: In this work, we investigate the potential of LLMs based agents to automate data science tasks, with the goal of comprehending task requirements, then building and training the best-fit machine learning models. Despite their widespread success, existing LLM agents are hindered by generating unreasonable experiment plans within this scenario. To this end, we present DS-Agent, a novel automatic framework that harnesses LLM agent and case-based reasoning (CBR). In the development stage, DS-Agent follows the CBR framework to structure an automatic iteration pipeline, which can flexibly capitalize on the expert knowledge from Kaggle, and facilitate consistent performance improvement through the feedback mechanism. Moreover, DS-Agent implements a low-resource deployment stage with a simplified CBR paradigm to adapt past successful solutions from the development stage for direct code generation, significantly reducing the demand on foundational capabilities of LLMs. Empirically, DS-Agent with GPT-4 achieves 100\% success rate in the development stage, while attaining 36\% improvement on average one pass rate across alternative LLMs in the deployment stage. In both stages, DS-Agent achieves the best rank in performance, costing \$1.60 and \$0.13 per run with GPT-4, respectively. Our data and code are open-sourced at https://github.com/guosyjlu/DS-Agent.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. Case-based reasoning: Foundational issues, methodological variations, and system approaches. AI communications, 7(1):39–59, 1994.
  2. Autonomous chemical research with large language models. Nature, 624(7992):570–578, 2023.
  3. Do as i can, not as i say: Grounding language in robotic affordances. In Conference on Robot Learning, pp.  287–318. PMLR, 2023.
  4. Grounding large language models in interactive environments with online reinforcement learning. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp.  3676–3713. PMLR, 23–29 Jul 2023.
  5. Chase, H. LangChain, October 2022. URL https://github.com/langchain-ai/langchain.
  6. Fireact: Toward language agent fine-tuning. arXiv preprint arXiv:2310.05915, 2023.
  7. Pangu-agent: A fine-tunable generalist agent with structured reasoning. arXiv preprint arXiv:2312.14878, 2023.
  8. Automating data science. Communications of the ACM, 65(3):76–87, 2022.
  9. K2: A foundation language model for geoscience knowledge understanding and utilization. 2023. URL https://api.semanticscholar.org/CorpusID:259108887.
  10. Autogluon-tabular: Robust and accurate automl for structured data. arXiv preprint arXiv:2003.06505, 2020.
  11. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2023.
  12. A case-based reasoning framework for adaptive prompting in cross-domain text-to-sql. arXiv preprint arXiv:2304.13301, 2023.
  13. Large language models for automated data science: Introducing caafe for context-aware automated feature engineering. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  14. MetaGPT: Meta programming for multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=VtmBAGCN7o.
  15. Benchmarking large language models as ai research agents. arXiv preprint arXiv:2310.03302, 2023.
  16. Automated machine learning: methods, systems, challenges. Springer Nature, 2019.
  17. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024.
  18. Automl to date and beyond: Challenges and opportunities. ACM Computing Surveys (CSUR), 54(8):1–36, 2021.
  19. Language models can solve computer tasks. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  20. Kolodner, J. L. An introduction to case-based reasoning. Artificial intelligence review, 6(1):3–34, 1992.
  21. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp.  611–626, 2023.
  22. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pp.  18319–18345. PMLR, 2023.
  23. H2o automl: Scalable automatic machine learning. In Proceedings of the AutoML Workshop at ICML, volume 2020. ICML, 2020.
  24. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474, 2020.
  25. Geogalactica: A scientific large language model in geoscience. ArXiv, abs/2401.00434, 2023. URL https://api.semanticscholar.org/CorpusID:266693296.
  26. Ml-bench: Large language models leverage open-source libraries for machine learning tasks. arXiv preprint arXiv:2311.09835, 2023.
  27. Chatgpt as your personal data scientist. arXiv e-prints, pp.  arXiv–2305, 2023.
  28. OpenAI. Introducing chatgpt. 2022. URL https://openai.com/blog/chatgpt.
  29. OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  30. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825–2830, 2011.
  31. Mathematical discoveries from program search with large language models. Nature, pp.  1–3, 2023.
  32. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp.  2655–2671, 2022.
  33. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. In Advances in Neural Information Processing Systems, 2023.
  34. Replug: Retrieval-augmented black-box language models. arXiv preprint arXiv:2301.12652, 2023.
  35. Reflexion: Language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  36. Significant Gravitas. AutoGPT, 2023. URL https://github.com/Significant-Gravitas/AutoGPT.
  37. Case-based reasoning with language models for classification of logical fallacies. arXiv preprint arXiv:2301.11879, 2023.
  38. Is ChatGPT good at search? investigating large language models as re-ranking agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.  14918–14937, 2023.
  39. Learning to retrieve in-context examples for large language models. arXiv preprint arXiv:2307.07164, 2023a.
  40. Learning to retrieve in-context examples for large language models. arXiv preprint arXiv:2307.07164, 2023b.
  41. Describe, explain, plan and select: Interactive planning with LLMs enables open-world multi-task agents. In Thirty-seventh Conference on Neural Information Processing Systems, 2023c. URL https://openreview.net/forum?id=KtvPdGb31Z.
  42. Case-based reasoning: A review. The knowledge engineering review, 9(4):327–354, 1994.
  43. Autogen: Enabling next-gen llm applications via multi-agent conversation framework. 2023.
  44. End-to-end case-based reasoning for commonsense knowledge base completion. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp.  3491–3504, 2023.
  45. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2022.
  46. Agenttuning: Enabling generalized agent abilities for llms. arXiv preprint arXiv:2310.12823, 2023.
  47. Mlcopilot: Unleashing the power of large language models in solving machine learning tasks. arXiv preprint arXiv:2304.14979, 2023a.
  48. Retrieve anything to augment large language models. arXiv preprint arXiv:2310.07554, 2023b.
  49. Automl-gpt: Automatic machine learning with gpt. arXiv preprint arXiv:2305.02499, 2023c.
  50. Expel: Llm agents are experiential learners. Thirty-Eighth AAAI Conference on Artificial Intelligence, 2024.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Cheng Deng (67 papers)
  2. Ying Wen (75 papers)
  3. Hechang Chen (26 papers)
  4. Yi Chang (150 papers)
  5. Jun Wang (990 papers)
  6. SiYuan Guo (20 papers)
Citations (15)
View on Semantic Scholar