Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
194 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Executing Natural Language-Described Algorithms with Large Language Models: An Investigation (2403.00795v2)

Published 23 Feb 2024 in cs.CL and cs.AI

Abstract: Executing computer programs described in natural language has long been a pursuit of computer science. With the advent of enhanced natural language understanding capabilities exhibited by LLMs, the path toward this goal has been illuminated. In this paper, we seek to examine the capacity of present-day LLMs to comprehend and execute algorithms outlined in natural language. We established an algorithm test set sourced from Introduction to Algorithm, a well-known textbook that contains many representative widely-used algorithms. To systematically assess LLMs' code execution abilities, we selected 30 algorithms, generated 300 random-sampled instances in total, and evaluated whether popular LLMs can understand and execute these algorithms. Our findings reveal that LLMs, notably GPT-4, can effectively execute programs described in natural language, as long as no heavy numeric computation is involved. We believe our findings contribute to evaluating LLMs' code execution abilities and would encourage further investigation and application for the computation power of LLMs.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (54)
  1. Alfred V. Aho and John E. Hopcroft. 1974. The Design and Analysis of Computer Algorithms, 1st edition. Addison-Wesley Longman Publishing Co., Inc., USA.
  2. Palm 2 technical report.
  3. Code generation tools (almost) for free? A study of few-shot, pre-trained language models on code. CoRR, abs/2206.01335.
  4. Richard Bellman. 1958. On a routing problem. Quarterly of applied mathematics, 16(1):87–90.
  5. Jon Bentley. 1984. Programming pearls: Algorithm design techniques. Commun. ACM, 27(9):865–873.
  6. On the computational power of transformers and its implications in sequence modeling. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 455–475, Online. Association for Computational Linguistics.
  7. Corrado Böhm and Giuseppe Jacopini. 1966. Flow diagrams, turing machines and languages with only two formation rules. Communications of the ACM, 9(5):366–371.
  8. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.
  9. Sparks of artificial general intelligence: Early experiments with GPT-4. CoRR, abs/2303.12712.
  10. Evaluating large language models trained on code. CoRR, abs/2107.03374.
  11. Livio Colussi. 1994. Fastest pattern matching in strings. Journal of Algorithms, 16(2):163–189.
  12. Introduction to algorithms.
  13. E. W. Dijkstra. 2022. A Note on Two Problems in Connexion with Graphs, 1 edition, page 287–290. Association for Computing Machinery, New York, NY, USA.
  14. Robert W. Floyd. 1962. Algorithm 97: Shortest path. Commun. ACM, 5(6):345.
  15. Fănică Gavril. 1972. Algorithms for minimum coloring, maximum clique, minimum covering by cliques, and maximum independent set of a chordal graph. SIAM Journal on Computing, 1(2):180–187.
  16. R.L. Graham. 1972. An efficient algorith for determining the convex hull of a finite planar set. Information Processing Letters, 1(4):132–133.
  17. Neural turing machines.
  18. Pre-trained models: Past, present and future. AI Open, 2:225–250.
  19. C. A. R. Hoare. 1961a. Algorithm 64: Quicksort. Commun. ACM, 4(7):321.
  20. C. A. R. Hoare. 1961b. Algorithm 65: Find. Commun. ACM, 4(7):321–322.
  21. An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Systems.
  22. R.A. Jarvis. 1973. On the identification of the convex hull of a finite set of points in the plane. Information Processing Letters, 2(1):18–21.
  23. GPT is becoming a turing machine: Here are some ways to program it. CoRR, abs/2303.14310.
  24. Repair is nearly generation: Multilingual program repair with llms. Proceedings of the AAAI Conference on Artificial Intelligence, 37(4):5131–5140.
  25. Scaling laws for neural language models. CoRR, abs/2001.08361.
  26. Donald E Knuth. 1973. Fundamental algorithms.
  27. Joseph B. Kruskal. 1956. On the shortest spanning subtree of a graph and the traveling salesman problem. Proceedings of the American Mathematical Society, 7(1):48–50.
  28. Eugene L Lawler. 1985. The traveling salesman problem: a guided tour of combinatorial optimization. Wiley-Interscience Series in Discrete Mathematics.
  29. Explaining competitive-level programming solutions using llms. CoRR, abs/2307.05337.
  30. Competition-level code generation with alphacode. Science, 378(6624):1092–1097.
  31. Code execution with pre-trained language models. arXiv preprint arXiv:2305.05383.
  32. Edward F Moore. 1959. The shortest path through a maze. In Proc. Int. Symp. Switching Theory, 1959, pages 285–292.
  33. Using an llm to help with code understanding.
  34. OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08774.
  35. Training language models to follow instructions with human feedback. In NeurIPS.
  36. R. C. Prim. 1957. Shortest connection networks and some generalizations. The Bell System Technical Journal, 36(6):1389–1401.
  37. On the turing completeness of modern neural network architectures. In International Conference on Learning Representations.
  38. Jean E. Sammet. 1966. The use of english as a programming language. Commun. ACM, 9(3):228–230.
  39. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations.
  40. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100.
  41. Dale Schuurmans. 2023. Memory augmented large language models are computationally universal.
  42. An empirical evaluation of using large language models for automated unit test generation. IEEE Transactions on Software Engineering, 50(1):85–105.
  43. H.T. Siegelmann and E.D. Sontag. 1995. On the computational power of neural nets. Journal of Computer and System Sciences, 50(1):132–150.
  44. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. CoRR, abs/2206.04615.
  45. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971.
  46. AM Turing. 1937. On computable numbers, with an application to the entscheidungsproblem. Proceedings of the London Mathematical Society, 2(1):230–230.
  47. The CLRS algorithmic reasoning benchmark. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 22084–22102. PMLR.
  48. Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5085–5109, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  49. Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformers. In Advances in Neural Information Processing Systems, volume 35, pages 12071–12083. Curran Associates, Inc.
  50. Finetuned language models are zero-shot learners. In International Conference on Learning Representations.
  51. Emergent abilities of large language models. Transactions on Machine Learning Research. Survey Certification.
  52. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
  53. On the practical computational power of finite precision RNNs for language recognition. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 740–745, Melbourne, Australia. Association for Computational Linguistics.
  54. John William Joseph Williams. 1964. Algorithm 232: heapsort. Communications of the ACM, 7(6):347–348.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets