Papers
Topics
Authors
Recent
Search
2000 character limit reached

PPTC-R benchmark: Towards Evaluating the Robustness of Large Language Models for PowerPoint Task Completion

Published 6 Mar 2024 in cs.CL | (2403.03788v1)

Abstract: The growing dependence on LLMs for finishing user instructions necessitates a comprehensive understanding of their robustness to complex task completion in real-world situations. To address this critical need, we propose the PowerPoint Task Completion Robustness benchmark (PPTC-R) to measure LLMs' robustness to the user PPT task instruction and software version. Specifically, we construct adversarial user instructions by attacking user instructions at sentence, semantic, and multi-language levels. To assess the robustness of LLMs to software versions, we vary the number of provided APIs to simulate both the newest version and earlier version settings. Subsequently, we test 3 closed-source and 4 open-source LLMs using a benchmark that incorporates these robustness settings, aiming to evaluate how deviations impact LLMs' API calls for task completion. We find that GPT-4 exhibits the highest performance and strong robustness in our benchmark, particularly in the version update and the multilingual settings. However, we find that all LLMs lose their robustness when confronted with multiple challenges (e.g., multi-turn) simultaneously, leading to significant performance drops. We further analyze the robustness behavior and error reasons of LLMs in our benchmark, which provide valuable insights for researchers to understand the LLM's robustness in task completion and develop more robust LLMs and agents. We release the code and data at \url{https://github.com/ZekaiGalaxy/PPTCR}.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. Palm 2 technical report. arXiv preprint arXiv:2305.10403.
  2. Do as i can, not as i say: Grounding language in robotic affordances. In Conference on Robot Learning, pages 287–318. PMLR.
  3. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712.
  4. Chatgpt’s one-year anniversary: Are open-source large language models catching up? arXiv preprint arXiv:2311.16989.
  5. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023).
  6. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  7. Language models can be logical solvers. arXiv preprint arXiv:2311.06158.
  8. How does chatgpt perform on the united states medical licensing examination? the implications of large language models for medical education and knowledge assessment. JMIR Medical Education, 9(1):e45312.
  9. Pptc benchmark: Evaluating large language models for powerpoint task completion.
  10. Prompt perturbation in retrieval-augmented generation based large language models. arXiv preprint arXiv:2402.07179.
  11. Mistral 7b. arXiv preprint arXiv:2310.06825.
  12. Is chatgpt a good translator? a preliminary study. arXiv preprint arXiv:2301.08745.
  13. Realtime qa: What’s the answer right now? Advances in Neural Information Processing Systems, 36.
  14. Gpt-4 passes the bar exam. Available at SSRN 4389233.
  15. Language models can solve computer tasks. arXiv preprint arXiv:2303.17491.
  16. Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In Conference on Robot Learning, pages 80–93. PMLR.
  17. Competition-level code generation with alphacode. Science, 378(6624):1092–1097.
  18. Taskmatrix. ai: Completing tasks by connecting foundation models with millions of apis. arXiv preprint arXiv:2303.16434.
  19. Reinforcement learning on web interfaces using workflow-guided exploration. In International Conference on Learning Representations (ICLR).
  20. Evaluating the logical reasoning ability of chatgpt and gpt-4. arXiv preprint arXiv:2304.03439.
  21. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. arXiv preprint arXiv:2305.01210.
  22. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688.
  23. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  24. Robustness over time: Understanding adversarial examples’ effectiveness on longitudinal versions of large language models. arXiv preprint arXiv:2308.07847.
  25. Stress test evaluation for natural language inference. arXiv preprint arXiv:1806.00692.
  26. Adversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599.
  27. OpenAI. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  28. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8494–8502.
  29. Is chatgpt a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476.
  30. Tool learning with foundation models.
  31. Android in the wild: A large-scale dataset for android device control. arXiv preprint arXiv:2307.10088.
  32. Beyond accuracy: Behavioral testing of nlp models with checklist. arXiv preprint arXiv:2005.04118.
  33. Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments. In Conference on Robot Learning, pages 477–490. PMLR.
  34. Safety assessment of chinese large language models. arXiv preprint arXiv:2304.10436.
  35. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  36. Decodingtrust: A comprehensive assessment of trustworthiness in gpt models. arXiv preprint arXiv:2306.11698.
  37. Adversarial glue: A multi-task benchmark for robustness evaluation of language models. arXiv preprint arXiv:2111.02840.
  38. Are large language models really robust to word-level perturbations? arXiv preprint arXiv:2309.11166.
  39. On the robustness of chatgpt: An adversarial and out-of-distribution perspective. arXiv preprint arXiv:2302.12095.
  40. Is chatgpt a good sentiment analyzer? a preliminary study. arXiv preprint arXiv:2304.04339.
  41. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244.
  42. On the tool manipulation capability of open-source large language models. arXiv preprint arXiv:2305.16504.
  43. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305.
  44. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744–20757.
  45. Adversarial textual robustness on visual dialog. In Findings of the Association for Computational Linguistics: ACL 2023, pages 3422–3438.
  46. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364.
  47. Promptbench: Towards evaluating the robustness of large language models on adversarial prompts. arXiv preprint arXiv:2306.04528.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up