Evaluating GPT-4's Vision Capabilities on Brazilian University Admission Exams (2311.14169v1)
Abstract: Recent advancements in LLMs have showcased human-comparable performance in academic entrance exams. However, existing studies often overlook questions that require the integration of visual comprehension, thus compromising the full spectrum and complexity inherent in real-world scenarios. To address this gap, we present a comprehensive framework to evaluate LLMs on entrance exams, which incorporates both textual and visual elements. We evaluate the two most recent editions of Exame Nacional do Ensino M\'edio (ENEM), the main standardized entrance examination adopted by Brazilian universities. Our study not only reaffirms the capabilities of GPT-4 as the state of the art for handling complex multidisciplinary questions, but also pioneers in offering a realistic assessment of multimodal LLMs on Portuguese examinations. One of the highlights is that text captions transcribing visual content outperform the direct use of images, suggesting that the vision model has room for improvement. Yet, despite improvements afforded by images or captions, mathematical questions remain a challenge for these state-of-the-art models. The code and data used on experiments are available at https://github.com/piresramon/gpt-4-enem.
- BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
- Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
- Scaling language models: Methods, analysis & insights from training gopher, 2022.
- Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
- Training compute-optimal large language models, 2022.
- Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
- OpenAI. Gpt-4 technical report, 2023.
- Evaluating the performance of large language models on gaokao benchmark, 2023.
- Evaluating gpt-3.5 and gpt-4 models on brazilian university admission exams, 2023.
- Bluex: A benchmark based on brazilian leading universities entrance exams. In Murilo C. Naldi and Reinaldo A. C. Bianchi, editors, Intelligent Systems, pages 337–347, Cham, 2023. Springer Nature Switzerland.
- Machine learning in education-a survey of current research trends. Annals of DAAAM & Proceedings, 29, 2018.
- A review of using machine learning approaches for precision education. Educational Technology & Society, 24(1):250–266, 2021.
- Gpt takes the bar exam. Available at SSRN 4314839, 2022.
- Gpt as Knowledge Worker: A Zero-Shot Evaluation of (AI)CPA Capabilities. SSRN Electronic Journal, 2023.
- Large language models encode clinical knowledge. arXiv preprint arXiv:2212.13138, 2022.
- Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213, 2022.
- Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022.
- Sabiá: Portuguese large language models. In Murilo C. Naldi and Reinaldo A. C. Bianchi, editors, Intelligent Systems, pages 226–240, Cham, 2023. Springer Nature Switzerland.
- Evaluating chatgpt efficacy in navigating the spanish medical residency entrance examination (mir): A new horizon for ai in clinical medicine. Preprints, September 2023.
- Advances in automatically solving the enem. In 2018 7th Brazilian Conference on Intelligent Systems (BRACIS), pages 43–48, Los Alamitos, CA, USA, oct 2018. IEEE Computer Society.
- Efficient Estimation of Word Representations in Vector Space.
- George A. Miller. Wordnet: A lexical database for english. Commun. ACM, 38(11):39–41, nov 1995.
- Adobe. Adobe pdf services python sdk. https://github.com/adobe/pdfservices-python-sdk, 2023.
- Chain of thought prompting elicits reasoning in large language models. CoRR, abs/2201.11903, 2022.
- Towards language models that can see: Computer vision through the lens of natural language, 2023.
- Ramon Pires (11 papers)
- Thales Sales Almeida (10 papers)
- Hugo Abonizio (12 papers)
- Rodrigo Nogueira (70 papers)