Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

IllusionX: An LLM-powered mixed reality personal companion (2402.07924v1)

Published 4 Feb 2024 in cs.HC and cs.MM

Abstract: Mixed Reality (MR) and AI are increasingly becoming integral parts of our daily lives. Their applications range in fields from healthcare to education to entertainment. MR has opened a new frontier for such fields as well as new methods of enhancing user engagement. In this paper, We propose a new system one that combines the power of LLMs and mixed reality (MR) to provide a personalized companion for educational purposes. We present an overview of its structure and components as well tests to measure its performance. We found that our system is better in generating coherent information, however it's rather limited by the documents provided to it. This interdisciplinary approach aims to provide a better user experience and enhance user engagement. The user can interact with the system through a custom-design smart watch, smart glasses and a mobile app.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (54)
  1. Affective computing. In Multimodal Affective Computing: Technologies and Applications in Learning Environments, pages 3–20. Springer, 2023.
  2. New speech emotion recognition analysis to increase accuracy using svm over decision tree. Journal of Survey in Fisheries Sciences, 10(1S):1943–1951, 2023.
  3. Facial emotion recognition using convolutional neural networks. Materials Today: Proceedings, 80:3560–3564, 2023.
  4. Cnn and lstm based ensemble learning for human emotion recognition using eeg recordings. Multimedia Tools and Applications, 82(4):4883–4896, 2023.
  5. Mohanad Halaweh. Chatgpt in education: Strategies for responsible implementation. 2023.
  6. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  7. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022.
  8. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  9. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020.
  10. How useful are educational questions generated by large language models? In International Conference on Artificial Intelligence in Education, pages 536–542. Springer, 2023.
  11. On the application of large language models for language teaching and assessment technology. arXiv preprint arXiv:2307.08393, 2023.
  12. Can large language models provide feedback to students? a case study on chatgpt. In 2023 IEEE International Conference on Advanced Learning Technologies (ICALT), pages 323–325. IEEE, 2023.
  13. Xi Lin. Exploring the role of chatgpt as a facilitator for motivating self-directed learning among adult learners. Adult Learning, page 10451595231184928, 2023.
  14. Harnessing the potential of large language models in medical education: promise and pitfalls. Journal of the American Medical Informatics Association, page ocad252, 2024.
  15. Malik Sallam. The utility of chatgpt as an example of large language models in healthcare education, research and practice: Systematic review on the future perspectives and potential limitations. medRxiv, pages 2023–02, 2023.
  16. Large language models in medical education: Opportunities, challenges, and future directions. JMIR Medical Education, 9(1):e48291, 2023.
  17. Large language models for software engineering: Survey and open problems. arXiv preprint arXiv:2310.03533, 2023.
  18. Exploring natural language processing in mechanical engineering education: Implications for academic integrity. International Journal of Mechanical Engineering Education, 52(1):88–105, 2024.
  19. Seed: Simple, efficient, and effective data management via large language models. arXiv preprint arXiv:2310.00749, 2023.
  20. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.
  21. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015.
  22. Adversarial training for large neural language models. arXiv preprint arXiv:2004.08994, 2020.
  23. Survey of vulnerabilities in large language models revealed by adversarial attacks. arXiv preprint arXiv:2310.10844, 2023.
  24. Cognitive mirage: A review of hallucinations in large language models. arXiv preprint arXiv:2309.06794, 2023.
  25. Large language models: a comprehensive survey of its applications, challenges, limitations, and future prospects. Authorea Preprints, 2023.
  26. State of the art of virtual reality technology. In 2016 IEEE aerospace conference, pages 1–19. IEEE, 2016.
  27. Frank Biocca. Virtual reality technology: A tutorial. Journal of communication, 42(4):23–72, 1992.
  28. Augmented reality: A comprehensive review. Archives of Computational Methods in Engineering, 30(2):1057–1080, 2023.
  29. A review on mixed reality: Current trends, challenges and prospects. Applied Sciences, 10(2):636, 2020.
  30. What is mixed reality? In Proceedings of the 2019 CHI conference on human factors in computing systems, pages 1–15, 2019.
  31. Using mixed reality (xr) immersive learning to enhance environmental education. The Journal of Environmental Education, 54(1):58–71, 2023.
  32. Designing immersive virtual reality simulation for environmental science education. Electronics, 12(2):315, 2023.
  33. A serious mixed-reality game for training police officers in tagging crime scenes. Applied Sciences, 13(2):1177, 2023.
  34. A roadmap for designing extended reality tools to teach unit operations in chemical engineering: Learning theories & shifting pedagogies. Digital Chemical Engineering, 6:100074, 2023.
  35. Yukang Xue. A review on intelligent wearables: Uses and risks. Human Behavior and Emerging Technologies, 1(4):287–294, 2019.
  36. A survey of knowledge-enhanced text generation. ACM Computing Surveys, 54(11s):1–38, 2022.
  37. Easyedit: An easy-to-use knowledge editing framework for large language models. arXiv preprint arXiv:2308.07269, 2023.
  38. Factuality enhanced language models for open-ended text generation. Advances in Neural Information Processing Systems, 35:34586–34599, 2022.
  39. Contrastive learning reduces hallucination in conversations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13618–13626, 2023.
  40. Knowledgpt: Enhancing large language models with retrieval and storage access on knowledge bases. arXiv preprint arXiv:2308.11761, 2023.
  41. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pages 2206–2240. PMLR, 2022.
  42. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. arXiv preprint arXiv:2212.10509, 2022.
  43. Reta-llm: A retrieval-augmented large language model toolkit. arXiv preprint arXiv:2306.05212, 2023.
  44. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020.
  45. Do language models know when they’re hallucinating references? arXiv preprint arXiv:2305.18248, 2023.
  46. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023.
  47. Hamid Reza Saeidnia. Welcome to the gemini era: Google deepmind and the information industry. Library Hi Tech News, 2023.
  48. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
  49. Malhar Lathkar. Getting started with fastapi. In High-Performance Web Apps with FastAPI: The Asynchronous Web Framework Based on Modern Python, pages 29–64. Springer, 2023.
  50. Malhar Lathkar. Introduction to fastapi. In High-Performance Web Apps with FastAPI: The Asynchronous Web Framework Based on Modern Python, pages 1–28. Springer, 2023.
  51. Mongodb vs postgresql: A comparative study on performance aspects. GeoInformatica, 25:243–268, 2021.
  52. Augmented reality smart glasses use and acceptance: A literature review. Computers & Education: X Reality, 2:100028, 2023.
  53. A comparison of perceived image quality between computer display monitors and augmented reality smart glasses. Radiography, 29(3):641–646, 2023.
  54. Wearable technologies: The role of usefulness and visibility in smartwatch adoption. Computers in Human Behavior, 65:276–284, 2016.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Ramez Yousri (1 paper)
  2. Zeyad Essam (1 paper)
  3. Yehia Kareem (1 paper)
  4. Youstina Sherief (1 paper)
  5. Sherry Gamil (1 paper)
  6. Soha Safwat (1 paper)
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets