Achieving Fairness in DareFightingICE Agents Evaluation Through a Delay Mechanism (2312.16010v1)
Abstract: This paper proposes a delay mechanism to mitigate the impact of latency differences in the gRPC framework--a high-performance, open-source universal remote procedure call (RPC) framework--between different programming languages on the performance of agents in DareFightingICE, a fighting game research platform. The study finds that gRPC latency differences between Java and Python can significantly impact real-time decision-making. Without a delay mechanism, Java-based agents outperform Python-based ones due to lower gRPC latency on the Java platform. However, with the proposed delay mechanism, both Java-based and Python-based agents exhibit similar performance, leading to a fair comparison between agents developed using different programming languages. Thus, this work underscores the crucial importance of considering gRPC latency when developing and evaluating agents in DareFightingICE, and the insights gained could potentially extend to other gRPC-based applications.
- I. Khan, T. V. Nguyen, X. Dai, R. Thawonmas, “DareFightingICE Competition: A Fighting Game Sound Design and AI Competition,” 2022 IEEE Conference on Games (CoG), pp. 478-485, August 2022.
- F. Lu, K. Yamamoto, L. H. Nomura, S. Mizuno, Y. Lee, and R. Thawonmas, “Fighting Game Artificial Intelligence Competition Platform,” Proceedings of the 2nd IEEE Global Conference on Consumer Electronics (GCCE), pp. 320-323, October 2013.
- “Introduction to gRPC,” gRPC. [Online]. Available: https://grpc.io/docs/what-is-grpc/introduction. [Accessed: May 14, 2023].
- C. Nimpattanavong, I. Khan, T. V. Nguyen, R. Thawonmas, W. Choensawat, K. Sookhanaphibarn “Improving Data Transfer Efficiency for AIs in the DareFightingICE using gRPC,” arXiv preprint arXiv:2303.10001, 2023 (accepted for oral representation at 2023 8th International Conference on Business and Industrial Research, May 2023).
- “Performance Best Practices,” gRPC. [Online]. Available: https://grpc.io/docs/guides/performance. [Accessed: May 14, 2023].
- J. Moon, Y. Choi, T. Park, J. Choi, J. Hong, and K. Kim, “Diversifying dynamic difficulty adjustment agent by integrating player state models into Monte-Carlo tree search,” Expert Systems with Applications, Volume 205, 2022.
- F. Waris, R. Reynolds, and J. Lee, “Evolving Deep Neural Networks with Cultural Algorithms for Real-Time Industrial Applications,” International Journal of Semantic Computing, Volume 16, No. 02, pp. 281-312, 2022.
- T. Van Nguyen, X. Dai, I. Khan, R. Thawonmas, and H. V. Pham, “A Deep Reinforcement Learning Blind AI in DareFightingICE,” 2022 IEEE Conference on Games (CoG), pp. 632–637, 2022.
- M. Ishihara, T. Miyazaki, C. Y. Chu, T. Harada, and R. Thawonmas, “Applying and improving Monte-Carlo Tree Search in a fighting game AI,” Proceedings of the 13th international conference on advances in computer entertainment technology, pp. 1-6, November 2016.