Dice Question Streamline Icon: https://streamlinehq.com

Effectiveness of Intelligent Tutoring Systems (ITSs) on Teaching and Learning

Determine whether Intelligent Tutoring Systems (ITSs) meaningfully impact teaching and learning outcomes by conducting rigorous, transparent evaluations that can resolve mixed evidence and address criticisms of existing evaluation protocols.

Information Square Streamline Icon: https://streamlinehq.com

Background

The paper reviews decades of work on Intelligent Tutoring Systems (ITSs), noting that despite broad adoption and initial optimism, empirical evidence on their effectiveness is mixed and evaluation methods have been criticized. This persistent uncertainty about impact, coupled with the absence of standardized evaluation practices, poses a foundational question for educational technology research and deployment.

Clarifying the actual effect of ITSs on teaching and learning is crucial for guiding investment, informing policy, and benchmarking newer generative AI tutoring approaches against established systems.

References

Despite initial excitement about the potential of ITSs to revolutionise education~\citep{davies2021mobilisation, seldon2020fourth}, and their broad adoption~\citep{becker2017artificial, miao2021ai}, it remains unclear if they can impact teaching and learning in a meaningful way~\citep{holmes2022artificial,zawacki2019systematic}: evidence of their effectiveness is mixed~\citep{holmes2022artificial,ilkka2018impact,foster2023edtech,kulik2016effectiveness}, and the underlying evaluation protocols have come under criticism~\citep{wollny2021we,okonkwo2021chatbots} (see Section~\ref{sec:evaluation_its} for more details).

— Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approach (2407.12687 - Jurenka et al., 21 May 2024) in Section 3.2 (Lack of transparency and common evaluation practices: lessons from EdTech)