International Journal of
Information and Education Technology

Abstract

In this work, the results of an analysis aimed at evaluating the effectiveness of a virtual assistant in helping undergraduate students with comprehension and critical reading of scientific texts are presented. The virtual assistant, based on Google generative AI Gemini 1.5, was specifically designed for this study and programmed in Python, using LangChain and LangGraph (platforms optimized for the design of applications based on generative artificial intelligence). A comparative analysis was conducted on two groups of students: a control group that was not provided with the virtual assistant and an experimental group that was granted the use of the virtual assistant. A set of reading comprehension questions was given to the students. The effectiveness of the virtual assistant was analyzed quantitatively using natural language processing techniques, measuring the cosine similarity between the students’ answers to the questions and the correct answers. The analysis showed a significant difference in the accuracy of responses to reading comprehension questions between the experimental and the control groups. This work has shown that the controlled use of generative artificial intelligence in the classroom can provide great benefits. The use of virtual assistants can be particularly helpful in distance learning courses or e-learning, since they can be used remotely and at a student’s own pace.