Using AI, university monitors online students at risk of dropping out

Dropping out early is one of the main concerns in online higher education, especially during the first year of a degree. An interdisciplinary team of researchers from the Universitat Oberta de Catalunya (UOC) has developed a new system based on artificial intelligence algorithms that makes it possible to identify on a daily basis those students that are at risk of failing and can automatically take early action by sending personalized messages to reverse the situation.

According to the researchers, this continuous monitoring helps reduce the time between the first signs of risk and the system's intervention to help prevent students from dropping out.

The technology has been pilot tested with 581 students enrolled on first-semester courses in a number of the UOC's Faculty of Economics and Business's bachelor's degree programs and has been found to reduce dropout rates and increase participation during the semester.

The research is led by David Bañeres, a member of the Systems, Software and Models Research Lab (SOM Research Lab) at the Internet Interdisciplinary Institute (IN3). He coordinates a multidisciplinary team featuring Ana Elena Guerrero, principal investigator of the Technology-Enhanced Knowledge and Interaction Group (TEKING) and a member of the Faculty of Computer Science, Multimedia and Telecommunications, María Elena Rodríguez-González, a member of the same research group and faculty, and Pau Cortadas, researcher at the Faculty of Economics and Business.

Prediction improvements for the LIS system

These promising results have been achieved thanks to the development of a new predictive model called Profiled Dropout At Risk (PDAR), which has been incorporated into the Learning Intelligent System (LIS). This system, developed by the same team of researchers, predicts whether students are at risk of failing and has been tested with good results in various pilot tests with UOC students since 2019.

Until now, the LIS system only had a predictive model to decide whether students were likely to complete an academic year based on historical course data—collected in the Datamart of the UOC's Institutional Project Evaluation Unit—and on the results of the continuous assessment activities carried out during the current academic year. Thus, after each activity, the LIS system predicts the minimum grade that should be obtained by the student in the next one in order to pass the course and assigns a level of risk of failing, which is then shown in the student's personal area in the form of a traffic light.

If a high level of risk is detected, the system activates the applicable intervention mechanisms in the form of messages to students. "This prediction, although very useful for students, has shortcomings, mainly because monitoring is limited to certain checkpoints after each activity (usually three or four each semester), which means that the associated intervention can be too late, when the student has already dropped out for the year," said Bañeres.

In contrast, the new PDAR model significantly improves monitoring, as it uses data on students' characteristics, their performance during the academic year and their clicks and other daily actions on the UOC's online campus to generate a daily prediction of the risk of dropping out.

"The model assesses whether the student's daily engagement matches the average for the course. In other words, this assessment is carried out based on each course and each activity," he said.

Avoiding false positives

One of the challenges of the new model has been to avoid false positives: people being wrongly identified by the system as being at risk. This error primarily affects students who are not always active in the virtual learning environment. Thus, the new model also takes into account a time window that is automatically calculated based on the academic year and on the type of activity and its difficulty.

In other words, in order to confirm that a student is really at risk of dropping out and activate the appropriate intervention mechanisms, the student must remain in the category of being at risk of dropping out for a given consecutive number of days, which is specified for each activity. If a student is at high risk of dropout, they are sent an automatic intervention message.

Personalized automatic messages

The aim of the intervention by the system is to increase students' motivation, for example by making recommendations on time management, setting short-term goals or providing information on the possible negative consequences of not completing the activity. It also provides additional learning materials and exercises to help students achieve their goals.

In addition, the teaching staff for the course can design and personalize the content of messages and adapt them according to the risk level. Finally, the tool has various different dashboards that allow both the teaching staff and the student to individually ascertain their current status and potential risks.

In order to assess the incorporation of this new predictive model, dropout results were compared with the group of students who had used the LIS system the previous year—when the PDAR had not yet been implemented—and with the group of students who chose not to participate in the study and took the course without the support of the LIS system.

The results, published in the International Journal of Educational Technology in Higher Education, which is co-edited by the UOC, showed that dropout rates at the end of the academic year fell significantly in all activities, with a difference of 12% between those students who participated in the pilot study and those who did not, and with a difference of 5% compared to the previous semester, when LIS was used on its own, without the new predictive model.

A tool at the service of teaching staff

This new system gives teaching staff the chance to intervene proactively to address their students' problems. "With this early detection system, students can be notified before the problem occurs, and we can monitor them 24/7," said Bañeres.

In addition, it is a scalable tool that allows teaching staff to comprehensively monitor courses with a high number of students. "For example, one of the pilot studies was carried out in a course with 1,500 students, and the system allowed teaching staff to monitor students at risk of dropping out without overloading the coordinating professor or the course instructors," he said.

A system that can be adapted to any online learning environment

A major advantage of the LIS system is that it does not rely on a particular learning management technology and can therefore be used in any environment that has students' academic data.

"This means that, even if the UOC is currently transitioning to a new campus based on another technology, such as Canvas, the system can still be used by acquiring data from the relevant information source," said Bañeres. For example, the team of researchers is currently working on a teaching personalization project for the European Patent Office, where the LIS system will be used to monitor students within its Moodle-based teaching platform.

In addition, LIS can be configured specifically for each program, adapting to the activities forming part of it and training the necessary prediction models with data from previous students who passed it.

Related research is also published in the journal IEEE Transactions on Learning Technologies.