Orientation: Case studies in learning analytics

Context:
Faculty of Organization and Informatics, University of Zagreb; courses Mathematics 1 (undergraduate programme) and Discrete Structures with Graph Theory (graduate programme). For both courses, the learning design was developed using the BDP tool (constructive alignment), and innovative learning and teaching strategies are applied.
Objective:
The objective is to compare the predictive power of formative assessment results (quizzes, homework assignments) and other student activities (videos, e-books, class attendance) for summative assessment outcomes (midterm exams) and to develop a reliable predictive model.
Data:
Data on assessment results (formative — quizzes, homework assignments; summative — midterm exams); activity records (logs) in the LMS (videos, e-books); class attendance data; responses to a student survey; a total of 813 students across two consecutive academic years (2021/2022 and 2022/2023).
Analysis:
A Random Forest machine learning algorithm (implemented in R) was used for predictions, including testing the efficiency of different models; distribution of students into classes and transitions between classes; analysis of student logs by class; and descriptive statistics for the student survey.
Results:
Formative assessment, together with previous summative assessment, proved to be a stronger predictor of summative assessment outcomes than other student activity data. The study highlighted the importance of data completeness and quality, as well as a clear alignment between assessment and learning outcomes when predicting student performance. Predictions were found to be less reliable for students with the lowest and highest results. It was also observed that other factors may influence predictions, such as the level of learning outcomes or factors that are difficult to capture from digital data, including the learning environment and individual learning strategies.
Full study:
Divjak, B., Svetec, B., & Horvat, D. (2024). How can valid and reliable automatic formative assessment predict the acquisition of learning outcomes? Journal of Computer Assisted Learning, 40(6), 2616–2632.