Videolesson: Trustworthiness of learning analytics - how to achieve it?

Watch the video lecture on the trustworthiness of learning analytics, which covers both social and technological aspects relevant to ensuring the trustworthiness of learning analytics. The video is based on a literature review and expert insights. The video lecture also includes an interview with an expert in this field.

	Welcome. In this video lesson we explore how the results of learning analytics are presented to users, students, teachers, and decision-makers through different forms of reporting. Learning analytics does not end with data collection and analysis. Reporting is a crucial phase because it enables insights from learning analytics to be applied in teaching and learning. There are various ways of reporting that are useful for different user groups. One example is statistical reports, which may include for instance, analysis of grades. More complex reports are also possible and may include not only descriptive but also predictive learning analytics. Such reports can be useful for teachers or decision makers, but they're not interactive and do not provide real-time data analysis. As such, there are not resources that students commonly use. Students benefit more from interactive forms of reporting, such as real-time messages in a learning management system. These represent a form of personalized feedback. For example, a student may automatically receive a reminder if they have been inactive for more than seven days, or receive feedback on their work. Messages that guide students through the learning process can increase motivation and support self-regulated learning. However, the most common form of reporting in learning analytics is the dashboard. Dashboards are visual interfaces that display key information and learning analytics results such as learning progress, engagement, achieved learning outcomes and risk alerts often in real time. They make it easier to track learning and provide the basis for reflection and intervention. They may be integrated into learning management systems, MSs (such as Moodle), and based on data from these systems, but they may also be standalone. Dashboards provide visualizations based on the digital footprint. Students leave in educational and environments, and they might also include recommendations. Data are collected, unobtrusively, or automatically without interrupting students or requiring teacher intervention. They usually include data on resource use, time spent, and student assessment. They can provide real-time reports on student progress and learning behavior. Their functionalities range from static reports to dynamic visualizations that can be customized. There are different types of dashboards intended for different stakeholders. Research has shown that most dashboards are aimed at teachers and that they insufficiently support students in developing learning related skills such as metacognition and self-regulation. In this context, the need to develop student-centered dashboards has increasingly been emphasized. A student dashboard can be defined as an interactive, historical, personalized, and analytical display that shows a student's learning patterns, status, performance, and interactions. In addition to insights into course progress, as an example of descriptive analytics, student dashboards may also include predictions related to achieving learning outcomes. As an example of predictive analytics, they may also include visual representations of learning behavior, highlight remaining tasks, show comparisons with other studentsand provide reminders, recommendations, and planning functions. Access to analytics can encourage reflection, behavioral change, and ultimately better achievement of learning outcomes. A study by Divjak et al. (2023) from 2023 examined what students actually want and expect from dashboards in particular, which functionalities they consider most useful for supporting their learning. It showed that students most value features related to immediate learning planning, such as schedules, calendars, notifications and overviews of points and grades. On the other hand, features related to competition and comparison were of less interest to students. However, other studies have shown that comparisons can be beneficial for students. Therefore, the development of dashboards must take into account differences in educational and cultural contexts, as well as individual differences. In this sense, opportunities for dashboard customization and personalization in line with the principle of human-centered learning analytics are particularly important. This allows students to use learning analytics in ways that match their individual needs. The same study also resulted in a proposal for a student dashboard model consisting of five components planning, which includes features that support time management, task tracking, reminders, and calendars. These features were rated as the most important by students comparisons. They enable comparison of a student's progress with that of others. While some students find this motivating. Others are concerned about competitiveness and stress. Predictions includes features related to predicting student success. These may support self-regulation, but may also have a potentially discouraging effect. Teachers includes features that enable students to provide feedback on teaching staff. Extracurricular includes badges, rewards and competition results, which students considered the least important. On the other hand, there are teacher dashboards. These provide insight into student activities, progress and achievement help identify students who need support and facilitate the adaptation of teaching and learning processes. While some dashboards simply present information and leave interpretation to the teacher, new approaches provide alerts about potential problems in individual students or groups and suggest possible interventions. They may include recommendations such as selecting students for group work or switching to another activity. However, most existing dashboards still focus on helping teachers understand the situation while fewer actively support specific pedagogical action. Teachers may also greatly benefit from learning design analytics displayed in dedicated dashboards such as those in the Balanced Design Planning BDP tool. These dashboards allow teachers to see detailed analysis of course, design from student workload, and the application of different learning types to constructive alignment. These analysis provide a foundation for improving learning design in accordance with key pedagogical principles and for applying innovative pedagogies. In learning analytics, the design and evaluation of dashboards is an important field with the goal of developing displays that gives students and teachers insight into data in a way that raises awareness and supports the transformation of teaching and learning. In line with human centered learning analytics, it is essential to involve students and teachers in the development of learning analytics systems and their dashboards. This ensures their purposefulness and usefulness in improving teaching and learning. Of course, students and teachers also share responsibility for actively engaging in these processes and contributing their perspectives. Dashboards and reports in learning analytics support students and teachers in making informed decisions. However, to be useful, they must be adapted to the context, needs, and users. Therefore, their proper development requires collaboration between designers, teachers, students, and other stakeholders.

Transcript

Welcome to this video lesson on the trustworthiness of learning analytics. After an introduction to learning analytics and a discussion of key concepts, we will talk about the aspects that influence the trustworthiness of learning analytics and the responsibilities of different stakeholders.

Learning analytics has been used for more than a decade to improve understanding of learning processes, prepare foundations for targeted learning support, enhance learning experiences, and encourage self-regulated learning. Although its benefits are clearly recognised in research and practice, learning analytics has not yet been widely implemented, especially outside higher education. This is due to a number of reasons, such as the lack of appropriate policies, strategies, and capacities. Another reason may be related to trust. Without trust, it is difficult to expect students, teachers, and other stakeholders such as institutions and decision-makers to participate in the implementation of learning analytics, or to use it as a basis for decision-making. Ensuring the trustworthiness of learning analytics is therefore a key prerequisite for achieving its full potential as a basis for informed decision-making in education.

While trust is subjective, trustworthiness can be considered an objective quality of learning analytics systems. A number of studies have addressed issues related to the trustworthiness of learning analytics. Learning analytics is closely connected with artificial intelligence, and so there are many parallels in the dimensions of trustworthiness. Particularly important questions are arising with the rapid development of generative AI. Taking all this into account, the dimensions of trustworthiness can be grouped into several key aspects.

We can talk about social, technological, and horizontal aspects. They are not isolated, but interwoven and interact to contribute to the trustworthiness of learning analytics.

Social Aspects

Social aspects focus on people, their roles and rights, and are closely related to ethics.

Learning analytics systems often collect sensitive data, such as grades and online activity. Therefore, privacy, data protection, and data security are crucial dimensions. It is important to ensure that personal data are protected from misuse, unauthorised access, and inappropriate analysis. For example, student data are anonymised before analysis.

Another important dimension is agency, autonomy, and control. Students should have a voice in how their data are used in learning analytics. They should be able to decide whether they want to make their data available, and be informed about how the data are used. Issues of control also include ownership of the data derived from the students’ raw data.

A further key dimension concerns stakeholders. Everyone involved in the educational process should be able to participate in the design and implementation of learning analytics, including teachers, students, and decision-makers. Ensuring that learning analytics systems reflect the interests of students and other stakeholders contributes to transparency and trustworthiness. It is also important to trust both higher education institutions and external stakeholders with regard to their competence in implementing learning analytics.

Technological Aspects

Technological aspects focus more on the technical features of learning analytics tools and systems, and are also linked to privacy and ethical data use.

The first dimension concerns data, algorithms, accuracy, and fairness. The accuracy, reliability, and fairness of learning analytics are closely related to the data on which they are based. However, there are challenges concerning incomplete or biased data, which can lead to biased algorithms and unreliable results. These issues may be caused by different data collection channels or students choosing not to participate with their data.

The second dimension is infrastructure, referring to the availability of appropriate software architecture and secure data management frameworks, including support for handling large datasets.

The third dimension is accessibility. Learning analytics systems should be accessible and understandable to different user groups, including people with disabilities (such as visual impairments), and speakers of less widely used languages.

Horizontal Aspects

These aspects relate to both social and technological dimensions.

The first is transparency. Students should know which data are collected, who collects them and for what purpose, and who has access to them. Transparency and explainability of algorithms are also essential, especially in the context of generative AI.

The second dimension is stakeholder responsibility.

Stakeholder responsibility

The areas of responsibility of different stakeholders vary and also overlap.

Educational systems and institutions

Decision-makers are responsible for strategic planning and leadership to ensure that learning analytics is implemented meaningfully, with focus and transparency. They are also responsible for investments, infrastructure, and providing technical and pedagogical support for teachers and students.

Teachers play a key role in implementing learning interventions based on learning analytics. They must ensure that teaching and learning processes are pedagogically grounded, including the design of learning activities. Without this foundation, the meaningfulness and explainability of learning analytics are questionable. Teachers are also crucial in interpreting learning analytics results and responding in ways that promote learning outcomes.

Both decision-makers and teachers must be aware of data limitations and potential biases caused by incomplete data. They must also be aware of the appropriate use of algorithms and statistical models, and of the risks related to generative AI. Understanding the specific educational and cultural context is essential for the correct interpretation and design of targeted interventions.

Students

The role of students is to provide feedback about which learning analytics are useful and the kind of support they need. They are also responsible for self-regulating their learning based on insights gained from learning analytics.Finally, all three stakeholder groups should be involved in the development and implementation of learning analytics systems. This helps to achieve human-centred learning analytics. It is also essential that stakeholders develop data literacy and AI literacy required to understand and implement learning analytics. Acting in accordance with ethical standards is a crucial prerequisite for trustworthiness.

We will talk about learning analytics to better understand challenges and opportunities in practice.

What is the greatest challenge in implementing trustworthy learning analytics at universities?Firstly, it is important to define the pedagogical foundation for learning analytics, and then ensure the technical conditions to implement it in a trustworthy way.

This requires high-quality data, which includes data cleaning, preparation, and alignment with teaching and learning processes. This takes a long time and requires ongoing improvement.

We must also be ready to admit when data are not trustworthy and to recognise specific biases.

This requires significant technical support, which is not negligible.

On the other hand, there are ethical aspects that must be respected: we must not collect data without consent, and when data are collected and analysed, the results must be used ethically and in pedagogically meaningful ways.

How can students and teachers actively participate in developing learning analytics systems?

Learning analytics is intended for both students and teachers. Therefore, from the beginning, they have to clearly express their needs.For example, research can be conducted on what students expect from learning analytics, which is what we did. Many students wanted an overview of upcoming activities.

However, when asked whether they wanted information on how they compare with the group, many said they did not need it.

Teachers also need to participate in defining the questions learning analytics should answer, and in evaluating results. Their interpretation of results is crucial. Therefore, teachers require significant professional development, and institutions should support this.

In short, both students and teachers have an active role.

What are the key ethical challenges when using generative AI in learning analytics?

The biggest issue is transparency.

When generative AI is used, we often do not know what is happening inside the “black box.” If teachers or students do not understand the results and accept them uncritically, then trust in the system cannot be established.

I therefore advocate the use of analytics where we can clearly explain what is happening inside that “black box.”Another major issue is data protection and privacy. Everyone needs to  know what is happening with their personal data.

We must also be aware of possible biases. For example, if certain groups, such as students with disabilities, choose not to share their data, the resulting analytics will be biased.

This shows that ensuring trustworthiness is not simple – neither for algorithms and data nor for pedagogically meaningful interpretation.

Thank you for the inspiring messages that we can apply in our educational contexts. Trustworthy learning analytics requires participation, understanding, and critical reflection. Thank you for following this video lesson.