Data literacy and artificial intelligence – biases and limitations

1. Introduction

One of the most significant challenges to the responsible use of AI in higher education is data bias, which can result from various factors, such as inadequate samples, collection and measurement errors, algorithm design limitations, or subjective interpretations reflected in the data. These biases can lead to unfair educational outcomes for students, undermine trust in AI technologies, and reduce their effectiveness in teaching and evaluation.

AI systems, though advanced, cannot independently grasp the complexity of educational environments. They rely on the quality and representativeness of the data provided, making human oversight and critical analysis of results essential. Teachers, researchers, IT professionals, and other educational stakeholders should work together to assess data sources, their interpretation, and the ethical implications of their use.

The following outlines the main types of data bias, their potential consequences in education, the limitations of AI systems, and recommendations for the ethical and responsible use of data in practice.

2. What are biases in data?

Bias in data refers to a systematic error that can lead to distorted or inaccurate analysis results, resulting in unfair decisions in education. It arises when certain groups of students, behavioural patterns, or conditions are not properly represented in the data, or when models are applied that do not account for the specifics of the learning environment.

Bias can occur at various stages of data collection and processing, and it is typically classified into several main groups.

• Sampling bias occurs when an artificial intelligence system collects data that is not representative of the actual population.
  

  Example: If an artificial intelligence system uses data only from students in technical faculties, its learning models cannot be accurately applied to students from other fields.

• Measurement bias occurs when artificial intelligence incorrectly records or interprets data.
  

  Example: If the system tracks only time spent in e-learning without distinguishing between active and passive participation, the results on student engagement may be inaccurate.

• Algorithm bias occurs when an AI is trained on biased data, reproducing existing inequalities.
  

  Example: An algorithm that recommends content may favour students with better prior performance.

• Confirmation bias refers to the tendency of an AI system (or its users) to interpret data in a way that confirms existing assumptions.
  

  Example: If an AI model learns that the speed of solving problems is an indicator of knowledge, it may confirm the false assumption that faster students are more successful, even though speed does not necessarily indicate better understanding.

Understanding the different types of bias is essential for the responsible use of AI in education. Only by identifying, analysing and mitigating them can we ensure fair, objective and reliable results.

3. Consequences of bias in the educational use of artificial intelligence

The following outlines the most important consequences of bias in the educational application of artificial intelligence and their possible effects on the quality and fairness of the educational process.

Examples of Situations of Bias and Unfair Outcomes in Education

Example 1 – Analysis of Written Assignments

For example, students who write more creatively, use non-standard expressions, or deviate from the prescribed format may receive lower grades, even when the content of their work is high quality and relevant. In this way, the algorithm does not recognise the complexity and diversity of academic expression, which can result in unfair outcomes and demotivate students.

Example 2 – Monitoring Participation in Online Classes

For example, a student who carefully studies materials outside the platform, takes their own notes, or collaborates with colleagues through other digital channels may be incorrectly labeled as “inactive.” Such bias in measuring engagement leads to inaccurate conclusions about students’ actual activity and can affect decisions about planning and adjusting classes or assessing student performance.

Example 3

Situations in which the system unknowingly favours students with better technical resources, stable internet access, or higher levels of digital skills, while students who face technical limitations or have different learning styles may be incorrectly assessed as less engaged.

Example 4 

If the algorithm makes recommendations based on previous results, students with lower initial success may be given easier tasks and thus have fewer opportunities to progress. This does not reduce existing differences, but further deepens them.

Example 5

From a legal perspective, the risks relate to the protection of personal data and student privacy. Irresponsible collection, storage, or processing of data may conflict with the General Data Protection Regulation (GDPR). Particular attention should be paid to informed consent, limitation of the purpose of processing, and minimisation of the amount of data collected. For example, if data on students’ behaviour in the digital environment is used for automated evaluation without their consent or the possibility of insight into the decision-making process, this violates the principle of transparency and threatens students’ right to fair treatment. For this reason, it is important for higher education institutions to develop clear data management policies, provide regular education on the ethical use of AI, and introduce ongoing human oversight of algorithmic systems. It is the only way to maintain fairness, transparency, and accountability in the educational process.

These examples demonstrate how biases in AI systems can have real and measurable consequences for the educational process. It is therefore crucial that teachers and educational professionals develop the ability to critically analyse the results generated by such systems, understand their limitations, and ensure that technology is used in a way that promotes fairness, transparency, and inclusiveness in education.

4. Limitations of AI in education

Despite the many benefits of AI systems in education, it is important to understand their limitations to ensure responsible, fair, and effective application in educational contexts.

Limited understanding of context
AI-based systems cannot fully capture the complexity of the educational process, teacher-student relationships, or individual learning circumstances. The recommendations they generate may be technically accurate but pedagogically inadequate or too general, as they do not recognise motivation, emotional factors, or the social dynamics of learning.

Dependence on data quality
The effectiveness of AI in education largely depends on the quality, accuracy, and representativeness of the data used in analysis. If the data is biased, outdated, or incomplete, the systems may produce inaccurate interpretations, unfair assessments of engagement, or inappropriate learning recommendations. These shortcomings can lead to erroneous decisions by teachers and reduce student trust in the technology.

The need for human oversight
Even the most advanced AI systems cannot make all decisions in education independently. Human oversight is essential for interpreting results, validating conclusions, and assessing pedagogical justification. Teachers and professionals must ensure that algorithmic decisions align with ethical principles, curriculum goals, and student well-being.

Limited adaptability
While AI enables personalisation of content based on data analysis, it still cannot replace the pedagogical insight, empathy, and contextual understanding that teachers bring to the teaching process. Human intervention remains necessary to tailor learning to the real needs of individuals and groups.

Ethical and responsible development
The introduction of AI into education requires careful consideration of ethical and legal aspects, particularly regarding privacy, transparency, and equal access. Only responsibly developed and overseen systems can ensure that the technology contributes to a just, inclusive, and safe educational environment.

5. Ethical data use and privacy protection in the application of AI in education

As AI systems collect, analyse, and interpret large amounts of educational data, every step in the process must be grounded in accountability, transparency, and respect for the privacy of students and teachers.

Adherence to the Principle of Transparency
All participants in the educational process should be clearly informed about how AI systems use their data. For example, students should know why data about their progress is collected, how it is processed, and how the system generates personalised recommendations or feedback. Transparency is essential for building trust and acceptance of AI technologies in education.

Obtaining Informed Consent
Data collection and processing are permitted only with the free and informed consent of the data subjects, in accordance with ethical and legal standards such as the General Data Protection Regulation (GDPR). Students should have the right to know what information is collected, how long it is retained, and for what purposes – especially when the data is used to train or adapt AI systems.

Ensuring Privacy and Data Security
When AI analyses educational data, it is essential to ensure that all data is adequately protected from unauthorised access, loss, or misuse. Security is achieved through technical and organisational measures such as encryption, anonymisation, and the use of trusted storage and processing systems. For example, when analysing student engagement in digital systems, students’ identities should remain hidden to preserve their privacy.

Promoting Fairness and Accountability in AI Systems
AI systems in education must be designed and implemented to prevent discrimination and bias. Regular testing and evaluation of models are necessary to verify that AI decisions are fair and aligned with pedagogical goals. For example, if the system recommends additional educational content, it is necessary to ensure that the recommendations are equally accessible to all students, regardless of their previous results, sociodemographic characteristics, or language abilities.

The Role of Teachers and Students in the Ethical Use of Artificial Intelligence
Teachers and students share the responsibility for the ethical handling of data and AI technologies. Teachers should ensure a safe digital environment, encourage open discussion about ethical challenges, and critically examine the results generated by AI systems. Students need to develop awareness of the importance of protecting personal data and understand how their interaction with AI tools can affect their learning experience.

Data literacy in the context of artificial intelligence involves not only understanding the technical processes of data collection and processing, but also recognising the biases, ethical challenges, and limitations that these systems carry. Only with a thoughtful, responsible, and transparent approach can it be ensured that artificial intelligence in education contributes to the creation of a fair, safe, and supportive learning environment.

6. Conclusion

In modern education, the abundance of data and the increasing use of artificial intelligence demand a new level of professional responsibility and understanding from teachers and students. Data literacy is becoming a core competency – it enables critical reflection on how data is collected, analysed, and interpreted, and it increases awareness of potential biases and ethical challenges that may arise from its use.

Developing data literacy also involves understanding how data-driven decisions affect learning, assessment, and lesson planning. Combined with ethical principles and pedagogical sensitivity, it supports the responsible use of artificial intelligence in education – respecting diversity, promoting fairness, and strengthening trust in digital tools and systems.

Ultimately, only data-literate participants in the educational process can make informed decisions and use artificial intelligence technologies in ways that truly improve the quality of learning and teaching, without undermining the fundamental principles.

Reflective activity:

Reflect on your own experience using digital tools and systems that incorporate elements of artificial intelligence in education.

• How do you use student data (or your own learning data) to improve learning outcomes?
• Are you aware of potential biases in the data you analyse or use?
• How could you improve your approach to data literacy to ensure that the use of AI in education is fair, safe, and transparent?

Write down a few concrete examples from your own practice or from your environment.

7. Literature

• Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2023). On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, 610–623.  
  • This paper examines the ethical challenges and potential risks related to the development and use of large-scale language models, highlighting issues such as bias, unexpected outcomes and limitations in understanding context.
• Chouldechova, A., & Roth, A. (2023). A snapshot of the frontiers of fairness in machine learning. Communications of the ACM, 66(3), 30–38.
  • The authors present an overview of current research on fairness in machine learning, highlighting various approaches, definitions and technical challenges in reducing bias.
• Cheong, B. C. (2024). Transparency and accountability in AI systems: Safeguarding wellbeing in the age of algorithmic decision-making. Frontiers in Human Dynamics. 
  
  • This review paper examines the main legal and ethical challenges related to implementing transparency and accountability in AI systems.
• Floridi, L., Cowls, J., Beltrametti, M., Chatila, R., Chazerand, P., Dignum, V., ... & Vayena, E. (2018). AI4People—An ethical framework for a good AI society: Opportunities, risks, principles, and recommendations. Minds and Machines, 28(4), 689–707.
  • This paper presents an ethical framework for the development and application of artificial intelligence aimed at socially beneficial outcomes, including the principles of transparency, fairness and accountability.
• UNESCO (2021). Recommendation on the ethics of artificial intelligence. 
  • The UNESCO Recommendation provides an international ethical framework for the development and use of artificial intelligence, emphasising human rights, transparency, privacy protection and the promotion of fairness.