Implications of using artificial intelligence in educational systems

1. Introduction

Education is undergoing a period of intense digital transformation, with artificial intelligence (AI) emerging as a key driver of this change. AI systems are already actively contributing to learning analytics, individualised teaching, automation of administrative processes, and the development of new forms of interaction between teachers and students. In this context, AI is viewed not only as a technological tool but also as a strategic resource that can enhance the quality of teaching, increase access to knowledge, and improve the overall efficiency of higher education institutions.

Encouraging student engagement, providing timely and personalised feedback, identifying risks of failure, and supporting specific student needs are now real and accessible possibilities thanks to advanced AI algorithms and models. The recent surge in the use of generative AI tools in academic learning is particularly notable, further highlighting the need for critical reflection on their role and consequences.

The EU AI Act recognises education as an area where AI systems have a significant impact on individual rights and, in certain cases, classifies such systems as high risk, especially when they influence decisions on access to studies or performance assessment.

Understanding the broader context of AI applications in education is essential for effective and ethically sound integration. Therefore, the following discusses the key aspects of AI applications in education: ethical, security, technical, and legislative.

2. Ethical implications of applying artificial intelligence in education

The application of artificial intelligence in higher education offers numerous benefits but also requires careful consideration of important ethical issues related to student rights, system fairness, teachers' professional responsibility, and the protection of data security in the digital environment.

In the educational context, artificial intelligence mediates processes that are fundamentally human: the development of knowledge, critical thinking, autonomy, and academic integrity. Therefore, it is necessary to ensure that the technology aligns with the values and norms of higher education and that its use is based on high standards of cybersecurity.

Compliance with the European Act on Artificial Intelligence

AI systems that affect student assessment, selection, and progression can be classified as high-risk systems, which obligate institutions to ensure:

• Demonstrable fairness and control of bias
• Transparency and explainability
• Clear human oversight of decisions
• Secure data management and protection.

Reflective question

• How can you ensure in your own educational practice that the use of AI is fair and ethically sustainable, protects student privacy and security, remains transparent and understandable, supports human judgment and knowledge, and preserves academic integrity and professional values?

3. Security aspects of the application of artificial intelligence in higher education

Artificial intelligence systems in the education process collect a wide range of personal and professional data. Data about students is particularly sensitive, including academic performance, learning habits, and patterns of behaviour in the digital environment. In addition, artificial intelligence processes data about teachers, administrative staff, researchers, and external collaborators. Any unauthorised access to this data can violate privacy, professional integrity, and trust in the institution.

Compliance with the European Act on Artificial Intelligence and the GDPR

According to the European Act on Artificial Intelligence, systems that affect access to education or student assessment are classified as high-risk, which imposes obligations on institutions to:

• Actively manage security risks.
• Ensure the cyber resilience of the system.
• Document security measures and verification.
• Provide human oversight in key decisions.
• Fully comply with the GDPR in the protection of personal data.

The aim is to preserve the rights, safety, and well-being of all stakeholders in the academic community.

Reflective question

• What security measures and procedures are already implemented in your educational environment, and how would you improve them to ensure that the application of artificial intelligence remains safe and reliable?

4. Technical aspects

The successful application of artificial intelligence in higher education requires appropriate technical infrastructure, high-quality data, and systematic technical support for users. Technical conditions directly affect the reliability, security, and pedagogical value of artificial intelligence systems.

Compliance with the European Artificial Intelligence Act

The EU AI Act emphasises that AI systems in education must be robust and technologically reliable, adapted to security and technical requirements, and subject to human oversight and evaluation.

Institutions are responsible for ensuring that technical conditions do not compromise academic outcomes or the rights of all educational stakeholders.

Reflective question:

• What technical steps should your institution take to increase the reliability and pedagogical effectiveness of AI systems when working with students?

5. Artificial Intelligence and the Legislative Framework in Higher Education: What does the European Artificial Intelligence Act (EU AI Act) introduce?

The European Act on Artificial Intelligence(the EU AI Act) is the first comprehensive legal act of the European Union to regulate AI systems within the common market framework. 
In terms of higher education, this means that institutions and faculties introducing AI tools and systems must assume a new level of responsibility: systems that can affect student outcomes, selection, or progression may be classified as “high-risk.” The EU AI Act introduces a categorisation of AI systems based on their potential risk: minimal, high-risk, and unacceptable.

For example, systems that automatically select candidates for admission or assign grades can fall into the high-risk category and must meet specific requirements.

Higher education institutions have several key steps ahead:

• Conduct a risk assessment and test AI tools before full implementation.
• Ensure that the data driving the system meets the criteria for quality, relevance, and initial representativeness.
• Ensure transparency: students and teachers must be clearly informed about the system’s role and limitations.
• Introduce human oversight of results that may affect students’ academic status.
• Ensure the protection of personal data in accordance with the General Data Protection Regulation (GDPR) and protect against unauthorised disclosure.

Rights of students and stakeholders in the educational process

Students and other users of the system have the right to:

• Clear and timely information about how AI is used in their education.
• Understand the basis on which the system makes assessments or recommendations.
• The possibility to object and request human review of decisions mediated by AI.

If a higher education institution uses an AI system to automatically grade student work, in order to comply with the EU AI Act, it must:

• Document the system’s reliability, including testing on a variety of data.
• Inform students about the grading process, algorithm, and appeal options.
• Ensure human oversight of final grades to prevent unfair or non-transparent decisions.

Implementation of the legislative framework (EU AI Act) and timeline

The EU AI Act entered into force on August 1, 2024, with the implementation of individual provisions in phases. Higher education institutions must prepare for the gradual implementation of this legislative framework and adapt their systems and practices accordingly.

The application of AI in higher education can be a powerful tool for improving learning and administration, but its success depends on the responsibility of all stakeholders. In accordance with the EU AI Act, higher education institutions must establish standards that guarantee safe, transparent, and fair use of AI, with human accountability.

Reflective questions:

• What artificial intelligence tools are you already using in your institution, and how do you plan to align them with the EU AI Act?
• How would you establish a procedure for human review of decisions made using artificial intelligence?
• What is the first step you would suggest to improve transparency in the use of artificial intelligence in the classroom?

6. Literature

• Veale, M., & Binns, R. (2017). Fairness and accountability design needs for algorithmic support in high-stakes public sector decision-making. Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, 388–398.
  
  • Description: Examines the design of AI systems to ensure fairness and accountability in significant decision-making, including educational processes.
    
    
• Mittelstadt, B. D. (2019). Principles alone cannot guarantee ethical AI. Nature Machine Intelligence, 1(11), 501–507.
  
  • Description: Argues that ethical principles are insufficient without legal frameworks for responsible AI use, which is essential for educational systems.
    
    
• Binns, R. (2017). On the fairness of machine learning in education. Communications of the ACM, 63(10), 34–37.
  
  • Description: Addresses equity challenges in applying machine learning in education, emphasising the importance of transparency and human oversight.
    
    
• Kaminski, M. E. (2018). The right to explanation, explained. Berkeley Technology Law Journal, 34(1), 189–218.
  
  • Description: Explains the right of users to understand AI system decisions, which is important for transparency in education.
    
    
• European Commission (2024). Artificial Intelligence Act. 
  
  • Description: Official EU law regulating the development and use of AI, emphasising security, transparency and accountability, including in educational systems.
    
    
• Jodi, K. H. M., Zulfadli, M., Azizan, A. S. M., Che Husain, F., & Haridi, N. H. M. (2025). Ethical framework for artificial intelligence in professional practices of higher education lecturers: A systematic review. International Journal of Education, Psychology and Counseling, 10(57), 194–212.
  
  • Description: This systematic review examines ethical frameworks for the use of artificial intelligence in the professional practice of higher education lecturers, highlighting challenges and recommendations for ethical application.