A Meta-Review of Artificial Intelligence in Higher Education: Trends, Gaps, and Future Directions

Table of Contents

Overall Summary

Overview

This meta-review examines the growing field of Artificial Intelligence in Education (AIEd), specifically in higher education (AIHEd). Think of it like taking a bird's-eye view of all the existing summaries of AIEd research to understand the big picture. The review analyzes 66 summaries of AIEd research, published between 2018 and 2023, to identify key trends, research gaps, and areas for improvement. It's like creating a map of the AIEd landscape to guide future exploration.

Key Findings

Strengths

Areas for Improvement

Significant Elements

Table 8

Description: This table lists the top research gaps identified in AIEd, such as ethical implications, methodological limitations, and the need for more diverse research contexts.

Relevance: It provides a roadmap for future research, highlighting the most pressing issues that need to be addressed.

Figure 5

Description: This figure shows the trend of AIEd publications over time, indicating a growing interest in the field.

Relevance: It provides context for the meta-review, showing how AIEd research has evolved over the past few years.

Conclusion

This meta-review reveals a rapidly growing yet unevenly developed field of AIEd in higher education. While AI offers great potential for personalized learning and improved educational outcomes, think of it like a powerful new tool that can be used for good or bad. Addressing the identified challenges, particularly ethical concerns and methodological limitations, is crucial for realizing AI's full potential and ensuring its responsible use in higher education. It's like building a bridge to the future of education – we need strong foundations and careful planning to make sure it's safe and effective.

Section Analysis

Abstract

Overview

This abstract summarizes a meta-review of research on Artificial Intelligence in Education (AIEd), specifically in higher education (AIHEd). It highlights the rapid growth of AIEd and the importance of a strong research base. The review synthesized secondary research, primarily systematic reviews, to explore the scope and nature of AIEd research, identifying key themes, research gaps, and suggestions for future research.

Key Aspects

Strengths

Suggestions for Improvement

Introduction

Overview

This introduction sets the stage for a meta-review of research on Artificial Intelligence in Education (AIEd), specifically in higher education. It emphasizes the growing importance of AIEd, the need for a solid research foundation, and the timeliness of this review due to the rapid evolution of AI and increased public discourse.

Key Aspects

Strengths

Suggestions for Improvement

Method

Overview

This section details the methods used to conduct a tertiary review (a review of reviews) of AI in higher education. It describes the search strategy, study selection process, data extraction methods, quality assessment criteria, and data synthesis approach. The goal is to provide a transparent and replicable methodology for mapping the AIEd field.

Key Aspects

Strengths

Suggestions for Improvement

Non-Text Elements

figure 2

Figure 2 presents the search string used for the tertiary review. It's organized into three main search components combined with "AND": AI, Education Sector, and Evidence Synthesis. Each component lists specific keywords or phrases used in the search, separated by "OR". For example, the AI component includes terms like "artificial intelligence," "machine learning," "chat bot*", and various other related terms. The Education Sector component specifies educational levels and settings like "higher education," "college*", "K-12", and other related terms. The Evidence Synthesis component lists different types of review methodologies, such as "systematic review," "scoping review," "meta-analysis," and many others.

First Mention

Text: "A search string was developed (see Fig. 2) based on the search strings from the two previous reviews"

Context: The authors explain how they developed their search string for the review, mentioning that it was based on previous reviews and focuses on AI, education settings, and evidence synthesis methods.

Relevance: This figure is crucial as it provides transparency and replicability for the review process. It shows exactly how the authors searched for relevant literature, allowing others to understand and potentially reproduce the search.

Critique
Visual Aspects
  • The figure clearly presents the search string in a structured and readable format.
  • The use of 'AND' and 'OR' operators is clearly shown, making the search logic easy to follow.
  • The grouping of search terms into categories (AI, Education Sector, Evidence Synthesis) enhances clarity.
Analytical Aspects
  • The search string appears comprehensive, covering a wide range of relevant terms for AI, education levels, and review types.
  • The use of wildcards (e.g., 'bot*') is helpful for capturing variations of search terms.
  • The search string could be improved by adding specific terms related to higher education contexts, such as 'university' or 'tertiary education', to further refine the search.
Numeric Data
figure 3

Figure 3, a PRISMA flow diagram, visually represents the process of selecting studies for inclusion in the meta-review. It starts with the initial number of records identified through database searching and other sources. Then, it shows the number of records after duplicates were removed. The diagram then details the screening process, showing how many records were screened based on title and abstract, and how many were excluded at this stage with reasons for exclusion. It proceeds to full-text screening, again showing exclusions and reasons. Finally, it shows the number of studies included in the review.

First Mention

Text: "The search strategy yielded 5609 items (see Fig. 3), which were exported as .ris or .txt files and imported into the evidence synthesis software EPPI Reviewer"

Context: This quote describes the initial stage of the study selection process, where the search results are imported into EPPI Reviewer software for further processing.

Relevance: This figure is essential for understanding the scope and rigor of the review. It clearly shows how many studies were considered and why some were excluded, ensuring transparency and allowing readers to assess the review's comprehensiveness.

Critique
Visual Aspects
  • The diagram is clear and easy to follow, using standard PRISMA formatting.
  • The numbers of included and excluded studies at each stage are clearly displayed.
  • The reasons for exclusion are provided, which is helpful for understanding the selection process.
Analytical Aspects
  • The diagram effectively communicates the iterative nature of the review process.
  • The large number of excluded studies highlights the importance of a systematic approach to literature selection.
  • The diagram could be further improved by providing more detail on the specific criteria used for each screening stage.
Numeric Data
table 2

Table 2 outlines the criteria used to include or exclude studies from the meta-review. It's divided into two columns: 'Inclusion criteria' and 'Exclusion criteria'. The 'Inclusion criteria' column lists factors like the publication date range (January 2018 to July 18, 2023), the focus on AI applications in formal education settings, the type of publication (journal articles or conference papers), the use of secondary research with a method section, and the language (English). The 'Exclusion criteria' column lists factors like publications before January 2018, studies not about AI or not in formal education settings, specific publication types (editorials, book chapters, etc.), primary research or literature reviews without a method section, and non-English publications.

First Mention

Text: "following lengthy discussion and agreement on the inclusion and exclusion criteria by all authors, two members of the team (MB and PP) double screened the first 100 items"

Context: This section describes the process of ensuring inter-rater reliability during the screening process, emphasizing the importance of agreed-upon inclusion and exclusion criteria.

Relevance: This table is crucial for understanding the scope and focus of the review. It clearly defines which studies were eligible for inclusion and why, ensuring transparency and allowing readers to assess the review's relevance to their own interests.

Critique
Visual Aspects
  • The table is clearly organized with distinct columns for inclusion and exclusion criteria.
  • The criteria are presented in a concise and easy-to-understand manner.
  • The table could be improved by adding a brief explanation of the rationale behind each criterion.
Analytical Aspects
  • The criteria appear well-defined and relevant to the review's focus on AI in higher education.
  • The exclusion of certain publication types (e.g., editorials, book chapters) helps to focus the review on research-based evidence.
  • The inclusion criteria could be strengthened by specifying the types of AI applications or educational contexts of interest.
Numeric Data
figure 4

Figure 4 provides a table outlining the criteria used for assessing the quality of the included reviews. Each criterion is listed along with a scoring system (Yes = 1, Partly = 0.5, No = 0) and an interpretation of what each score represents. The criteria include aspects like the presence of research questions, the clarity of inclusion/exclusion criteria, the definition of publication years, the adequacy of the search strategy, the reporting of inter-rater reliability, and the provision of a data extraction coding scheme. It also assesses whether a quality assessment was conducted, if sufficient details about the included studies were provided, and if the review reflects on its limitations.

First Mention

Text: "To answer sub-question 1f about the quality of AIHEd secondary research, the decision was made to use the DARE tool (Centre for Reviews and Dissemination, 1995), which has been used in previous tertiary reviews (e.g., Kitchenham et al., 2009; Tran et al., 2021)."

Context: This section of the paper discusses the quality assessment methods employed in the meta-review. It explains the rationale for choosing the DARE tool and lists the criteria used for evaluating the quality of the included reviews. The criteria are presented in a table format in Figure 4.

Relevance: This figure is crucial as it makes the review process transparent and allows readers to understand how the quality of the included studies was judged. By outlining the specific criteria and their scoring, it provides a clear framework for evaluating the rigor and reliability of the synthesized evidence. This helps establish the trustworthiness of the meta-review's findings.

Critique
Visual Aspects
  • The table format is clear and easy to understand.
  • The use of simple language for the criteria and interpretations makes it accessible to a wider audience.
  • The scoring system is straightforward and easy to apply.
Analytical Aspects
  • The criteria cover important aspects of review quality, such as the clarity of research questions, the adequacy of the search strategy, and the reporting of inter-rater reliability.
  • The inclusion of criteria related to transparency, like the provision of a data extraction coding scheme, strengthens the rigor of the quality assessment.
  • The criteria align with established quality assessment tools like DARE and AMSTAR 2, enhancing the credibility of the evaluation.
Numeric Data
figure 6

Figure 6 is a bar chart showing the overall quality assessment of the 66 AIHEd reviews included in the meta-review. The chart categorizes the reviews into five quality levels: Critically Low, Low, Medium, High, and Excellent. The height of each bar represents the number of reviews that fall into each quality category.

First Mention

Text: "The reviews were given an overall quality assessment score out of 10 (see Fig. 6), averaging 6.57 across the corpus."

Context: This part of the paper discusses the overall quality of the AIHEd reviews included in the meta-review. It mentions that each review received a score out of 10 and that the average score was 6.57. Figure 6 visually represents the distribution of these quality scores.

Relevance: This figure is important because it provides a visual summary of the overall quality of the reviews included in the meta-review. It helps readers quickly grasp the distribution of quality levels and understand the general rigor of the synthesized evidence. This is essential for interpreting the findings and conclusions of the meta-review.

Critique
Visual Aspects
  • The bar chart format is effective for showing the distribution of quality levels.
  • Clear labels on the x and y axes make the chart easy to interpret.
  • The use of distinct colors for each bar enhances visual clarity.
Analytical Aspects
  • The categorization of reviews into five quality levels provides a nuanced view of the quality assessment.
  • The chart highlights the proportion of reviews that fall into each quality category, allowing for a quick assessment of the overall quality of the included studies.
  • The visual representation of the quality assessment complements the detailed criteria provided in Figure 4, offering a more comprehensive understanding of the review process.
Numeric Data
  • Critically Low: 2 reviews
  • Low: 8 reviews
  • Medium: 33 reviews
  • High: 17 reviews
  • Excellent: 6 reviews

Findings

Overview

This section presents the findings of the meta-review on AI in higher education. It covers the publication trends, types of reviews conducted, author demographics, quality assessment of the reviews, common AI applications, benefits and challenges, and identified research gaps.

Key Aspects

Strengths

Suggestions for Improvement

Non-Text Elements

figure 5

Figure 5 is a bar chart illustrating the number of AIEd evidence syntheses focused on higher education published each year from 2018 to 2023. Each bar represents a year, and its height corresponds to the number of publications. It shows a low number of publications in the initial years (2 in 2018, 10 in 2019), a dip in 2020 (6), a significant rise in 2021 and 2022 (16 and 20 respectively), and a slight decrease in 2023 (12).

First Mention

Text: "there was a slight reduction in the number published in 2020 before rising again (see Fig. 5)."

Context: The authors are discussing the general publication characteristics of the AIEd evidence syntheses included in their review. They note a decrease in publications in 2020, likely due to the COVID-19 pandemic, before the numbers rise again. Figure 5 visually represents this trend.

Relevance: This figure helps visualize the growth and trends in AIEd research publications specifically focused on higher education. It provides context for the review by showing the increasing interest in this area over recent years, while also acknowledging the impact of external factors like the pandemic.

Critique
Visual Aspects
  • The bar chart is clear and easy to understand, effectively showing the trend of publications over time.
  • The labels for the years and the number of publications are clear and easy to read.
  • The use of color enhances the visual appeal and readability of the chart.
Analytical Aspects
  • The figure clearly demonstrates the increasing interest in AIEd research in higher education.
  • The dip in publications in 2020 provides an interesting data point for further investigation and discussion.
  • The figure could be improved by adding a trend line or a cumulative count of publications to further emphasize the growth in this area.
Numeric Data
  • 2018: 2 publications
  • 2019: 10 publications
  • 2020: 6 publications
  • 2021: 16 publications
  • 2022: 20 publications
  • 2023: 12 publications
table 3

Table 3 shows the top nine most productive countries in terms of authorship in AIEd evidence syntheses focused on higher education. It lists the countries, their rank, the number of publications from each country, and the percentage of the total publications each country represents. The United States is the most productive, followed by Canada and Australia.

First Mention

Text: "Whilst it was the most productive country (see Table 3), the United States was closely followed by Canada and Australia."

Context: The authors are discussing the geographical distribution of AIEd evidence synthesis authorship. They mention that the US is the most productive country, but Canada and Australia are close behind. Table 3 provides the data supporting this statement.

Relevance: This table provides insights into the global distribution of research on AI in higher education. It shows which countries are leading in this area and can be used to identify potential collaborations or areas for future research development.

Critique
Visual Aspects
  • The table is clear and easy to read, with a simple and effective layout.
  • The inclusion of both count and percentage data provides a comprehensive view of each country's contribution.
  • The ranking helps to quickly identify the most productive countries.
Analytical Aspects
  • The table shows a relatively diverse range of countries contributing to AIEd research, which is a positive sign for the field.
  • The dominance of North America and Australia could be further explored to understand the factors contributing to their higher publication rates.
  • The table could be enhanced by adding information about the types of AI applications or research methods used in each country.
Numeric Data
  • United States: 11 publications
  • Canada: 9 publications
  • Australia: 7 publications
  • South Africa: 6 publications
  • China: 5 publications
  • Saudi Arabia: 4 publications
  • Spain: 4 publications
  • Germany: 3 publications
  • India: 3 publications
table 4

Table 4 presents a quality assessment of the 66 AIEd evidence syntheses included in the review. It lists ten criteria used to evaluate the quality of each review, along with the percentage of reviews that fully met (Yes), partially met (Partly), did not meet (No), or for which the criteria were not applicable (N/A). The criteria include having research questions, inclusion/exclusion criteria, defined publication years, an adequate search, a provided search string, reported inter-rater reliability, a data extraction coding scheme, a quality assessment, sufficient details about included studies, and a reflection on limitations.

First Mention

Text: "The AIHEd reviews in the corpus were assessed against 10 quality assessment criteria (see Table 4), based on the DARE (Centre for Reviews and Dissemination, 1995; Kitchenham et al., 2009) and AMSTAR 2 (Shea et al., 2017) tools, as well as the method by Buntins et al. (2023)."

Context: The authors are explaining how they assessed the quality of the AIHEd reviews included in their meta-review. They mention using a combination of criteria from the DARE and AMSTAR 2 tools, as well as a method by Buntins et al. (2023). Table 4 details these criteria and the results of the quality assessment.

Relevance: This table is crucial for understanding the rigor and reliability of the included reviews. It provides a transparent overview of the quality assessment process and allows readers to assess the trustworthiness of the meta-review's findings.

Critique
Visual Aspects
  • The table is well-organized and easy to read, with clear headings and labels.
  • The use of percentages allows for easy comparison across criteria.
  • The inclusion of 'N/A' acknowledges that some criteria may not be applicable to all review types.
Analytical Aspects
  • The criteria cover important aspects of review quality, such as the clarity of research questions, the adequacy of the search strategy, and the reporting of inter-rater reliability.
  • The table highlights areas where AIEd reviews could be improved, such as the reporting of inter-rater reliability and the provision of data extraction coding schemes.
  • The quality assessment provides valuable insights into the methodological rigor of the AIEd research landscape.
Numeric Data
  • Research questions, aims or objectives: 92.4 %
  • Inclusion/exclusion criteria reported and appropriate: 77.3 %
  • Publication years included defined: 87.9 %
  • Search adequately conducted and likely to have covered all relevant studies: 68.2 %
  • Search string provided in full: 68.2 %
  • Inter-rater reliability reported: 51.5 %
  • Data extraction coding scheme provided: 24.2 %
  • Quality assessment undertaken: 45.5 %
  • Sufficient details provided about the individual included studies: 65.2 %
  • Reflection on review limitations: 65.2 %
table 5

Table 5 shows the distribution of AI applications that were the primary focus of the 66 reviews analyzed. It categorizes the reviews based on their main AI focus: General AIEd (covering various AI applications), Profiling and Prediction, Adaptive Systems and Personalisation, Assessment and Evaluation, and Intelligent Tutoring Systems. For each category, it provides the number (n) and percentage of reviews that fell under that focus.

First Mention

Text: "The reviews were categorised using Zawacki-Richter et al.’s (2019) classification (profiling and prediction; intelligent tutoring systems; adaptive systems and personalisation; assessment and evaluation; see Fig. 1), depending upon their purported focus within the title, abstract, keywords or search terms, with any reviews not specifying a particular focus categorised as ‘General AIEd’ (see Table 5)."

Context: This introduces Table 5 and explains how the reviews were categorized based on their focus, using the classification by Zawacki-Richter et al. (2019).

Relevance: This table is important because it shows the main areas of focus within AIEd research in higher education. It helps to understand which AI applications are receiving the most attention in research and which areas might be under-researched.

Critique
Visual Aspects
  • The table is clear and easy to understand, with clear labels for each category.
  • The inclusion of both count and percentage for each category makes it easy to compare the relative focus on different AI applications.
  • The table could be visually enhanced by using color-coding or other visual cues to highlight the most prominent categories.
Analytical Aspects
  • The table provides a good overview of the distribution of AI applications in the reviewed studies.
  • The high percentage of reviews focusing on General AIEd suggests that many studies explore a broad range of AI applications rather than focusing on a specific one.
  • The relatively low percentages for Assessment and Evaluation and Intelligent Tutoring Systems suggest these areas might be under-researched compared to others.
Numeric Data
  • General AIEd: 31 reviews
  • Profiling and Prediction: 19 reviews
  • Adaptive Systems and Personalisation: 18 reviews
  • Assessment and Evaluation: 3 reviews
  • Intelligent Tutoring Systems: 1 reviews
table 6

Table 6 presents the top six reported benefits of using AI in higher education, based on the analysis of 31 reviews. It lists benefits like personalized learning, greater insight into student understanding, positive influence on learning outcomes, reduced planning and administration time for teachers, greater equity in education, and precise assessment & feedback. For each benefit, the table shows the number of reviews that mentioned it and the corresponding percentage.

First Mention

Text: "Twelve benefits were identified across the 31 reviews (see Additional file 12: Appendix L), with personalised learning the most prominent (see Table 6)."

Context: This introduces Table 6, highlighting that it shows the top benefits of AI in higher education identified across the 31 general AIEd reviews.

Relevance: This table is important because it summarizes the perceived advantages of using AI in higher education. It highlights the potential positive impacts of AI on various aspects of teaching, learning, and administration, which can inform decisions about AI adoption and implementation.

Critique
Visual Aspects
  • The table is clear and concise, presenting the benefits in a ranked order based on frequency.
  • The inclusion of both count and percentage for each benefit allows for easy comparison.
  • The table could be visually improved by using icons or other visual elements to represent each benefit.
Analytical Aspects
  • The table effectively highlights the most frequently mentioned benefits of AI in higher education.
  • The prominence of personalized learning as a benefit aligns with the focus on adaptive systems and personalization in Table 5.
  • The table could be strengthened by providing more context or specific examples of how each benefit is realized in practice.
Numeric Data
  • Personalized learning: 12 reviews
  • Greater insight into student understanding: 10 reviews
  • Positive influence on learning outcomes: 10 reviews
  • Reduced planning and administration time for teachers: 10 reviews
  • Greater equity in education: 7 reviews
  • Precise assessment & feedback: 7 reviews
table 7

Table 7 lists the top five challenges of implementing AI in higher education as identified across 31 reviews. These challenges include lack of ethical consideration, curriculum development needs, infrastructure limitations, lack of teacher technical knowledge, and shifting authority. The table provides the number and percentage of reviews that mentioned each challenge.

First Mention

Text: "The 31 reviews found 17 challenges, but these were mentioned in fewer studies than the benefits (see Additional file 12: Appendix L). Nine studies (see Table 7) reported a lack of ethical consideration, followed by curriculum development, infrastructure, lack of teacher technical knowledge, and shifting authority"

Context: This introduces Table 7 and explains that it presents the top five challenges identified in the 31 general AIEd reviews.

Relevance: This table is important because it highlights the key obstacles to successful AI implementation in higher education. Understanding these challenges is crucial for developing strategies to overcome them and effectively integrate AI into educational settings.

Critique
Visual Aspects
  • The table is clear and easy to understand, with concise labels for each challenge.
  • The inclusion of both count and percentage for each challenge facilitates comparison.
  • The table could be visually enhanced by using color-coding or other visual cues to emphasize the most significant challenges.
Analytical Aspects
  • The table effectively summarizes the most frequently mentioned challenges of AI implementation.
  • The prominence of ethical considerations as a challenge aligns with the research gaps identified in Table 8.
  • The table could be strengthened by providing more context or specific examples of how each challenge manifests in practice.
Numeric Data
  • Lack of ethical consideration: 9 reviews
  • Curriculum development: 7 reviews
  • Infrastructure: 7 reviews
  • Lack of teacher technical knowledge: 7 reviews
  • Shifting Authority: 7 reviews
table 8

Table 8 shows the top ten research gaps identified across the 66 studies included in the review. It lists each gap, the number of studies (n) that mentioned it, and the percentage (%) of the total studies that mentioned it. The gaps include ethical implications, the need for more diverse methodological approaches, more research within the field of Education, research with a wider range of stakeholders, interdisciplinary approaches, research beyond specific disciplines, research in a wider range of countries (especially developing countries), stronger theoretical foundations, longitudinal studies, and research beyond a few limited topics.

First Mention

Text: "Each review in this corpus (n = 66) was searched for any research gaps that had been identified within the primary studies, which were then coded inductively (see Additional file 1: Appendix A)."

Context: This explains that the research gaps were identified from the included studies and coded inductively. Appendix A is referenced for a full list.

Relevance: This table is highly relevant as it summarizes the main areas where future research is needed in AIHEd, according to the synthesized reviews. It provides a clear direction for future research efforts and highlights the current limitations of the field.

Critique
Visual Aspects
  • The table is clear and easy to understand, with clear headings and a simple structure.
  • The use of both raw counts (n) and percentages (%) helps to understand the prevalence of each gap.
  • The table could be visually improved by ordering the rows by percentage or count, to highlight the most prominent gaps.
Analytical Aspects
  • The identified gaps are relevant and important for the future development of AIHEd.
  • The gaps cover a range of issues, from ethical considerations to methodological limitations and the need for more diverse research contexts.
  • The table could be strengthened by providing more specific examples of research questions or topics within each gap area.
Numeric Data
  • Ethical implications: 27 studies
  • More methodological approaches needed: 24 studies
  • More research in Education needed: 22 studies
  • More research with a wider range of stakeholders: 14 studies
  • Interdisciplinary approaches required: 11 studies
  • Research limited to specific discipline areas: 11 studies
  • More research in a wider range of countries, esp. developing: 10 studies
  • Greater emphasis on theoretical foundations needed: 9 studies
  • Longitudinal studies recommended: 8 studies
  • Research limited to a few topics: 8 studies

Discussion

Overview

This discussion section summarizes the key findings of the meta-review on AI in higher education, highlighting the prevalence of adaptive systems and personalization, along with profiling and prediction. It emphasizes the need for increased ethics, collaboration, and rigor in future AIHEd research and practice. The discussion also addresses the global distribution of AIHEd research and the importance of open access publishing.

Key Aspects

Strengths

Suggestions for Improvement

Conclusion

Overview

This conclusion summarizes the meta-review's findings, emphasizing the dominance of adaptive systems and personalization in AIHEd research. It reiterates the need for increased ethics, collaboration, and rigor in the field, while also highlighting the global distribution of research and advocating for open access publishing.

Key Aspects

Strengths

Suggestions for Improvement

↑ Back to Top