learning literature research

• Open access
• Published: 23 October 2020

A systematic literature review of personalized learning terms

• Atikah Shemshack   ORCID: orcid.org/0000-0003-4964-6171 1 &
• Jonathan Michael Spector 1  

Smart Learning Environments volume  7 , Article number:  33 ( 2020 ) Cite this article

54k Accesses

101 Citations

4 Altmetric

Metrics details

Learning is a natural human activity that is shaped by personal experiences, cognitive awareness, personal bias, opinions, cultural background, and environment. Learning has been defined as a stable and persistent change in what a person knows and can do. Learning is formed through an individual’s interactions, including the conveyance of knowledge and skills from others and experiences. So, learning is a personalized experience that allows one to expand their knowledge, perspective, skills, and understanding. Therefore, personalized learning models can help to meet individual needs and goals. Furthermore, to personalize the learning experience, technology integration can play a crucial role. This paper provides a review of the recent research literature on personalized learning as technology is changing how learning can be effectively personalized. The emphasis is on the terms used to characterize learning as those can suggest a framework for personalized and will eventually be used in meta-analyses of research on personalized learning, which is beyond the scope of this paper.

Introduction

Personalized learning has been a topic of research for a long time. However, around 2008, personalized learning started to draw more attention and take on a transformed meaning as seen in Fig.  1 . However, we believe the variety of terms that have been used for personalized learning seems to be an obstacle to the progress of personalized learning theories and research. Although there exists an abundance of the resources/studies on personalized learning, not having a readily agreed-upon term of personalized learning might be the obstacle in research progress on personalized learning. In response to this need, this paper is focused on analyzing the terms that have been used for personalized learning. A distinctly personalized learning approach can help the educational researchers to build up research on previous data, instead of trying to start new research from scratch each time. This paper will present a research-based framework for personalized learning and discuss future research directions, issues, and challenges through an in-depth analysis of the definitions and terms used for personalized learning.

The number of published papers on “personalized learning ”

Personalized learning has existed for hundreds of years in the form of apprenticeship and mentoring. As educational technologies began to mature in the last half of the previous century, personalized learning took the form of intelligent tutoring systems. In this century, big data and learning analytics are poised to transform personalized learning once again. Learning has been characterized as a stable and persistent change in what a person knows and can do (Spector, 2015 ). Personalized learning is a complex activity approach that is the product of self-organization (Chatti, 2010 ; Miliband, 2006 ) or learning and customized instruction that considers individual needs and goals. Personalized learning can be an efficient approach that can increase motivation, engagement and understanding (Pontual Falcão, e Peres, Sales de Morais and da Silva Oliveira, 2018 ), maximizing learner satisfaction, learning efficiency, and learning effectiveness (Gómez, Zervas, Sampson and Fabregat, 2014 ). However, while such personalized learning is now possible, it remains as one of the biggest challenges in modern educational systems. In this paper a review of progress in personalized learning using current technologies is provided. The emphasis is on the characteristics of personalized learning that need to be taken into consideration to have a well-developed concept of personalized learning.

We started with the definition of personalized learning suggested by Spector ( 2014 , 2018 ) and others that are discussed below, which requires a digital learning environment to be classified as a personalized learning environment to be adaptive to individual knowledge, experience and interests and to be effective and efficient in supporting and promoting desired learning outcomes. These characteristics are those which are typically discussed in the research community although we found it challenging to find a sufficient number of published cases that reported effect sizes and details of the sample in order to conduct a formal meta-analysis. Lacking those cases suggests that personalized learning in the digital era is still in its infancy. As a result, we conducted a more informal albeit systematic review of published research on personalized learning.

Furthermore, we, along with many educational technologists, believe an efficient personalized learning approach can increase learners’ motivation and engagement in learning activities so that improved learning results. While that outcome now seems achievable, it remains a largely unrealized opportunity according to this research review. Truong ( 2016 ) stated that providing the same content to students with different qualifications and personal traits and having different interests and needs is not considered adequate anymore when learning can now be personalized. Miliband ( 2006 , as cited in Lee, Huh, Lin and Reigeluth, 2018 ) promoted personalized learning to be the solution to tailoring the learning according to individuals’ needs and prior experience so as to allow everyone to reach their maximum potential through customized instruction (Hsieh and Chen, 2016 ; Lin, Yeh, Hung and Chang, 2013 ).The customized instruction that includes what is taught, how it is taught, and the pace at which it is taught. This allows learning to meet individual needs, interests and circumstances which can be quite diverse (Brusilovsky and Peylo, 2003 ; Liu and Yu, 2011 ). Furthermore, FitzGerald et al. ( 2018 ) pointed out the personalization of learning is now a recurring trend across government agencies, popular media, conferences, research papers, and technological innovations.

Personalized learning is in demand (Huang, Liang, Su and Chen, 2012 ) due to new technologies involving big data and learning analytics. It should be tailored to and continuously modified to an individual learner’s conditions, abilities, preferences, background knowledge, interests, and goals and adaptable to the learner’s evolving skills and knowledge (Sampson, Karagiannidis and Kinshuk, 2002 ; Sharples, 2000 ). Today’s personalized learning theories are inspired by educational philosophy from the progressive era in the previous century, especially John Dewey’s ( 1915 , 1998 ) emphasis on experiential, learner-centered learning, social learning, extension of the curriculum, and fitting for a changing world. McCombs and Whisler ( 1997 ; as cited in Lee et al., 2018 ) claimed that a learner-centered environment develops as it considers learners’ unique characteristics using the best knowledge of teaching and learning which are available. Furthermore, Lockspeiser and Kaul ( 2016 ) claimed that individualized learning is a tool to facilitate learner-centered education. FitzGerald et al. ( 2018 ) pointed out that personalization is a crucial topic of current interest in technology-oriented learning design and discussion for government policymakers, but less so in educational research. This might be a good explanation of disunity of personalized learning approaches.

On the other hand, Niknam and Thulasiraman ( 2020 ) argued that educational society has been interested in having a personalized learning system that adjusts the pedagogy, curriculum, and learning environment for learners to meet their learning needs and preferences. A personalized learning system can adapt itself when providing learning support to different learners to defeat the weakness of one-size-fits-all approaches in technology-enabled learning systems. The goal is to have a learning system that can dynamically adapt itself based on a learner’s characteristics and needs to provide personalized learning. Human one-on-one tutors can do this and now it is possible for digital systems to do so as well. Schmid and Petko ( 2019 ) pointed out that a look at international research literature shows that personalized learning is a multilayered construct with numerous definitions and various forms of implementation. Which supports our claim that one of the most critical problems with personalized learning is, there is no readily agreed-upon meaning of the phrase ‘personalized learning’. Schmid and Petko ( 2019 ) supported this claim by stating that a clearly defined concept of personalized learning is still lacking; instead, it serves as an umbrella term for educational strategies that try to do justice to the individual’s abilities, knowledge, and learning needs of each student. Spector ( 2013 ) claimed that there would be more robust information to support personalized learning as technology develops. So many different terms have been used in the replacement of ‘personalized learning’. Researchers could not locate a systematic literature review on personalized learning terms that review the terms that have been used for personalized learning, and it is important to address this need. Therefore, this review was done to close that gap and respond to the need for a unified, personalized learning term. As a result, personalized learning definitions and the terms that have been used interchangeably, such as adaptive learning, individualized instruction, and customized learning are analyzed in this paper. These terms were chosen because they have been most used in the education field (Reisman, 2014 ). In the next several sections, each term will be defined, and their relationship with personalized learning will be discussed. The analysis of these terms guided the systematic review of the research literature that follows.

• Adaptive learning

Most educators recognize the advantages of adaptive learning, but evidence-based research stays limited as adaptive learning is still evolving (Liu, McKelroy, Corliss and Carrigan, 2017 ). Adaptive learning is one of the terms that has been used interchangeably with personalized learning. The adaptive learning system is built on principles that have been around for a very long time dating back to the era of apprenticeship training and human tutoring. However, many other labels such as individualized instruction, self-paced instruction, and personalized instruction were used interchangeably while trying to produce the most suitable sequence of learning units for each learner (Garcia-Cabot, De-Marcos and Garcia-Lopez, 2015 ; Reisman, 2014 ). While early forms of adaptive learning (e.g., apprenticeship training and human tutoring) only dealt with one or a very small number of learners, the current interest is using adaptive learning for large numbers of learners, which is why there is such interest in big data and learning analytics.

For instance, adaptive learning has been interchangeably used by Yang, Hwang and Yang ( 2013 ) in their study that focused on the development of adaptive learning by considering students’ preferences (Dwivedi and Bharadwaj, 2013 ) and characteristics, including learning styles (Çakıroğlu, 2014 ; Klašnja-Milićević, Vesin, Ivanović and Budimac, 2011 ) and cognitive styles (Lo, Chan and Yeh, 2012 ) which concluded to be effective. Wang and Liao ( 2011 ) defined adaptive learning as a developed system (Lu, Chang, Kinshuk, Huang and Chen, 2014 ) to accommodate a variety of individual differences (Scheiter et al., 2019 ; Wang & Liao, 2011 ) such as gender, learning motivation, cognitive type, and learning style to determine optimal adaptive learning experience that accommodates a variety of individual differences (Afini Normadhi et al., 2019 ) to remove barriers of time and location. Griff and Matter ( 2013 ) discussed that adaptive learning is also referred to as computer-based learning, adaptive educational hypermedia, and intelligent tutoring. Furthermore, Hooshyar, Ahmad, Yousefi, Yusop and Horng ( 2015 ) used personalized and adaptive learning to explain the importance of the Intelligent Tutoring System (Aeiad and Meziane, 2019 ) for implementing one-to-one personalized and adaptive teaching. “Although the terms ‘personalized learning’ and ‘adaptive learning’ are different, they are often used interchangeably in various studies” (Aroyo et al., 2006 ; Göbel et al., 2010 ; Gómez et al., 2014 ; Lin et al., 2013 , as cited in Xie, Chu, Hwang and Wang, 2019 , p.2).

Based on this review, adaptive learning systems are defined as those that are computerized learning systems that adapt learning content, presentation styles, or learning paths based on individual students’ profiles, learning status, or human factors (Chen, Liu and Chang, 2006 ; Tseng et al., 2008 ; Yang et al., 2013 ).

Individualized instruction

Individualized instruction is one of the terms that are often used to talk about the specific needs and goals of individuals to be addressed during instruction. U.S. Department of Education ( 2010 ) defined personalized learning as involving customizing the learning pace to individual learners (individualization), tailoring instructional methods (differentiation), and personalizing learning content. This notion has evolved from one-on-one human tutoring. It is not agreed upon whether individualization is a component of personalized learning or another term that can be used in place of personalized learning. The review results show that instead of being a component, individualized instruction has been used as a replacement term for personalized learning and is a product of personalized learning. Chatti, Jarke and Specht ( 2010 ) and Chou, Lai, Chao, Lan and Chen ( 2015 ) had used both terms without defining/explaining how they relate to each other . Bahçeci and Gürol ( 2016 ) created a portal that offers individualized learning content based on the individual’s level of cognitive knowledge. Bahçeci and Gürol ( 2016 ) stated that education should be done by recognizing the individual differences of the students such as students learning styles (Çakıroğlu, 2014 ; Klašnja-Milićević et al., 2011 ) and characteristics. The researchers observed that Bahçeci and Gürol ( 2016 ) used individualized learning and personalized learning interchangeably without pointing out that they were doing so.

Also, most individualized learning studies have used individualized instruction to refer to IEP (individualized educational plans) for students with disabilities to accommodate their needs and goals. Even though individualized instruction is suggested as an approach that individualize material to improve the learning experience for students with learning disabilities, it can benefit all students (Barrio et al., 2017 ; Ko, Chiang, Lin and Chen, 2011 ). Personalized learning considers students’ interests, needs, readiness, and motivation and adapts to their progress by situating the learner at the center of the learning process. Individualized learning allows for individualization of learning based on the learner’s unique needs (Cavanagh, 2014 ; Lockspeiser & Kaul, 2016 ). While a learner-centered paradigm of education has influenced personalized learning, the current teacher-student ratios in school systems seem to be an obstacle to make learning experiences personalized for individual students without technology (Lee et al., 2018 ), with the exception of the requirement for IEPs in many school districts. We follow the definition offered by the U.S. Department of Education and note that individualized learning in school systems requires significant technology support, such as big data and learning analytics.

Customized learning

While Lee et al. ( 2018 ) suggested a learner-centered system that supports diverse needs and development of individual learners’ potentials. This system develops customized instructional methods and learning content for individual learners with unique characteristics and interests. Lee et al. ( 2018 ) suggested that learner-centered learning and personalized learning are blended and considered together. Lee et al. ( 2018 ) defined a personalized learning plan (PLP) that refers to a customized instructional plan (Somyürek, 2015 ) that considers individual differences and needs, characteristics, interests, and academic mastery. The PLP includes the notions of individualization, differentiation, and personalization that allows learning to be personally relevant, engaging, appropriate to the learners’ capabilities, and respectful of individual differences, making learning useful and motivational.

The review of those three terms reveals a great deal of overlap with an emphasis on the need to use technology to support such efforts. This study reviews definitions of personalized learning terms used in research papers from 2010 to 2020 by systematically reviewing the literature to compare the similarities and differences in definitions of each of these terms. The hope is to synthesize the terms used for personalized learning so the researchers can analyze and go through the research in the field and conduct meta-analyses and syntheses of the research literature. Also, analyzing the definitions of the term ‘personalized learning’, ‘adaptive learning’, ‘individualized instruction’, and ‘customized learning’ that have been used can help to develop a unified definition for personalized learning that can lead a framework. The framework can help with having a common understanding of personalized learning rather than a collection of loosely defined systems. A unified description of personalized learning and analyzing the studies related to personalized learning can help consolidate findings and suggest new areas to explore.

Our idea of personalized learning rests on the foundation that humans learn through experience and by constructing knowledge. Constructivism claims that learners’ acquired knowledge and understanding, determine learning ability and that knowledge acquisition is a process of construction according to individuals’ experience (Ormrod, 2011 ). Personalized learning is influenced by a learner’s prior experiences, backgrounds, interests, needs, goals, and motivation. Moreover, it is accomplished via meaningful interactions in individual learners’ lives. Furthermore, no conscious effort is needed to be actively learning while engaged in everyday life (Kinshuk, 2012 ) although reflection and mega-cognition can promote learning.

Adaptive instruction, blended instruction, differentiation, customized instruction, individualized learning, adaptive learning, proactive supports, real-world connections, and applications are hallmarks of good personalized learning. In general, personalized-learning models seek to adapt to the pace of learning and the instructional strategies, content and activities being used to fit best each learner’s strengths, weaknesses, and interests. Personalized learning is about giving students some control over their learning (Benhamdi, Babouri and Chiky, 2017 ; Jung, Kim, Yoon, Park and Oakley, 2019 ; Tomberg, Laanpere, Ley and Normak, 2013 ), differentiating instruction for each learner, and providing real-time individualized feedback to teachers and learners (Nedungadi and Raman, 2012 ), which is all effortlessly blended throughout the learning activity. Putting a framework together can help with a practical personalized learning model for all. The model can be developed and evolved as technology develops and we learn more about human learning and machine learning.

Research methodology

For this review, the guidelines published by Okoli to conducting a systematic literature review for Information Systems Research were adapted (Okoli, 2015 ). Okoli’s work provides a detailed framework for writing a systematic literature review with its roots in information technology. As this systematic literature review is rooted in information technology, it was deemed appropriate to use Okoli’s work as the basis for this body of work.

Okoli presented eight significant steps that need to be followed to conduct a scientifically, rigorous systematic literature review. These steps are listed below:

Identify the purpose: The researchers identified the purpose and intended goals of the study to ensure the review is clear to readers.

Draft protocol and train the team: Reviewers agreed on procedures to follow to ensure consistency in how they complete the review.

Apply practical inclusion screen: Reviewers were specific about what studies they considered for review and which ones they eliminated without further examination. The reviewers created four phases to review papers to produce the final papers to review.

Search for literature: Reviewers described the literature search details and justified how they ensured the search’s comprehensiveness.

Extract data: After reviewers identified all the studies to be included in the review, they systematically extract the applicable information from each study by going through four review phases they explained in search query.

Appraise quality: The reviewers explicitly listed the criteria used to decide which papers they will exclude for insufficient quality in the search query. Researchers reviewed all papers and decided on final papers after explicit four search phases. They finalized the papers to be reviewed, depending on the content of the papers’ content and quality.

Synthesize studies: The researchers analyzed the data obtained from the studies using appropriate qualitative techniques.

Write the review: The process of a systematic literature review was explicitly described in adequate detail that other researchers can independently reproduce the review’s results.

Research question

This literature review promotes research around personalized learning in informational education. To fulfill answer of “What are the similarities and differences of different terms used for personalized learning approaches?” we need a research base and theoretical framework that provides answers to basic questions. Furthermore, the following questions are sub-questions to be considered during the study.

How is personalized learning defined?

How adaptive learning has been used and how it relates to personalized learning?

How individualized instruction has been used and how it relates to personalized learning?

How is customized learning connected to personalized learning?

What components need to be included in a well-defined personalized learning term?

Also, researchers are seeking a unified definition of personalized learning that will include all those different components. That is the focus of this literature review was conducted.

Sources of literature

To answer the research question, the researchers have selected the following well known and reputable databases to base this literature review: Scopus, Science Direct, EBSCOhost, IEEE Xplore, JSTOR, and Web of Science to ensure all related journals of the field are included. The most relevant journals for the systematic review were chosen consistently from these databases. Also, Google Scholar h5-index for the category “Educational technology” was used as the starting point since this category is a specific category for personalized learning studies.

Databases in which to base this literature review are listed in Table  1 .

The top nine journals from the “Educational Technology” category from google scholar h5-index selected to keep the range of the papers manageable while trying to ensure the review is broad enough to include enough studies that can satisfactorily answer the research question. Later, most of the journals about educational technology were indexed. SJR (SCIMAGO JOURNAL RANK) was used to validate the impact of the selected journals. Even though the impact factor is not perfectly aligned with Google Scholar’s h5-index order, the selected journals listed the most impactful journals in the educational technology field. Also, even though Journal of Learning Analytics was listed on google scholar and showed having a high impact on education technology, researchers have not located any qualified paper according to selection procedures, thus this journal was eliminated from review.

This review solely retrieved peer-reviewed article papers from online journals because those online academic journals are known to be reliable and authoritative. They allow the readers to verify the facts from their sources, which increases the reliability of enriched studies filled with data and facts. They enable the readers to perform comprehensive research and allow the reader to access more data without the limitations of space and time. A defined method was set in this research for selecting journals, to keep the process methodologically reliable and scientifically consistent. The researchers review the main databases for educational technology to ensure all related journals of the field are included. This review is only focused on journals to keep the scope of the review manageable and provide reviewed data to create a resource for future studies.

Journals in which to base this literature review are listed in Table  2 .

Supplementary procedures

Relevant papers were initially identified through traditional searches of online databases and journals. These papers were subsequently analyzed to determine their applicability to the study.

Search query

An appropriate search query was formulated that would find relevant personalized learning papers. The search query was as follows: ( Publication Title : (“journal name”)) AND (“term”) and the journals listed in the table were searched for each of following terms: “personalized learning”, “adaptive learning”, “individualized instruction”, and “customized learning.”

Inclusion/exclusion criteria

Four phases were determined to meet the paper’s inclusion criteria in the final set to be reviewed. First phase was initial search, searching each term ‘ personalized learning,’ ‘adaptive learning,’ ‘individualized instruction,’ ‘customized learning ,’ filtered years to 2010–2020 to review personalized learning papers which has been a hot topic for the research and policymakers. The language was filtered to English only to not wait on translation, the paper addressed technology integration , and type of the paper research articles that published in one of the peer-reviewed scientific journals listed to keep the scope manageable. The second search phase was eliminating by title, reviewing the abstract and keywords; researchers went through titles, abstracts, and keywords of each result of the initial search and included the ones look related to the term.

The next search phase, reading the abstract of each paper of the second search result-set, looking for a definition to see if it mentions the definition and/or terms that have been used for the term and the paper was available at one of the free online databases or the researchers’ university library. The fourth step was to download all those papers to Mendeley (indexing database) and index them under sub-folders for each journal database. Then the entire paper was read to determine if the paper was to be included in the literature review by looking for components and definitions of personalized learning and star the ones to be included in the review. Each paper that met the inclusion criteria was read in its entirety a second time to validate the paper’s decision in the final data set.

An initial search on google scholar on ‘ personalized learning’ shows that the number of published papers on personalized learning has progressively increased year by year; especially there is a jump in 2008 as seen in Fig. 1 . The date range of 2010 to the present day was chosen as this when personalized learning term started to gain more attention to research due to technology usage increase in education. The first smartphone was released in June 2007, which might be an element of the increase due to flexibility and access it provides. Cheung and Hew ( 2009 ) claimed that handheld devices are increasingly being used in educational settings. Primarily, papers published after the 2000s are focused on more technology-enhanced personalized learning. Figure 1 shows the results of the initial google scholar search on “personalized learning” published papers (Fig. 1 ).

Nine journals were determined as the source of papers to be reviewed for this study. Each journal was searched for “personalized learning,” “adaptive learning,” “individualized instruction,” “customized learning,” and each result gone through the inclusion criteria and final phase; papers were saved in Mendeley under subfolders for each journal. Table 3 are search results for each phase by journals.

The title, abstract, and when necessary, the full paper was reviewed to decide if the paper met the inclusion criteria. This process helped to finalize the papers that will be used for this study, and the result set for “personalized learning” and the result set for each term to be reviewed is shown in Table 3 . Some of the papers that did not fit the inclusion criteria are referenced in this paper as they provide valuable information about personalized learning. We reviewed 978 papers, and 4 phases of inclusion ended up with 56 relevant, high-quality papers. The 56 papers identified are marked in the references section with an asterisk. The systematic review methodology was used, and our literature search resulted in 56 relevant studies meeting the inclusion criteria. As shown in Table  4 , 56 papers met the minimum quality criteria and were examined in detail; 33 of them use personalized learning, 17 adaptive learning, three individualized instruction, and three customized learning as the main term in the paper.

Our findings revealed that although so many terms are used in education settings, by policymakers and cooperate settings, in the research field, the terms used for personalized learning are unified, and mostly personalized learning and/or adaptive learning is being used. For example, Chatti et al. ( 2010 ) and Peng, Ma and Spector ( 2019 ) are the ones who put the two most common terms used for personalized learning together and started to use “personalized adaptive learning,” which might be a good lead for future studies. However, future research needs to focus on components included in the personalized adaptive learning term’s definition, and components are included in it. Chatti et al. ( 2010 ) and Peng et al. ( 2019 ) s paper put all together very well, and Peng et al. ( 2019 ) called it a personalized adaptive smart learning environment. Future studies can focus on what components are being used for each personalized learning approach and, at the same time, acknowledge it is a term that will evolve by time as we learn more about human learning and as technology develop. Table  4 shows the results of the searches for each term by journals.

Existing and emerging trends

Miliband ( 2006 , as cited in Schmid & Petko, 2019 ) pointed out that the Organisation for Economic Co-operation and Development OECD ( 2006 ) was among the first to use personalized learning term and described personalized learning in the report “Schooling for Tomorrow– Personalising Education” as a critical trend. According to this educational policy report, personalized learning is characterized by changes concerning five dimensions: assessment for learning by giving students individual feedback and setting suitable learning objectives, teaching and learning strategies based on the individual needs, curriculum choices (Tomberg et al., 2013 ), student-centered approach to school organization, and strong partnerships beyond the school.

According to the United States National Education Technology Plan 2017 , personalized learning is defined as “instruction in which the pace of learning and the instructional approach are optimized for each learner’s needs. Learning objectives, instructional strategies, and instructional content (Shute and Rahimi, 2017 ) may differ depending on learner needs. Besides, learning activities are meaningful and relevant to learners, driven by their interests, and often self-initiated.” (p. 9).

American Psychological Association Presidential Task Force on Psychology in Education (1993, as cited in Lee et al., 2018 ) explained that a personalized learning plan (PLP) refers to a customized instructional plan that considers individual differences or needs such as career goals, characteristics, interests, and academic mastery. This includes the notions of individualization, differentiation, and personalization. Preparing and implementing PLPs allows for adjusting the pace to individual learners, adjusting instructional methods to individual characteristics, and having different learning goals tailored to individual interests. Furthermore, Sungkur, Antoaroo and Beeharry ( 2016 ) suggested an eye-tracking system to determine the user’s interest and behavior. The PLPs allow learning to be personally relevant, engaging, appropriate to the learners’ capabilities, and respectful of individual differences, making learning useful and motivational.

Learning analytics seems to grow to ensure the process of personalizing the content which allows mechanisms to identify student characteristics and associate them with a learning pattern (Ramos de Melo et al., 2014 ). Also, the ability to reactively organize personalized content may be a favorable factor in promoting the study support in virtual learning environments, respecting students’ different individualities, preferences (Erümit and Çetin, 2020 ) and difficulty factors.

There is a research gap in an adaptive learning environment that needs to focus on emotions and personality which play a significant role in parts of adaptive systems, such as feedback (Fatahi, 2019 ). Furthermore, Junokas, Lindgren, Kang and Morphew ( 2018 ) created a system based on multimodal educational environments that integrate gesture-recognition systems and found that it is effective in improving the learning experience.

The personalization of learning has been achieved using various methods that have been made available by the rapid development of information communication technology (ICT) (Dawson, Heathcote and Poole, 2010 ). Furthermore, Ramos de Melo et al. ( 2014 ) stated that personalization is customizing the content that allows present parts of the content as needed by the student. That is one of the most common themes among most of the personalized learning approaches which can be done by using adaptive learning systems that can present personalized content for individual students (Hwang, Sung, Hung and Huang, 2013 ).

The higher-order thinking skills and communication had attracted little attention in terms of both learning outcomes and the process of adaptive/personalized learning due to the difficulty of measurement and the limited learning support types. Furthermore, virtual reality techniques might be a solution to this need. Developing learning approaches that build on students’ current ability and support efficacy beliefs by allowing autonomy with a proper challenge to promote academic attainment (Foshee, Elliott and Atkinson, 2016 ; Xie et al., 2019 ). Future studies can focus on higher-order thinking skills cultivation by supporting these skills through personalized learning environments.

The idea of personalized learning rests on the foundation that humans learn through experience and by constructing knowledge. It is heavily influenced by a learner’s prior experiences and is accomplished via language and social interaction. Personalized learning is not the only way to think about teaching and learning. Moreover, learning will and should take many different forms. Proper instruction, blended instruction, differentiation, proactive supports, real-world connections, and applications are hallmarks of good, sound personalized learning. In general, personalized-learning models seek to adapt to the pace of learning and the instructional strategies, content and activities being used to fit best each learner’s strengths, weaknesses, and interests. Personalized learning is about giving students control over their learning, differentiating instruction for each child, and providing real-time feedback. Putting a framework together can help with practical personalized learning for all and can be developed as it faces challenges. The framework can help with having a structured common-sense personalized learning instead of a learning system that is being interpreted differently. In conjunction with a well-designed curriculum, instructional practice plays a crucial role in how children learn.

Most of the current personalized learning models/ideas are built on technology integration. For example, while Chen, Lee and Chen ( 2005 ) proposed a personalized system that provides learning paths (Nabizadeh, Gonçalves, Gama, Jorge and Rafsanjani, 2020 ) that can be adapted to various levels of difficulty of course materials (Zou and Xie, 2018 ) and various abilities of learners (p. 239). Klašnja-Milićević et al. ( 2011 ) stated that personalized learning occurs when e-learning systems make deliberate efforts to design educational experiences (Flores, Ari, Inan and Arslan-Ari, 2012 ) that fit the needs, goals, talents, motivations, and interests of their learners (p. 885). The term of needs is not specified to clarify what needs of the learner need to be considered for robust personalized learning. Considering the needs of the learner is one of the most common components used in personalized learning. However, only a few studies clarify what needs are mentioned to be considered, such as emotional needs, social needs, learning needs, knowledge needs, etc. Even if we agree on a unified definition with each component commonly agreed on, we need to ensure that each component is well defined.

In the past decades, many methods and systems have been proposed to accommodate students’ needs by proposing learning environments that consider personal factors. Learning styles (Çakıroğlu, 2014 ; Klašnja-Milićević et al., 2011 ; Latham, Crockett and McLean, 2014 ) have been among the broadly chosen components in previous studies as a reference for adapting learning. For example, George and Lal ( 2019 ) argued that personalized learning is meant to incorporate a learner’s varied attributes, including learning style, knowledge level on a subject, preferences, and learner’s prior knowledge while they discussed adaptive learning is adapting content according to learner’s choice and pace. Chen, Huang, Shih and Chang ( 2016 ) brought up the gender component to personalized learning. Furthermore, Atkinson ( 2006 ) found that there was a significant difference in learning achievement between male and female students, and among students who used different learning styles (Çakıroğlu, 2014 ; Klašnja-Milićević et al., 2011 ; Latham et al., 2014 ).

Our findings revealed that individualized instruction mostly focuses on special education students or students are limited in way compared to their peers. These students have IEPs (individualized educational plans) mandated by the state to be followed to ensure the schools are accommodating these students’ needs. One goal could be to create IEPs for all learners.

Moreover, it seems in education industry terms are quite varied, but when it comes to academia, it is mostly adaptive learning and personalized learning being used interchangeably Rastegarmoghadam and Ziarati ( 2017 ); however, mostly adaptive learning is being used when it is technology-enhanced learning. Adaptivity is typically referring to content being adjusted according to prior knowledge (Huang and Shiu, 2012 ), while personalized learning is being used for more broad adjustments according to different needs, interests, and goals of individuals.

Another finding is that adaptive learning is the most used term follows personalized learning. Individualized learning and customized learning, even though they are being used by cooperative, they are not commonly used in research. As shown in Table  4 , we have found 56 papers met the minimum quality criteria and were examined in detail; 33 of them use personalized learning, 17 adaptive learning, three individualized instruction, and three customized learning.

However, it seems that also the lack of a commonly identified personalized learning approach is an obstacle. This might be due to the nature of technology involvement; due to the rapid development increase in technology that makes personalized learning an evolving approach. That is fine if we all can agree that it should evolve as technology improves, and we learn more about humans and how human-machine interaction can improve the learning process.

Besides, another obstacle is the researchers and policymakers should show the same interest to personalized learning so the demand and research can align. Educators fear that machines will take over the teaching job if they allow technology to be used for teaching. Kinshuk, Huang, Sampson and Chen ( 2013 ) argued that the benefits of technology in education caught the interest of researchers, governments, and funding agencies. Computer systems were funded to help students in the learning process, consequently decreasing teachers’ workload. As a result, educational technology research was able to study advanced issues such as intelligent tutoring, simulations, advanced learning management systems, automatic assessment systems, and adaptive systems. Some educators believe that since technology involves big budgets, the interest of policymakers is not due to the interest of improving learning experience (Troussas, Krouska and Sgouropoulou, 2020 ); their interest is due to the monetary benefit they gain from increased use of technology in education. In addition, Kinshuk et al. ( 2013 ) pointed out that practitioners in education could not take advantage of all that research at an equally fast pace, and the implementation lagged severely behind. Researchers need to keep up with the demand of personalized learning. The alignment will help to ensure the practices policy makers discuss are research based efficient approaches that will increase efficiency of learning/teaching.

The progress of the research in personalized learning shows that by technological improvement, personalized learning becomes more embedded with technology and taking advantage of the benefits technology can offer. Some of these advantages are gathering data of learners’ emotions by using bio-trackers, which might bring up some privacy concerns.

Limitations

This study encountered several shortcomings during the review and in its attempt to answer all the research questions. The enormous number of published papers might lead to some missing relevant papers; numerous literature review studies face this problem. Furthermore, the immense effort to construct a search by identifying the keywords is crucial for the search process. The keyword determination method was conducted using a snowballing process to identify the reflections or keywords relevant to this study. Overlooking articles by omitting relevant information or keyword combinations is likewise possible due to the limited time frame.

Nevertheless, this study also faces the possible limitation caused by the selection criteria. For example, this study focused on only journal articles and was limited to only documents written in English. Therefore, other pertinent articles that are not written in English and were not published in journals might have not included.

Future research

Our findings revealed that there is no unified agreement on what components to consider planning a personalized adaptive learning environment. Future research can focus on components included in different personalized adaptive learning systems and the term’s definition to build a unified approach and definition. Future studies focusing on what components are being used for each personalized learning approach simultaneously need to acknowledge it as a term that will evolve by time as we learn more about human psychology and develop more technologies. Chatti et al. ( 2010 ) and Peng et al. ( 2019 ) paper put it all together very well, and Peng et al. ( 2019 ) called it a personalized adaptive learning. Future studies can be built on this approach to develop a general framework.

Also, a focus on higher-order thinking skills is not a common theme in the existing literature. This gap can be filled up by focusing on higher-order thinking skills cultivation by supporting these skills through personalized learning environments. Future studies can also focus on adding higher-order thinking skills as an outcome of personalized learning models and seek embedding of virtual reality techniques with considering ethical and privacy concerns.

Furthermore, a in depth study is needed to review current personalized adaptive learning platforms/systems and see if different systems work better for different goals and needs.

Conclusions

In conclusion, this study found and analyzed 56 relevant studies based on the research protocol. The findings from this study support that adaptive/personalized learning has become a fundamental learning paradigm in the research community of educational technologies. Firstly, the findings are presented as they relate to the R.Q. (Research Question) s; then, the future direction and limitations are discussed. The SLR results show that using personality traits and their identification techniques has an enormously positive influence in adaptive learning environments. This study is related to several significant domains of psychology, education, and computer science. It likewise reveals the integration of personal traits in the adaptive learning environment, which involves many personality traits and identification techniques that can influence learning. Also, it found that there is an increase of interest in two areas that are oriented towards the incorporation and exploration of significant data capabilities in education: Educational Data Mining (EDM) and Learning Analytics (LA) and their respective communities (Papamitsiou and Economides, 2014 ) which seems to adding another perspective to personalized learning and make it easier modify the learning according individuals.

It seems the personalized learning models gain more attention from governments and policymakers than educators and researchers. We need to focus on the obstacles of lack of interest to motivate the educators and researchers, the experts of the field, to voice their concerns and look for solutions to come up with a robust personalized learning model that will satisfy both instructor and learners’ expectations. Personalized learning cannot be a solution to learning until it is defined better and developed more thoroughly. Personalized learning for everyone looks different according to the needs and goals of the individual. Ennouamani, Mahani and Akharraz ( 2020 ) argued that learners are different in terms of their needs, knowledge, personality, behavior (Pliakos et al., 2019 ) preferences, learning style, culture, as well as the parameters of the mobile devices that they use. Furthermore, the increasing involvement of the researchers and educators in proposing personalized learning approaches can increase the trust towards the ICT supported personalized learning models.

In this review study, we have answered some critical research questions, including the issues with different terms that have been used for personalized learning, components of personalized learning, and obstacles to the development of personalized learning. We need more research to be done about personalized learning. We also need the involvement of experts in the field, educators, pedagogues, researchers, software engineers, and programmers to create teams to work on the same goal to produce stable, unified, personalized learning systems/models.

Also, some research issues and potential future development directions are discussed. According to the discussions and results, it was found that adaptive/personalized learning systems seem to evolve as technology develops, however, a unified agreement on the components needs to be included in personalized learning models still needed. These components may evolve as we learn more about human-machine interaction and learn to take advantage of the technology to improve learning experiences. We suggest that researchers might use the consolidated terms of this review to guide future meta-analyses of the impact of personalized learning on student learning and performance.

To sum up, this study discusses the potential obstacles to personalized learning and practical solutions for these issues. We also discussed different components used for personalized learning models and how personalized learning evolves as technology develops, and we learn more about human-machine interaction.

Availability of data and materials

Not applicable.

Abbreviations

Individualized educational plans

Personalized learning plan

Scimago Journal Rank

Organisation for Economic Co-operation and Development

Information communication technology

Systematic Literature Review

Research Question

Educational Data Mining

Learning Analytics

The papers identified for systematic literature review are marked in the references with an asterisk (*)

*Aeiad, E., & Meziane, F. (2019). An adaptable and personalised elearning system applied to computer. Education and Information Technologies , 78 , 674–681.

Google Scholar  

*Afini Normadhi, N. B., Shuib, L., Md Nasir, H. N., Bimba, A., Idris, N., & Balakrishnan, V. (2019). Identification of personal traits in adaptive learning environment: Systematic literature review. Computers & Education , 130 , 168–190. https://doi.org/10.1016/j.compedu.2018.11.005 .

Article   Google Scholar  

Aroyo, L., Dolog, P., Houben, G.-J., Kravcik, M., Naeve, A., Nilsson, M., & Wild, F. (2006). Interoperability in personalized adaptive learning. Educational Technology & Society, 9 (2), 4–18.

Atkinson, S. (2006). Factors influencing successful achievement in contrasting design and technology activities in higher education. International Journal of Technology and Design Education , 16 , 193–213.

Bahçeci, F., & Gürol, M. (2016). The effect of individualized instruction system on the academic achievement scores of students. Education Research International , 2016 , 1–9. https://doi.org/10.1155/2016/7392125 .

Barrio, B. L., Miller, D., Hsiao, Y. J., Dunn, M., Petersen, S., Hollingshead, A., & Banks, S. (2017). Designing culturally responsive and relevant individualized educational programs. Intervention in School and Clinic , 53 (2), 114–119. https://doi.org/10.1177/1053451217693364 .

*Benhamdi, S., Babouri, A., & Chiky, R. (2017). Personalized recommender system for e-Learning environment. Education and Information Technologies , 22 (4), 1455–1477. https://doi.org/10.1007/s10639-016-9504-y .

Brusilovsky, P., & Peylo, C. (2003). Adaptive and intelligent web-based educational systems adaptive and intelligent technologies for web-based educational systems. International Journal of Artificial Intelligence in Education , 13 , 156–169.

*Çakıroğlu, Ü. (2014). Analyzing the effect of learning styles and study habits of distance learners on learning performances: a case of an introductory programming course. International Review of Research in Open and Distance Learning , 15 (4), 161–185.

Cavanagh, S. (2014). What is “personalized learning”? Educators seek clarity. Education Week , 34 (9), S2–S4.

Chatti, M. A. (2010). Personalization in technology enhanced learning: a social software perspective . Aachen: Shaker Verlag.

*Chatti, M. A., Jarke, M., & Specht, M. (2010). The 3P learning model. Educational Technology & Society , 13 (4), 74–85.

Chen, C. M., Lee, H. M., & Chen, Y. H. (2005). Personalized e-learning system using item response theory. Computers & Education , 44 (3), 237–255.

Chen, C. M., Liu, C. Y., & Chang, M. H. (2006). Personalized curriculum sequencing utilizing modified item response theory for web-based instruction. Expert Systems with Applications , 30 (2), 378–396.

Chen, S. Y., Huang, P. R., Shih, Y. C., & Chang, L. P. (2016). Investigation of multiple human factors in personalized learning. Interactive Learning Environments , 24 (1), 119–141.

Cheung, W. S., & Hew, K. F. (2009). A review of research methodologies used in studies on mobile handheld devices in K-12 and higher education settings. Australasian Journal of Educational Technology , 25 (2), 153–183.

*Chou, C. Y., Lai, K. R., Chao, P. Y., Lan, C. H., & Chen, T. H. (2015). Negotiation based adaptive learning sequences: combining adaptivity and adaptability. Computers & Education , 88 , 215–226. https://doi.org/10.1016/j.compedu.2015.05.007 .

Dawson, S., Heathcote, L., & Poole, G. (2010). Harnessing ICT potential: the adoption and analysis of ICT systems for enhancing the student learning experience. International Journal of Educational Management , 24 (2), 116–128.

Dewey, J. (1915). The school and society . Chicago: Chicago Press.

Dewey, J. (1998). Experience and education (60th anniversary edn.) . West Lafayette: Kappa Delta.

*Dwivedi, P., & Bharadwaj, K. K. (2013). Effective trust-aware E-learning recommender system based on learning styles and knowledge levels. Educational Technology & Society , 16 (4), 201–216.

*Ennouamani, S., Mahani, Z., & Akharraz, L. (2020). A context-aware mobile learning system for adapting learning content and format of presentation: design, validation, and evaluation. Education and Information Technologies . https://doi.org/10.1007/s10639-020-10149-9 .

*Erümit, A. K., & Çetin, İ. (2020). Design framework of adaptive intelligent tutoring systems. Education and Information Technologies . https://doi.org/10.1007/s10639-020-10182-8 .

*Fatahi, S. (2019). An experimental study on an adaptive e-learning environment based on learner’s personality and emotion. Education and Information Technologies , 24 (4), 2225–2241. https://doi.org/10.1007/s10639-019-09868-5 .

*FitzGerald, E., Kucirkova, N., Jones, A., Cross, S., Ferguson, R., Herodotou, C., … Scanlon, E. (2018). Dimensions of personalisation in technology-enhanced learning: A framework and implications for design. British Journal of Educational Technology , 49 (1), 165–181. https://doi.org/10.1111/bjet.12534 .

*Flores, R., Ari, F., Inan, F. A., & Arslan-Ari, I. (2012). The impact of adapting content for students with individual differences. Educational Technology & Society , 15 (3), 251–261.

*Foshee, C. M., Elliott, S. N., & Atkinson, R. K. (2016). Technology-enhanced learning in college mathematics remediation. British Journal of Educational Technology , 47 (5), 893–905. https://doi.org/10.1111/bjet.12285 .

*Garcia-Cabot, A., De-Marcos, L., & Garcia-Lopez, E. (2015). An empirical study on m-learning adaptation: Learning performance and learning contexts. Computers & Education , 82 , 450–459. https://doi.org/10.1016/j.compedu.2014.12.007 .

*George, G., & Lal, A. M. (2019). Review of ontology-based recommender systems in e-learning. Computers & Education , 142 (July), 103,642 https://doi.org/10.1016/j.compedu.2019.103642 .

Göbel, S., Wendel, V., Ritter, C., & Steinmetz, R. (2010). Personalized, adaptive digital educational games using narrative game-based learning objects. In International conference on technologies for E-learning and digital entertainment , (pp. 438–445). Berlin. Springer.

Gómez, S., Zervas, P., Sampson, D. G., & Fabregat, R. (2014). Context-aware adaptive and personalized mobile learning delivery supported by UoLmP. Journal of King Saud University - Computer and Information Sciences , 26 (1), 47–61. https://doi.org/10.1016/j.jksuci.2013.10.008 .

*Griff, E. R., & Matter, S. F. (2013). Evaluation of an adaptive online learning system. British Journal of Educational Technology , 44 (1), 170–176. https://doi.org/10.1111/j.1467-8535.2012.01300.x .

*Hooshyar, D., Ahmad, R. B., Yousefi, M., Yusop, F. D., & Horng, S. J. (2015). A flowchart-based intelligent tutoring system for improving problem-solving skills of novice programmers. Journal of Computer Assisted Learning , 31 (4), 345–361. https://doi.org/10.1111/jcal.12099 .

*Hsieh, C. W., & Chen, S. Y. (2016). A cognitive style perspective to handheld devices: customization vs. personalization. International Review of Research in Open and Distance Learning , 17 (1), 1–22. https://doi.org/10.19173/irrodl.v17i1.2168 .

*Huang, S. L., & Shiu, J. H. (2012). A user-centric adaptive learning system for e-learning 2.0. Educational Technology & Society , 15 (3), 214–225.

Huang, Y. M., Liang, T. H., Su, Y. N., & Chen, N. S. (2012). Empowering personalized learning with an interactive e-book learning system for elementary school students. Educational Technology Research and Development , 60 (4), 703–722. https://doi.org/10.1007/s11423-012-9237-6 .

*Hwang, G. J., Sung, H. Y., Hung, C. M., & Huang, I. (2013). A learning style perspective to investigate the necessity of developing adaptive learning systems. Educational Technology & Society , 16 (2), 188–197.

*Jung, E., Kim, D., Yoon, M., Park, S., & Oakley, B. (2019). The influence of instructional design on learner control, sense of achievement, and perceived effectiveness in a supersize MOOC course. Computers & Education , 128 (October 2018), 377–388 https://doi.org/10.1016/j.compedu.2018.10.001 .

*Junokas, M. J., Lindgren, R., Kang, J., & Morphew, J. W. (2018). Enhancing multimodal learning through personalized gesture recognition. Journal of Computer Assisted Learning , 34 (4), 350–357. https://doi.org/10.1111/jcal.12262 .

*Kinshuk (2012). Guest editorial: personalized learning. Educational Technology Research and Development , 60 (4), 561–562. https://doi.org/10.1007/s11423-012-9248-3 .

*Kinshuk, Huang, H.-W., Sampson, D., & Chen, N.-S. (2013). Trends in educational technology through the lens of the highly cited articles published in the Journal of Educational Technology and Society. Educational Technology & Society , 16 (2), 3–20.

*Klašnja-Milićević, A., Vesin, B., Ivanović, M., & Budimac, Z. (2011). E-Learning personalization based on hybrid recommendation strategy and learning style identification. Computers & Education , 56 (3), 885–899. https://doi.org/10.1016/j.compedu.2010.11.001 .

*Ko, C. C., Chiang, C. H., Lin, Y. L., & Chen, M. C. (2011). An individualized e-Reading system developed based on multirepresentations approach. Educational Technology & Society , 14 (4), 88–98.

*Latham, A., Crockett, K., & McLean, D. (2014). An adaptation algorithm for an intelligent natural language tutoring system. Computers & Education , 71 , 97–110. https://doi.org/10.1016/j.compedu.2013.09.014 .

*Lee, D., Huh, Y., Lin, C. Y., & Reigeluth, C. M. (2018). Technology functions for personalized learning in learner-centered schools. Educational Technology Research and Development , 6 (5), 1269–1302. https://doi.org/10.1007/s11423-018-9615-9 .

*Lin, C. F., Yeh, Y. C., Hung, Y. H., & Chang, R. I. (2013). Data mining for providing a personalized learning path in creativity: an application of decision trees. Computers & Education , 68 , 199–210. https://doi.org/10.1016/j.compedu.2013.05.009 .

*Liu, M., McKelroy, E., Corliss, S. B., & Carrigan, J. (2017). Investigating the effect of an adaptive learning intervention on students’ learning. Educational Technology Research and Development , 65 (6), 1605–1625. https://doi.org/10.1007/s11423-017-9542-1 .

*Liu, M.-T., & Yu, P.-T. (2011). International forum of educational technology & Society aberrant learning achievement detection based on person-fit statistics in personalized e-learning systems. Journal of Educational Technology & Society , 14 (1), 107–120. https://doi.org/10.2307/jeductechsoci.14.1.107

*Lo, J. J., Chan, Y. C., & Yeh, S. W. (2012). Designing an adaptive web-based learning system based on students’ cognitive styles identified online. Computers & Education , 58 (1), 209–222. https://doi.org/10.1016/j.compedu.2011.08.018 .

Lockspeiser, T. M., & Kaul, P. (2016). Using individualized learning plans to facilitate learner-centered teaching. Journal of Pediatric and Adolescent Gynecology , 29 (3), 214–217. https://doi.org/10.1016/j.jpag.2015.10.020 .

*Lu, C., Chang, M., Kinshuk, Huang, E., & Chen, C. W. (2014). Context-aware mobile role-playing game for learning. Lecture Notes in Educational Technology , 17 (9783642382901), 131–146. https://doi.org/10.1007/978-3-642-38291-8_8 .

McCombs, B. L., & Whisler, J. S. (1997). The learner-centered classroom and school: Strategies for increasing student motivation and achievement , (1st ed.). San Francisco: Jossey-Bass.

Miliband, D. (2006). Choice and voice in personalised learning. In OECD (Ed.), Schooling for tomorrow: personalising education , (pp. 21–30). Paris: OECD Publishing.

*Nabizadeh, A. H., Gonçalves, D., Gama, S., Jorge, J., & Rafsanjani, H. N. (2020). Adaptive learning path recommender approach using auxiliary learning objects. Computers & Education , 147 (November 2019), 103777 https://doi.org/10.1016/j.compedu.2019.103777 .

*Nedungadi, P., & Raman, R. (2012). A new approach to personalization: Integrating e-learning and m-learning. Educational Technology Research and Development , 60 (4), 659–678. https://doi.org/10.1007/s11423-012-9250-9 .

*Niknam, M., & Thulasiraman, P. (2020). LPR: a bio-inspired intelligent learning path recommendation system based on meaningful learning theory. Education and Information Technologies . https://doi.org/10.1007/s10639-020-10133-3 .

Okoli, C. (2015). A guide to conducting a standalone systematic literature review. Communications of the Association for Information Systems , 37 (1), 879–910. https://doi.org/10.17705/1cais.03743 .

Organisation for Economic Co-operation and Development (OECD). (2006). Are students ready for a technology-rich world? What PISA studies tell us. Retrieved from http://www.oecd.org

Ormrod, J. E. (2011). Human learning . London: Pearson Higher.

Papamitsiou, Z., & Economides, A. A. (2014). Learning analytics and educational data mining in practice: a systemic literature review of empirical evidence. Educational Technology & Society , 17 (4), 49–64.

Peng, H., Ma, S., & Spector, J. M. (2019). Personalized adaptive learning: an emerging pedagogical approach enabled by a smart learning environment. Lecture Notes in Educational Technology , 171–176. https://doi.org/10.1007/978-981-13-6908-7_24 .

*Pliakos, K., Joo, S. H., Park, J. Y., Cornillie, F., Vens, C., & Van den Noortgate, W. (2019). Integrating machine learning into item response theory for addressing the cold start problem in adaptive learning systems. Computers & Education , 137 (April), 91–103. https://doi.org/10.1016/j.compedu.2019.04.009 .

*Pontual Falcão, T., e Peres, F. M. A., Sales de Morais, D. C., & da Silva Oliveira, G. (2018). Participatory methodologies to promote student engagement in the development of educational digital games. Computers & Education , 116 , 161–175. https://doi.org/10.1016/j.compedu.2017.09.006 .

*Ramos De Melo, F., Flôres, E. L., Diniz De Carvalho, S., Gonçalves De Teixeira, R. A., Batista Loja, L. F., & De Sousa Gomide, R. (2014). Computational organization of didactic contents for personalized virtual learning environments. Computers & Education , 79 , 126–137. https://doi.org/10.1016/j.compedu.2014.07.012 .

*Rastegarmoghadam, M., & Ziarati, K. (2017). Improved modeling of intelligent tutoring systems using ant colony optimization. Education and Information Technologies , 22 (3), 1067–1087. https://doi.org/10.1007/s10639-016-9472-2 .

Reisman, S. (2014). The future of online instruction, Part 2. Computer , 47 (6), 82–84. https://doi.org/10.1109/MC.2014.168 .

Sampson, D., Karagiannidis, C., & Kinshuk (2002). Personalised learning: educational, technological and standardisation perspective. Interactive Educational Multimedia: IEM , 4 (4), 24–39.

*Scheiter, K., Schubert, C., Schüler, A., Schmidt, H., Zimmermann, G., Wassermann, B., … Eder, T. (2019). Adaptive multimedia: using gaze-contingent instructional guidance to provide personalized processing support. Computers & Education , 139 , 31–47. https://doi.org/10.1016/j.compedu.2019.05.005 .

*Schmid, R., & Petko, D. (2019). Does the use of educational technology in personalized learning environments correlate with self-reported digital skills and beliefs of secondary-school students? Computers & Education , 136 (March), 75–86. https://doi.org/10.1016/j.compedu.2019.03.006 .

Sharples, M. (2000). The design of personal mobile technologies for lifelong learning. Computers & Education , 34 (3–4), 177–193. https://doi.org/10.1016/s0360-1315(99)00044-5 .

*Shute, V. J., & Rahimi, S. (2017). Review of computer-based assessment for learning in elementary and secondary education. Journal of Computer Assisted Learning , 33 (1), 1–19. https://doi.org/10.1111/jcal.12172 .

*Somyürek, S. (2015). The new trends in adaptive educational hypermedia systems. International Review of Research in Open and Distance Learning , 16 (1), 221–241. https://doi.org/10.19173/irrodl.v16i1.1946 .

*Spector, J. M. (2013). Emerging educational technologies and research directions. Educational Technology & Society , 16 (2), 21–30.

Spector, J. M. (2014). Conceptualizing the emerging field of smart learning environments. Smart Learning Environments , 1 , 2. https://doi.org/10.1186/s40561-014-0002-7 .

Spector, J. M. (2015). Foundations of educational technology: integrative approaches and interdisciplinary perspectives , (2nd ed.). New York: Routledge.

Book   Google Scholar  

Spector, J. M. (2018). The potential of smart technologies for learning and instruction. International Journal of Smart Technology & Learning , 1 (1), 21–32. https://doi.org/10.1504/IJSMARTTL.2016.078163 .

Sungkur, R. K., Antoaroo, M. A., & Beeharry, A. (2016). Eye tracking system for enhanced learning experiences. Education and Information Technologies , 21 (6), 1785–1806. https://doi.org/10.1007/s10639-015-9418-0 .

*Tomberg, V., Laanpere, M., Ley, T., & Normak, P. (2013). Sustaining teacher control in a blog-based personal learning environment. International Review of Research in Open and Distance Learning , 14 (3), 109–133 https://doi.org/10.19173/irrodl.v14i3.1397 .

*Troussas, C., Krouska, A., & Sgouropoulou, C. (2020). Collaboration and fuzzy-modeled personalization for mobile game-based learning in higher education. Computers & Education , 144 (September 2019), 103698 https://doi.org/10.1016/j.compedu.2019.103698 .

Truong, H. M. (2016). Integrating learning styles and adaptive e-learning system: current developments, problems, and opportunities. Computers in Human Behavior , 55 , 1185–1193 https://doi.org/10.1016/j.chb.2015.02.014 .

Tseng, S.-S., Su, J.-M., Hwang, G.-J., Hwang, G.-H., Tsai, C.-C., & Tsai, C.-J. (2008). An object- oriented course framework for developing adaptive learning systems. Educational Technology & Society , 11 (2), 171–191.

U.S. Department of Education (2010). Transforming American education: Learning powered by technology . Washington, DC: Office of Educational Technology.

U.S. Department of Education, Office of Educational Technology (2017 ). Reimagining the role of technology in education: 2017 national education technology plan update. Available at: https://tech.ed.gov/files/2017/01/NETP17.pdf .

*Wang, Y. H., & Liao, H. C. (2011). Adaptive learning for ESL based on computation. British Journal of Educational Technology , 42 (1), 66–87. https://doi.org/10.1111/j.1467-8535.2009.00981.x .

*Xie, H., Chu, H. C., Hwang, G. J., & Wang, C. C. (2019). Trends and development in technology-enhanced adaptive/personalized learning: a systematic review of journal publications from 2007 to 2017. Computers & Education , 140 (June), 103599 https://doi.org/10.1016/j.compedu.2019.103599 .

Yang, T. C., Hwang, G. J., & Yang, S. J. H. (2013). Development of an adaptive learning system with multiple perspectives based on students’ learning styles and cognitive styles. Educational Technology & Society , 16 (4), 185–200.

*Zou, D., & Xie, H. (2018). Personalized word-learning based on technique feature analysis and learning analytics. Educational Technology & Society , 21 (2), 233–244.

Download references

Acknowledgements

Author information, authors and affiliations.

University of North Texas, Denton, USA

Atikah Shemshack & Jonathan Michael Spector

You can also search for this author in PubMed   Google Scholar

Contributions

Each author contributed evenly to this paper. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Atikah Shemshack .

Ethics declarations

Competing interests.

The authors have no conflict of interest to declare.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Shemshack, A., Spector, J.M. A systematic literature review of personalized learning terms. Smart Learn. Environ. 7 , 33 (2020). https://doi.org/10.1186/s40561-020-00140-9

Download citation

Received : 05 August 2020

Accepted : 13 September 2020

Published : 23 October 2020

DOI : https://doi.org/10.1186/s40561-020-00140-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Personalized learning
• Intelligent tutoring systems
• Learning analytics
• Personalized adaptive learning
• Systematic review

Libraries | Research Guides

Literature reviews, what is a literature review, learning more about how to do a literature review.

• Planning the Review
• The Research Question
• Choosing Where to Search
• Organizing the Review
• Writing the Review

A literature review is a review and synthesis of existing research on a topic or research question. A literature review is meant to analyze the scholarly literature, make connections across writings and identify strengths, weaknesses, trends, and missing conversations. A literature review should address different aspects of a topic as it relates to your research question. A literature review goes beyond a description or summary of the literature you have read. 

• Sage Research Methods Core Collection This link opens in a new window SAGE Research Methods supports research at all levels by providing material to guide users through every step of the research process. SAGE Research Methods is the ultimate methods library with more than 1000 books, reference works, journal articles, and instructional videos by world-leading academics from across the social sciences, including the largest collection of qualitative methods books available online from any scholarly publisher. – Publisher

• Next: Planning the Review >>
• Last Updated: May 2, 2024 10:39 AM
• URL: https://libguides.northwestern.edu/literaturereviews

Maureen and Mike Mansfield Library

View All Hours | My Library Accounts

Research and Find Materials

Technology and Spaces

Archives and Special Collections

Teaching and Learning Research Guide

• Children's & YA Literature
• Conduct a Literature Review
• Keep Current
• Citation Style Guides & Management Tools This link opens in a new window

Literature Review Overview

A literature review involves both the literature searching and the writing. The purpose of the literature search is to:

• reveal existing knowledge
• identify areas of consensus and debate
• identify gaps in knowledge
• identify approaches to research design and methodology
• identify other researchers with similar interests
• clarify your future directions for research

List above from Conducting A Literature Search , Information Research Methods and Systems, Penn State University Libraries

A literature review provides an evaluative review and documentation of what has been published by scholars and researchers on a given topic. In reviewing the published literature, the aim is to explain what ideas and knowledge have been gained and shared to date (i.e., hypotheses tested, scientific methods used, results and conclusions), the weakness and strengths of these previous works, and to identify remaining research questions: A literature review provides the context for your research, making clear why your topic deserves further investigation.

Before You Search

• Select and understand your research topic and question.
• Identify the major concepts in your topic and question.
• Brainstorm potential keywords/terms that correspond to those concepts.
• Identify alternative keywords/terms (narrower, broader, or related) to use if your first set of keywords do not work.
• Determine (Boolean*) relationships between terms.
• Begin your search.
• Review your search results.
• Revise & refine your search based on the initial findings.

*Boolean logic provides three ways search terms/phrases can be combined, using the following three operators: AND, OR, and NOT.

Search Process

The type of information you want to find and the practices of your discipline(s) drive the types of sources you seek and where you search.

For most research you will use multiple source types such as: annotated bibliographies; articles from journals, magazines, and newspapers; books; blogs; conference papers; data sets; dissertations; organization, company, or government reports; reference materials; systematic reviews; archival materials; curriculum materials; and more. It can be helpful to develop a comprehensive approach to review different sources and where you will search for each. Below is an example approach.

Utilize Current Awareness Services  Identify and browse current issues of the most relevant journals for your topic; Setup email or RSS Alerts, e.g., Journal Table of Contents, Saved Searches

Consult Experts   Identify and search for the publications of or contact educators, scholars, librarians, employees etc. at schools, organizations, and agencies

• Annual Reviews and Bibliographies   e.g., Annual Review of Psychology
• Internet   e.g., Discussion Groups, Listservs, Blogs, social networking sites
• Grant Databases   e.g., Foundation Directory Online, Grants.gov
• Conference Proceedings   e.g., American Educational Research Association Online Paper Repository
• Newspaper Indexes   e.g., Access World News, Ethnic NewsWatch, New York Times Historical
• Be sure to follow the tips in the "Finding Empirical Studies" box on the right side of the page if you need to find an empirical study.
• Citation Indexes   e.g., ERIC (Education Resources Information Center), Educational Administration Abstracts, PsycINFO
• Specialized Data   e.g., GEMS ( Growth and Enhancement of Montana Students) , IPEDS ( Integrated Postsecondary Education Data System)
• Book Catalogs – e.g., local library catalog or discovery search, WorldCat
• Library Web Scale Discovery Service  e.g., OneSearch
• Web Search Engines   e.g., Google, Yahoo
• Digital Collections   e.g., Archives & Special Collections Digital Collections, Digital Public Library of America
• Associations/Community groups/Institutions/Organizations   e.g., Association for Supervision and Curriculum Development, Montana Office of Public Instruction, National Education Association

Remember there is no one portal for all information!

Database Searching Videos, Guides, and Examples

ProQuest (platform for ERIC, PsycINFO, and Dissertations & Theses Global databases, among other databases) search videos:

• Basic Search
• Advanced Search
• Search Results

ERIC (Educational Resources Information Center)

• Comprehensive guide to the database, including Sample Searches
• Searchable Fields
• Comprehensive guide to the database
• Education topic guide
• Child Development topic guide
• Performing Basic Searches
• Performing Advanced Searches
• Search Tips

If you are new to research , check out the Searching for Information tutorials and videos for foundational information.

Finding Empirical Studies

In ERIC : Check the box next to “143: Reports - Research” under "Document type" from the Advanced Search page

In PsycINFO : Check the box next to “Empirical Study” under "Methodology" from the Advanced Search page

In OneSearch : There is not a specific way to limit to empirical studies in OneSearch, you can limit your search results to peer-reviewed journals and or dissertations, and then identify studies by reading the source abstract to determine if you’ve found an empirical study or not.

Summarize Studies in a Meaningful Way

The Writing and Public Speaking Center at UM provides not only tutoring but many other resources for writers and presenters. Three with key tips for writing a literature review are:

• Literature Reviews Defined
• Tracking, Organizing, and Using Sources
• Organizing and Integrating Sources

If you are new to research , check out the Presenting and Organizing Information tutorials and videos for foundational information. You may also want to consult the Purdue OWL Academic Writing resources or APA Style Workshop content.

• << Previous: Children's & YA Literature
• Next: Keep Current >>
• Last Updated: Mar 21, 2024 1:47 PM
• URL: https://libguides.lib.umt.edu/teach

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Clinics (Sao Paulo)

Approaching literature review for academic purposes: The Literature Review Checklist

Debora f.b. leite.

I Departamento de Ginecologia e Obstetricia, Faculdade de Ciencias Medicas, Universidade Estadual de Campinas, Campinas, SP, BR

II Universidade Federal de Pernambuco, Pernambuco, PE, BR

III Hospital das Clinicas, Universidade Federal de Pernambuco, Pernambuco, PE, BR

Maria Auxiliadora Soares Padilha

Jose g. cecatti.

A sophisticated literature review (LR) can result in a robust dissertation/thesis by scrutinizing the main problem examined by the academic study; anticipating research hypotheses, methods and results; and maintaining the interest of the audience in how the dissertation/thesis will provide solutions for the current gaps in a particular field. Unfortunately, little guidance is available on elaborating LRs, and writing an LR chapter is not a linear process. An LR translates students’ abilities in information literacy, the language domain, and critical writing. Students in postgraduate programs should be systematically trained in these skills. Therefore, this paper discusses the purposes of LRs in dissertations and theses. Second, the paper considers five steps for developing a review: defining the main topic, searching the literature, analyzing the results, writing the review and reflecting on the writing. Ultimately, this study proposes a twelve-item LR checklist. By clearly stating the desired achievements, this checklist allows Masters and Ph.D. students to continuously assess their own progress in elaborating an LR. Institutions aiming to strengthen students’ necessary skills in critical academic writing should also use this tool.

INTRODUCTION

Writing the literature review (LR) is often viewed as a difficult task that can be a point of writer’s block and procrastination ( 1 ) in postgraduate life. Disagreements on the definitions or classifications of LRs ( 2 ) may confuse students about their purpose and scope, as well as how to perform an LR. Interestingly, at many universities, the LR is still an important element in any academic work, despite the more recent trend of producing scientific articles rather than classical theses.

The LR is not an isolated section of the thesis/dissertation or a copy of the background section of a research proposal. It identifies the state-of-the-art knowledge in a particular field, clarifies information that is already known, elucidates implications of the problem being analyzed, links theory and practice ( 3 - 5 ), highlights gaps in the current literature, and places the dissertation/thesis within the research agenda of that field. Additionally, by writing the LR, postgraduate students will comprehend the structure of the subject and elaborate on their cognitive connections ( 3 ) while analyzing and synthesizing data with increasing maturity.

At the same time, the LR transforms the student and hints at the contents of other chapters for the reader. First, the LR explains the research question; second, it supports the hypothesis, objectives, and methods of the research project; and finally, it facilitates a description of the student’s interpretation of the results and his/her conclusions. For scholars, the LR is an introductory chapter ( 6 ). If it is well written, it demonstrates the student’s understanding of and maturity in a particular topic. A sound and sophisticated LR can indicate a robust dissertation/thesis.

A consensus on the best method to elaborate a dissertation/thesis has not been achieved. The LR can be a distinct chapter or included in different sections; it can be part of the introduction chapter, part of each research topic, or part of each published paper ( 7 ). However, scholars view the LR as an integral part of the main body of an academic work because it is intrinsically connected to other sections ( Figure 1 ) and is frequently present. The structure of the LR depends on the conventions of a particular discipline, the rules of the department, and the student’s and supervisor’s areas of expertise, needs and interests.

Interestingly, many postgraduate students choose to submit their LR to peer-reviewed journals. As LRs are critical evaluations of current knowledge, they are indeed publishable material, even in the form of narrative or systematic reviews. However, systematic reviews have specific patterns 1 ( 8 ) that may not entirely fit with the questions posed in the dissertation/thesis. Additionally, the scope of a systematic review may be too narrow, and the strict criteria for study inclusion may omit important information from the dissertation/thesis. Therefore, this essay discusses the definition of an LR is and methods to develop an LR in the context of an academic dissertation/thesis. Finally, we suggest a checklist to evaluate an LR.

WHAT IS A LITERATURE REVIEW IN A THESIS?

Conducting research and writing a dissertation/thesis translates rational thinking and enthusiasm ( 9 ). While a strong body of literature that instructs students on research methodology, data analysis and writing scientific papers exists, little guidance on performing LRs is available. The LR is a unique opportunity to assess and contrast various arguments and theories, not just summarize them. The research results should not be discussed within the LR, but the postgraduate student tends to write a comprehensive LR while reflecting on his or her own findings ( 10 ).

Many people believe that writing an LR is a lonely and linear process. Supervisors or the institutions assume that the Ph.D. student has mastered the relevant techniques and vocabulary associated with his/her subject and conducts a self-reflection about previously published findings. Indeed, while elaborating the LR, the student should aggregate diverse skills, which mainly rely on his/her own commitment to mastering them. Thus, less supervision should be required ( 11 ). However, the parameters described above might not currently be the case for many students ( 11 , 12 ), and the lack of formal and systematic training on writing LRs is an important concern ( 11 ).

An institutional environment devoted to active learning will provide students the opportunity to continuously reflect on LRs, which will form a dialogue between the postgraduate student and the current literature in a particular field ( 13 ). Postgraduate students will be interpreting studies by other researchers, and, according to Hart (1998) ( 3 ), the outcomes of the LR in a dissertation/thesis include the following:

• To identify what research has been performed and what topics require further investigation in a particular field of knowledge;
• To determine the context of the problem;
• To recognize the main methodologies and techniques that have been used in the past;
• To place the current research project within the historical, methodological and theoretical context of a particular field;
• To identify significant aspects of the topic;
• To elucidate the implications of the topic;
• To offer an alternative perspective;
• To discern how the studied subject is structured;
• To improve the student’s subject vocabulary in a particular field; and
• To characterize the links between theory and practice.

A sound LR translates the postgraduate student’s expertise in academic and scientific writing: it expresses his/her level of comfort with synthesizing ideas ( 11 ). The LR reveals how well the postgraduate student has proceeded in three domains: an effective literature search, the language domain, and critical writing.

Effective literature search

All students should be trained in gathering appropriate data for specific purposes, and information literacy skills are a cornerstone. These skills are defined as “an individual’s ability to know when they need information, to identify information that can help them address the issue or problem at hand, and to locate, evaluate, and use that information effectively” ( 14 ). Librarian support is of vital importance in coaching the appropriate use of Boolean logic (AND, OR, NOT) and other tools for highly efficient literature searches (e.g., quotation marks and truncation), as is the appropriate management of electronic databases.

Language domain

Academic writing must be concise and precise: unnecessary words distract the reader from the essential content ( 15 ). In this context, reading about issues distant from the research topic ( 16 ) may increase students’ general vocabulary and familiarity with grammar. Ultimately, reading diverse materials facilitates and encourages the writing process itself.

Critical writing

Critical judgment includes critical reading, thinking and writing. It supposes a student’s analytical reflection about what he/she has read. The student should delineate the basic elements of the topic, characterize the most relevant claims, identify relationships, and finally contrast those relationships ( 17 ). Each scientific document highlights the perspective of the author, and students will become more confident in judging the supporting evidence and underlying premises of a study and constructing their own counterargument as they read more articles. A paucity of integration or contradictory perspectives indicates lower levels of cognitive complexity ( 12 ).

Thus, while elaborating an LR, the postgraduate student should achieve the highest category of Bloom’s cognitive skills: evaluation ( 12 ). The writer should not only summarize data and understand each topic but also be able to make judgments based on objective criteria, compare resources and findings, identify discrepancies due to methodology, and construct his/her own argument ( 12 ). As a result, the student will be sufficiently confident to show his/her own voice .

Writing a consistent LR is an intense and complex activity that reveals the training and long-lasting academic skills of a writer. It is not a lonely or linear process. However, students are unlikely to be prepared to write an LR if they have not mastered the aforementioned domains ( 10 ). An institutional environment that supports student learning is crucial.

Different institutions employ distinct methods to promote students’ learning processes. First, many universities propose modules to develop behind the scenes activities that enhance self-reflection about general skills (e.g., the skills we have mastered and the skills we need to develop further), behaviors that should be incorporated (e.g., self-criticism about one’s own thoughts), and each student’s role in the advancement of his/her field. Lectures or workshops about LRs themselves are useful because they describe the purposes of the LR and how it fits into the whole picture of a student’s work. These activities may explain what type of discussion an LR must involve, the importance of defining the correct scope, the reasons to include a particular resource, and the main role of critical reading.

Some pedagogic services that promote a continuous improvement in study and academic skills are equally important. Examples include workshops about time management, the accomplishment of personal objectives, active learning, and foreign languages for nonnative speakers. Additionally, opportunities to converse with other students promotes an awareness of others’ experiences and difficulties. Ultimately, the supervisor’s role in providing feedback and setting deadlines is crucial in developing students’ abilities and in strengthening students’ writing quality ( 12 ).

HOW SHOULD A LITERATURE REVIEW BE DEVELOPED?

A consensus on the appropriate method for elaborating an LR is not available, but four main steps are generally accepted: defining the main topic, searching the literature, analyzing the results, and writing ( 6 ). We suggest a fifth step: reflecting on the information that has been written in previous publications ( Figure 2 ).

First step: Defining the main topic

Planning an LR is directly linked to the research main question of the thesis and occurs in parallel to students’ training in the three domains discussed above. The planning stage helps organize ideas, delimit the scope of the LR ( 11 ), and avoid the wasting of time in the process. Planning includes the following steps:

• Reflecting on the scope of the LR: postgraduate students will have assumptions about what material must be addressed and what information is not essential to an LR ( 13 , 18 ). Cooper’s Taxonomy of Literature Reviews 2 systematizes the writing process through six characteristics and nonmutually exclusive categories. The focus refers to the reviewer’s most important points of interest, while the goals concern what students want to achieve with the LR. The perspective assumes answers to the student’s own view of the LR and how he/she presents a particular issue. The coverage defines how comprehensive the student is in presenting the literature, and the organization determines the sequence of arguments. The audience is defined as the group for whom the LR is written.
• Designating sections and subsections: Headings and subheadings should be specific, explanatory and have a coherent sequence throughout the text ( 4 ). They simulate an inverted pyramid, with an increasing level of reflection and depth of argument.
• Identifying keywords: The relevant keywords for each LR section should be listed to guide the literature search. This list should mirror what Hart (1998) ( 3 ) advocates as subject vocabulary . The keywords will also be useful when the student is writing the LR since they guide the reader through the text.
• Delineating the time interval and language of documents to be retrieved in the second step. The most recently published documents should be considered, but relevant texts published before a predefined cutoff year can be included if they are classic documents in that field. Extra care should be employed when translating documents.

Second step: Searching the literature

The ability to gather adequate information from the literature must be addressed in postgraduate programs. Librarian support is important, particularly for accessing difficult texts. This step comprises the following components:

• Searching the literature itself: This process consists of defining which databases (electronic or dissertation/thesis repositories), official documents, and books will be searched and then actively conducting the search. Information literacy skills have a central role in this stage. While searching electronic databases, controlled vocabulary (e.g., Medical Subject Headings, or MeSH, for the PubMed database) or specific standardized syntax rules may need to be applied.

In addition, two other approaches are suggested. First, a review of the reference list of each document might be useful for identifying relevant publications to be included and important opinions to be assessed. This step is also relevant for referencing the original studies and leading authors in that field. Moreover, students can directly contact the experts on a particular topic to consult with them regarding their experience or use them as a source of additional unpublished documents.

Before submitting a dissertation/thesis, the electronic search strategy should be repeated. This process will ensure that the most recently published papers will be considered in the LR.

• Selecting documents for inclusion: Generally, the most recent literature will be included in the form of published peer-reviewed papers. Assess books and unpublished material, such as conference abstracts, academic texts and government reports, are also important to assess since the gray literature also offers valuable information. However, since these materials are not peer-reviewed, we recommend that they are carefully added to the LR.

This task is an important exercise in time management. First, students should read the title and abstract to understand whether that document suits their purposes, addresses the research question, and helps develop the topic of interest. Then, they should scan the full text, determine how it is structured, group it with similar documents, and verify whether other arguments might be considered ( 5 ).

Third step: Analyzing the results

Critical reading and thinking skills are important in this step. This step consists of the following components:

• Reading documents: The student may read various texts in depth according to LR sections and subsections ( defining the main topic ), which is not a passive activity ( 1 ). Some questions should be asked to practice critical analysis skills, as listed below. Is the research question evident and articulated with previous knowledge? What are the authors’ research goals and theoretical orientations, and how do they interact? Are the authors’ claims related to other scholars’ research? Do the authors consider different perspectives? Was the research project designed and conducted properly? Are the results and discussion plausible, and are they consistent with the research objectives and methodology? What are the strengths and limitations of this work? How do the authors support their findings? How does this work contribute to the current research topic? ( 1 , 19 )
• Taking notes: Students who systematically take notes on each document are more readily able to establish similarities or differences with other documents and to highlight personal observations. This approach reinforces the student’s ideas about the next step and helps develop his/her own academic voice ( 1 , 13 ). Voice recognition software ( 16 ), mind maps ( 5 ), flowcharts, tables, spreadsheets, personal comments on the referenced texts, and note-taking apps are all available tools for managing these observations, and the student him/herself should use the tool that best improves his/her learning. Additionally, when a student is considering submitting an LR to a peer-reviewed journal, notes should be taken on the activities performed in all five steps to ensure that they are able to be replicated.

Fourth step: Writing

The recognition of when a student is able and ready to write after a sufficient period of reading and thinking is likely a difficult task. Some students can produce a review in a single long work session. However, as discussed above, writing is not a linear process, and students do not need to write LRs according to a specific sequence of sections. Writing an LR is a time-consuming task, and some scholars believe that a period of at least six months is sufficient ( 6 ). An LR, and academic writing in general, expresses the writer’s proper thoughts, conclusions about others’ work ( 6 , 10 , 13 , 16 ), and decisions about methods to progress in the chosen field of knowledge. Thus, each student is expected to present a different learning and writing trajectory.

In this step, writing methods should be considered; then, editing, citing and correct referencing should complete this stage, at least temporarily. Freewriting techniques may be a good starting point for brainstorming ideas and improving the understanding of the information that has been read ( 1 ). Students should consider the following parameters when creating an agenda for writing the LR: two-hour writing blocks (at minimum), with prespecified tasks that are possible to complete in one section; short (minutes) and long breaks (days or weeks) to allow sufficient time for mental rest and reflection; and short- and long-term goals to motivate the writing itself ( 20 ). With increasing experience, this scheme can vary widely, and it is not a straightforward rule. Importantly, each discipline has a different way of writing ( 1 ), and each department has its own preferred styles for citations and references.

Fifth step: Reflecting on the writing

In this step, the postgraduate student should ask him/herself the same questions as in the analyzing the results step, which can take more time than anticipated. Ambiguities, repeated ideas, and a lack of coherence may not be noted when the student is immersed in the writing task for long periods. The whole effort will likely be a work in progress, and continuous refinements in the written material will occur once the writing process has begun.

LITERATURE REVIEW CHECKLIST

In contrast to review papers, the LR of a dissertation/thesis should not be a standalone piece or work. Instead, it should present the student as a scholar and should maintain the interest of the audience in how that dissertation/thesis will provide solutions for the current gaps in a particular field.

A checklist for evaluating an LR is convenient for students’ continuous academic development and research transparency: it clearly states the desired achievements for the LR of a dissertation/thesis. Here, we present an LR checklist developed from an LR scoring rubric ( 11 ). For a critical analysis of an LR, we maintain the five categories but offer twelve criteria that are not scaled ( Figure 3 ). The criteria all have the same importance and are not mutually exclusive.

First category: Coverage

1. justified criteria exist for the inclusion and exclusion of literature in the review.

This criterion builds on the main topic and areas covered by the LR ( 18 ). While experts may be confident in retrieving and selecting literature, postgraduate students must convince their audience about the adequacy of their search strategy and their reasons for intentionally selecting what material to cover ( 11 ). References from different fields of knowledge provide distinct perspective, but narrowing the scope of coverage may be important in areas with a large body of existing knowledge.

Second category: Synthesis

2. a critical examination of the state of the field exists.

A critical examination is an assessment of distinct aspects in the field ( 1 ) along with a constructive argument. It is not a negative critique but an expression of the student’s understanding of how other scholars have added to the topic ( 1 ), and the student should analyze and contextualize contradictory statements. A writer’s personal bias (beliefs or political involvement) have been shown to influence the structure and writing of a document; therefore, the cultural and paradigmatic background guide how the theories are revised and presented ( 13 ). However, an honest judgment is important when considering different perspectives.

3. The topic or problem is clearly placed in the context of the broader scholarly literature

The broader scholarly literature should be related to the chosen main topic for the LR ( how to develop the literature review section). The LR can cover the literature from one or more disciplines, depending on its scope, but it should always offer a new perspective. In addition, students should be careful in citing and referencing previous publications. As a rule, original studies and primary references should generally be included. Systematic and narrative reviews present summarized data, and it may be important to cite them, particularly for issues that should be understood but do not require a detailed description. Similarly, quotations highlight the exact statement from another publication. However, excessive referencing may disclose lower levels of analysis and synthesis by the student.

4. The LR is critically placed in the historical context of the field

Situating the LR in its historical context shows the level of comfort of the student in addressing a particular topic. Instead of only presenting statements and theories in a temporal approach, which occasionally follows a linear timeline, the LR should authentically characterize the student’s academic work in the state-of-art techniques in their particular field of knowledge. Thus, the LR should reinforce why the dissertation/thesis represents original work in the chosen research field.

5. Ambiguities in definitions are considered and resolved

Distinct theories on the same topic may exist in different disciplines, and one discipline may consider multiple concepts to explain one topic. These misunderstandings should be addressed and contemplated. The LR should not synthesize all theories or concepts at the same time. Although this approach might demonstrate in-depth reading on a particular topic, it can reveal a student’s inability to comprehend and synthesize his/her research problem.

6. Important variables and phenomena relevant to the topic are articulated

The LR is a unique opportunity to articulate ideas and arguments and to purpose new relationships between them ( 10 , 11 ). More importantly, a sound LR will outline to the audience how these important variables and phenomena will be addressed in the current academic work. Indeed, the LR should build a bidirectional link with the remaining sections and ground the connections between all of the sections ( Figure 1 ).

7. A synthesized new perspective on the literature has been established

The LR is a ‘creative inquiry’ ( 13 ) in which the student elaborates his/her own discourse, builds on previous knowledge in the field, and describes his/her own perspective while interpreting others’ work ( 13 , 17 ). Thus, students should articulate the current knowledge, not accept the results at face value ( 11 , 13 , 17 ), and improve their own cognitive abilities ( 12 ).

Third category: Methodology

8. the main methodologies and research techniques that have been used in the field are identified and their advantages and disadvantages are discussed.

The LR is expected to distinguish the research that has been completed from investigations that remain to be performed, address the benefits and limitations of the main methods applied to date, and consider the strategies for addressing the expected limitations described above. While placing his/her research within the methodological context of a particular topic, the LR will justify the methodology of the study and substantiate the student’s interpretations.

9. Ideas and theories in the field are related to research methodologies

The audience expects the writer to analyze and synthesize methodological approaches in the field. The findings should be explained according to the strengths and limitations of previous research methods, and students must avoid interpretations that are not supported by the analyzed literature. This criterion translates to the student’s comprehension of the applicability and types of answers provided by different research methodologies, even those using a quantitative or qualitative research approach.

Fourth category: Significance

10. the scholarly significance of the research problem is rationalized.

The LR is an introductory section of a dissertation/thesis and will present the postgraduate student as a scholar in a particular field ( 11 ). Therefore, the LR should discuss how the research problem is currently addressed in the discipline being investigated or in different disciplines, depending on the scope of the LR. The LR explains the academic paradigms in the topic of interest ( 13 ) and methods to advance the field from these starting points. However, an excess number of personal citations—whether referencing the student’s research or studies by his/her research team—may reflect a narrow literature search and a lack of comprehensive synthesis of ideas and arguments.

11. The practical significance of the research problem is rationalized

The practical significance indicates a student’s comprehensive understanding of research terminology (e.g., risk versus associated factor), methodology (e.g., efficacy versus effectiveness) and plausible interpretations in the context of the field. Notably, the academic argument about a topic may not always reflect the debate in real life terms. For example, using a quantitative approach in epidemiology, statistically significant differences between groups do not explain all of the factors involved in a particular problem ( 21 ). Therefore, excessive faith in p -values may reflect lower levels of critical evaluation of the context and implications of a research problem by the student.

Fifth category: Rhetoric

12. the lr was written with a coherent, clear structure that supported the review.

This category strictly relates to the language domain: the text should be coherent and presented in a logical sequence, regardless of which organizational ( 18 ) approach is chosen. The beginning of each section/subsection should state what themes will be addressed, paragraphs should be carefully linked to each other ( 10 ), and the first sentence of each paragraph should generally summarize the content. Additionally, the student’s statements are clear, sound, and linked to other scholars’ works, and precise and concise language that follows standardized writing conventions (e.g., in terms of active/passive voice and verb tenses) is used. Attention to grammar, such as orthography and punctuation, indicates prudence and supports a robust dissertation/thesis. Ultimately, all of these strategies provide fluency and consistency for the text.

Although the scoring rubric was initially proposed for postgraduate programs in education research, we are convinced that this checklist is a valuable tool for all academic areas. It enables the monitoring of students’ learning curves and a concentrated effort on any criteria that are not yet achieved. For institutions, the checklist is a guide to support supervisors’ feedback, improve students’ writing skills, and highlight the learning goals of each program. These criteria do not form a linear sequence, but ideally, all twelve achievements should be perceived in the LR.

CONCLUSIONS

A single correct method to classify, evaluate and guide the elaboration of an LR has not been established. In this essay, we have suggested directions for planning, structuring and critically evaluating an LR. The planning of the scope of an LR and approaches to complete it is a valuable effort, and the five steps represent a rational starting point. An institutional environment devoted to active learning will support students in continuously reflecting on LRs, which will form a dialogue between the writer and the current literature in a particular field ( 13 ).

The completion of an LR is a challenging and necessary process for understanding one’s own field of expertise. Knowledge is always transitory, but our responsibility as scholars is to provide a critical contribution to our field, allowing others to think through our work. Good researchers are grounded in sophisticated LRs, which reveal a writer’s training and long-lasting academic skills. We recommend using the LR checklist as a tool for strengthening the skills necessary for critical academic writing.

AUTHOR CONTRIBUTIONS

Leite DFB has initially conceived the idea and has written the first draft of this review. Padilha MAS and Cecatti JG have supervised data interpretation and critically reviewed the manuscript. All authors have read the draft and agreed with this submission. Authors are responsible for all aspects of this academic piece.

ACKNOWLEDGMENTS

We are grateful to all of the professors of the ‘Getting Started with Graduate Research and Generic Skills’ module at University College Cork, Cork, Ireland, for suggesting and supporting this article. Funding: DFBL has granted scholarship from Brazilian Federal Agency for Support and Evaluation of Graduate Education (CAPES) to take part of her Ph.D. studies in Ireland (process number 88881.134512/2016-01). There is no participation from sponsors on authors’ decision to write or to submit this manuscript.

No potential conflict of interest was reported.

1 The questions posed in systematic reviews usually follow the ‘PICOS’ acronym: Population, Intervention, Comparison, Outcomes, Study design.

2 In 1988, Cooper proposed a taxonomy that aims to facilitate students’ and institutions’ understanding of literature reviews. Six characteristics with specific categories are briefly described: Focus: research outcomes, research methodologies, theories, or practices and applications; Goals: integration (generalization, conflict resolution, and linguistic bridge-building), criticism, or identification of central issues; Perspective: neutral representation or espousal of a position; Coverage: exhaustive, exhaustive with selective citations, representative, central or pivotal; Organization: historical, conceptual, or methodological; and Audience: specialized scholars, general scholars, practitioners or policymakers, or the general public.

Along with Stanford news and stories, show me:

• Student information
• Faculty/Staff information

We want to provide announcements, events, leadership messages and resources that are relevant to you. Your selection is stored in a browser cookie which you can remove at any time using “Clear all personalization” below.

Global history is not just significant events on a timeline, it is also the ordinary, mundane moments that people experience in between. Graphic novels can capture this multidimensionality in ways that are difficult, and sometimes impossible, in more traditional media formats, says Stanford history professor Tom Mullaney .

Tom Mullaney, a professor of history in the School of Humanities and Sciences, uses graphic novels in his teachings to help students appreciate different experiences and perspectives throughout history. (Image credit: Ilmiyah Achmad)

Mullaney has incorporated graphic novels in some of his Stanford courses since 2017; in 2020, he taught a course dedicated to the study of world history through comic strip formats.

While graphic novels are not a substitute for academic literature, he said he finds them a useful teaching and research tool. They not only portray the impact of historic events on everyday lives, but because they can be read in one or two sittings, they get to it at a much faster rate than say a 10,000 word essay or autobiography could.

“It accelerates the process of getting to subtlety,” said Mullaney, a professor of history at Stanford’s School of Humanities and Sciences . “There’s just so much you can do, and so many questions you can ask, and so many perspective shifts you can carry out – like that! You can just do it – you show them something, they read it and BOOM! It’s like an accelerant. It’s awesome.”

For example, in Thi Bui’s graphic novel The Best We Could Do , themes of displacement and diaspora emerge as she talks about her family’s escape from war-torn Vietnam to the United States. The illustrated memoir shows Bui’s upbringing in suburban California and the complicated memories her parents carry with them as they move about their new life in America. In other chapters, she depicts her mother and father back in Vietnam and what their own childhood was like amidst the country’s upheaval.

Graphic novels like The Best We Could Do and also Maus , Art Spiegelman’s seminal portrayal of his Jewish family’s experience during the Holocaust, illustrate the challenges and subtleties of memory – particularly family memory – and the entanglements that arise when narrating history, Mullaney said.

Graphic novels prime readers for the complexity of doing and reading historical research and how there is no simplistic, narrative arc of history. “When I read a graphic novel, I feel prepared to ask questions that allow me to go into the harder core, peer-reviewed material,” Mullaney said.

The everyday moments that graphic novels are exceptionally good at capturing also raise questions in the reader’s mind, Mullaney said. Readers sit in the family living room and at the kitchen table with Spiegelman and Bui and follow along as their characters try to understand what their parent’s generation went through and discover it’s not always easy to grasp. “Not everything mom and dad say makes sense,” said Mullaney.

These seemingly mundane moments also present powerful opportunities for inquiry. “The ordinary  is where the explanation lives and where you can start asking questions,” Mullaney added.

Graphic novels can also depict how in periods of war and conflict, violence can become part of everyday existence and survival. The simplicity of the format allows heavy, painful experiences to emerge from a panel untethered and unweighted from lengthy descriptions or dramatizations.

“They’re banal. They’re not dramatic. There are no strings attached. In a work of nonfiction, in an article or book, it would be almost impossible to do that. There would have to be so much expository writing and so much description that you would lose the horror of it,” Mullaney said.

A ‘fundamental misunderstanding’

Graphic novels like Maus and The Best We Could Do were included in Mullaney’s 2020 Stanford class, Global History Through Graphic Novels .

In 2020, Tom Mullaney, a professor of history, taught Global History Through Graphic Novels , a course that paired graphic novels such as Art Spiegelman’s Maus with archival materials and historical essays to examine modern world history from the 18th to the 21st century. He created a poster for the class, as shown here. (Image credit: Tom Mullaney)

In the course, Mullaney paired graphic novels with archival materials and historical essays to examine modern world history from the 18th to the 21st century.

The course syllabus also included the graphic novels Showa , Shigeru Mizuki ’s manga series about growing up in Japan before World War II, and Such a Lovely Little War , about Marcelino Truong’s experience as a child in Saigon during the Vietnam War.

Most recently, Mullaney has offered to teach a variation of the Stanford course to the public, free for high school and college students , this summer.

Registration for the online course opened shortly after news emerged and made international headlines that Maus was banned by a Tennessee school board for its depiction of nudity and use of swear words.

Within two days of Mullaney’s course registration opening, over 200 students from across the world signed up.

Mullaney believes that there is a “fundamental misunderstanding” about what young people can process when it comes to negotiating complex themes and topics – whether it is structural racism or the Holocaust. They just need some guidance, which he hopes as a teacher, he can provide.

“I think students of high school age or even younger, if they have the scaffolding they need – which is the job of educators to give them – they can handle the structural inequalities, darknesses and horrors of life,” he said.

Mullaney noted that many teenagers are already exposed to many of these difficult issues through popular media. “But they’re just doing it on their own and figuring it out for themselves – that’s not a good idea,” he said.

Mullaney said he hopes he can teach Global History Through Graphic Novels to Stanford students again this fall.

For Stanford scholars interested in learning more about the intersection of graphic novels and scholarship, there is a newly established working group through the Division of Literatures, Cultures and Languages, Comics, More than Words .

Media Contacts

Melissa De Witte, Stanford News Service:  [email protected]

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 26 May 2024

A double machine learning model for measuring the impact of the Made in China 2025 strategy on green economic growth

• Jie Yuan 1 &
• Shucheng Liu 2  

Scientific Reports volume  14 , Article number:  12026 ( 2024 ) Cite this article

Metrics details

• Environmental economics
• Environmental impact
• Sustainability

The transformation and upgrading of China’s manufacturing industry is supported by smart and green manufacturing, which have great potential to empower the nation’s green development. This study examines the impact of the Made in China 2025 industrial policy on urban green economic growth. This study applies the super-slacks-based measure model to measure cities’ green economic growth, using the double machine learning model, which overcomes the limitations of the linear setting of traditional causal inference models and maintains estimation accuracy under high-dimensional control variables, to conduct an empirical analysis based on panel data of 281 Chinese cities from 2006 to 2021. The results reveal that the Made in China 2025 strategy significantly drives urban green economic growth, and this finding holds after a series of robustness tests. A mechanism analysis indicates that the Made in China 2025 strategy promotes green economic growth through green technology progress, optimizing energy consumption structure, upgrading industrial structure, and strengthening environmental supervision. In addition, the policy has a stronger driving effect for cities with high manufacturing concentration, industrial intelligence, and digital finance development. This study provides valuable theoretical insights and policy implications for government planning to promote high-quality development through industrial policy.

Similar content being viewed by others

The economic commitment of climate change

Systematic review and meta-analysis of ex-post evaluations on the effectiveness of carbon pricing

The impact of artificial intelligence on employment: the role of virtual agglomeration

Introduction.

Since China’s reform and opening up, the nation’s economy has experienced rapid growth for more than 40 years. According to the National Bureau of Statistics, China’s per capita GDP has grown from 385 yuan in 1978 to 85,698 yuan in 2022, with an average annual growth rate of 13.2%. However, obtaining this growth miracle has come at considerable social and environmental costs 1 . Current pollution prevention and control systems have not yet fundamentally alleviated the structural and root causes, impairing China’s economic progress toward high-quality development 2 . The report of the 20th National Congress of the Communist Party of China proposed that the future will be focused on promoting the formation of green modes of production and lifestyles and advancing the harmonious coexistence of human beings and nature. This indicates that transforming the mode of economic development is now the focus of the government’s attention, calling for advancing the practices of green growth aimed at energy conservation, emissions reduction, and sustainability while continuously increasing economic output 3 . As a result, identifying approaches to balance economic growth and green environmental protection in the development process and realize green economic growth has become an arduous challenge and a crucially significant concern for China’s high-quality economic development.

An intrinsic driver of urban economic growth, manufacturing is also the most energy-intensive and pollution-emitting industry, and greatly constrains urban green development 4 . China’s manufacturing industry urgently needs to advance the formation of a resource-saving and environmentally friendly industrial structure and manufacturing system through transformation and upgrading to support for green economic growth 5 . As an incentive-based industrial policy that emphasizes an innovation-driven and eco-civilized development path through the development and implementation of an intelligent and green manufacturing system, Made in China 2025 is a significant initiative for promoting the manufacturing industry’s transformation and upgrading, providing solid economic support for green economic growth 6 . To promote the effective implementation of this industrial policy, fully mobilize localities to explore new modes and paths of manufacturing development, and strengthen the urban manufacturing industry’s influential demonstration role in advancing the green transition, the Ministry of Industry and Information Technology of China successively launched 30 Made in China 2025 pilot cities (city clusters) in 2016 and 2017. The Pilot Demonstration Work Program for “Made in China 2025” Cities specified that significant results should be achieved within three to 5 years. After several years of implementation, has the Made in China 2025 pilot policy promoted green economic growth? What are the policy’s mechanisms of action? Are there differences in green economic growth effects in pilot cities based on various urban development characteristics? This study’s theoretical interpretation and empirical examination of the above questions can add to the growing body of related research and provide valuable insights for cities to comprehensively promote the transformation and upgrading of manufacturing industry to advance China’s high-quality development.

This study constructs an analytical framework at the theoretical level to analyze the impact of the Made in China 2025 strategy on urban green economic growth, and uses the double machine learning (ML) model to test its green economic growth effect. The contributions of this study are as follows. First, focusing on the field of urban green development, the study incorporates variables representing the potential economic and environmental effects of the Made in China 2025 policy into a unified framework to systematically examine the impact of the Made in China 2025 pilot policy on the urban green economic growth, providing a novel perspective for assessing the effects of industrial policies. Second, we investigate potential transmission mechanisms of the Made in China 2025 strategy affecting green economic growth from the perspectives of green technology advancement, energy consumption structure optimization, industrial structure upgrading, and environmental supervision strengthening, establishing a useful supplement for related research. Third, leveraging the advantage of ML algorithms in high-dimensional and nonparametric prediction, we apply a double ML model assess the policy effects of the Made in China 2025 strategy to avoid the “curse of dimensionality” and the inherent biases of traditional econometric models, and improve the credibility of our research conclusions.

The remainder of this paper is structured as follows. Section “ Literature review ” presents a literature review. Section “ Policy background and theoretical analysis ” details our theoretical analysis and research hypotheses. Section “ Empirical strategy ” introduces the model setting and variables selection for the study. Section “ Empirical result ” describes the findings of empirical testing and analyzes the results. Section “ Conclusion and policy recommendation ” summarizes our conclusions and associated policy implications.

Literature review

Measurement and influencing factors of green economic growth.

The Green Economy Report, which was published by the United Nations Environment Program in 2011, defined green economy development as facilitating more efficient use of natural resources and sustainable growth than traditional economic models, with a more active role in promoting combined economic development and environmental protection. The Organization for Economic Co-operation and Development defined green economic growth as promoting economic growth while ensuring that natural assets continue to provide environmental resources and services; a concept that is shared by a large number of institutions and scholars 7 , 8 , 9 . A considerable amount of research has assessed green economic growth, primarily using three approaches. First, single-factor indicators, such as sulfur dioxide emissions, carbon dioxide emissions intensity, and other quantified forms; however, this approach neglects the substitution of input factors such as capital and labor for the energy factor, which has certain limitations 5 , 10 . Second, studies have been based on neoclassical economic growth theory, incorporating factors of capital, technology, energy, and the environment, and constructing a green Solow model to measure green total factor productivity (GTFP) 11 , 12 . Third, based on neoclassical economic growth theory, some studies have simultaneously considered desirable and undesirable output, applying Shepard’s distance function, the directional distance function, and data envelopment analysis to measure GTFP 13 , 14 , 15 .

Economic growth is an extremely complex process, and green economic growth is also subject to a combination of multiple complex factors. Scholars have explored the influence mechanisms of green economic growth from perspectives of resource endowment 16 , technological innovation 17 , industrial structure 18 , human capital 19 , financial support 20 , government regulation 21 , and globalization 22 . In the field of policy effect assessment, previous studies have confirmed the green development effects of pilot policies such as innovative cities 23 , Broadband China 24 , smart cities 25 , and low-carbon cities 26 . However, few studies have focused on the impact of Made in China 2025 strategy on urban green economic growth and identified its underlying mechanisms.

The impact of Made in China 2025 strategy

Since the industrial policy of Made in China 2025 was proposed, scholars have predominantly focused on exploring its economic effects on technological innovation 27 , digital transformation 28 , and total factor productivity (TFP) 29 , while the potential environmental effects have been neglected. Chen et al. (2024) 30 found that Made in China 2025 promotes firm innovation through tax incentives, public subsidies, convenient financing, academic collaboration and talent incentives. Xu (2022) 31 point out that Made in China 2025 policy has the potential to substantially improve the green innovation of manufacturing enterprises, which can boost the green transformation and upgrading of China’s manufacturing industry. Li et al. (2024) 32 empirically investigates the positive effect of Made in China 2025 strategy on digital transformation and exploratory innovation in advanced manufacturing firms. Moreover, Liu and Liu (2023) 33 take “Made in China 2025” as an exogenous shock and find that the pilot policy has a positive impact on the high-quality development of enterprises and capital markets. Unfortunately, scholars have only discussed the impact of Made in China 2025 strategy on green development and environmental protection from a theoretical perspective and lack empirical analysis. Li (2018) 27 has compared Germany’s “Industry 4.0” and China’s “Made in China 2025”, and point out that “Made in China 2025” has clear goals, measures and sector focus. Its guiding principles are to enhance industrial capability through innovation-driven manufacturing, optimize the structure of Chinese industry, emphasize quality over quantity, train and attract talent, and achieve green manufacturing and environment. Therefore, it is necessary to systematically explore the impact and mechanism of Made in China 2025 strategy on urban green economic growth from both theoretical and empirical perspectives.

Causal inference based on double ML

The majority of previous studies have used traditional causal inference models to assess policy effects; however, some limitations are inherent to the application of these models. For example, the parallel trend test of the difference-in-differences model has stringent requirements on appropriate sample data; the synthetic control method can construct a virtual control group that conforms to the parallel trend, but it requires that the treatment group does not have the extreme value characteristics, and it is only applicable to “one-to-many” circumstances; and the propensity score matching (PSM) method involves a considerable amount of subjectivity in selecting matching variables. To compensate for the shortcomings of traditional models, scholars have started to explore the application of ML in the field of causal inference 34 , 35 , 36 , and double ML is a typical representative.

Double ML was formalized in 2018 34 , and the relevant research falls into two main categories. The first strand of literature applies double ML to assess causality concerning economic phenomena. Yang et al. (2020) 37 applied double ML using a gradient boosting algorithm to explore the average treatment effect of top-ranked audit firms, verifying its robustness compared with the PSM method. Zhang et al. (2022) 38 used double ML to quantify the impact of nighttime subway services on the nighttime economy, house prices, traffic accidents, and crime following the introduction of nighttime subway services in London in 2016. Farbmacher et al. (2022) 39 combined double ML with mediating effects analysis to assess the causal relationship between health insurance coverage and youth wellness and examine the indirect mechanisms of regular medical checkups, based on a national longitudinal health survey of youth conducted by the US Bureau of Labor Statistics. The second strand of literature has innovated methodological theory based on double ML. Chiang et al. (2022) 40 proposed an improved multidirectional cross-fitting double ML method, obtaining regression results for high-dimensional parameters while estimating robust standard errors for dual clustering, which can effectively adapt to multidirectional clustered sampled data and improve the validity of estimation results. Bodory et al. (2022) 41 combined dynamic analysis with double ML to measure the causal effects of multiple treatment variables over time, using weighted estimation to assess the dynamic treatment effects of specific subsamples, which enriched the dynamic quantitative extension of double ML.

In summary, previous research has conducted some useful investigations regarding the impact of socioeconomic policies on green development, but limited studies have explored the relationship between the Made in China 2025 strategy and green economic growth. This study takes 281 Chinese cities as the research object, and applies the super-slacks-based measure (SBM) model to quantify Chinese cities’ green economic growth from 2006 to 2021. Based on a quasi-natural experiment of Made in China 2025 pilot policy implementation, we use the double ML model to test the impact and transmission mechanisms of the policy on urban green economic growth. We also conduct a heterogeneity analysis of cities based on different levels of manufacturing agglomeration, industrial intelligence, and digital finance. This study applies a novel approach and provides practical insights for research in the field of industrial policy assessment.

Policy background and theoretical analysis

Policy background.

The Made in China 2025 strategy aims to encourage and support local exploration of new paths and models for the transformation and upgrading of the manufacturing industry, and to drive the improvement of manufacturing quality and efficiency in other regions through demonstration effects. According to the Notice of Creating “Made in China 2025” National Demonstration Zones issued by the State Council, municipalities directly under the central government, sub-provincial cities, and prefecture-level cities can apply for the creation of demonstration zones. Cities with proximity and high industrial correlation can jointly apply for urban agglomeration demonstration zones. The Notice clarifies the goals and requirements for creating demonstration zones in areas such as green manufacturing, clean production, and environmental protection. In 2016, Ningbo became the first Made in China 2025 pilot city, and a total of 12 cities and 4 city clusters were included in the list of Made in China 2025 national demonstration zones. In 2018, the State Council issued the Evaluation Guidelines for “Made in China 2025” National Demonstration Zone, which further clarified the evaluation process and indicator system of the demonstration zone. Seven primary indicators and 29 secondary indicators were formulated, including innovation driven, quality first, green development, structural optimization, talent oriented, organizational implementation, and coordinated development of urban agglomerations. This indicator system can evaluate the creation process and overall effectiveness of pilot cities (city clusters), which is beneficial for the promotion of successful experiences and models in demonstration areas.

Advancing green urban development is a complex systematic project that requires structural adjustment and technological and institutional changes in the socioeconomic system 42 . The Made in China 2025 strategy emphasizes the development and application of smart and green manufacturing systems, which can unblock technological bottlenecks in the manufacturing sector in terms of industrial production, energy consumption, and waste emissions, and empower cities to operate in a green manner. In addition, the Made in China 2025 policy established requirements for promoting technological innovation to advance energy saving and environmental protection, improving the rate of green energy use, transforming traditional industries, and strengthening environmental supervision. For pilot cities, green economy development requires the support of a full range of positive factors. Therefore, this study analyzes the mechanisms by which the Made in China 2025 strategy affects urban green economic growth from the four paths of green technology advancement, energy consumption structure optimization, industrial structure upgrading, and environmental supervision strengthening.

Theoretical analysis and research hypotheses

As noted, the Made in China 2025 strategy emphasizes strengthening the development and application of energy-saving and environmental protection technologies to advance cleaner production. Pilot cities are expected to prioritize the driving role of green innovation, promote clustering carriers and innovation platforms for high-tech enterprises, and guide the progress of enterprises’ implementation of green technology. Specifically, pilot cities are encouraged to optimize the innovation environment by increasing scientific and technological investment and financial subsidies in key areas such as smart manufacturing and high-end equipment and strengthening intellectual property protection to incentivize enterprises to conduct green research and development (R&D) activities. These activities subsequently promote the development of green innovation technologies and industrial transformation 43 . Furthermore, since quality human resources are a core aspect of science and technology innovation 44 , pilot cities prioritize the cultivation and attraction of talent to establish a stable human capital guarantee for enterprises’ ongoing green technology innovation, transform and upgrade the manufacturing industry, and advance green urban development. Green technology advances also contribute to urban green economic growth. First, green technology facilitates enterprises’ adoption of improved production equipment and innovation in green production technology, accelerating the change of production mode and driving the transformation from traditional crude production to a green and intensive approach 45 , promoting green urban development. Second, green technology advancement accelerates green innovations such as clean processes, pollution control technologies, and green equipment, and facilitates the effective supply of green products, taking full advantage of the benefits of green innovations 46 and forming a green economic development model to achieve urban green economic growth.

The Made in China 2025 pilot policy endeavors to continuously increase the rate of green and low-carbon energy use and reduce energy consumption. Under target constraints of energy saving and carbon control, pilot cities will accelerate the cultivation of high-tech industries in green environmental protection and high-end equipment manufacturing with advantages of sustainability and low resource inputs 47 to improve the energy consumption structure. Pilot cities also advance new energy sector development by promoting clean energy projects, subsidizing new energy consumption, and supporting green infrastructure construction and other policy measures 48 to optimize the energy consumption structure. Energy consumption structure optimization can have a profound impact on green economy development. Optimization means that available energy tends to be cleaner, which can reduce the manufacturing industry’s dependence on traditional fossil energy and raise the proportion of clean energy 49 , ultimately promoting green urban development. Pilot cities also provide financial subsidies for new energy technology R&D, which promotes the innovation and application of new technologies, energy-saving equipment, efficient resource use, and energy-saving diagnostics, which allow enterprises to save energy and reduce consumption and improve energy use efficiency and TFP 50 , advancing the growth of urban green economy.

At its core, the Made in China 2025 strategy promotes the transformation and upgrading of the manufacturing sector. Pilot cities guide and develop technology-intensive high-tech industries, adjust the proportion of traditional heavy industry, and improve the urban industrial structure. Pilot cities also implement the closure, merger, and transformation of pollution-intensive industries; guide the fission of professional advantages of manufacturing enterprises 51 ; and expand the establishment and development of service-oriented manufacturing and productive service industries to promote the evolution of the industrial structure toward rationalization and high-quality development 52 . Upgrading the industrial structure can also contribute to urban green economic growth. First, industrial structure upgrading promotes the transition from labor- and capital-intensive industries to knowledge- and technology-intensive industries, which optimizes the industrial distribution patterns of energy consumption and pollutant emissions and promotes the transformation of economic growth dynamics and pollutant emissions control, providing a new impetus for cities’ sustainable development 53 . Second, changes in industrial structure and scale can have a profound impact on the type and quantity of pollutant emissions. By introducing high-tech industries, service-oriented manufacturing, and production-oriented service industries, pilot cities can promote the transformation of pollution-intensive industries, promoting the adjustment and optimization of industrial structure and scale 54 to achieve the purpose of driving green urban development.

The Made in China 2025 strategy proposes strengthening green supervision and conducting green evaluations, establishing green development goals for the manufacturing sector in terms of emissions and consumption reduction and water conservation. This requires pilot cities to implement stringent environmental regulatory policies, such as higher energy efficiency and emissions reduction targets and sewage taxes and charges, strict penalties for excess emissions, and project review criteria 55 , which consolidates the effectiveness of green development. Under the framework of environmental authoritarianism, strengthening environmental supervision is a key measure for achieving pollution control and improving environmental quality 56 . Therefore, environmental regulatory enhancement can help cities achieve green development goals. First, according to the Porter hypothesis 57 , strong environmental regulatory policies encourage firms to internalize the external costs of environmental supervision, stimulate technological innovation, and accelerate R&D and application of green technologies. This response helps enterprises improve input–output efficiency, achieve synergy between increasing production and emissions reduction, partially or completely offset the “environmental compliance cost” from environmental supervision, and realize the innovation compensation effect 58 . Second, strict environmental regulations can effectively mitigate the complicity of local governments and enterprises in focusing on economic growth while neglecting environmental protection 59 , urging local governments to constrain enterprises’ emissions, which compels enterprises to conduct technological innovation and pursue low-carbon transformation, promoting urban green economic growth.

Based on the above analysis, we propose the mechanisms that promote green economic growth through Made in China 2025 strategy, as shown in Fig.  1 . The proposed research hypotheses are as follows:

Mechanism analysis of Made in China 2025 strategy and green economic growth.

Hypothesis 1

The Made in China 2025 strategy promotes urban green economic growth.

Hypothesis 2

The Made in China 2025 strategy drives urban green economic growth through four channels: promoting green technology advancement, optimizing energy consumption structure, upgrading industrial structure, and strengthening environmental supervision.

Empirical strategy

Double ml model.

Compared with traditional causal inference models, double ML has unique advantages in variable selection and model estimation, and is also more applicable to the research problem of this study. Green economic growth is a comprehensive indicator of transformative urban growth that is influenced by many socioeconomic factors. To ensure the accuracy of our policy effects estimation, the interference of other factors on urban green economic growth must be controlled as much as possible; however, when introducing high-dimensional control variables, traditional regression models may face the “curse of dimensionality” and multicollinearity, rendering the accuracy of the estimates questionable. Double ML uses ML and regularization algorithms to automatically filter the preselected set of high-dimensional control variables to obtain an effective set of control variables with higher prediction accuracy. This approach avoids the “curse of dimensionality” caused by redundant control variables and mitigates the estimation bias caused by the limited number of primary control variables 39 . Furthermore, nonlinear relationships between variables are the norm in the evolution of economic transition, and ordinary linear regression may suffer from model-setting bias producing estimates that lack robustness. Double ML effectively overcomes the problem of model misspecification by virtue of the advantages of ML algorithms in handling nonlinear data 37 . In addition, based on the idea of instrumental variable functions, two-stage predictive residual regression, and sample split fitting, double ML mitigates the “regularity bias” in ML estimation and ensures unbiased estimates of the treatment coefficients in small samples 60 .

Based on the analysis above, this study uses the double ML model to assess the policy effects of the Made in China 2025 strategy. The partial linear double ML model is constructed as follows:

where i denotes the city, t denotes the year, and Y it represents green economic growth. Policy it represents the policy variable of Made in China 2025, which is set as 1 if the pilot is implemented and 0 otherwise. θ 0 is the treatment coefficient that is the focus of this study. X it denotes the set of high-dimensional control variables, and the ML algorithm is used to estimate the specific functional form \(\hat{g}(X_{it} )\) . U it denotes the error term with a conditional mean of zero.

Direct estimation of Eqs. ( 1 ) and ( 2 ) yields the following estimate of the treatment coefficient:

where n denotes the sample size.

Notably, the double ML model uses a regularization algorithm to estimate the specific functional form \(\hat{g}(X_{it} )\) , which prevents the variance of the estimate from being too large, but inevitably introduces a “regularity bias,” resulting in a biased estimate. To speed up the convergence of the \(\hat{g}(X_{it} )\) directions so that the estimates of the treatment coefficients satisfy unbiasedness with small samples, the following auxiliary regression is constructed:

where \(m(X_{it} )\) is the regression function of the treatment variable on the high-dimensional control variable, using ML algorithms to estimate the specific functional form \(\hat{m}(X_{it} )\) . V it is the error term with a conditional mean of zero.

The specific operation process follows three stages. First, we use the ML algorithm to estimate the auxiliary regression \(\hat{m}(X_{it} )\) and take its residuals \(\hat{V}_{it} = Policy_{it} - \hat{m}(X_{it} )\) . Second, we use the ML algorithm to estimate \(\hat{g}(X_{it} )\) and change the form of the main regression \(Y_{it} - \hat{g}(X_{it} ) = \theta_{0} Policy_{it} + U_{it}\) . Finally, we regress \(\hat{V}_{it}\) as an instrumental variable for Policy it , obtaining unbiased estimates of the treatment coefficients as follows:

Variable selection

• Green economic growth

We apply the super-SBM model to measure urban green economic growth. The super-SBM model is compatible with radial and nonradial characteristics, which avoids inflated results due to ignoring slack variables and deflated results due to ignoring the linear relationships between elements, and can truly reflect relative efficiency 61 . The SBM model reflects the nature of green economic growth more accurately compared with other models, and has been widely adopted by scholars 62 . The expression of the super-SBM model considering undesirable output is as follows:

where x is the input variable; y and z are the desirable and undesirable output variables, respectively; m denotes the number of input indicators; s 1 and s 2 represent the respective number of indicators for desirable and undesirable outputs; k denotes the period of production; i , r , and t are the decision units for the inputs, desirable outputs, and undesirable outputs, respectively; \(s^{ - }\) , \(s^{ + }\) , and \(s^{z - }\) are the respective slack variables for the inputs, desirable outputs, and undesirable outputs; and γ is a vector of weights. A larger \(\rho_{SE}\) value indicates greater efficiency. If \(\rho_{SE}\)  = 1, the decision unit is effective; if \(\rho_{SE}\)  < 1, the decision unit is relatively ineffective, indicating a loss of efficiency.

Referencing Sarkodie et al. (2023) 63 , the evaluation index system of green economic growth is constructed as shown in Table 1 .

Made in China 2025 pilot policy

The list of Made in China 2025 pilot cities (city clusters) published by the Ministry of Industry and Information Technology of China in 2016 and 2017 is matched with the city-level data to obtain 30 treatment group cities and 251 control group cities. The policy dummy variable of Made in China 2025 is constructed by combining the implementation time of the pilot policies.

Mediating variables

This study also examines the transmission mechanism of the Made in China 2025 strategy affecting urban green economic growth from four perspectives, including green technology advancement, energy consumption structure optimization, industrial structure upgrading, and strengthening of environmental supervision. (1) The number of green patent applications is adopted to reflect green technology advancement. (2) Energy consumption structure is quantified using the share of urban domestic electricity consumption in total energy consumption. (3) The industrial structure upgrading index is calculated using the formula \(\sum\nolimits_{i = 1}^{3} {i \times (GDP_{i} /GDP)}\) , where GDP i denotes the added value of primary, secondary, or tertiary industries. (4) The frequency of words related to the environment in government work reports is the proxy for measuring the intensity of environmental supervision 64 .

Control variables

Double ML can effectively accommodate the case of high-dimensional control variables using regularization algorithms. To control for the effect of other urban characteristics on green economic growth, this study introduces the following 10 control variables. We measure education investment by the ratio of education expenditure to GDP. Technology investment is the ratio of technology expenditure to GDP. The study measures urbanization using the share of urban built-up land in the urban area. Internet penetration is the number of internet users as a share of the total population at the end of the year. We measure resident consumption by the total retail sales of consumer goods per capita. The unemployment rate is the ratio of the number of registered unemployed in urban areas at the end of the year to the total population at the end of the year. Financial scale is the ratio of the balance of deposits and loans of financial institutions at the end of the year to the GDP. Human capital is the natural logarithm of the number of students enrolled in elementary school, general secondary schools, and general tertiary institutions per 10,000 persons. Transportation infrastructure is the natural logarithm of road and rail freight traffic. Finally, openness to the outside world is reflected by the ratio of actual foreign investment to GDP. Quadratic terms for the control variables are also included in the regression analysis to improve the accuracy of the model’s fit. We introduce city and time fixed effects as individual and year dummy variables to avoid missing information on city and time dimensions.

Data sources

This study uses 281 Chinese cities spanning from 2006 to 2021 as the research sample. Data sources include the China City Statistical Yearbook, the China Economic and Social Development Statistics Database, and the EPS Global Statistics Database. We used the average annual growth rate method to fill the gaps for the minimal missing data. To remove the effects of price changes, all data measured in monetary units are deflated using the consumer price index for each province for the 2005 base period. The descriptive statistics of the data are presented in Table 2 .

Empirical result

Baseline results.

The sample split ratio of the double ML model is set to 1:4, and we use the Lasso algorithm to predict and solve the main and auxiliary regressions, presenting the results in Table 3 . Column (1) does not control for fixed effects or control variables, column (2) introduces city and time fixed effects, and columns (3) and (4) add control variables to columns (1) and (2), respectively. The regressions in columns (1) and (2) are highly significant, regardless of whether city and time fixed effects are controlled. Column (4) controls for city fixed effects, time fixed effects, and the primary term of the control variable over the full sample interval, revealing that the regression coefficient of the Made in China 2025 pilot policy on green economic growth is positive and significant at the 1% level, confirming that the Made in China 2025 strategy significantly promotes urban green economic growth. Column (5) further incorporates the quadratic terms of the control variables and the regression coefficients remain significantly positive with little change in values. Therefore, Hypothesis 1 is verified.

Parallel trend test

The prerequisite for the establishment of policy evaluation is that the development status of cities before the pilot policy is introduced is similar. Referring to Liu et al. (2022) 29 , we adopt a parallel trend test to verify the effectiveness of Made in China 2025 pilot policy. Figure  2 shows the result of parallel trend test. None of the coefficient estimates before the Made in China 2025 pilot policy are significant, indicating no significant difference between the level of green economic growth in pilot and nonpilot cities before implementing the policy, which passes the parallel trend test. The coefficient estimates for all periods after the policy implementation are significantly positive, indicating that the Made in China 2025 pilot policy can promote urban green economic growth.

Parallel trend test.

Robustness tests

Replace explained variable.

Referencing Oh and Heshmati (2010) 65 and Tone and Tsutsui (2010) 66 , we use the Malmquist–Luenberger index under global production technology conditions (GML) and an epsilon-based measure (EBM) model to recalculate urban green economic growth. The estimation results in columns (1) and (2) of Table 4 show that the estimated coefficients of the Made in China 2025 pilot policy remain significantly positive after replacing the explanatory variables, validating the robustness of the baseline findings.

Adjusting the research sample

Considering the large gaps in the manufacturing development base between different regions in China, using all cities in the regression analysis may lead to biased estimation 67 . Therefore, we exclude cities in seven provinces with a poor manufacturing development base (Gansu, Qinghai, Ningxia, Xinjiang, Tibet, Yunnan, and Guizhou) and four municipalities with a better development base (Beijing, Tianjin, Shanghai, and Chongqing). The other city samples are retained to rerun the regression analysis, and the results are presented in column (3) of Table 4 . The first batch of pilot cities of the Made in China 2025 strategy was released in 2016, and the second batch of pilot cities was released in 2017. To exclude the effect of point-in-time samples that are far from the time of policy promulgation, the regression is also rerun by restricting the study interval to the three years before and after the promulgation of the policy (2013–2020), and the results are presented in column (4) of Table 4 . The coefficients of the Made in China 2025 pilot policy effect on urban green economic growth decrease after adjusting for the city sample and the time interval, but remain significantly positive at the 1% level. This, once again, verifies the robustness of the benchmark regression results.

Eliminating the impact of potential policies

During the same period of the Made in China 2025 strategy implementation, urban green economy growth may be affected by other relevant policies. To ensure the accuracy of the policy effect estimates, four representative policy categories overlapping with the sample period, including smart cities, low-carbon cities, Broadband China, and innovative cities, were collected and organized. Referencing Zhang and Fan (2023) 25 , dummy variables for these policies are included in the benchmark regression model and the results are presented in Table 5 . The estimated coefficient of the Made in China 2025 pilot policy decreases after controlling for the effects of related policies, but remains significantly positive at the 1% level. This suggests that the positive impact of the Made in China 2025 strategy on urban green economic growth, although overestimated, does not affect the validity of the study’s findings.

Reset double ML model

To avoid the impact of the double ML model imparting bias on the conclusions, we conduct robustness tests by varying the sample splitting ratio, the ML algorithm, and the model estimation form. First, we change the sample split ratio of the double ML model from 1:4 to 3:7 and 1:3. Second, we replace the Lasso ML algorithm with random forest (RF), gradient boosting (GBT), and BP neural network (BNN). Third, we replace the partial linear model based on the dual ML with a more generalized interactive model, using the following main and auxiliary regressions for the analysis:

among them, the meanings of each variable are the same as Eqs. ( 1 ) and ( 2 ).

The estimated coefficients for the treatment effects are obtained from the interactive model as follows:

Table 6 presents the regression results after resetting the double ML model, revealing that the sample split ratio, ML algorithm, and the model estimation form in double ML model did not affect the conclusion that the Made in China 2025 strategy promotes urban green economic growth, and only alters the magnitude of the policy effect, once again validating the robustness of our conclusions.

Difference-in-differences model

To further verify the robustness of the estimation results, we use traditional econometric models for regression. Based on the difference-in-differences (DID) model, a synthetic difference-in-differences (SDID) model is constructed by combining the synthetic control method 68 . It constructs a composite control group with a similar pre-trend to the treatment group by linearly combining several individuals in the control group, and compares it with the treatment group 69 . Table 7 presents the regression results of traditional DID model and SDID model. The estimated coefficient of the Made in China 2025 policy remains significantly positive at the 1% level, which once again verifies the robustness of the study’s findings.

Mechanism verification

This section conducts mechanism verification from four perspectives of green technology advancement, energy consumption structure, industrial structure, and environmental supervision. The positive impacts of the Made in China 2025 strategy on green technology advancement, energy consumption structure optimization, industrial structure upgrading, and strengthening environmental supervision are empirically examined using a dual ML model (see Table A.1 in the Online Appendix for details). Referencing Farbmacher et al. (2022) 39 for causal mediating effect analysis of double ML (see the Appendix for details), we test the transmission mechanism of the Made in China 2025 strategy on green economic growth based on the Lasso algorithm, presenting the results in Table 8 . The findings show that the total effects under different mediating paths are all significantly positive at the 1% level, verifying that the Made in China 2025 strategy positively promotes urban green economic growth.

Mechanism of green technology advancement

The indirect effect of green technological innovation is significantly positive for both the treatment and control groups. After stripping out the path of green technology advancement, the direct effects of the treatment and control groups remain significantly positive, indicating that the increase in the level of green technological innovation brought about by the Made in China 2025 strategy significantly promotes urban green economic growth. The Made in China 2025 strategy proposes to strengthen financial and tax policy support, intellectual property protection, and talent training systems. Through the implementation of policy incentives, pilot cities have fostered the concentration of high-technology enterprises and scientific and technological talent cultivation, exerting a knowledge spillover effect that further promotes green technology advancement. At the same time, policy preferences have stimulated the demand for innovation in energy conservation and emissions reduction, which raises enterprises’ motivation to engage in green innovation activities. Green technology advancement helps cities achieve an intensive development model, bringing multiple dividends such as lower resource consumption, reduced pollution emissions, and improved production efficiency, which subsequently promotes green economic growth.

Mechanism of energy consumption structure

The indirect effect of energy consumption structure is significantly positive for the treatment and control groups, while the direct effect of the Made in China 2025 pilot policy on green economic growth remains significantly positive, indicating that the policy promotes urban green economic growth through energy consumption structure optimization. The policy encourages the introduction of clean energy into production processes, reducing pressure on enterprise performance and the cost of clean energy use, which helps enterprises to reduce traditional energy consumption that is dominated by coal and optimize the energy structure to promote green urban development.

Mechanism of industrial structure

The indirect effects of industrial structure on the treatment and control groups are significantly positive. After stripping out the path of industrial structure upgrading, the direct effects remain significantly positive for both groups, indicating that the Made in China 2025 strategy promotes urban green economic growth through industrial structure optimization. Deepening the restructuring of the manufacturing industry is a strategic task specified in Made in China 2025. Pilot cities focus on transforming and guiding the traditional manufacturing industry toward high-end, intelligent equipment upgrades and digital transformation, driving the regional industrial structure toward rationalization and advancement to achieve rational allocation of resources. Upgrading industrial structure is a prerequisite for cities to advance intensive growth and sustainable development. By assuming the roles of “resource converter” and “pollutant controller,” industrial upgrading can continue to release the dividends of industrial structure, optimize resource allocation, and improve production efficiency, establishing strong support for green economic growth.

Mechanism of environmental supervision

The treatment and control groups of environmental supervision has a positive indirect effect in the process of the Made in China 2025 pilot policy affecting green economic growth that is significant at the 1% level, affirming the transmission path of environmental supervision. The Made in China 2025 strategy states that energy consumption, material consumption, and pollutant emissions per unit of industrial added value in key industries should reach the world’s advanced level by 2025. This requires pilot cities to consolidate and propagate the effectiveness of green development by strengthening environmental supervision while promoting the manufacturing sector’s green development. Strengthening environmental supervision promotes enterprises’ energy saving and emissions reduction through innovative compensation effects, while restraining enterprises’ emissions behaviors by tightening environmental protection policies, promoting environmental legislation, and increasing penalties to advance green urban development. Based on the above analysis, Hypothesis 2 is validated.

Heterogeneity analysis

Heterogeneity of manufacturing agglomeration.

To reduce production and transaction costs and realize economies of scale and scope, the manufacturing industry tends to accelerate its growth through agglomeration, exerting an “oasis effect” 70 . Cities with a high degree of manufacturing agglomeration are prone to scale and knowledge spillover effects, which amplify the agglomeration functions of talent, capital, and technology, strengthening the effectiveness of pilot policies. Based on this, we use the locational entropy of manufacturing employees to measure the degree of urban manufacturing agglomeration in the year (2015) before policy implementation, using the median to divide the full sample of cities into high and low agglomeration groups. Columns (1) and (2) in Table 9 reveal that the Made in China 2025 pilot policy has a stronger effect in promoting green economic growth in cities with high manufacturing concentration compared to those with low concentration. The rationale for this outcome may be that cities with a high concentration of manufacturing industries has large population and developed economy, which is conducive to leveraging agglomeration economies and knowledge spillover effects. Meanwhile, they are able to offer greater policy concessions by virtue of economic scale, public services, infrastructure, and other advantages. These benefits can attract the clustering of productive services and the influx of innovative elements such as R&D talent, accelerating the transformation and upgrading of the manufacturing industry and the integration and advancement of green technologies, empowering the green urban development.

Heterogeneity of industrial intelligence

As a landmark technology for the integration of the new scientific and technological revolution with manufacturing, industrial intelligence is a new approach for advancing the green transformation of manufacturing production methods. Based on this, we use the density of industrial robot installations to measure the level of industrial intelligence in cities in the year (2015) prior to policy implementation 71 , using the median to classify the full sample of cities into high and low level groups. Columns (3) and (4) in Table 9 reveals that the Made in China 2025 pilot policy has a stronger driving effect on the green economic growth of highly industrial intelligent cities. The rationale for this outcome may be that with the accumulation of smart factories, technologies, and equipment, a high degree of industrial intelligence is more likely to leverage the green development effects of pilot policies. For cities where the development of industrial intelligence is in its infancy or has not yet begun, the cost of information and knowledge required for enterprises to undertake technological R&D is higher, reducing the motivation and incentive to conduct innovative activities, diminishing the pilot policy’s contribution to green economic growth.

Heterogeneity of digital finance

As a fusion of traditional finance and information technology, digital finance has a positive impact on the development of the manufacturing industry by virtue of its advantages of low financing thresholds, fast mobile payments, and wide range of services 72 . Cities with a high degree of digital finance development have abundant financial resources and well-developed financial infrastructure that provide enterprises with more complete financial services, with subsequent influence on the effects of pilot policies. We use the Peking University Digital Inclusive Finance Index to measure the level of digital financial development in cities in the year (2015) prior to policy implementation, using the median to divide the full sample of cities into high and low level groups. Columns (5) and (6) in Table 9 reveal that the Made in China 2025 pilot policy has a stronger driving effect on the green economic growth of cities with highly developed digital finance. The rationale for this outcome may be that cities with a high degree of digital finance development can fully leverage the universality of financial resources, provide financial supply for environmentally friendly and technology-intensive enterprises, effectively alleviate the mismatch of financial capital supply, and provide financial security for enterprises to conduct green technology R&D. Digital finance also makes enterprises’ information more transparent through a rich array of data access channels, which strengthens government pollution regulation and public environmental supervision and compels enterprises to engage in green technological innovation to promote green economic growth.

Conclusion and policy recommendation

Conclusions.

This study examines the impact of the Made in China 2025 strategy on urban green economic growth using the double ML model based on panel data for 281 Chinese cities from 2006 to 2021. The relevant research results are threefold. First, the Made in China 2025 strategy significantly promotes urban green economic growth; a conclusion that is supported by a series of robustness tests. Second, regarding mechanisms, the Made in China 2025 strategy promotes urban green economic growth through green technology advancement, energy consumption structure optimization, industrial structure upgrading, and strengthening of environmental supervision. Third, the heterogeneity analysis reveals that the Made in China 2025 strategy has a stronger driving effect on green economic growth for cities with a high concentration of manufacturing and high degrees of industrial intelligence and digital finance.

policy recommendations

We next propose specific policy recommendations based on our findings. First, policymakers should summarize the experience of building pilot cities and create a strategic model to advance the transformation and upgrading of the manufacturing industry to drive green urban development. The Made in China 2025 pilot policy effectively promotes green economic growth and highlights the significance of the transformation and upgrading of the manufacturing industry to empower sustainable urban development. The government should strengthen the model and publicize summaries of successful cases of manufacturing development in pilot cities to promote the experience of manufacturing transformation and upgrading by producing typical samples to guide the transformation of the manufacturing industry to intelligence and greening. Policies should endeavor to optimize the industrial structure and production system of the manufacturing industry to create a solid real economy support for high-quality urban development.

Second, policymakers should explore the multidimensional driving paths of urban green economic growth and actively stimulate the green development dividend of pilot policies by increasing support for enterprise-specific technologies, subsidizing R&D in areas of energy conservation and emissions reduction, consumption reduction and efficiency, recycling and pollution prevention, and promoting the progress of green technologies. The elimination of outdated production capacity must be accelerated and the low-carbon transformation of traditional industries must be targeted, while guiding the clustering of high-tech industries, optimizing cities’ industrial structure, and driving industrial structure upgrading. Policymakers can regulate enterprises’ production practices and enhance the effectiveness of environmental supervision by improving the system of environmental information disclosure and mechanisms of rewards and penalties for pollution discharge. In addition, strategies should consider cities’ own resource endowment, promote large-scale production of new energy, encourage enterprises to increase the proportion of clean energy use, and optimize the structure of energy consumption.

Third, policymakers should engage a combination of urban development characteristics and strategic policy implementation to empower green urban development, actively promoting optimization of manufacturing industry structure, and accelerating the development of high-technology industries under the guidance of policies and the market to promote high-quality development and agglomeration of the manufacturing industry. At the same time, the government should strive to popularize the industrial internet, promote the construction of smart factories and the application of smart equipment, increase investment in R&D to advance industrial intelligence, and actively cultivate new modes and forms of industrial intelligence. In addition, new infrastructure construction must be accelerated, the application of information technology must be strengthened, and digital financial services must be deepened to ease the financing constraints for enterprises conducting R&D on green technologies and to help cities develop in a high-quality manner.

Data availability

The datasets used or analysed during the current study are available from the corresponding author on reasonable request.

Cheng, K. & Liu, S. Does urbanization promote the urban–rural equalization of basic public services? Evidence from prefectural cities in China. Appl. Econ. 56 (29), 3445–3459. https://doi.org/10.1080/00036846.2023.2206625 (2023).

Article   Google Scholar  

Yin, X. & Xu, Z. An empirical analysis of the coupling and coordinative development of China’s green finance and economic growth. Resour. Policy 75 , 102476. https://doi.org/10.1016/j.resourpol.2021.102476 (2022).

Fernandes, C. I., Veiga, P. M., Ferreira, J. J. M. & Hughes, M. Green growth versus economic growth: Do sustainable technology transfer and innovations lead to an imperfect choice?. Bus. Strateg. Environ. 30 (4), 2021–2037. https://doi.org/10.1002/bse.2730 (2021).

Orsatti, G., Quatraro, F. & Pezzoni, M. The antecedents of green technologies: The role of team-level recombinant capabilities. Res. Policy 49 (3), 103919. https://doi.org/10.1016/j.respol.2019.103919 (2020).

Lin, B. & Zhou, Y. Measuring the green economic growth in China: Influencing factors and policy perspectives. Energy 241 (15), 122518. https://doi.org/10.1016/j.energy.2021.122518 (2022).

Fang, M. & Chang, C. L. Nexus between fiscal imbalances, green fiscal spending, and green economic growth: Empirical findings from E-7 economies. Econ. Change Restruct. 55 , 2423–2443. https://doi.org/10.1007/s10644-022-09392-6 (2022).

Qian, Y., Liu, J. & Forrest, J. Y. L. Impact of financial agglomeration on regional green economic growth: Evidence from China. J. Environ. Plan. Manag. 65 (9), 1611–1636. https://doi.org/10.1080/09640568.2021.1941811 (2022).

Awais, M., Afzal, A., Firdousi, S. & Hasnaoui, A. Is fintech the new path to sustainable resource utilisation and economic development?. Resour. Policy 81 , 103309. https://doi.org/10.1016/j.resourpol.2023.103309 (2023).

Ahmed, E. M. & Elfaki, K. E. Green technological progress implications on long-run sustainable economic growth. J. Knowl. Econ. https://doi.org/10.1007/s13132-023-01268-y (2023).

Shen, F. et al. The effect of economic growth target constraints on green technology innovation. J. Environ. Manag. 292 (15), 112765. https://doi.org/10.1016/j.jenvman.2021.112765 (2021).

Zhao, L. et al. Enhancing green economic recovery through green bonds financing and energy efficiency investments. Econ. Anal. Policy 76 , 488–501. https://doi.org/10.1016/j.eap.2022.08.019 (2022).

Ferreira, J. J. et al. Diverging or converging to a green world? Impact of green growth measures on countries’ economic performance. Environ. Dev. Sustain. https://doi.org/10.1007/s10668-023-02991-x (2023).

Article   PubMed   PubMed Central   Google Scholar  

Song, X., Zhou, Y. & Jia, W. How do economic openness and R&D investment affect green economic growth?—Evidence from China. Resour. Conserv. Recycl. 149 , 405–415. https://doi.org/10.1016/j.resconrec.2019.03.050 (2019).

Xu, J., She, S., Gao, P. & Sun, Y. Role of green finance in resource efficiency and green economic growth. Resour. Policy 81 , 103349 (2023).

Zhou, Y., Tian, L. & Yang, X. Schumpeterian endogenous growth model under green innovation and its enculturation effect. Energy Econ. 127 , 107109. https://doi.org/10.1016/j.eneco.2023.107109 (2023).

Luukkanen, J. et al. Resource efficiency and green economic sustainability transition evaluation of green growth productivity gap and governance challenges in Cambodia. Sustain. Dev. 27 (3), 312–320. https://doi.org/10.1002/sd.1902 (2019).

Wang, K., Umar, M., Akram, R. & Caglar, E. Is technological innovation making world “Greener”? An evidence from changing growth story of China. Technol. Forecast. Soc. Change 165 , 120516. https://doi.org/10.1016/j.techfore.2020.120516 (2021).

Talebzadehhosseini, S. & Garibay, I. The interaction effects of technological innovation and path-dependent economic growth on countries overall green growth performance. J. Clean. Prod. 333 (20), 130134. https://doi.org/10.1016/j.jclepro.2021.130134 (2022).

Ge, T., Li, C., Li, J. & Hao, X. Does neighboring green development benefit or suffer from local economic growth targets? Evidence from China. Econ. Modell. 120 , 106149. https://doi.org/10.1016/j.econmod.2022.106149 (2023).

Lin, B. & Zhu, J. Fiscal spending and green economic growth: Evidence from China. Energy Econ. 83 , 264–271. https://doi.org/10.1016/j.eneco.2019.07.010 (2019).

Sohail, M. T., Ullah, S. & Majeed, M. T. Effect of policy uncertainty on green growth in high-polluting economies. J. Clean. Prod. 380 (20), 135043. https://doi.org/10.1016/j.jclepro.2022.135043 (2022).

Sarwar, S. Impact of energy intensity, green economy and blue economy to achieve sustainable economic growth in GCC countries: Does Saudi Vision 2030 matters to GCC countries. Renew. Energy 191 , 30–46. https://doi.org/10.1016/j.renene.2022.03.122 (2022).

Park, J. & Page, G. W. Innovative green economy, urban economic performance and urban environments: An empirical analysis of US cities. Eur. Plann. Stud. 25 (5), 772–789. https://doi.org/10.1080/09654313.2017.1282078 (2017).

Feng, Y., Chen, Z. & Nie, C. The effect of broadband infrastructure construction on urban green innovation: Evidence from a quasi-natural experiment in China. Econ. Anal. Policy 77 , 581–598. https://doi.org/10.1016/j.eap.2022.12.020 (2023).

Zhang, X. & Fan, D. Collaborative emission reduction research on dual-pilot policies of the low-carbon city and smart city from the perspective of multiple innovations. Urban Climate 47 , 101364. https://doi.org/10.1016/j.uclim.2022.101364 (2023).

Cheng, J., Yi, J., Dai, S. & Xiong, Y. Can low-carbon city construction facilitate green growth? Evidence from China’s pilot low-carbon city initiative. J. Clean. Prod. 231 (10), 1158–1170. https://doi.org/10.1016/j.jclepro.2019.05.327 (2019).

Li, L. China’s manufacturing locus in 2025: With a comparison of “Made-in-China 2025” and “Industry 4.0”. Technol. Forecast. Soc. Change 135 , 66–74. https://doi.org/10.1016/j.techfore.2017.05.028 (2018).

Wang, J., Wu, H. & Chen, Y. Made in China 2025 and manufacturing strategy decisions with reverse QFD. Int. J. Prod. Econ. 224 , 107539. https://doi.org/10.1016/j.ijpe.2019.107539 (2020).

Liu, X., Megginson, W. L. & Xia, J. Industrial policy and asset prices: Evidence from the Made in China 2025 policy. J. Bank. Finance 142 , 106554. https://doi.org/10.1016/j.jbankfin.2022.106554 (2022).

Chen, K. et al. How does industrial policy experimentation influence innovation performance? A case of Made in China 2025. Humanit. Soc. Sci. Commun. 11 , 40. https://doi.org/10.1057/s41599-023-02497-x (2024).

Article   CAS   Google Scholar  

Xu, L. Towards green innovation by China’s industrial policy: Evidence from Made in China 2025. Front. Environ. Sci. 10 , 924250. https://doi.org/10.3389/fenvs.2022.924250 (2022).

Li, X., Han, H. & He, H. Advanced manufacturing firms’ digital transformation and exploratory innovation. Appl. Econ. Lett. https://doi.org/10.1080/13504851.2024.2305665 (2024).

Liu, G. & Liu, B. How digital technology improves the high-quality development of enterprises and capital markets: A liquidity perspective. Finance Res. Lett. 53 , 103683 (2023).

Chernozhukov, V. et al. Double/debiased machine learning for treatment and structural parameters. Econom. J. 21 (1), C1–C68. https://doi.org/10.1111/ectj.12097 (2018).

Article   MathSciNet   Google Scholar  

Athey, S., Tibshirani, J. & Wager, S. Generalized random forests. Ann. Stat. 47 (2), 1148–1178. https://doi.org/10.1214/18-AOS1709 (2019).

Knittel, C. R. & Stolper, S. Machine learning about treatment effect heterogeneity: The case of household energy use. AEA Pap. Proc. 111 , 440–444 (2021).

Yang, J., Chuang, H. & Kuan, C. Double machine learning with gradient boosting and its application to the Big N audit quality effect. J. Econom. 216 (1), 268–283. https://doi.org/10.1016/j.jeconom.2020.01.018 (2020).

Zhang, Y., Li, H. & Ren, G. Quantifying the social impacts of the London Night Tube with a double/debiased machine learning based difference-in-differences approach. Transp. Res. Part A Policy Pract. 163 , 288–303. https://doi.org/10.1016/j.tra.2022.07.015 (2022).

Farbmacher, H., Huber, M., Lafférs, L., Langen, H. & Spindler, M. Causal mediation analysis with double machine learning. Econom. J. 25 (2), 277–300. https://doi.org/10.1093/ectj/utac003 (2022).

Chiang, H., Kato, K., Ma, Y. & Sasaki, Y. Multiway cluster robust double/debiased machine learning. J. Bus. Econ. Stat. 40 (3), 1046–1056. https://doi.org/10.1080/07350015.2021.1895815 (2022).

Bodory, H., Huber, M. & Lafférs, L. Evaluating (weighted) dynamic treatment effects by double machine learning. Econom. J. 25 (3), 628–648. https://doi.org/10.1093/ectj/utac018 (2022).

Waheed, R., Sarwar, S. & Alsaggaf, M. I. Relevance of energy, green and blue factors to achieve sustainable economic growth: Empirical study of Saudi Arabia. Technol. Forecast. Soc. Change 187 , 122184. https://doi.org/10.1016/j.techfore.2022.122184 (2023).

Taskin, D., Vardar, G. & Okan, B. Does renewable energy promote green economic growth in OECD countries?. Sustain. Account. Manag. Policy J. 11 (4), 771–798. https://doi.org/10.1108/SAMPJ-04-2019-0192 (2020).

Ding, X. & Liu, X. Renewable energy development and transportation infrastructure matters for green economic growth? Empirical evidence from China. Econ. Anal. Policy 79 , 634–646. https://doi.org/10.1016/j.eap.2023.06.042 (2023).

Ferguson, P. The green economy agenda: Business as usual or transformational discourse?. Environ. Polit. 24 (1), 17–37. https://doi.org/10.1080/09644016.2014.919748 (2015).

Pan, D., Yu, Y., Hong, W. & Chen, S. Does campaign-style environmental regulation induce green economic growth? Evidence from China’s central environmental protection inspection policy. Energy Environ. https://doi.org/10.1177/0958305X231152483 (2023).

Zhang, Q., Qu, Y. & Zhan, L. Great transition and new pattern: Agriculture and rural area green development and its coordinated relationship with economic growth in China. J. Environ. Manag. 344 , 118563. https://doi.org/10.1016/j.jenvman.2023.118563 (2023).

Li, J., Dong, K. & Dong, X. Green energy as a new determinant of green growth in China: The role of green technological innovation. Energy Econ. 114 , 106260. https://doi.org/10.1016/j.eneco.2022.106260 (2022).

Herman, K. S. et al. A critical review of green growth indicators in G7 economies from 1990 to 2019. Sustain. Sci. 18 , 2589–2604. https://doi.org/10.1007/s11625-023-01397-y (2023).

Mura, M., Longo, M., Zanni, S. & Toschi, L. Exploring socio-economic externalities of development scenarios. An analysis of EU regions from 2008 to 2016. J. Environ. Manag. 332 , 117327. https://doi.org/10.1016/j.jenvman.2023.117327 (2023).

Huang, S. Do green financing and industrial structure matter for green economic recovery? Fresh empirical insights from Vietnam. Econ. Anal. Policy 75 , 61–73. https://doi.org/10.1016/j.eap.2022.04.010 (2022).

Li, J., Dong, X. & Dong, K. Is China’s green growth possible? The roles of green trade and green energy. Econ. Res.-Ekonomska Istraživanja 35 (1), 7084–7108. https://doi.org/10.1080/1331677X.2022.2058978 (2022).

Zhang, H. et al. Promoting eco-tourism for the green economic recovery in ASEAN. Econ. Change Restruct. 56 , 2021–2036. https://doi.org/10.1007/s10644-023-09492-x (2023).

Article   ADS   Google Scholar  

Ahmed, F., Kousar, S., Pervaiz, A. & Shabbir, A. Do institutional quality and financial development affect sustainable economic growth? Evidence from South Asian countries. Borsa Istanbul Rev. 22 (1), 189–196. https://doi.org/10.1016/j.bir.2021.03.005 (2022).

Yuan, S., Li, C., Wang, M., Wu, H. & Chang, L. A way toward green economic growth: Role of energy efficiency and fiscal incentive in China. Econ. Anal. Policy 79 , 599–609. https://doi.org/10.1016/j.eap.2023.06.004 (2023).

Capasso, M., Hansen, T., Heiberg, J., Klitkou, A. & Steen, M. Green growth – A synthesis of scientific findings. Technol. Forecast. Soc. Change 146 , 390–402. https://doi.org/10.1016/j.techfore.2019.06.013 (2019).

Wei, X., Ren, H., Ullah, S. & Bozkurt, C. Does environmental entrepreneurship play a role in sustainable green development? Evidence from emerging Asian economies. Econ. Res. Ekonomska Istraživanja 36 (1), 73–85. https://doi.org/10.1080/1331677X.2022.2067887 (2023).

Iqbal, K., Sarfraz, M. & Khurshid,. Exploring the role of information communication technology, trade, and foreign direct investment to promote sustainable economic growth: Evidence from Belt and Road Initiative economies. Sustain. Dev. 31 (3), 1526–1535. https://doi.org/10.1002/sd.2464 (2023).

Li, Y., Zhang, J. & Lyu, Y. Toward inclusive green growth for sustainable development: A new perspective of labor market distortion. Bus. Strategy Environ. 32 (6), 3927–3950. https://doi.org/10.1002/bse.3346 (2023).

Chernozhukov, V. et al. Double/Debiased/Neyman machine learning of treatment effects. Am. Econ. Rev. 107 (5), 261–265. https://doi.org/10.1257/aer.p20171038 (2017).

Chen, C. Super efficiencies or super inefficiencies? Insights from a joint computation model for slacks-based measures in DEA. Eur. J. Op. Res. 226 (2), 258–267. https://doi.org/10.1016/j.ejor.2012.10.031 (2013).

Article   ADS   MathSciNet   Google Scholar  

Tone, K., Chang, T. & Wu, C. Handling negative data in slacks-based measure data envelopment analysis models. Eur. J. Op. Res. 282 (3), 926–935 (2020).

Sarkodie, S. A., Owusu, P. A. & Taden, J. Comprehensive green growth indicators across countries and territories. Sci. Data 10 , 413. https://doi.org/10.1038/s41597-023-02319-4 (2023).

Jiang, Z., Wang, Z. & Lan, X. How environmental regulations affect corporate innovation? The coupling mechanism of mandatory rules and voluntary management. Technol. Soc. 65 , 101575 (2021).

Oh, D. H. & Heshmati, A. A sequential Malmquist-Luenberger productivity index: Environmentally sensitive productivity growth considering the progressive nature of technology. Energy Econ. 32 (6), 1345–1355. https://doi.org/10.1016/j.eneco.2010.09.003 (2010).

Tone, K. & Tsutsui, M. An epsilon-based measure of efficiency in DEA - A third pole of technical efficiency. Eur. J. Op. Res. 207 (3), 1554–1563. https://doi.org/10.1016/j.ejor.2010.07.014 (2010).

Lv, C., Song, J. & Lee, C. Can digital finance narrow the regional disparities in the quality of economic growth? Evidence from China. Econ. Anal. Policy 76 , 502–521. https://doi.org/10.1016/j.eap.2022.08.022 (2022).

Arkhangelsky, D., Athey, S., Hirshberg, D. A., Imbens, G. W. & Wager, S. Synthetic difference-in-differences. Am. Econ. Rev. 111 (12), 4088–4118 (2021).

Abadie, A., Diamond, A. & Hainmueller, J. Synthetic control methods for comparative case studies: Estimating the effect of California’s tobacco control program. J. Am. Stat. Assoc. 105 (490), 493–505 (2010).

Article   MathSciNet   CAS   Google Scholar  

Fang, J., Tang, X., Xie, R. & Han, F. The effect of manufacturing agglomerations on smog pollution. Struct. Change Econ. Dyn. 54 , 92–101. https://doi.org/10.1016/j.strueco.2020.04.003 (2020).

Yang, S. & Liu, F. Impact of industrial intelligence on green total factor productivity: The indispensability of the environmental system. Ecol. Econ. 216 , 108021. https://doi.org/10.1016/j.ecolecon.2023.108021 (2024).

Zhang, P., Wang, Y., Wang, R. & Wang, T. Digital finance and corporate innovation: Evidence from China. Appl. Econ. 56 (5), 615–638. https://doi.org/10.1080/00036846.2023.2169242 (2024).

Download references

Acknowledgements

This work was supported by the Major Program of National Fund of Philosophy and Social Science of China (20&ZD133).

Author information

Authors and affiliations.

School of Public Finance and Taxation, Zhejiang University of Finance and Economics, Hangzhou, 310018, China

School of Economics, Xiamen University, Xiamen, 361005, China

Shucheng Liu

You can also search for this author in PubMed   Google Scholar

Contributions

J.Y.: Methodology, Validation. S.L.: Writing - Reviewing and Editing, Validation, Methodology. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Shucheng Liu .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary information., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Yuan, J., Liu, S. A double machine learning model for measuring the impact of the Made in China 2025 strategy on green economic growth. Sci Rep 14 , 12026 (2024). https://doi.org/10.1038/s41598-024-62916-0

Download citation

Received : 05 March 2024

Accepted : 22 May 2024

Published : 26 May 2024

DOI : https://doi.org/10.1038/s41598-024-62916-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Made in China 2025
• Industrial policy
• Double machine learning
• Causal inference

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing: Anthropocene newsletter — what matters in anthropocene research, free to your inbox weekly.

A systematic literature review of empirical research on ChatGPT in education

• Open access
• Published: 26 May 2024
• Volume 3 , article number  60 , ( 2024 )

Cite this article

You have full access to this open access article

• Yazid Albadarin   ORCID: orcid.org/0009-0005-8068-8902 1 ,
• Mohammed Saqr 1 ,
• Nicolas Pope 1 &
• Markku Tukiainen 1  

Over the last four decades, studies have investigated the incorporation of Artificial Intelligence (AI) into education. A recent prominent AI-powered technology that has impacted the education sector is ChatGPT. This article provides a systematic review of 14 empirical studies incorporating ChatGPT into various educational settings, published in 2022 and before the 10th of April 2023—the date of conducting the search process. It carefully followed the essential steps outlined in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA 2020) guidelines, as well as Okoli’s (Okoli in Commun Assoc Inf Syst, 2015) steps for conducting a rigorous and transparent systematic review. In this review, we aimed to explore how students and teachers have utilized ChatGPT in various educational settings, as well as the primary findings of those studies. By employing Creswell’s (Creswell in Educational research: planning, conducting, and evaluating quantitative and qualitative research [Ebook], Pearson Education, London, 2015) coding techniques for data extraction and interpretation, we sought to gain insight into their initial attempts at ChatGPT incorporation into education. This approach also enabled us to extract insights and considerations that can facilitate its effective and responsible use in future educational contexts. The results of this review show that learners have utilized ChatGPT as a virtual intelligent assistant, where it offered instant feedback, on-demand answers, and explanations of complex topics. Additionally, learners have used it to enhance their writing and language skills by generating ideas, composing essays, summarizing, translating, paraphrasing texts, or checking grammar. Moreover, learners turned to it as an aiding tool to facilitate their directed and personalized learning by assisting in understanding concepts and homework, providing structured learning plans, and clarifying assignments and tasks. However, the results of specific studies (n = 3, 21.4%) show that overuse of ChatGPT may negatively impact innovative capacities and collaborative learning competencies among learners. Educators, on the other hand, have utilized ChatGPT to create lesson plans, generate quizzes, and provide additional resources, which helped them enhance their productivity and efficiency and promote different teaching methodologies. Despite these benefits, the majority of the reviewed studies recommend the importance of conducting structured training, support, and clear guidelines for both learners and educators to mitigate the drawbacks. This includes developing critical evaluation skills to assess the accuracy and relevance of information provided by ChatGPT, as well as strategies for integrating human interaction and collaboration into learning activities that involve AI tools. Furthermore, they also recommend ongoing research and proactive dialogue with policymakers, stakeholders, and educational practitioners to refine and enhance the use of AI in learning environments. This review could serve as an insightful resource for practitioners who seek to integrate ChatGPT into education and stimulate further research in the field.

Avoid common mistakes on your manuscript.

1 Introduction

Educational technology, a rapidly evolving field, plays a crucial role in reshaping the landscape of teaching and learning [ 82 ]. One of the most transformative technological innovations of our era that has influenced the field of education is Artificial Intelligence (AI) [ 50 ]. Over the last four decades, AI in education (AIEd) has gained remarkable attention for its potential to make significant advancements in learning, instructional methods, and administrative tasks within educational settings [ 11 ]. In particular, a large language model (LLM), a type of AI algorithm that applies artificial neural networks (ANNs) and uses massively large data sets to understand, summarize, generate, and predict new content that is almost difficult to differentiate from human creations [ 79 ], has opened up novel possibilities for enhancing various aspects of education, from content creation to personalized instruction [ 35 ]. Chatbots that leverage the capabilities of LLMs to understand and generate human-like responses have also presented the capacity to enhance student learning and educational outcomes by engaging students, offering timely support, and fostering interactive learning experiences [ 46 ].

The ongoing and remarkable technological advancements in chatbots have made their use more convenient, increasingly natural and effortless, and have expanded their potential for deployment across various domains [ 70 ]. One prominent example of chatbot applications is the Chat Generative Pre-Trained Transformer, known as ChatGPT, which was introduced by OpenAI, a leading AI research lab, on November 30th, 2022. ChatGPT employs a variety of deep learning techniques to generate human-like text, with a particular focus on recurrent neural networks (RNNs). Long short-term memory (LSTM) allows it to grasp the context of the text being processed and retain information from previous inputs. Also, the transformer architecture, a neural network architecture based on the self-attention mechanism, allows it to analyze specific parts of the input, thereby enabling it to produce more natural-sounding and coherent output. Additionally, the unsupervised generative pre-training and the fine-tuning methods allow ChatGPT to generate more relevant and accurate text for specific tasks [ 31 , 62 ]. Furthermore, reinforcement learning from human feedback (RLHF), a machine learning approach that combines reinforcement learning techniques with human-provided feedback, has helped improve ChatGPT’s model by accelerating the learning process and making it significantly more efficient.

This cutting-edge natural language processing (NLP) tool is widely recognized as one of today's most advanced LLMs-based chatbots [ 70 ], allowing users to ask questions and receive detailed, coherent, systematic, personalized, convincing, and informative human-like responses [ 55 ], even within complex and ambiguous contexts [ 63 , 77 ]. ChatGPT is considered the fastest-growing technology in history: in just three months following its public launch, it amassed an estimated 120 million monthly active users [ 16 ] with an estimated 13 million daily queries [ 49 ], surpassing all other applications [ 64 ]. This remarkable growth can be attributed to the unique features and user-friendly interface that ChatGPT offers. Its intuitive design allows users to interact seamlessly with the technology, making it accessible to a diverse range of individuals, regardless of their technical expertise [ 78 ]. Additionally, its exceptional performance results from a combination of advanced algorithms, continuous enhancements, and extensive training on a diverse dataset that includes various text sources such as books, articles, websites, and online forums [ 63 ], have contributed to a more engaging and satisfying user experience [ 62 ]. These factors collectively explain its remarkable global growth and set it apart from predecessors like Bard, Bing Chat, ERNIE, and others.

In this context, several studies have explored the technological advancements of chatbots. One noteworthy recent research effort, conducted by Schöbel et al. [ 70 ], stands out for its comprehensive analysis of more than 5,000 studies on communication agents. This study offered a comprehensive overview of the historical progression and future prospects of communication agents, including ChatGPT. Moreover, other studies have focused on making comparisons, particularly between ChatGPT and alternative chatbots like Bard, Bing Chat, ERNIE, LaMDA, BlenderBot, and various others. For example, O’Leary [ 53 ] compared two chatbots, LaMDA and BlenderBot, with ChatGPT and revealed that ChatGPT outperformed both. This superiority arises from ChatGPT’s capacity to handle a wider range of questions and generate slightly varied perspectives within specific contexts. Similarly, ChatGPT exhibited an impressive ability to formulate interpretable responses that were easily understood when compared with Google's feature snippet [ 34 ]. Additionally, ChatGPT was compared to other LLMs-based chatbots, including Bard and BERT, as well as ERNIE. The findings indicated that ChatGPT exhibited strong performance in the given tasks, often outperforming the other models [ 59 ].

Furthermore, in the education context, a comprehensive study systematically compared a range of the most promising chatbots, including Bard, Bing Chat, ChatGPT, and Ernie across a multidisciplinary test that required higher-order thinking. The study revealed that ChatGPT achieved the highest score, surpassing Bing Chat and Bard [ 64 ]. Similarly, a comparative analysis was conducted to compare ChatGPT with Bard in answering a set of 30 mathematical questions and logic problems, grouped into two question sets. Set (A) is unavailable online, while Set (B) is available online. The results revealed ChatGPT's superiority in Set (A) over Bard. Nevertheless, Bard's advantage emerged in Set (B) due to its capacity to access the internet directly and retrieve answers, a capability that ChatGPT does not possess [ 57 ]. However, through these varied assessments, ChatGPT consistently highlights its exceptional prowess compared to various alternatives in the ever-evolving chatbot technology.

The widespread adoption of chatbots, especially ChatGPT, by millions of students and educators, has sparked extensive discussions regarding its incorporation into the education sector [ 64 ]. Accordingly, many scholars have contributed to the discourse, expressing both optimism and pessimism regarding the incorporation of ChatGPT into education. For example, ChatGPT has been highlighted for its capabilities in enriching the learning and teaching experience through its ability to support different learning approaches, including adaptive learning, personalized learning, and self-directed learning [ 58 , 60 , 91 ]), deliver summative and formative feedback to students and provide real-time responses to questions, increase the accessibility of information [ 22 , 40 , 43 ], foster students’ performance, engagement and motivation [ 14 , 44 , 58 ], and enhance teaching practices [ 17 , 18 , 64 , 74 ].

On the other hand, concerns have been also raised regarding its potential negative effects on learning and teaching. These include the dissemination of false information and references [ 12 , 23 , 61 , 85 ], biased reinforcement [ 47 , 50 ], compromised academic integrity [ 18 , 40 , 66 , 74 ], and the potential decline in students' skills [ 43 , 61 , 64 , 74 ]. As a result, ChatGPT has been banned in multiple countries, including Russia, China, Venezuela, Belarus, and Iran, as well as in various educational institutions in India, Italy, Western Australia, France, and the United States [ 52 , 90 ].

Clearly, the advent of chatbots, especially ChatGPT, has provoked significant controversy due to their potential impact on learning and teaching. This indicates the necessity for further exploration to gain a deeper understanding of this technology and carefully evaluate its potential benefits, limitations, challenges, and threats to education [ 79 ]. Therefore, conducting a systematic literature review will provide valuable insights into the potential prospects and obstacles linked to its incorporation into education. This systematic literature review will primarily focus on ChatGPT, driven by the aforementioned key factors outlined above.

However, the existing literature lacks a systematic literature review of empirical studies. Thus, this systematic literature review aims to address this gap by synthesizing the existing empirical studies conducted on chatbots, particularly ChatGPT, in the field of education, highlighting how ChatGPT has been utilized in educational settings, and identifying any existing gaps. This review may be particularly useful for researchers in the field and educators who are contemplating the integration of ChatGPT or any chatbot into education. The following research questions will guide this study:

What are students' and teachers' initial attempts at utilizing ChatGPT in education?

What are the main findings derived from empirical studies that have incorporated ChatGPT into learning and teaching?

2 Methodology

To conduct this study, the authors followed the essential steps of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA 2020) and Okoli’s [ 54 ] steps for conducting a systematic review. These included identifying the study’s purpose, drafting a protocol, applying a practical screening process, searching the literature, extracting relevant data, evaluating the quality of the included studies, synthesizing the studies, and ultimately writing the review. The subsequent section provides an extensive explanation of how these steps were carried out in this study.

2.1 Identify the purpose

Given the widespread adoption of ChatGPT by students and teachers for various educational purposes, often without a thorough understanding of responsible and effective use or a clear recognition of its potential impact on learning and teaching, the authors recognized the need for further exploration of ChatGPT's impact on education in this early stage. Therefore, they have chosen to conduct a systematic literature review of existing empirical studies that incorporate ChatGPT into educational settings. Despite the limited number of empirical studies due to the novelty of the topic, their goal is to gain a deeper understanding of this technology and proactively evaluate its potential benefits, limitations, challenges, and threats to education. This effort could help to understand initial reactions and attempts at incorporating ChatGPT into education and bring out insights and considerations that can inform the future development of education.

2.2 Draft the protocol

The next step is formulating the protocol. This protocol serves to outline the study process in a rigorous and transparent manner, mitigating researcher bias in study selection and data extraction [ 88 ]. The protocol will include the following steps: generating the research question, predefining a literature search strategy, identifying search locations, establishing selection criteria, assessing the studies, developing a data extraction strategy, and creating a timeline.

2.3 Apply practical screen

The screening step aims to accurately filter the articles resulting from the searching step and select the empirical studies that have incorporated ChatGPT into educational contexts, which will guide us in answering the research questions and achieving the objectives of this study. To ensure the rigorous execution of this step, our inclusion and exclusion criteria were determined based on the authors' experience and informed by previous successful systematic reviews [ 21 ]. Table 1 summarizes the inclusion and exclusion criteria for study selection.

2.4 Literature search

We conducted a thorough literature search to identify articles that explored, examined, and addressed the use of ChatGPT in Educational contexts. We utilized two research databases: Dimensions.ai, which provides access to a large number of research publications, and lens.org, which offers access to over 300 million articles, patents, and other research outputs from diverse sources. Additionally, we included three databases, Scopus, Web of Knowledge, and ERIC, which contain relevant research on the topic that addresses our research questions. To browse and identify relevant articles, we used the following search formula: ("ChatGPT" AND "Education"), which included the Boolean operator "AND" to get more specific results. The subject area in the Scopus and ERIC databases were narrowed to "ChatGPT" and "Education" keywords, and in the WoS database was limited to the "Education" category. The search was conducted between the 3rd and 10th of April 2023, which resulted in 276 articles from all selected databases (111 articles from Dimensions.ai, 65 from Scopus, 28 from Web of Science, 14 from ERIC, and 58 from Lens.org). These articles were imported into the Rayyan web-based system for analysis. The duplicates were identified automatically by the system. Subsequently, the first author manually reviewed the duplicated articles ensured that they had the same content, and then removed them, leaving us with 135 unique articles. Afterward, the titles, abstracts, and keywords of the first 40 manuscripts were scanned and reviewed by the first author and were discussed with the second and third authors to resolve any disagreements. Subsequently, the first author proceeded with the filtering process for all articles and carefully applied the inclusion and exclusion criteria as presented in Table  1 . Articles that met any one of the exclusion criteria were eliminated, resulting in 26 articles. Afterward, the authors met to carefully scan and discuss them. The authors agreed to eliminate any empirical studies solely focused on checking ChatGPT capabilities, as these studies do not guide us in addressing the research questions and achieving the study's objectives. This resulted in 14 articles eligible for analysis.

2.5 Quality appraisal

The examination and evaluation of the quality of the extracted articles is a vital step [ 9 ]. Therefore, the extracted articles were carefully evaluated for quality using Fink’s [ 24 ] standards, which emphasize the necessity for detailed descriptions of methodology, results, conclusions, strengths, and limitations. The process began with a thorough assessment of each study's design, data collection, and analysis methods to ensure their appropriateness and comprehensive execution. The clarity, consistency, and logical progression from data to results and conclusions were also critically examined. Potential biases and recognized limitations within the studies were also scrutinized. Ultimately, two articles were excluded for failing to meet Fink’s criteria, particularly in providing sufficient detail on methodology, results, conclusions, strengths, or limitations. The review process is illustrated in Fig.  1 .

The study selection process

2.6 Data extraction

The next step is data extraction, the process of capturing the key information and categories from the included studies. To improve efficiency, reduce variation among authors, and minimize errors in data analysis, the coding categories were constructed using Creswell's [ 15 ] coding techniques for data extraction and interpretation. The coding process involves three sequential steps. The initial stage encompasses open coding , where the researcher examines the data, generates codes to describe and categorize it, and gains a deeper understanding without preconceived ideas. Following open coding is axial coding , where the interrelationships between codes from open coding are analyzed to establish more comprehensive categories or themes. The process concludes with selective coding , refining and integrating categories or themes to identify core concepts emerging from the data. The first coder performed the coding process, then engaged in discussions with the second and third authors to finalize the coding categories for the first five articles. The first coder then proceeded to code all studies and engaged again in discussions with the other authors to ensure the finalization of the coding process. After a comprehensive analysis and capturing of the key information from the included studies, the data extraction and interpretation process yielded several themes. These themes have been categorized and are presented in Table  2 . It is important to note that open coding results were removed from Table  2 for aesthetic reasons, as it included many generic aspects, such as words, short phrases, or sentences mentioned in the studies.

2.7 Synthesize studies

In this stage, we will gather, discuss, and analyze the key findings that emerged from the selected studies. The synthesis stage is considered a transition from an author-centric to a concept-centric focus, enabling us to map all the provided information to achieve the most effective evaluation of the data [ 87 ]. Initially, the authors extracted data that included general information about the selected studies, including the author(s)' names, study titles, years of publication, educational levels, research methodologies, sample sizes, participants, main aims or objectives, raw data sources, and analysis methods. Following that, all key information and significant results from the selected studies were compiled using Creswell’s [ 15 ] coding techniques for data extraction and interpretation to identify core concepts and themes emerging from the data, focusing on those that directly contributed to our research questions and objectives, such as the initial utilization of ChatGPT in learning and teaching, learners' and educators' familiarity with ChatGPT, and the main findings of each study. Finally, the data related to each selected study were extracted into an Excel spreadsheet for data processing. The Excel spreadsheet was reviewed by the authors, including a series of discussions to ensure the finalization of this process and prepare it for further analysis. Afterward, the final result being analyzed and presented in various types of charts and graphs. Table 4 presents the extracted data from the selected studies, with each study labeled with a capital 'S' followed by a number.

This section consists of two main parts. The first part provides a descriptive analysis of the data compiled from the reviewed studies. The second part presents the answers to the research questions and the main findings of these studies.

3.1 Part 1: descriptive analysis

This section will provide a descriptive analysis of the reviewed studies, including educational levels and fields, participants distribution, country contribution, research methodologies, study sample size, study population, publication year, list of journals, familiarity with ChatGPT, source of data, and the main aims and objectives of the studies. Table 4 presents a comprehensive overview of the extracted data from the selected studies.

3.1.1 The number of the reviewed studies and publication years

The total number of the reviewed studies was 14. All studies were empirical studies and published in different journals focusing on Education and Technology. One study was published in 2022 [S1], while the remaining were published in 2023 [S2]-[S14]. Table 3 illustrates the year of publication, the names of the journals, and the number of reviewed studies published in each journal for the studies reviewed.

3.1.2 Educational levels and fields

The majority of the reviewed studies, 11 studies, were conducted in higher education institutions [S1]-[S10] and [S13]. Two studies did not specify the educational level of the population [S12] and [S14], while one study focused on elementary education [S11]. However, the reviewed studies covered various fields of education. Three studies focused on Arts and Humanities Education [S8], [S11], and [S14], specifically English Education. Two studies focused on Engineering Education, with one in Computer Engineering [S2] and the other in Construction Education [S3]. Two studies focused on Mathematics Education [S5] and [S12]. One study focused on Social Science Education [S13]. One study focused on Early Education [S4]. One study focused on Journalism Education [S9]. Finally, three studies did not specify the field of education [S1], [S6], and [S7]. Figure  2 represents the educational levels in the reviewed studies, while Fig.  3 represents the context of the reviewed studies.

Educational levels in the reviewed studies

Context of the reviewed studies

3.1.3 Participants distribution and countries contribution

The reviewed studies have been conducted across different geographic regions, providing a diverse representation of the studies. The majority of the studies, 10 in total, [S1]-[S3], [S5]-[S9], [S11], and [S14], primarily focused on participants from single countries such as Pakistan, the United Arab Emirates, China, Indonesia, Poland, Saudi Arabia, South Korea, Spain, Tajikistan, and the United States. In contrast, four studies, [S4], [S10], [S12], and [S13], involved participants from multiple countries, including China and the United States [S4], China, the United Kingdom, and the United States [S10], the United Arab Emirates, Oman, Saudi Arabia, and Jordan [S12], Turkey, Sweden, Canada, and Australia [ 13 ]. Figures  4 and 5 illustrate the distribution of participants, whether from single or multiple countries, and the contribution of each country in the reviewed studies, respectively.

The reviewed studies conducted in single or multiple countries

The Contribution of each country in the studies

3.1.4 Study population and sample size

Four study populations were included: university students, university teachers, university teachers and students, and elementary school teachers. Six studies involved university students [S2], [S3], [S5] and [S6]-[S8]. Three studies focused on university teachers [S1], [S4], and [S6], while one study specifically targeted elementary school teachers [S11]. Additionally, four studies included both university teachers and students [S10] and [ 12 , 13 , 14 ], and among them, study [S13] specifically included postgraduate students. In terms of the sample size of the reviewed studies, nine studies included a small sample size of less than 50 participants [S1], [S3], [S6], [S8], and [S10]-[S13]. Three studies had 50–100 participants [S2], [S9], and [S14]. Only one study had more than 100 participants [S7]. It is worth mentioning that study [S4] adopted a mixed methods approach, including 10 participants for qualitative analysis and 110 participants for quantitative analysis.

3.1.5 Participants’ familiarity with using ChatGPT

The reviewed studies recruited a diverse range of participants with varying levels of familiarity with ChatGPT. Five studies [S2], [S4], [S6], [S8], and [S12] involved participants already familiar with ChatGPT, while eight studies [S1], [S3], [S5], [S7], [S9], [S10], [S13] and [S14] included individuals with differing levels of familiarity. Notably, one study [S11] had participants who were entirely unfamiliar with ChatGPT. It is important to note that four studies [S3], [S5], [S9], and [S11] provided training or guidance to their participants before conducting their studies, while ten studies [S1], [S2], [S4], [S6]-[S8], [S10], and [S12]-[S14] did not provide training due to the participants' existing familiarity with ChatGPT.

3.1.6 Research methodology approaches and source(S) of data

The reviewed studies adopted various research methodology approaches. Seven studies adopted qualitative research methodology [S1], [S4], [S6], [S8], [S10], [S11], and [S12], while three studies adopted quantitative research methodology [S3], [S7], and [S14], and four studies employed mixed-methods, which involved a combination of both the strengths of qualitative and quantitative methods [S2], [S5], [S9], and [S13].

In terms of the source(s) of data, the reviewed studies obtained their data from various sources, such as interviews, questionnaires, and pre-and post-tests. Six studies relied on interviews as their primary source of data collection [S1], [S4], [S6], [S10], [S11], and [S12], four studies relied on questionnaires [S2], [S7], [S13], and [S14], two studies combined the use of pre-and post-tests and questionnaires for data collection [S3] and [S9], while two studies combined the use of questionnaires and interviews to obtain the data [S5] and [S8]. It is important to note that six of the reviewed studies were quasi-experimental [S3], [S5], [S8], [S9], [S12], and [S14], while the remaining ones were experimental studies [S1], [S2], [S4], [S6], [S7], [S10], [S11], and [S13]. Figures  6 and 7 illustrate the research methodologies and the source (s) of data used in the reviewed studies, respectively.

Research methodologies in the reviewed studies

Source of data in the reviewed studies

3.1.7 The aim and objectives of the studies

The reviewed studies encompassed a diverse set of aims, with several of them incorporating multiple primary objectives. Six studies [S3], [S6], [S7], [S8], [S11], and [S12] examined the integration of ChatGPT in educational contexts, and four studies [S4], [S5], [S13], and [S14] investigated the various implications of its use in education, while three studies [S2], [S9], and [S10] aimed to explore both its integration and implications in education. Additionally, seven studies explicitly explored attitudes and perceptions of students [S2] and [S3], educators [S1] and [S6], or both [S10], [S12], and [S13] regarding the utilization of ChatGPT in educational settings.

3.2 Part 2: research questions and main findings of the reviewed studies

This part will present the answers to the research questions and the main findings of the reviewed studies, classified into two main categories (learning and teaching) according to AI Education classification by [ 36 ]. Figure  8 summarizes the main findings of the reviewed studies in a visually informative diagram. Table 4 provides a detailed list of the key information extracted from the selected studies that led to generating these themes.

The main findings in the reviewed studies

4 Students' initial attempts at utilizing ChatGPT in learning and main findings from students' perspective

4.1 virtual intelligent assistant.

Nine studies demonstrated that ChatGPT has been utilized by students as an intelligent assistant to enhance and support their learning. Students employed it for various purposes, such as answering on-demand questions [S2]-[S5], [S8], [S10], and [S12], providing valuable information and learning resources [S2]-[S5], [S6], and [S8], as well as receiving immediate feedback [S2], [S4], [S9], [S10], and [S12]. In this regard, students generally were confident in the accuracy of ChatGPT's responses, considering them relevant, reliable, and detailed [S3], [S4], [S5], and [S8]. However, some students indicated the need for improvement, as they found that answers are not always accurate [S2], and that misleading information may have been provided or that it may not always align with their expectations [S6] and [S10]. It was also observed by the students that the accuracy of ChatGPT is dependent on several factors, including the quality and specificity of the user's input, the complexity of the question or topic, and the scope and relevance of its training data [S12]. Many students felt that ChatGPT's answers were not always accurate and most of them believed that it requires good background knowledge to work with.

4.2 Writing and language proficiency assistant

Six of the reviewed studies highlighted that ChatGPT has been utilized by students as a valuable assistant tool to improve their academic writing skills and language proficiency. Among these studies, three mainly focused on English education, demonstrating that students showed sufficient mastery in using ChatGPT for generating ideas, summarizing, paraphrasing texts, and completing writing essays [S8], [S11], and [S14]. Furthermore, ChatGPT helped them in writing by making students active investigators rather than passive knowledge recipients and facilitated the development of their writing skills [S11] and [S14]. Similarly, ChatGPT allowed students to generate unique ideas and perspectives, leading to deeper analysis and reflection on their journalism writing [S9]. In terms of language proficiency, ChatGPT allowed participants to translate content into their home languages, making it more accessible and relevant to their context [S4]. It also enabled them to request changes in linguistic tones or flavors [S8]. Moreover, participants used it to check grammar or as a dictionary [S11].

4.3 Valuable resource for learning approaches

Five studies demonstrated that students used ChatGPT as a valuable complementary resource for self-directed learning. It provided learning resources and guidance on diverse educational topics and created a supportive home learning environment [S2] and [S4]. Moreover, it offered step-by-step guidance to grasp concepts at their own pace and enhance their understanding [S5], streamlined task and project completion carried out independently [S7], provided comprehensive and easy-to-understand explanations on various subjects [S10], and assisted in studying geometry operations, thereby empowering them to explore geometry operations at their own pace [S12]. Three studies showed that students used ChatGPT as a valuable learning resource for personalized learning. It delivered age-appropriate conversations and tailored teaching based on a child's interests [S4], acted as a personalized learning assistant, adapted to their needs and pace, which assisted them in understanding mathematical concepts [S12], and enabled personalized learning experiences in social sciences by adapting to students' needs and learning styles [S13]. On the other hand, it is important to note that, according to one study [S5], students suggested that using ChatGPT may negatively affect collaborative learning competencies between students.

4.4 Enhancing students' competencies

Six of the reviewed studies have shown that ChatGPT is a valuable tool for improving a wide range of skills among students. Two studies have provided evidence that ChatGPT led to improvements in students' critical thinking, reasoning skills, and hazard recognition competencies through engaging them in interactive conversations or activities and providing responses related to their disciplines in journalism [S5] and construction education [S9]. Furthermore, two studies focused on mathematical education have shown the positive impact of ChatGPT on students' problem-solving abilities in unraveling problem-solving questions [S12] and enhancing the students' understanding of the problem-solving process [S5]. Lastly, one study indicated that ChatGPT effectively contributed to the enhancement of conversational social skills [S4].

4.5 Supporting students' academic success

Seven of the reviewed studies highlighted that students found ChatGPT to be beneficial for learning as it enhanced learning efficiency and improved the learning experience. It has been observed to improve students' efficiency in computer engineering studies by providing well-structured responses and good explanations [S2]. Additionally, students found it extremely useful for hazard reporting [S3], and it also enhanced their efficiency in solving mathematics problems and capabilities [S5] and [S12]. Furthermore, by finding information, generating ideas, translating texts, and providing alternative questions, ChatGPT aided students in deepening their understanding of various subjects [S6]. It contributed to an increase in students' overall productivity [S7] and improved efficiency in composing written tasks [S8]. Regarding learning experiences, ChatGPT was instrumental in assisting students in identifying hazards that they might have otherwise overlooked [S3]. It also improved students' learning experiences in solving mathematics problems and developing abilities [S5] and [S12]. Moreover, it increased students' successful completion of important tasks in their studies [S7], particularly those involving average difficulty writing tasks [S8]. Additionally, ChatGPT increased the chances of educational success by providing students with baseline knowledge on various topics [S10].

5 Teachers' initial attempts at utilizing ChatGPT in teaching and main findings from teachers' perspective

5.1 valuable resource for teaching.

The reviewed studies showed that teachers have employed ChatGPT to recommend, modify, and generate diverse, creative, organized, and engaging educational contents, teaching materials, and testing resources more rapidly [S4], [S6], [S10] and [S11]. Additionally, teachers experienced increased productivity as ChatGPT facilitated quick and accurate responses to questions, fact-checking, and information searches [S1]. It also proved valuable in constructing new knowledge [S6] and providing timely answers to students' questions in classrooms [S11]. Moreover, ChatGPT enhanced teachers' efficiency by generating new ideas for activities and preplanning activities for their students [S4] and [S6], including interactive language game partners [S11].

5.2 Improving productivity and efficiency

The reviewed studies showed that participants' productivity and work efficiency have been significantly enhanced by using ChatGPT as it enabled them to allocate more time to other tasks and reduce their overall workloads [S6], [S10], [S11], [S13], and [S14]. However, three studies [S1], [S4], and [S11], indicated a negative perception and attitude among teachers toward using ChatGPT. This negativity stemmed from a lack of necessary skills to use it effectively [S1], a limited familiarity with it [S4], and occasional inaccuracies in the content provided by it [S10].

5.3 Catalyzing new teaching methodologies

Five of the reviewed studies highlighted that educators found the necessity of redefining their teaching profession with the assistance of ChatGPT [S11], developing new effective learning strategies [S4], and adapting teaching strategies and methodologies to ensure the development of essential skills for future engineers [S5]. They also emphasized the importance of adopting new educational philosophies and approaches that can evolve with the introduction of ChatGPT into the classroom [S12]. Furthermore, updating curricula to focus on improving human-specific features, such as emotional intelligence, creativity, and philosophical perspectives [S13], was found to be essential.

5.4 Effective utilization of CHATGPT in teaching

According to the reviewed studies, effective utilization of ChatGPT in education requires providing teachers with well-structured training, support, and adequate background on how to use ChatGPT responsibly [S1], [S3], [S11], and [S12]. Establishing clear rules and regulations regarding its usage is essential to ensure it positively impacts the teaching and learning processes, including students' skills [S1], [S4], [S5], [S8], [S9], and [S11]-[S14]. Moreover, conducting further research and engaging in discussions with policymakers and stakeholders is indeed crucial for the successful integration of ChatGPT in education and to maximize the benefits for both educators and students [S1], [S6]-[S10], and [S12]-[S14].

6 Discussion

The purpose of this review is to conduct a systematic review of empirical studies that have explored the utilization of ChatGPT, one of today’s most advanced LLM-based chatbots, in education. The findings of the reviewed studies showed several ways of ChatGPT utilization in different learning and teaching practices as well as it provided insights and considerations that can facilitate its effective and responsible use in future educational contexts. The results of the reviewed studies came from diverse fields of education, which helped us avoid a biased review that is limited to a specific field. Similarly, the reviewed studies have been conducted across different geographic regions. This kind of variety in geographic representation enriched the findings of this review.

In response to RQ1 , "What are students' and teachers' initial attempts at utilizing ChatGPT in education?", the findings from this review provide comprehensive insights. Chatbots, including ChatGPT, play a crucial role in supporting student learning, enhancing their learning experiences, and facilitating diverse learning approaches [ 42 , 43 ]. This review found that this tool, ChatGPT, has been instrumental in enhancing students' learning experiences by serving as a virtual intelligent assistant, providing immediate feedback, on-demand answers, and engaging in educational conversations. Additionally, students have benefited from ChatGPT’s ability to generate ideas, compose essays, and perform tasks like summarizing, translating, paraphrasing texts, or checking grammar, thereby enhancing their writing and language competencies. Furthermore, students have turned to ChatGPT for assistance in understanding concepts and homework, providing structured learning plans, and clarifying assignments and tasks, which fosters a supportive home learning environment, allowing them to take responsibility for their own learning and cultivate the skills and approaches essential for supportive home learning environment [ 26 , 27 , 28 ]. This finding aligns with the study of Saqr et al. [ 68 , 69 ] who highlighted that, when students actively engage in their own learning process, it yields additional advantages, such as heightened motivation, enhanced achievement, and the cultivation of enthusiasm, turning them into advocates for their own learning.

Moreover, students have utilized ChatGPT for tailored teaching and step-by-step guidance on diverse educational topics, streamlining task and project completion, and generating and recommending educational content. This personalization enhances the learning environment, leading to increased academic success. This finding aligns with other recent studies [ 26 , 27 , 28 , 60 , 66 ] which revealed that ChatGPT has the potential to offer personalized learning experiences and support an effective learning process by providing students with customized feedback and explanations tailored to their needs and abilities. Ultimately, fostering students' performance, engagement, and motivation, leading to increase students' academic success [ 14 , 44 , 58 ]. This ultimate outcome is in line with the findings of Saqr et al. [ 68 , 69 ], which emphasized that learning strategies are important catalysts of students' learning, as students who utilize effective learning strategies are more likely to have better academic achievement.

Teachers, too, have capitalized on ChatGPT's capabilities to enhance productivity and efficiency, using it for creating lesson plans, generating quizzes, providing additional resources, generating and preplanning new ideas for activities, and aiding in answering students’ questions. This adoption of technology introduces new opportunities to support teaching and learning practices, enhancing teacher productivity. This finding aligns with those of Day [ 17 ], De Castro [ 18 ], and Su and Yang [ 74 ] as well as with those of Valtonen et al. [ 82 ], who revealed that emerging technological advancements have opened up novel opportunities and means to support teaching and learning practices, and enhance teachers’ productivity.

In response to RQ2 , "What are the main findings derived from empirical studies that have incorporated ChatGPT into learning and teaching?", the findings from this review provide profound insights and raise significant concerns. Starting with the insights, chatbots, including ChatGPT, have demonstrated the potential to reshape and revolutionize education, creating new, novel opportunities for enhancing the learning process and outcomes [ 83 ], facilitating different learning approaches, and offering a range of pedagogical benefits [ 19 , 43 , 72 ]. In this context, this review found that ChatGPT could open avenues for educators to adopt or develop new effective learning and teaching strategies that can evolve with the introduction of ChatGPT into the classroom. Nonetheless, there is an evident lack of research understanding regarding the potential impact of generative machine learning models within diverse educational settings [ 83 ]. This necessitates teachers to attain a high level of proficiency in incorporating chatbots, such as ChatGPT, into their classrooms to create inventive, well-structured, and captivating learning strategies. In the same vein, the review also found that teachers without the requisite skills to utilize ChatGPT realized that it did not contribute positively to their work and could potentially have adverse effects [ 37 ]. This concern could lead to inequity of access to the benefits of chatbots, including ChatGPT, as individuals who lack the necessary expertise may not be able to harness their full potential, resulting in disparities in educational outcomes and opportunities. Therefore, immediate action is needed to address these potential issues. A potential solution is offering training, support, and competency development for teachers to ensure that all of them can leverage chatbots, including ChatGPT, effectively and equitably in their educational practices [ 5 , 28 , 80 ], which could enhance accessibility and inclusivity, and potentially result in innovative outcomes [ 82 , 83 ].

Additionally, chatbots, including ChatGPT, have the potential to significantly impact students' thinking abilities, including retention, reasoning, analysis skills [ 19 , 45 ], and foster innovation and creativity capabilities [ 83 ]. This review found that ChatGPT could contribute to improving a wide range of skills among students. However, it found that frequent use of ChatGPT may result in a decrease in innovative capacities, collaborative skills and cognitive capacities, and students' motivation to attend classes, as well as could lead to reduced higher-order thinking skills among students [ 22 , 29 ]. Therefore, immediate action is needed to carefully examine the long-term impact of chatbots such as ChatGPT, on learning outcomes as well as to explore its incorporation into educational settings as a supportive tool without compromising students' cognitive development and critical thinking abilities. In the same vein, the review also found that it is challenging to draw a consistent conclusion regarding the potential of ChatGPT to aid self-directed learning approach. This finding aligns with the recent study of Baskara [ 8 ]. Therefore, further research is needed to explore the potential of ChatGPT for self-directed learning. One potential solution involves utilizing learning analytics as a novel approach to examine various aspects of students' learning and support them in their individual endeavors [ 32 ]. This approach can bridge this gap by facilitating an in-depth analysis of how learners engage with ChatGPT, identifying trends in self-directed learning behavior, and assessing its influence on their outcomes.

Turning to the significant concerns, on the other hand, a fundamental challenge with LLM-based chatbots, including ChatGPT, is the accuracy and quality of the provided information and responses, as they provide false information as truth—a phenomenon often referred to as "hallucination" [ 3 , 49 ]. In this context, this review found that the provided information was not entirely satisfactory. Consequently, the utilization of chatbots presents potential concerns, such as generating and providing inaccurate or misleading information, especially for students who utilize it to support their learning. This finding aligns with other findings [ 6 , 30 , 35 , 40 ] which revealed that incorporating chatbots such as ChatGPT, into education presents challenges related to its accuracy and reliability due to its training on a large corpus of data, which may contain inaccuracies and the way users formulate or ask ChatGPT. Therefore, immediate action is needed to address these potential issues. One possible solution is to equip students with the necessary skills and competencies, which include a background understanding of how to use it effectively and the ability to assess and evaluate the information it generates, as the accuracy and the quality of the provided information depend on the input, its complexity, the topic, and the relevance of its training data [ 28 , 49 , 86 ]. However, it's also essential to examine how learners can be educated about how these models operate, the data used in their training, and how to recognize their limitations, challenges, and issues [ 79 ].

Furthermore, chatbots present a substantial challenge concerning maintaining academic integrity [ 20 , 56 ] and copyright violations [ 83 ], which are significant concerns in education. The review found that the potential misuse of ChatGPT might foster cheating, facilitate plagiarism, and threaten academic integrity. This issue is also affirmed by the research conducted by Basic et al. [ 7 ], who presented evidence that students who utilized ChatGPT in their writing assignments had more plagiarism cases than those who did not. These findings align with the conclusions drawn by Cotton et al. [ 13 ], Hisan and Amri [ 33 ] and Sullivan et al. [ 75 ], who revealed that the integration of chatbots such as ChatGPT into education poses a significant challenge to the preservation of academic integrity. Moreover, chatbots, including ChatGPT, have increased the difficulty in identifying plagiarism [ 47 , 67 , 76 ]. The findings from previous studies [ 1 , 84 ] indicate that AI-generated text often went undetected by plagiarism software, such as Turnitin. However, Turnitin and other similar plagiarism detection tools, such as ZeroGPT, GPTZero, and Copyleaks, have since evolved, incorporating enhanced techniques to detect AI-generated text, despite the possibility of false positives, as noted in different studies that have found these tools still not yet fully ready to accurately and reliably identify AI-generated text [ 10 , 51 ], and new novel detection methods may need to be created and implemented for AI-generated text detection [ 4 ]. This potential issue could lead to another concern, which is the difficulty of accurately evaluating student performance when they utilize chatbots such as ChatGPT assistance in their assignments. Consequently, the most LLM-driven chatbots present a substantial challenge to traditional assessments [ 64 ]. The findings from previous studies indicate the importance of rethinking, improving, and redesigning innovative assessment methods in the era of chatbots [ 14 , 20 , 64 , 75 ]. These methods should prioritize the process of evaluating students' ability to apply knowledge to complex cases and demonstrate comprehension, rather than solely focusing on the final product for assessment. Therefore, immediate action is needed to address these potential issues. One possible solution would be the development of clear guidelines, regulatory policies, and pedagogical guidance. These measures would help regulate the proper and ethical utilization of chatbots, such as ChatGPT, and must be established before their introduction to students [ 35 , 38 , 39 , 41 , 89 ].

In summary, our review has delved into the utilization of ChatGPT, a prominent example of chatbots, in education, addressing the question of how ChatGPT has been utilized in education. However, there remain significant gaps, which necessitate further research to shed light on this area.

7 Conclusions

This systematic review has shed light on the varied initial attempts at incorporating ChatGPT into education by both learners and educators, while also offering insights and considerations that can facilitate its effective and responsible use in future educational contexts. From the analysis of 14 selected studies, the review revealed the dual-edged impact of ChatGPT in educational settings. On the positive side, ChatGPT significantly aided the learning process in various ways. Learners have used it as a virtual intelligent assistant, benefiting from its ability to provide immediate feedback, on-demand answers, and easy access to educational resources. Additionally, it was clear that learners have used it to enhance their writing and language skills, engaging in practices such as generating ideas, composing essays, and performing tasks like summarizing, translating, paraphrasing texts, or checking grammar. Importantly, other learners have utilized it in supporting and facilitating their directed and personalized learning on a broad range of educational topics, assisting in understanding concepts and homework, providing structured learning plans, and clarifying assignments and tasks. Educators, on the other hand, found ChatGPT beneficial for enhancing productivity and efficiency. They used it for creating lesson plans, generating quizzes, providing additional resources, and answers learners' questions, which saved time and allowed for more dynamic and engaging teaching strategies and methodologies.

However, the review also pointed out negative impacts. The results revealed that overuse of ChatGPT could decrease innovative capacities and collaborative learning among learners. Specifically, relying too much on ChatGPT for quick answers can inhibit learners' critical thinking and problem-solving skills. Learners might not engage deeply with the material or consider multiple solutions to a problem. This tendency was particularly evident in group projects, where learners preferred consulting ChatGPT individually for solutions over brainstorming and collaborating with peers, which negatively affected their teamwork abilities. On a broader level, integrating ChatGPT into education has also raised several concerns, including the potential for providing inaccurate or misleading information, issues of inequity in access, challenges related to academic integrity, and the possibility of misusing the technology.

Accordingly, this review emphasizes the urgency of developing clear rules, policies, and regulations to ensure ChatGPT's effective and responsible use in educational settings, alongside other chatbots, by both learners and educators. This requires providing well-structured training to educate them on responsible usage and understanding its limitations, along with offering sufficient background information. Moreover, it highlights the importance of rethinking, improving, and redesigning innovative teaching and assessment methods in the era of ChatGPT. Furthermore, conducting further research and engaging in discussions with policymakers and stakeholders are essential steps to maximize the benefits for both educators and learners and ensure academic integrity.

It is important to acknowledge that this review has certain limitations. Firstly, the limited inclusion of reviewed studies can be attributed to several reasons, including the novelty of the technology, as new technologies often face initial skepticism and cautious adoption; the lack of clear guidelines or best practices for leveraging this technology for educational purposes; and institutional or governmental policies affecting the utilization of this technology in educational contexts. These factors, in turn, have affected the number of studies available for review. Secondly, the utilization of the original version of ChatGPT, based on GPT-3 or GPT-3.5, implies that new studies utilizing the updated version, GPT-4 may lead to different findings. Therefore, conducting follow-up systematic reviews is essential once more empirical studies on ChatGPT are published. Additionally, long-term studies are necessary to thoroughly examine and assess the impact of ChatGPT on various educational practices.

Despite these limitations, this systematic review has highlighted the transformative potential of ChatGPT in education, revealing its diverse utilization by learners and educators alike and summarized the benefits of incorporating it into education, as well as the forefront critical concerns and challenges that must be addressed to facilitate its effective and responsible use in future educational contexts. This review could serve as an insightful resource for practitioners who seek to integrate ChatGPT into education and stimulate further research in the field.

Data availability

The data supporting our findings are available upon request.

Abbreviations

• Artificial intelligence

AI in education

Large language model

Artificial neural networks

Chat Generative Pre-Trained Transformer

Recurrent neural networks

Long short-term memory

Reinforcement learning from human feedback

Natural language processing

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

AlAfnan MA, Dishari S, Jovic M, Lomidze K. ChatGPT as an educational tool: opportunities, challenges, and recommendations for communication, business writing, and composition courses. J Artif Intell Technol. 2023. https://doi.org/10.37965/jait.2023.0184 .

Article   Google Scholar  

Ali JKM, Shamsan MAA, Hezam TA, Mohammed AAQ. Impact of ChatGPT on learning motivation. J Engl Stud Arabia Felix. 2023;2(1):41–9. https://doi.org/10.56540/jesaf.v2i1.51 .

Alkaissi H, McFarlane SI. Artificial hallucinations in ChatGPT: implications in scientific writing. Cureus. 2023. https://doi.org/10.7759/cureus.35179 .

Anderson N, Belavý DL, Perle SM, Hendricks S, Hespanhol L, Verhagen E, Memon AR. AI did not write this manuscript, or did it? Can we trick the AI text detector into generated texts? The potential future of ChatGPT and AI in sports & exercise medicine manuscript generation. BMJ Open Sport Exerc Med. 2023;9(1): e001568. https://doi.org/10.1136/bmjsem-2023-001568 .

Ausat AMA, Massang B, Efendi M, Nofirman N, Riady Y. Can chat GPT replace the role of the teacher in the classroom: a fundamental analysis. J Educ. 2023;5(4):16100–6.

Google Scholar  

Baidoo-Anu D, Ansah L. Education in the Era of generative artificial intelligence (AI): understanding the potential benefits of ChatGPT in promoting teaching and learning. Soc Sci Res Netw. 2023. https://doi.org/10.2139/ssrn.4337484 .

Basic Z, Banovac A, Kruzic I, Jerkovic I. Better by you, better than me, chatgpt3 as writing assistance in students essays. 2023. arXiv preprint arXiv:2302.04536 .‏

Baskara FR. The promises and pitfalls of using chat GPT for self-determined learning in higher education: an argumentative review. Prosiding Seminar Nasional Fakultas Tarbiyah dan Ilmu Keguruan IAIM Sinjai. 2023;2:95–101. https://doi.org/10.47435/sentikjar.v2i0.1825 .

Behera RK, Bala PK, Dhir A. The emerging role of cognitive computing in healthcare: a systematic literature review. Int J Med Inform. 2019;129:154–66. https://doi.org/10.1016/j.ijmedinf.2019.04.024 .

Chaka C. Detecting AI content in responses generated by ChatGPT, YouChat, and Chatsonic: the case of five AI content detection tools. J Appl Learn Teach. 2023. https://doi.org/10.37074/jalt.2023.6.2.12 .

Chiu TKF, Xia Q, Zhou X, Chai CS, Cheng M. Systematic literature review on opportunities, challenges, and future research recommendations of artificial intelligence in education. Comput Educ Artif Intell. 2023;4:100118. https://doi.org/10.1016/j.caeai.2022.100118 .

Choi EPH, Lee JJ, Ho M, Kwok JYY, Lok KYW. Chatting or cheating? The impacts of ChatGPT and other artificial intelligence language models on nurse education. Nurse Educ Today. 2023;125:105796. https://doi.org/10.1016/j.nedt.2023.105796 .

Cotton D, Cotton PA, Shipway JR. Chatting and cheating: ensuring academic integrity in the era of ChatGPT. Innov Educ Teach Int. 2023. https://doi.org/10.1080/14703297.2023.2190148 .

Crawford J, Cowling M, Allen K. Leadership is needed for ethical ChatGPT: Character, assessment, and learning using artificial intelligence (AI). J Univ Teach Learn Pract. 2023. https://doi.org/10.53761/1.20.3.02 .

Creswell JW. Educational research: planning, conducting, and evaluating quantitative and qualitative research [Ebook]. 4th ed. London: Pearson Education; 2015.

Curry D. ChatGPT Revenue and Usage Statistics (2023)—Business of Apps. 2023. https://www.businessofapps.com/data/chatgpt-statistics/

Day T. A preliminary investigation of fake peer-reviewed citations and references generated by ChatGPT. Prof Geogr. 2023. https://doi.org/10.1080/00330124.2023.2190373 .

De Castro CA. A Discussion about the Impact of ChatGPT in education: benefits and concerns. J Bus Theor Pract. 2023;11(2):p28. https://doi.org/10.22158/jbtp.v11n2p28 .

Deng X, Yu Z. A meta-analysis and systematic review of the effect of Chatbot technology use in sustainable education. Sustainability. 2023;15(4):2940. https://doi.org/10.3390/su15042940 .

Eke DO. ChatGPT and the rise of generative AI: threat to academic integrity? J Responsib Technol. 2023;13:100060. https://doi.org/10.1016/j.jrt.2023.100060 .

Elmoazen R, Saqr M, Tedre M, Hirsto L. A systematic literature review of empirical research on epistemic network analysis in education. IEEE Access. 2022;10:17330–48. https://doi.org/10.1109/access.2022.3149812 .

Farrokhnia M, Banihashem SK, Noroozi O, Wals AEJ. A SWOT analysis of ChatGPT: implications for educational practice and research. Innov Educ Teach Int. 2023. https://doi.org/10.1080/14703297.2023.2195846 .

Fergus S, Botha M, Ostovar M. Evaluating academic answers generated using ChatGPT. J Chem Educ. 2023;100(4):1672–5. https://doi.org/10.1021/acs.jchemed.3c00087 .

Fink A. Conducting research literature reviews: from the Internet to Paper. Incorporated: SAGE Publications; 2010.

Firaina R, Sulisworo D. Exploring the usage of ChatGPT in higher education: frequency and impact on productivity. Buletin Edukasi Indonesia (BEI). 2023;2(01):39–46. https://doi.org/10.56741/bei.v2i01.310 .

Firat, M. (2023). How chat GPT can transform autodidactic experiences and open education.  Department of Distance Education, Open Education Faculty, Anadolu Unive .‏ https://orcid.org/0000-0001-8707-5918

Firat M. What ChatGPT means for universities: perceptions of scholars and students. J Appl Learn Teach. 2023. https://doi.org/10.37074/jalt.2023.6.1.22 .

Fuchs K. Exploring the opportunities and challenges of NLP models in higher education: is Chat GPT a blessing or a curse? Front Educ. 2023. https://doi.org/10.3389/feduc.2023.1166682 .

García-Peñalvo FJ. La percepción de la inteligencia artificial en contextos educativos tras el lanzamiento de ChatGPT: disrupción o pánico. Educ Knowl Soc. 2023;24: e31279. https://doi.org/10.14201/eks.31279 .

Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor A, Chartash D. How does ChatGPT perform on the United States medical Licensing examination? The implications of large language models for medical education and knowledge assessment. JMIR Med Educ. 2023;9: e45312. https://doi.org/10.2196/45312 .

Hashana AJ, Brundha P, Ayoobkhan MUA, Fazila S. Deep Learning in ChatGPT—A Survey. In   2023 7th international conference on trends in electronics and informatics (ICOEI) . 2023. (pp. 1001–1005). IEEE. https://doi.org/10.1109/icoei56765.2023.10125852

Hirsto L, Saqr M, López-Pernas S, Valtonen T. (2022). A systematic narrative review of learning analytics research in K-12 and schools.  Proceedings . https://ceur-ws.org/Vol-3383/FLAIEC22_paper_9536.pdf

Hisan UK, Amri MM. ChatGPT and medical education: a double-edged sword. J Pedag Educ Sci. 2023;2(01):71–89. https://doi.org/10.13140/RG.2.2.31280.23043/1 .

Hopkins AM, Logan JM, Kichenadasse G, Sorich MJ. Artificial intelligence chatbots will revolutionize how cancer patients access information: ChatGPT represents a paradigm-shift. JNCI Cancer Spectr. 2023. https://doi.org/10.1093/jncics/pkad010 .

Househ M, AlSaad R, Alhuwail D, Ahmed A, Healy MG, Latifi S, Sheikh J. Large Language models in medical education: opportunities, challenges, and future directions. JMIR Med Educ. 2023;9: e48291. https://doi.org/10.2196/48291 .

Ilkka T. The impact of artificial intelligence on learning, teaching, and education. Minist de Educ. 2018. https://doi.org/10.2760/12297 .

Iqbal N, Ahmed H, Azhar KA. Exploring teachers’ attitudes towards using CHATGPT. Globa J Manag Adm Sci. 2022;3(4):97–111. https://doi.org/10.46568/gjmas.v3i4.163 .

Irfan M, Murray L, Ali S. Integration of Artificial intelligence in academia: a case study of critical teaching and learning in Higher education. Globa Soc Sci Rev. 2023;8(1):352–64. https://doi.org/10.31703/gssr.2023(viii-i).32 .

Jeon JH, Lee S. Large language models in education: a focus on the complementary relationship between human teachers and ChatGPT. Educ Inf Technol. 2023. https://doi.org/10.1007/s10639-023-11834-1 .

Khan RA, Jawaid M, Khan AR, Sajjad M. ChatGPT—Reshaping medical education and clinical management. Pak J Med Sci. 2023. https://doi.org/10.12669/pjms.39.2.7653 .

King MR. A conversation on artificial intelligence, Chatbots, and plagiarism in higher education. Cell Mol Bioeng. 2023;16(1):1–2. https://doi.org/10.1007/s12195-022-00754-8 .

Kooli C. Chatbots in education and research: a critical examination of ethical implications and solutions. Sustainability. 2023;15(7):5614. https://doi.org/10.3390/su15075614 .

Kuhail MA, Alturki N, Alramlawi S, Alhejori K. Interacting with educational chatbots: a systematic review. Educ Inf Technol. 2022;28(1):973–1018. https://doi.org/10.1007/s10639-022-11177-3 .

Lee H. The rise of ChatGPT: exploring its potential in medical education. Anat Sci Educ. 2023. https://doi.org/10.1002/ase.2270 .

Li L, Subbareddy R, Raghavendra CG. AI intelligence Chatbot to improve students learning in the higher education platform. J Interconnect Netw. 2022. https://doi.org/10.1142/s0219265921430325 .

Limna P. A Review of Artificial Intelligence (AI) in Education during the Digital Era. 2022. https://ssrn.com/abstract=4160798

Lo CK. What is the impact of ChatGPT on education? A rapid review of the literature. Educ Sci. 2023;13(4):410. https://doi.org/10.3390/educsci13040410 .

Luo W, He H, Liu J, Berson IR, Berson MJ, Zhou Y, Li H. Aladdin’s genie or pandora’s box For early childhood education? Experts chat on the roles, challenges, and developments of ChatGPT. Early Educ Dev. 2023. https://doi.org/10.1080/10409289.2023.2214181 .

Meyer JG, Urbanowicz RJ, Martin P, O’Connor K, Li R, Peng P, Moore JH. ChatGPT and large language models in academia: opportunities and challenges. Biodata Min. 2023. https://doi.org/10.1186/s13040-023-00339-9 .

Mhlanga D. Open AI in education, the responsible and ethical use of ChatGPT towards lifelong learning. Soc Sci Res Netw. 2023. https://doi.org/10.2139/ssrn.4354422 .

Neumann, M., Rauschenberger, M., & Schön, E. M. (2023). “We Need To Talk About ChatGPT”: The Future of AI and Higher Education.‏ https://doi.org/10.1109/seeng59157.2023.00010

Nolan B. Here are the schools and colleges that have banned the use of ChatGPT over plagiarism and misinformation fears. Business Insider . 2023. https://www.businessinsider.com

O’Leary DE. An analysis of three chatbots: BlenderBot, ChatGPT and LaMDA. Int J Intell Syst Account, Financ Manag. 2023;30(1):41–54. https://doi.org/10.1002/isaf.1531 .

Okoli C. A guide to conducting a standalone systematic literature review. Commun Assoc Inf Syst. 2015. https://doi.org/10.17705/1cais.03743 .

OpenAI. (2023). https://openai.com/blog/chatgpt

Perkins M. Academic integrity considerations of AI large language models in the post-pandemic era: ChatGPT and beyond. J Univ Teach Learn Pract. 2023. https://doi.org/10.53761/1.20.02.07 .

Plevris V, Papazafeiropoulos G, Rios AJ. Chatbots put to the test in math and logic problems: A preliminary comparison and assessment of ChatGPT-3.5, ChatGPT-4, and Google Bard. arXiv (Cornell University) . 2023. https://doi.org/10.48550/arxiv.2305.18618

Rahman MM, Watanobe Y (2023) ChatGPT for education and research: opportunities, threats, and strategies. Appl Sci 13(9):5783. https://doi.org/10.3390/app13095783

Ram B, Verma P. Artificial intelligence AI-based Chatbot study of ChatGPT, google AI bard and baidu AI. World J Adv Eng Technol Sci. 2023;8(1):258–61. https://doi.org/10.30574/wjaets.2023.8.1.0045 .

Rasul T, Nair S, Kalendra D, Robin M, de Oliveira Santini F, Ladeira WJ, Heathcote L. The role of ChatGPT in higher education: benefits, challenges, and future research directions. J Appl Learn Teach. 2023. https://doi.org/10.37074/jalt.2023.6.1.29 .

Ratnam M, Sharm B, Tomer A. ChatGPT: educational artificial intelligence. Int J Adv Trends Comput Sci Eng. 2023;12(2):84–91. https://doi.org/10.30534/ijatcse/2023/091222023 .

Ray PP. ChatGPT: a comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet Things Cyber-Phys Syst. 2023;3:121–54. https://doi.org/10.1016/j.iotcps.2023.04.003 .

Roumeliotis KI, Tselikas ND. ChatGPT and Open-AI models: a preliminary review. Future Internet. 2023;15(6):192. https://doi.org/10.3390/fi15060192 .

Rudolph J, Tan S, Tan S. War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education. J Appl Learn Teach. 2023. https://doi.org/10.37074/jalt.2023.6.1.23 .

Ruiz LMS, Moll-López S, Nuñez-Pérez A, Moraño J, Vega-Fleitas E. ChatGPT challenges blended learning methodologies in engineering education: a case study in mathematics. Appl Sci. 2023;13(10):6039. https://doi.org/10.3390/app13106039 .

Sallam M, Salim NA, Barakat M, Al-Tammemi AB. ChatGPT applications in medical, dental, pharmacy, and public health education: a descriptive study highlighting the advantages and limitations. Narra J. 2023;3(1): e103. https://doi.org/10.52225/narra.v3i1.103 .

Salvagno M, Taccone FS, Gerli AG. Can artificial intelligence help for scientific writing? Crit Care. 2023. https://doi.org/10.1186/s13054-023-04380-2 .

Saqr M, López-Pernas S, Helske S, Hrastinski S. The longitudinal association between engagement and achievement varies by time, students’ profiles, and achievement state: a full program study. Comput Educ. 2023;199:104787. https://doi.org/10.1016/j.compedu.2023.104787 .

Saqr M, Matcha W, Uzir N, Jovanović J, Gašević D, López-Pernas S. Transferring effective learning strategies across learning contexts matters: a study in problem-based learning. Australas J Educ Technol. 2023;39(3):9.

Schöbel S, Schmitt A, Benner D, Saqr M, Janson A, Leimeister JM. Charting the evolution and future of conversational agents: a research agenda along five waves and new frontiers. Inf Syst Front. 2023. https://doi.org/10.1007/s10796-023-10375-9 .

Shoufan A. Exploring students’ perceptions of CHATGPT: thematic analysis and follow-up survey. IEEE Access. 2023. https://doi.org/10.1109/access.2023.3268224 .

Sonderegger S, Seufert S. Chatbot-mediated learning: conceptual framework for the design of Chatbot use cases in education. Gallen: Institute for Educational Management and Technologies, University of St; 2022. https://doi.org/10.5220/0010999200003182 .

Book   Google Scholar  

Strzelecki A. To use or not to use ChatGPT in higher education? A study of students’ acceptance and use of technology. Interact Learn Environ. 2023. https://doi.org/10.1080/10494820.2023.2209881 .

Su J, Yang W. Unlocking the power of ChatGPT: a framework for applying generative AI in education. ECNU Rev Educ. 2023. https://doi.org/10.1177/20965311231168423 .

Sullivan M, Kelly A, McLaughlan P. ChatGPT in higher education: Considerations for academic integrity and student learning. J ApplLearn Teach. 2023;6(1):1–10. https://doi.org/10.37074/jalt.2023.6.1.17 .

Szabo A. ChatGPT is a breakthrough in science and education but fails a test in sports and exercise psychology. Balt J Sport Health Sci. 2023;1(128):25–40. https://doi.org/10.33607/bjshs.v127i4.1233 .

Taecharungroj V. “What can ChatGPT do?” analyzing early reactions to the innovative AI chatbot on Twitter. Big Data Cognit Comput. 2023;7(1):35. https://doi.org/10.3390/bdcc7010035 .

Tam S, Said RB. User preferences for ChatGPT-powered conversational interfaces versus traditional methods. Biomed Eng Soc. 2023. https://doi.org/10.58496/mjcsc/2023/004 .

Tedre M, Kahila J, Vartiainen H. (2023). Exploration on how co-designing with AI facilitates critical evaluation of ethics of AI in craft education. In: Langran E, Christensen P, Sanson J (Eds).  Proceedings of Society for Information Technology and Teacher Education International Conference . 2023. pp. 2289–2296.

Tlili A, Shehata B, Adarkwah MA, Bozkurt A, Hickey DT, Huang R, Agyemang B. What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education. Smart Learn Environ. 2023. https://doi.org/10.1186/s40561-023-00237-x .

Uddin SMJ, Albert A, Ovid A, Alsharef A. Leveraging CHATGPT to aid construction hazard recognition and support safety education and training. Sustainability. 2023;15(9):7121. https://doi.org/10.3390/su15097121 .

Valtonen T, López-Pernas S, Saqr M, Vartiainen H, Sointu E, Tedre M. The nature and building blocks of educational technology research. Comput Hum Behav. 2022;128:107123. https://doi.org/10.1016/j.chb.2021.107123 .

Vartiainen H, Tedre M. Using artificial intelligence in craft education: crafting with text-to-image generative models. Digit Creat. 2023;34(1):1–21. https://doi.org/10.1080/14626268.2023.2174557 .

Ventayen RJM. OpenAI ChatGPT generated results: similarity index of artificial intelligence-based contents. Soc Sci Res Netw. 2023. https://doi.org/10.2139/ssrn.4332664 .

Wagner MW, Ertl-Wagner BB. Accuracy of information and references using ChatGPT-3 for retrieval of clinical radiological information. Can Assoc Radiol J. 2023. https://doi.org/10.1177/08465371231171125 .

Wardat Y, Tashtoush MA, AlAli R, Jarrah AM. ChatGPT: a revolutionary tool for teaching and learning mathematics. Eurasia J Math, Sci Technol Educ. 2023;19(7):em2286. https://doi.org/10.29333/ejmste/13272 .

Webster J, Watson RT. Analyzing the past to prepare for the future: writing a literature review. Manag Inf Syst Quart. 2002;26(2):3.

Xiao Y, Watson ME. Guidance on conducting a systematic literature review. J Plan Educ Res. 2017;39(1):93–112. https://doi.org/10.1177/0739456x17723971 .

Yan D. Impact of ChatGPT on learners in a L2 writing practicum: an exploratory investigation. Educ Inf Technol. 2023. https://doi.org/10.1007/s10639-023-11742-4 .

Yu H. Reflection on whether Chat GPT should be banned by academia from the perspective of education and teaching. Front Psychol. 2023;14:1181712. https://doi.org/10.3389/fpsyg.2023.1181712 .

Zhu C, Sun M, Luo J, Li T, Wang M. How to harness the potential of ChatGPT in education? Knowl Manag ELearn. 2023;15(2):133–52. https://doi.org/10.34105/j.kmel.2023.15.008 .

Download references

The paper is co-funded by the Academy of Finland (Suomen Akatemia) Research Council for Natural Sciences and Engineering for the project Towards precision education: Idiographic learning analytics (TOPEILA), Decision Number 350560.

Author information

Authors and affiliations.

School of Computing, University of Eastern Finland, 80100, Joensuu, Finland

Yazid Albadarin, Mohammed Saqr, Nicolas Pope & Markku Tukiainen

You can also search for this author in PubMed   Google Scholar

Contributions

YA contributed to the literature search, data analysis, discussion, and conclusion. Additionally, YA contributed to the manuscript’s writing, editing, and finalization. MS contributed to the study’s design, conceptualization, acquisition of funding, project administration, allocation of resources, supervision, validation, literature search, and analysis of results. Furthermore, MS contributed to the manuscript's writing, revising, and approving it in its finalized state. NP contributed to the results, and discussions, and provided supervision. NP also contributed to the writing process, revisions, and the final approval of the manuscript in its finalized state. MT contributed to the study's conceptualization, resource management, supervision, writing, revising the manuscript, and approving it.

Corresponding author

Correspondence to Yazid Albadarin .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

See Table  4

The process of synthesizing the data presented in Table  4 involved identifying the relevant studies through a search process of databases (ERIC, Scopus, Web of Knowledge, Dimensions.ai, and lens.org) using specific keywords "ChatGPT" and "education". Following this, inclusion/exclusion criteria were applied, and data extraction was performed using Creswell's [ 15 ] coding techniques to capture key information and identify common themes across the included studies.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Albadarin, Y., Saqr, M., Pope, N. et al. A systematic literature review of empirical research on ChatGPT in education. Discov Educ 3 , 60 (2024). https://doi.org/10.1007/s44217-024-00138-2

Download citation

Received : 22 October 2023

Accepted : 10 May 2024

Published : 26 May 2024

DOI : https://doi.org/10.1007/s44217-024-00138-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Large language models
• Educational technology
• Systematic review

Advertisement

• Find a journal
• Publish with us
• Track your research