systematic literature review using scopus

• Topics Computer Setup Software Installation Developing Coding Skills Research Skills & Theses Writing an academic paper Templates and Dynamic Documents Navigating Scopus for Effective Literature Research Collect & Store Data Collection Data Storage Manage and Manipulate Data Loading and Importing Data Manipulating and Cleaning Data Analyze Data Regression Analysis Tests Causal Inference Machine Learning and Predictive Modeling Visualizing and Reporting Data Visualization Reporting Tables Automations and Workflows Workflows Automation Tools Replicability and Environment Management Version Control and Repository Management Using Artificial Intelligence Collaborate and Share Project Management and Team Sciences Share your Work
• Examples A Reproducible Research Workflow with AirBnB Data A Simple Reproducible Research Workflow A Simple Reproducible Web Scraping Example An Interactive Shiny App of Google's COVID-19 Data Exploring Trends in the Cars Dataset with Regression Analysis A Reproducible Workflow Using Snakemake and R Find Keywords and Sentences from Multiple PDF Files A Primer on Bayesian Inference for Accounting Research
• About About Tilburg Science Hub Meet our contributors Visit our blog

Navigating Scopus for Effective Literature Research

Introduction.

In the realm of academia, conducting a comprehensive literature review is a crucial step towards building a strong foundation for your scholarly work, whether it’s a paper, thesis, or any other research project. As you embark on this journey of knowledge discovery, having the right tools at your disposal is essential, and one such powerful tool is Scopus.

This tutorial is designed for individuals who are new to Scopus, offering a guided tour through its features and functionalities. Whether you’re a student navigating the intricacies of academic research or a newcomer to the world of literature reviews, this tutorial aims to simplify the process of utilizing Scopus to its full potential.

By the end of this tutorial, you’ll not only be comfortable using Scopus but also equipped with the skills to conduct a thorough and well-informed literature review, laying the groundwork for impactful research.

The structure of this tutorial is as follows:

• Introduction & access
• Searching for documents
• Advanced search options

Let’s embark on this journey together and unlock what Scopus has to offer!

What is Scopus?

Scopus is a comprehensive abstract and citation database designed to facilitate academic research. Developed by Elsevier, Scopus covers a vast range of disciplines, including science, technology, medicine, social sciences, and arts and humanities. It provides researchers, scholars, and students with a centralized platform to access a wealth of scholarly literature, including academic journals, conference proceedings, patents, and other scientific publications.

You might be asking yourself: why Scopus? Well, as you will see during the course of this tutorial, Scopus is not only a comprehensive database for a variety of disciplines, it also has personalized features that make your research process more efficient and organized.

How to access

Scopus Database use typically requires a subscription or to be granted access through a research institution, university or library.

Subscription access to Scopus is only available for organizations and enterprises, hence check if that may be the case for your organization.

There is also an option to use Scopus for free, by scrolling down on the Elsevier Scopus landing page , find the button that says “View Scopus Preview”

Institutional access

The most common option to access Scopus is through a university and/or library, either by means of an authentication or just by simply being connected to the Wi-Fi.

Follow these steps for a smooth start to the Scopus experience:

Make sure you are connected to the university Wi-Fi or the VPN.

Navigate to the Scopus landing page , it should look as follows:

Click on the button “Sign in”.

• A box should appear, click on the button “Sign in via your organization”:

Type in the box the name of your university, click on the result.

You will be now asked to authenticate your university account. Once verified, you should have full access to the Scopus Database.

On the top left corner of the home page you can verify your institutional access. On the top right corner you can also check your personal account, where you can explore personalized features that we will dive into later in the tutorial.

Now that you are set up, we can get into exploring what Scopus has to offer.

Introduction to the User Interface

Personalized features.

By clicking on your account icon on the top right corner, a menu opens with the following personalized items:

• Saved lists: allows you to rename, edit, delete, add to or export your saved lists of papers or documents.
• Saved searches: allows you to rename, edit, delete, combine or set an alert for saved searches. You can also run a saved search to view the results since the search was last run.
• Alerts: allows you to edit, delete or change the status of your alerts. You can also check for new results based upon the date that the alert was created. Alerts are personalized notifications that you can set up to be updated when new research comes out on a topic of your interest.
• Export preferences: allows you to choose a preferred file type or reference management tool when exporting documents. This can be very useful when exporting references or citations of the literature you plan to use. In this section you can select the export settings of your preference, which will then be applied to your search sessions.

How to search & analyze results

Let’s start exploring the search features of Scopus!

For the purpose of this tutorial we will use the topic of unemployment, but you are free to choose whatever you feel more comfortable with.

Enter unemployment in the search box as follows:

By default, Scopus will search the word(s) in the Article title, Abstract and Keywords of documents. You can specify in which fields to search using the drop-down menu. Some options include: authors, sources, affiliation. For the purpose of this tutorial, we will stick with the default option, but feel free to play around!

To expand the search to additional fields, click the “+ Add search field”. A new search bar will appear, let’s use long term for this search.

Notice that the two searches will be connected by the logical operator AND, meaning that the search aims to include both terms. Alternatively, you can opt for OR and AND NOT operators, depending on your purpose, but don’t worry about this as the topic of logical operators will be covered later in this tutorial.

• You can also set a specific date range for search results in the following way:

For more information on setting up a search query in Scopus, you can select ‘Search tips.’

Click the search button and let’s see what the results are!

Refining your search

Below is the overview of the search results. As you can see there is a large number of documents, they might not all be relevant for your research. To this purpose, let’s see how you can refine your search in a few easy steps to get closer to the literature you actually need.

On the left side of the page, the column “refine search” has a few filters, we will walk through them and use some which aim at our research:

• Year: if you haven’t set this filter previously, there is still the possibility to do so.
• Author name: In case you are searching by author or combinations of authors.
• Subject area: You can restrict the search to your area of interest, in our case we will stick to social sciences
• Document type: Choose among different formats, we will choose article .
• Source title
• Publication stage
• Keyword: You can select more keywords to further restrict your search, let’s try selecting unemployment, female and adult .
• Affiliation
• Funding sponsor
• Source type
• Language: Select your language of interest, it is sometimes overlooked as a way to reduce the number of documents you will have to look through!
• Open access

Once you are done with your selection, don’t forget to click Limit to .

There are a few more adjustments you can perform to the output of the search. Let’s have a look at them.

By default, the search results are sorted by date. Use the ‘Sort by’ drop-down menu to sort in a different order. One that might be useful if you are writing a thesis is sorting by highest cited, this allows you to immediately point out reliable sources of literature.

Another feature is the possibility to show the abstract by clicking ‘Show abstract’, useful to have a good first impression of a paper instead of opening it.

Analyze results

By clicking the feature ‘Analyze results’ on a search results page provides an analysis of your search and shows you the number of documents in your results broken down (on separate tabs) by year, source, author, affiliation, country, document type, subject area and funding sponsor. You can click on individual cards to expand and view additional data.

In the above image you can see the amount of documents published on our topic through time, it depicts a clear increase in research especially in the last decade. This can be very useful for your research as it points out which years were more prolific for researchers.

By scrolling down you can inspect additional data. Authors with the most documents, countries, affiliations and subject areas can be inspected in more detail.

Working with a document

If you find a title or abstract interesting, it is always a good idea to open the page for that document as you can see below:

Scopus offers the following features and insights:

• Click an author name to go to the details page for that author.
• You can see the button ‘View PDF’ if you have direct access to the pdf version of the file; otherwise click on ‘Full text options’ to check other access types.
• View the three most recent documents to cite this article in the top right corner.
• ‘Metrics’ are article level metrics which allow you to evaluate both citation impact and levels of community engagement around an article.
• By scrolling down you can view the ‘References’ cited in this document. The titles link to the abstract pages for those articles.

Saving your search

Keeping track of the literature you come across and that could be useful for your literature review is extremely important. Scopus offers the possibility to save an article thanks to the ‘Save to list’ button easily indicated by a star.

By clicking that, a box will appear asking you to save the document in either a new or existing list as you can see in the image below:

The files will then be stored in the personalized area ‘Saved lists’ in ‘My Scopus’ which you can access by clicking on your personal profile:

You can do something similar also with your search query. Navigate back to the search results page:

Click on save search and choose a name to easily identify your query. It will be stored in your personal area under ‘Saved searches’.

Advanced search

The advanced search feature allows you to create a more complex search using field codes, proximity operators or Boolean operators.

For this part of the tutorial let’s change our research topic to how does climate change affect coffee production.

Type in the search bar coffee and then click on ‘Advanced document search’ as you can see below:

To start this part of the tutorial reset the previous search by removing filters and keywords in the search bar.

The keyword ‘coffee’ was carried over to the advanced search tab.

On the right side of the page you can explore the different operators, by positioning the cursor on the operator you can get a brief description of its function. By clicking on the ‘+’ a paragraph explaining the function pops up and the operator will also be added to the query.

Let’s start by adding AND to our search, as you type, notice that a drop down menu opens with potential suggestions, follow the operator with ‘TITLE-ABS-KEY’ and within the parenthesis write production so that both will be searched in the title, abstract and keywords of documents. Using the same logic, add also ‘climate’.

This is what our search query should look like: TITLE-ABS-KEY(coffee) AND TITLE-ABS-KEY(production) AND TITLE-ABS-KEY(climate).

Now that the keywords are defined let’s scroll down and explore the filters, which are essentially the same ones as the previous section.

Let’s select:

• Under document filter, only open access files.
• Subject areas: agricultural and biological sciences (under life sciences) and economics, econometrics and finance (under social sciences).

Click on ‘search’, you will be directed to the search results page we are already familiar with.

If you realize you have limited your results too far you can modify your search, perhaps remove some filters. It is also possible to edit your advanced search string, for example you can remove the ‘open access’ requirement as your institution might have free access to a lot of resources.

To view the search string more clearly as you modify it click on ‘Outline query’, highlight the unwanted items and delete them as you can see below:

Accessing Scopus : Check institutional access or use the free preview option on the Elsevier Scopus landing page.

Utilizing Personalized Features : Use ‘Saved lists’, ‘Saved searches’, and set up ‘Alerts’ to organize and track research.

Basic Search Techniques : Enter keywords and refine search results using filters.

Analyzing Search Results : Use ‘Analyze results’ to gain insights on research trends and evaluate citation impact.

Saving Relevant Documents : Use ‘Save to list’ to save articles and queries for future reference.

Advanced Search Techniques : Utilize Boolean operators and field codes to create complex search queries and refine results.

By following these key steps, you can effectively utilize Scopus for comprehensive literature research, making your academic journey smoother and more productive!

Related Posts

Chapter structure of a thesis or academic paper.

The thesis or paper structure, with tips for the content of each chapter

Coding, Workflow, and Data Management in a Research Project/Thesis

Coding, workflow and data management as the core foundation of your research.

Master Thesis Resource Bundle

The Master Thesis Guide will serve as a guidance throughout the whole writing process of a thesis that consists of an empirical research project, for studies in marketing, economics, management, etc..

Google Analytics (functional)

This code gives us insight into the number of people that visit our website, where they are from and what they are clicking on.

Google Tag Manager (functional)

This code manages the deployment of various marketing and analytics tags on the website.

Want to know more? Check our Disclaimer .

Conducting a Literature Review (PGBS)

• Preparing to Do a Lit Review

Why use Scopus?

Scopus search video tutorials.

• ProQuest Dissertations & Theses
• Business Source Complete + PsycINFO
• Pepperdine University's Library Catalog
• Evaluating Citations
• Tracking an Article in Time
• Google Scholar: Promises & Pitfalls
• Resources for Citing Your Sources

Drescher Graduate Campus Librarian

Pepperdine's subscription to the Scopus database provides the easiest method to trace citations in both directions - backwards and forwards. While Scopus doesn't provide full-text access to articles, it does assist in tracking and analyzing the output (writing) of researchers. The focus of this page is on providing brief video tutorials for using  Scopus .

• Scopus Scopus is the largest abstract and citation database of peer-reviewed literature: scientific journals, books and conference proceedings.
• Searching for documents This tutorial demonstrates how to create and run a search using the Scopus Document search form
• Reviewing search results This tutorial demonstrates features available on the Search Results page
• Using Scopus article metrics This tutorial demonstrates how Article Metrics are used in Scopus
• << Previous: Searching the Literature
• Next: ProQuest Dissertations & Theses >>
• Last Updated: May 24, 2024 9:28 AM
• URL: https://infoguides.pepperdine.edu/litreview

Explore. Discover. Create.

Copyright ©  2022  Pepperdine University

Systematic Review

• Systematic reviews

Being systematic

Search terms, choosing databases, finding additional resources.

• Search techniques
• Systematically search databases
• Appraisal & synthesis
• Reporting findings
• Systematic review tools

Searching literature systematically is useful for all types of literature reviews!

However, if you are writing a systematic literature review the search needs to be particularly well planned and structured to ensure it is:

• comprehensive
• transparent

These help ensure bias is eliminated and the review is methodologically sound.

To achieve the above goals, you will need to:

• create a search strategy and ensure it is reviewed by your research group
• document each stage of your literature searching
• report each stage of quality appraisal 

Identify the key concepts in your research question

The first step in developing your search strategy is identifying the key concepts your research question covers.

• A preliminary search is often done to understand the topic and to refine your research question. 

Identify search terms

Use an iterative process to identify useful search terms for conducting your search. 

• Brainstorm keywords and phrases that can describe each concept you have identified in your research question.
• Create a table to record these keywords
• Select your keywords carefully
• Check against inclusion/exclusion criteria
• Repeated testing   is required to create a robust search strategy for a systematic review
• Run your search on your primary database and evaluate the first page of records to see how suitable your search is
• Identify reasons for irrelevant results and adjust your keywords accordingly 
• Consider whether it would be useful to use broader or narrower terms for your concepts
• Identify keywords in relevant results that you could add to your search to retrieve more relevant resources

Using a concept map or a mind map may help you clarify concepts and the relationships between or within concepts. Watch these YouTube videos for some ideas: 

• How to make a concept map  (by Lucidchart)
• Make sense of this mess world - mind maps  (by Sheng Huang)

Example keywords table:

Research question: What is the relationship between adverse childhood experiences and depression in mothers during the perinatal period? 

Revise your strategy/search terms until :

• the results match your research question
• you are confident you will find all the relevant literature on your topic

See Creating search strings for information on how to enter your search terms into databases. 

Example search string (using Scopus's Advanced search option) for the terms in the above table:

(TITLE-ABS-KEY("advserse childhood experienc*" OR ACE OR "childhood trauma") AND TITLE-ABS-KEY("perinatal depress*" OR "postpartum depress*" OR "postnatal depress*" OR "maternal mental health" OR "maternal psychological distress") AND TITLE-ABS-KEY(mother* OR women*))

See Subject headings  for information on including these database specific terms to your search terms.

Systematic reviewers usually use several databases to search for literature. This ensures that the searching is comprehensive and biases are minimised. 

Use both subject-specific and multidisciplinary databases to find resources relevant to your research question:

• Subject-specific databases: in-depth coverage of literature specific to a research field.
• Multi-disciplinary databases: literature from many research fields - help you find resources from disciplines you may not have considered.

Check for databases in your subject area via the Databases tab > Find by subject on the library homepage .

Find the  key databases that are often used for systematic reviews in this guide. 

Test searches to determine database usefulness. You can consult your Liaison Librarians to finalise the list of databases for your review.

Recommendations:

For all systematic reviews we recommend using Scopus , a high-quality, multidisciplinary database:

• Scopus is an abstract and citation database with links to full text on publisher websites or in other databases.
• Scopus indexes a curated collection of high quality journals along with books and conference proceedings.
• Research outputs are across a range of fields - science, technology, medicine, social science, arts and humanities.

For systematic reviews within the health/biomedical field, we recommend including Medline as one of the databases for your review:

MEDLINE  (via Ebsco, via Ovid, via PubMed)

• Medline is the National Library of Medicine’s (NLM) article citation database.
• Medline is hosted individually on a variety of platforms (EBSCO, OVID) and comprises the majority of PubMed.
• Articles in Medline are indexed using MeSH headings. See Subject headings for more information on MeSH.

Note: PubMed contains all of Medline and additional citations, e.g. books, manuscripts, citations that predate Medline.

To ensure your search is comprehensive you may need to search beyond academic databases when conducting a systematic review, particularly to find grey literature  (literature not published commercially and outside traditional academic sources such as journals).

Google Scholar

Google Scholar contains academic resources across disciplines and sources types. These come from academic publishers, professional societies, online repositories, universities and web sites.

Use Google Scholar

• as an additional tool to locate relevant publications not included in high-level academic databases
• for finding grey literature such as postgraduate theses and conference proceedings

You can limit your search to the type of websites by using site:ac . nz; site:edu

Note that Google Scholar searches are not as replicable or transparent as academic database searches, and may find large numbers of results.

Other sources of grey literature

• Grey literature checklist  (health related grey literature)
• OpenGrey  
• Public health Ontario guide to appraising grey literature
• Institutional Repository for Information Sharing (IRIS)
• Google search: use it for finding government reports, policies, theses, etc. You can limit your search to a particular type of websites by including site : govt.nz, site: . gov, site: . ac . nz, site: . edu, in your search

Watch our Finding grey literature  video (3.49 mins) online.

• << Previous: Planning
• Next: Search techniques >>
• Last Updated: May 29, 2024 9:20 AM
• URL: https://aut.ac.nz.libguides.com/systematic_reviews

Gerstein Science Information Centre

Searching the literature: a guide to comprehensive searching in the health sciences.

• Formulate your question
• Precision vs. Sensitivity
• Gather synonyms
• Construct strategy using OR/AND
• Use subject headings & textwords
• Choose your database(s)
• Conduct your search
• Test Your Search
• Translating your Search
• Supplement your search
• Manage your results
• Locating Full Text
• OVID Medline
• OVID Embase
• Ebsco CINAHL
• Cochrane Central

SCOPUS Video Tutorials

• Web of Science
• ERIC (ProQuest)
• AgeLine (EBSCO)
• Grey Literature
• Search Filters
• Saving and Documenting Your Search
• Can ChatGPT write a comprehensive search strategy?
• Systematic & Scoping Review Methods
• Searching Series Workshop: Course Materials This link opens in a new window

This series of videos is designed to guide you through searching comprehensively in SCOPUS. 

1. Introduction to SCOPUS

2. Searching with Textwords

3. Advanced Textword Searching

4. Combining Search Lines

5. Saving and Documenting Searches

6. Exporting Searches

7. Search Analyzer

This work is openly licensed via  CC BY-NC-SA 4.0 .  For information on this guide contact  Erica Nekolaichuk , Faculty Liaison & Instruction Librarian at the Gerstein Science Information Centre. 

• << Previous: Cochrane Central
• Next: Web of Science >>
• Last Updated: May 16, 2024 12:05 PM
• URL: https://guides.library.utoronto.ca/comprehensivesearching

Library links

• Gerstein Home
• U of T Libraries Home
• Renew items and pay fines
• Library hours
• Contact Gerstein
• University of Toronto Libraries
• UT Mississauga Library
• UT Scarborough Library
• Information Commons
• All libraries

© University of Toronto . All rights reserved.

Connect with us

• UNC Libraries
• HSL Academic Process
• Systematic Reviews
• Step 3: Conduct Literature Searches

Systematic Reviews: Step 3: Conduct Literature Searches

Created by health science librarians.

• Step 1: Complete Pre-Review Tasks
• Step 2: Develop a Protocol

About Step 3: Conduct Literature Searches

Partner with a librarian, systematic searching process, choose a few databases, search with controlled vocabulary and keywords, acknowledge outdated or offensive terminology, helpful tip - building your search, use nesting, boolean operators, and field tags, build your search, translate to other databases and other searching methods, document the search, updating your review.

• Searching FAQs
• Step 4: Manage Citations
• Step 5: Screen Citations
• Step 6: Assess Quality of Included Studies
• Step 7: Extract Data from Included Studies
• Step 8: Write the Review

  Check our FAQ's

   Email us

   Call (919) 962-0800

   Make an appointment with a librarian

  Request a systematic or scoping review consultation

Search the FAQs

In Step 3, you will design a search strategy to find all of the articles related to your research question. You will:

• Define the main concepts of your topic
• Choose which databases you want to search
• List terms to describe each concept
• Add terms from controlled vocabulary like MeSH
• Use field tags to tell the database where to search for terms
• Combine terms and concepts with Boolean operators AND and OR
• Translate your search strategy to match the format standards for each database
• Save a copy of your search strategy and details about your search

There are many factors to think about when building a strong search strategy for systematic reviews. Librarians are available to provide support with this step of the process.

Click an item below to see how it applies to Step 3: Conduct Literature Searches.

Reporting your review with PRISMA

For PRISMA, there are specific items you will want to report from your search.  For this step, review the PRISMA-S checklist.

• PRISMA-S for Searching
• Specify all databases, registers, websites, organizations, reference lists, and other sources searched or consulted to identify studies. Specify the date when each source was last searched or consulted. Present the full search strategies for all databases, registers and websites, including any filters and limits used.
• For information on how to document database searches and other search methods on your PRISMA flow diagram, visit our FAQs "How do I document database searches on my PRISMA flow diagram?" and "How do I document a grey literature search for my PRISMA flow diagram?"

Managing your review with Covidence

For this step of the review, in Covidence you can:

• Document searches in Covidence review settings so all team members can view
• Add keywords from your search to be highlighted in green or red while your team screens articles in your review settings

How a librarian can help with Step 3

When designing and conducting literature searches, a librarian can advise you on :

• How to create a search strategy with Boolean operators, database-specific syntax, subject headings, and appropriate keywords 
• How to apply previously published systematic review search strategies to your current search
• How to test your search strategy's performance 
• How to translate a search strategy from one database's preferred structure to another

The goal of a systematic retrieve is to find all results that are relevant to your topic. Because systematic review searches can be quite extensive and retrieve large numbers of results, an important aspect of systematic searching is limiting the number of irrelevant results that need to be screened. Librarians are experts trained in literature searching and systematic review methodology. Ask us a question or partner with a librarian to save time and improve the quality of your review. Our comparison chart detailing two tiers of partnership provides more information on how librarians can collaborate with and contribute to systematic review teams.

Search Process

• Use controlled vocabulary, if applicable
• Include synonyms/keyword terms
• Choose databases, websites, and/or registries to search
• Translate to other databases
• Search using other methods (e.g. hand searching)
• Validate and peer review the search

Databases can be multidisciplinary or subject specific. Choose the best databases for your research question. Databases index various journals, so in order to be comprehensive, it is important to search multiple databases when conducting a systematic review. Consider searching databases with more diverse or global coverage (i.e., Global Index Medicus) when appropriate. A list of frequently used databases is provided below. You can access UNC Libraries' full listing of databases on the HSL website (arranged alphabetically or by subject ).

Generally speaking, when literature searching, you are not searching the full-text article. Instead, you are searching certain citation data fields, like title, abstract, keyword, controlled vocabulary terms, and more. When developing a literature search, a good place to start is to identify searchable concepts of the research question, and then expand by adding other terms to describe those concepts. Read below for more information and examples on how to develop a literature search, as well as find tips and tricks for developing more comprehensive searches.

Identify search concepts and terms for each

Start by identifying the main concepts of your research question. If unsure, try using a question framework to help identify the main searchable concepts. PICO is one example of a question framework and is used specifically for clinical questions. If your research question doesn't fit into the PICO model well, view other examples of question frameworks and try another!

View our example in PICO format

Question: for patients 65 years and older, does an influenza vaccine reduce the future risk of pneumonia, controlled vocabulary.

Controlled vocabulary is a set of terminology assigned to citations to describe the content of each reference. Searching with controlled vocabulary can improve the relevancy of search results. Many databases assign controlled vocabulary terms to citations, but their naming schema is often specific to each database. For example, the controlled vocabulary system searchable via PubMed is MeSH, or Medical Subject Headings. More information on searching MeSH can be found on the HSL PubMed Ten Tips Legacy Guide .

Note: Controlled vocabulary may be outdated, and some databases allow users to submit requests to update terminology.

View Controlled Vocabulary for our example PICO

As mentioned above, databases with controlled vocabulary often use their own unique system. A listing of controlled vocabulary systems by database is shown below.

Keyword Terms

Not all citations are indexed with controlled vocabulary terms, however, so it is important to combine controlled vocabulary searches with keyword, or text word, searches. 

Authors often write about the same topic in varied ways and it is important to add these terms to your search in order to capture most of the literature. For example, consider these elements when developing a list of keyword terms for each concept:

• American versus British spelling
• hyphenated terms
• quality of life
• satisfaction
• vaccination
• influenza vaccination

There are several resources to consider when searching for synonyms. Scan the results of preliminary searches to identify additional terms. Look for synonyms, word variations, and other possibilities in Wikipedia, other encyclopedias or dictionaries, and databases. For example, PubChem lists additional drug names and chemical compounds.

Display Controlled Vocabulary and Keywords for our example PICO

Combining controlled vocabulary and text words in PubMed would look like this:

"Influenza Vaccines"[Mesh] OR "influenza vaccine" OR "influenza vaccines" OR "flu vaccine" OR "flu vaccines" OR "flu shot" OR "flu shots" OR "influenza virus vaccine" OR "influenza virus vaccines"

Social and cultural norms have been rapidly changing around the world. This has led to changes in the vocabulary used, such as when describing people or populations. Library and research terminology changes more slowly, and therefore can be considered outdated, unacceptable, or overly clinical for use in conversation or writing.

For our example with people 65 years and older, APA Style Guidelines recommend that researchers use terms like “older adults” and “older persons” and forgo terms like “senior citizens” and “elderly” that connote stereotypes. While these are current recommendations, researchers will recognize that terms like “elderly” have previously been used in the literature. Therefore, removing these terms from the search strategy may result in missed relevant articles. 

Research teams need to discuss current and outdated terminology and decide which terms to include in the search to be as comprehensive as possible. The research team or a librarian can search for currently preferred terms in glossaries, dictionaries, published guidelines, and governmental or organizational websites. The University of Michigan Library provides suggested wording to use in the methods section when antiquated, non-standard, exclusionary, or potentially offensive terms are included in the search.

Check the methods sections or supplementary materials of published systematic reviews for search strategies to see what terminology they used. This can help inform your search strategy by using MeSH terms or keywords you may not have thought of. However, be aware that search strategies will differ in their comprehensiveness.

You can also run a preliminary search for your topic, sort the results by Relevance or Best Match, and skim through titles and abstracts to identify terminology from relevant articles that you should include in your search strategy.

Nesting is a term that describes organizing search terms inside parentheses. This is important because, just like their function in math, commands inside a set of parentheses occur first. Parentheses let the database know in which order terms should be combined. 

Always combine terms for a single concept inside a parentheses set. For example: 

( "Influenza Vaccines"[Mesh] OR "influenza vaccine" OR "influenza vaccines" OR "flu vaccine" OR "flu vaccines" OR "flu shot" OR "flu shots" OR "influenza virus vaccine" OR "influenza virus vaccines" )

Additionally, you may nest a subset of terms for a concept inside a larger parentheses set, as seen below. Pay careful attention to the number of parenthesis sets and ensure they are matched, meaning for every open parentheses you also have a closed one.

( "Influenza Vaccines"[Mesh] OR "influenza vaccine" OR "influenza vaccines" OR "flu vaccine" OR "flu vaccines" OR "flu shot" OR "flu shots" OR "influenza virus vaccine" OR "influenza virus vaccines" OR   (( flu OR influenza ) AND ( vaccine OR vaccines OR vaccination OR immunization )))

Boolean operators

Boolean operators are used to combine terms in literature searches. Searches are typically organized using the Boolean operators OR or AND. OR is used to combine search terms for the same concept (i.e., influenza vaccine). AND is used to combine different concepts (i.e., influenza vaccine AND older adults AND pneumonia). An example of how Boolean operators can affect search retrieval is shown below. Using AND to combine the three concepts will only retrieve results where all are present. Using OR to combine the concepts will retrieve results that use all separately or together. It is important to note that, generally speaking, when you are performing a literature search you are only searching the title, abstract, keywords and other citation data. You are not searching the full-text of the articles.

The last major element to consider when building systematic literature searches are field tags. Field tags tell the database exactly where to search. For example, you can use a field tag to tell a database to search for a term in just the title, the title and abstract, and more. Just like with controlled vocabulary, field tag commands are different for every database.

If you do not manually apply field tags to your search, most databases will automatically search in a set of citation data points. Databases may also overwrite your search with algorithms if you do not apply field tags. For systematic review searching, best practice is to apply field tags to each term for reproducibility.

For example:

("Influenza Vaccines"[Mesh] OR "influenza vaccine"[tw] OR "influenza vaccines"[tw] OR "flu vaccine"[tw] OR "flu vaccines"[tw] OR "flu shot"[tw] OR "flu shots"[tw] OR "influenza virus vaccine"[tw] OR "influenza virus vaccines"[tw] OR ((flu[tw] OR influenza[tw]) AND (vaccine[tw] OR vaccines[tw] OR vaccination[tw] OR immunization[tw])))

View field tags for several health databases

For more information about how to use a variety of databases, check out our guides on searching.

• Searching PubMed guide Guide to searching Medline via the PubMed database
• Searching Embase guide Guide to searching Embase via embase.com
• Searching Scopus guide Guide to searching Scopus via scopus.com
• Searching EBSCO Databases guide Guide to searching CINAHL, PsycInfo, Global Health, & other databases via EBSCO

Combining search elements together

Organizational structure of literature searches is very important. Specifically, how terms are grouped (or nested) and combined with Boolean operators will drastically impact search results. These commands tell databases exactly how to combine terms together, and if done incorrectly or inefficiently, search results returned may be too broad or irrelevant.

For example, in PubMed:

(influenza OR flu) AND vaccine is a properly combined search and it produces around 50,000 results.

influenza OR flu AND vaccine is not properly combined.  Databases may read it as everything about influenza OR everything about (flu AND vaccine), which would produce more results than needed.

We recommend one or more of the following:

• put all your synonyms together inside a set of parentheses, then put AND between the closing parenthesis of one set and the opening parenthesis of the next set
• use a separate search box for each set of synonyms
• run each set of synonyms as a separate search, and then combine all your searches
• ask a librarian if your search produces too many or too few results

View the proper way to combine MeSH terms and Keywords for our example PICO

Question: for patients 65 years and older, does an influenza vaccine reduce the future risk of pneumonia , translating search strategies to other databases.

Databases often use their own set of terminology and syntax. When searching multiple databases, you need to adjust the search slightly to retrieve comparable results. Our sections on Controlled Vocabulary and Field Tags have information on how to build searches in different databases.  Resources to help with this process are listed below.

• Polyglot search A tool to translate a PubMed or Ovid search to other databases
• Search Translation Resources (Cornell) A listing of resources for search translation from Cornell University
• Advanced Searching Techniques (King's College London) A collection of advanced searching techniques from King's College London

Other searching methods

Hand searching.

Literature searches can be supplemented by hand searching. One of the most popular ways this is done with systematic reviews is by searching the reference list and citing articles of studies included in the review. Another method is manually browsing key journals in your field to make sure no relevant articles were missed. Other sources that may be considered for hand searching include: clinical trial registries, white papers and other reports, pharmaceutical or other corporate reports, conference proceedings, theses and dissertations, or professional association guidelines.

Searching grey literature

Grey literature typically refers to literature not published in a traditional manner and often not retrievable through large databases and other popular resources. Grey literature should be searched for inclusion in systematic reviews in order to reduce bias and increase thoroughness. There are several databases specific to grey literature that can be searched.

• Open Grey Grey literature for Europe
• OAIster A union catalog of millions of records representing open access resources from collections worldwide
• Grey Matters: a practical tool for searching health-related grey literature (CADTH) From CADTH, the Canadian Agency for Drugs and Technologies in Health, Grey Matters is a practical tool for searching health-related grey literature. The MS Word document covers a grey literature checklist, including national and international health technology assessment (HTA) web sites, drug and device regulatory agencies, clinical trial registries, health economics resources, Canadian health prevalence or incidence databases, and drug formulary web sites.
• Duke Medical Center Library: Searching for Grey Literature A good online compilation of resources by the Duke Medical Center Library.

Systematic review quality is highly dependent on the literature search(es) used to identify studies. To follow best practices for reporting search strategies, as well as increase reproducibility and transparency, document various elements of the literature search for your review. To make this process more clear, a statement and checklist for reporting literature searches has been developed and and can be found below.

• PRISMA-S: Reporting Literature Searches in Systematic Reviews
• Section 4.5 Cochrane Handbook - Documenting and reporting the search process

At a minimum, document and report certain elements, such as databases searched, including name (i.e., Scopus) and platform (i.e. Elsevier), websites, registries, and grey literature searched. In addition, this also may include citation searching and reaching out to experts in the field. Search strategies used in each database or source should be documented, along with any filters or limits, and dates searched. If a search has been updated or was built upon previous work, that should be noted as well. It is also helpful to document which search terms have been tested and decisions made for term inclusion or exclusion by the team. Last, any peer review process should be stated as well as the total number of records identified from each source and how deduplication was handled. 

If you have a librarian on your team who is creating and running the searches, they will handle the search documentation.

You can document search strategies in word processing software you are familiar with like Microsoft Word or Excel, or Google Docs or Sheets. A template, and separate example file, is provided below for convenience. 

• Search Strategy Documentation Template
• Search Strategy Documentation Example

*Some databases like PubMed are being continually updated with new technology and algorithms. This means that searches may retrieve different results than when originally run, even with the same filters, date limits, etc.

When you decide to update a systematic review search, there are two ways of identifying new articles:  

1. rerun the original search strategy without any changes. .

Rerun the original search strategy without making any changes.  Import the results into your citation manager, and remove all articles duplicated from the original set of search results.

2. Rerun the original search strategy and add an entry date filter.

Rerun the original search strategy and add a date filter for when the article was added to the database ( not the publication date).  An entry date filter will find any articles added to the results since you last ran the search, unlike a publication date filter, which would only find more recent articles.

Some examples of entry date filters for articles entered since December 31, 2021 are:

• PubMed:   AND ("2021/12/31"[EDAT] : "3000"[EDAT])
• Embase: AND [31-12-2021]/sd
• CINAHL:   AND EM 20211231-20231231
• PsycInfo: AND RD 20211231-20231231
• Scopus:   AND LOAD-DATE AFT 20211231  

Your PRISMA flow diagram

For more information about updating the PRISMA flow diagram for your systematic review, see the information on filling out a PRISMA flow diagram for review updates on the Step 8: Write the Review page of the guide.

• << Previous: Step 2: Develop a Protocol
• Next: Step 4: Manage Citations >>
• Last Updated: May 16, 2024 3:24 PM
• URL: https://guides.lib.unc.edu/systematic-reviews

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List

Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study

Wichor m. bramer.

1 Medical Library, Erasmus MC, Erasmus University Medical Centre Rotterdam, 3000 CS Rotterdam, the Netherlands

Melissa L. Rethlefsen

2 Spencer S. Eccles Health Sciences Library, University of Utah, Salt Lake City, Utah USA

Jos Kleijnen

3 Kleijnen Systematic Reviews Ltd., York, UK

4 School for Public Health and Primary Care (CAPHRI), Maastricht University, Maastricht, the Netherlands

Oscar H. Franco

5 Department of Epidemiology, Erasmus MC, Erasmus University Medical Centre Rotterdam, Rotterdam, the Netherlands

Associated Data

The datasets generated and/or analyzed during the current study are available from the corresponding author on a reasonable request.

Within systematic reviews, when searching for relevant references, it is advisable to use multiple databases. However, searching databases is laborious and time-consuming, as syntax of search strategies are database specific. We aimed to determine the optimal combination of databases needed to conduct efficient searches in systematic reviews and whether the current practice in published reviews is appropriate. While previous studies determined the coverage of databases, we analyzed the actual retrieval from the original searches for systematic reviews.

Since May 2013, the first author prospectively recorded results from systematic review searches that he performed at his institution. PubMed was used to identify systematic reviews published using our search strategy results. For each published systematic review, we extracted the references of the included studies. Using the prospectively recorded results and the studies included in the publications, we calculated recall, precision, and number needed to read for single databases and databases in combination. We assessed the frequency at which databases and combinations would achieve varying levels of recall (i.e., 95%). For a sample of 200 recently published systematic reviews, we calculated how many had used enough databases to ensure 95% recall.

A total of 58 published systematic reviews were included, totaling 1746 relevant references identified by our database searches, while 84 included references had been retrieved by other search methods. Sixteen percent of the included references (291 articles) were only found in a single database; Embase produced the most unique references ( n  = 132). The combination of Embase, MEDLINE, Web of Science Core Collection, and Google Scholar performed best, achieving an overall recall of 98.3 and 100% recall in 72% of systematic reviews. We estimate that 60% of published systematic reviews do not retrieve 95% of all available relevant references as many fail to search important databases. Other specialized databases, such as CINAHL or PsycINFO, add unique references to some reviews where the topic of the review is related to the focus of the database.

Conclusions

Optimal searches in systematic reviews should search at least Embase, MEDLINE, Web of Science, and Google Scholar as a minimum requirement to guarantee adequate and efficient coverage.

Electronic supplementary material

The online version of this article (10.1186/s13643-017-0644-y) contains supplementary material, which is available to authorized users.

Investigators and information specialists searching for relevant references for a systematic review (SR) are generally advised to search multiple databases and to use additional methods to be able to adequately identify all literature related to the topic of interest [ 1 – 6 ]. The Cochrane Handbook, for example, recommends the use of at least MEDLINE and Cochrane Central and, when available, Embase for identifying reports of randomized controlled trials [ 7 ]. There are disadvantages to using multiple databases. It is laborious for searchers to translate a search strategy into multiple interfaces and search syntaxes, as field codes and proximity operators differ between interfaces. Differences in thesaurus terms between databases add another significant burden for translation. Furthermore, it is time-consuming for reviewers who have to screen more, and likely irrelevant, titles and abstracts. Lastly, access to databases is often limited and only available on subscription basis.

Previous studies have investigated the added value of different databases on different topics [ 8 – 15 ]. Some concluded that searching only one database can be sufficient as searching other databases has no effect on the outcome [ 16 , 17 ]. Nevertheless others have concluded that a single database is not sufficient to retrieve all references for systematic reviews [ 18 , 19 ]. Most articles on this topic draw their conclusions based on the coverage of databases [ 14 ]. A recent paper tried to find an acceptable number needed to read for adding an additional database; sadly, however, no true conclusion could be drawn [ 20 ]. However, whether an article is present in a database may not translate to being found by a search in that database. Because of this major limitation, the question of which databases are necessary to retrieve all relevant references for a systematic review remains unanswered. Therefore, we research the probability that single or various combinations of databases retrieve the most relevant references in a systematic review by studying actual retrieval in various databases.

The aim of our research is to determine the combination of databases needed for systematic review searches to provide efficient results (i.e., to minimize the burden for the investigators without reducing the validity of the research by missing relevant references). A secondary aim is to investigate the current practice of databases searched for published reviews. Are included references being missed because the review authors failed to search a certain database?

Development of search strategies

At Erasmus MC, search strategies for systematic reviews are often designed via a librarian-mediated search service. The information specialists of Erasmus MC developed an efficient method that helps them perform searches in many databases in a much shorter time than other methods. This method of literature searching and a pragmatic evaluation thereof are published in separate journal articles [ 21 , 22 ]. In short, the method consists of an efficient way to combine thesaurus terms and title/abstract terms into a single line search strategy. This search is then optimized. Articles that are indexed with a set of identified thesaurus terms, but do not contain the current search terms in title or abstract, are screened to discover potential new terms. New candidate terms are added to the basic search and evaluated. Once optimal recall is achieved, macros are used to translate the search syntaxes between databases, though manual adaptation of the thesaurus terms is still necessary.

Review projects at Erasmus MC cover a wide range of medical topics, from therapeutic effectiveness and diagnostic accuracy to ethics and public health. In general, searches are developed in MEDLINE in Ovid (Ovid MEDLINE® In-Process & Other Non-Indexed Citations, Ovid MEDLINE® Daily and Ovid MEDLINE®, from 1946); Embase.com (searching both Embase and MEDLINE records, with full coverage including Embase Classic); the Cochrane Central Register of Controlled Trials (CENTRAL) via the Wiley Interface; Web of Science Core Collection (hereafter called Web of Science); PubMed restricting to records in the subset “as supplied by publisher” to find references that not yet indexed in MEDLINE (using the syntax publisher [sb]); and Google Scholar. In general, we use the first 200 references as sorted in the relevance ranking of Google Scholar. When the number of references from other databases was low, we expected the total number of potential relevant references to be low. In this case, the number of hits from Google Scholar was limited to 100. When the overall number of hits was low, we additionally searched Scopus, and when appropriate for the topic, we included CINAHL (EBSCOhost), PsycINFO (Ovid), and SportDiscus (EBSCOhost) in our search.

Beginning in May 2013, the number of records retrieved from each search for each database was recorded at the moment of searching. The complete results from all databases used for each of the systematic reviews were imported into a unique EndNote library upon search completion and saved without deduplication for this research. The researchers that requested the search received a deduplicated EndNote file from which they selected the references relevant for inclusion in their systematic review. All searches in this study were developed and executed by W.M.B.

Determining relevant references of published reviews

We searched PubMed in July 2016 for all reviews published since 2014 where first authors were affiliated to Erasmus MC, Rotterdam, the Netherlands, and matched those with search registrations performed by the medical library of Erasmus MC. This search was used in earlier research [ 21 ]. Published reviews were included if the search strategies and results had been documented at the time of the last update and if, at minimum, the databases Embase, MEDLINE, Cochrane CENTRAL, Web of Science, and Google Scholar had been used in the review. From the published journal article, we extracted the list of final included references. We documented the department of the first author. To categorize the types of patient/population and intervention, we identified broad MeSH terms relating to the most important disease and intervention discussed in the article. We copied from the MeSH tree the top MeSH term directly below the disease category or, in to case of the intervention, directly below the therapeutics MeSH term. We selected the domain from a pre-defined set of broad domains, including therapy, etiology, epidemiology, diagnosis, management, and prognosis. Lastly, we checked whether the reviews described limiting their included references to a particular study design.

To identify whether our searches had found the included references, and if so, from which database(s) that citation was retrieved, each included reference was located in the original corresponding EndNote library using the first author name combined with the publication year as a search term for each specific relevant publication. If this resulted in extraneous results, the search was subsequently limited using a distinct part of the title or a second author name. Based on the record numbers of the search results in EndNote, we determined from which database these references came. If an included reference was not found in the EndNote file, we presumed the authors used an alternative method of identifying the reference (e.g., examining cited references, contacting prominent authors, or searching gray literature), and we did not include it in our analysis.

Data analysis

We determined the databases that contributed most to the reviews by the number of unique references retrieved by each database used in the reviews. Unique references were included articles that had been found by only one database search. Those databases that contributed the most unique included references were then considered candidate databases to determine the most optimal combination of databases in the further analyses.

In Excel, we calculated the performance of each individual database and various combinations. Performance was measured using recall, precision, and number needed to read. See Table  1 for definitions of these measures. These values were calculated both for all reviews combined and per individual review.

Definitions of general measures of performance in searches

Performance of a search can be expressed in different ways. Depending on the goal of the search, different measures may be optimized. In the case of a clinical question, precision is most important, as a practicing clinician does not have a lot of time to read through many articles in a clinical setting. When searching for a systematic review, recall is the most important aspect, as the researcher does not want to miss any relevant references. As our research is performed on systematic reviews, the main performance measure is recall.

We identified all included references that were uniquely identified by a single database. For the databases that retrieved the most unique included references, we calculated the number of references retrieved (after deduplication) and the number of included references that had been retrieved by all possible combinations of these databases, in total and per review. For all individual reviews, we determined the median recall, the minimum recall, and the percentage of reviews for which each single database or combination retrieved 100% recall.

For each review that we investigated, we determined what the recall was for all possible different database combinations of the most important databases. Based on these, we determined the percentage of reviews where that database combination had achieved 100% recall, more than 95%, more than 90%, and more than 80%. Based on the number of results per database both before and after deduplication as recorded at the time of searching, we calculated the ratio between the total number of results and the number of results for each database and combination.

Improvement of precision was calculated as the ratio between the original precision from the searches in all databases and the precision for each database and combination.

To compare our practice of database usage in systematic reviews against current practice as evidenced in the literature, we analyzed a set of 200 recent systematic reviews from PubMed. On 5 January 2017, we searched PubMed for articles with the phrase “systematic review” in the title. Starting with the most recent articles, we determined the databases searched either from the abstract or from the full text until we had data for 200 reviews. For the individual databases and combinations that were used in those reviews, we multiplied the frequency of occurrence in that set of 200 with the probability that the database or combination would lead to an acceptable recall (which we defined at 95%) that we had measured in our own data.

Our earlier research had resulted in 206 systematic reviews published between 2014 and July 2016, in which the first author was affiliated with Erasmus MC [ 21 ]. In 73 of these, the searches and results had been documented by the first author of this article at the time of the last search. Of those, 15 could not be included in this research, since they had not searched all databases we investigated here. Therefore, for this research, a total of 58 systematic reviews were analyzed. The references to these reviews can be found in Additional file 1 . An overview of the broad topical categories covered in these reviews is given in Table  2 . Many of the reviews were initiated by members of the departments of surgery and epidemiology. The reviews covered a wide variety of disease, none of which was present in more than 12% of the reviews. The interventions were mostly from the chemicals and drugs category, or surgical procedures. Over a third of the reviews were therapeutic, while slightly under a quarter answered an etiological question. Most reviews did not limit to certain study designs, 9% limited to RCTs only, and another 9% limited to other study types.

Description of topics of included references (only values above 5% are shown)

Together, these reviews included a total of 1830 references. Of these, 84 references (4.6%) had not been retrieved by our database searches and were not included in our analysis, leaving in total 1746 references. In our analyses, we combined the results from MEDLINE in Ovid and PubMed (the subset as supplied by publisher) into one database labeled MEDLINE.

Unique references per database

A total of 292 (17%) references were found by only one database. Table  3 displays the number of unique results retrieved for each single database. Embase retrieved the most unique included references, followed by MEDLINE, Web of Science, and Google Scholar. Cochrane CENTRAL is absent from the table, as for the five reviews limited to randomized trials, it did not add any unique included references. Subject-specific databases such as CINAHL, PsycINFO, and SportDiscus only retrieved additional included references when the topic of the review was directly related to their special content, respectively nursing, psychiatry, and sports medicine.

Number of unique included references by each specific database

Overall performance

The four databases that had retrieved the most unique references (Embase, MEDLINE, Web of Science, and Google Scholar) were investigated individually and in all possible combinations (see Table  4 ). Of the individual databases, Embase had the highest overall recall (85.9%). Of the combinations of two databases, Embase and MEDLINE had the best results (92.8%). Embase and MEDLINE combined with either Google Scholar or Web of Science scored similarly well on overall recall (95.9%). However, the combination with Google Scholar had a higher precision and higher median recall, a higher minimum recall, and a higher proportion of reviews that retrieved all included references. Using both Web of Science and Google Scholar in addition to MEDLINE and Embase increased the overall recall to 98.3%. The higher recall from adding extra databases came at a cost in number needed to read (NNR). Searching only Embase produced an NNR of 57 on average, whereas, for the optimal combination of four databases, the NNR was 73.

Performance of several databases and database combinations in terms of sensitivity and precision

a Overall recall: The total number of included references retrieved by the databases divided by the total number of included references retrieved by all databases

b Median recall: The median value of recall per review

c Minimum recall: The lowest value of recall per review

d Percentage 100% recall: The percentage of reviews for which the database combination retrieved all included references

e Precision: The number of included references divided by the total number of references retrieved

f Number Needed to Read: The total number of references retrieved divided by the number of included references

Probability of appropriate recall

We calculated the recall for individual databases and databases in all possible combination for all reviews included in the research. Figure  1 shows the percentages of reviews where a certain database combination led to a certain recall. For example, in 48% of all systematic reviews, the combination of Embase and MEDLINE (with or without Cochrane CENTRAL; Cochrane CENTRAL did not add unique relevant references) reaches a recall of at least 95%. In 72% of studied systematic reviews, the combination of Embase, MEDLINE, Web of Science, and Google Scholar retrieved all included references. In the top bar, we present the results of the complete database searches relative to the total number of included references. This shows that many database searches missed relevant references.

Percentage of systematic reviews for which a certain database combination reached a certain recall. The X -axis represents the percentage of reviews for which a specific combination of databases, as shown on the y -axis, reached a certain recall (represented with bar colors). Abbreviations: EM Embase, ML MEDLINE, WoS Web of Science, GS Google Scholar. Asterisk indicates that the recall of all databases has been calculated over all included references. The recall of the database combinations was calculated over all included references retrieved by any database

Differences between domains of reviews

We analyzed whether the added value of Web of Science and Google Scholar was dependent of the domain of the review. For 55 reviews, we determined the domain. See Fig.  2 for the comparison of the recall of Embase, MEDLINE, and Cochrane CENTRAL per review for all identified domains. For all but one domain, the traditional combination of Embase, MEDLINE, and Cochrane CENTRAL did not retrieve enough included references. For four out of five systematic reviews that limited to randomized controlled trials (RCTs) only, the traditional combination retrieved 100% of all included references. However, for one review of this domain, the recall was 82%. Of the 11 references included in this review, one was found only in Google Scholar and one only in Web of Science.

Percentage of systematic reviews of a certain domain for which the combination Embase, MEDLINE and Cochrane CENTRAL reached a certain recall

Reduction in number of results

We calculated the ratio between the number of results found when searching all databases, including databases not included in our analyses, such as Scopus, PsycINFO, and CINAHL, and the number of results found searching a selection of databases. See Fig.  3 for the legend of the plots in Figs.  4 and ​ and5. 5 . Figure  4 shows the distribution of this value for individual reviews. The database combinations with the highest recall did not reduce the total number of results by large margins. Moreover, in combinations where the number of results was greatly reduced, the recall of included references was lower.

Legend of Figs. 3 and ​ and4 4

The ratio between number of results per database combination and the total number of results for all databases

The ratio between precision per database combination and the total precision for all databases

Improvement of precision

To determine how searching multiple databases affected precision, we calculated for each combination the ratio between the original precision, observed when all databases were searched, and the precision calculated for different database combinations. Figure  5 shows the improvement of precision for 15 databases and database combinations. Because precision is defined as the number of relevant references divided by the number of total results, we see a strong correlation with the total number of results.

Status of current practice of database selection

From a set of 200 recent SRs identified via PubMed, we analyzed the databases that had been searched. Almost all reviews (97%) reported a search in MEDLINE. Other databases that we identified as essential for good recall were searched much less frequently; Embase was searched in 61% and Web of Science in 35%, and Google Scholar was only used in 10% of all reviews. For all individual databases or combinations of the four important databases from our research (MEDLINE, Embase, Web of Science, and Google Scholar), we multiplied the frequency of occurrence of that combination in the random set, with the probability we found in our research that this combination would lead to an acceptable recall of 95%. The calculation is shown in Table  5 . For example, around a third of the reviews (37%) relied on the combination of MEDLINE and Embase. Based on our findings, this combination achieves acceptable recall about half the time (47%). This implies that 17% of the reviews in the PubMed sample would have achieved an acceptable recall of 95%. The sum of all these values is the total probability of acceptable recall in the random sample. Based on these calculations, we estimate that the probability that this random set of reviews retrieved more than 95% of all possible included references was 40%. Using similar calculations, also shown in Table  5 , we estimated the probability that 100% of relevant references were retrieved is 23%.

Calculation of probability of acceptable recall of a PubMed sample of systematic reviews

a Two reviews did not use any of the databases used in this evaluation

Our study shows that, to reach maximum recall, searches in systematic reviews ought to include a combination of databases. To ensure adequate performance in searches (i.e., recall, precision, and number needed to read), we find that literature searches for a systematic review should, at minimum, be performed in the combination of the following four databases: Embase, MEDLINE (including Epub ahead of print), Web of Science Core Collection, and Google Scholar. Using that combination, 93% of the systematic reviews in our study obtained levels of recall that could be considered acceptable (> 95%). Unique results from specialized databases that closely match systematic review topics, such as PsycINFO for reviews in the fields of behavioral sciences and mental health or CINAHL for reviews on the topics of nursing or allied health, indicate that specialized databases should be used additionally when appropriate.

We find that Embase is critical for acceptable recall in a review and should always be searched for medically oriented systematic reviews. However, Embase is only accessible via a paid subscription, which generally makes it challenging for review teams not affiliated with academic medical centers to access. The highest scoring database combination without Embase is a combination of MEDLINE, Web of Science, and Google Scholar, but that reaches satisfactory recall for only 39% of all investigated systematic reviews, while still requiring a paid subscription to Web of Science. Of the five reviews that included only RCTs, four reached 100% recall if MEDLINE, Web of Science, and Google Scholar combined were complemented with Cochrane CENTRAL.

The Cochrane Handbook recommends searching MEDLINE, Cochrane CENTRAL, and Embase for systematic reviews of RCTs. For reviews in our study that included RCTs only, indeed, this recommendation was sufficient for four (80%) of the reviews. The one review where it was insufficient was about alternative medicine, specifically meditation and relaxation therapy, where one of the missed studies was published in the Indian Journal of Positive Psychology . The other study from the Journal of Advanced Nursing is indexed in MEDLINE and Embase but was only retrieved because of the addition of KeyWords Plus in Web of Science. We estimate more than 50% of reviews that include more study types than RCTs would miss more than 5% of included references if only traditional combination of MEDLINE, Embase, and Cochrane CENTAL is searched.

We are aware that the Cochrane Handbook [ 7 ] recommends more than only these databases, but further recommendations focus on regional and specialized databases. Though we occasionally used the regional databases LILACS and SciELO in our reviews, they did not provide unique references in our study. Subject-specific databases like PsycINFO only added unique references to a small percentage of systematic reviews when they had been used for the search. The third key database we identified in this research, Web of Science, is only mentioned as a citation index in the Cochrane Handbook, not as a bibliographic database. To our surprise, Cochrane CENTRAL did not identify any unique included studies that had not been retrieved by the other databases, not even for the five reviews focusing entirely on RCTs. If Erasmus MC authors had conducted more reviews that included only RCTs, Cochrane CENTRAL might have added more unique references.

MEDLINE did find unique references that had not been found in Embase, although our searches in Embase included all MEDLINE records. It is likely caused by difference in thesaurus terms that were added, but further analysis would be required to determine reasons for not finding the MEDLINE records in Embase. Although Embase covers MEDLINE, it apparently does not index every article from MEDLINE. Thirty-seven references were found in MEDLINE (Ovid) but were not available in Embase.com . These are mostly unique PubMed references, which are not assigned MeSH terms, and are often freely available via PubMed Central.

Google Scholar adds relevant articles not found in the other databases, possibly because it indexes the full text of all articles. It therefore finds articles in which the topic of research is not mentioned in title, abstract, or thesaurus terms, but where the concepts are only discussed in the full text. Searching Google Scholar is challenging as it lacks basic functionality of traditional bibliographic databases, such as truncation (word stemming), proximity operators, the use of parentheses, and a search history. Additionally, search strategies are limited to a maximum of 256 characters, which means that creating a thorough search strategy can be laborious.

Whether Embase and Web of Science can be replaced by Scopus remains uncertain. We have not yet gathered enough data to be able to make a full comparison between Embase and Scopus. In 23 reviews included in this research, Scopus was searched. In 12 reviews (52%), Scopus retrieved 100% of all included references retrieved by Embase or Web of Science. In the other 48%, the recall by Scopus was suboptimal, in one occasion as low as 38%.

Of all reviews in which we searched CINAHL and PsycINFO, respectively, for 6 and 9% of the reviews, unique references were found. For CINAHL and PsycINFO, in one case each, unique relevant references were found. In both these reviews, the topic was highly related to the topic of the database. Although we did not use these special topic databases in all of our reviews, given the low number of reviews where these databases added relevant references, and observing the special topics of those reviews, we suggest that these subject databases will only add value if the topic is related to the topic of the database.

Many articles written on this topic have calculated overall recall of several reviews, instead of the effects on all individual reviews. Researchers planning a systematic review generally perform one review, and they need to estimate the probability that they may miss relevant articles in their search. When looking at the overall recall, the combination of Embase and MEDLINE and either Google Scholar or Web of Science could be regarded sufficient with 96% recall. This number however is not an answer to the question of a researcher performing a systematic review, regarding which databases should be searched. A researcher wants to be able to estimate the chances that his or her current project will miss a relevant reference. However, when looking at individual reviews, the probability of missing more than 5% of included references found through database searching is 33% when Google Scholar is used together with Embase and MEDLINE and 30% for the Web of Science, Embase, and MEDLINE combination. What is considered acceptable recall for systematic review searches is open for debate and can differ between individuals and groups. Some reviewers might accept a potential loss of 5% of relevant references; others would want to pursue 100% recall, no matter what cost. Using the results in this research, review teams can decide, based on their idea of acceptable recall and the desired probability which databases to include in their searches.

Strengths and limitations

We did not investigate whether the loss of certain references had resulted in changes to the conclusion of the reviews. Of course, the loss of a minor non-randomized included study that follows the systematic review’s conclusions would not be as problematic as losing a major included randomized controlled trial with contradictory results. However, the wide range of scope, topic, and criteria between systematic reviews and their related review types make it very hard to answer this question.

We found that two databases previously not recommended as essential for systematic review searching, Web of Science and Google Scholar, were key to improving recall in the reviews we investigated. Because this is a novel finding, we cannot conclude whether it is due to our dataset or to a generalizable principle. It is likely that topical differences in systematic reviews may impact whether databases such as Web of Science and Google Scholar add value to the review. One explanation for our finding may be that if the research question is very specific, the topic of research might not always be mentioned in the title and/or abstract. In that case, Google Scholar might add value by searching the full text of articles. If the research question is more interdisciplinary, a broader science database such as Web of Science is likely to add value. The topics of the reviews studied here may simply have fallen into those categories, though the diversity of the included reviews may point to a more universal applicability.

Although we searched PubMed as supplied by publisher separately from MEDLINE in Ovid, we combined the included references of these databases into one measurement in our analysis. Until 2016, the most complete MEDLINE selection in Ovid still lacked the electronic publications that were already available in PubMed. These could be retrieved by searching PubMed with the subset as supplied by publisher. Since the introduction of the more complete MEDLINE collection Epub Ahead of Print , In-Process & Other Non-Indexed Citations , and Ovid MEDLINE® , the need to separately search PubMed as supplied by publisher has disappeared. According to our data, PubMed’s “as supplied by publisher” subset retrieved 12 unique included references, and it was the most important addition in terms of relevant references to the four major databases. It is therefore important to search MEDLINE including the “Epub Ahead of Print, In-Process, and Other Non-Indexed Citations” references.

These results may not be generalizable to other studies for other reasons. The skills and experience of the searcher are one of the most important aspects in the effectiveness of systematic review search strategies [ 23 – 25 ]. The searcher in the case of all 58 systematic reviews is an experienced biomedical information specialist. Though we suspect that searchers who are not information specialists or librarians would have a higher possibility of less well-constructed searches and searches with lower recall, even highly trained searchers differ in their approaches to searching. For this study, we searched to achieve as high a recall as possible, though our search strategies, like any other search strategy, still missed some relevant references because relevant terms had not been used in the search. We are not implying that a combined search of the four recommended databases will never result in relevant references being missed, rather that failure to search any one of these four databases will likely lead to relevant references being missed. Our experience in this study shows that additional efforts, such as hand searching, reference checking, and contacting key players, should be made to retrieve extra possible includes.

Based on our calculations made by looking at random systematic reviews in PubMed, we estimate that 60% of these reviews are likely to have missed more than 5% of relevant references only because of the combinations of databases that were used. That is with the generous assumption that the searches in those databases had been designed sensitively enough. Even when taking into account that many searchers consider the use of Scopus as a replacement of Embase, plus taking into account the large overlap of Scopus and Web of Science, this estimate remains similar. Also, while the Scopus and Web of Science assumptions we made might be true for coverage, they are likely very different when looking at recall, as Scopus does not allow the use of the full features of a thesaurus. We see that reviewers rarely use Web of Science and especially Google Scholar in their searches, though they retrieve a great deal of unique references in our reviews. Systematic review searchers should consider using these databases if they are available to them, and if their institution lacks availability, they should ask other institutes to cooperate on their systematic review searches.

The major strength of our paper is that it is the first large-scale study we know of to assess database performance for systematic reviews using prospectively collected data. Prior research on database importance for systematic reviews has looked primarily at whether included references could have theoretically been found in a certain database, but most have been unable to ascertain whether the researchers actually found the articles in those databases [ 10 , 12 , 16 , 17 , 26 ]. Whether a reference is available in a database is important, but whether the article can be found in a precise search with reasonable recall is not only impacted by the database’s coverage. Our experience has shown us that it is also impacted by the ability of the searcher, the accuracy of indexing of the database, and the complexity of terminology in a particular field. Because these studies based on retrospective analysis of database coverage do not account for the searchers’ abilities, the actual findings from the searches performed, and the indexing for particular articles, their conclusions lack immediate translatability into practice. This research goes beyond retrospectively assessed coverage to investigate real search performance in databases. Many of the articles reporting on previous research concluded that one database was able to retrieve most included references. Halladay et al. [ 10 ] and van Enst et al. [ 16 ] concluded that databases other than MEDLINE/PubMed did not change the outcomes of the review, while Rice et al. [ 17 ] found the added value of other databases only for newer, non-indexed references. In addition, Michaleff et al. [ 26 ] found that Cochrane CENTRAL included 95% of all RCTs included in the reviews investigated. Our conclusion that Web of Science and Google Scholar are needed for completeness has not been shared by previous research. Most of the previous studies did not include these two databases in their research.

We recommend that, regardless of their topic, searches for biomedical systematic reviews should combine Embase, MEDLINE (including electronic publications ahead of print), Web of Science (Core Collection), and Google Scholar (the 200 first relevant references) at minimum. Special topics databases such as CINAHL and PsycINFO should be added if the topic of the review directly touches the primary focus of a specialized subject database, like CINAHL for focus on nursing and allied health or PsycINFO for behavioral sciences and mental health. For reviews where RCTs are the desired study design, Cochrane CENTRAL may be similarly useful. Ignoring one or more of the databases that we identified as the four key databases will result in more precise searches with a lower number of results, but the researchers should decide whether that is worth the >increased probability of losing relevant references. This study also highlights once more that searching databases alone is, nevertheless, not enough to retrieve all relevant references.

Future research should continue to investigate recall of actual searches beyond coverage of databases and should consider focusing on the most optimal database combinations, not on single databases.

Additional files

Reviews included in the research . References to the systematic reviews published by Erasmus MC authors that were included in the research. (DOCX 19 kb)

Acknowledgements

Not applicable

Melissa Rethlefsen receives funding in part from the National Center for Advancing Translational Sciences of the National Institutes of Health under Award Number UL1TR001067. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Availability of data and materials

Authors’ contributions.

WB, JK, and OF designed the study. WB designed the searches used in this study and gathered the data. WB and ML analyzed the data. WB drafted the first manuscript, which was revised critically by the other authors. All authors have approved the final manuscript.

Ethics approval and consent to participate

Consent for publication, competing interests.

WB has received travel allowance from Embase for giving a presentation at a conference. The other authors declare no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Contributor Information

Wichor M. Bramer, Phone: +31 10 7043785, Email: [email protected] .

Melissa L. Rethlefsen, Email: moc.liamg@nesfelhterlm .

Jos Kleijnen, Email: moc.sweiver-citametsys@soj .

Oscar H. Franco, Email: [email protected] .

Doing the literature review: Scoping search

• The literature review: why?
• Types of literature review
• Selecting databases

Scoping search

• Using a database thesaurus
• Advanced search in a database
• Citation information
• Using a reference manager
• Reporting your search strategy
• Writing & structuring
• When to stop

Creating an overview of keywords used

Example of an author keyword map

Activity: Creating a keyword map with VOSviewer

In the handout VOSviewer Keywords Map   (PDF)   we explain how you can create such a keyword map for your own research topic. We use Scopus as the example database, but you can also use data from Web of Science and PubMed and from other databases (as long as you can export data in RIS-format and the file contains keyword-data).

Email the Information skills team

• << Previous: Selecting databases
• Next: Using a database thesaurus >>
• Last Updated: Jun 5, 2024 9:34 AM
• URL: https://libguides.eur.nl/informationskillslitreview

Methodology

• Open access
• Published: 28 April 2022

An intelligent literature review: adopting inductive approach to define machine learning applications in the clinical domain

• Renu Sabharwal   ORCID: orcid.org/0000-0001-9728-8001 1 &
• Shah J. Miah 1  

Journal of Big Data volume  9 , Article number:  53 ( 2022 ) Cite this article

6832 Accesses

9 Citations

Metrics details

Big data analytics utilizes different techniques to transform large volumes of big datasets. The analytics techniques utilize various computational methods such as Machine Learning (ML) for converting raw data into valuable insights. The ML assists individuals in performing work activities intelligently, which empowers decision-makers. Since academics and industry practitioners have growing interests in ML, various existing review studies have explored different applications of ML for enhancing knowledge about specific problem domains. However, in most of the cases existing studies suffer from the limitations of employing a holistic, automated approach. While several researchers developed various techniques to automate the systematic literature review process, they also seemed to lack transparency and guidance for future researchers. This research aims to promote the utilization of intelligent literature reviews for researchers by introducing a step-by-step automated framework. We offer an intelligent literature review to obtain in-depth analytical insight of ML applications in the clinical domain to (a) develop the intelligent literature framework using traditional literature and Latent Dirichlet Allocation (LDA) topic modeling, (b) analyze research documents using traditional systematic literature review revealing ML applications, and (c) identify topics from documents using LDA topic modeling. We used a PRISMA framework for the review to harness samples sourced from four major databases (e.g., IEEE, PubMed, Scopus, and Google Scholar) published between 2016 and 2021 (September). The framework comprises two stages—(a) traditional systematic literature review consisting of three stages (planning, conducting, and reporting) and (b) LDA topic modeling that consists of three steps (pre-processing, topic modeling, and post-processing). The intelligent literature review framework transparently and reliably reviewed 305 sample documents.

Introduction

Organizations are continuously harnessing the power of various big data adopting different ML techniques. Captured insights from big data may create a greater impact to reshape their business operations and processes. As a vital technique, big data analytics methods are used to transform complicated and huge amounts of data, known as ‘Big Data, in order to uncover hidden patterns, new learning, untold facts or associations, anomalies, and other perceptions [ 41 ]. Big Data alludes to the enormous amount of data that a traditional database management system cannot handle. In most of the cases, traditional software functions would be inadequate to analyze or process them. Big data are characterized by the 5 V’s, which refers to volume, variety, velocity, veracity, and value [ 22 ]. ML is a vital approach to design useful big data analytics techniques, which is a rapidly growing sub-field in information sciences that deals with all these characteristics. ML employs numerous methods for machines to learn from past experiences (e.g., past datasets) reducing the extra burden of writing codes in traditional programming [ 7 , 26 ]. Clinical care enterprises face a huge challenge due to the increasing demand of big data processing to improve clinical care outcomes. For example, an electronic health record contains a huge amount of patient information, drug administration, imaging data using various modalities. The variety and quantity of the huge data provide in the clinical domain as an ideal topic to appraise the value of ML in research.

Existing ML approaches, such as Oala et al. [ 35 ] proposed an algorithmic framework that give a path towards the effective and reliable application of ML in the healthcare domain. In conjunction with their systematic review, our research offers a smart literature review that consolidates a traditional literature review followed the PRISMA framework guidelines and topic modeling using LDA, focusing on the clinical domain. Most of the existing literature focused on the healthcare domain [ 14 , 42 , 49 ] are more inclusive and of a broader scope with a requisite of medical activities, whereas our research is primarily focused is clinical, which assist in diagnosing and treating patients as well as includes clinical aspects of medicine.

Since clinical research has developed, the area has become increasingly attractive to clinical researchers, in particular for learning insights of ML applications in clinical practices . This is because of its practical pertinence to clinical patients, professionals, clinical application designers, and other specialists supported by the omnipresence of clinical disease management techniques. Although the advantage is presumed for the target audience, such as self-management abilities (self-efficacy and investment behavior) and physical or mental condition of life amid long-term ill patients, clinical care specialists (such as further developing independent direction and providing care support to patients), their clinical care have not been previously assessed and conceptualized as a well-defined and essential sub-field of health care research. It is important to portray similar studies utilizing different types of review approaches in the aspect of the utilization of ML/DL and its value. Table 1 represents some examples of existing studies with various points and review approaches in the domain.

Although the existing studies included in Table 1 give an understanding of designated aspects of ML/DL utilization in clinical care, they show a lack of focus on how key points addressed in existing ML/DL research are developing. Further to this, they indicate a clear need towards an understanding of multidisciplinary affiliations and profiles of ML/DL that could provide significant knowledge to new specialists or professionals in this space. For instance, Brnabic and Hess [ 8 ] recommended a direction for future research by stating that “ Future work should routinely employ ensemble methods incorporating various applications of machine learning algorithms” (p. 1).

ML tools have become the central focus of modern biomedical research, because of better admittance to large datasets, exponential processing power, and key algorithmic developments allowing ML models to handle increasingly challenging data [ 19 ]. Different ML approaches can analyze a huge amount of data, including difficult and abnormal patterns. Most studies have focused on ML and its impacts on clinical practices [ 2 , 9 , 10 , 24 , 26 , 34 , 43 ]. Fewer studies have examined the utilization of ML algorithms [ 11 , 20 , 45 , 48 ] for more holistic benefits for clinical researchers.

ML becomes an interdisciplinary science that integrates computer science, mathematics, and statistics. It is also a methodology that builds smart machines for artificial intelligence. Its applications comprise algorithms, an assortment of instructions to perform specific tasks, crafted to independently learn from data without human intercession. Over time, ML algorithms improve their prediction accuracy without a need for programming. Based on this, we offer an intelligent literature review using traditional literature review and Latent Dirichlet Allocation (LDA Footnote 1 ) topic modeling in order to meet knowledge demands in the clinical domain. Theoretical measures direct the current study results because previous literature provides a strong foundation for future IS researchers to investigate ML in the clinical sector. The main aim of this study is to develop an intelligent literature framework using traditional literature. For this purpose, we employed four digital databases -IEEE, Google Scholar, PubMed, and Scopus then performed LDA topic modeling, which may assist healthcare or clinical researchers in analyzing many documents intelligently with little effort and a small amount of time.

Traditional systematic literature is destined to be obsolete, time-consuming with restricted processing power, resulting in fewer sample documents investigated. Academic and practitioner-researchers are frequently required to discover, organize, and comprehend new and unexplored research areas. As a part of a traditional literature review that involves an enormous number of papers, the choice for a researcher is either to restrict the number of documents to review a priori or analyze the study using some other methods.

The proposed intelligent literature review approach consists of Part A and Part B, a combination of traditional systematic literature review and topic modeling that may assist future researchers in using appropriate technology, producing accurate results, and saving time. We present the framework below in Fig.  1 .

Proposed intelligent literature review framework

The traditional literature review identified 534,327 articles embraces Scopus (24,498), IEEE (2558), PubMed (11,271), and Google Scholar (496,000) articles, which went through three stages–Planning the review, conducting the review, and reporting the review and analyzed 305 articles, where we performed topic modeling using LDA.

We follow traditional systematic literature review methodologies [ 25 , 39 , 40 ] including a PRISMA framework [ 37 ]. We review four digital databases and deliberately develop three stages entailing planning, conducting, and reporting the review (Fig.  2 ).

Traditional literature review three stages

Planning the review

Research articles : the research articles are classified using some keywords mentioned below in Tables 2 , 3 .

Digital database : Four databases (IEEE, PubMed, Scopus, and Google Scholar) were used to collect details for reviewing research articles.

Review protocol development : We first used Scopus to search the information and found many studies regarding this review. We then searched PubMed, IEEE, and Google scholar for articles and extracted only relevant papers matching our keywords and review context based on their full-text availability.

Review protocol evaluation : To support the selection of research articles and inclusion and exclusion criteria, the quality of articles was explored and assessed to appraise their suitability and impartiality [ 44 ]. Only articles with keywords “machine learning” and “clinical” in document titles and abstracts were selected.

Conducting the review

The second step is conducting the review, which includes a description of Search Syntax and data synthesis.

Search syntax Table 4 details the syntax used to select research articles.

Data synthesis

We used a qualitative meta-synthesis technique to understand the methodology, algorithms, applications, qualities, results, and current research impediments. Qualitative meta-synthesis is a coherent approach for analyzing data across qualitative studies [ 4 ]. Our first search identified 534,327 papers, comprising Scopus (24,498), IEEE (2,558), PubMed (11,271), and Google Scholar (496,000) articles with the selected keywords. After subjecting this dataset to our inclusion and exclusion criteria, articles were reduced to Scopus (181), IEEE (62), PubMed (37), and Google Scholar (46) (Fig.  3 ).

PRISMA framework of traditional literature review

Reporting the review

This section displays the result of the traditional literature review.

Demonstration of findings

A search including linear literature and citation chaining was acted in digital databases, and the resulted papers were thoroughly analyzed to choose only the most pertinent articles, at last, 305 articles were included for the Part B review. Information of such articles were classified, organized, and demonstrated to show the finding.

Report the findings

The word cloud is displayed on the selected 305 research articles which give an overview of the frequency of the word within those 305 research articles. The chosen articles are moved to the next step to perform the conversion of PDF files to text documents for performing LDA topic modeling (Fig. 4 ).

Word cloud on 305 articles

Conversion of pdf files to a text document

The Python coding is used to convert pdf files shared on GitHub https://github.com/MachineLearning-UON/Topic-modeling-using-LDA.git . The one text document is prepared with 305 research papers collected from a traditional literature review.

Topic modelling for intelligent literature review

Our intelligent literature review is developed using a combination of traditional literature review and topic modeling [ 22 ]. We use topic modeling—probability generating, a text-mining technique widely used in computer science for text mining and data recovery. Topic modeling is used in numerous papers to analyze [ 1 , 5 , 17 , 36 ] and use various ML algorithms [ 38 ] such as Latent Semantic Indexing (LSI), Latent Semantic Analysis (LSA), Latent Dirichlet Allocation (LDA), Non-Negative Matrix Factorization (NMF), Parallel Latent Dirichlet Allocation (PLDA), and Pachinko Allocation Model (PAM). We developed the LDA-based methodological framework so it would be most widely and easily used [ 13 , 17 , 21 ] as a very elementary [ 6 ] approach. LDA is an unsupervised and probabilistic ML algorithm that discovers topics by calculating patterns of word co-occurrence across many documents or corpus [ 16 ]. Each LDA topic is distributed across each document as a probability.

While there are numerous ways of conducting a systematic literature review, most strategies require a high expense of time and prior knowledge of the area in advance. This study examined the expense of various text categorization strategies, where the assumptions and cost of the strategy are analyzed [ 5 ]. Interestingly, except manually reading the articles and topic modeling, all the strategies require prior knowledge of the articles' categories and high pre-examination costs. However, topic modeling can be automated, alternate the utilization of researchers' time, demonstrating a perfect match for the utilization of topic modeling as a part of an Intelligent literature review. Topic modeling has been used in a few papers to categorize research papers presented in Table 5 .

The articles/papers in the above table analyzed are speeches, web documents, web posts, press releases, and newspapers. However, none of those have developed the framework to perform traditional literature reviews from digital databases then use topic modeling to save time. However, this research points out the utilization of LDA in academics and explores four parameters—text pre-processing, model parameters selection, reliability, and validity [ 5 ]. Topic modeling identifies patterns of the repetitive word across a corpus of documents. Patterns of word co-occurrence are conceived as hidden ‘topics’ available in the corpus. First, documents must be modified to be machine-readable, with only their most informative features used for topic modeling. We modify documents in a three-stage process entailing pre-processing, topic modeling, and post-processing, as defined in Fig.  1 earlier.

The utilization of topic modeling presents an opportunity for researchers to use advanced technology for the literature review process. Topic modeling has been utilized online and requires many statistical skills, which not all researchers have. Therefore, we have shared the codes in GitHub with the default parameter for future researchers.

Pre-processing

Székely and Brocke [ 46 ] explained that pre-processing is a seven-step process which explored below and mentioned in Fig.  1 as part B:

Load data—The text data file is imported using the python command.

Optical character recognition—using word cloud, characters are recognized.

Filtering non-English words—non-English words are removed.

Document tokenization—Split the text into sentences and the sentences into words. Lowercase the words and remove punctuation.

Text cleaning—the text has been cleaned using portstemmer.

Word lemmatization—words in the third person are changed to the first person, and past and future verb tenses are changed into the present.

Stop word removal—All stop words are removed.

Topic modelling using LDA

Several research articles have been selected to run LDA topic modeling, explained in Table 5 . LDA model results present the coherence score for all the selected topics and a list of the most frequently used words for each.

Post-processing

The goal of the post-processing stage is to identify and label topics and topics relevant for use in the literature review. The result of the LDA model is presented as a list of topics and probabilities of each document (paper). The list is utilized to assign a paper to a topic by arranging the list by the highest probability for each paper for each topic. All the topics contain documents that are like each other. To reduce the risk of error in topic identification, a combination of inspecting the most frequent words for each topic and a paper view is used. After the topic review, it will present in the literature review.

Following the intelligent literature review, results of the LDA model should be approved or validated by statistical, semantic, or predictive means. Statistical validation defines the mutual information tests of result fit to model assumptions; semantics validation requires hand-coding to decide if the importance of specific words varies significantly and as expected with tasks to different topics which is used in the current study to validate LDA model result; and predictive validation refers to checking if events that ought to have expanded the prevalence of particular topic if out interpretations are right, did so [ 6 , 21 ].

LDA defines that each word in each document comes from a topic, and the topic is selected from a set of keywords. So we have two matrices:

ϴtd = P(t|d) which is the probability distribution of topics in documents

Фwt = P(w|t), which is the probability distribution of words in topics

And, we can say that the probability of a word given document, i.e., P(w|d), is equal to:

where T is the total number of topics; likewise, let’s assume there are W keywords for all the documents.

If we assume conditional independence, we can say that

And hence P(w|d) is equal to

that is the dot product of ϴtd and Фwt for each topic t.

Our systematic literature review identified 305 research papers after performing a traditional literature review. After executing LDA topic modeling, only 115 articles show the relevancy with our topic "machine learning application in clinical domain'. The following stages present LDA topic modeling process.

The 305 research papers were stacked into a Python environment then converted into a single text file. The seven steps have been carried out, described earlier in Pre-processing .

• Topic modeling

The two main parameters of the LDA topic model are the dictionary (id2word)-dictionary and the corpus—doc_term_matrix. The LDA model is created by running the command:

# Creating the object for LDA model using gensim library

LDA = gensim.models.ldamodel.LdaModel

# Build LDA model

lda_model = LDA(corpus=doc_term_matrix, id2word = dictionary, num_topics=20, random_state=100,

chunksize = 1000, passes=50,iterations=100)

In this model, ‘num_topics’ = 20, ‘chunksize’ is the number of documents used in each training chunk, and ‘passes’ is the total number of training passes.

Firstly, the LDA model is built with 20 topics; each topic is represented by a combination of 20 keywords, with each keyword contributing a certain weight to a topic. Topics are viewed and interpreted in the LDA model, such as Topic 0, represented as below:

(0, '0.005*"analysis" + 0.005*"study" + 0.005*"models" + 0.004*"prediction" + 0.003*"disease" + 0.003*"performance" + 0.003*"different" + 0.003*"results" + 0.003*"patient" + 0.002*"feature" + 0.002*"system" + 0.002*"accuracy" + 0.002*"diagnosis" + 0.002*"classification" + 0.002*"studies" + 0.002*"medicine" + 0.002*"value" + 0.002*"approach" + 0.002*"variables" + 0.002*"review"'),

Our approach to finding the ideal number of topics is to construct LDA models with different numbers of topics as K and select the model with the highest coherence value. Selecting the ‘K' value that denotes the end of the rapid growth of topic coherence ordinarily offers significant and interpretable topics. Picking a considerably higher value can provide more granular sub-topics if the ‘K’ selection is too large, which can cause the repetition of keywords in multiple topics.

Model perplexity and topic coherence values are − 8.855378536321144 and 0.3724024189689453, respectively. To measure the efficiency of the LDA model is lower the perplexity, the better the model is. Topics and associated keywords were then examined in an interactive chart using the pyLDAvis package, which presents the topics are 20 and most salient terms in those 20 topics, but these 20 topics overlap each other as shown in Fig.  5 , which means the keywords are repeated in these 20 topics and topics are overlapped, which means so decided to use num_topics = 9 and presented PyLDAvis Figure below. Each bubble on the left-hand side plot represents a topic. The bigger the bubble is, the more predominant that topic is. A decent topic will have a genuinely big, non-overlapping bubble dispersed throughout the graph instead of grouped in one quadrant. A topic model with many topics will typically have many overlaps, small-sized bubbles clustered in one locale of the graph, as shown in Fig.  6 .

PyLDAvis graph with 20 topics in the clinical domain

PyLDAvis graph with nine vital topics in the clinical domain

Each bubble addresses a generated topic. The larger the bubble, the higher percentage of the number of keywords in the corpus is about that topic which can be seen on the GitHub file. Blue bars address the general occurrence of each word in the corpus. If no topic is selected, the blue bars of the most frequently used words are displayed, as depicted in Fig.  6 .

The further the bubbles are away from each other, the more various they are. For example, we can tell that topic 1 is about patient information and studies utilized deep learning to analyze the disease, which can be seen in GitHub file codes ( https://github.com/MachineLearning-UON/Topic-modeling-using-LDA.git ) and presented in Fig.  7 .

PyLDAvis graph with topic 1

Red bars give the assessed number of times a given topic produced a given term. As you can see from Fig.  7 , there are around 4000 of the word 'analysis', and this term is utilized 1000 times inside topic 1. The word with the longest red bar is the most used by the keywords having a place with that topic.

A good topic model will have big and non-overlapping bubbles dispersed throughout the chart. As we can see from Fig.  6 , the bubbles are clustered within one place. One of the practical applications of topic modeling is discovering the topic in a provided document. We find out the topic number with the highest percentage contribution in that document, as shown in Fig.  8 .

Dominant topics with topic percentage contribution

The next stage is to process the discoveries and find a satisfactory depiction of the topics. A combination of evaluating the most continuous words utilized to distinguish the topic. For example, the most frequent words for the papers in topic 2 are "study" and "analysis", which indicate frequent words for ML usage in the clinical domain.

The topic name is displayed with the topic number from 0 to 8, which represents in the Table 6 , which includes the Topic number and Topic words.

The result represents the percentage of the topics in all documents, which presents that topic 0 and topic 6 have the highest percentage and used in 58 and 57 documents, respectively, with 115 papers. The result of this research was an overview of the exploration areas inside the paper corpus, addressed by 9 topics.

This paper presented a new methodology that is uncommon in scholarly publications. The methodology utilizes ML to investigate sample articles/papers to distinguish research directions. Even though the structure of the ML-based methodology has its restrictions, the outcomes and its ease of use leave a promising future for topic modeling-based systematic literature reviews.

The principal benefit of the methodological framework is that it gives information about an enormous number of papers, with little effort on the researcher's part, before time-exorbitant manual work is to be finished. By utilizing the framework, it is conceivable to rapidly explore a wide range of paper corpora and assess where the researcher's time and concentration should be spent. This is particularly significant for a junior researcher with minimal earlier information on a research field. If default boundaries and cleaning settings can be found for the steps in the framework, a completely programmed gathering of papers could be empowered, where limited works have been introduced to accomplish an overview of research directions.

From a literature review viewpoint, the advantage of utilizing the proposed framework is that the inclusion and exclusion selection of papers for a literature review will be delayed to a later stage where more information is given, resulting in a more educated dynamic interaction. The framework empowers reproducibility, as every step can be reproduced in the systematic review process that ultimately empowers with transparency. The whole process has been demonstrated as a case concept on GitHub by future researchers.

The study has introduced an intelligent literature review framework that uses ML to analyze existing research documents or articles. We demonstrate how topic modeling can assist literature review by reducing the manual screening of huge quantities of literature for more efficient use of researcher time. An LDA algorithm provides default parameters and data cleaning steps, reducing the effort required to review literature. An additional advantage of our framework is that the intelligent literature review offers accurate results with little time, and it comprises traditional ways to analyze literature and LDA topic modeling.

This framework is constructed in a step-by-step manner. Researchers can use it efficiently because it requires less technical knowledge than other ML algorithms. There is no restriction on the quantity of the research papers it can measure. This research extends knowledge to similar studies in this field [ 12 , 22 , 23 , 26 , 30 , 46 ] which present topic modeling. The study acknowledges the inspiring concept of smart literature defined by Asmussen and Møller [ 3 ]. The researchers previously provided a brief description of how LDA is utilized in topic modeling. Our research followed the basic idea but enhanced its significance to broaden its scale and focus on a specific domain such as the clinical domain to produce insights from existing research articles. For instance, Székely and Vom [ 46 ] utilized natural language processing to analyze 9514 sustainability reports published between 1999 and 2015. They identified 42 topics but did not develop any framework for future researchers. This was considered a significant gap in the research. Similarly, Kushwaha et al. [ 22 ] used a network analysis approach to analyze 10-year papers without providing any clear transparent outcome (e.g., how the research step-by-step produces an outcome). Likewise, Asmussen and Møller [ 3 ] developed a smart literature review framework that was limited to analyzing 650 sample articles through a single method. However, in our research, we developed an intelligent literature review that combines traditional and LDA topic modeling, so that future researchers can get assistance to gain effective knowledge regarding literature review when it becomes a state-of-the-art in research domains.

Our research developed a more effective intelligent framework, which combines traditional literature review and topic modeling using LDA, which provides more accurate and transparent results. The results are shared via public access on GitHub using this link https://github.com/MachineLearning-UON/Topic-modeling-using-LDA.git .

This paper focused on creating a methodological framework to empower researchers, diminishing the requirement for manually scanning documents and assigning the possibility to examine practically limitless. It would assist in capturing insights of an enormous number of papers quicker, more transparently, with more reliability. The proposed framework utilizes the LDA's topic model, which gathers related documents into topics.

A framework employed topic modeling for rapidly and reliably investigating a limitless number of papers, reducing their need to read individually, is developed. Topic modeling using the LDA algorithm can assist future researchers as they often need an outline of various research fields with minimal pre-existing knowledge. The proposed framework can empower researchers to review more papers in less time with more accuracy. Our intelligent literature review framework includes a holistic literature review process (conducting, planning, and reporting the review) and an LDA topic modeling (pre-processing, topic modeling, and post-processing stages), which conclude the results of 115 research articles are relevant to the search.

The automation of topic modeling with default parameters could also be explored to benefit non-technical researchers to explore topics or related keywords in any problem domain. For future directions, the principal points should be addressed. Future researchers in other research fields should apply the proposed framework to acquire information about the practical usage and gain ideas for additional advancement of the framework. Furthermore, research in how to consequently specify model parameters could extraordinarily enhance the ease of use for the utilization of topic modeling for non-specialized researchers, as the determination of model parameters enormously affects the outcome of the framework.

Future research may be utilized more ML analytics tools as complete solution artifacts to analyze different forms of big data. This could be adopting design science research methodologies for benefiting design researchers who are interested in building ML-based artifacts [ 15 , 28 , 29 , 31 , 32 , 33 ].

Availability of data and materials

Data will be supplied upon request.

LDA is a probabilistic method for topic modeling in text analysis, providing both a predictive and latent topic representation.

Abbreviations

The Institute of Electrical and Electronics Engineers

• Machine learning
• Latent Dirichlet Allocation

Organizational Capacity

Latent Semantic Indexing

Latent Semantic Analysis

Non-Negative Matrix Factorization

Parallel Latent Dirichlet Allocation

Pachinko Allocation Model

Abuhay TM, Kovalchuk SV, Bochenina K, Mbogo G-K, Visheratin AA, Kampis G, et al. Analysis of publication activity of computational science society in 2001–2017 using topic modelling and graph theory. J Comput Sci. 2018;26:193–204.

Article   Google Scholar  

Adlung L, Cohen Y, Mor U, Elinav E. Machine learning in clinical decision making. Med. 2021;2(6):642–65.

Asmussen CB, Møller C. Smart literature review: a practical topic modeling approach to exploratory literature review. J Big Data. 2019;6(1):1–18.

Beck CT. A meta-synthesis of qualitative research. MCN Am J Mater Child Nurs. 2002;27(4):214–21.

Behera RK, Bala PK, Dhir A. The emerging role of cognitive computing in healthcare: a systematic literature review. Int J Med Informatics. 2019;129:154–66.

Blei DM. Probabilistic topic models. Commun ACM. 2012;55(4):77–84.

Blei DM, Ng AY, Jordan MI. Latent Dirichlet allocation. J Mach Learn Res. 2003;3:993–1022.

MATH   Google Scholar  

Brnabic A, Hess LM. Systematic literature review of machine learning methods used in the analysis of real-world data for patient-provider decision making. BMC Med Inform Decis Mak. 2021;21(1):1–19.

Cabitza F, Locoro A, Banfi G. Machine learning in orthopedics: a literature review. Front Bioeng Biotechnol. 2018;6:75.

Chang C-H, Lin C-H, Lane H-Y. Machine learning and novel biomarkers for the diagnosis of Alzheimer’s disease. Int J Mol Sci. 2021;22(5):2761.

Connor KL, O’Sullivan ED, Marson LP, Wigmore SJ, Harrison EM. The future role of machine learning in clinical transplantation. Transplantation. 2021;105(4):723–35.

Dias R, Torkamani A. Artificial intelligence in clinical and genomic diagnostics. Genome Med. 2019;11(1):1–12.

DiMaggio P, Nag M, Blei D. Exploiting affinities between topic modeling and the sociological perspective on culture: application to newspaper coverage of US government arts funding. Poetics. 2013;41(6):570–606.

Forest P-G, Martin D. Fit for Purpose: Findings and recommendations of the external review of the Pan-Canadian Health Organizations: Summary Report: Health Canada Ottawa, ON; 2018.

Genemo H, Miah SJ, McAndrew A. A design science research methodology for developing a computer-aided assessment approach using method marking concept. Educ Inf Technol. 2016;21(6):1769–84.

Greene D, Cross JP. Exploring the political agenda of the european parliament using a dynamic topic modeling approach. Polit Anal. 2017;25(1):77–94.

Grimmer J. A Bayesian hierarchical topic model for political texts: measuring expressed agendas in Senate press releases. Polit Anal. 2010;18(1):1–35.

Grimmer J, Stewart BM. Text as data: the promise and pitfalls of automatic content analysis methods for political texts. Polit Anal. 2013;21(3):267–97.

Hassan N, Slight R, Weiand D, Vellinga A, Morgan G, Aboushareb F, et al. Preventing sepsis; how can artificial intelligence inform the clinical decision-making process? A systematic review. Int J Med Inform. 2021;150:104457.

Hirt R, Koehl NJ, Satzger G, editors. An end-to-end process model for supervised machine learning classification: from problem to deployment in information systems. Designing the Digital Transformation: DESRIST 2017 Research in Progress Proceedings of the 12th International Conference on Design Science Research in Information Systems and Technology Karlsruhe, Germany 30 May-1 Jun; 2017: Karlsruher Institut für Technologie (KIT).

Koltsova O, Koltcov S. Mapping the public agenda with topic modeling: the case of the Russian live journal. Policy Internet. 2013;5(2):207–27.

Kushwaha AK, Kar AK, Dwivedi YK. Applications of big data in emerging management disciplines: a literature review using text mining. Int J Inf Manag Data Insights. 2021;1(2):100017.

Google Scholar  

Li S, Wang H. Traditional literature review and research synthesis. The Palgrave handbook of applied linguistics research methodology. 2018:123–44.

Magrabi F, Ammenwerth E, McNair JB, De Keizer NF, Hyppönen H, Nykänen P, et al. Artificial intelligence in clinical decision support: challenges for evaluating AI and practical implications. Yearb Med Inform. 2019;28(01):128–34.

Maier D, Waldherr A, Miltner P, Wiedemann G, Niekler A, Keinert A, et al. Applying LDA topic modeling in communication research: toward a valid and reliable methodology. Commun Methods Meas. 2018;12(2–3):93–118.

Mårtensson G, Ferreira D, Granberg T, Cavallin L, Oppedal K, Padovani A, et al. The reliability of a deep learning model in clinical out-of-distribution MRI data: a multicohort study. Med Image Anal. 2020;66:101714.

Mendo IR, Marques G, de la Torre DI, López-Coronado M, Martín-Rodríguez F. Machine learning in medical emergencies: a systematic review and analysis. J Med Syst. 2021;45(10):1–16.

Miah SJ. An ontology based design environment for rural business decision support. Nathan: Griffith University Nathan; 2008.

Miah SJ, A new semantic knowledge sharing approach for e-government systems. 4th IEEE International Conference on Digital Ecosystems and Technologies; 2010: IEEE.

Miah SJ, Camilleri E, Vu HQ. Big Data in healthcare research: a survey study. J Comput Inf Syst. 2021. https://doi.org/10.1080/08874417.2020.1858727 .

Miah SJ, Gammack J, Kerr D, Ontology development for context-sensitive decision support. Third International Conference on Semantics, Knowledge and Grid (SKG 2007); 2007: IEEE.

Miah SJ, Gammack JG. Ensemble artifact design for context sensitive decision support. Australas J Inf Syst. 2014. https://doi.org/10.3127/ajis.v18i2.898 .

Miah SJ, Gammack JG, McKay J. A metadesign theory for tailorable decision support. J Assoc Inf Syst. 2019;20(5):4.

Mimno D, Blei D, editors. Bayesian checking for topic models. Proceedings of the 2011 conference on empirical methods in natural language processing; 2011.

Oala L, Murchison AG, Balachandran P, Choudhary S, Fehr J, Leite AW, et al. Machine learning for health: algorithm auditing & quality control. J Med Syst. 2021;45(12):1–8.

Ouhbi S, Idri A, Fernández-Alemán JL, Toval A. Requirements engineering education: a systematic mapping study. Requir Eng. 2015;20(2):119–38.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2020;372:n71.

Quinn KM, Monroe BL, Colaresi M, Crespin MH, Radev DR. How to analyze political attention with minimal assumptions and costs. Am J Polit Sci. 2010;54(1):209–28.

Rowley J, Slack F. Conducting a literature review. Management research news. 2004.

Rozas LW, Klein WC. The value and purpose of the traditional qualitative literature review. J Evid Based Soc Work. 2010;7(5):387–99.

Sabharwal R, Miah SJ. A new theoretical understanding of big data analytics capabilities in organizations: a thematic analysis. J Big Data. 2021;8(1):1–17.

Salazar-Reyna R, Gonzalez-Aleu F, Granda-Gutierrez EM, Diaz-Ramirez J, Garza-Reyes JA, Kumar A. A systematic literature review of data science, data analytics and machine learning applied to healthcare engineering systems. Management Decision. 2020.

Shah P, Kendall F, Khozin S, Goosen R, Hu J, Laramie J, et al. Artificial intelligence and machine learning in clinical development: a translational perspective. NPJ Digit Med. 2019;2(1):1–5.

Sone D, Beheshti I. Clinical application of machine learning models for brain imaging in epilepsy: a review. Front Neurosci. 2021;15:761.

Spasic I, Nenadic G. Clinical text data in machine learning: systematic review. JMIR Med Inform. 2020;8(3):e17984.

Székely N, Vom Brocke J. What can we learn from corporate sustainability reporting? Deriving propositions for research and practice from over 9,500 corporate sustainability reports published between 1999 and 2015 using topic modelling technique. PLoS ONE. 2017;12(4):e0174807.

Verma D, Bach K, Mork PJ, editors. Application of machine learning methods on patient reported outcome measurements for predicting outcomes: a literature review. Informatics; 2021: Multidisciplinary Digital Publishing Institute.

Weng W-H. Machine learning for clinical predictive analytics. Leveraging data science for global health. Cham: Springer; 2020. p. 199–217.

Book   Google Scholar  

Yin Z, Sulieman LM, Malin BA. A systematic literature review of machine learning in online personal health data. J Am Med Inform Assoc. 2019;26(6):561–76.

Download references

Acknowledgements

Not applicable.

Author information

Authors and affiliations.

Newcastle Business School, The University of Newcastle, Newcastle, NSW, Australia

Renu Sabharwal & Shah J. Miah

You can also search for this author in PubMed   Google Scholar

Contributions

The first author conducted the research, while the second author has ensured quality standards and rewritten the entire findings linking to underlying theories. Both authors read and approved the final manuscript.

Corresponding author

Correspondence to Renu Sabharwal .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests, additional information, publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Sabharwal, R., Miah, S.J. An intelligent literature review: adopting inductive approach to define machine learning applications in the clinical domain. J Big Data 9 , 53 (2022). https://doi.org/10.1186/s40537-022-00605-3

Download citation

Received : 18 November 2021

Accepted : 06 April 2022

Published : 28 April 2022

DOI : https://doi.org/10.1186/s40537-022-00605-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Clinical research
• Systematic literature review

• Open access
• Published: 19 April 2023

AI literacy in K-12: a systematic literature review

• Lorena Casal-Otero   ORCID: orcid.org/0000-0002-0906-4321 1 ,
• Alejandro Catala   ORCID: orcid.org/0000-0002-3677-672X 2 , 3 ,
• Carmen Fernández-Morante   ORCID: orcid.org/0000-0003-4398-3361 1 ,
• Maria Taboada   ORCID: orcid.org/0000-0002-2353-596X 2 ,
• Beatriz Cebreiro   ORCID: orcid.org/0000-0003-2064-915X 1 &
• Senén Barro   ORCID: orcid.org/0000-0001-6035-540X 3  

International Journal of STEM Education volume  10 , Article number:  29 ( 2023 ) Cite this article

38k Accesses

45 Citations

28 Altmetric

Metrics details

The successful irruption of AI-based technology in our daily lives has led to a growing educational, social, and political interest in training citizens in AI. Education systems now need to train students at the K-12 level to live in a society where they must interact with AI. Thus, AI literacy is a pedagogical and cognitive challenge at the K-12 level. This study aimed to understand how AI is being integrated into K-12 education worldwide. We conducted a search process following the systematic literature review method using Scopus. 179 documents were reviewed, and two broad groups of AI literacy approaches were identified, namely learning experience and theoretical perspective. The first group covered experiences in learning technical, conceptual and applied skills in a particular domain of interest. The second group revealed that significant efforts are being made to design models that frame AI literacy proposals. There were hardly any experiences that assessed whether students understood AI concepts after the learning experience. Little attention has been paid to the undesirable consequences of an indiscriminate and insufficiently thought-out application of AI. A competency framework is required to guide the didactic proposals designed by educational institutions and define a curriculum reflecting the sequence and academic continuity, which should be modular, personalized and adjusted to the conditions of the schools. Finally, AI literacy can be leveraged to enhance the learning of disciplinary core subjects by integrating AI into the teaching process of those subjects, provided the curriculum is co-designed with teachers.

Introduction

In recent years, the convergence of huge computing power, massive amounts of data and improved machine learning algorithms have led to remarkable advances in Artificial Intelligence (AI) based technologies, which are set to be the most socially and economically disruptive technologies ever developed (Russell, 2021 ). The irruption of AI-based technology in our daily lives (e.g., robot vacuum cleaners, real-time location and search systems, virtual assistants, etc.) has generated a growing social and political interest in educating citizens about AI. The scientific community has also begun to engage in this education after detecting a significant gap in the understanding of AI, based on comments and fears expressed by citizens about this technology (West & Allen, 2018 ). Therefore, integrating AI into curricula is necessary to train citizens who must increasingly live and act in a world with a significant presence of AI.

It is worth noting that AI education addresses not only the learning of the scientific and technological foundations of AI, but also the knowledge and critical reflection on how a trustworthy AI should be developed and the consequences of not doing so. Hence, it is crucial to incorporate AI teaching from the earliest stages of education (Heintz, 2021 ). However, although some countries are making significant efforts to promote AI teaching in K-12 (Touretzky et al., 2019a ), this is being implemented through highly varied AI training experiences, such as data-driven design (Vartiainen et al., 2021 ), interactive data visualizations (Chittora & Baynes, 2020 ; von Wangenheim et al., 2021 ), virtual reality and robotics (Narahara & Kobayashi, 2018 ), games (Giannakos et al., 2020 ), or even based on combined workshop series (Lee et al., 2021 ). To date, there are very few methodological proposals on how to introduce the AI curriculum in K-12 education (Lee et al., 2020 ).

Since the development of a field requires prior research, we propose in this paper to identify and examine the way in which AI literacy is developing in K-12 around the world, to draw conclusions and guide teaching proposals for AI literacy in K-12. By highlighting and discussing the pros and cons of the different approaches and experiences in the literature, we aim to inspire new initiatives and guide the actors involved, from decisions-makers, for example in education policy, to teachers involved in their conception, design and implementation. We also hope to raise awareness of the importance of learning about AI from an early age, emphasizing the key aspects of this training and, hopefully, fueling the debate that needs to be fostered in our research community.

Integration of AI into the K12 curriculum

As a scientific-technological field, AI is just a few decades old. The name was coined in 1956, and since then different disciplines (such as computer science, mathematics, philosophy, neuroscience, or psychology) have contributed to its development from an interdisciplinary focus. AI is oriented to comprehend, model, and replicate human intelligence and cognitive processes into artificial systems. Currently, it covers a wide range of subfields such as machine learning, perception, natural language processing, knowledge representation and reasoning, computer vision, among many others (Russell & Norvig, 2021 ).

Starting in the 1970s, AI began to emerge in educational contexts through tools specifically designed to support learning, teaching, and the management of educational institutions. Since many jobs are now AI-related and will continue to increase in the coming years, some researchers believe that AI education should be considered as important as literacy in reading and writing (Kandlhofer et al., 2016 ). The highly interdisciplinary character is also another factor to consider. AI literacy can be defined as a set of skills that enable a solid understanding of AI through three priority axes: learning about AI, learning about how AI works, and learning for life with AI (Long & Magerko, 2020 ; Miao et al., 2021 ). The first axis focuses on understanding AI concepts and techniques to enable the recognition of which artifacts/platforms use AI and which do not. The second axis addresses the understanding of how AI works, to effectively interact with it. The third axis seeks to understand how AI can affect our lives, allowing us to critically evaluate its technology. Thus, AI literacy goes beyond the use of AI applications in education, such as Intelligent Tutoring Systems (ITS) (du Boulay, 2016 ).

The teaching of knowledge in AI has traditionally been carried out at the university level, focused on students who study disciplines closely related to computing and ICT in general. In recent years, AI learning has also started to be relevant both in university programs with diverse study backgrounds (Kong et al., 2021 ), as well as at the K-12 level (Kandlhofer & Steinbauer, 2021 ; Tedre et al., 2021 ). However, teaching AI at the K-12 level is not yet prevalent in formal settings and is considered challenging. Experts believe it is important to have some thought on what AI education should look like at the K-12 level so that future generations can become informed citizens who understand the technologies they interact with in their daily lives (Touretzky et al., 2019a ). Training children and teenagers will allow them to understand the basics of the science and technology that underpins AI, its possibilities, its limits and its potential social and economic impact. It also stimulates and better prepares them to pursue further studies related to AI or even to become creators and developers of AI themselves (Heintz, 2021 ).

Nowadays, research on AI teaching is still scarce (Chai et al., 2020a , 2020b ; Lee et al., 2020 ). The acquisition of knowledge in AI represents a great pedagogical challenge for both experts and teachers, and a cognitive challenge for students (Micheuz, 2020 ). Some countries are making significant efforts to promote AI education in K-12 (Touretzky et al., 2019b ), by developing relatively comprehensive curriculum guidelines (Yue et al., 2021 ). Through interviews with practitioners and policy makers from three different continents (America, Asia and Europe), some studies report on continuing works to introduce AI in K-12 education (He et al., 2020 ). Some other work focuses on examining and comparing AI curricula in several countries (Yue et al., 2021 ). In addition, there are a growing number of AI training experiences that explore pathways to optimize AI learning for K-12 students. However, most of them are somehow thematically limited as they do not adequately address key areas of AI, such as planning, knowledge representation and automated reasoning (Nisheva-Pavlova, 2021 ). Additionally, due to the rapid growth of AI, there is a need to understand how educators can best leverage AI techniques for the academic success of their students. Zhai et al. ( 2021 ) recommend that educators work together with AI experts to bridge the gap between technique and pedagogy.

Using a systematic review method, our research aims to present an overview of current approaches to understand how AI is taught worldwide. Several studies have conducted systematic reviews concerning applications of AI in education. Zhai et al. ( 2021 ) analyzed how AI was applied to the education domain from 2010 to 2020. Their review covers research on AI-based learning environments, from their construction to their application and integration in the educational environment. Guan et al. ( 2020 ) reviewed the main themes and trends in AI research in education over the past two decades. The authors found that research on the use of AI techniques to support teaching or learning has stood the test of time and that learner profiling models and learning analytics have proliferated in the last two decades. Ng et al. ( 2022 ) examined learner types, teaching tools and pedagogical approaches in AI teaching and learning, mainly in university computer science education. Chen et al. ( 2020 ) covered education enhanced by AI techniques aimed to back up teaching and learning. All these studies have focused on the main role that AI has played in educational applications over the last decades. However, in light of the recent need to consider how AI education should be approached at the K-12 level (Kandlhofer et al., 2016 ; Long & Magerko, 2020 ; Miao et al., 2021 ; Touretzky et al., 2019b ), it would be of great value to structure and characterize the different approaches used so far to develop AI literacy in K-12, as well as to identify research gaps to be explored. Recently, Yue et al. ( 2022 ) analyzed the main components of the pedagogical design in 32 empirical studies in K-12 AI education and Su et al. ( 2022 ) examined 14 learning experiences carried out in the Asian-Pacific region. These components included target audience, setting, duration, contents, pedagogical approaches to teaching, and assessment methods. Sanusi et al. ( 2022 ) reviewed research on teaching machine learning in K-12 from four perspectives: curriculum development, technology development, pedagogical development, and teacher training development. The findings of the study revealed that more studies are needed on how to integrate machine learning into subjects other than computer science. Crompton et al. ( 2022 ) carried out a systematic review on the use of AI as a supporting tool in K-12 teaching, which entails an interesting but narrower scope. Our study extends previous reviews on K-12 AI research by emphasizing how the current approaches are integrating AI literacy in K-12 education worldwide.

Research question

To begin the systematic review, a single research question (RQ) was formulated.

RQ: How are current approaches integrating AI literacy into K-12 education worldwide?

In essence, the RQ aims to investigate the characterization of the different approaches being employed to incorporate AI education in K-12. The following subsections in the methodology describe the search and the data collection process in such a way that an answer to the RQ can be provided in a replicable and objective fashion.

The research method chosen to conduct this research was the systematic literature review (SLR), following the guidelines posed by Kitchenham ( 2004 ). Accordingly, the following subsections summarize and document the key steps implemented in this research method.

Search process

We used Scopus to implement the search process. Scopus provides an integrated search facility to find relevant papers in its database based on curated metadata. It includes primary bibliographic sources published by Elsevier , Springer , ACM , and IEEE , among others. It provides a comprehensive coverage of journals and top-ranked conferences within fields of interest. We did not limit our search to specific journals or regular conference proceedings, as there is not yet a clearly established body of literature on the subject. All searches were performed based on title, keywords and abstract, and conducted between 21 October 2021 and 9 March 2023.

To decide the search string, we ran an initial search and found only a few papers focused on ‘literacy’ whereas the vast majority referred to the broader term ‘education’. Therefore, we decided to use both search terms (key issue 1 in Table 1 ). As some recent works combine the terms ‘Artificial Intelligence’ and ‘education’/’literacy’ into single terms such as ‘AI literacy’ or ‘AI education’, these were added to the search string (key issue 2 in Table 1 ). The educational stage was also included in the search string (key issue 3 in Table 1 ). As the search term ‘education’ also returns AI-based learning environments which are outside the scope of our review, we explicitly considered negated terms to leave out both computer-based learning and intelligent tutoring systems (key issue 4 in Table 1 ). A final decision was whether to use the term ‘Artificial Intelligence’ as a single umbrella term or to add narrower terms related to AI subfields (e.g., machine learning). After a preliminary inspection of a few relevant papers, we observed that such additional specific terms usually co-occur with the string ‘Artificial Intelligence’ in education, and they were therefore regarded as unnecessary. Thus, to capture the essence of our RQ and to build up the complete search string, we considered the search terms as shown in Table 1 . Eventually, this resulted in the following complete search string in Scopus:

TITLE-ABS-KEY ( ( ( ( literacy OR education) AND ( ( artificial AND intelligence))) OR ( "AI literacy" OR "AI education")) AND ( "primary school" OR "secondary school" OR k-12 OR "middle school"

OR "high school") AND NOT ( "computer-based learning") AND NOT ( "intelligent tutoring system")).

We included peer-reviewed papers published on topics related to literacy and education on AI at school. Then we excluded papers whose usage of AI was limited to 1) supporting computer-based learning only, with no focus on learning about AI; 2) supporting assessment/tutoring based on AI. We also excluded papers that targeted college students and those that were limited to K-12 programming/CS concepts as a prerequisite for learning about AI in the future. Following these inclusion and exclusion criteria, our search in Scopus returned an initial list of 750 documents. After we inspected the title, abstract, keywords and full-text screening, we obtained a final list of 179 documents.

Data collection extraction and synthesis strategy

Data collection extraction was performed, discussed, and coordinated through regular meetings. After inspecting and discussing 10% of the papers over multiple meetings, the authors agreed on the annotations presented in Table 2 . This process is important as it allowed us to build a data annotation scheme empirically emerging from the sampled papers. A copy of the papers was also kept for easy review in case of doubts or disagreements.

The data resulted in a spreadsheet with the metadata of the papers which passed the inclusion and exclusion criteria, and a document with the list of paper IDs together with the rest of annotations. Some Python scripts were used to further process metadata (e.g., counting participating countries, frequencies, etc.) and produce a more complete bibliographic report with histograms and overview counting. A more qualitative analysis was carried out to answer the research question based on paper reading and annotations.

The results were organized into two subsections. The first subsection is a bibliometric analysis of the reviewed studies, which is based on the metadata provided by Scopus. The second subsection provides a qualitative analysis of the studies, which is based on the extracted data annotations (see Table 2 ). Both analyses are complementary and together deliver a better understanding of the research articles retrieved.

Bibliometric analysis

Figure  1 shows that the annual scientific production has been modest. It gained traction in 2016 and increased sharply in 2020.

Annual scientific production: number of papers by year

Most of the contributions are conference publications (126 papers), while 52 are journal articles and one is a book chapter (Fig.  2 ).

Type of contributions: number of papers by type

Eighty out of 179 papers have at least a citation in Scopus. There are 13 papers that have 10 or more citations, and the most cited papers are Long and Magerko ( 2020 ) and Touretzky et al. ( 2019b ). Figure  3 summarizes the number of contributions by publishers, where Springer, IEEE and ACM stand out, followed by Elsevier. As for journals, there are no single journals concentrating the publication of articles. Nevertheless, there are some journals that are especially relevant and well-known by the community such as the International Journal of Child-Computer Interaction, Computers and Education: Artificial Intelligence, International Journal of Artificial Intelligence in Education, or IEEE Transactions on Education.

Frequency of publishers: number of papers by publisher

As for conferences, Fig.  4 summarizes the main conference events where papers are published. It includes flagship conferences Footnote 1 such as CHI and AAAI, top-ranked conferences such as HRI or SIGCSE and several noteworthy events (IDC, ICALT, ITiCSE, VL/HCC, to name a few). It is worth mentioning that AAAI is receiving contributions from recent years, which confirms the interest in the field in broadening the discussion to education. There are some additional publications associated with satellite AAAI events, such as workshops in CEUR-WS that deal with the issue under study. Although such contributions may sometimes be short, we decided to include them as they were relevant. For instance, the works published in (Herrero et al., 2020 ) and (Micheuz, 2020 ) include the German countrywide proposal for educating about AI, through a 6-module course focusing on explaining how AI works, the social discourse on AI and reducing existing misconceptions. On the other hand, Aguar et al. ( 2016 ) talk about teaching AI via an optional course which does not contribute to the final grades.

Main conference events: number of papers by conference

The analysis did not reveal particularly outstanding institutions (see Table 3 for a summary). Among the 299 affiliated institutions, we mostly find universities and research centers along with a few collaboration associations. The most active institutions are the Chinese University of Hong Kong, University of Eastern Finland and MIT, whose authors participated in a total of 19, 11 and 10 contributions, respectively.

Finally, the retrieved papers were co-authored by 643 different authors affiliated to research institutions from 42 countries. Figure  5 shows the histogram of participation by country. Of the 179 papers reviewed, most papers were written by authors affiliated with institutions in the same country. Only 32 papers involved authors from several countries. It is remarkable that in these cases at least one author is from the US, Hong Kong or China.

Country participation: number of papers by country

Literature analysis

By analyzing the data extracted, the papers were classified into two broad thematic categories according to the type of educational approach, namely, learning experience and theoretical perspective. The first category covers AI learning experiences focused on understanding a particular AI concept/technique or using specific tools/platforms to illustrate some AI concepts. The second category involves initiatives for the implementation of AI education for K-12 through the development of guidelines, curriculum design or teacher training, among others. Each main category was further subdivided into other subcategories to structure the field and characterize the different approaches used in developing AI literacy in K-12. Figure  6 shows all the identified categories and subcategories.

Taxonomy of approaches to AI learning in K-12

Learning experiences focused on understanding AI

This category covers learning experiences aimed at experimenting and becoming familiar with AI concepts and techniques. Based on the priority axes in AI literacy (Long & Magerko, 2020 ; Miao et al., 2021 ), we identified experiences aimed at acquiring basic AI knowledge to recognize artifacts using AI, learning how AI works, learning tools for AI and learning to live with AI.

Learning to recognize artifacts using AI

This subcategory refers to experiences that aim to understand AI concepts and techniques enabling the recognition of which artifacts/platforms use AI and which do not. Four studies were found in this subcategory. They are proposals aimed at helping young people to understand and demystify AI through different types of activities. These activities included conducting discussions after watching AI-related movies (Tims et al., 2012 ), carrying out computer-based simulations of human-like behaviors (Ho et al., 2019 ), experimenting as active users of social robots (Gonzalez et al., 2017 ) and programming AI-based conversational agents (Van Brummelen et al., 2021b ).

Learning about how AI works

This topic covers proposals designed to understand how AI works to make user interaction with AI easier and more effective. In this type of proposal, the focus is on methodology and learning is achieved through technology (Kim et al., 2023 ). The objective is to provide a better understanding of a particular aspect of reality in order to carry out a project or solve a problem (Lenoir & Hasni, 2016 ). The activities are supported by active experiences based on building and creating intelligent devices to achieve the understanding of AI concepts following the idea of Papert’s constructionism.

These experiences are mainly focused on teaching AI subfields such as ML or AI algorithms applied to robotics. Understanding the principles of ML, its workflows and its role in everyday practices to solve real-life problems has been the main objective of some studies (Burgsteiner et al., 2016 ; Evangelista et al., 2019 ; Lee et al., 2020 ; Sakulkueakulsuk et al., 2019 ; Vartiainen et al., 2021 ). In addition, there are also experiences focused on unplugged activities that simulate AI algorithms. For example, through classic games such as Mystery Hunt, one can learn how to traverse a graph without being able to see beyond the next path to be traversed (blind search) (Kandlhofer et al., 2016 ). Similarly, the AI4K12 initiative (Touretzky et al., 2019b ) collects a large set of activities and resources to simulate AI algorithms.

Learning tools for AI

This topic includes approaches that involve learning about AI support tools. The development of intelligent devices in the context of teaching AI requires specific programming languages or age-appropriate tools. Many of the tools currently available are focused on ML, with the aim of demystifying this learning in K-12 education (Wan et al., 2020 ). Some of them are integrated into block-based programming languages (such as Scratch or App Inventor) (Toivonen et al., 2020 ; von Wangenheim et al., 2021 ), enabling the deployment of the ML models built into games or mobile applications. Other approaches use data visualization and concepts of gamification to engage the student in the learning process (Reyes et al., 2020 ; Wan et al., 2020 ) or combine traditional programming activities with ML model building (Rodríguez-García et al., 2020 ).

This type of proposal aims to introduce AI through tools that enable the use of AI techniques. It is therefore an approach focused on learning by using AI-oriented tools. In this vein, different experiences have focused on learning programming tools for applications based on Machine Learning (Reyes et al., 2020 ; Toivonen et al., 2020 ; von Wangenheim et al., 2021 ; Wan et al., 2020 ), robotics (Chen et al., 2017 ; Eguchi, 2021 ; Eguchi & Okada, 2020 ; Holowka, 2020 ; Narahara & Kobayashi, 2018 ; Nurbekova et al., 2018 ; Verner et al., 2021 ), programming and the creation of applications (Chittora & Baynes, 2020 ; Giannakos et al., 2020 ; Kahn et al., 2018 ; Kelly et al., 2008 ; Park et al., 2021 ). Some of these tools use Scratch-based coding platforms to make AI-based programming attractive to children. In (Kahn et al., 2018 ), students play around with machine learning to classify self-captured images, using a block-based coding platform.

There are also experiences in which other types of environments are used to facilitate learning (Aung et al., 2022 ). In (Holowka, 2020 ; Verner et al., 2021 ), students can learn reinforcement learning through online simulation. In (Narahara & Kobayashi, 2018 ), a virtual environment helps students generate data in a playful setting, which is then used to train a neural network for the autonomous driving of a toy car-lab. In (Avanzato, 2009 ; Croxell et al., 2007 ), students experiment with different AI-based tasks through robotics-oriented competitions.

Learning for life with AI

This subcategory covers experiences aimed at understanding how AI can affect our lives thus providing us with skills to critically assess its technology. In (Vachovsky et al., 2016 ), technically rigorous AI concepts are contextualized through the impact on society. There are also experiences where students explore how a robot equipped with AI components can be used in society (Eguchi & Okada, 2018 ), program conversational agents (Van Brummelen et al., 2021b ), or learn to recognize credible but fake media products (video, photos), which have been generated using AI-based techniques ( 2021b ; Ali et al., 2021a ).

The ethical and philosophical implications of AI have also been addressed in some experiences ( 2021b ; Ali et al., 2021a ; Ellis et al., 2005 ), whereas others focus on training students to participate in present-day society and become critical consumers of AI (Alexandre et al., 2021 ; Cummings et al., 2021 ; Díaz et al., 2015 ; Kaspersen et al., 2022 ; Lee et al., 2021 ; Vartiainen et al., 2020 ).

Proposals for implementation of AI learning at the K-12 level

Some countries are making efforts to promote AI education in K-12. In the U.S., intense work is being carried out on the integration of AI in schools and among these schemes, AI4K12 stands out (Heintz, 2021 ). This scheme is especially interesting since it defines the national guidelines for future curricula, highlighting the essential collaborative work between developers, teachers and students (Touretzky et al., 2019a ). This idea of co-creation is also stressed in other schemes (Chiu, 2021 ). In the U.S. we can also mention the proposal made by the Massachusetts Institute of Technology, which is an AI curriculum that aims to engage students with its social and ethical implications (Touretzky et al., 2019a ). Although the United States is working intensively on the design of integrating this knowledge into the curriculum, so far AI is not widely offered in most K-12 schools (Heintz, 2021 ).

In China, the Ministry of Education has integrated AI into the compulsory secondary school curriculum (Ottenbreit-Leftwich et al., 2021 ; Xiao & Song, 2021 ). Among their schemes we can reference the AI4Future initiative of the Chinese University of Hong Kong (CUHK), which promotes the co-creation process to implement AI education (Chiu et al., 2021 ). In Singapore, a program for AI learning in schools has also been developed, where K-12 children learn AI interactively. However, the program is hindered by a lack of professionals (teachers) with adequate training (Heintz, 2021 ). In Germany, there are also several initiatives to pilot AI-related projects and studies (Micheuz, 2020 ), including the launch of a national initiative to teach a holistic view of AI. This initiative consists of a 6-module course aimed at explaining how AI works, stimulating a social discourse on AI and clarifying the abundant existing misconceptions (Micheuz, 2020 ). Canada has also designed an AI course for high schools. The course is intended to empower students with knowledge about AI, covering both its philosophical and conceptual underpinnings as well as its practical aspects. The latter are achieved by building AI projects that solve real-life problems (Nisheva-Pavlova, 2021 ).

The literature also highlights the different approaches that AI literacy should focus on: curriculum design, AI subject design, student perspective, teacher training, resource design and gender diversity. All these approaches are described in depth below.

AI literacy curriculum design

Approaches to curriculum development differ widely, ranging from the product-centered model (technical-scientific perspective) to the process-centered model (learner perspective) (Yue et al., 2021 ). AI literacy can be launched in primary and secondary education depending on the age and computer literacy of the students. To do this, it is necessary to define the core competencies for AI literacy according to three dimensions: AI concepts, AI applications and AI ethics and security (Long & Magerko, 2020 ; Wong et al., 2020 ). Research has focused on the understanding of the concepts, the functional roles of AI, and the development of problem-solving skills (Woo et al., 2020 ). This has led to proposing a redefinition of the curriculum (Han et al., 2019 ; Malach & Vicherková, 2020 ; Zhang et al., 2020 ) supported by different ideas that K-12 students should know (Chiu et al., 2021 ; Sabuncuoglu, 2020 ; Touretzky et al., 2019b ). Several countries have already made different curricular proposals (Alexandre et al., 2021 ; Micheuz, 2020 ; Nisheva-Pavlova, 2021 ; Ottenbreit-Leftwich et al., 2021 ; Touretzky et al., 2019b ; Xiao & Song, 2021 ), where they argue that the curricular design must include different elements such as content, product, process and praxis (Chiu, 2021 ). It is also convenient for learning in AI to follow the computational thinking model (Shin, 2021 ), contextualizing the proposed curriculum (Eguchi et al., 2021 ; Wang et al., 2020 ) and providing it with the necessary resources for teachers (Eguchi et al., 2021 ). In this sense, emerging initiatives highlight the need to involve teachers in the process of co-creating a curriculum associated to their context (Barlex et al., 2020 ; Chiu et al., 2021 ; Dai et al., 2023 ; Lin & Brummelen, 2021 ; Yau et al., 2022 ).

AI as a subject in K-12 education

Traditionally, including computer science or new technologies in the educational system has been carried out through a specific subject integrated into the curriculum or through the offer of extracurricular activities. In this sense, different proposals have suggested the integration of AI as a subject in K-12 education (Ellis et al., 2009 ; Knijnenburg et al., 2021 ; Micheuz, 2020 ; Sperling & Lickerman, 2012 ), in short-term courses (around 15 h) and divided into learning modules focused on classical and modern AI (Wong, 2020 ) or through MOOCs (Alexandre et al., 2021 ).

Student perspective on AI Literacy

Student-focused studies explore and analyze attitudes and previous knowledge to make didactic proposals adapted to the learner. Some of them measure their intention and interest in learning AI (Bollin et al., 2020 ; Chai et al., 2021 , 2020a , 2020b ; Gao & Wang, 2019 ; Harris et al., 2004 ; Sing, et al., 2022 ; Suh & Ahn, 2022 ), whereas others discuss their views on the integration of technologies in the education system (Sorensen & Koefoed, 2018 ) and on teaching–learning support tools in AI (Holstein et al., 2019 ).

Teacher training in AI

Teachers are key players for the integration of AI literacy in K-12, as proven by the numerous studies that examine this issue (An et al., 2022 ; Bai & Yang, 2019 ; Chiu & Chai, 2020 ; Chiu et al., 2021 ; Chounta et al., 2021 ; Judd, 2020 ; Kandlhofer et al., 2019 , 2021 ; Kim et al., 2021 ; Korenova, 2016 ; Lin et al., 2022 ; Lindner & Berges, 2020 ; Oh, 2020 ; Summers et al., 1995 ; Wei et al., 2020 ; Wu et al., 2020 ; Xia & Zheng, 2020 ). This approach places teachers at the center, bearing in mind what they need to know so as to integrate AI into K-12 (Itmazi & Khlaif, 2022 ; Kim et al., 2021 ). The literature analyzed reports on the factors that influence the knowledge of novice teachers (Wei, 2021 ) and focuses on teacher training in AI (Lindner & Berges, 2020 ; Olari & Romeike, 2021 ). Thus, AI training proposals can be found aimed at both teachers in training (Xia & Zheng, 2020 ) and practicing educators. Training schemes focus on their knowledge in technologies to facilitate their professional development (Wei et al., 2020 ) through the TPACK (Technological, Pedagogical and Content Knowledge) teaching knowledge model (Gutiérrez-Fallas & Henriques, 2020 ). Studies focusing on teachers’ opinions on curriculum development in AI are relevant (Chiu & Chai, 2020 ), as are their self-efficacy in relation to ICT (Wu et al., 2020 ), their opinions on the tools that support the teaching–learning process in AI (Holstein et al., 2019 ) and their teacher training in technologies (Cheung et al, 2018 ; Jaskie et al., 2021 ). These elements are central to the design of an AI literacy strategy in K-12. Both the co-design of ML curricula between AI researchers and K-12 teachers, and the assessment of the impact of these educational interventions on K-12 are important issues today. At present, there is a shortage of teachers with training in AI and working with teachers in training (Xia & Zheng, 2020 ) or with teachers in schools (Chiu et al., 2021 ) is proposed as an effective solution. One of the most interesting analyses of teacher competency proposes the acquisition of this skill for the teaching of AI in K-12, through the analysis of the curricula and resources of AI using TPACK. This model was formulated by (Mishra & Koehler, 2006 ) and aims to define the different types of knowledge that teachers need to integrate ICT effectively in the classroom. In this regard, it is suggested that teachers imparting AI to K-12 students require TPACK to build an environment and facilitate project-based classes that solve problems using AI technologies (Kim et al., 2021 ).

AI literacy support resources

Research using this approach focuses on presenting resources that support AI literacy (Kandlhofer & Steinbauer, 2021 ), considering that the creation of resources and repositories is a priority in supporting this teaching–learning process (Matarić et al., 2007 ; Mongan & Regli, 2008 ). However, these resources largely do not meet an interdisciplinary approach and do not embody a general approach to AI development (Sabuncuoglu, 2020 ).

Gender diversity in AI literacy

AI education, as a broad branch of computer science, also needs to address the issue of gender diversity. Lack of gender diversity can impact the lives of the people for whom AI-based systems are developed. The literature highlights the existence of proposals designed with a perspective toward gender, where the activities designed are specifically aimed at girls (Ellis et al., 2009 ; Jagannathan & Komives, 2019 ; Perlin et al., 2005 ; Summers et al., 1995 ; Vachovsky et al., 2016 ; Xia et al., 2022 ).

The huge impact that AI is having on our lives, at work and in every type of organization and business sector is easily recognizable today. No one doubts that AI is one of the most disruptive technologies in history, if not the most. In recent years, the expectations generated by AI, far from being deflated, have only grown. We are still a long way from general-purpose AI, but the application of AI to solve real problems has already taken hold for a wide range of purposes. It is therefore necessary for young people to know how AI works, as this learning will make it easier for them to use these technologies in their daily lives, both to learn and to interact with others.

Like any other technology, the potential uses and abuses of AI go hand in hand with its disruptive capacity. Many social groups and governments are expressing concern about the possible negative consequences of AI misuse. Although it is crucial to adequately regulate the use of AI, education is as important, if not more important, than regulation. Everything, whether good or bad, stems from the education received. Thus, education systems must prepare students for a society in which they will have to live and interact with AI. AI education will enable young people to discover how these tools work and, consequently, to act responsibly and critically. Therefore, AI literacy has become a relevant and strategic issue (Chiu & Chai, 2020 ).

This systematic review has focused on analyzing AI teaching–learning proposals in K-12 globally. The results confirm that the teaching of basic AI- related concepts and techniques at the K-12 level is scarce (Kandlhofer et al., 2016 ). Our work shows that there have been, on the one hand, different AI learning experiences and, on the other hand, proposals for the implementation of AI literacy, made at the political level and by different experts. The learning experiences described show that AI literacy in schools has focused on technical, conceptual, and applied skills in some domains of interest. Proposals for AI implementation, especially those defined by the US and China, reveal that significant efforts are being made to design models that frame AI literacy proposals.

We also found that there are hardly any AI learning experiences that have analyzed learning outcomes, e.g., through assessments of learners’ understanding of AI concepts. Obviously, this is a result of the infancy of these AI learning experiences at the K-12 level. However, it is important for learning experiences to be based on clearly defined competencies in a particular AI literacy framework, such as those proposed in the literature (Alexandre et al., 2021 ; Han et al., 2019 ; Long & Magerko, 2020 ; Malach & Vicherková, 2020 ; Micheuz, 2020 ; Ottenbreit-Leftwich et al., 2021 ; Touretzky et al., 2019a ; Wong et al., 2020 ; Xiao & Song, 2021 ; Zhang et al., 2020 ). Recently, Van Brummelen et al. ( 2021a ) designed a curriculum for a five-day online workshop based on the specific AI competencies proposed by Long and Magerko ( 2020 ). They used several types of questionnaires to assess the quality of the program through the knowledge acquired by the students in these competencies. Therefore, clearly defined competency-based learning experiences can provide a rigorous assessment of student learning outcomes.

The research shows that clear guidelines are needed on what students are expected to learn about AI in K-12 (Chiu, 2021 ; Chiu & Chai, 2020 ; Lee et al., 2020 ). These studies highlight the need for a competency framework to guide the design of didactic proposals for AI literacy in K-12 in educational institutions. This framework would provide a benchmark for describing the areas of competency that K-12 learners should develop and which specific educational projects can be designed. Furthermore, it would support the definition of a curriculum reflecting sequence and academic continuity (Woo et al., 2020 ). Such a curriculum should be modular and personalized (Gong et al., 2019 ) and adjusted to the conditions of the schools (Wang et al., 2020 ). In the teaching of AI, an exploratory education should be adopted, which integrates science, computer science and integral practice (Wang et al., 2020 ). It should also address issues related to the ethical dimension, which is fundamental to the literacy of K-12 students as it enables them to understand the basic principles of AI (Henry et al., 2021 ). This training facilitates the development of students’ critical capacity, and this is necessary to understand that technology is not neutral and to benefit from and make appropriate use of it. Ethics, complementary to legal norms, enhances the democratic quality of society by setting legitimate limits in the shaping of technological life. In this sense, different AI literacy proposals in K-12 already support the addressing of ethical, social and security issues linked to AI technologies (Eguchi et al., 2021 ; Micheuz, 2020 ; Wong et al., 2020 ). Moreover, considering designing for social good could foster or help to motivate learning about AI (Chai et al., 2021 ). Without a doubt, all this will impact on the achievement of a more democratic society. Due to the gender gap in issues related to computer science, it is also necessary to address the gender perspective. In this vein, the research proposes, among other strategies, to focus AI literacy on real-world elements since this approach favors the motivation of girls and greater involvement in learning (Jagannathan & Komives, 2019 ). However, little attention is paid to the undesirable consequences of an indiscriminate and insufficiently thought-out application of AI, both in higher education and especially in K-12. For example, the increase in socio-economic inequality between countries and within countries, resulting from the increasing automation of employment, is of particular concern. This is leading to growing inequality in wages and preservation of human employment, but it is not usually a subject of interest in education.

Currently, the challenges of this AI literacy require an interdisciplinary and critical approach (Henry et al., 2021 ). We believe that AI literacy can be leveraged to enhance the learning of disciplinary core subjects by integrating AI into the teaching process of those subjects. AI literacy should rely on transferring AI knowledge and methods to core subjects, allowing education to cross disciplinary boundaries, but staying within the framework of disciplinary core subjects. To achieve this change, educators need to take a closer look at the current capabilities of AI. This would enable them to identify all options to improve the core of educational practice and thus optimize the educational process. For example, understanding and using word clouds is a powerful educational strategy to enhance education in core subjects such as science (e.g., to facilitate object classification), language (e.g. to enable the matching of different topics or authors’ works), music (e.g., to support the analysis of song lyrics) or social sciences (e.g., to assist in comparing different discourses). Since AI is highly interdisciplinary in nature, it has a broad projection on multiple fields and problems that require a transversal and applied approach. For example, the basic algorithms of ML could be taught in Mathematics and related disciplines, the design of supervised classifiers could be performed for the study of taxonomies in Biology, natural language processing could be used to make the study of a language more attractive, or the ethical issues surrounding AI could be discussed in Philosophy and Social Sciences subjects.

Finally, for this meaningful learning to take place, AI teaching must be addressed through holistic, active, and collaborative pedagogical strategies in which real problem solving is the starting point of the learning process. An important gap regarding the integration of AI in K-12 concerns teachers, as it is unclear how to prepare and involve them in the process (Chiu & Chai, 2020 ). Teachers’ attitudes towards AI have a significant influence on the effectiveness of using AI in education. Teachers can swing between total resistance and overconfidence. The first could arise from inadequate, inappropriate, irrelevant, or outdated professional development. On the one hand, teachers must be digitally-competent enough to integrate AI into the teaching–learning processes of their subjects. Therefore, teacher training is also necessary following a framework of standard competencies. This should include new ways of organizing the professional role of teachers, as well as enhancing students’ attitudes towards these changes. On the other hand, research reveals that it is essential for didactic proposals to be co-designed and implemented by the teachers at those schools involved (Henry et al., 2021 ), to undergo training in the specific AI subjects and for this knowledge to be integrated into non-computer subjects (Lin & Brummelen, 2021 ). To this end, it is crucial to identify the perception and knowledge that teachers have about AI and involve them in the design of curricular proposals (Chiu, 2021 ; Chiu & Chai, 2020 ; Chiu et al., 2021 ).

This study aimed to understand how AI literacy is being integrated into K-12 education. To achieve this, we conducted a search process following the systematic literature review method and using Scopus. Two broad groups of AI literacy approaches were identified, namely learning experiences and theoretical perspective. The study revealed that learning experiences in schools have focused mainly on technical and applied skills limited to a specific domain without rigorously assessing student learning outcomes. In contrast, the US and China are leading the way in AI literacy implementation schemes which are broader in scope and involve a more ambitious approach. However, there is still a need to test these initiatives through comprehensive learning experiences that incorporate an analysis of learning outcomes. This work has allowed us to draw several conclusions that can be considered in the design of AI literacy proposals in K-12. Firstly, AI literacy should be based on an interdisciplinary and competency-based approach and integrated into the school curriculum. There is no need to include a new AI subject in the curriculum, but rather to build on the competencies and content of disciplinary subjects and then integrate AI literacy into those subjects. Given the interdisciplinary nature of AI, AI education can break disciplinary boundaries and adopt a global, practical, and active approach in which project-based and contextualized work plays an important role. Secondly, AI literacy should be leveraged to extend and enhance learning in curricular subjects. As a final point, AI literacy must prioritize the competency of teachers and their active participation in the co-design of didactic proposals, together with pedagogues and AI experts.

Availability of data and materials

Last revision round required update the review. Thus, Additional file 1 contains a.csv file with the listing of papers that are not cited but are part of the reviewed papers. The papers cited in text already appear in the Reference section and, therefore, not in the Additional file.

1 Conference categorization and ranking based on the GII-GRIN-SCIE (GGS) Conference Ratings: https://scie.lcc.uma.es/

Aguar, K., Arabnia, H. R., Gutierrez, J. B., Potter, W. D., & Taha, T. R. (2016). Making cs inclusive: An overview of efforts to expand and diversify cs education. In International Conference on Computational Science and Computational Intelligence (CSCI). (pp. 321–326). https://doi.org/10.1109/CSCI.2016.0067

Alexandre, F., Becker, J., Comte, M. H., Lagarrigue, A., Liblau, R., Romero, M., & Viéville, T. (2021). Why, What and How to help each citizen to understand artificial intelligence? KI Kunstliche Intelligenz, 35 (2), 191–199. https://doi.org/10.1007/s13218-021-00725-7

Article   Google Scholar  

Ali, S., DiPaola, D., Lee, I., Hong, J., & Breazeal, C. (2021a). Exploring generative models with middle school students. In Proceedings of the 2021a CHI Conference on Human Factors in Computing Systems. (pp. 1–13). https://doi.org/10.1145/3411764.3445226

Ali, S., DiPaola, D., Lee, I., Sindato, V., Kim, G., Blumofe, R., & Breazeal, C. (2021b). Children as creators, thinkers and citizens in an AI-driven future. Computers and Education Artificial Intelligence . https://doi.org/10.1016/j.caeai.2021.100040

An, X., Chai, C. S., Li, Y., Zhou, Y., Shen, X., Zheng, C., & Chen, M. (2022). Modeling English teachers’ behavioral intention to use artificial intelligence in middle schools. Education and Information Technologies . https://doi.org/10.1007/s10639-022-11286-z

Aung, Z. H., Sanium, S., Songsaksuppachok, C., Kusakunniran, W., Precharattana, M., Chuechote, S., & Ritthipravat, P. (2022). Designing a novel teaching platform for AI: A case study in a Thai school context. Journal of Computer Assisted Learning, 38 (6), 1714–1729. https://doi.org/10.1111/jcal.12706

Avanzato, R. L. (2009). Autonomous Outdoor Mobile Robot Challenge. Computer in Education Journal (July-September 2009).

Bai, H., & Yang, S. (2019, October). Research on the Sustainable Development Model of Information Technology Literacy of Normal Students Based on Deep Learning Recommendation System. In 2019 4th International Conference on Mechanical, Control and Computer Engineering (ICMCCE). (pp. 837–840). https://doi.org/10.1109/ICMCCE48743.2019.00192

Barlex, D., Steeg, T., & Givens, N. (2020). Teaching about disruption: A key feature of new and emerging technologies. Learning to Teach Design and Technology in the Secondary School, 4 , 137–154. https://doi.org/10.4324/9780429321191-9

Bollin, A., Kesselbacher, M., & Mößlacher, C. (2020). Ready for computing science? A closer look at personality, interests and self-concept of girls and boys at secondary level. In  Informatics in Schools. Engaging Learners in Computational Thinking: 13th International Conference, ISSEP. (pp. 107–118). https://doi.org/10.1007/978-3-030-63212-0_9

Burgsteiner, H., Kandlhofer, M., & Steinbauer, G. (2016). IRobot: teaching the basics of artificial intelligence in high schools. Proceedings of the AAAI Conference on Artificial Intelligence . https://doi.org/10.1609/aaai.v30i1.9864

Chai, C.S., Lin, P.-Y., Jong, M.S.-Y., Dai, Y., Chiu, T.K., & Huang, B. (2020a). Factors influencing students' behavioral intention to continue artificial intelligence learning. In 2020a International Symposium on Educational Technology (ISET). (pp. 147–150).  https://doi.org/10.1109/ISET49818.2020.00040

Chai, C. S., Wang, X., & Xu, C. (2020b). An extended theory of planned behavior for the modelling of chinese secondary school students’ intention to learn artificial intelligence. Mathematics, 8 (11), 1–18. https://doi.org/10.3390/math8112089

Chai, C.S., Lin, P.-Y., Jong, M.S.-Y., Dai, Y., Chiu, T.K., & Qin, J. (2021). Perceptions of and behavioral intentions towards learning artificial intelligence in primary school students. Educational Technology & Society , 24 (3), 89–101. Retrieved from https://www.jstor.org/stable/27032858

Chen, S., Qian, B., & Cheng, H. (2017). Voice recognition for STEM education using robotics. In Volume 9: 13th ASME/IEEE International Conference on Mechatronic and Embedded Systems and Applications. ASME . https://doi.org/10.1115/DETC2017-68368

Chen, M., Zhou, C., & Wu, Y. (2020). Research on the model and development status of information literacy self-improvement ability of primary and secondary school teachers. In Ninth International Conference of Educational Innovation through Technology (EITT). (pp. 87–91). https://doi.org/10.1109/EITT50754.2020.00021

Cheung, S. K., Lam, J., Li, K. C., Au, O., Ma, W. W., & Ho, W. S. (Eds.). (2018).  Technology in Education. Innovative Solutions and Practices: Third International Conference, ICTE 2018. Springer.

Chittora, S., & Baynes, A. (2020, October). Interactive Visualizations to Introduce Data Science for High School Students. In  Proceedings of the 21st Annual Conference on Information Technology Education.  (pp. 236–241). https://doi.org/10.1145/3368308.3415360

Chiu, T. K. F. (2021). A holistic approach to the design of artificial intelligence (AI) education for k-12 schools. TechTrends, 65 (5), 796–807. https://doi.org/10.1007/s11528-021-00637-1

Chiu, T. K., & Chai, C.-S. (2020). Sustainable curriculum planning for artificial intelligence education: A self-determination theory perspective. Sustainability (switzerland) . https://doi.org/10.3390/su12145568

Chiu, T. K., Meng, H., Chai, C. S., King, I., Wong, S., & Yam, Y. (2021). Creation and evaluation of a pretertiary artificial intelligence (AI) curriculum. IEEE Transactions on Education, 65 (1), 30–39. https://doi.org/10.1109/TE.2021.3085878

Chounta, I.-A., Bardone, E., Raudsep, A., & Pedaste, M. (2021). Exploring teachers’ perceptions of artificial intelligence as a tool to support their practice in estonian k-12 education. International Journal of Artificial Intelligence in Education, 32 , 725–755. https://doi.org/10.1007/s40593-021-00243-5

Crompton, H., Jones, M. V., & Burke, D. (2022). Affordances and challenges of artificial intelligence in K-12 education: a systematic review. Journal of Research on Technology in Education . https://doi.org/10.1080/15391523.2022.2121344

Croxell, J., Mead, R., & Weinberg, J. (2007). Designing robot competitions that promote ai solutions: Lessons learned competing and designing. Technical Report of the 2007 American Association of Artificial Intelligence. Spring Symposia , SS-07–09. (pp. 29–34).

Cummings D., Anthony M., Watson C., Watson A., & Boone S. (2021). Combating social injustice and misinformation to engage minority youth in computing sciences. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education. (pp.1006–1012). https://doi.org/10.1145/3408877.3432452 .

Dai, Y., Liu, A., Qin, J., Guo, Y., Jong, M. S. Y., Chai, C. S., & Lin, Z. (2023). Collaborative construction of artificial intelligence curriculum in primary schools. Journal of Engineering Education, 112 (1), 23–42. https://doi.org/10.1002/jee.20503

Díaz, J., Queiruga, C., Tzancoff, C., Fava, L., & Harari, V. (2015). Educational robotics and videogames in the classroom. In 2015 10th Iberian Conference on Information Systems and Technologies (CISTI) . Aveiro, Portugal. (pp. 1–6). https://doi.org/10.1109/CISTI.2015.7170616

du Boulay, B. (2016). Artificial intelligence as an effective classroom assistant. IEEE Intelligent Systems, 31 (6), 76–81. https://doi.org/10.1109/MIS.2016.93

Eguchi, A. (2021). AI-robotics and ai literacy. Studies in Computational Intelligence, 982 , 75–85. https://doi.org/10.1007/978-3-030-77022-8

Eguchi, A., & Okada, H. (2018). If you give students a social robot? - world robot summit pilot study. In  Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction.  (pp. 103–104). https://doi.org/10.1145/3173386.3177038

Eguchi, A., & Okada, H. (2020). Imagine the Future with Social Robots - World Robot Summit’s Approach: Preliminary Investigation. In M. Moro, D. Alimisis, & L. Iocchi, L. (Eds) Educational Robotics in the Context of the Maker Movement. Edurobotics 2018. Advances in Intelligent Systems and Computing, (p. 946). Springer. https://doi.org/10.1007/978-3-030-18141-3_10

Eguchi, A., Okada, H., & Muto, Y. (2021). Contextualizing AI education for k- 12 students to enhance their learning of ai literacy through culturally responsive approaches. KI Kunstliche Intelligenz, 35 (2), 153–161. https://doi.org/10.1007/s13218-021-00737-3

Ellis, G., Ory, E., Bhushan, N. (2005). Organizing a K-12 AI curriculum using philosophy of the mind. Engineering: Faculty Publications, Smith College. Retrieved from https://scholarworks.smith.edu/egr_facpubs/96

Ellis, G., Silva, K., Epstein, T., & Giammaria, N. (2009). Artificial intelligence in pre-college education: Learning within a philosophy of the mind framework. International Journal of Engineering Education, 25 (3), 511–522.

Google Scholar  

Evangelista, I., Blesio, G., & Benatti, E. (2019). Why are we not teaching machine learning at high school? a proposal. In 2018 World Engineering Education Forum - Global Engineering Deans Council (WEEF-GEDC). (pp. 1–6). https://doi.org/10.1109/WEEF-GEDC.2018.8629750

Gao, J., & Wang, L. (2019). Reverse thinking teaching discussion in high school information technology under new curriculum standards. In 14th International Conference on Computer Science & Education (ICCSE). (pp. 222–226). https://doi.org/10.1109/ICCSE.2019.8845429

Giannakos, M., Voulgari, I., Papavlasopoulou, S., Papamitsiou, Z., & Yannakakis, G. (2020). Games for artificial intelligence and machine learning education: Review and perspectives. Lecture Notes in Educational Technology . https://doi.org/10.1007/978-981-15-6747-6_7

Gong, X., Zhao, L., Tang, R., Guo, Y., Liu, X., He, J., … Wang, X. (2019). AI education system for primary and secondary schools. In  2019 ASEE Annual Conference & Exposition .

Gonzalez, A. J., Hollister, J. R., DeMara, R. F., Leigh, J., Lanman, B., Lee, S. Y., & Wilder, B. (2017). AI in informal science education: bringing turing back to life to perform the turing test. International Journal of Artificial Intelligence in Education, 27 (2), 353–384. https://doi.org/10.1007/s40593-017-0144-1

Guan, C., Mou, J., & Jiang, Z. (2020). Artificial intelligence innovation in education: A twenty-year data-driven historical analysis. International Journal of Innovation Studies, 4 (4), 134–147. https://doi.org/10.1016/j.ijis.2020.09.001

Gutiérrez, L. F., & Henriques, A. (2020). Prospective mathematics teachers’ tpack in a context of a teacher education experiment. Revista Latinoamericana De Investigación En Matemática Educativa, 23 (2), 175–202. https://doi.org/10.12802/relime.20.2322

Han, X., Hu, F., Xiong, G., Liu, X., Gong, X., Niu, X., … Wang, X. (2019). Design of AI + curriculum for primary and secondary schools in Qingdao. In Chinese Automation Congress (CAC) . (pp. 4135–4140). https://doi.org/10.1109/CAC.2018.8623310

Harris, E., Lamonica, A., & Weinberg. JB. (2004) Interfacing the public and technology: a web controlled mobile robot. In Accessible hands-on artificial intelligence and robotics education: working papers of the 2004. AAAI spring symposium series. AAAI Press. (pp.106–110)

He, Y.-T., Guo, B.-J., Lu, J., Xu, Y.-P., & Gong, M. (2020). Research of scratch programming recommendation system based on med and knowledge graph. In 2020 5th International Conference on Mechanical, Control and Computer Engineering (ICMCCE) . (pp. 2158–2163). https://doi.org/10.1109/ICMCCE51767.2020.00469

Heintz, F. (2021). Three interviews about k-12 ai education in america, europe, and singapore. KI Kunstliche Intelligenz, 35 (2), 233–237. https://doi.org/10.1007/s13218-021-00730-w

Henry, J., Hernalesteen, A., & Collard, A.-S. (2021). Teaching artificial intelli- gence to k-12 through a role-playing game questioning the intelligence concept. KI Kunstliche Intelligenz, 35 (2), 171–179. https://doi.org/10.1007/s13218-021-00733-7

Herrero, Á., Cambra, C., Urda, D., Sedano, J., Quintián, H., & Corchado, E. (Eds) (2020). 11th International Conference on European Transnational Educational (ICEUTE 2020). ICEUTE 2020. Advances in Intelligent Systems and Computing, 1266 . Springer. https://doi.org/10.1007/978-3-030-57799-5_8

Ho, J. W., Scadding, M., Kong, S. C., Andone, D., Biswas, G., Hoppe, H. U., & Hsu, T. C. (2019). Classroom activities for teaching artificial intelligence to primary school students. In  Proceedings of international conference on computational thinking education. The Education University of Hong Kong. (pp. 157–159).

Holowka, P. (2020). Teaching robotics during COVID-19: Machine learning, simulation, and aws deepracer. In 17th International Conference on Cognition and Exploratory Learning in Digital Age, CELDA .

Holstein, K., McLaren, B. M., & Aleven, V. (2019). Designing for complementarity: Teacher and student needs for orchestration support in ai-enhanced classrooms. In S. Isotani, E. Millán, A. Ogan, P. Hastings, B. McLaren, & R. Luckin (Eds.), Artificial intelligence in education. AIED 2019. Lecture notes in computer science (p. 11625). Springer. 10.1007/978-3-030-23204-7_14.

Itmazi, J., & Khlaif, Z. N. (2022). Science education in Palestine: Hope for a better future. Lecture Notes in Educational Technology . https://doi.org/10.1007/978-981-16-6955-2_9

Jagannathan, R. K., & Komives, C. (2019). Teaching by induction: Project- based learning for Silicon Valley. Journal of Engineering Education Transformations, 33 (1), 22–26. https://doi.org/10.16920/jeet/2019/v33i1/149003

Jaskie, K., Larson, J., Johnson, M., Turner, K., O’Donnell, M., Christen, J.B., & Spanias, A. (2021). Research experiences for teachers in machine learning. In IEEE Frontiers in Education Conference (FIE) . Lincoln, NE, USA. (pp. 1–5). https://doi.org/10.1109/FIE49875.2021.9637132

Judd, S. (2020). Activities for Building Understanding: How AI4ALL Teaches AI to Diverse High School Students. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education. (pp. 633–634). https://doi.org/10.1145/3328778.3366990

Kahn, K., Megasari, R., Piantari, E., & Junaeti, E. (2018). AI programming by children using Snap! block programming in a developing country. In Thirteenth European Conference on Technology Enhanced Learning . (p. 11082). https://doi.org/10.1007/978-3-319-98572-5

Kandlhofer, M., Steinbauer, G., Hirschmugl-Gaisch, S., & Huber, P. (2016). Artificial intelligence and computer science in education: From kinder- garten to university. In IEEE Frontiers in Education Conference. (pp. 1–9). https://doi.org/10.1109/FIE.2016.7757570

Kandlhofer, M., Steinbauer, G., Lasnig, J.P., Baumann, W., Plomer, S., Ballagi, A., & Alfoldi, I. (2019). Enabling the creation of intelligent things: Bringing artificial intelligence and robotics to schools. In IEEE Frontiers in Education Conference (FIE) . (pp. 1–5). https://doi.org/10.1109/FIE43999.2019.9028537

Kandlhofer, M., & Steinbauer, G. (2021). AI k-12 education service. KI Kunstliche Intelligenz, 35 (2), 125–126. https://doi.org/10.1007/s13218-021-00715-9

Kandlhofer, M., Steinbauer, G., Lassnig, J., Menzinger, M., Baumann, W., Ehardt-Schmiederer, M., & Szalay, I. (2021). EDLRIS: A European driving license for robots and intelligent systems. KI Kunstliche Intelligenz, 35 (2), 221–232. https://doi.org/10.1007/s13218-021-00716-8

Kaspersen, M. H., Bilstrup, K. E. K., Van Mechelen, M., Hjort, A., Bouvin, N. O., & Petersen, M. G. (2022). High school students exploring machine learning and its societal implications Opportunities and challenges. International Journal of Child-Computer Interaction, 34 , 1–12. https://doi.org/10.1016/j.ijcci.2022.100539

Kelly, J., Binney, J., Pereira, A., Khan, O., & Sukhatme, G. (2008). Just add wheels: Leveraging commodity laptop hardware for robotics and ai education. In  Proceedings of AAAI Education Colloquium , 22.

Kim, K., Kwon, K., Ottenbreit-Leftwich, A., Bae, H., & Glazewski, K. (2023). Exploring middle school students’ common naive conceptions of Artificial Intelligence concepts, and the evolution of these ideas. Education and Information Technologies . https://doi.org/10.1007/s10639-023-11600-3

Kim, S., Jang, Y., Choi, S., Kim, W., Jung, H., Kim, S., & Kim, H. (2021). Analyzing teacher competency with tpack for k-12 ai education. KI Kunstliche Intelligenz, 35 (2), 139–151. https://doi.org/10.1007/s13218-021-00731-9

Kitchenham, B. (2004). Procedures for performing systematic reviews (Vol. 33, pp. 1–26). Keele: Keele University.

Knijnenburg, B., Bannister, N., & Caine, K. (2021). Using mathematically- grounded metaphors to teach ai-related cybersecurity. In IJCAI-21 Workshop on Adverse Impacts and Collateral Effects of Artificial Intelligence Technologies (AIofAI) .

Kong, S. C., ManYinCheung, W., & Zhang, G. (2021). Evaluation of an artificial intelligence literacy course for university students with diverse study backgrounds. Computers and Education: Artificial Intelligence . https://doi.org/10.1016/j.caeai.2021.100026

Korenova, L. (2016). Digital technologies in teaching mathematics on the faculty of education of the Comenius University in Bratislava. In 15 Conference on Applied Mathematics. Slovak University of Technology in Bratislava. (p. 690–699).

Lee, S., Mott, B., Ottenbriet-Leftwich, A., Scribner, A., Taylor, S., Glazewski, K.,…Lester, J. (2020). Designing a collaborative game-based learning environment for ai-infused inquiry learning in elementary school class- rooms. In Proceedings of the 2020 ACM conference on innovation and technology in computer science education . (pp. 566–566). https://doi.org/10.1145/3341525.3393981

Lee, I., Ali, S., Zhang, H., Dipaola, D., & Breazeal, C. (2021). Developing middle school students’ ai literacy. In Association for Computing Machinery, Inc. (pp. 191–197). https://doi.org/10.1145/3408877.3432513

Lenoir, Y., & Hasni, A. (2016). Interdisciplinarity in primary and secondary school: Issues and perspectives. Creative Education, 7 (16), 2433–2458. https://doi.org/10.4236/ce.2016.716233

Lin, P., & Brummelen, J. (2021). Engaging teachers to co-design integrated ai curriculum for k-12 classrooms. In CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems . (pp.1–12). https://doi.org/10.1145/3411764.3445377

Lin, X. F., Chen, L., Chan, K. K., Peng, S., Chen, X., Xie, S., & Hu, Q. (2022). Teachers’ perceptions of teaching sustainable artificial intelligence: A design frame perspective. Sustainability, 14 (13), 1–20. https://doi.org/10.3390/su14137811

Lindner, A., & Berges, M. (2020). Can you explain ai to me? teachers’ pre- concepts about artificial intelligence. In IEEE Frontiers in Education Conference (FIE). (pp. 1–9). https://doi.org/10.1109/FIE44824.2020.9274136

Long, D., & Magerko, B. (2020). What is AI literacy? Competencies and design considerations. In Proceedings of the 2020 chi conference on human factors in computing systems. (pp. 1–16). https://doi.org/10.1145/3313831.3376727

Malach, J., & Vicherková, D. (2020). Background of the Revision of the Secondary School Engineering Curriculum in the Context of the Society 4.0. In M. Auer, H. Hortsch & P. Sethakul (Eds). The Impact of the 4th Industrial Revolution on Engineering Education. ICL Advances in Intelligent Systems and Computing, vol 1135. Springer. https://doi.org/10.1007/978-3-030-40271-6_27

Matarić, M.J., Koenig, N., & Feil-Seifer, D. (2007). Materials for enabling hands- on robotics and stem education. In AAAI Spring Symposium: Semantic Scientific Knowledge Integration. (pp. 99–102). http://www.aaai.org/Papers/Symposia/Spring/2007/SS-07-09/SS07-09-022.pdf

Miao, F., Holmes, W., Huang, R., & Zhang, H. (2021). AI and education: A guidance for policymakers . UNESCO Publishing.

Micheuz, P. (2020). Approaches to Artificial Intelligence as a Subject in School Education. In T. Brinda, D. Passey, & T. Keane (Eds), Empowering Teaching for Digital Equity and Agency. OCCE 2020. IFIP Advances in Information and Communication Technology, 595. Springer. https://doi.org/10.1007/978-3-030-59847-1_1

Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge: A framework for teacher knowledge. Teachers College Record, 108 (6), 1017–1054. https://doi.org/10.1111/j.1467-9620.2006.00684.x

Mongan, W.M., & Regli, W.C. (2008). A cyber-infrastructure for supporting k-12 engineering education through robotics, WS-08-02, 68–73.

Narahara, T., & Kobayashi, Y. (2018). Personalizing homemade bots with plug & play ai for steam education. In SIGGRAPH Asia 2018 technical briefs . (pp. 1–4). https://doi.org/10.1145/3283254.3283270

Ng, D. T. K., Lee, M., Tan, R. J. Y., Hu, X., Downie, J. S., & Chu, S. K. W. (2022). A review of AI teaching and learning from 2000 to 2020. Education and Information Technologies . https://doi.org/10.1007/s10639-022-11491-w

Nisheva-Pavlova, M.M. (2021). Ai courses for secondary and high school - comparative analysis and conclusions. In CEUR Workshop Proceedings, 3061. (pp. 9–16).

Nurbekova, Z., Mukhamediyeva, K., & Assainova, A. (2018). Educational robotics technologies in Kazakhstan and in the world: Comparative analysis, current state and perspectives. Astra Salvensis, 6 (1), 665–686.

Oh, W. (2020). Physics teachers’ perception of it convergence-based physics education. New Physics: Sae Mulli, 70 (8), 660–666. https://doi.org/10.3938/NPSM.70.660

Olari, V., & Romeike, R. (2021). Addressing ai and data literacy in teacher education: A review of existing educational frameworks. In WiPSCE '21: The 16th Workshop in Primary and Secondary Computing Education, 17. (pp. 1–2) https://doi.org/10.1145/3481312.3481351

Ottenbreit-Leftwich, A., Glazewski, K., Jeon, M., Hmelo-Silver, C., Mott, B., Lee, S., & Lester, J. (2021). How do elementary students conceptualize artificial intelligence? In SIGCSE '21: Proceedings of the 52nd ACM Technical Symposium on Computer Science Education . (pp. 1261). https://doi.org/10.1145/3408877.3439642

Park, K., Mott, B., Lee, S., Glazewski, K., Scribner, J., Ottenbreit-Leftwich, A., & Lester, J. (2021). Designing a visual interface for elementary students to formulate ai planning tasks. In IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). (pp. 1–9). https://doi.org/10.1109/VL/HCC51201.2021.9576163

Perlin, K., Flanagan, M., & Hollingshead, A. (2005). The Rapunsel Project. In Subsol, G. (Eds). Virtual Storytelling. Using Virtual Reality Technologies for Storytelling. ICVS 2005. Lecture Notes in Computer Science , 3805. Springer. https://doi.org/10.1007/11590361_29

Reyes, A., Elkin, C., Niyaz, Q., Yang, X., Paheding, S., & Devabhaktuni, V. (2020). A preliminary work on visualization-based education tool for high school machine learning education. In IEEE Integrated STEM Education Conference (ISEC). (pp. 1–5). https://doi.org/10.1109/ISEC49744.2020.9280629

Rodríguez-García, J., Moreno-León, J., Román-González, M., & Robles, G. (2020). Introducing artificial intelligence fundamentals with learning ML: Artificial intelligence made easy. In TEEM'20: Eighth International Conference on Technological Ecosystems for Enhancing Multiculturality . (pp. 18–20). https://doi.org/10.1145/3434780.3436705

Russell, S. (2021). The history and future of AI. Oxford Review of Economic Policy, 37 (3), 509–520. https://doi.org/10.1093/oxrep/grab013

Russell, S., & Norvig, P. (2021). Artificial Intelligence, global edition a modern approach . Pearson Deutschland.

Sabuncuoglu, A. (2020). Designing one year curriculum to teach artificial intelligence for middle school. In Proceedings of the 2020 ACM conference on innovation and technology in computer science education . (pp. 96–102). https://doi.org/10.1145/3341525.3387364

Sakulkueakulsuk, B., Witoon, S., Ngarmkajornwiwat, P., Pataranutapom, P., Surareungchai, W., Pataranutaporn, P., & Subsoontorn, P. (2019). Kids making ai: Integrating machine learning, gamification, and social context. In 2018 IEEE international conference on teaching, assessment, and learning for engineering (TALE) . (pp. 1005–1010). https://doi.org/10.1109/TALE.2018.8615249

Sanusi, I. T., Oyelere, S. S., Vartiainen, H., Suhonen, J., & Tukiainen, M. (2022). A systematic review of teaching and learning machine learning in K-12 education. Education and Information Technologies . https://doi.org/10.1007/s10639-022-11416-7

Shin, S. (2021). A study on the framework design of artificial intelligence thinking for artificial intelligence education. International Journal of Information and Education Technology, 11 (9), 392–397. https://doi.org/10.18178/ijiet.2021.11.9.1540

Sing, C. C., Teo, T., Huang, F., Chiu, T. K., & Xing Wei, W. (2022). Secondary school students’ intentions to learn AI: Testing moderation effects of readiness, social good and optimism. Educational Technology Research and Development, 70 (3), 765–782. https://doi.org/10.1007/s11423-022-10111-1

Sorensen, L., & Koefoed, N. (2018). The future of teaching—what are students’ expectations. In 2018 11th CMI International Conference: Prospects and Challenges Towards Developing a Digital Economy within the EU . (pp. 62–66). https://doi.org/10.1109/PCTDDE.2018.8624771

Sperling, A., & Lickerman, D. (2012). Integrating AI and machine learning in software engineering course for high school students. In Proceedings of the 17th ACM annual conference on Innovation and technology in computer science education . (pp. 244–249). https://doi.org/10.1145/2325296.2325354

Su, J., Zhong, Y., & Ng, D. T. K. (2022). A meta-review of literature on educational approaches for teaching AI at the K-12 levels in the Asia-Pacific region. Computers and Education: Artificial Intelligence, 3 , 1–18. https://doi.org/10.1016/j.caeai.2022.100065

Suh, W., & Ahn, S. (2022). Development and validation of a scale measuring student attitudes toward artificial intelligence. SAGE Open, 12 (2), 1–12. https://doi.org/10.1177/21582440221100463

Summers, B.G., Hicks, H., & Oliver, C. (1995). Reaching minority, female and disadvantaged students. In Proceedings Frontiers in Education 1995 25th Annual Conference. Engineering Education for the 21st Century , 1 . (992a4–16). https://doi.org/10.1109/FIE.1995.483030

Tedre, M., Toivonen, T., Kahila, J., Vartiainen, H., Valtonen, T., Jormanainen, I., & Pears, A. (2021). Teaching machine learning in k-12 classroom: Pedagogical and technological trajectories for artificial intelligence education. IEEE Access, 9 , 110558–110572. https://doi.org/10.1109/ACCESS.2021.3097962

Tims, H., Turner III, G., Cazes, G., & Marshall, J. (2012). Junior cyber discovery: Creating a vertically integrated middle school cyber camp. In  2012 ASEE Annual Conference & Exposition.  (pp. 25–867). Retrieved from https://peer.asee.org/21624

Toivonen, T., Jormanainen, I., Kahila, J., Tedre, M., Valtonen, T., Vartiainen, H. (2020). Co-designing machine learning apps in k-12 with primary school children. In  2020 IEEE 20th International Conference on Advanced Learning Technologies (ICALT). IEEE. (pp. 308–310). https://doi.org/10.1109/ICALT49669.2020.00099

Touretzky, D., Gardner-McCune, C., Breazeal, C., Martin, F., & Seehorn, D. (2019a). A year in k-12 ai education. AI Magazine, 40 (4), 88–90. https://doi.org/10.1609/aimag.v40i4.5289

Touretzky, D., Gardner-McCune, C., Martin, F., & Seehorn, D. (2019b). Envisioning ai for k-12: What should every child know about ai? Proceedings of the AAAI Conference on Artificial Intelligence, 33 (01), 9795–9799. https://doi.org/10.1609/aaai.v33i01.33019795

Vachovsky, M., Wu, G., Chaturapruek, S., Russakovsky, O., Sommer, R., & Fei-Fei, L. (2016). Towards more gender diversity in cs through an arti- ficial intelligence summer program for high school girls. In  Proceedings of the 47th ACM technical symposium on computing science education.  (pp. 303–308). https://doi.org/10.1145/2839509.2844620

Van Brummelen, J., Heng, T., & Tabunshchyk, V. (2021a). Teaching tech to talk: K-12 conversational artificial intelligence literacy curriculum and development tools. Proceedings of the AAAI Conference on Artificial Intelligence, 35 (17), 15655–15663. https://doi.org/10.1609/aaai.v35i17.17844

Van Brummelen, J., Tabunshchyk, V., & Heng, T. (2021b). Alexa, can i program you? Student perceptions of conversational artificial intelligence before and after programming Alexa. In IDC '21: Interaction Design and ChildrenJune. (pp. 305–313) https://doi.org/10.1145/3459990.3460730

Vartiainen, H., Tedre, M., & Valtonen, T. (2020). Learning machine learning with very young children: Who is teaching whom? International Journal of Child-Computer Interaction, 25 , 1–11. https://doi.org/10.1016/j.ijcci.2020.100182

Vartiainen, H., Toivonen, T., Jormanainen, I., Kahila, J., Tedre, M., & Valtonen, T. (2021). Machine learning for middle schoolers: Learning through data- driven design. International Journal of Child-Computer Interaction, 29 , 1–12. https://doi.org/10.1016/j.ijcci.2021.100281

Verner, I., Cuperman, D., & Reitman, M. (2021). Exploring robot connectivity and collaborative sensing in a high-school enrichment program. Robotics, 10 (1), 1–19. https://doi.org/10.3390/robotics10010013

von Wangenheim, C. G., Hauck, J. C., Pacheco, F. S., & Bueno, M. F. B. (2021). Visual tools for teaching machine learning in K-12: A ten-year systematic mapping. Education and Information Technologies, 26 (5), 5733–5778. https://doi.org/10.1007/s10639-021-10570-8

Wan, X., Zhou, X., Ye, Z., Mortensen, C., & Bai, Z. (2020). Smileyclus- ter: Supporting accessible machine learning in k-12 scientific discovery. In  proceedings of the Interaction Design and Children Conference.  (pp. 23–35). https://doi.org/10.1145/3392063.3394440

Wang, H., Liu, Y., Han, Z., & Wu, J. (2020). Extension of media literacy from the perspective of artificial intelligence and implementation strategies of artificial intelligence courses in junior high schools. In  2020 International Conference on Artificial Intelligence and Education (ICAIE).  (pp. 63–66). https://doi.org/10.1109/ICAIE50891.2020.00022

Wei, Y. (2021). Influence factors of using modern teaching technology in the classroom of junior middle school teachers under the background of artificial intelligence-analysis based on HLM. Advances in Intelligent Systems and Computing, 1282 , 110–118. https://doi.org/10.1007/978-3-030-62743-0_16

Wei, Q., Li, M., Xiang, K., & Qiu, X. (2020). Analysis and strategies of the professional development of information technology teachers under the vision of artificial intelligence. In  2020 15th International Conference on Computer Science & Education (ICCSE).  (pp. 716–721). https://doi.org/10.1109/ICCSE49874.2020.9201652

West, D.M., & Allen, J.R. (2018). How artificial intelligence is transforming the world. Report. Retrieved April 24, 2018, f rom https://www.brookings.edu/research/how-artificial-intelligence-is-transforming-the-world/

Wong, K.-C. (2020). Computational thinking and artificial intelligence education: A balanced approach using both classical AI and modern AI. CoolThink@ JC , 108.

Wong, G. K., Ma, X., Dillenbourg, P., & Huen, J. (2020). Broadening artificial intelligence education in k-12: Where to start? ACM Inroads, 11 (1), 20–29. https://doi.org/10.1145/3381884

Woo, H., Kim, J., Kim, J., & Lee, W. (2020). Exploring the ai topic composition of k-12 using nmf-based topic modeling. International Journal on Advanced Science, Engineering and Information Technology, 10 (4), 1471–1476. https://doi.org/10.18517/ijaseit.10.4.12787

Wu, D., Zhou, C., Meng, C., & Chen, M. (2020). Identifying multilevel factors influencing ICT self-efficacy of k-12 teachers in China. In  Blended Learning. Education in a Smart Learning Environment: 13th International Conference, ICBL 2020. (pp. 303–314). Springer International Publishing. https://doi.org/10.1007/978-3-030-51968-1

Xia, Q., Chiu, T. K., & Chai, C. S. (2022). The moderating effects of gender and need satisfaction on self-regulated learning through Artificial Intelligence (AI). Education and Information Technologies . https://doi.org/10.1007/s10639-022-11547-x

Xia, L., & Zheng, G. (2020). To meet the trend of AI: The ecology of developing ai talents for pre-service teachers in China. International Journal of Learning, 6 (3), 186–190. https://doi.org/10.18178/IJLT.6.3.186-190

Xiao, W., & Song, T. (2021). Current situation of artificial intelligence education in primary and secondary schools in China. In  The Sixth International Conference on Information Management and Technology. (pp. 1–4). https://doi.org/10.1145/3465631.3465980

Yau, K. W., Chai, C. S., Chiu, T. K., Meng, H., King, I., Wong, S. W. H., & Yam, Y. (2022). Co-designing artificial intelligence curriculum for secondary schools: A grounded theory of teachers' experience. In 2022 International Symposium on Educational Technology (ISET). (pp. 58–62). https://doi.org/10.1109/ISET55194.2022.00020

Yue, M., Dai, Y., Siu-Yung, M., & Chai, C.-S. (2021). An analysis of k-12 artificial intelligence curricula in eight countries. In  Proceedings of the 29th International Conference on Computers in Education.  (pp. 22–26).

Yue, M., Jong, M. S. Y., & Dai, Y. (2022). Pedagogical design of K-12 artificial intelligence education: A systematic review. Sustainability, 14 (23), 15620. https://doi.org/10.3390/su142315620

Zhai, X., Chu, X., Chai, C. S., Jong, M. S. Y., Istenic, A., Spector, M., & Li, Y. (2021). A review of artificial intelligence (AI) in education from 2010 to 2020. Complexity . https://doi.org/10.1155/2021/8812542

Zhang, N., Biswas, G., McElhaney, K.W., Basu, S., McBride, E., & Chiu, J.L. (2020). Studying the interactions between science, engineering, and computational thinking in a learning-by-modeling environment. In  International conference on artificial intelligence in education. (pp. 598–609). Springer.

Download references

Acknowledgements

Authors would like to thank the reviewers and editors, whose comments and feedback helped us to improve the original manuscript.

This work has partially been funded by the Spanish Ministry of Science, Innovation and Universities (PID2021-123152OB-C21), and the Consellería de Educación, Universidade e Formación Profesional (accreditation 2019–2022 ED431C2022/19 and reference competitive group, ED431G2019/04) and the European Regional Development Fund (ERDF), which acknowledges the CiTIUS—Centro Singular de Investigación en Tecnoloxías Intelixentes da Universidade de Santiago de Compostela as a Research Center of the Galician University System. This work also received support from the Educational Knowledge Transfer (EKT), the Erasmus + project (reference number 612414-EPP-1-2019-1-ES-EPPKA2-KA) and the Knowledge Alliances call (Call EAC/A03/2018).

Author information

Authors and affiliations.

Pedagogy and Didactics Department, University of Santiago de Compostela, 15782, Santiago de Compostela, Spain

Lorena Casal-Otero, Carmen Fernández-Morante & Beatriz Cebreiro

Departamento de Electrónica e Computación, Universidade de Santiago de Compostela, 15782, Santiago de Compostela, Spain

Alejandro Catala & Maria Taboada

Centro Singular de Investigación en Tecnoloxías Intelixentes (CiTIUS), Universidade de Santiago de Compostela, 15782, Santiago de Compostela, Spain

Alejandro Catala & Senén Barro

You can also search for this author in PubMed   Google Scholar

Contributions

All authors have contributed significantly to the authorship of this work in all stages of conceptualization, discussions, definition of the methodology, carrying out the analysis as well as writing—review & editing. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Alejandro Catala .

Ethics declarations

Ethics approval and consent to participate.

This research is carried out in accordance to ethics recommendations. As it focuses on a systematic literature review as a research method, ethics approval by the University ethics committee does not apply.

Competing interests

The authors declare that they have no competing interests. The authors have no other relevant financial or non-financial interests to disclose and no further competing interests to declare that are relevant to the content of this article.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1..

Additional listing.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Casal-Otero, L., Catala, A., Fernández-Morante, C. et al. AI literacy in K-12: a systematic literature review. IJ STEM Ed 10 , 29 (2023). https://doi.org/10.1186/s40594-023-00418-7

Download citation

Received : 06 September 2022

Accepted : 30 March 2023

Published : 19 April 2023

DOI : https://doi.org/10.1186/s40594-023-00418-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Secondary education
• Teaching/learning strategies
• Twenty-first century skills
• Cultural and social implications

• Open access
• Published: 30 May 2024

Barriers and facilitators to mental health treatment access and engagement for LGBTQA+ people with psychosis: a scoping review protocol

• Cláudia C. Gonçalves   ORCID: orcid.org/0000-0001-6767-0920 1 ,
• Zoe Waters 2 ,
• Shae E. Quirk 1 ,
• Peter M. Haddad 1 , 3 ,
• Ashleigh Lin 4 ,
• Lana J. Williams 1 &
• Alison R. Yung 1 , 5  

Systematic Reviews volume  13 , Article number:  143 ( 2024 ) Cite this article

124 Accesses

Metrics details

The prevalence of psychosis has been shown to be disproportionately high amongst sexual and gender minority individuals. However, there is currently little consideration of the unique needs of this population in mental health treatment, with LGBTQA+ individuals facing barriers in accessing timely and non-stigmatising support for psychotic experiences. This issue deserves attention as delays to help-seeking and poor engagement with treatment predict worsened clinical and functional outcomes for people with psychosis. The present protocol describes the methodology for a scoping review which will aim to identify barriers and facilitators faced by LGBTQA+ individuals across the psychosis spectrum in help-seeking and accessing mental health support.

A comprehensive search strategy will be used to search Medline, PsycINFO, Embase, Scopus, LGBTQ+ Source, and grey literature. Original studies of any design, setting, and publication date will be included if they discuss barriers and facilitators to mental health treatment access and engagement for LGBTQA+ people with experiences of psychosis. Two reviewers will independently screen titles/abstracts and full-text articles for inclusion in the review. Both reviewers will then extract the relevant data according to pre-determined criteria, and study quality will be assessed using the Joanna Briggs Institute (JBI) critical appraisal checklists. Key data from included studies will be synthesised in narrative form according to the Guidance on the Conduct of Narrative Synthesis in Systematic Reviews.

The results of this review will provide a comprehensive account of the current and historical barriers and facilitators to mental healthcare faced by LGBTQA+ people with psychotic symptoms and experiences. It is anticipated that the findings from this review will be relevant to clinical and community services and inform future research. Findings will be disseminated through publication in a peer-reviewed journal and presented at conferences.

Scoping review registration

This protocol is registered in Open Science Framework Registries ( https://doi.org/10.17605/OSF.IO/AT6FC ).

Peer Review reports

The prevalence of psychotic disorders in the general population has been estimated to be around 0.27–0.75% [ 1 , 2 ], with the lifetime prevalence of ever having a psychotic experience being estimated at 5.8% [ 3 ]. However, rates of psychotic symptoms and experiences are disproportionately high amongst LGBTQA+ populations, with non-heterosexual individuals estimated to be 1.99–3.75 times more likely to experience psychosis than their heterosexual peers [ 4 , 5 , 6 , 7 ]. Additionally, it has been estimated that transgender or gender non-conforming (henceforth trans) individuals are 2.46–49.7 times more likely than their cisgender peers (i.e. individuals whose gender identity is the same as their birth registered sex) to receive a psychotic disorder diagnosis [ 8 , 9 ]. The increased rates of psychotic experiences noted amongst gender and sexual minorities may be explained by evidence indicating that LGBTQA+ people are also exposed to risk factors for psychosis at a far greater rate than members of the general population, such as childhood adversity [ 10 , 11 , 12 ], minority stress [ 13 ], discrimination [ 14 ], and stigma [ 15 , 16 ]. Furthermore, there is added potential for diagnostic biases leading to over-diagnosing psychosis in gender diverse individuals, whose gender expression and dysphoria may be pathologized by mental health service providers [ 8 ].

Despite these concerning statistics, there is very little research examining the experiences of LGBTQA+ people with psychosis, and limited consideration of the unique needs these individuals may have in accessing and engaging with mental health services. While timely access to treatment has consistently been associated with better symptomatic and functional outcomes for people with psychosis [ 17 , 18 ], there are often delays to treatment initiation which are worsened for LGBTQA+ individuals [ 19 , 20 ]. These individuals face additional barriers to accessing adequate mental health support compared to cisgender/heterosexual people [ 19 ] and may need to experiment with several mental health services before finding culturally competent care [ 20 ]. This in turn may lead to longer duration of untreated psychosis. Additionally, there seems to be a lack of targeted support for this population from healthcare providers, with LGBTQA+ individuals with serious mental health concerns reporting higher rates of dissatisfaction with psychiatric services than their cisgender and heterosexual counterparts [ 7 , 14 , 21 ]. However, the extent of these differences varies across contexts [ 22 ], potentially due to improved education around stigma and LGBTQA+ issues within a subset of mental health services.

Nonetheless, stigma remains one of the highest cited barriers to help-seeking for mental health problems, particularly with regard to concerns around disclosure [ 23 ], which can be particularly challenging for people experiencing psychosis [ 24 , 25 ]. Stigma stress in young people at risk for psychosis is associated with less positive attitudes towards help-seeking regarding both psychiatric medication and psychotherapy [ 26 ], potentially partly due to fears of judgement and being treated differently by service providers [ 27 ]. This issue may be compounded for people who also belong to minoritized groups [ 23 , 28 ], particularly as LGBTQA+ individuals have reported experiencing frequent stigma and encountering uninformed staff when accessing mental healthcare [ 7 , 29 ]. Furthermore, stigma-fuelled hesitance to access services may be heightened for trans people [ 30 ] whose identities have historically been pathologized and conflated with experiences of psychosis [ 31 ].

Even when individuals manage to overcome barriers to access support, there are added challenges to maintaining adequate treatment engagement. In a large online study, half of trans and nearly one third of LGB participants reported having stopped using mental health services in the past because of negative experiences related to their gender identity or sexuality [ 20 ]. This can be particularly problematic as experiences of stigma predict poorer medication adherence in psychosis [ 32 ] which subsequently multiplies the risk for relapse and suicide [ 33 ]. While no research to date has explored non-adherence rates in people with psychosis who are LGBTQA+, concerns around suicidality are heightened for individuals who are gender and sexuality diverse [ 34 , 35 , 36 ].

Generally, there is rising demand for mental healthcare that specifically addresses the needs of gender and sexual minority individuals and promotes respect for diversity, equity, and inclusion [ 29 , 37 ]. This is particularly salient as positive relationships with staff are associated with better medication adherence for people with psychosis [ 38 ] and healthcare providers with LGBTQA+-specific mandates have demonstrated higher satisfaction rates for LGBTQA+ individuals [ 20 ]. Mental health services need to adapt treatment options to acknowledge minority stress factors for those with stigmatised identities and, perhaps more importantly, how these intersect and interact to increase inequalities in people from minoritized groups accessing and benefiting from treatment [ 37 , 39 ].

Additionally, gender affirming care needs to be recognised as an important facet of mental health treatment for many trans individuals, as it is associated with positive outcomes such as improvements in quality of life and psychological functioning [ 40 , 41 , 42 ] and reductions in psychiatric symptom severity and need for subsequent mental health treatment [ 8 , 43 ]. While there are additional barriers in access to gender affirming care for individuals with psychosis, this treatment has shown success in parallel with treatment to address psychosis symptom stabilisation [ 19 , 44 ]. The importance of affirmation is echoed by the finding that many negative experiences of LGBTQA+ participants with mental health services could be avoided simply by respecting people’s pronouns and using gender-neutral language [ 20 ].

To ensure timely access to appropriate treatment for LGBTQA+ people with psychosis, there is a need for improved understanding of the factors which challenge and facilitate help-seeking and engagement with mental health support. A preliminary search of Google Scholar, Medline, the Cochrane Database of Systematic Reviews, and PROSPERO was conducted and revealed no existing or planned reviews exploring benefits and/or obstacles to mental health treatment specific to this population. Therefore, the proposed review seeks to comprehensively search and appraise the existing literature to identify and summarise a range of barriers and facilitators to adequate mental health support faced by LGBTQA+ people with experiences of psychosis. This will allow for the mapping of the types of evidence available and identification of any knowledge gaps. Moreover, we hope to guide future decision-making in mental healthcare to improve service accessibility for LGBTQA+ individuals with psychosis and to set the foundations for future research that centres this marginalised population. Based on published guidance [ 45 , 46 , 47 ], a scoping review methodology was identified as the most appropriate approach to address these aims.

Selection criteria

This scoping review protocol has been developed in compliance with the JBI Manual for Evidence Synthesis [ 48 ] and, where relevant, the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) checklist [ 49 ] (see Additional file 1). In the event of protocol amendments, the date, justification, and description for each amendment will be provided.

Due to the limited literature around the topic of this review, any primary original study design, setting, and publication date will be considered for inclusion. Publications written in English will be included, and articles in other languages may be considered pending time and cost constraints around translation. Publications will be excluded if the full text is not available upon request from authors.

The PCC (Population, Concept, Context) framework was used to develop the inclusion criteria for this scoping review:

This review will include individuals of any age who are LGBTQA+ and have had experiences of psychosis. For the purposes of this review, ‘LGBTQA+ individuals’ will be broadly defined as any individual that is not heterosexual and/or cisgender or anyone who engages in same-gender sexual behaviour. Studies may include participants who are cisgender and heterosexual if they separately report outcomes for LGBTQA+ individuals. Within this review, the term ‘psychosis’ includes (i) any diagnosis of a psychotic disorder, such as schizophrenia spectrum disorders, mood disorders with psychotic features, delusional disorders, and drug-induced psychotic disorders, (ii) sub-threshold psychotic symptoms, such as those present in ultra-high risk (UHR), clinical high risk (CHR), or at risk mental state (ARMS) individuals, and (iii) any psychotic-like symptoms or experiences. Studies may include participants with multiple diagnoses if they separately report outcomes for individuals on the psychosis spectrum.

This review will include publications which discuss potential barriers and/or facilitators to mental health help-seeking and/or engagement with mental health treatment. ‘Barriers’ will be operationalised as any factors which may delay or prevent individuals from accessing and engaging with appropriate mental health support. These may include lack of mental health education, experienced or internalised stigma, experiences of discrimination from health services, and lack of inclusivity in health services. ‘Facilitators’ will be operationalised as any factors which may promote timely help-seeking and engagement with sources of support. These may include improved access to mental health education, positive sources of social support, and welcoming and inclusive services. Mental health help-seeking will be broadly defined as any attempt to seek and access formal or informal support to address a mental health concern related to experiences of psychosis (e.g. making an initial appointment with a service provider, seeking help from a friend). Mental health treatment engagement will be broadly defined as adherence and active participation in the treatment that is offered by a source of support (e.g. attending scheduled appointments, taking medication as prescribed, openly communicating with service providers).

This review may include research encompassing any setting in which mental healthcare is provided. This is likely to include formal healthcare settings such as community mental health teams or inpatient clinics as well as informal settings such as LGBTQA+ spaces or informal peer support. Studies will be excluded if they focus exclusively on physical health treatment.

Search strategy

Database searches will be conducted in Medline, PsycINFO, Embase, Scopus, and LGBTQ+ Source. The full search strategy for this protocol is available (see Additional file 2). This strategy has been collaboratively developed and evaluated by a scholarly services health librarian. Searches will include subject headings relevant to each database and title/abstract keywords relating to three main concepts: (i) LGBTQA+ identity, (ii) experiences of psychosis, and (iii) mental health treatment. Keywords for each concept will be combined using the Boolean operator ‘OR’, and the three concepts will be combined using ‘AND’. This search strategy was appropriately translated for each of the selected databases. There will be no limitations on language or publication date at this stage to maximise the breadth of the literature captured. Publications returned from these searches will be exported to EndNote. Searches will be re-run prior to the final analysis to capture any newly published studies.

The database searches will be supplemented by searching the grey literature as per the eligibility criteria detailed above. These may include theses and dissertations, conference proceedings, reports from mental health services, and policy documents from LGBTQA+ groups. Google and Google Scholar will be searched using a combination of clauses for psychosis (Psychosis OR psychotic OR schizophrenia OR schizoaffective), treatment (treatment or “help-seeking”), and queer identity. The latter concept will have three clauses for three separate searches, with one including broad queer identity (LGBT), one specific to non-heterosexual individuals (gay OR lesbian OR homosexual OR bisexual OR queer OR asexual), and one specific to trans individuals (transgender OR transsexual OR transexual OR “non-binary” OR “gender minority”). Additionally, reference lists and citing literature will be manually searched for each paper included in the review to capture any articles and policy documents not previously identified.

Data selection

Search results will be imported into Covidence using EndNote, and duplicates will be eliminated. Titles and abstracts will be screened by the first and second authors according to pre-defined screening criteria, which will be discussed by the authors and piloted prior to screening. These criteria will consider whether the articles included LGBTQA+ participants with experiences of psychosis (as operationalised above) in relation to mental health help-seeking and/or treatment. Full texts of relevant articles will then be obtained and screened by the first and second reviewer in accordance with the full inclusion and exclusion criteria after initial piloting to maximise inter-rater reliability. Decisions on inclusion and exclusion will be blinded and recorded on Covidence. Potential discrepancies will be resolved through discussion, and when consensus cannot be reached, these will be resolved by the supervising author. The process of study selection will be documented using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram [ 50 ].

Data extraction

Data extraction will be performed independently by two reviewers using Covidence. Prior to beginning final extraction, both reviewers will independently pilot the extraction tool using a sample of five included studies and discuss any necessary changes. Information extracted is planned to include the following: title, author name(s), year of publication, country in which the study was conducted, study design, sample size, population of focus (i.e. sexual minorities, gender minorities, or both), sample demographics (i.e. age, gender identity, and sexual orientation), setting (e.g. early intervention service, community mental health team, etc.), psychosis characteristics (e.g. diagnoses included, severity of symptoms, etc.), type of treatment (e.g. cognitive behavioural therapy, antipsychotic medication, etc.), and any barriers and/or facilitators identified according to the aforementioned operationalised definitions. Disagreements will be resolved through discussion between the two reviewers and, when necessary, final decisions will be made by a senior supervisor. Once extracted, information will be recorded in Excel. Lead authors of papers will be contacted by the primary review author in cases where there is missing or insufficient data.

Quality assessment

Due to the expected heterogeneity in the types of studies that may be included in this review (e.g. qualitative studies, randomised controlled trials, case control studies, case reports), the relevant revised Joanna Briggs Institute (JBI) critical appraisal checklists [ 51 ] will be used to assess risk of bias and study quality for each study design. Two reviewers will independently use these checklists to assess each paper that is included following the full-text screening. If there are discrepancies in article ratings, these will be resolved through discussion between the two authors. If no consensus is reached, discrepancies will be resolved by a senior supervisor. In line with the scoping nature of this review, low-quality studies will not be excluded from the synthesis.

Evidence synthesis

Data from included studies will be synthesised using a narrative synthesis approach in accordance with the Guidance on the Conduct of Narrative Synthesis in Systematic Reviews [ 52 ]. A preliminary descriptive synthesis will be conducted by tabulating the extracted data elements from each study alongside quality assessment results and developing an initial description of the barriers and facilitators to (1) accessing and (2) engaging with mental health support that are identified in the literature. This initial synthesis will then be interrogated and refined to contextualise these barriers and facilitators in the setting, population, and methodology of each study to form the basis for an interpretative synthesis.

This review will not use a pre-existing thematic framework to categorise barriers and facilitators as it is expected that the factors identified will not neatly fit into existing criteria. Instead, these will be conceptualised according to overarching themes as interrelated factors, so that potentially complex interactions between barriers and facilitators within and across relevant studies may be explored through concept mapping. If most of the studies included are qualitative, there may also be scope for a partial meta-synthesis. To avoid oversimplifying the concept of ‘barriers and facilitators’ (see criticism by Bach-Mortensen & Verboom [ 53 ]), this data synthesis will be followed by a critical reflection of the findings through the lens of the socio-political contexts which may give rise to the barriers and facilitators identified, exploring the complexities necessary for any changes to be implemented in mental health services.

If the extracted data indicate that gender minority and sexual minority individuals experience unique or different barriers and/or facilitators to each other, these population groups will be analysed separately as opposed to findings being generalised across the LGBTQA+ spectrum. Furthermore, if there is scope to do so, analyses may be conducted to investigate how perceived barriers and facilitators for this population may have changed over time (i.e. according to publication date) as definitions of psychosis evolve and LGBTQA+ individuals gain visibility in clinical services.

The proposed review will add to the literature around mental health treatment for LGBTQA+ people with psychosis. It will provide a thorough account of the barriers and facilitators to accessing and engaging with support faced by this population and may inform future research and clinical practice.

In terms of limitations, this review will be constrained by the existing literature and may therefore not be sufficiently comprehensive in reflecting the barriers and facilitators experienced by subgroups within the broader LGBTQA+ community. Additionally, although broad inclusion criteria are necessary to capture the full breadth of research conducted in this topic, included studies are likely to be heterogeneous and varied in terms of their methodology and population which may complicate data synthesis.

Nonetheless, it is anticipated that the findings from this review will provide the most comprehensive synthesis to date of the issues driving low help-seeking and treatment engagement in people across the psychosis spectrum who are LGBTQA+. This review will likely also identify gaps in the literature which may inform avenues for future research, and the factors identified in this review will be considered in subsequent research by the authors.

Additionally, findings will be relevant to healthcare providers that offer support to people with psychosis who may have intersecting LGBTQA+ identities as well as LGBTQA+ organisations which offer support to LGBTQA+ people who may be experiencing distressing psychotic experiences. These services are likely to benefit from an increased awareness of the factors which may improve or hinder accessibility for these subsets of their target populations. Therefore, results from this review may inform decision-making around the implementation of service-wide policy changes.

The findings of this review will be disseminated through the publication of an article in a peer-reviewed journal and presented at relevant conferences in Australia and/or internationally. Additionally, the completed review will form part of the lead author’s doctoral thesis.

Availability of data and materials

Not applicable for this protocol.

Abbreviations

• At risk mental state

Clinical high risk for psychosis

Joanna Briggs Institute

Lesbian, gay, and bisexual

Lesbian, gay, bisexual, transgender, queer or questioning, asexual or aromantic, and more

Population, Concept, Context

Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols

Ultra-high risk for psychosis

Institute for Health Metrics and Evaluation (IHME). Global Burden of Disease Study (GBD) Results. Seattle, WA: IHME, University of Washington. 2020. https://vizhub.healthdata.org/gbd-results/ . Accessed 26 May 2023

Moreno-Kustner B, Martin C, Pastor L. Prevalence of psychotic disorders and its association with methodological issues. A systematic review and meta-analyses. PLoS One. 2018;13(4):e0195687. https://doi.org/10.1371/journal.pone.0195687 .

Article   CAS   PubMed   PubMed Central   Google Scholar  

McGrath JJ, Saha S, Al-Hamzawi A, Alonso J, Bromet EJ, Bruffaerts R, et al. Psychotic experiences in the general population: a cross-national analysis based on 31 261 respondents from 18 countries. JAMA Psychiatry. 2015;72(7):697–705. https://doi.org/10.1001/jamapsychiatry.2015.0575 .

Article   PubMed   PubMed Central   Google Scholar  

Chakraborty A, McManus S, Brugha TS, Bebbington P, King M. Mental health of the non-heterosexual population of England. Br J Psychiatry. 2011;198(2):143–8. https://doi.org/10.1192/bjp.bp.110.082271 .

Article   PubMed   Google Scholar  

Gevonden M, Selten J, Myin-Germeys I, De Graaf R, Ten Have M, Van Dorsselaer S, et al. Sexual minority status and psychotic symptoms: findings from the Netherlands Mental Health Survey and Incidence Studies (NEMESIS). Psychol Med. 2014;44(2):421–33. https://doi.org/10.1017/S0033291713000718 .

Article   CAS   PubMed   Google Scholar  

Jacob L, Smith L, McDermott D, Haro JM, Stickley A, Koyanagi A. Relationship between sexual orientation and psychotic experiences in the general population in England. Psychol Med. 2021;51(1):138–46. https://doi.org/10.1017/S003329171900309X .

Welch S, Collings SCD, Howden-Chapman P. Lesbians in New Zealand: their mental health and satisfaction with mental health services. Aust N Z J Psychiatry. 2000;34(2):256–63. https://doi.org/10.1080/j.1440-1614.2000.00710.x .

Barr SM, Roberts D, Thakkar KN. Psychosis in transgender and gender non-conforming individuals: a review of the literature and a call for more research. Psychiatry Res. 2021;306:114272. https://doi.org/10.1016/j.psychres.2021.114272 .

Hanna B, Desai R, Parekh T, Guirguis E, Kumar G, Sachdeva R. Psychiatric disorders in the U.S. transgender population. Ann Epidemiol. 2019;39:1–7. https://doi.org/10.1016/j.annepidem.2019.09.009 .

Schneeberger AR, Dietl MF, Muenzenmaier KH, Huber CG, Lang UE. Stressful childhood experiences and health outcomes in sexual minority populations: a systematic review. Soc Psychiatry Psychiatr Epidemiol. 2014;49:1427–45. https://doi.org/10.1007/s00127-014-0854-8 .

Stanton KJ, Denietolis B, Goodwin BJ, Dvir Y. Childhood trauma and psychosis: an updated review. Child Adolesc Psychiatr Clin N Am. 2020;29(1):115–29. https://doi.org/10.1016/j.chc.2019.08.004 .

Varese F, Smeets F, Drukker M, Lieverse R, Lataster T, Viechtbauer W, et al. Childhood adversities increase the risk of psychosis: a meta-analysis of patient-control, prospective- and cross-sectional cohort studies. Schizophr Bull. 2012;38(4):661–71. https://doi.org/10.1093/schbul/sbs050 .

Mongelli F, Perrone D, Balducci J, Saccheti A, Ferrari S, Mattei G, et al. Minority stress and mental health among LGBT populations: an update on the evidence. Minerva Psichiatr. 2019;60(1):27–50. https://doi.org/10.23736/S0391-1772.18.01995-7 .

Article   Google Scholar  

Kidd SA, Howison M, Pilling M, Ross LE, McKenzie K. Severe mental illness in LGBT populations: a scoping review. Psychiatr Serv. 2016;67(7):779–83. https://doi.org/10.1176/appi.ps.201500209 .

Hatzenbuehler ML, Pachankis JE. Stigma and minority stress as social determinants of health among lesbian, gay, bisexual, and transgender youth. Pediatr Clin North Am. 2016;63(6):985–97. https://doi.org/10.1016/j.pcl.2016.07.003 .

Rüsch N, Heekeren K, Theodoridou A, Müller M, Corrigan PW, Mayer B, et al. Stigma as a stressor and transition to schizophrenia after one year among young people at risk of psychosis. Schizophr Res. 2015;166(1–3):43–8. https://doi.org/10.1016/j.schres.2015.05.027 .

Howes OD, Whitehurst T, Shatalina E, Townsend L, Onwordi EC, Mak TLA, et al. The clinical significance of duration of untreated psychosis: an umbrella review and random-effects meta-analysis. World Psychiatry. 2021;20(1):75–95. https://doi.org/10.1002/wps.20822 .

McGorry PD. Early intervention in psychosis: obvious, effective, overdue. J Nerv Ment Dis. 2015;203(5):310–308. https://doi.org/10.1097/NMD.0000000000000284 .

Peta JL. The Oxford Handbook of Sexual and Gender Minority Mental Health. Oxford (GB): Oxford University Press; 2020. Chapter 11, Schizophrenia spectrum and other psychotic disorders among sexual and gender minority populations; 125-134. Available from: The Oxford Handbook of Sexual and Gender Minority Mental Health - Google Books. Accessed 01 Jun 2023

Simeonov D, Steele LS, Anderson S, Ross LE. Perceived satisfaction with mental health services in the lesbian, gay, bisexual, transgender, and transsexual communities in Ontario, Canada: an internet-based survey. Can J Commun Ment Health. 2015;34(1):31–44. https://doi.org/10.7870/cjcmh-2014-037 .

Avery AM, Hellman RE, Sudderth LK. Satisfaction with mental health services among sexual minorities with major mental illness. Am J Public Health. 2001;91(6):990–1. https://doi.org/10.2105/AJPH.91.6.990 .

Plöderl M, Mestel R, Fartacek C. Differences by sexual orientation in treatment outcome and satisfaction with treatment among inpatients of a German psychiatric clinic. PLoS ONE. 2022;17(1):e0262928. https://doi.org/10.1371/journal.pone.0262928 .

Clement S, Schauman O, Graham T, Maggioni F, Evans-Lacko S, Bezborodovs N, et al. What is the impact of mental health-related stigma on help-seeking? A systematic review of quantitative and qualitative studies. Psychol Med. 2015;45(1):11–27. https://doi.org/10.1017/S0033291714000129 .

Pyle M, Morrison AP. “It’s just a very taboo and secretive kind of thing”: making sense of living with stigma and discrimination from accounts of people with psychosis. Psychosis. 2014;6(3):195–205. https://doi.org/10.1080/17522439.2013.834458 .

Wood L, Burke E, Byrne R, Pyle M, Chapman N, Morrison AP. Stigma in psychosis: a thematic synthesis of current qualitative evidence. Psychosis. 2015;7(2):152–65. https://doi.org/10.1080/17522439.2014.926561 .

Rüsch N, Heekeren K, Theodoridou A, Dvorsky D, Müller M, Paust T, et al. Attitudes towards help-seeking and stigma among young people at risk for psychosis. Psychiatry Res. 2013;210(3):1313–5. https://doi.org/10.1016/j.psychres.2013.08.028 .

Gronholm PC, Thornicroft G, Laurens KR, Evans-Lacko S. Mental health-related stigma and pathways to care for people at risk of psychotic disorders or experiencing first-episode psychosis: a systematic review. Psychol Med. 2017;47(11):1867–79. https://doi.org/10.1017/S0033291717000344 .

Rossman K, Salamanca P, Macapagal K. A qualitative study examining young adults’ experiences of disclosure and nondisclosure of LGBTQ identity to health care providers. J Homosex. 2017;64(10):1390–410. https://doi.org/10.1080/00918369.2017.1321379 .

Rees SM, Crowe M, Harris S. The lesbian, gay, bisexual and transgender communities’ mental health care needs and experiences of mental health services: an integrative review of qualitative studies. J Psychiat Mental Health Nurs. 2020;28(4):578–89. https://doi.org/10.1111/jpm.12720 .

Shipherd JC, Green KE, Abramovitz A. Transgender clients: identifying and minimizing barriers to mental health treatment. J Gay Lesbian Ment Health. 2010;14(2):94–108. https://doi.org/10.1080/19359701003622875 .

Hoening J, Kenna JC. The nosological position of transsexualism. Arch Sex Behav. 1974;3:273–87. https://doi.org/10.1007/BF01541490 .

Eliasson ET, McNamee L, Swanson L, Lawrie SM, Schwannauer M. Unpacking stigma: meta-analyses of correlates and moderators of personal stigma in psychosis. Clin Psychol Rev. 2021;89:102077. https://doi.org/10.1016/j.cpr.2021.102077 .

Lally J, MacCabe JH. Antipsychotic medication in schizophrenia: a review. Br Med Bull. 2015;114(1):169–79. https://doi.org/10.1093/bmb/ldv017 .

De Lange J, Baams L, van Bergen D, Bos HMW, Bosker RJ. Minority stress and suicidal ideation and suicide attempts among LGBT adolescents and young adults: a meta-analysis. LGBT Health. 2022;9(4):222–37. https://doi.org/10.1089/lgbt.2021.0106 .

Skerrett DM, Kolves K, De Leo D. Are LGBT populations at higher risk for suicidal behaviours in Australia? Research findings and implications. J Homosex. 2015;62(7):883–901. https://doi.org/10.1080/00918369.2014.1003009 .

Strauss P, Cook A, Winter S, Watson V, Toussaint DW, Lin A. Trans Pathways: the mental health experiences and care pathways of trans young people: summary of results. Perth, Australia: Telethon Kids Institute. 2017. trans-pathways-report.pdf (telethonkids.org.au). Accessed 16 June 2023

DeLuca JS, Novacek DM, Adery LH, Herrera SN, Landa Y, Corcoran CM, et al. Equity in mental health services for youth at clinical high risk for psychosis: considering marginalized identities and stressors. Evid Based Pract Child Adolesc Ment Health. 2022;7(2):176–97. https://doi.org/10.1080/23794925.2022.2042874 .

Day JC, Bentall RP, Roberts C, Randall F, Rogers A, Cattell D, et al. Attitudes toward antipsychotic medication: the impact of clinical variables and relationships with health professionals. Arch Gen Psychiatry. 2005;62(7):717–24. https://doi.org/10.1001/archpsyc.62.7.717 .

Coyne CA, Poquiz JL, Janssen A, Chen D. Evidence-based psychological practice for transgender and non-binary youth: defining the need, framework for treatment adaptation, and future directions. Evid Based Pract Child Adolesc Ment Health. 2020;5(3):340–53. https://doi.org/10.1080/23794925.2020.1765433 .

Baker KE, Wilson LM, Sharma R, Dukhanin V, McArthur K, Robinson KA. Hormone therapy, mental health, and quality of life among transgender people: a systematic review. J Endocr Soc. 2021;5(4):bvab011. https://doi.org/10.1210/jendso/bvab011 .

Hughto JMW, Reisner SL. A systematic review of the effects of hormone therapy on psychological functioning and quality of life in transgender individuals. Transgend Health. 2016;1(1):21–31. https://doi.org/10.1089/trgh.2015.0008 .

Wernick JA, Busa S, Matouk K, Nicholson J, Janssen A. A systematic review of the psychological benefits of gender-affirming surgery. Urol Clin North Am. 2019;46(4):475–86. https://doi.org/10.1016/j.ucl.2019.07.002 .

Bränström R, Packanhkis JE. Reductions in mental health treatment utilization among transgender individuals after gender-affirming surgeries: a total population study. Am J Psychiatry. 2019;177(8):727–34. https://doi.org/10.1176/appi.ajp.2019.19010080 .

Meijer JH, Eeckhout GM, van Vlerken RHT, de Vries ALC. Gender dysphoria and co-existing psychosis: review and four case examples of successful gender affirmative treatment. LGBT Health. 2017;4(2):106–14. https://doi.org/10.1089/lgbt.2016.0133 .

Munn Z, Peters MDJ, Stern C, Tufanaru C, McArthur A, Aromataris E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med Res Methodol. 2018;18(1):143. https://doi.org/10.1186/s12874-018-0611-x .

Munn Z, Pollock D, Khalil H, Alexander L, McInerney P, Godfrey CM, et al. What are scoping reviews? Providing a formal definition of scoping reviews as a type of evidence synthesis. JBI Evid Synth. 2022;20(4):950–2. https://doi.org/10.11124/JBIES-21-00483 .

Peters MDJ, Marnie C, Colquhoun H, Garritty CM, Hempel S, Horsley T, et al. Scoping reviews: reinforcing and advancing the methodology and application. Syst Rev. 2021;10(1):263. https://doi.org/10.1186/s13643-021-01821-3 .

Aromataris E, Lockwood C, Porritt K, Pilla B, Jordan Z, editors. JBI Manual for Evidence Synthesis. JBI; 2024. https://synthesismanual.jbi.global . https://doi.org/10.46658/JBIMES-24-01

Shamseer L, Moher D, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ. 2015;350: g7647. https://doi.org/10.1136/bmj.g7647 .

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Syst Rev. 2021;372:n71. https://doi.org/10.1136/bmj.n71 .

Barker TH, Stone JC, Sears K, Klugar M, Leonardi-Bee J, Tufanaru C, et al. Revising the JBI quantitative critical appraisal tools to improve their applicability: an overview of methods and the development process. JBI Evid Synth. 2023;21(3):478–93. https://doi.org/10.11124/JBIES-22-00125 .

Popay J, Roberts H, Sowden A, Petticrew M, Arai L, Rodgers M, et al. Guidance on the conduct of narrative synthesis in systematic reviews: a product from the ESRC methods programme. Lancaster University; 2006. https://doi.org/10.13140/2.1.1018.4643

Bach-Mortensen AM, Verboom B. Barriers and facilitators systematic reviews in health: a methodological review and recommendations for reviewers. Res Synth Methods. 2020;11(6):743–59. https://doi.org/10.1002/jrsm.1447 .

Download references

Acknowledgements

The authors would like to acknowledge the support of Ms Olivia Larobina, Scholarly Services Librarian (STEMM) at Deakin University, in the development of the search strategy.

CCG is funded by a Deakin University Postgraduate Research (DUPR) Scholarship. ZW is funded by a University of Western Australia Research Training Program (RTP) Scholarship. AL is supported by a National Health and Medical Research Council (NHMRC) Emerging Leaders Fellowship (2010063). LJW is supported by a NHMRC Emerging Leaders Fellowship (1174060). ARY is supported by a NHMRC Principal Research Fellowship (1136829). The funding providers had no role in the design and conduct of the study, or in the preparation, review, or approval of this manuscript.

Author information

Authors and affiliations.

Institute for Mental and Physical Health and Clinical Translation, Deakin University, Geelong, VIC, 3220, Australia

Cláudia C. Gonçalves, Shae E. Quirk, Peter M. Haddad, Lana J. Williams & Alison R. Yung

Telethon Kids Institute, University of Western Australia, Perth, WA, 6009, Australia

University Hospital Geelong, Barwon Health, Geelong, VIC, 3220, Australia

Peter M. Haddad

School of Population and Global Health, University of Western Australia, Perth, WA, 6009, Australia

Ashleigh Lin

School of Health Sciences, University of Manchester, Manchester, M13 9PL, UK

Alison R. Yung

You can also search for this author in PubMed   Google Scholar

Contributions

CCG is the guarantor. CCG conceptualised the review, developed the study design, and drafted the manuscript. CCG, ZW, and SQ collaborated with OL (Scholarly Services Librarian) to develop the search strategy. All authors critically reviewed the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Cláudia C. Gonçalves .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

In the last 3 years, PMH has received honoraria for lecturing from Janssen, NewBridge Pharmaceuticals, and Otsuka and royalties from edited textbooks (Cambridge University Press, Oxford University Press).

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1. prisma-p 2015 checklist. completed prisma-p checklist for this systematic review protocol., 13643_2024_2566_moesm2_esm.docx.

Additional file 2. Search Strategy. Detailed search strategy for this systematic review, including search terms and relevant controlled vocabulary terms for each included database.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Gonçalves, C.C., Waters, Z., Quirk, S.E. et al. Barriers and facilitators to mental health treatment access and engagement for LGBTQA+ people with psychosis: a scoping review protocol. Syst Rev 13 , 143 (2024). https://doi.org/10.1186/s13643-024-02566-5

Download citation

Received : 04 July 2023

Accepted : 17 May 2024

Published : 30 May 2024

DOI : https://doi.org/10.1186/s13643-024-02566-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Treatment access
• Treatment engagement
• Facilitators
• Clinical high risk
• Ultra-high risk
• Psychotic experiences

Systematic Reviews

ISSN: 2046-4053

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

• Systematic Review
• Open access
• Published: 30 May 2024

Patient experiences: a qualitative systematic review of chemotherapy adherence

• Amineh Rashidi 1 ,
• Susma Thapa 1 ,
• Wasana Sandamali Kahawaththa Palliya Guruge 1 &
• Shubhpreet Kaur 1  

BMC Cancer volume  24 , Article number:  658 ( 2024 ) Cite this article

152 Accesses

Metrics details

Adherence to chemotherapy treatment is recognized as a crucial health concern, especially in managing cancer patients. Chemotherapy presents challenges for patients, as it can lead to potential side effects that may adversely affect their mobility and overall function. Patients may sometimes neglect to communicate these side effects to health professionals, which can impact treatment management and leave their unresolved needs unaddressed. However, there is limited understanding of how patients’ experiences contribute to improving adherence to chemotherapy treatment and the provision of appropriate support. Therefore, gaining insights into patients’ experiences is crucial for enhancing the accompaniment and support provided during chemotherapy.

This review synthesizes qualitative literature on chemotherapy adherence within the context of patients’ experiences. Data were collected from Medline, Web of Science, CINAHL, PsychINFO, Embase, Scopus, and the Cochrane Library, systematically searched from 2006 to 2023. Keywords and MeSH terms were utilized to identify relevant research published in English. Thirteen articles were included in this review. Five key themes were synthesized from the findings, including positive outlook, receiving support, side effects, concerns about efficacy, and unmet information needs. The review underscores the importance for healthcare providers, particularly nurses, to focus on providing comprehensive information about chemotherapy treatment to patients. Adopting recommended strategies may assist patients in clinical practice settings in enhancing adherence to chemotherapy treatment and improving health outcomes for individuals living with cancer.

Peer Review reports

Introduction

Cancer can affect anyone and is recognized as a chronic disease characterized by abnormal cell multiplication in the body [ 1 ]. While cancer is prevalent worldwide, approximately 70% of cancer-related deaths occur in low- to middle-income nations [ 1 ]. Disparities in cancer outcomes are primarily attributed to variations in the accessibility of comprehensive diagnosis and treatment among countries [ 1 , 2 ]. Cancer treatment comes in various forms; however, chemotherapy is the most widely used approach [ 3 ]. Patients undergoing chemotherapy experience both disease-related and treatment-related adverse effects, significantly impacting their quality of life [ 4 ]. Despite these challenges, many cancer patients adhere to treatment in the hope of survival [ 5 ]. However, some studies have shown that concerns about treatment efficacy may hinder treatment adherence [ 6 ]. Adherence is defined as “the extent to which a person’s behaviour aligns with the recommendations of healthcare providers“ [ 7 ]. Additionally, treatment adherence is influenced by the information provided by healthcare professionals following a cancer diagnosis [ 8 ]. Patient experiences suggest that the decision to adhere to treatment is often influenced by personal factors, with family support playing a crucial role [ 8 ]. Furthermore, providing adequate information about chemotherapy, including its benefits and consequences, can help individuals living with cancer gain a better understanding of the advantages associated with adhering to chemotherapy treatment [ 9 ].

Recognizing the importance of adhering to chemotherapy treatment and understanding the impact of individual experiences of chemotherapy adherence would aid in identifying determinants of adherence and non-adherence that are modifiable through effective interventions [ 10 ]. Recently, systematic reviews have focused on experiences and adherence in breast cancer [ 11 ], self-management of chemotherapy in cancer patients [ 12 ], and the influence of medication side effects on adherence [ 13 ]. However, these reviews were narrow in scope, and to date, no review has integrated the findings of qualitative studies designed to explore both positive and negative experiences regarding chemotherapy treatment adherence. This review aims to synthesize the qualitative literature on chemotherapy adherence within the context of patients’ experiences.

This review was conducted in accordance with the Joanna Briggs Institute [ 14 ] guidelines for systemic review involving meta-aggregation. This review was registered in PROSPERO (CRD42021270459).

Search methods

The searches for peer reviewed publications in English from January 2006-September 2023 were conducted by using keywords, medical subject headings (MeSH) terms and Boolean operators ‘AND’ and ‘OR’, which are presented in the table in Appendix 1 . The searches were performed in a systematic manner in core databases such including Embase, Medline, PsycINFO, CINAHL, Web of Science, Cochrane Library, Scopus and the Joanna Briggs Institute (JBI). The search strategy was developed from keywords and medical subject headings (MeSH) terms. Librarian’s support and advice were sought in forming of the search strategies.

Study selection and inclusion criteria

The systematic search was conducted on each database and all articles were exported to Endnote and duplicates records were removed. Then, title and abstract of the full text was screened by two independent reviewers against the inclusion criteria. For this review, populations were patients aged 18 and over with cancer, the phenomenon of interest was experiences on chemotherapy adherence and context was considered as hospitals, communities, rehabilitation centres, outpatient clinics, and residential aged care. All peer-reviewed qualitative study design were also considered for inclusion. Studies included in this review were classified as primary research, published in English since 2006, some intervention implemented to improve adherence to treatment. This review excluded any studies that related to with cancer and mental health condition, animal studies and grey literature.

Quality appraisal and data extraction

The JBI Qualitative Assessment and Review Instrument for qualitative studies was used to assess the methodological quality of the included studies, which was conducted by the primary and second reviewers independently. There was no disagreement between the reviews. The qualitative data on objectives, study population, context, study methods, and the phenomena of interest and findings form the included studies were extracted.

Data synthesis

The meta-aggregation approach was used to combine the results with similar meaning. The primary and secondary reviewers created categories based on the meanings and concept. These categories were supported by direct quotations from participants. The findings were assess based on three levels of evidence, including unequivocal, credible, and unsupported [ 15 , 16 ]. Findings with no quotation were not considered for synthesis in this review. The categories and findings were also discussed by the third and fourth reviewers until a consensus was reached. The review was approved by the Edith Cowan University Human Research Ethics Committee (2021–02896).

Study inclusion

A total of 4145 records were identified through a systematic search. Duplicates ( n  = 647) were excluded. Two independent reviewers conducted screening process. The remaining articles ( n  = 3498) were examined for title and abstract screening. Then, the full text screening conducted, yielded 13 articles to be included in the final synthesis see Appendix 2 .

Methodological quality of included studies

All included qualitative studies scored between 7 and 9, which is displayed in Appendix 3 . The congruity between the research methodology and the research question or objectives, followed by applying appropriate data collection and data analysis were observed in all included studies. Only one study [ 17 ] indicated the researcher’s statement regarding cultural or theoretical perspectives. Three studies [ 18 , 19 , 20 ] identified the influence of the researcher on the research and vice-versa.

Characteristics of included studies

Most of studies conducted semi-structured and in-depth interviews, one study used narrative stories [ 19 ], one study used focus group discussion [ 21 ], and one study combined focus group and interview [ 22 ] to collect data. All studies conducted outpatient’s clinic, community, or hospital settings [ 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 ]. The study characteristics presented in Appendix 4 .

Review findings

Eighteen findings were extracted and synthesised into five categories: positive outlook, support, side effects, concern about efficacy and unmet information needs.

Positive outlook

Five studies discussed the link between positivity and hope and chemotherapy adherence [ 19 , 20 , 23 , 27 , 28 ]. Five studies commented that feeling positive and avoid the negativity and worry could encourage people to adhere in their mindset chemotherapy: “ I think the main thing for me was just keeping a positive attitude and not worrying, not letting myself worry about it ” [ 20 ]. Participants also considered the positive thoughts as a coping mechanism, that would help them to adhere and complete chemotherapy: “ I’m just real positive on how everything is going. I’m confident in the chemo, and I’m hoping to get out of her soon ” [ 23 ]. Viewing chemotherapy as part of their treatment regimen and having awareness of negative consequences of non-adherence to chemotherapy encouraged them to adhere chemotherapy: “ If I do not take medicine, I do not think I will be able to live ” [ 28 ]. Adhering chemotherapy was described as a survivor tool which helped people to control cancer-related symptoms: “ it is what is going to restore me. If it wasn’t this treatment, maybe I wasn’t here talking to you. So, I have to focus in what he is going to give me, life !” [ 27 ]. Similarly, people accepted the medical facts and prevent their life from worsening; “ without the treatment, it goes the wrong way. It is hard, but I have accepted it from the beginning, yes. This is how it is. I cannot do anything about it. Just have to accept it ” [ 19 ].

Finding from six studies contributed to this category [ 20 , 21 , 23 , 24 , 25 , 29 ]. Providing support from families and friends most important to the people. Receiving support from family members enhanced a sense responsibility towards their families, as they believed to survive for their family even if suffered: “ yes, I just thought that if something comes back again and I say no, then I have to look my family and friends in the eye and say I could have prevented it, perhaps. Now, if something comes back again, I can say I did everything I could. Cancer is bad enough without someone saying: It’s your own fault!!” [ 29 ]. Also, emotional support from family was described as important in helping and meeting their needs, and through facilitation helped people to adhere chemotherapy: “ people who genuinely mean the support that they’re giving […] just the pure joy on my daughter’s face for helping me. she was there day and night for me if I needed it, and that I think is the main thing not to have someone begrudgingly looking after you ” [ 20 ]. Another study discussed the role family, friends and social media as the best source of support during their treatment to adhere and continue “ I have tons of friends on Facebook, believe it or not, and it’s amazing how many people are supportive in that way, you know, just sending get-well wishes. I can’t imagine going through this like 10 years ago whenever stuff like that wasn’t around ” [ 23 ]. Receiving support from social workers was particularly helpful during chemotherapy in encouraging adherence to the chemotherapy: “ the social worker told me that love is courage. That was a huge encouragement, and I began to encourage myself ” [ 25 ].

Side effects

Findings from five studies informed this category [ 17 , 21 , 22 , 25 , 26 ]. Physical side effects were described by some as the most unpleasure experience: “ the side effects were very uncomfortable. I felt pain, fatigue, nausea, and dizziness that limited my daily activities. Sometimes, I was thinking about not keeping to my chemotherapy schedule due to those side effect ” [ 17 ]. The impact of side effects affected peoples’ ability to maintain their independence and self-care: “ I couldn’t walk because I didn’t have the energy, but I wouldn’t have dared to go out because the diarrhoea was so bad. Sometimes I couldn’t even get to the toilet; that’s very embarrassing because you feel like you’re a baby ” [ 26 ]. Some perceived that this resulted in being unable to perform independently: “ I was incredibly weak and then you still have to do things and you can’t manage it ” [ 22 ]. These side effect also decreased their quality of life “ I felt nauseated whenever I smelled food. I simply had no appetite when food was placed in front of me. I lost my sense of taste. Food had no taste anymore ” [ 25 ]. Although, the side effects impacted on patients´ leisure and free-time activities, they continued to undertake treatment: “ I had to give up doing the things I liked the most, such as going for walks or going to the beach. Routines, daily life in general were affected ” [ 21 ].

Concern about efficacy

Findings form four studies informed this category [ 17 , 18 , 24 , 28 ]. Although being concerned about the efficacy of the chemotherapy and whether or not chemotherapy treatment would be successful, one participant who undertook treatment described: “the efficacy is not so great. It is said to expect about 10% improvement, but I assume that it declines over time ” [ 28 ]. People were worried that such treatment could not cure their cancer and that their body suffered more due to the disease: “ I was really worried about my treatment effectiveness, and I will die shortly ” [ 17 ]. There were doubts expressed about remaining the cancer in the body after chemotherapy: “ there’s always sort of hidden worries in there that whilst they’re not actually taking the tumour away, then you’re wondering whether it’s getting bigger or what’s happening to it, whether it’s spreading or whatever, you know ” [ 24 ]. Uncertainty around the outcome of such treatment, or whether recovering from cancer or not was described as: “it makes you feel confused. You don’t know whether you are going to get better or else whether the illness is going to drag along further” [ 18 ].

Unmet information needs

Five studies contributed to this category [ 17 , 21 , 22 , 23 , 26 ]. The need for adequate information to assimilate information and provide more clarity when discussing complex information were described. Providing information from clinicians was described as minimal: “they explain everything to you and show you the statistics, then you’re supposed to take it all on-board. You could probably go a little bit slower with the different kinds of chemo and grappling with these statistics” [ 26 ]. People also used the internet search to gain information about their cancer or treatments, “I’ve done it (consult google), but I stopped right away because there’s so much information and you don’t know whether it’s true or not ” [ 21 ]. The need to receive from their clinicians to obtain clearer information was described as” I look a lot of stuff up online because it is not explained to me by the team here at the hospital ” [ 23 ]. Feeling overwhelmed with the volume of information could inhibit people to gain a better understanding of chemotherapy treatment and its relevant information: “ you don’t absorb everything that’s being said and an awful lot of information is given to you ” [ 22 ]. People stated that the need to know more information about their cancer, as they were never dared to ask from their clinicians: “ I am a low educated person and come from a rural area; I just follow the doctor’s advice for my health, and I do not dare to ask anything” [ 17 ].

The purpose of this review was to explore patient’s experiences about the chemotherapy adherence. After finalizing the searches, thirteen papers were included in this review that met the inclusion criteria.

The findings of the present review suggest that social support is a crucial element in people’s positive experiences of adhering to chemotherapy. Such support can lead to positive outcomes by providing consistent and timely assistance from family members or healthcare professionals, who play vital roles in maintaining chemotherapy adherence [ 30 ]. Consistent with our study, previous research has highlighted the significant role of family members in offering emotional and physical support, which helps individuals cope better with chemotherapy treatment [ 31 , 32 ]. However, while receiving support from family members reinforces individuals’ sense of responsibility in managing their treatment and their family, it also instils a desire to survive cancer and undergo chemotherapy. One study found that assuming self-responsibility empowers patients undergoing chemotherapy, as they feel a sense of control over their therapy and are less dependent on family members or healthcare professionals [ 33 ]. A qualitative systematic review reported that support from family members enables patients to become more proactive and effective in adhering to their treatment plan [ 34 ]. This review highlights the importance of maintaining a positive outlook and rational beliefs as essential components of chemotherapy adherence. Positive thinking helps individuals recognize their role in chemotherapy treatment and cope more effectively with their illness by accepting it as part of their treatment regimen and viewing it as a tool for survival. This finding is supported by previous studies indicating that positivity and positive affirmations play critical roles in helping individuals adapt to their reality and construct attitudes conducive to chemotherapy adherence [ 35 , 36 ]. Similarly, maintaining a positive mindset can foster more favourable thoughts regarding chemotherapy adherence, ultimately enhancing adherence and overall well-being [ 37 ].

This review identified side effects as a significant negative aspect of the chemotherapy experience, with individuals expressing concerns about how these side effects affected their ability to perform personal self-care tasks and maintain independent living in their daily lives. Previous studies have shown that participants with a history of chemotherapy drug side effects were less likely to adhere to their treatment regimen due to worsening symptoms, which increased the burden of medication side effects [ 38 , 39 ]. For instance, cancer patients who experienced minimal side effects from chemotherapy were at least 3.5 times more likely to adhere to their treatment plan compared to those who experienced side effects [ 40 ]. Despite experiencing side effects, patients were generally willing to accept and adhere to their treatment program, although one study in this review indicated that side effects made some patients unable to maintain treatment adherence. Side effects also decreased quality of life and imposed restrictions on lifestyle, as seen in another study where adverse effects limited individuals in fulfilling daily commitments and returning to normal levels of functioning [ 41 ]. Additionally, unmet needs regarding information on patients’ needs and expectations were common. Healthcare professionals were considered the most important source of information, followed by consultation with the internet. Providing information from healthcare professionals, particularly nurses, can support patients effectively and reinforce treatment adherence [ 42 , 43 ]. Chemotherapy patients often preferred to base their decisions on the recommendations of their care providers and required adequate information retention. Related studies have highlighted that unmet needs among cancer patients are known factors associated with chemotherapy adherence, emphasizing the importance of providing precise information and delivering it by healthcare professionals to improve adherence [ 44 , 45 ]. Doubts about the efficacy of chemotherapy treatment, as the disease may remain latent, were considered negative experiences. Despite these doubts, patients continued their treatment, echoing findings from a study where doubts regarding efficacy were identified as a main concern for chemotherapy adherence. Further research is needed to understand how doubts about treatment efficacy can still encourage patients to adhere to chemotherapy treatment.

Strengths and limitation

The strength of this review lies in its comprehensive search strategy across databases to select appropriate articles. Additionally, the use of JBI guidelines provided a comprehensive and rigorous methodological approach in conducting this review. However, the exclusion of non-English studies, quantitative studies, and studies involving adolescents and children may limit the generalizability of the findings. Furthermore, this review focuses solely on chemotherapy treatment and does not encompass other types of cancer treatment.

Conclusion and practical implications

Based on the discussion of the findings, it is evident that maintaining a positive mentality and receiving social support can enhance chemotherapy adherence. Conversely, experiencing treatment side effects, concerns about efficacy, and unmet information needs may lead to lower adherence. These findings present an opportunity for healthcare professionals, particularly nurses, to develop standardized approaches aimed at facilitating chemotherapy treatment adherence, with a focus on providing comprehensive information. By assessing patients’ needs, healthcare professionals can tailor approaches to promote chemotherapy adherence and improve the survival rates of people living with cancer. Raising awareness and providing education about cancer and chemotherapy treatment can enhance patients’ understanding of the disease and its treatment options. Utilizing videos and reading materials in outpatient clinics and pharmacy settings can broaden the reach of educational efforts. Policy makers and healthcare providers can collaborate to develop sustainable patient education models to optimize patient outcomes in the context of cancer care. A deeper understanding of individual processes related to chemotherapy adherence is necessary to plan the implementation of interventions effectively. Further research examining the experiences of both adherent and non-adherent patients is essential to gain a comprehensive understanding of this topic.

Data availability

The datasets used and/or analysed during the current study available from the corresponding author on reasonable request. on our submission system as well.

World Health Organization. Cancer 2021 [ https://www.who.int/news-room/fact-sheets/detail/cancer .

Klapheke A, Yap SA, Pan K, Cress RDDHSDCA. Sociodemographic disparities in chemotherapy treatment and impact on survival among patients with metastatic bladder cancer. Urologic Oncology: Seminars Original Investigations. 2018;36(6):19–308.

Article   Google Scholar  

Moth EB, Kiely BE, Naganathan V, Martin A, Blinman P. How do oncologists make decisions about chemotherapy for their older patients with cancer? A survey of Australian oncologists. Support Care Cancer. 2018;26(2):451–60.

Article   CAS   PubMed   Google Scholar  

Khamboon T, Pakanta I. Intervention for symptom cluster management of fatigue, loss of appetite, and anxiety among patients with lung cancer undergoing chemotherapy. Asia-Pacific J Oncol Nurs. 2021;8(3):267–75.

Garcia ACM, Camargos Junior JB, Sarto KK, Silva Marcelo CAd, Paiva EMC, Nogueira DA, Mills J. Quality of life, self-compassion and mindfulness in cancer patients undergoing chemotherapy: a cross-sectional study. Eur J Oncol Nurs. 2021;51:N.PAG-N.PAG.

Horne R, Chapman SCE, Parham R, Freemantle N, Forbes A, Cooper V. Understanding patients’ adherence-related beliefs about Medicines prescribed for long-term conditions: a Meta-Analytic Review of the necessity-concerns Framework. PLoS ONE. 2013;8(12):e80633.

Article   PubMed   PubMed Central   Google Scholar  

WHO. Adherence to long-term therapies: evidence for action. Geneva, Switzerland: World Health Organisation; 2003.

Google Scholar  

Warby A, Dhillon HM, Kao S, Vardy JL. A survey of patient and caregiver experience with malignant pleural mesothelioma. Support Care Cancer. 2019;27(12):4675–86.

Article   PubMed   Google Scholar  

Arunachalam SS, Shetty AP, Panniyadi N, Meena C, Kumari J, Rani B, et al. Study on knowledge of chemotherapy’s adverse effects and their self-care ability to manage - the cancer survivors impact. Clin Epidemiol Global Health. 2021;11:100765.

Article   CAS   Google Scholar  

Nizet P, Touchefeu Y, Pecout S, Cauchin E, Beaudouin E, Mayol S, et al. Exploring the factors influencing adherence to oral anticancer drugs in patients with digestive cancer: a qualitative study. Support Care Cancer. 2022;30(3):2591–604.

Clancy C, Lynch J, Oconnor P, Dowling M. Breast cancer patients’ experiences of adherence and persistence to oral endocrine therapy: a qualitative evidence synthesis. Eur J Oncol Nurs. 2020;44.

Magalhães B, Fernandes C, Lima L, Martinez-Galiano JM, Santos C. Cancer patients’ experiences on self-management of chemotherapy treatment-related symptoms: A systematic review and thematic synthesis. Eur J Oncol Nurs. 2020;49.

Peddie N, Agnew S, Crawford M, Dixon D, MacPherson I, Fleming L. The impact of medication side effects on adherence and persistence to hormone therapy in breast cancer survivors: a qualitative systematic review and thematic synthesis. Breast. 2021;58:147–59.

Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ: Br Med J. 2009;339(7716):332–6.

Joanna Briggs Institute. The Joanna Briggs Institute critical appraisal tools for use in JBI systematic reviews. Checklist for qualitative research. 2017.

Zachary M, Kylie P, Craig L, Edoardo A, Alan P. Establishing confidence in the output of qualitative research synthesis: the ConQual approach. BMC Med Res Methodol [Internet]. 2014;14(1):108.

Iskandarsyah A, de Klerk C, Suardi DR, Soemitro MP, Sadarjoen SS, Passchier J. Psychosocial and cultural reasons for Delay in seeking help and Nonadherence to treatment in Indonesian women with breast Cancer: a qualitative study. Health Psychol. 2014;33(3):214–21.

Chircop D, Scerri J. The lived experience of patients with Non-hodgkin’s lymphoma undergoing chemotherapy. Eur J Oncol Nurs. 2018;35:117–21.

Kvåle K, Synnes O. Living with life-prolonging chemotherapy—control and meaning‐making in the tension between life and death. Eur J Cancer Care. 2018;27(1):1.

Staneva AA, Beesley VL, Niranjan N, Gibson AF, Rowlands I, Webb PM. I wasn’t gonna let it stop me: exploring women’s experiences of getting through chemotherapy for ovarian cancer. Cancer Nurs. 2019;42(2):E31–8.

Talens A, Guilabert M, Lumbreras B, Aznar MT, López-Pintor E. Medication Experience and Adherence to Oral Chemotherapy: A Qualitative Study of Patients’ and Health Professionals’ Perspectives. Int J Environ Res Public Health. 2021;18(8).

Dumas L, Lidington E, Appadu L, Jupp P, Husson O, Banerjee S, et al. Exploring older women’s attitudes to and experience of treatment for advanced ovarian cancer: a qualitative phenomenological study. Cancers. 2021;13(6):1207.

Albrecht TA, Keim-Malpass J, Boyiadzis M, Rosenzweig M. Psychosocial experiences of young adults diagnosed with acute leukemia during hospitalization for induction chemotherapy treatment. J Hospice Palliat Nurs. 2019;21(2):167–73.

Beaver K, Williamson S, Briggs J. Exploring patient experiences of neo-adjuvant chemotherapy for breast cancer. Eur J Oncol Nurs. 2016;20:77–86.

Chou J-F, Lu YY. Intraperitoneal chemotherapy: the lived experiences of Taiwanese patients with ovarian cancer. Clin J Oncol Nurs. 2019;23(6):E100–6.

Farrell C, Heaven C. Understanding the impact of chemotherapy on dignity for older people and their partners. Eur J Oncol Nurs. 2018;36:82–8.

Wakiuchi J, Silva Marcon S, de Oliveira DC, Aparecida Sales C. Rebuilding subjectivity from the experience of cancer and its treatment. Revista Brasileira De Enfermagem. 2019;72(1):125–33.

Yagasaki K, Komatsu H, Takahashi T. Inner conflict in patients receiving oral anticancer agents: a qualitative study. BMJ Open [Internet]. 2015; 5(4).

Gassmann C, Kolbe N, Brenner A. Experiences and coping strategies of oncology patients undergoing oral chemotherapy: first steps of a grounded theory study. Eur J Oncol Nurs. 2016;23:106–14.

Tang GX, Yan PP, Yan CL, Fu B, Zhu SJ, Zhou LQ, et al. Determinants of suicidal ideation in gynecological cancer patients. Psycho-oncology. 2016;25(1):97–103.

Oven Ustaalioglu B, Acar E, Caliskan M. The predictive factors for perceived social support among cancer patients and caregiver burden of their family caregivers in Turkish population. Int J Psychiatry Clin Pract. 2018;22(1):63–9.

Levkovich I, Cohen M, Karkabi K. The experience of fatigue in breast Cancer patients 1–12 Month Post-chemotherapy: a qualitative study. Behav Med. 2019;45(1):7–18.

Simchowitz B, Shiman L, Spencer J, Brouillard D, Gross A, Connor M, Weingart SN. Perceptions and experiences of patients receiving oral chemotherapy. Clin J Oncol Nurs. 2010;14(4):447–53.

Rashidi A, Kaistha P, Whitehead L, Robinson S. Factors that influence adherence to treatment plans amongst people living with cardiovascular disease: a review of published qualitative research studies. Int J Nurs Stud 2020;110(103727).

Aydogan U, Doganer YC, Komurcu S, Ozturk B, Ozet A, Saglam K. Coping attitudes of cancer patients and their caregivers and quality of life of caregivers. Indian J Palliat Care. 2016;22(2):150–6.

Langford DJ, Morgan S, Cooper B, Paul S, Kober K, Wright F, et al. Association of personality profiles with coping and adjustment to cancer among patients undergoing chemotherapy. Psycho-oncology. 2020;29(6):1060–7.

Jamie MJ, Pensak NA, Sporn NJ, MacDonald JJ, Lennes IT, Safren SA et al. Treatment satisfaction and adherence to oral chemotherapy in patients with Cancer. J Oncol Pract. 2017;13(2).

Tsai Y-F, Huang W-C, Cho S-F, Hsiao H-H, Liu Y-C, Lin S-F, et al. Side effects and medication adherence of tyrosine kinase inhibitors for patients with chronic myeloid leukemia in Taiwan. Medicine. 2018;97(26):415.

D S, M P, G R, S H. Importance of medication adherence and factors affecting it. IP Int J Compr Adv Pharmacolog. 2020;3(2):69–77.

Bekalu YE, Wudu MA, Gashu AW. Adherence to Chemotherapy and Associated factors among patients with Cancer in Amhara Region, Northeastern Ethiopia, 2022. A cross-sectional study. Cancer Control. 2023;30.

Hsu H-C, Liou W-S, Chiang A-J, Tsai S-Y, Jeang S-R, Wu S-L, et al. Longitudinal perceptions of the side effects of chemotherapy in patients with gynecological cancer. Support Care Cancer. 2017;25(11):3457–64.

Gow K, Rashidi A, Whithead L. Factors influencing medication adherence among adults living with diabetes and comorbidities: a qualitative systematic review. Curr Diab Rep. 2023:1–7.

Rashidi A, Whitehead L, Kaistha P. Nurses’ perceptions of factors influencing treatment engagement among patients with cardiovascular diseases: a systematic review. BMC Nurs. 2021;20(1):251.

Zebrack BJ, Block R, Hayes-Lattin B, Embry L, Aguilar C, Meeske KA, et al. Psychosocial service use and unmet need among recently diagnosed adolescent and young adult cancer patients. Cancer. 2013;119(1):201–14.

Timmers L, Boons CCLM, van den Verbrugghe M, Van Hecke A, Hugtenburg JG. Supporting adherence to oral anticancer agents: clinical practice and clues to improve care provided by physicians, nurse practitioners, nurses and pharmacists. BMC Cancer. 2017;17(1).

Download references

Acknowledgements

Not applicable.

Author information

Authors and affiliations.

School of Nursing and Midwifery, Edith Cowan University, 270 Joondalup Drive, Joondalup, Perth, WA, 6027, Australia

Amineh Rashidi, Susma Thapa, Wasana Sandamali Kahawaththa Palliya Guruge & Shubhpreet Kaur

You can also search for this author in PubMed   Google Scholar

Contributions

First author (AR) and second author (ST) conceived the review and the second author oversight for all stages of the review provided by the second author. All authors (AR), (ST), (WG) and (SK) undertook the literature search. Data extraction, screening the included papers and quality appraisal were undertaken by all authors (AR), (ST), (WG) and (SK). First and second authors (AR) and (ST) analysed the data and wrote the first draft of the manuscript and revised the manuscript and all authors (AR), (ST), (WG) and (SK) approved the final version of the manuscript.

Corresponding author

Correspondence to Amineh Rashidi .

Ethics declarations

Ethics approval and consent to participate.

The review was approved by the Edith Cowan University Human Research Ethics Committee (2021–02896). A proposal for the systematic review was assessed by the Edith Cowan University Human Research Ethics Committee and deemed not appropriate for full ethical review. However, a Data Management Plan (2021-02896-RASHIDI) was approved and monitored as part of this procedure. Raw data was extracted from the published manuscripts and authors could not identify individual participants during or after this process.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary material 2, supplementary material 3, supplementary material 4, supplementary material 5, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Rashidi, A., Thapa, S., Kahawaththa Palliya Guruge, W. et al. Patient experiences: a qualitative systematic review of chemotherapy adherence. BMC Cancer 24 , 658 (2024). https://doi.org/10.1186/s12885-024-12353-z

Download citation

Received : 17 November 2023

Accepted : 07 May 2024

Published : 30 May 2024

DOI : https://doi.org/10.1186/s12885-024-12353-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Chemotherapy treatment
• Medication adherence
• Qualitative research
• Patients experiences

ISSN: 1471-2407

• Submission enquiries: [email protected]
• General enquiries: [email protected]

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Review Article
• Open access
• Published: 25 May 2024

Relationship between obstructive sleep apnoea syndrome and gastrointestinal diseases: a systematic review and Meta-analysis

• Liubin Cao 1   na1 ,
• Chengpei Zhou 2   na1 ,
• Rupei Zhang 1 ,
• Shan Zhou 1 ,
• Xiaolei Sun 2 &
• Jun Yan 1  

npj Primary Care Respiratory Medicine volume  34 , Article number:  12 ( 2024 ) Cite this article

254 Accesses

Metrics details

• Outcomes research
• Respiratory distress syndrome

Studies exploring the association between obstructive sleep apnoea syndrome (OSA) and gastrointestinal diseases (GID) are important for enhancing clinical outcomes. This study aimed to systematically assess the association between these two diseases. Adhering to PRISMA guidelines, a comprehensive literature search was conducted across databases including PubMed, Web of Science, Willey Library, Cochrane Library and Scopus. This search focused on English literature published up to January 2024. Literature screening, quality assessment (using the NOS scale) and data extraction were performed by two independent researchers. Statistical analyses were performed using the meta-package of the R.4.2.2 software. An initial screening of 2178 papers was conducted and 11 studies were included. Meta-analysis results showed a significant association between OSA and GID ( p  < 0.01). Subgroup analyses further indicated a stronger association between OSA and GID in Asian populations compared to Europe and the United States. In addition, both benign and malignant GID were significantly associated with OSA, with a pronounced association for malignant GID than for benign GID. The results of publication bias analysis revealed no significant bias (Begg’s test p  = 0.45, Egger’s test p  = 0.60). This study uncovers a notable association between OSA and GID, especially in Asian populations, suggesting that clinicians should consider the potential connection between these two diseases during diagnosis and treatment. However, due to the heterogeneity and limitations of the study, these conclusions need to be further validated through more comprehensive research.

Similar content being viewed by others

Obstructive sleep apnoea and glaucoma: a systematic review and meta-analysis

A systematic review on the association of sleep-disordered breathing with cardiovascular pathology in adults

Transforming electronic health record polysomnographic data into the Observational Medical Outcome Partnership's Common Data Model: a pilot feasibility study

Introduction.

Obstructive sleep apnoea syndrome (OSA) is a prevalent sleep breathing disorder characterised by recurrent partial or complete obstruction of the upper airway during sleep, leading to apnoea or hypopnoea 1 , 2 . OSA not only diminishes sleep quality, but is also associated with a wide range of health problems such as cardiovascular disease, metabolic disorders, and neurocognitive dysfunction 3 , 4 , 5 , thereby emerging as an important public health concern.

Recently, the potential link between OSA and gastrointestinal diseases (GID) has garnered increasing attention from the medical community 6 , 7 , 8 , 9 , 10 . GID, including gastroesophageal reflux disease (GERD) and inflammatory bowel disease (IBD), commonly affects the health of the global population. It has been suggested that symptoms of GERD are more prevalent in patients with OSA, and these gastrointestinal symptoms may in turn exacerbate the manifestations of OSA. GID affects not only the digestive system but may also cause a range of systemic health problems.

In their Meta-analysis, Nabil El Hage Chehade and other researchers 7 suggested an association between OSA and GERD, independent of screening or diagnostic methods. They noted that GERD’s presence does not directly affect the severity of OSA. This finding implies a more complex interaction mechanism between OSA and GID, possibly involving a variety of physiological and pathological processes including intermittent hypoxaemia 10 , 11 , nocturnal respiratory abnormalities 8 , and comorbid obesity and metabolic syndrome 12 .

Despite studies that have been conducted to explore the potential association between OSA and GID, the results of these studies have been inconsistent 6 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 . Therefore, this study aims to comprehensively evaluate the association between OSA and various types of GID to provide stronger evidence support for clinical diagnosis and treatment, thereby improving the diagnostic and treatment protocols and quality of life for patients.

Results of literature screening

The literature screening process for this study was meticulously executed in strict adherence to the pre-established inclusion and exclusion criteria. The initial search yielded a total of 2,178 pieces of literature, with 2,173 obtained through electronic database searches, and 5 obtained through manual searches from references. After removing 483 duplicates, 1408 pieces of literature were excluded due to titles and abstracts not aligning with the study’s focus and 5 pieces of literature were not accessible in full text. Subsequently, the remaining 282 literatures were then reviewed in full text, leading to the exclusion of an additional 271 papers, primarily due to non-compliance with the study design, or incomplete or substandard data quality. Ultimately, a total of 11 pieces of literature were eventually included in the final analysis, as seen in Fig. 1 . Details of the included literature are given in Table 1 .

Schematic diagram of the literature screening process.

Literature quality evaluation results

The quality of the literature included in this study was evaluated using the NOS scale. The evaluation revealed that seven literatures scored more than 7, which showed high research quality, and four literatures scored 6, which was medium quality and met the quality requirements of this paper, as shown in Table 2 .

Association between OSA and GID

The meta-analysis incorporated eleven studies with a total of 2729 patients to assess the association between OSA and GID. Heterogeneity analysis found substantial variability I 2  = 88%. By using a random-effects model, the Meta-analysis showed a statistically significant correlation between the two diseases (OR = 1.37, 95% CI [1.13, 1.66], p  < 0.01) (Fig. 2 ). For subgroup analysis, GID was categorized into benign and malignant conditions, observing a significant difference ( p  < 0.01) between benign (OR = 1.25, 95% CI [0.88, 1.79]) and malignant GID (OR = 1.42, 95% CI [1.16, 1.75]). When analysed using a fixed-effects model, a notably stronger association was observed in malignant GID (Fig. 3 ). Subgroup analyses based on regional classification revealed that the correlation between OSA and GID was significantly different between geographic regions using fixed-effect model analysis ( P  < 0.01) (Fig. 4 ), with the correlation between OSA and GID being stronger in Asia (OR = 1.57, 95% CI [1.14, 2.15]) than Europe and the United States (OR = 1.24, 95% CI [1.13, 1.66]) was stronger (Fig. 4 ).

OR odds ratio, SE standard error, CI confidence interval, OSA obstructive sleep apnoea syndrome, GID gastrointestinal diseases.

Factors influencing the correlation between OSA and GID

In analysing the factors influencing the OSA-GID correlation, both age and gender demonstrated a high degree of heterogeneity for both the age and gender factors ( I 2  = 97% and I 2  = 99%), both analysed using a random-effects model. The findings suggested that neither age (OR = 1.20, 95% CI [0.99, 1.46], P  = 0.07) nor gender (OR = 1.33, 95% CI [0.95, 1.86], P  = 0.10) were not key factors influencing the correlation between OSA and GID, as seen in Figs. 5 , 6 .

Publication bias analysis

To assess the possible presence of publication bias in this Meta-analysis, several statistical methods were employed. Visual inspection of funnel plots indicated good symmetry, despite some studies falling outside the funnel. Additionally, the outcomes of Begg’s test ( z  = 0.76, p  = 0.45) and Egger’s test ( t  = −0.54, p  = 0.60) showed that the included literature was free of publication bias, as seen in Fig. 7 .

OSA obstructive sleep apnoea syndrome, GID gastrointestinal diseases.

Sensitivity analysis

The results of the sensitivity analyses showed that neither the combined effect sizes nor the heterogeneity indicators of the Meta-analyses changed significantly after the exclusion of any single study, suggesting that the results of our Meta-analyses were highly robust and that no single study had an excessive effect on the overall results.

The results of this systematic evaluation and Meta-analysis suggest that there is a strong interconnection between OSA and GID, especially for malignant GID. This finding provides new insights into the mechanisms of OSA and GID interactions and suggests the need for optimisation in clinical practice.

Numerous studies have been conducted in recent years to investigate the mechanisms linking OSA and GID and to establish a causal relationship between 2 , 3 , 23 . Recent studies have revealed that OSA-induced physiological changes, such as intermittent hypoxaemia and sleep disruption, may increase the risk of GID, including gastroesophageal reflux disease, gastric ulcers, and another digestive disorders 20 , 21 , 22 . The increased stress on the body’s internal environment caused by OSA may precipitate gastrointestinal dysfunction 18 , 19 . Furthermore, the high prevalence of obesity and metabolic syndrome among OSA patients may also further exacerbate the risk of GID 24 . However, some studies have yielded inconsistent results 6 , 17 , 18 , 19 . Therefore, our systematic evaluation and meta-analysis aimed to provide more comprehensive and precise evidence by synthesising data from a large number of studies to address the existing controversies regarding the association of OSA with GID. We focused on parsing differences across regions, ethnicities, and lifestyle contexts, and on exploring potential biological mechanisms, aiming to offer deeper insights into the field. Our study not only focuses on the aggregation of existing evidence but also endeavours to identify potential biases in research and suggest future research directions to provide more instructive information for clinical practice and patient management.

From a pathophysiological perspective, the correlation between OSA and GID may involve a series of complex biological pathways. Intermittent hypoxaemia, which often occurs at night in patients with OSA, may trigger a systemic inflammatory response. This inflammatory response is particularly closely related to benign GID, which implies that OSA may cause damage to the gastrointestinal mucosal barrier and increase the probability of GERD and other benign GID 4 , 25 . In addition, OSA-induced changes in intrathoracic and intra-abdominal pressure may disrupt the lower oesophageal sphincter’s function, promoting gastric acid reflux 26 . These changes can directly impair gastrointestinal health and may also indirectly affect the digestive system through neuroreflex mechanisms. The prevalence of obesity in patients with OSA is also an important factor exacerbating the risk of GID. Obesity may exacerbate the symptoms of GID by increasing the intra-abdominal pressure and causing metabolic disturbances 27 . In addition, obesity may also affect the rate of gastric emptying and the balance of intestinal flora, which may further exacerbate gastrointestinal problems 28 .

In exploring the geographic distribution of the OSA-GID relationship, our study found a particularly significant association in Asian populations 13 , 15 , 19 . This pattern may be due to the unique genetic background, dietary habits, and lifestyle of the Asian region. The high content of salt and fat in the Asian diet may significantly influence the pathogenesis of GID, such as GERD 18 , promoting increased gastric acid secretion and exacerbating GERD symptoms, which may trigger or worsen GID in patients with OSA. In addition, the high carbohydrate intake in Asian populations may be associated with an increased prevalence of GID, especially among patients with diabetes mellitus 29 . Genetic factors There may be specific genetic variants or genetic predispositions in the development of OSA and GID that make Asian populations more susceptible to both diseases. For example, genetic variants associated with obesity, diabetes, and other metabolic diseases may increase the risk of both OSA and GID 30 . Nevertheless, research in this domain is nascent, and more comprehensive genetic epidemiological studies are needed to uncover specific molecular mechanisms and genetic predispositions.

These findings highlight the importance of identifying and managing patients with both OSA and GID in daily clinical practice. Primary care physicians should consider the interrelationship between OSA and GID, particularly in Asian, and conduct a comprehensive evaluation of the medical history including pay attention to the patient’s dietary habits, lifestyle, and genetic background. It is recommended that during the diagnosis, a comprehensive evaluation should be performed for GID patients with OSA symptoms. Multi-disciplinary treatment should be considered for these patients, including Department of Gastroenterology and Department of Respiration.

This study analyzed the association between OSA and GID, and explored the association between OSA and malignant or benign GID through subgroup analysis, as well as subgroup analysis of the population, to obtain more comprehensive analysis results. Although there are some limitations. Most included studies were observational, causality could not be established. In addition, the high heterogeneity of the included studies may affect the robustness and generalisability of the results, and future studies should adopt a more rigorous study design and uniform diagnostic criteria to further validate our findings.

In conclusion, this study not only highlights the importance of considering patients’ gastrointestinal conditions in the diagnosis and treatment of OSA, but also sheds light on biological mechanisms that need to be investigated in greater depth. Future studies should aim to clarify the causal relationship between OSA and GID, analyse regional and racial differences, and delve deeper into the underlying molecular and cellular mechanisms, so as to provide more comprehensive treatment strategies for the clinic.

As this study involves the summary and analysis of other studies, it does not involve medical ethics approval or patient-informed consent.

Search strategy

This study was conducted in accordance with the PRISMA guidelines 23 , and to systematically assess the relationship between OSA and GID, the study was searched in several databases, including PubMed, Web of science, Willey Library, Cochrane Library, and Scopus. The search strategy included keywords and medical subject terms related to OSA and various GID. The search language was English, and the search period was from the establishment of each database to January 2024.

The search terms included “Obstructive Sleep Apnea”, “OSA”, “Gastrointestinal Diseases”, “Gastroesophageal Reflux Disease (GERD)”, “Irritable Bowel Syndrome (IBS)”. “Inflammatory Bowel Disease (IBD)”, “Crohn’s Disease”, “Ulcerative Colitis “, “Gastritis”, “Peptic Ulcer Disease”, “Gastroenteritis “, “Colorectal Cancer”. The search was performed using a combination of subject terms and free words with matching truncation, using PubMed as an example: (“Obstructive Sleep Apnea” [MeSH Terms] OR “OSA” [Title/Abstract]) AND (“Gastrointestinal Diseases” [MeSH Terms] OR “Gastroesophageal Reflux Disease (GERD)”[Title/Abstract] OR “Irritable Bowel Syndrome (IBS)”[Title/Abstract] OR “Inflammatory Bowel Disease (IBD)”[Title/Abstract] OR “Crohn’s Disease”[Title/Abstract] OR “Ulcerative Colitis”[Title/Abstract] OR “Gastritis”[Title/Abstract] OR “Peptic Ulcer Disease”[Title/Abstract] OR “Gastroenteritis”[Title/Abstract] OR “Colorectal Cancer”[Title/Abstract]).

Literature inclusion and exclusion criteria

Inclusion criteria included (1) studies with a clear diagnosis of OSA, and the basis for the diagnosis should include standard sleep monitoring data or recognised clinical diagnostic guidelines; (2) studies assessed the association between OSA and GID (such as GERD, IBS, IBD, Crohn’s Disease, Ulcerative Colitis); (3) studies provided quantitative data, such as risk ratios (RR), odds ratios (OR), or correlation data; (4) the publication was a peer-reviewed full-text article; and (5) the article was written in English.

Exclusion criteria included: (1) interventional studies, case reports, commentaries, expert opinions, conference abstracts, review articles, or non-peer-reviewed publications; (2) studies that did not provide sufficient data to estimate risk ratios or odds ratios; (3) studies in which the diagnostic criteria for OSA or GID were unclear or did not meet widely accepted clinical guidelines; (4) studies that were repetitively published or had duplicated data; (5) studies that were not original, such as secondary data based on published data.

Literature screening

The literature screening process for this study followed a rigorous process, wherein the titles and abstracts of retrieved literature were initially examined by two intended researchers. The preliminary screening aimed to exclude irrelevant studies, specifically those not addressing OSA or a specific GID. Subsequently, a detailed full-text review was undertaken for literature that seems to the inclusion criteria, to determine whether the inclusion criteria were fully met. Discrepancies in opinion were resolved through discussion or, if necessary, by consulting third-party experts.

Literature quality assessment

The Newcastle-Ottawa Scale (NOS), a standardised tool for appraising the quality of observational studies, particularly cohort and case-control studies, was used in this study for literature quality evaluation. The NOS encompasses three main domains: selectivity (choice of the study population), comparability (comparisons between study groups), and outcome (assessment of the study’s results.) The NOS scale reflects the methodological quality of each study by providing it with an overall score of up to 9 points. In this study, each study meeting the inclusion criteria was independently assessed by two researchers using the NOS scale. Disagreements during the scoring process were addressed through discussion, and third-party experts were consulted as needed.

Data extraction

In this study, the data extraction process was carried out independently by two researchers to ensure the accuracy and completeness of the data. A comprehensive data extraction form was designed to gather key information from each study, including (1) basic information about the study, such as authors, year of publication, and type of study design; (2) characteristics of the study population, including sample size, age range, and gender ratio; (3) primary and secondary outcome indicators, including all relevant clinical outcomes and measurements; (4) study results, such as the odds ratio (OR), correlation coefficient, and so on, as well as their confidence intervals and statistical significance levels. Upon completion of each data extraction, investigators cross-verified the data to eliminate potential errors or biases. Any disagreements identified during the data extraction process were resolved through discussion, and third-party expert consultation was sought as necessary.

Statistical methods

The Meta-analysis was conducted using the meta-package in R software (R 4.2.2). Firstly, we will extract the corrected Odds Ratio (OR) of the outcome metrics and their 95% Confidence Interval (CI) from each included study. Subsequently, these ORs and 95% CIs were converted to logORs and their standard errors (Standard Error, SE) for meta-analysis. For the heterogeneity test, I ² statistic was used to assess the heterogeneity between studies, when I ² ≤ 50%, indicating small heterogeneity, a fixed-effect model (FEM) was employed for meta-analysis. If I ² > 50%, indicating a high degree of heterogeneity, in which case a random-effects model was used to analyse the data. To assess the quality of the studies, the Newcastle-Ottawa Scale (NOS) was used for cohort studies. In addition, for studies numbering 10 or more, funnel plots, Begg’s test, and Egger’s test were utilized to assess possible publication bias. A statistically significant difference of P  < 0.05 was used as the criterion for all tests.

Data availability

The data used to support the findings of this study are included within the article.

Lv, R. et al. Pathophysiological mechanisms and therapeutic approaches in obstructive sleep apnea syndrome. Signal Transduct. Target. Ther. 8 , 218 (2023).

Article   PubMed   PubMed Central   Google Scholar  

Lee, J. J. & Sundar, K. M. Evaluation and management of adults with obstructive sleep apnea syndrome. Lung 199 , 87–101 (2021).

Article   PubMed   Google Scholar  

Jaimchariyatam, N., Tantipornsinchai, W., Desudchit, T. & Gonlachanvit, S. Association between respiratory events and nocturnal gastroesophageal reflux events in patients with coexisting obstructive sleep apnea and gastroesophageal reflux disease. Sleep Med. 22 , 33–38 (2016).

Orr, W. C., Fass, R., Sundaram, S. S. & Scheimann, A. O. The effect of sleep on gastrointestinal functioning in common digestive diseases. Lancet Gastroenterol. Hepatol. 5 , 616–624 (2020).

Roy N., Merrill R. M., Pierce J., Sundar K. M. Evidence of possible irritable larynx syndrome in obstructive sleep apnea: an epidemiologic approach. J. Voice 35 , 932.e29-932.e38 (2021).

Hoffman K. et al. Prevalence of obstructive sleep apnea is increased in patients with inflammatory bowel disease: a large, multi-network study. Crohns. Colitis. 360 4 , otac026 (2022).

El Hage Chehade, N. et al. Association between obstructive sleep apnea and gastroesophageal reflux disease: A systematic review and meta-analysis. J. Gastroenterol. Hepatol. 38 , 1244–1251 (2023).

Wu, Z. H., Yang, X. P., Niu, X., Xiao, X. Y. & Chen, X. The relationship between obstructive sleep apnea hypopnea syndrome and gastroesophageal reflux disease: a meta-analysis. Sleep Breath. 23 , 389–397 (2019).

Mashaqi, S. et al. Biomarkers of gut barrier dysfunction in obstructive sleep apnea: a systematic review and meta-analysis. Sleep Med. Rev. 69 , 101774 (2023).

Article   CAS   PubMed   Google Scholar  

Basoglu, O. K. et al. Obstructive sleep apnea syndrome and gastroesophageal reflux disease: the importance of obesity and gender. Sleep Breath 19 , 585–592 (2015).

Barnes A. et al. Simple novel screening tool for obstructive sleep apnea in inflammatory bowel disease. Crohns. Colitis. 360 5 , otad016 (2023).

Drager, L. F., Togeiro, S. M., Polotsky, V. Y. & Lorenzi-Filho, G. Obstructive sleep apnea: a cardiometabolic risk in obesity and the metabolic syndrome. J. Am. Coll. Cardiol. 62 , 569–576 (2013).

Chen, C. Y. et al. Increased incidence of colorectal cancer with obstructive sleep apnea: a nationwide population-based cohort study. Sleep Med. 66 , 15–20 (2020).

Cummings, L. C. et al. Barrett’s esophagus and the risk of obstructive sleep apnea: a case-control study. BMC Gastroenterol. 13 , 82 (2013).

Fang, H. F., Miao, N. F., Chen, C. D., Sithole, T. & Chung, M. H. Risk of cancer in patients with insomnia, parasomnia, and obstructive sleep apnea: a nationwide nested case-control study. J. Cancer 6 , 1140–1147 (2015).

Ghiasi, F., Amra, B., Sebghatollahi, V. & Azimian, F. Association of irritable bowel syndrome and sleep apnea in patients referred to sleep laboratory. J. Res. Med. Sci. 22 , 72 (2017).

Gozal, D., Ham, S. A. & Mokhlesi, B. Sleep apnea and cancer: analysis of a nationwide population sample. Sleep 39 , 1493–1500 (2016).

Huang T. et al. Associations of self-reported obstructive sleep apnea with total and site-specific cancer risk in older women: a prospective study. Sleep 44 , zsaa198. (2021).

Kang, H. H., Lim, C. H., Oh, J. H., Cho, M. J. & Lee, S. H. The influence of gastroesophageal reflux disease on daytime sleepiness and depressive symptom in patients with obstructive sleep apnea. J. Neurogastroenterol. Motil. 27 , 215–222 (2021).

Valentin, S. et al. Prevalence of and factors associated with respiratory symptoms among patients with inflammatory bowel disease: a prospective study. Inflamm. Bowel Dis. 29 , 207–216 (2023).

Vela, M. F., Kramer, J. R., Richardson, P. A., Dodge, R. & El-Serag, H. B. Poor sleep quality and obstructive sleep apnea in patients with GERD and Barrett’s esophagus. Neurogastroenterol. Motil. 26 , 346–352 (2014).

You, C. R. et al. Association between non-erosive reflux disease and high risk of obstructive sleep apnea in Korean population. J. Neurogastroenterol. Motil. 20 , 197–204 (2014).

Liberati A. et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med . https://doi.org/10.1371/journal.pmed.1000100 (2009).

Teo, Y. H. et al. Obstructive sleep apnea and the incidence and mortality of gastrointestinal cancers: a systematic review and meta-analysis of 5,120,837 participants. J. Gastrointest. Oncol. 13 , 2789–2798 (2022).

Lim, K. G., Morgenthaler, T. I. & Katzka, D. A. Sleep and nocturnal gastroesophageal reflux: an update. Chest 154 , 963–971 (2018).

Zhang, Y. et al. Obesity: pathophysiology and intervention. Nutrients. 6 , 5153–5183 (2014).

Article   CAS   PubMed   PubMed Central   Google Scholar  

Behnoush, A. H. et al. Triglyceride-glucose index and obstructive sleep apnea: a systematic review and meta-analysis. Lipids Health Dis. 23 , 4 (2024).

Avirineni B. S., Singh A., Zapata R. C., Phillips C. D., Chelikani P. K. Dietary whey and egg proteins interact with inulin fiber to modulate energy balance and gut microbiota in obese rats. J. Nutr. Biochem . 99 , 108860 (2022).

Kuribayashi, S. et al. Upper esophageal sphincter and gastroesophageal junction pressure changes act to prevent gastroesophageal and esophagopharyngeal reflux during apneic episodes in patients with obstructive sleep apnea. Chest 137 , 769–776 (2010).

Reutrakul, S. & Mokhlesi, B. Obstructive sleep apnea and diabetes: a state of the art review. Chest 152 , 1070–1086 (2017).

Download references

Acknowledgements

We are grateful for the support of the National Natural Science Foundation of China (No.: 81300346).

Author information

These authors contributed equally: Liubin Cao, Chengpei Zhou.

Authors and Affiliations

Department of Forensic Medicine, Nantong University Medical College, Nantong, China

Liubin Cao, Rupei Zhang, Shan Zhou & Jun Yan

Department of Pathogenic Biology, Nantong University Medical College, Nantong, China

Chengpei Zhou & Xiaolei Sun

You can also search for this author in PubMed   Google Scholar

Contributions

Conception and design: Liubin Cao and Chengpei Zhou. Administrative support: Rupei Zhang. Collection and assembly of data: Shan Zhou. Data analysis and interpretation: Xiaolei Sun. Drafting the work: Jun Yan. Revising the work: Jun Yan. Final approval of manuscript: All authors. Liubin Cao and Chengpei Zhou contributed equally to this paper.

Corresponding author

Correspondence to Jun Yan .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Cao, L., Zhou, C., Zhang, R. et al. Relationship between obstructive sleep apnoea syndrome and gastrointestinal diseases: a systematic review and Meta-analysis. npj Prim. Care Respir. Med. 34 , 12 (2024). https://doi.org/10.1038/s41533-024-00373-y

Download citation

Received : 29 January 2024

Accepted : 08 May 2024

Published : 25 May 2024

DOI : https://doi.org/10.1038/s41533-024-00373-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

From economic wealth to well-being: exploring the importance of happiness economy for sustainable development through systematic literature review

• Open access
• Published: 23 May 2024

Cite this article

You have full access to this open access article

• Shruti Agrawal   ORCID: orcid.org/0000-0002-1620-9429 1 , 5 ,
• Nidhi Sharma 1 , 5 ,
• Karambir Singh Dhayal   ORCID: orcid.org/0000-0002-0000-4330 2 &
• Luca Esposito   ORCID: orcid.org/0000-0001-5983-6898 3 , 4  

552 Accesses

Explore all metrics

The pursuit of happiness has been an essential goal of individuals and countries throughout history. In the past few years, researchers and academicians have developed a huge interest in the notion of a ‘happiness economy’ that aims to prioritize subjective well-being and life satisfaction over traditional economic indicators such as Gross Domestic Product (GDP). Over the past few years, many countries have adopted a happiness and well-being-oriented framework to re-design the welfare policies and assess environmental, social, economic, and sustainable progress. Such a policy framework focuses on human and planetary well-being instead of material growth and income. The present study offers a comprehensive summary of the existing studies on the subject, exploring how a happiness economy framework can help achieve sustainable development. For this purpose, a systematic literature review (SLR) summarised 257 research publications from 1995 to 2023. The review yielded five major thematic clusters, namely- (i) Going beyond GDP: Transition towards happiness economy, (ii) Rethinking growth for sustainability and ecological regeneration, (iii) Beyond money and happiness policy, (iv) Health, human capital and wellbeing and (v) Policy push for happiness economy. Furthermore, the study proposes future research directions to help researchers and policymakers build a happiness economy framework.

Similar content being viewed by others

Beyond GDP: A Movement Toward Happiness Economy to Achieve Sustainability

National progress, sustainability and higher goals: the case of Bhutan’s Gross National Happiness

Economic Performance, Happiness, and Sustainable Development in OECD Countries

Avoid common mistakes on your manuscript.

1 Introduction

Happiness is considered the ultimate goal of human beings (Ikeda, 2010 ; Lama, 2012 ). All economic, social, environmental and political human activities are aligned towards achieving this goal. This fundamental pursuit of human life introduces a new scope of research, namely the ‘happiness economy’ (Agrawal and Sharma 2023 ). The happiness economy is an emerging economic domain wherein many countries are working to envision and implement a happiness-oriented framework by expanding how they measure economic success, which includes wellbeing and sustainability (Cook and Davíðsdóttir 2021 ; Forgeard et al., 2011 ). The investigation of happiness, life-satisfaction and subjective well-being has witnessed increasing research interest across the disciplines- from psychology, philosophy, psychiatry, and cognitive neuroscience to sociology, economics and management (Diener 1984 ; Hallberg and Kullenberg, 2019 ).

In the post-Covid era, the world seeks an enormous transformation shift in the public system (Costanza 2020 ). However, public authorities need more time to realize such needs. To experience the ‘policy transformation’ within the coming few years, we require a paradigm shift that helps warm peoples’ hearts and minds. The new economic paradigm can penetrate the policy processes in advanced economies and every part of the world affected by the epidemic with the support of intellectuals, researchers, entrepreneurs and professionals.

OECD ( 2016 ) proposed a well-being economy framework to measure living conditions and people’s well-being. In 2020, developed countries like Finland, New Zealand, Iceland, Scotland and Wales have become members of the Wellbeing Economy Government (WEGo) (Abrar 2021 ). Since then, the network of government and international authorities across the globe has gained a quick momentum concerning an increasing tendency about a growing tendency to concentrate governmental decisions around human well-being rather than wealth and economic growth (Coscieme et al. 2019 ; Costanza et al. 2020 ).

In light of these circumstances, the purpose of this article is to describe the concept of a “happiness economy” or one that seeks to give everyone fair possibilities for growth, a sense of social inclusion, and stability that can support human resilience (Coyne and Boettke 2006 ). It provides a promising route towards improved social well-being and environmental health and is oriented towards serving individuals and communities (Skul’skaya & Shirokova, 2010 ). Moreover, the happiness economy paradigm is a transition from material production and consumption of commodities and services as the only means to economic development towards embracing a considerable variety of economic, social, environmental and subjective well-being dynamics that are considered fundamental contributors to human happiness (Atkinson et al., 2012 ; King et al., 2014 ; Agrawal and Sharma 2023 ). In following so, it reflects the ‘beyond growth’ approach that empathizes with the revised concept of growth, which is not centred around an increase in income or material production; instead it is grounded in the philosophy of achieving greater happiness for more people (Fioramonti et al. 2019a ).

Whereas the other critiques of economic growth emphasize contraction, frugality and deprivation, the happiness economy relies on a cumulative approach of humanity, hope and well-being, with a perceptive to build a ‘forward-looking’ narrative of ways for humans to live a happy and motivated life by inspiring the cumulative actions and encouraging policy-reforms in the measuring growth of an economy (Stucke 2013 ). Agrawal et al. ( 2023a , b ) explore the domain of happiness economics through a review of the various trends coupled with the future directions and highlight why it needs to be supported for a well-managed economic system and a happy society.

In this paper, we define a “happiness economy as an economy that aims to achieve the well-being of individuals in a nation, promoting human happiness, environmental up-gradation, and sustainability. Alternatively, as an economy where the wellbeing of people counts more than the goals of production and income”. Moreover, we have examined the existing body of research on the happiness economy and analyzed the emerging research themes related to rethinking the conventional approach to economic growth. We conclude by discussing how the happiness economy concept has been accepted so far and realizing its importance by triggering policy reforms at the societal level, by outlining potential future directions that might be included into the current national post-growth policies.

Various researchers and experts in the field of happiness economy support the idea that there is a lack of thorough studies related to the concept, definitions, and themes of the happiness economy model in the nations. This gap has motivated us to conduct a SLR in order to identify the evolution in the domain of happiness economy and to identify the emerging themes in this context. Therefore, this present study seeks to offer a holistic outline of the emerging research area of the happiness economy and helps to understand how the happiness economy can accelerate sustainable development. With the following research questions, this study seeks to give an all-encompassing review of this subject.

What is the annual publication trend in this domain and the most contributing authors, journals, countries etc?

Which themes and upcoming research areas are present in this field?

What directions will the happiness economics study field go in the future?

The SCOPUS database was used to achieve the above research objectives. We have selected 257 articles for examination by hand-selecting the pertinent keywords and going over each one. In the methods section, a thorough explanation of the procedures for gathering, reviewing, and selecting documents is provided.

The remainder of this paper is structured as follows; A thorough survey of the literature on the happiness economy is provided in Sect.  2 . The research approach employed in the study is presented in Sect.  3 . A thorough data analysis of the research findings is given in Sect.  4 . After discussing the results in Sect.  5 , Sect.  6 suggests areas for further research in this field. The study is summarised with a conclusion in Sect.  7 . Section  8 outlines the study’s limitation.

2 Literature review

The supporters of conventional economic growth proclaim that the material production of goods and services and consumption is vital to enhancing one’s living standards. The statement is true to some degree, mainly in countries of enormous deprivation. Some studies have found significantly less correlation between growth and happiness after fulfilling minimum threshold needs (Easterlin 1995 ; Kahneman and Krueger, 2006 ; Inglehart et al., 2008 ). These studies recommend that rather than concentrating solely on economic growth, governmental policy should give priority to non-economic aspects of human existence above a particular income level. According to some researchers, it is challenging to distinguish between the use and emissions of natural resources and economic growth (absolute decoupling) because of the interdependence between socioeconomic conditions and their biophysical basis (Wiedenhofer et al. 2020 ; Wang and Su, 2019 ; Wu et al., 2018 ). However, a shred of increasing evidence shows that it could be possible for humans to maintain a quality of life and a decent standard of living inside the ecological frontier of the environment, given that a contemporary perspective on the production and use of materials are adopted in conjunction with more fair wealth distribution (Millward-Hopkins et al. 2020 ; Bengtsson et al., 2018 ; Ni et al., 2022 ).

The scholarly discourse and institutional framework on the relationship between happiness and economic progress are synthesised in the happiness economy (Frey and Gallus 2012 ; Sohn, 2010 ; Clark et al., 2016 ; Easterlin, 2015 ; Su et al., 2022 ). From a happiness economy perspective, extreme materialism is unsustainable as it significantly impacts natural resources and hinders social coherence and individuals psychological and physical well-being (Fioramonti et al. 2022a ). Additionally, inequalities within countries have grown, while psychological suffering has increased, especially during accelerated growth (Vicente 2020 ; Galbraith, 2009 ). The modern world is witnessing anxiety, depression, wars, reduction of empathy, climate change, pandemics, loss of social bonds and other psychological disorders (Brahmi et al., 2022 ; Santini et al., 2015 ).

It has been scientifically proven that cordial human relations, care-based activity, voluntary activities and the living environment immensely impact a person’s health and societal well-being (Bowler et al. 2010 ; Keniger et al., 2013 ). Ecological economists demonstrated that free ecosystem services have enhanced human well-being (Fang et al. 2022 ). Social epidemiologists have long argued that an increase in inequalities has a negative influence on society while providing equality tends to improve significant objective ways of well-being, from healthier communities to happier communities, declining hate and crime and enhancing social cohesion, productivity, unity and mutual trust (Aiyar and Ebeke 2020 ; Ferriss, 2010 ).

From moving beyond materialistic growth, the happiness economy promotes, appreciates, and protects the environmental, societal, and human capital contributions that lead to cummalative well-being. In a happiness economy framework, a multidimensional approach is needed to evaluate the level of development based on the environmental parameters, health outcomes, as well as public trust, hope, value-creating education and social bonds (Agrawal and Sharma 2023 ; Bayani et al. 2023 ; Lavrov, 2010 ). Such factors have consistently been excluded from any traditional concept or assessment of economic growth. As a result, countries have promoted more industrial activities that deteriorate the authentic ways of human well-being and, hence, the foundations of economic progress.

An excess of production can create a detrimental effect on climate and people’s health, thereby creating a negative externality for society (Fioramonti et al. 2022b ). Moderation of output may be more efficient and desirable than hyper/over-production, as the former can reduce negative environmental externalities (e.g. waste, climate change) and create positive externalities (e.g. employment of the local resources and community) (Kim et al. 2019 ; Kinman and Jones, 2008 ). Moreover, people can also be productive in other contexts outside of the workplace, such as as volunteers, business owners, artists, friends, or members of the community (Fioramonti et al. 2022a ).

Various scholars and scientific research have established that the essential contributions to happiness in one’s life are made by natural surroundings, green and blue spaces, eco-friendly environment, healthy social relations, spirituality, good health, responsible consumption and value-creating education (Helliwell et al. 2021 ; Francart et al., 2018 ; Armstrong et al., 2016 ; Gilead, 2016 ; Giannetti et al., 2015 ). Unfortunately, existing conventional growth theories have ignored all these significant contributions. For example, GDP considers natural ecosystems as economically helpful only up until they are mined and their products are traded (Carrero et al. 2020 ). The non-market benefits they generate, such as natural fertilization, soil regeneration, climate regulation, clean air and maintenance of biodiversity, are entirely ignored (Boyd 2007 ; Hirschauer et al., 2014). The quality time people spend with their families and communities for leisure, educating future generations and making a healthy communal harmony is regarded meaningless, even in the event that they are important to enhance people’s well-being and, hence, to assist any dimension of economic engagement (Griep et al. 2015 ; Agrawal et al., 2020 ). Similarly, if an economy is focusing on people’s healthy lifestyle (for example, by providing comfortable working hours, improving work-life balance, emphasizing mental health, focusing on healthy food, reducing pollution, and promoting sustainable consumption), it is not considered in sync with the growth paradigm (Roy 2021 ; Scrieciu et al., 2013; Shrivastava and Zsolnai 2022 ; Lauzon et al., 2023 ).

Among the latest reviews, Bayani et al. ( 2023 ) highlight that the economics of happiness helps reduce the country’s financial crime by providing a livelihood that reduces financial delinquency. Chen ( 2023 ) highlights that smart city performance enhances urban happiness by adopting green spaces, reusing and recycling products, and controlling pollution. The study by (Agrawal and Sharma 2023 ) proposed a conceptual framework for a happiness economy to achieve sustainability by going beyond GDP. Similarly, Fioramonti et al. ( 2019b ) explored going beyond GDP for a transition towards a happy and well-being economy. The article by Laurent et al. ( 2022 ) has intensively reviewed the well-being indicators in Rome and proposed a conceptual framework for it.

Table  1 provides a thorough summary of the prior review studies about the happiness economy and its contribution to public policy and sustainable development.

3 Research methodology

In the current study, we have adopted an integrative review approach of SLR and bibliometric analysis of the academic literature to get a detailed knowledge of the study, which could also help propose future research avenues. The existing scientific production’s qualitative and quantitative context must be incorporated for a conclusive decision. The study by Meredith ( 1993 ) defines that SLR enables an “integrating several different works on the same topic, summarising the common elements, contrasting the differences, and extending the work in some fashion”. In the present study, the “Preferred Reporting Items for Systematic Reviews and Meta-Analyses” (PRISMA) is applied to perform the SLR to follow systematic and transparent steps for the research methodology, as shown in Fig.  1 . The PRISMA technique includes the identification, screening, eligibility, and exclusion criteria parts of the review process.

Additionally, examples of the data abstraction and analysis processes are provided (Mengist et al. 2020 ; Moher et al., 2015 ). The four main phases of the PRISMA process are eligibility, identification, screening, and data abstraction and analysis. Because the PRISMA technique employs sequential steps to accomplish the study’s purpose, it benefits SLR research. Moreover, the bibliometric analysis helps summarise the existing literature’s bibliographic data and determine the emerging condition of the intellectual structure and developing tendencies in the specified research domain (Dervis 2019 ).

3.1 Identification

The step to conduct the PRISMA is the identification of the relevant keywords to initiate the search for material. Next, search strings for the digital library’s search services are created using the selected keywords. The basic search query is for digital library article titles, keywords, and abstracts. Next, a Boolean AND or OR operator is used to generate the search string (Boolean combinations of the operators may also be used).

There are different search databases to conduct the review studies, such as Scopus, Sage, Web of Science, IEEE, and Google Scholar. Among all the available search databases, we have used the Scopus database to identify the articles; since 84% of the material on Web of Science (WoS) overlaps with Scopus, very few authors have addressed the benefits of adopting Scopus over WoS (Mongeon and Paul-Hus 2016 ). Scopus is widely used by academicians and researchers for quantitative analysis (Donthu et al. 2021 ). It is the biggest database of scientific research and contains citations and abstracts from peer-reviewed publications consisting of journal research articles, books and conference articles (Farooque et al., 2019 ; Dhayal et al., 2022 ; Brahmi et al., 2022 ). The following search term was used: (TITLE-ABS-KEY (“happiness economy” OR “economics of happiness” OR “happiness in economy” OR “economy of happiness” OR “economy of wellbeing” OR “wellbeing economy” OR “wellbeing in economy” OR “beyond growth”). This process yields 380 artciles in the initial phase.

3.2 Screening

The second phase is completed by all identified articles from the Scopus database obtained from the search string in the identification phase. The publications are either included or excluded throughout the screening process based on the standards established by the authors and with the aid of particular databases. Exclusion and inclusion criteria are shown during the screening phase to identify pertinent articles for the systematic review procedure. The timeline of this study’s selected articles is from 1995 to 2023. The first article related to the research domain was published in 1995. The second criterion for the inclusion includes the types of documents. In the present research, the authors have regarded only peer-reviewed journals and review articles. Other types of articles, such as books, book chapters, conference articles, notes, and editorials, are excluded to maintain the quality of the review. The third inclusion and exclusion criterion is based on language. All the non-English language documents are excluded to avoid translation confusion; hence, only the English language articles are considered for the final review. After the screening process, 297 articles are obtained.

3.3 Eligibility

Articles are manually selected or excluded depending on specific criteria specified by the authors during the eligibility process. During the elimination process, the authors excluded the articles that did not fit into the scope of review after manual screening of the articles. Two hundred fifty-seven articles were selected after the eligibility procedure. These selected articles are carefully reviewed for the study by reviewing the titles, abstracts, and standards from earlier screening processes.

3.4 Data abstraction and analysis

Analysis and abstraction of data are part of the fourth step. Finally, 257 papers were taken into account for final review. After that, the studies are culled to identify pertinent themes and subthemes for the current investigation by thoroughly reviewing each article’s text. An integrative review is a form of study that combines mixed, qualitative, and quantitative research procedures. It is carried out as shown in Fig.  1 . R-studio Bibliometrix and VOSviewer version 1.6.18 were used to evaluate the final study dataset corpus of 257 articles. Since the Bibliometrix software package is a free-source tool programmed in the R language. It is proficient of conducting comprehensive scientific mapping. It also contains several graphical and statistical features with flexible and frequent updates (Agrawal et al. 2023a , b ).

Extraction of articles and selection process

This section provides an answer to the first research question, RQ1, by indicating the main information of corpus data, research publication trends, influential prolific authors, journals, countries and most used keywords, etc. (Refer to Tables  2 , 3 and 4 ) and (Refer to Figs.  2 , 3 , 4 , 5 and 6 ).

4.1 Bibliometric analysis

Table  2 shows the relevant information gathered from the publication-related details. It presents the cognitive knowledge of the research area, for instance, details about authors, annual average publication, average citations and collaboration index. By observing the rate of document publishing, the study illustrates how much has already been done and how much remains to be investigated.

The annual publication trend is shown in Fig.  2 . It is reflected that the first article related to happiness in an economy was released in the year 1995 when (Bowling 1995 ) published the article “What things are important in people’s lives? A survey of the public’s judgements to inform scales of health related quality of life” where the article discussed “quality of life” and “happiness” as an essential component of a healthy life. Oswald ( 1997 ) brought the concept of happiness and economics together and raised questions such as “Does money buy happiness?” or “Do you think your children’s lives will be better than your own?”. Eventually, the gross national product of the past year and the coming year’s exchange rate was no longer the concern; instead, happiness as the sublime moment became more accurate (Schyns 1998 ; Easterlin, 2001; Frey and Stutzer, 2005 ). Post-2013, we can see exponential growth in the publication trend, and the reason behind the growth is the report published by the “ Stiglitz-Sen-Fitoussi” Commission, which has identified limitations of GDP and questioned the metric of wealth, economic and societal progress. The affirmed questions have gained the attention of researchers and organizations, and thus, they have explored the alternatives to GDP. As a result, the “Organization for Economic Co-operation and Development” (OECD) have proposed a wellbeing framework. Some research work has significantly impacted that time, contributing to the immense growth in this research area (Sangha et al. 2015 ; Spruk and Kešeljević, 2015 ; Nunes et al., 2016 ).

Publication trend

Table  3 shows the top prolific journals concerning the topmost publications in the domain of happiness economy for the corpus of 257 articles, namely “International Journal of Environmental Research and Public Health”, “Ecological Economics”, “Ecological Indicators”, “Sustainability” and “Journal of Cleaner Production” with 5, 4, 4,4 and 4 articles respectively (Refer to Table  4 ). Moreover, the most influential journals with maximum citations are “Nature Human Behavior”, “Quality of Life Research”, “Journal of Applied Behavior Analysis”, “Journal of Cleaner Production” and “Ecological Economics”, with 219, 205, 186, 154 and 142 citations, respectively. “Journal of Cleaner Production” and “Ecological Economics” are highly prolific and the most influential journals in the happiness economy research domain.

Table  4 shows the most influential authors. Baños, R.M. and Botella, C. are the two most contributing authors with maximum publications. For the maximum number of citations, Zheng G. and Coscieme L. are the topmost authors for their research work. The nations were sorted according to the quantity of publications, and Fig.  3 showed where the top ten countries with the highest number of publications are listed originated. It can be seen from the figure that the United Stated has contributed the maximum publications, 66, followed by the United Kingdom with 41 articles, followed by Germany with 32 articles. It is worth noting that emerging nation such as India and China have also made significant contributions.

Top ten contributing countries

Figure  4 shows semantic network analysis in which the relationships between words in individual texts are performed. In the present study, we have identified word frequency distributions and the co-occurrences of the authors’ keywords in this study. We employed co-word analysis to find repeated keywords or terms in the title, abstract, or body of a text. In Fig.  5 , the circle’s colour represents a particular cluster, and the circle’s radius indicates how frequently the words occur. The size of a keyword’s node indicates how frequently that keyword appears. The arcs connecting the nodes represent their co-occurrence in the same publication. The greater the distance between two nodes, the more often the two terms co-occur. It can be seen that “happiness” is linked with “growth” and “life satisfaction”. The nodes of “green economy”, “ecological economics”, and “climate change” are in a separate cluster that shows they are emerging areas, and future studies can explore the relationship between happiness economy with these keywords.

Co-ocurrance of author’s keyword (Author’s compilation)

4.2 Thematic map analysis through R studio

The thematic analysis map, as shown in Fig.  5 , displays, beneath the author’s keywords, the visualisation of four distinct topic typologies produced via a biblioshiny interface. The thematic map shows nine themes/clusters under four quadrants segregated in “Callon’s centrality” and “density value”. The degree of interconnectedness between networks is determined by Callon’s centrality, while Callon’s density determines the internal strength of networks. (Chen et al. 2019 ). The rectangular boxes in Fig.  5 represent the subthemes under each topic or cluster that are either directly or indirectly connected to the major themes, based on the available research. In the upper-right quadrant, four themes have appeared, namely “circular economy”, “well-being economy”, “depression”, and “sustainable development”, they fall under the category of motor themes since they are extremely pertinent to the research field, highly repetitious, and well-developed. When compared to other issues with internal linkages but few exterior relations, “urban population” in the upper-left quadrant is seen as a niche concern since it is not as significant. This cluster may have affected the urban population’s happiness (Knickel et al. 2021 ). “Social innovation” is categorised as an emerging or declining subject with low centrality and density, meaning it is peripheral and undeveloped. It is positioned in the lower-left quadrant. Last but not least, the transversal and fundamental themes “happiness economy”, “subjective well-being”, and “climate change” in the lower-right quadrant are seen to be crucial to the happiness economy study field but are still in the early stages of development. As a result, future research must place greater emphasis on the quantitative and qualitative growth of the study area in light of the key themes that have been identified.

Thematic map analysis

4.3 Science mapping through cluster analysis

In the study, science mapping was conducted to examine the interrelationship between the research domains that could be intellectual (Aria and Cuccurullo 2017 ; Donthu et al. 2021 ). It includes various techniques, such as co-authorship analysis, co-occurrence analysis, bibliographic coupling, etc. We have used R-Studio for the study’s temporal analysis by cluster analysis. To answer RQ2, the authors have performed a qualitative examination of the emerging cluster themes through the science mapping of the existing research corpus of 257 articles by performing bibliographic coupling of documents. Bibliographic coupling analysis helps identify clusters reflecting the most recent research themes in the happiness economy field to illuminate the field’s current areas of interest.

The visual presentation of science mapping relied on VoSviewer version 1.6.18 (refer to Fig.  6 ). Five significant clusters emerged in this research domain (refer to Table  5 ). Going beyond GDP: Transition towards happiness economy, rethinking growth for sustainability and ecological regeneration, beyond money and happiness policy, health, human capital and wellbeing and Policy-Push for happiness economy. A thorough examination identified cluster analyzes has also assists us in identifying potential future research proposals. (Franceschet 2009 )

4.4 Cluster 1: Going beyond GDP: transition towards happiness economy

It depicts from the green colour circles and nodes, where seven research articles were identified with a common theme of beyond GDP that can be seen in Fig.  6 . Cook and Davíðsdóttir ( 2021 ) investigated the linkages between the alternative measure of the beyond growth approach such as a well-being economy prespective and the SDGs. They proposed a conceptual model of a well-being economy consisting of four capital assets interrelated with SDGs that promote well-being goals and domains. To extend the concept of going beyond GDP, various economic well-being indicators are being aligned with the different economic, environmental, and social dimensions to target the set goals of SDG. It is found that the “Genuine Progress Indicator” (GPI) is consider as the most extensive method that covers the fourteen targets among the seventeen’s SDG’s. Cook et al. ( 2022 ) consider SDGs to represent the classical, neoclassical and growth-based economy model and as an emerging paradigm for a well-being economy. The significance of GDP is more recognized within the goals of sustainable development.

GPI is considered an alternative indicator of economic well-being. On this basis, excess consumption of high-quality energy will expand macro-economic activity, which GDP measures. For such, a conceptual exploration of the study is conducted on how pursuing “Sustainable Energy Development” (SED) that can increase the GPI results. As the study’s outcome, according to the GPI, SED will have a significant advantage in implementing energy and environment policy and will also contribute to the advancement of social and economic well-being. Coscieme et al. ( 2020a ) explored the connection between the unconditional growth of GDP and SDG. The author considered that policy coherence for sustainable development should lessen the damaging effects of cyclic manufacturing on the ecosystem. Thus, the services considered free of charge in the GDP model should be valued as a component of society. Generally, such services include ecosystem services and a myriad of “economic” functions like rainfall and carbon sequestration. To work for SDG 8, defined by the “United Nations Sustainable Development Goals” (UNSDGs), a higher GDP growth rate would eventually make it more difficult to achieve environmental targets and lessen inequality. Various guidelines were proposed to select alternative variables for SDG-8 to enhance coherence among all the SDG and other policies for sustainability.

Fioramonti et al. ( 2019a ) state their focus is to go beyond GDP toward a well-being economy rather than material output with the help of convergence reforms in policies and economic shifts. To achieve the SDG through protecting the environment, promoting equality, equitable development and sharing economy. The authors have developed the Sustainable Well-Being Index (SWBI) to consolidate the “Beyond GDP” streams as a metric of well-being matched with the objectives to achieve SDG. The indicators of well-being for an economy have enough possibility to connect current transformations in the economic policies and the economy that, generally, GDP is unable to capture.

Fioramonti et al. ( 2022a ) investigate the critical features of the Wellbeing Economy (WE), including its various parameters like work, technology, and productivity. Posting a WE framework that works for mainstream post-growth policy at the national and international levels was the study’s primary goal. The authors have focused on building a society that promotes well-being that should be empowering, adaptable, and integrative. A well-being economic model should develop new tools and indicators to monitor all ecological and human well-being contributors. A multidimensional approach including critical components for a well-being economy was proposed that creates value to re-focus on economic, societal, personal, and natural aspects. Rubio-Mozos et al. ( 2019 ) conducted in-depth interviews with Fourth Sector business leaders, entrepreneurs, and academicians to investigate the function of small and medium-sized businesses and the pressing need to update the economic model using a new measure in line with UN2030. They have proposed a network from “limits to growth” to a “sustainable well-being economy”.

4.5 Cluster 2: Rethinking growth for sustainability and ecological regeneration

Figure  6 depicts it from blue circles and nodes, wherein four papers were identified. Knickel et al. ( 2021 ) proposed an analytical approach by collecting the data from 11 European areas to examine the existing conditions, difficulties, and anticipated routes forward. The goal of the study is to define the many ideas of a sustainable well-being economy and territorial development plans that adhere to the fundamental characteristics of a well-being economy. A transition from a conventional economic viewpoint to a broader view of sustainable well-being is centred on regional development plans and shifting rural-urban interactions.

Pillay ( 2020 ) investigates the new theories of de-growth, ecosocialism, well-being and happiness economy to break the barriers of traditional economic debates by investigating ways to commercialise and subjugate the state to a society in line with non-human nature. The significant indicator of Gross National Happiness (GNH) is an alternative working indicator of development; thus, the Chinese wall between Buddha and Marx has been built. They questioned the perspective of Buddha and Marx, whether they were harmonized or became a counter-hegemonic movement. In order to determine if the happiness principle is grounded in spiritual values and aligns with the counter-hegemonic ecosocialist movement, the author examined the ecosocialist perspective. Shrivastava and Zsolnai ( 2022 ) have investigated the theoretical and practical ramifications of creative organisations for well-being rooted in the drive for a well-being economy. Wellbeing and happiness-focused economic frameworks are emerging primarily in developed countries. This new policy framework also abolishes GDP-based economic growth and prioritizes individual well-being and ecological regeneration. To understand its application and interpretation, Van Niekerk ( 2019 ) develops a conceptual framework and theoretical analysis of inclusive economics. It contributes to developing a new paradigm for economic growth, both theoretically and practically.

4.6 Cluster 3: ‘Beyond money’ and happiness policy

It depicts pink circles and nodes, wherein five articles were identified, as shown in Fig.  6 . According to Diener and Seligman ( 2004a ) economic indicators are critical in the early phases of economic growth when meeting basic requirements is the primary focus. However, as society becomes wealthier, an individual’s well-being becomes less dependent on money and more on social interactions and job satisfaction. Individuals reporting high well-being outperform those reporting low well-being in terms of income and performance. A national well-being index is required to evaluate well-being variables and shape policies systematically. Diener and Seligman ( 2018 ) propounded the ‘Beyond Money’ concept in 2004. In response to the shortcomings of GDP and economic measures, other quality-of-life indicators, such as health and education, have been created. The national account of well-being has been proposed as a common path to provide societies with an overall quality of life metric. While measuring the subjective well-being of people, the authors reasoned a societal indicator of the quality of life. In this article, the authors have proposed an economy of well-being model by combining subjective and objective measures to convince policymakers and academicians to enact policies that enhance human welfare. The well-being economy includes quality of life indicators and life satisfaction, subjective well-being and happiness.

Frey and Stutzer ( 2000 ) perceived the microeconomic well-being variables in countries. In the study, survey data was used from 6000 individuals in Switzerland and showed that the individuals are happier in developed democracies and institutions (government federalization). They analyzed the reported subjective well-being data to determine the function of federal and democratic institutions on an individual’s satisfaction with life. The study found a negative relationship between income and unemployment. Three criteria have been employed in the study to determine happiness: demographic and psychological traits, macro- and microeconomic factors, and constitutional circumstances. Thus, a new pair of determinants reflects happiness’s effect on individuals’ income, unemployment, inflation and income growth.

Happiness policy, according to Frey and Gallus ( 2013b ), is an intrinsic aspect of the democratic process in which various opinions are collected and examined. “Happiness policy” is far more critical than continuing a goal such as increasing national income and instead considered an official policy goal. The article focuses on how politicians behave differently when they believe that achieving happiness is the primary objective of policy. Frey et al. ( 2014 ) explored the three critical areas of happiness, which are positive and negative shocks on happiness, choice of comparison and its extent to derive the theoretical propositions that can be investigated in future research. It discussed the areas where a more novel and comprehensive theoretical framework is needed: comparison, adaptation, and happiness policy. Wolfgramm et al. ( 2020 ) derived a value-driven transformation framework in Māori economics of wellbeing. It contributes to a multilevel and comprehensive review of Māori economics and well-being. The framework is adopted to advance the policies and implement economies of well-being.

4.7 Cluster 4: Health, human capital and wellbeing

It is depicted as a red colour circle and nodes in Fig.  6 , and only three papers on empirical investigations were found. Laurent et al. ( 2022 ) investigated the Health-Environment Nexus report published by the “Wellbeing Economy Alliance”. In place of increased production and consumption, they suggested a comprehensive framework for human health and the environment that includes six essential paths. The six key pathways are well-being energy, sustainable food, health care, education, social cooperation and health-environment nexus. The proposed variables yield the co-benefits for the climate, health and sustainable economy. Steer clear of the false perception of trade-offs, such as balancing the economy against the environment or the need to save lives. McKinnon and Kennedy ( 2021 ) focuses on community economics of well-being that benefits entrepreneurs and employees. They investigated the interactions of four social enterprises that work for their employees inside and within the broader community. Cylus et al. ( 2020 ) proposed the opportunities and challenges in adopting the model of happiness or well-being in an economy as an alternative measure of GDP. Orekhov et al. ( 2020 ) proposed the derivation of happiness from the World Happiness Index (WHI) data to estimate the regression model for developed countries.

4.8 Cluster 5: Policy-push for happiness economy

It is depicted as an orange circle and nodes in Fig.  6 , and only five papers on empirical and review investigations were found. Oehler-Șincai et al. ( 2023 ) proposed the conceptual and practical perspective of household-income-labour dynamics for policy formulation. It discusses the measurement of well-being as a representation of various policies focusing on health, productivity, and longevity. It focuses on the role of policy in building the subjective and objective dimensions of well-being, defines the correlation between well-being, employment policies, and governance, is inclined to the well-being performance of various countries, and underscores present risks that jeopardize well-being. Musa et al. ( 2018 ) have developed a “community happiness index” by incorporating the four aspects of sustainability—economic, social, environmental, and urban governance—as well as the other sustainability domains, such as human well-being and eco-environmental well-being. From then onwards, community happiness and sustainable urban development emerged. Chernyahivska et al. ( 2020 ) developed strategies to raise the standard of living for people in countries undergoing economic transition by using the quality of life index. The methods uncovered are enhancing employment opportunities and uplifting the international labour market in urban and rural areas, prioritizing human capital, eliminating gender inequality, focusing on improving the individual’s health, and enhancing social protection. Zheng et al. ( 2019 ) investigated the livelihood and well-being index of the population that makes liveable conditions and city construction in society based on people’s happiness index. The structure of a liveable city should be emphasised on sustainable development. The growth strategy in urban areas is an essential aspect of building a liveable city. Frey and Gallus ( 2013a ) criticised the National Happiness Index as a policy goal in a country because it cannot be measured and thus fails to measure the true happiness of people. To measure real happiness, the government should establish living conditions that enable individuals to become happy. The rule of law and human rights must support the process.

The structure of a liveable city should be emphasized in sustainable development. The growth strategy in urban areas is an essential aspect of building a liveable city. Frey and Gallus ( 2013a ) criticized the National Happiness Index as a policy goal in a country because it cannot be measured and thus fails to identify the true individuals happiness. To measure real happiness, the government should establish living conditions that enable individuals to be happy. The process needs to be supported by human rights and the rule of law.

Visualization of cluster analysis

5 Discussion of findings

Concerns like the improved quality-of-life and a decent standard of living within the ecological frontier of the environment have various effects on individuals overall well-being and life satisfaction. The ‘beyond growth’ approach empathized with the revised concept of growth, which is based on the idea of maximising happiness for a larger number of people rather than being driven by a desire for financial wealth or production. In that aspect, the notion of happiness economy is designed that prioritizes serving both people and the environment over the other. This present article has focused on the beyond growth approach and towards a new economic paradigm by doing bibliometric and visual analysis on the dataset that was obtained from Scopus, helping to determine which nations, publications, and authors were most significant in this field of study.

In this field of study, developed nations have made significant contributions as compared to the developing nations. In total, 59 countries have made the substantial contribution to the beyond growth approach literature an some of them have proposed their respective national well-being economy framwework. Among 59 countries the United States and the United Kingdom have been crucial to the publishing. With the exception of five of the top 10 nations, Europe contributes the most to scientific research. The existing research shows the inclination of developed and developing countries to build a new economic paradigm that goes beyond growth by prioritizing the happiness level at individual as well as at collective level.

The most prolific journals in this research domain are the “International Journal of Environmental Research” and “Public Health” with the total publication of 5 and 4. The top two cited journals were the “ Nature Human Behavior” with 219 citations and the “Quality of Life Research” with 205 citations. Due to various economic and non-economic factors, these journals struggled to strike a balance between scientific accuracy and timeliness, and it became vital to spread accurate and logical knowledge. For, example, discussing the relationship between inequality and well-being, exploring the challenges and opportunites of happiness economy in different countries, assessing the role of health in all policies to support the transition to the well-being economy. Visualization of semantic network analysis of co-ocurrance of authors keywords from the VOSviewer showed the future research scope to explore the association between happiness economy along with green economy, climate change, spirituality and sustainability. However, in the thematic mapping, the motor themes denotes the themes that are well-developed and repetative in research, such as, well-being economy, depression, sustainable development and circular economy. The basic themes depicts the developing and transveral themes such as happiness economy, subjective well-being and climate condition. As a result, future research must place greater emphasis on the theoretical and practical expansion of the research field in view of the determined major subjects.

The present study have performed the cluster analysis to identify the emerging research themes in this domain through VOSviewer that helps to analyze the network of published documents. Based on published papers, the author can analyse the interconnected network structure with the use of cluster analysis. We have identified the top five clusters from the study. Each cluster denote the specific and defined theme of the research in this domain. In cluster 1, the majorly of the authors are working in the area of going beyond GDP and transition towards happiness economy, which consists of empirical and review studies. Cluster 2 represents that authors are exploring the relationship between rethinking growth for sustainability and ecological regeneration to evaluate the transition from a conventional economic thought to a broader view of sustainable well-being which is centred on regional development plans and shifting rural-urban interactions. In cluster 3, the authors are exploring the beyond money and happiness policy themes and identified the shortcomings of GDP and economic measures, other quality-of-life indicators, such as health and education. They have proposed the well-being index to evaluate the well-being variables and shape socio-economic policies systematically. The authors have proposed an economy of well-being model by combining subjective and objective measures to convince policymakers and academicians to enact policies that enhance human welfare. The well-being economy includes quality of life indicators and life satisfaction, subjective well-being and happiness. In cluster 4, the authors are working of related theme of Health, human capital and wellbeing, whereby they have put up a comprehensive framework for health and the environment that includes several important avenues for prioritising human and ecological well-being over increased production and consumption. In cluster 5, the authors have suggested the policy-push for happiness economy in which they have identified the conceptual and practical perspective of household-income-labour dynamics for policy formulation. Majorly of the authors in this clutster have focused on the role of policy in building the subjective and objective dimensions of well-being, defines the correlation between well-being, employment policies, and governance, is inclined to the well-being performance of various countries, and underscores present risks that jeopardize well-being. Hence, the present study will give academics, researchers, and policymakers a thorough understanding of the productivity, features, key factors, and research outcomes in this field of study.

6 Scope for future research avenues

The emergence of a happiness economy will transform society’s traditional welfare measure. Such changes will generate more reliable and practical means to measure the well-being or welfare of an economy. After a rigorous analysis of the existing literature, we have proposed the scope for future research in Table  6 .

7 Conclusion

In 2015, the United Nations proposed the pathbreaking and ambitious seventeen “Sustainable Development Goals” (SDGs) for countries to steer their policies toward achieving them by 2030. In reality, economic growth remains central to the agenda for SDGs, demonstrating the absence of a ground-breaking and inspirational vision that might genuinely place people and their happiness at the core of a new paradigm for development. As this research has reflect, there are various evidence that the happiness economy strategy is well-suited to permeate policies geared towards sustainable development. In this context, ‘happiness’ may be a strong concept that ensures the post-2030 growth will resonate with the socioeconomic and environmental traits of everyone around the world while motivating public policies for happiness.

The current research has emphasized the many dynamics of the happiness economy by using a bibliometric analytic study of 257 articles. We have concluded that the happiness economy is an emerging area that includes different dimensions of happiness, such as ecological regeneration, circular economy, sustainability, sustainable well-being, economic well-being, subjective well-being, and well-being economy. In addition to taking into consideration the advantages and disadvantages of human participation in the market, a happiness-based economic system would offer new metrics to assess all contributions to human and planetary well-being. In terms of theoretical ramifications, we suggest that future scholars concentrate on fusing the welfare and happiness theory with economic policy. As countries are predisposed to generate disharmony and imbalance, maximizing societal well-being now entails expanding sustainable development. Since the happiness economy is still a relatively novel field, it offers numerous potential research opportunities.

8 Limitations

Similar to every other research, this one has significant restrictions as well. We are primarily concerned that all our data were extracted from the Scopus database. Furthermore, future research can utilize other software like BibExcel and Gephi to expound novel variables and linkages. Given the research limitations, this article still provides insightful and relevant direction to policymakers, scholars, and those intrigued by the idea of happiness and well-being in mainstream economics.

The study offers scope for future research in connecting the happiness economy framework with different SDGs. Future studies can also carry empirical research towards creating a universally acceptable ‘happiness economy index’ with human and planetary well-being at its core.

Data availability

Data not used in this article.

Abrar, R.: Building the transition together: WEAll’s perspective on creating a Wellbeing Economy. Well-Being Transition. 157–180 (2021). https://doi.org/10.1007/978-3-030-67860-9_9/COVER

Agrawal, R., Agrawal, S., Samadhiya, A., Kumar, A., Luthra, S., Jain, V.: Adoption of green finance and green innovation for achieving circularity: An exploratory review and future directions. Geosci. Front. 101669 (2023a). https://doi.org/10.1016/J.GSF.2023.101669

Agrawal, S., Sharma, N., Singh, M.: Employing CBPR to understand the well-being of higher education students during covid-19 lockdown in India. SSRN Electron. J. (2020). https://doi.org/10.2139/ssrn.3628458

Agrawal, S., Sharma, N.: Beyond GDP: A movement toward happiness economy to achieve sustainability. Sustain. Green. Future. 95–114 (2023). https://doi.org/10.1007/978-3-031-24942-6_5

Agrawal, S., Sharma, N., Bruni, M.E., Iazzolino, G.: Happiness economics: Discovering future research trends through a systematic literature review. J. Clean. Prod. 416 , 137860 (2023b). https://doi.org/10.1016/j.jclepro.2023.137860

Article   Google Scholar  

Aiyar, S., Ebeke, C.: Inequality of opportunity, inequality of income and economic growth. World Dev. 136 , 105115 (2020). https://doi.org/10.1016/J.WORLDDEV.2020.105115

Armstrong, C.M.J., Connell, K.Y.H., Lang, C., Ruppert-Stroescu, M., LeHew, M.L.A.: Educating for sustainable fashion: using clothing acquisition abstinence to explore sustainable consumption and life beyond growth. J Consum Policy. 39 (4), 417–439 (2016). https://doi.org/10.1007/s10603-016-9330-z

Approaches to Improving the Quality of Life: How to Enhance the Quality of Life - Abbott L. Ferriss - Google Books . (n.d.). Retrieved April 25, from (2023). https://books.google.co.in/books?hl=en&lr=&id=9AKdtNzGsGcC&oi=fnd&pg=PR8&dq=equality+tends+to+improve+major+objective+ways+of+wellbeing,+from+healthier+communities+to+happier+communities,+from+declining+hate+and+crime+and+to+improved+social+cohesion,+productivity,+unity+and+interpersonal+trust&ots=pZ5kbKdqrC&sig=vfwoVTo2Aur-nV9J9HNF4rbF74o&redir_esc=y#v=onepage&q&f=false

Aria, M., Cuccurullo, C.: Bibliometrix: An R-tool for comprehensive science mapping analysis. J. Informetrics. 11 (4), 959–975 (2017). https://doi.org/10.1016/J.JOI.2017.08.007

Atkinson, S., Fuller, S., Painter, J.: Wellbeing and place, pp. 1–14. Ashgate Publishing (2012). https://researchers.mq.edu.au/en/publications/wellbeing-and-place

Bengtsson, M., Alfredsson, E., Cohen, M., Lorek, S., Schroeder, P.: Transforming systems of consumption and production for achieving the sustainable development goals: moving beyond efficiency. Sustain. Sci. 13 (6), 1533–1547 (2018). https://doi.org/10.1007/s11625-018-0582-1

Bayani, E., Ahadi, F., Beigi, J.: The preventive impact of Happiness Economy on Financial Delinquency. Political Sociol. Iran. 5 (11), 4651–4670 (2023). https://doi.org/10.30510/PSI.2022.349645.3666

Better Life Initiative: Measuring Well-Being and Progress - OECD . (n.d.). Retrieved December 8, from (2022). https://www.oecd.org/wise/better-life-initiative.htm

Bowler, D.E., Buyung-Ali, L.M., Knight, T.M., Pullin, A.S.: A systematic review of evidence for the added benefits to health of exposure to natural environments. BMC Public. Health. 10 (1), 1–10 (2010). https://doi.org/10.1186/1471-2458-10-456/TABLES/1

Bowling, A.: What things are important in people’s lives? A survey of the public’s judgements to inform scales of health related quality of life. Soc. Sci. Med. 41 (10), 1447–1462 (1995). https://doi.org/10.1016/0277-9536(95)00113-L

Boyd, J.: Nonmarket benefits of nature: What should be counted in green GDP? Ecol. Econ. 61 (4), 716–723 (2007). https://doi.org/10.1016/J.ECOLECON.2006.06.016

Brahmi, M., Aldieri, L., Dhayal, K.S., Agrawal, S.: Education 4.0: can it be a component of the sustainable well-being of students? pp. 215–230 (2022). https://doi.org/10.4018/978-1-6684-4981-3.ch014

Carrero, G.C., Fearnside, P.M., Valle, D. R., de Alves, S., C: Deforestation trajectories on a Development Frontier in the Brazilian Amazon: 35 years of settlement colonization, policy and economic shifts, and Land Accumulation. Environ. Manage. 2020. 66:6 (6), 966–984 (2020). https://doi.org/10.1007/S00267-020-01354-W 66

Chen, C.W.: Can smart cities bring happiness to promote sustainable development? Contexts and clues of subjective well-being and urban livability. Developments Built Environ. 13 , 100108 (2023). https://doi.org/10.1016/J.DIBE.2022.100108

Chen, X., Lun, Y., Yan, J., Hao, T., Weng, H.: Discovering thematic change and evolution of utilizing social media for healthcare research. BMC Med. Inf. Decis. Mak. 19 (2), 39–53 (2019). https://doi.org/10.1186/S12911-019-0757-4/FIGURES/10

Chernyahivska, V.V., Bilyk, O.I., Charkina, A.O., Zhayvoronok, I., Farynovych, I.V.: Strategy for improving the quality of life in countries with economies in transition. Int. J. Manag. 11 (4), 523–531 (2020).

Clark, A.E., Flèche, S., Senik, C.: Economic growth evens out happiness: evidence from six surveys. Rev Income Wealth. 62 (3), 405–419 (2016). https://doi.org/10.1111/roiw.12190

Construction strategies and evaluation models of livable city based on the happiness index | IEEE Conference Publication | IEEE Xplore . (n.d.). Retrieved April 1, from (2023). https://ieeexplore.ieee.org/abstract/document/6640911

Cook, D., Davíðsdóttir, B.: An appraisal of interlinkages between macro-economic indicators of economic well-being and the sustainable development goals. Ecol. Econ. 184 (2021). https://doi.org/10.1016/j.ecolecon.2021.106996

Cook, D., Davíðsdóttir, B., Gunnarsdóttir, I.: A conceptual exploration of how the pursuit of sustainable Energy Development is implicit in the genuine Progress Indicator. Energies. 15 (6) (2022). https://doi.org/10.3390/en15062129

Coscieme, L., Sutton, P., Mortensen, L.F., Kubiszewski, I., Costanza, R., Trebeck, K., Pulselli, F.M., Giannetti, B.F., Fioramonti, L.: Overcoming the myths of mainstream economics to enable a newwellbeing economy. Sustain. (Switzerland). 11 (16) (2019). https://doi.org/10.3390/su11164374

Coscieme, L., Mortensen, L.F., Anderson, S., Ward, J., Donohue, I., Sutton, P.C.: Going beyond gross domestic product as an indicator to bring coherence to the Sustainable Development Goals. J. Clean. Prod. 248 , 119232 (2020). https://doi.org/10.1016/J.JCLEPRO.2019.119232

Coscieme, L., Mortensen, L.F., Anderson, S., Ward, J., Donohue, I., Sutton, P.C.: Going beyond gross domestic product as an indicator to bring coherence to the Sustainable Development Goals. J. Clean. Prod. 248 (2020a). https://doi.org/10.1016/j.jclepro.2019.119232

Costanza, R.: Ecological economics in 2049: Getting beyond the argument culture to the world we all want. Ecol. Econ. 168 , 106484 (2020). https://doi.org/10.1016/J.ECOLECON.2019.106484

Costanza, R., Caniglia, E., Fioramonti, L., Kubiszewski, I., Lewis, H., Lovins, H., McGlade, J., Mortensen, L.F., Philipsen, D., Pickett, K.E., Ragnarsdottir, K.V., Roberts, D.: Toward a Sustainable Wellbeing Economy. Solutions: For a Sustainable and Desirable Future . (2020). https://openresearch-repository.anu.edu.au/handle/1885/205271

Coyne, C.J., Boettke, P.J.: Economics and Happiness Research: Insights from Austrian and Public Choice Economics. Happiness Public. Policy. 89–105 (2006). https://doi.org/10.1057/9780230288027_5

Cylus, J., Smith, P.C., Smith, P.C.: The economy of wellbeing: What is it and what are the implications for health? BMJ. 369 (2020). https://doi.org/10.1136/bmj.m1874

Dervis, H.: Bibliometric analysis using bibliometrix an R package. J. Scientometr. Res. 8 (3), 156–160 (2019). https://doi.org/10.5530/JSCIRES.8.3.32

Dhayal, K.S., Brahmi, M., Agrawal, S., Aldieri, L., Vinci, C.P.: A paradigm shift in education systems due to COVID-19, pp. 157–166 (2022)

Diener, E.: Subjective well-being. Psychol. Bull. 95 (3), 542–575. (1984). https://doi.org/10.1037/0033-2909.95.3.542. https://psycnet.apa.org/record/1984-23116-001

Diener, E., Seligman, M.E.P.: Beyond money: Toward an economy of well-being. Psychological Science in the Public Interest, Supplement , 5 (1). (2004a). https://www.scopus.com/inward/record.uri?eid=2-s2.0-3142774261&partnerID=40&md5=e86b2c930837502a9ce9cbd057c0df82

Diener, E., Seligman, M.E.P.: Beyond money: Toward an economy of well-being. Psychol. Sci. Public. Interest. 5 (1), 1–31 (2004b). https://doi.org/10.1111/j.0963-7214.2004.00501001.x

Diener, E., Seligman, M.E.P.: Beyond money: Progress on an economy of well-being. Perspect. Psychol. Sci. 13 (2), 171–175 (2018). https://doi.org/10.1177/1745691616689467

Dolan, P., Peasgood, T., White, M.: Do we really know what makes us happy? A review of the economic literature on the factors associated with subjective well-being. J. Econ. Psychol. 29 (1), 94–122 (2008). https://doi.org/10.1016/J.JOEP.2007.09.001

Donthu, N., Kumar, S., Mukherjee, D., Pandey, N., Lim, W.M.: How to conduct a bibliometric analysis: An overview and guidelines. J. Bus. Res. 133 , 285–296 (2021). https://doi.org/10.1016/j.jbusres.2021.04.070

Easterlin, R.A.: Will raising the incomes of all increase the happiness of all? J. Econ. Behav. Organ. 27 (1), 35–47 (1995). https://doi.org/10.1016/0167-2681(95)00003-B

Easterlin, R.A.: Happiness and economic growth - the evidence. In: Global Handbook of Quality of Life, pp. 283–299. Springer, Netherlands (2015). https://doi.org/10.1007/978-94-017-9178-6_12

Fang, Z., Wang, H., Xue, S., Zhang, F., Wang, Y., Yang, S., Zhou, Q., Cheng, C., Zhong, Y., Yang, Y., Liu, G., Chen, J., Qiu, L., Zhi, Y.: A comprehensive framework for detecting economic growth expenses under ecological economics principles in China. Sustainable Horizons. 4 , 100035 (2022). https://doi.org/10.1016/J.HORIZ.2022.100035

Farooque, M., Zhang, A., Thürer, M., Qu, T., Huisingh, D.: Circular supply chain management: a definition and structured literature review. J. Clean. Prod. 228 , 882–900 (2019). https://doi.org/10.1016/J.JCLEPRO.2019.04.303

Ferriss, A.L.: Approaches to improving the quality of life?: how to enhance the quality of life. 150 (2010)

Fioramonti, L., Coscieme, L., Mortensen, L.F.: From gross domestic product to wellbeing: How alternative indicators can help connect the new economy with the Sustainable Development Goals: Https://Doi.Org/10.1177/2053019619869947 , 6 (3), 207–222. (2019). https://doi.org/10.1177/2053019619869947

Fioramonti, L., Coscieme, L., Mortensen, L.F.: From gross domestic product to wellbeing: How alternative indicators can help connect the new economy with the Sustainable Development Goals. Anthropocene Rev. 6 (3), 207–222 (2019a). https://doi.org/10.1177/2053019619869947

Fioramonti, L., Coscieme, L., Costanza, R., Kubiszewski, I., Trebeck, K., Wallis, S., Roberts, D., Mortensen, L.F., Pickett, K.E., Wilkinson, R., Ragnarsdottír, K.V., McGlade, J., Lovins, H., De Vogli, R.: Wellbeing economy: An effective paradigm to mainstream post-growth policies? Ecol. Econ. 192 , 107261 (2022b). https://doi.org/10.1016/j.ecolecon.2021.107261

Fioramonti, L., Coscieme, L., Costanza, R., Kubiszewski, I., Trebeck, K., Wallis, S., Roberts, D., Mortensen, L.F., Pickett, K.E., Wilkinson, R., Ragnarsdottír, K.V., McGlade, J., Lovins, H., De Vogli, R.: Wellbeing economy: An effective paradigm to mainstream post-growth policies? Ecol. Econ. 192 (2022a). https://doi.org/10.1016/j.ecolecon.2021.107261

Forgeard, M.J.C., Jayawickreme, E., Kern, M.L., Seligman, M.E.P.: Doing the right thing: measuring wellbeing for public policy. Int J Wellbeing. 1 (1), 79–106 (2011). https://doi.org/10.5502/ijw.v1i1.15

Francart, N., Malmqvist, T., Hagbert, P.: Climate target fulfilment in scenarios for a sustainable Swedish built environment beyond growth. Futures.  98 , 1–18 (2018). https://doi.org/10.1016/J.FUTURES.2017.12.001

Franceschet, M.: A cluster analysis of scholar and journal bibliometric indicators. J. Am. Soc. Inform. Sci. Technol. 60 (10), 1950–1964 (2009). https://doi.org/10.1002/ASI.21152

Frey, B.S., Gallus, J.: Happiness policy and economic development. Int. J. Happiness Dev. 1 (1), 102 (2012). https://doi.org/10.1504/IJHD.2012.050835

Frey, B.S., Gallus, J.: Political economy of happiness. Appl. Econ. 45 (30), 4205–4211 (2013a). https://doi.org/10.1080/00036846.2013.778950

Frey, B.S., Gallus, J.: Subjective well-being and policy. Topoi. 32 (2), 207–212 (2013b). https://doi.org/10.1007/S11245-013-9155-1/METRICS

Frey, B.S., Stutzer, A.: Happiness, economy and institutions. Econ. J. 110 (466), 918–938 (2000). https://doi.org/10.1111/1468-0297.00570

Frey, B.S., Stutzer, A.: What can economists learn from Happiness Research? Source: J. Economic Literature. 40 (2), 402–435 (2002). https://doi.org/10.1257/002205102320161320

Frey, B.S., Stutzer, A.: Happiness research: State and prospects. Rev. Soc. Econ. 63 (2), 207–228 (2005). https://doi.org/10.1080/00346760500130366

Frey, B.S., Gallus, J., Steiner, L.: Open issues in happiness research. Int. Rev. Econ. 61 (2), 115–125 (2014). https://doi.org/10.1007/s12232-014-0203-y

Frijters, P., Clark, A.E., Krekel, C., Layard, R.: A happy choice: Wellbeing as the goal of government. Behav. Public. Policy. 4 (2), 126–165 (2020). https://doi.org/10.1017/BPP.2019.39

Galbraith, J.K.: Inequality, unemployment and growth: new measures for old controversies. J. Econ. Inequal. 7 (2), 189–206 (2009). https://doi.org/10.1007/s10888-008-9083-2

Giannetti, B.F., Agostinho, F., Almeida, C.M.V.B., Huisingh, D.: A review of limitations of GDP and alternative indices to monitor human wellbeing and to manage eco-system functionality. J. Clean. Prod. 87 (1), 11–25. (2015). https://doi.org/10.1016/j.jclepro.2014.10.051

Gilead, T.: Education’s role in the economy: towards a new perspective. 47 (4), 457–473. (2016). https://doi.org/10.1080/0305764X.2016.1195790

Griep, Y., Hyde, M., Vantilborgh, T., Bidee, J., De Witte, H., Pepermans, R.: Voluntary work and the relationship with unemployment, health, and well-being: A two-year follow-up study contrasting a materialistic and psychosocial pathway perspective. J. Occup. Health Psychol. 20 (2), 190–204 (2015). https://doi.org/10.1037/A0038342

Hallberg, M., Kullenberg, C.: Happiness studies. Nord. J. Work. Life Stud. 7 (1), 42–50 (2019). https://doi.org/10.5324/NJSTS.V7I1.2530

Helliwell, J., Layard, R., Sachs, J., Neve, J.-E.: World Happiness Report 2021. Happiness and Subjective Well-Being . (2021). https://www.wellbeingintlstudiesrepository.org/hw_happiness/5

Ikeda, D. (2010). A New Humanism: The University Addresses of Daisaku Ikeda - Daisaku Ikeda - Google Books . books.google.co.in/books?hl = en&lr=&id = 17aKDwAAQBAJ&oi = fnd&pg = PP1&ots = gQvBHjJA7P&sig = wVOxQ_XlCIrj39Q08W-kxc_sPjA&redir_esc = y#v = onepage&q&f = false

Inglehart, R., Foa, R., Peterson, C., Welzel, C.: Development, freedom, and rising happiness: a global perspective (1981–2007). 3 (4), 264–285 (2008). https://doi.org/10.1111/j.1745-6924.2008.00078.x

Kahneman, D., Krueger, A.B.: Developments in the measurement of subjective well-being. J. Econ. Perspect. 20 (1), 3–24 (2006). https://doi.org/10.1257/089533006776526030

Keniger, L.E., Gaston, K.J., Irvine, K.N., Fuller, R.A.: What are the benefits of interacting with nature? Int. J. Environ. Res. Public Health. 10 (3), 913–935 (2013). https://doi.org/10.3390/IJERPH10030913

Kinman, G., Jones, F.: A life beyond work? job demands, work-life balance, and wellbeing in UK academics. 17 (1–2), 41–60 (2008). https://doi.org/10.1080/10911350802165478

Kim, K.H., Kang, E., Yun, Y.H.: Public support for health taxes and media regulation of harmful products in South Korea. BMC Public. Health. 19 (1), 1–12 (2019). https://doi.org/10.1186/S12889-019-7044-2/TABLES/5

King, M.F., Renó V.F., Novo, E.M.L.M.: The concept, dimensions and methods of assessment of human well-being within a socioecological context: a literature review. Soc Indic Res. 116 (3), 681–698 (2014). https://doi.org/10.1007/s11205-013-0320-0

Knickel, K., Almeida, A., Galli, F., Hausegger-Nestelberger, K., Goodwin-Hawkins, B., Hrabar, M., Keech, D., Knickel, M., Lehtonen, O., Maye, D., Ruiz-Martinez, I., Šūmane, S., Vulto, H., Wiskerke, J.S.C.: Transitioning towards a sustainable wellbeing economy—implications for rural–urban relations. Land. 10 (5) (2021). https://doi.org/10.3390/land10050512

Kullenberg, C., Nelhans, G.: The happiness turn? Mapping the emergence of happiness studies using cited references. Scientometrics. 103 (2), 615–630 (2015). https://doi.org/10.1007/S11192-015-1536-3/FIGURES/5

Lama, D.: A human approach to world peace: his holiness the Dalai Lama. J. Hum. Values. 18 (2), 91–100 (2012). https://doi.org/10.1177/0971685812454479

Laurent, É., Galli, A., Battaglia, F., Libera Marchiori, D., G., Fioramonti, L.: Toward health-environment policy: Beyond the Rome Declaration. Global Environmental Change , 72 . (2022). https://doi.org/10.1016/j.gloenvcha.2021.102418

Lauzon, C., Stevenson, A., Peel, K., Brinsdon, S.: A “bottom up” health in all policies program: supporting local government wellbeing approaches. Health Promot J Austr. (2023). https://doi.org/10.1002/hpja.712

Lavrov, I.: Prospect as a model of the future in the happiness economy - new normative theory of wellbeing. Published Papers (2010). https://ideas.repec.org/p/rnp/ppaper/che5.html

McKinnon, K., Kennedy, M.: Community economies of wellbeing: How social enterprises contribute to surviving well together. In: Social Enterprise, Health, and Wellbeing: Theory, Methods, and Practice. Taylor and Francis Inc (2021). https://doi.org/10.4324/9781003125976-5

Mengist, W., Soromessa, T., Legese, G.: Method for conducting systematic literature review and meta-analysis for environmental science research. MethodsX. 7 (2020). https://doi.org/10.1016/j.mex.2019.100777

Meredith, J.: Theory building through conceptual methods. Int. J. Oper. Prod. Manage. 13 (5), 3–11 (1993). https://doi.org/10.1108/01443579310028120

Millward-Hopkins, J., Steinberger, J.K., Rao, N.D., Oswald, Y.: Providing decent living with minimum energy: A global scenario. Glob. Environ. Change. 65 , 102168 (2020). https://doi.org/10.1016/J.GLOENVCHA.2020.102168

Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L.A. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst. Rev. 4 (1), 1 (2015). https://doi.org/10.1186/2046-4053-4-1

Mongeon, P., Paul-Hus, A.: The journal coverage of web of Science and Scopus: A comparative analysis. Scientometrics. 106 (1), 213–228 (2016). https://doi.org/10.1007/S11192-015-1765-5/FIGURES/6

Musa, H.D., Yacob, M.R., Abdullah, A.M., Ishak, M.Y.: Enhancing subjective well-being through strategic urban planning: Development and application of community happiness index. Sustainable Cities Soc. 38 , 184–194 (2018). https://doi.org/10.1016/J.SCS.2017.12.030

Ni, Z., Yang, J., Razzaq, A.: How do natural resources, digitalization, and institutional governance contribute to ecological sustainability through load capacity factors in highly resource-consuming economies? Resour. Policy. 79 , 103068 (2022). https://doi.org/10.1016/J.RESOURPOL.2022.103068

Nunes, A.R., Lee, K., O’Riordan, T.: The importance of an integrating framework for achieving the sustainable development goals: the example of health and well-being. BMJ Glob Health. 1 (3), e000068 (2016). https://doi.org/10.1136/BMJGH-2016-000068

Ócsai, A.: The future of ecologically conscious business. Palgrave Stud. Sustainable Bus. Association Future Earth. 259–274 (2021). https://doi.org/10.1007/978-3-030-60918-4_7

OECD.: Measuring and assessing well-being in Israel (2016).  https://doi.org/10.1787/9789264246034-EN

Oehler-Șincai, I.M.: Well-Being, Quality of Governance, and Employment Policies: International Perspectives. (2023). https://doi.org/10.1080/07360932.2023.2189078

Orekhov, V.D., Prichina, O.S., Loktionova, Y.N., Yanina, O.N., Gusareva, N.B.: Scientific analysis of the happiness index in regard to the human capital development. J. Adv. Res. Dyn. Control Syst. 12 (4 Special Issue), 467–478 (2020). https://doi.org/10.5373/JARDCS/V12SP4/20201512

Oswald, A.J.: Happiness and economic performance. Econ. J. 107 (445), 1815–1831 (1997). https://doi.org/10.1111/J.1468-0297.1997.TB00085.X

Pillay, D.: Happiness, wellbeing and ecosocialism–a radical humanist perspective. Globalizations. 17 (2), 380–396 (2020). https://doi.org/10.1080/14747731.2019.1652470

Roy, M.J.: Towards a ‘Wellbeing Economy’: What Can We Learn from Social Enterprise? 269–284. (2021). https://doi.org/10.1007/978-3-030-68295-8_13

Rubio-Mozos, E., García-Muiña, F.E., Fuentes-Moraleda, L.: Rethinking 21st-century businesses: An approach to fourth sector SMEs in their transition to a sustainable model committed to SDGs. Sustain. (Switzerland). 11 (20) (2019). https://doi.org/10.3390/su11205569

Santini, Z.I., Koyanagi, A., Tyrovolas, S., Mason, C., Haro, J.M.: The association between social relationships and depression: a systematic review. J. Affect. Disord. 175 , 53–65 (2015). https://doi.org/10.1016/J.JAD.2014.12.049

Sangha, K.K., Le Brocque, A., Costanza, R., Cadet-James, Y.: Ecosystems and indigenous well-being: An integrated framework. Global Ecol. Conserv. 4 , 197–206 (2015). https://doi.org/10.1016/J.GECCO.2015.06.008

Schyns, P.: Crossnational differences in happiness: Economic and cultural factors explored. Soc. Indic. Res. 43 (1–2), 3–26 (1998). https://doi.org/10.1023/A:1006814424293/METRICS

Shrivastava, P., Zsolnai, L.: Wellbeing-oriented organizations: Connecting human flourishing with ecological regeneration. Bus. Ethics Environ. Responsib. 31 (2), 386–397 (2022). https://doi.org/10.1111/beer.12421

Skul’skaya, L.V., Shirokova, T.K.: Is it possible to build an economy of happiness? (On the book by E.E. Rumyantseva Economy of Happiness (INFRA-M, Moscow, 2010) [in Russian]). Studies on Russian Economic Development 2010 21:4 , 21 (4), 455–456. (2010). https://doi.org/10.1134/S1075700710040131

Sohn, K.: Considering happiness for economic development: determinants of happiness in Indonesia. SSRN Electronic J. (2010). https://doi.org/10.2139/SSRN.2489785

Spruk, R., Kešeljević, A.: Institutional origins of subjective well-being: estimating the effects of economic freedom on national happiness. J. Happiness Stud. 17 (2), 659–712 (2015). https://doi.org/10.1007/S10902-015-9616-X

Su, Y.S., Lien, D., Yao, Y.: Economic growth and happiness in China: a Bayesian multilevel age-period-cohort analysis based on the CGSS data 2005–2015. Int. Rev. Econ. Finance. 77 , 191–205 (2022). https://doi.org/10.1016/J.IREF.2021.09.018

Stucke, M.E.: Should competition policy promote happiness? Fordham Law Review , 81 (5), 2575–2645. (2013). https://www.scopus.com/inward/record.uri?eid=2-s2.0-84878026480&partnerID=40&md5=66be6f8cd1fe9df6ca216e7724c928b3

van Niekerk, A.: A conceptual framework for inclusive economics. South. Afr. J. Economic Manage. Sci. 22 (1) (2019). https://doi.org/10.4102/sajems.v22i1.2915

Vicente, M.J.V.: How more equal societies reduce stress, restore sanity and improve everyone’s well-being por Richard Wilkinson y Kate Pickett. Sistema: Revista de Ciencias Sociales, ISSN 0210–0223, No 257, 2020, Págs. 135–140 , 257 , 135–140. (2020). https://dialnet.unirioja.es/servlet/articulo?codigo=7999261

Wang, Q., Su, M.: The effects of urbanization and industrialization on decoupling economic growth from carbon emission – a case study of China. Sustain Cities Soc. 51 , 101758 (2019). https://doi.org/10.1016/J.SCS.2019.101758

Wiedenhofer, D., Virág, D., Kalt, G., Plank, B., Streeck, J., Pichler, M., Mayer, A., Krausmann, F., Brockway, P., Schaffartzik, A., Fishman, T., Hausknost, D., Leon-Gruchalski, B., Sousa, T., Creutzig, F., Haberl, H.: A systematic review of the evidence on decoupling of GDP, resource use and GHG emissions, part I: Bibliometric and conceptual mapping. Environ. Res. Lett. 15 (6), 063002 (2020). https://doi.org/10.1088/1748-9326/AB8429

Wolfgramm, R., Spiller, C., Henry, E., Pouwhare, R.: A culturally derived framework of values-driven transformation in Māori economies of well-being (Ngā Hono ōhanga oranga). AlterNative. 16 (1), 18–28 (2020). https://doi.org/10.1177/1177180119885663

Wu, Y., Zhu, Q., Zhu, B.: Decoupling analysis of world economic growth and CO2 emissions: a study comparing developed and developing countries. J. Clean. Prod. 190 , 94–103 (2018) https://doi.org/10.1016/j.jclepro.2018.04.139

Zheng, S., Wang, J., Sun, C., Zhang, X., Kahn, M.E.: Air pollution lowers Chinese urbanites’ expressed happiness on social media. Nat. Hum. Behav. 3 (3), 237–243 (2019). https://doi.org/10.1038/s41562-018-0521-2

Download references

The authors declare that no funds, grants, or other support were received during the preparation of this manuscript.

Open access funding provided by Università degli Studi di Salerno within the CRUI-CARE Agreement.

Author information

Authors and affiliations.

Department of Humanities and Social Sciences, Malaviya National Institute of Technology Jaipur, Malaviya Nagar, J.L.N. Marg, Jaipur, Rajasthan, 302017, India

Shruti Agrawal & Nidhi Sharma

Department of Economics and Finance, Birla Institute of Technology and Science (BITS), Pilani, Rajasthan, India

Karambir Singh Dhayal

Department of Economics and Statistics, University of Salerno, Fisciano, Italy

Luca Esposito

Karelian Institute, University of Eastern Finland, Joensuu, Finland

Erasmus Happiness Economics Research Organisation, Erasmus University, Rotterdam, Netherlands

You can also search for this author in PubMed   Google Scholar

Contributions

All authors contributed to the study conception and design. Shruti Agrawal: Conceptualization, Material preparation, Data Collection, Formal analysis, Methodology, Writing - Original Draft, Review and Editing. Nidhi Sharma: Validation, Project Administration, Supervision, and Writing - Review & Editing. Karambir Singh Dhayal: Validation, Formal analysis, Methodology, Writing - Review and Editing. Luca Esposito: Validation, Writing - Review and Editing. The first draft of the manuscript was written by Shruti Agrawal and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript

Corresponding author

Correspondence to Luca Esposito .

Ethics declarations

Ethical approval.

This material is the authors’ own original work, which has not been previously published elsewhere. The paper is not currently being considered for publication elsewhere. The paper reflects the authors’ own research and analysis in a truthful and complete manner.

Competing interests

The authors declare no conflict of interest.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Agrawal, S., Sharma, N., Dhayal, K.S. et al. From economic wealth to well-being: exploring the importance of happiness economy for sustainable development through systematic literature review. Qual Quant (2024). https://doi.org/10.1007/s11135-024-01892-z

Download citation

Accepted : 21 April 2024

Published : 23 May 2024

DOI : https://doi.org/10.1007/s11135-024-01892-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Happiness economy
• Sustainability
• Find a journal
• Publish with us
• Track your research