analysing a systematic literature review

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Systematic Review | Definition, Example, & Guide

Systematic Review | Definition, Example & Guide

Published on June 15, 2022 by Shaun Turney . Revised on November 20, 2023.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question “What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?”

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

Table of contents

What is a systematic review, systematic review vs. meta-analysis, systematic review vs. literature review, systematic review vs. scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, other interesting articles, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce bias . The methods are repeatable, and the approach is formal and systematic:

• Formulate a research question
• Develop a protocol
• Search for all relevant studies
• Apply the selection criteria
• Extract the data
• Synthesize the data
• Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesizing all available evidence and evaluating the quality of the evidence. Synthesizing means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

Systematic reviews often quantitatively synthesize the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesize results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarize and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimize bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

Prevent plagiarism. Run a free check.

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

• A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
• If you’re doing a systematic review on your own (e.g., for a research paper or thesis ), you should take appropriate measures to ensure the validity and reliability of your research.
• Access to databases and journal archives. Often, your educational institution provides you with access.
• Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
• Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

• They minimize research bias by considering all available evidence and evaluating each study for bias.
• Their methods are transparent , so they can be scrutinized by others.
• They’re thorough : they summarize all available evidence.
• They can be replicated and updated by others.

Systematic reviews also have a few cons .

• They’re time-consuming .
• They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

• Allow you to more effectively communicate your research to other researchers and practitioners
• Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

• Population(s) or problem(s)
• Intervention(s)
• Comparison(s)

You can rearrange these four components to write your research question:

• What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fifth component, the type of study design . In this case, the acronym is PICOT .

• Type of study design(s)
• The population of patients with eczema
• The intervention of probiotics
• In comparison to no treatment, placebo , or non-probiotic treatment
• The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
• Randomized control trials, a type of study design

Their research question was:

• What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

• Background information : Provide the context of the research question, including why it’s important.
• Research objective (s) : Rephrase your research question as an objective.
• Selection criteria: State how you’ll decide which studies to include or exclude from your review.
• Search strategy: Discuss your plan for finding studies.
• Analysis: Explain what information you’ll collect from the studies and how you’ll synthesize the data.

If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

• Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
• Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
• Gray literature: Gray literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of gray literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of gray literature.
• Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

• Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
• Handsearch: Conference proceedings and reference lists of articles
• Gray literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
• Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

You should apply the selection criteria in two phases:

• Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
• Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarize what you did using a PRISMA flow diagram .

Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

• Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
• Your judgment of the quality of the evidence, including risk of bias .

You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomized into the control and treatment groups.

Step 6: Synthesize the data

Synthesizing the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesizing the data:

• Narrative ( qualitative ): Summarize the information in words. You’ll need to discuss the studies and assess their overall quality.
• Quantitative : Use statistical methods to summarize and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analyzed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

• Abstract : A summary of the review
• Introduction : Including the rationale and objectives
• Methods : Including the selection criteria, search method, data extraction method, and synthesis method
• Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
• Discussion : Including interpretation of the results and limitations of the review
• Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

In their report, Boyle and colleagues concluded that probiotics cannot be recommended for reducing eczema symptoms or improving quality of life in patients with eczema. Note Generative AI tools like ChatGPT can be useful at various stages of the writing and research process and can help you to write your systematic review. However, we strongly advise against trying to pass AI-generated text off as your own work.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Student’s  t -distribution
• Normal distribution
• Null and Alternative Hypotheses
• Chi square tests
• Confidence interval
• Quartiles & Quantiles
• Cluster sampling
• Stratified sampling
• Data cleansing
• Reproducibility vs Replicability
• Peer review
• Prospective cohort study

Research bias

• Implicit bias
• Cognitive bias
• Placebo effect
• Hawthorne effect
• Hindsight bias
• Affect heuristic
• Social desirability bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Turney, S. (2023, November 20). Systematic Review | Definition, Example & Guide. Scribbr. Retrieved September 3, 2024, from https://www.scribbr.com/methodology/systematic-review/

Is this article helpful?

Shaun Turney

Other students also liked, how to write a literature review | guide, examples, & templates, how to write a research proposal | examples & templates, what is critical thinking | definition & examples, "i thought ai proofreading was useless but..".

I've been using Scribbr for years now and I know it's a service that won't disappoint. It does a good job spotting mistakes”

Reference management. Clean and simple.

How to write a systematic literature review [9 steps]

What is a systematic literature review?

Where are systematic literature reviews used, what types of systematic literature reviews are there, how to write a systematic literature review, 1. decide on your team, 2. formulate your question, 3. plan your research protocol, 4. search for the literature, 5. screen the literature, 6. assess the quality of the studies, 7. extract the data, 8. analyze the results, 9. interpret and present the results, registering your systematic literature review, frequently asked questions about writing a systematic literature review, related articles.

A systematic literature review is a summary, analysis, and evaluation of all the existing research on a well-formulated and specific question.

Put simply, a systematic review is a study of studies that is popular in medical and healthcare research. In this guide, we will cover:

• the definition of a systematic literature review
• the purpose of a systematic literature review
• the different types of systematic reviews
• how to write a systematic literature review

➡️ Visit our guide to the best research databases for medicine and health to find resources for your systematic review.

Systematic literature reviews can be utilized in various contexts, but they’re often relied on in clinical or healthcare settings.

Medical professionals read systematic literature reviews to stay up-to-date in their field, and granting agencies sometimes need them to make sure there’s justification for further research in an area. They can even be used as the starting point for developing clinical practice guidelines.

A classic systematic literature review can take different approaches:

• Effectiveness reviews assess the extent to which a medical intervention or therapy achieves its intended effect. They’re the most common type of systematic literature review.
• Diagnostic test accuracy reviews produce a summary of diagnostic test performance so that their accuracy can be determined before use by healthcare professionals.
• Experiential (qualitative) reviews analyze human experiences in a cultural or social context. They can be used to assess the effectiveness of an intervention from a person-centric perspective.
• Costs/economics evaluation reviews look at the cost implications of an intervention or procedure, to assess the resources needed to implement it.
• Etiology/risk reviews usually try to determine to what degree a relationship exists between an exposure and a health outcome. This can be used to better inform healthcare planning and resource allocation.
• Psychometric reviews assess the quality of health measurement tools so that the best instrument can be selected for use.
• Prevalence/incidence reviews measure both the proportion of a population who have a disease, and how often the disease occurs.
• Prognostic reviews examine the course of a disease and its potential outcomes.
• Expert opinion/policy reviews are based around expert narrative or policy. They’re often used to complement, or in the absence of, quantitative data.
• Methodology systematic reviews can be carried out to analyze any methodological issues in the design, conduct, or review of research studies.

Writing a systematic literature review can feel like an overwhelming undertaking. After all, they can often take 6 to 18 months to complete. Below we’ve prepared a step-by-step guide on how to write a systematic literature review.

• Decide on your team.
• Formulate your question.
• Plan your research protocol.
• Search for the literature.
• Screen the literature.
• Assess the quality of the studies.
• Extract the data.
• Analyze the results.
• Interpret and present the results.

When carrying out a systematic literature review, you should employ multiple reviewers in order to minimize bias and strengthen analysis. A minimum of two is a good rule of thumb, with a third to serve as a tiebreaker if needed.

You may also need to team up with a librarian to help with the search, literature screeners, a statistician to analyze the data, and the relevant subject experts.

Define your answerable question. Then ask yourself, “has someone written a systematic literature review on my question already?” If so, yours may not be needed. A librarian can help you answer this.

You should formulate a “well-built clinical question.” This is the process of generating a good search question. To do this, run through PICO:

• Patient or Population or Problem/Disease : who or what is the question about? Are there factors about them (e.g. age, race) that could be relevant to the question you’re trying to answer?
• Intervention : which main intervention or treatment are you considering for assessment?
• Comparison(s) or Control : is there an alternative intervention or treatment you’re considering? Your systematic literature review doesn’t have to contain a comparison, but you’ll want to stipulate at this stage, either way.
• Outcome(s) : what are you trying to measure or achieve? What’s the wider goal for the work you’ll be doing?

Now you need a detailed strategy for how you’re going to search for and evaluate the studies relating to your question.

The protocol for your systematic literature review should include:

• the objectives of your project
• the specific methods and processes that you’ll use
• the eligibility criteria of the individual studies
• how you plan to extract data from individual studies
• which analyses you’re going to carry out

For a full guide on how to systematically develop your protocol, take a look at the PRISMA checklist . PRISMA has been designed primarily to improve the reporting of systematic literature reviews and meta-analyses.

When writing a systematic literature review, your goal is to find all of the relevant studies relating to your question, so you need to search thoroughly .

This is where your librarian will come in handy again. They should be able to help you formulate a detailed search strategy, and point you to all of the best databases for your topic.

➡️ Read more on on how to efficiently search research databases .

The places to consider in your search are electronic scientific databases (the most popular are PubMed , MEDLINE , and Embase ), controlled clinical trial registers, non-English literature, raw data from published trials, references listed in primary sources, and unpublished sources known to experts in the field.

➡️ Take a look at our list of the top academic research databases .

Tip: Don’t miss out on “gray literature.” You’ll improve the reliability of your findings by including it.

Don’t miss out on “gray literature” sources: those sources outside of the usual academic publishing environment. They include:

• non-peer-reviewed journals
• pharmaceutical industry files
• conference proceedings
• pharmaceutical company websites
• internal reports

Gray literature sources are more likely to contain negative conclusions, so you’ll improve the reliability of your findings by including it. You should document details such as:

• The databases you search and which years they cover
• The dates you first run the searches, and when they’re updated
• Which strategies you use, including search terms
• The numbers of results obtained

➡️ Read more about gray literature .

This should be performed by your two reviewers, using the criteria documented in your research protocol. The screening is done in two phases:

• Pre-screening of all titles and abstracts, and selecting those appropriate
• Screening of the full-text articles of the selected studies

Make sure reviewers keep a log of which studies they exclude, with reasons why.

➡️ Visit our guide on what is an abstract?

Your reviewers should evaluate the methodological quality of your chosen full-text articles. Make an assessment checklist that closely aligns with your research protocol, including a consistent scoring system, calculations of the quality of each study, and sensitivity analysis.

The kinds of questions you'll come up with are:

• Were the participants really randomly allocated to their groups?
• Were the groups similar in terms of prognostic factors?
• Could the conclusions of the study have been influenced by bias?

Every step of the data extraction must be documented for transparency and replicability. Create a data extraction form and set your reviewers to work extracting data from the qualified studies.

Here’s a free detailed template for recording data extraction, from Dalhousie University. It should be adapted to your specific question.

Establish a standard measure of outcome which can be applied to each study on the basis of its effect size.

Measures of outcome for studies with:

• Binary outcomes (e.g. cured/not cured) are odds ratio and risk ratio
• Continuous outcomes (e.g. blood pressure) are means, difference in means, and standardized difference in means
• Survival or time-to-event data are hazard ratios

Design a table and populate it with your data results. Draw this out into a forest plot , which provides a simple visual representation of variation between the studies.

Then analyze the data for issues. These can include heterogeneity, which is when studies’ lines within the forest plot don’t overlap with any other studies. Again, record any excluded studies here for reference.

Consider different factors when interpreting your results. These include limitations, strength of evidence, biases, applicability, economic effects, and implications for future practice or research.

Apply appropriate grading of your evidence and consider the strength of your recommendations.

It’s best to formulate a detailed plan for how you’ll present your systematic review results. Take a look at these guidelines for interpreting results from the Cochrane Institute.

Before writing your systematic literature review, you can register it with OSF for additional guidance along the way. You could also register your completed work with PROSPERO .

Systematic literature reviews are often found in clinical or healthcare settings. Medical professionals read systematic literature reviews to stay up-to-date in their field and granting agencies sometimes need them to make sure there’s justification for further research in an area.

The first stage in carrying out a systematic literature review is to put together your team. You should employ multiple reviewers in order to minimize bias and strengthen analysis. A minimum of two is a good rule of thumb, with a third to serve as a tiebreaker if needed.

Your systematic review should include the following details:

A literature review simply provides a summary of the literature available on a topic. A systematic review, on the other hand, is more than just a summary. It also includes an analysis and evaluation of existing research. Put simply, it's a study of studies.

The final stage of conducting a systematic literature review is interpreting and presenting the results. It’s best to formulate a detailed plan for how you’ll present your systematic review results, guidelines can be found for example from the Cochrane institute .

• A-Z Publications

Annual Review of Psychology

Volume 70, 2019, review article, how to do a systematic review: a best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses.

• Andy P. Siddaway 1 , Alex M. Wood 2 , and Larry V. Hedges 3
• View Affiliations Hide Affiliations Affiliations: 1 Behavioural Science Centre, Stirling Management School, University of Stirling, Stirling FK9 4LA, United Kingdom; email: [email protected] 2 Department of Psychological and Behavioural Science, London School of Economics and Political Science, London WC2A 2AE, United Kingdom 3 Department of Statistics, Northwestern University, Evanston, Illinois 60208, USA; email: [email protected]
• Vol. 70:747-770 (Volume publication date January 2019) https://doi.org/10.1146/annurev-psych-010418-102803
• First published as a Review in Advance on August 08, 2018
• Copyright © 2019 by Annual Reviews. All rights reserved

Systematic reviews are characterized by a methodical and replicable methodology and presentation. They involve a comprehensive search to locate all relevant published and unpublished work on a subject; a systematic integration of search results; and a critique of the extent, nature, and quality of evidence in relation to a particular research question. The best reviews synthesize studies to draw broad theoretical conclusions about what a literature means, linking theory to evidence and evidence to theory. This guide describes how to plan, conduct, organize, and present a systematic review of quantitative (meta-analysis) or qualitative (narrative review, meta-synthesis) information. We outline core standards and principles and describe commonly encountered problems. Although this guide targets psychological scientists, its high level of abstraction makes it potentially relevant to any subject area or discipline. We argue that systematic reviews are a key methodology for clarifying whether and how research findings replicate and for explaining possible inconsistencies, and we call for researchers to conduct systematic reviews to help elucidate whether there is a replication crisis.

Article metrics loading...

Full text loading...

Literature Cited

• APA Publ. Commun. Board Work. Group J. Artic. Rep. Stand. 2008 . Reporting standards for research in psychology: Why do we need them? What might they be?. Am. Psychol . 63 : 848– 49 [Google Scholar]
• Baumeister RF 2013 . Writing a literature review. The Portable Mentor: Expert Guide to a Successful Career in Psychology MJ Prinstein, MD Patterson 119– 32 New York: Springer, 2nd ed.. [Google Scholar]
• Baumeister RF , Leary MR 1995 . The need to belong: desire for interpersonal attachments as a fundamental human motivation. Psychol. Bull. 117 : 497– 529 [Google Scholar]
• Baumeister RF , Leary MR 1997 . Writing narrative literature reviews. Rev. Gen. Psychol. 3 : 311– 20 Presents a thorough and thoughtful guide to conducting narrative reviews. [Google Scholar]
• Bem DJ 1995 . Writing a review article for Psychological Bulletin. Psychol . Bull 118 : 172– 77 [Google Scholar]
• Borenstein M , Hedges LV , Higgins JPT , Rothstein HR 2009 . Introduction to Meta-Analysis New York: Wiley Presents a comprehensive introduction to meta-analysis. [Google Scholar]
• Borenstein M , Higgins JPT , Hedges LV , Rothstein HR 2017 . Basics of meta-analysis: I 2 is not an absolute measure of heterogeneity. Res. Synth. Methods 8 : 5– 18 [Google Scholar]
• Braver SL , Thoemmes FJ , Rosenthal R 2014 . Continuously cumulating meta-analysis and replicability. Perspect. Psychol. Sci. 9 : 333– 42 [Google Scholar]
• Bushman BJ 1994 . Vote-counting procedures. The Handbook of Research Synthesis H Cooper, LV Hedges 193– 214 New York: Russell Sage Found. [Google Scholar]
• Cesario J 2014 . Priming, replication, and the hardest science. Perspect. Psychol. Sci. 9 : 40– 48 [Google Scholar]
• Chalmers I 2007 . The lethal consequences of failing to make use of all relevant evidence about the effects of medical treatments: the importance of systematic reviews. Treating Individuals: From Randomised Trials to Personalised Medicine PM Rothwell 37– 58 London: Lancet [Google Scholar]
• Cochrane Collab. 2003 . Glossary Rep., Cochrane Collab. London: http://community.cochrane.org/glossary Presents a comprehensive glossary of terms relevant to systematic reviews. [Google Scholar]
• Cohn LD , Becker BJ 2003 . How meta-analysis increases statistical power. Psychol. Methods 8 : 243– 53 [Google Scholar]
• Cooper HM 2003 . Editorial. Psychol. Bull. 129 : 3– 9 [Google Scholar]
• Cooper HM 2016 . Research Synthesis and Meta-Analysis: A Step-by-Step Approach Thousand Oaks, CA: Sage, 5th ed.. Presents a comprehensive introduction to research synthesis and meta-analysis. [Google Scholar]
• Cooper HM , Hedges LV , Valentine JC 2009 . The Handbook of Research Synthesis and Meta-Analysis New York: Russell Sage Found, 2nd ed.. [Google Scholar]
• Cumming G 2014 . The new statistics: why and how. Psychol. Sci. 25 : 7– 29 Discusses the limitations of null hypothesis significance testing and viable alternative approaches. [Google Scholar]
• Earp BD , Trafimow D 2015 . Replication, falsification, and the crisis of confidence in social psychology. Front. Psychol. 6 : 621 [Google Scholar]
• Etz A , Vandekerckhove J 2016 . A Bayesian perspective on the reproducibility project: psychology. PLOS ONE 11 : e0149794 [Google Scholar]
• Ferguson CJ , Brannick MT 2012 . Publication bias in psychological science: prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychol. Methods 17 : 120– 28 [Google Scholar]
• Fleiss JL , Berlin JA 2009 . Effect sizes for dichotomous data. The Handbook of Research Synthesis and Meta-Analysis H Cooper, LV Hedges, JC Valentine 237– 53 New York: Russell Sage Found, 2nd ed.. [Google Scholar]
• Garside R 2014 . Should we appraise the quality of qualitative research reports for systematic reviews, and if so, how. Innovation 27 : 67– 79 [Google Scholar]
• Hedges LV , Olkin I 1980 . Vote count methods in research synthesis. Psychol. Bull. 88 : 359– 69 [Google Scholar]
• Hedges LV , Pigott TD 2001 . The power of statistical tests in meta-analysis. Psychol. Methods 6 : 203– 17 [Google Scholar]
• Higgins JPT , Green S 2011 . Cochrane Handbook for Systematic Reviews of Interventions, Version 5.1.0 London: Cochrane Collab. Presents comprehensive and regularly updated guidelines on systematic reviews. [Google Scholar]
• John LK , Loewenstein G , Prelec D 2012 . Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol. Sci. 23 : 524– 32 [Google Scholar]
• Juni P , Witschi A , Bloch R , Egger M 1999 . The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 282 : 1054– 60 [Google Scholar]
• Klein O , Doyen S , Leys C , Magalhães de Saldanha da Gama PA , Miller S et al. 2012 . Low hopes, high expectations: expectancy effects and the replicability of behavioral experiments. Perspect. Psychol. Sci. 7 : 6 572– 84 [Google Scholar]
• Lau J , Antman EM , Jimenez-Silva J , Kupelnick B , Mosteller F , Chalmers TC 1992 . Cumulative meta-analysis of therapeutic trials for myocardial infarction. N. Engl. J. Med. 327 : 248– 54 [Google Scholar]
• Light RJ , Smith PV 1971 . Accumulating evidence: procedures for resolving contradictions among different research studies. Harvard Educ. Rev. 41 : 429– 71 [Google Scholar]
• Lipsey MW , Wilson D 2001 . Practical Meta-Analysis London: Sage Comprehensive and clear explanation of meta-analysis. [Google Scholar]
• Matt GE , Cook TD 1994 . Threats to the validity of research synthesis. The Handbook of Research Synthesis H Cooper, LV Hedges 503– 20 New York: Russell Sage Found. [Google Scholar]
• Maxwell SE , Lau MY , Howard GS 2015 . Is psychology suffering from a replication crisis? What does “failure to replicate” really mean?. Am. Psychol. 70 : 487– 98 [Google Scholar]
• Moher D , Hopewell S , Schulz KF , Montori V , Gøtzsche PC et al. 2010 . CONSORT explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ 340 : c869 [Google Scholar]
• Moher D , Liberati A , Tetzlaff J , Altman DG PRISMA Group. 2009 . Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 339 : 332– 36 Comprehensive reporting guidelines for systematic reviews. [Google Scholar]
• Morrison A , Polisena J , Husereau D , Moulton K , Clark M et al. 2012 . The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies. Int. J. Technol. Assess. Health Care 28 : 138– 44 [Google Scholar]
• Nelson LD , Simmons J , Simonsohn U 2018 . Psychology's renaissance. Annu. Rev. Psychol. 69 : 511– 34 [Google Scholar]
• Noblit GW , Hare RD 1988 . Meta-Ethnography: Synthesizing Qualitative Studies Newbury Park, CA: Sage [Google Scholar]
• Olivo SA , Macedo LG , Gadotti IC , Fuentes J , Stanton T , Magee DJ 2008 . Scales to assess the quality of randomized controlled trials: a systematic review. Phys. Ther. 88 : 156– 75 [Google Scholar]
• Open Sci. Collab. 2015 . Estimating the reproducibility of psychological science. Science 349 : 943 [Google Scholar]
• Paterson BL , Thorne SE , Canam C , Jillings C 2001 . Meta-Study of Qualitative Health Research: A Practical Guide to Meta-Analysis and Meta-Synthesis Thousand Oaks, CA: Sage [Google Scholar]
• Patil P , Peng RD , Leek JT 2016 . What should researchers expect when they replicate studies? A statistical view of replicability in psychological science. Perspect. Psychol. Sci. 11 : 539– 44 [Google Scholar]
• Rosenthal R 1979 . The “file drawer problem” and tolerance for null results. Psychol. Bull. 86 : 638– 41 [Google Scholar]
• Rosnow RL , Rosenthal R 1989 . Statistical procedures and the justification of knowledge in psychological science. Am. Psychol. 44 : 1276– 84 [Google Scholar]
• Sanderson S , Tatt ID , Higgins JP 2007 . Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int. J. Epidemiol. 36 : 666– 76 [Google Scholar]
• Schreiber R , Crooks D , Stern PN 1997 . Qualitative meta-analysis. Completing a Qualitative Project: Details and Dialogue JM Morse 311– 26 Thousand Oaks, CA: Sage [Google Scholar]
• Shrout PE , Rodgers JL 2018 . Psychology, science, and knowledge construction: broadening perspectives from the replication crisis. Annu. Rev. Psychol. 69 : 487– 510 [Google Scholar]
• Stroebe W , Strack F 2014 . The alleged crisis and the illusion of exact replication. Perspect. Psychol. Sci. 9 : 59– 71 [Google Scholar]
• Stroup DF , Berlin JA , Morton SC , Olkin I , Williamson GD et al. 2000 . Meta-analysis of observational studies in epidemiology (MOOSE): a proposal for reporting. JAMA 283 : 2008– 12 [Google Scholar]
• Thorne S , Jensen L , Kearney MH , Noblit G , Sandelowski M 2004 . Qualitative meta-synthesis: reflections on methodological orientation and ideological agenda. Qual. Health Res. 14 : 1342– 65 [Google Scholar]
• Tong A , Flemming K , McInnes E , Oliver S , Craig J 2012 . Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Med. Res. Methodol. 12 : 181– 88 [Google Scholar]
• Trickey D , Siddaway AP , Meiser-Stedman R , Serpell L , Field AP 2012 . A meta-analysis of risk factors for post-traumatic stress disorder in children and adolescents. Clin. Psychol. Rev. 32 : 122– 38 [Google Scholar]
• Valentine JC , Biglan A , Boruch RF , Castro FG , Collins LM et al. 2011 . Replication in prevention science. Prev. Sci. 12 : 103– 17 [Google Scholar]
• Article Type: Review Article

Most Read This Month

Most cited most cited rss feed, job burnout, executive functions, social cognitive theory: an agentic perspective, on happiness and human potentials: a review of research on hedonic and eudaimonic well-being, sources of method bias in social science research and recommendations on how to control it, mediation analysis, missing data analysis: making it work in the real world, grounded cognition, personality structure: emergence of the five-factor model, motivational beliefs, values, and goals.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

Systematic Reviews and Meta-Analysis: A Guide for Beginners

Affiliation.

• 1 Department of Pediatrics, Advanced Pediatrics Centre, PGIMER, Chandigarh. Correspondence to: Prof Joseph L Mathew, Department of Pediatrics, Advanced Pediatrics Centre, PGIMER Chandigarh. [email protected].
• PMID: 34183469
• PMCID: PMC9065227
• DOI: 10.1007/s13312-022-2500-y

Systematic reviews involve the application of scientific methods to reduce bias in review of literature. The key components of a systematic review are a well-defined research question, comprehensive literature search to identify all studies that potentially address the question, systematic assembly of the studies that answer the question, critical appraisal of the methodological quality of the included studies, data extraction and analysis (with and without statistics), and considerations towards applicability of the evidence generated in a systematic review. These key features can be remembered as six 'A'; Ask, Access, Assimilate, Appraise, Analyze and Apply. Meta-analysis is a statistical tool that provides pooled estimates of effect from the data extracted from individual studies in the systematic review. The graphical output of meta-analysis is a forest plot which provides information on individual studies and the pooled effect. Systematic reviews of literature can be undertaken for all types of questions, and all types of study designs. This article highlights the key features of systematic reviews, and is designed to help readers understand and interpret them. It can also help to serve as a beginner's guide for both users and producers of systematic reviews and to appreciate some of the methodological issues.

PubMed Disclaimer

Similar articles

• The future of Cochrane Neonatal. Soll RF, Ovelman C, McGuire W. Soll RF, et al. Early Hum Dev. 2020 Nov;150:105191. doi: 10.1016/j.earlhumdev.2020.105191. Epub 2020 Sep 12. Early Hum Dev. 2020. PMID: 33036834
• How to review and assess a systematic review and meta-analysis article: a methodological study (secondary publication). Myung SK. Myung SK. J Educ Eval Health Prof. 2023;20:24. doi: 10.3352/jeehp.2023.20.24. Epub 2023 Aug 27. J Educ Eval Health Prof. 2023. PMID: 37619974 Free PMC article.
• How to Critically Appraise and Interpret Systematic Reviews and Meta-Analyses of Diagnostic Accuracy: A User Guide. Frank RA, Salameh JP, Islam N, Yang B, Murad MH, Mustafa R, Leeflang M, Bossuyt PM, Takwoingi Y, Whiting P, Dawit H, Kang SK, Ebrahimzadeh S, Levis B, Hutton B, McInnes MDF. Frank RA, et al. Radiology. 2023 May;307(3):e221437. doi: 10.1148/radiol.221437. Epub 2023 Mar 14. Radiology. 2023. PMID: 36916896 Free PMC article. Review.
• Detecting, quantifying and adjusting for publication bias in meta-analyses: protocol of a systematic review on methods. Mueller KF, Meerpohl JJ, Briel M, Antes G, von Elm E, Lang B, Gloy V, Motschall E, Schwarzer G, Bassler D. Mueller KF, et al. Syst Rev. 2013 Jul 25;2:60. doi: 10.1186/2046-4053-2-60. Syst Rev. 2013. PMID: 23885765 Free PMC article.
• Characteristics, quality and volume of the first 5 months of the COVID-19 evidence synthesis infodemic: a meta-research study. Abbott R, Bethel A, Rogers M, Whear R, Orr N, Shaw L, Stein K, Thompson Coon J. Abbott R, et al. BMJ Evid Based Med. 2022 Jun;27(3):169-177. doi: 10.1136/bmjebm-2021-111710. Epub 2021 Jun 3. BMJ Evid Based Med. 2022. PMID: 34083212 Free PMC article. Review.
• Italian and Middle Eastern adherence to Mediterranean diet in relation to Body Mass Index and non-communicable diseases: nutritional adequacy of simulated weekly food plans. Luconi E, Tosi M, Boracchi P, Colonna I, Rappocciolo E, Ferraretto A, Lorenzini EC. Luconi E, et al. J Transl Med. 2024 Jul 30;22(1):703. doi: 10.1186/s12967-024-05325-1. J Transl Med. 2024. PMID: 39080668 Free PMC article.
• A commentary on 'Efficacy of electroacupuncture in improving postoperative ileus in patients receiving colorectal surgery: a systematic review and meta-analysis'. Zhou M, Ye H, Ma R. Zhou M, et al. Int J Surg. 2024 Aug 1;110(8):5290-5291. doi: 10.1097/JS9.0000000000001564. Int J Surg. 2024. PMID: 38716889 Free PMC article. No abstract available.
• Evidence, Experience, and Eminence: Building Blocks for Pediatric Pulmonology Practice and Research in India. Mathew JL. Mathew JL. Indian J Pediatr. 2023 Jul;90(7):690-692. doi: 10.1007/s12098-023-04649-y. Epub 2023 Jun 2. Indian J Pediatr. 2023. PMID: 37264278 Free PMC article. No abstract available.
• Efficiency, Economy and Excellence: Experimental Exploration of Evidence-based Guideline Development in Resource-Constrained Settings. Mathew JL. Mathew JL. Indian J Pediatr. 2023 Jul;90(7):700-707. doi: 10.1007/s12098-023-04575-z. Epub 2023 Apr 28. Indian J Pediatr. 2023. PMID: 37106227 Free PMC article. Review.
• Sackett D, Strauss S, Richardson W, et al. Evidence-Based Medicine: How to practice and teach EBM. 2nd ed. Churchill Livingstone: 2000.
• Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: Synthesis of best evidence for clinical decisions. Ann Intern Med. 1997;126:376–80. doi: 10.7326/0003-4819-126-5-199703010-00006. - DOI - PubMed
• PennState Eberley College of Science. Lesson 4: Bias and Random Error. Accessed October 01, 2020. Available from: https://online.stat.psu.edu/stat509/node/26/
• Comprehensive Meta-analysis. Accessed October 01, 2020. Available from: https://www.meta-analysis.com/pages/why_do.php?cart=
• National Institute for Health Research. PROSPERO International prospective register of systematic reviews. Accessed October 01, 2020. Available from: https://utas.libguides.com/SystematicReviews/Protocol

Publication types

• Search in MeSH

LinkOut - more resources

Full text sources.

• Europe PubMed Central
• Indian Pediatrics
• PubMed Central

• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Systematic Reviews and Meta Analysis

• Getting Started
• Guides and Standards
• Review Protocols
• Databases and Sources
• Randomized Controlled Trials
• Controlled Clinical Trials
• Observational Designs
• Tests of Diagnostic Accuracy
• Software and Tools
• Where do I get all those articles?
• Collaborations
• EPI 233/528
• Countway Mediated Search
• Risk of Bias (RoB)

Systematic review Q & A

What is a systematic review.

A systematic review is guided filtering and synthesis of all available evidence addressing a specific, focused research question, generally about a specific intervention or exposure. The use of standardized, systematic methods and pre-selected eligibility criteria reduce the risk of bias in identifying, selecting and analyzing relevant studies. A well-designed systematic review includes clear objectives, pre-selected criteria for identifying eligible studies, an explicit methodology, a thorough and reproducible search of the literature, an assessment of the validity or risk of bias of each included study, and a systematic synthesis, analysis and presentation of the findings of the included studies. A systematic review may include a meta-analysis.

For details about carrying out systematic reviews, see the Guides and Standards section of this guide.

Is my research topic appropriate for systematic review methods?

A systematic review is best deployed to test a specific hypothesis about a healthcare or public health intervention or exposure. By focusing on a single intervention or a few specific interventions for a particular condition, the investigator can ensure a manageable results set. Moreover, examining a single or small set of related interventions, exposures, or outcomes, will simplify the assessment of studies and the synthesis of the findings.

Systematic reviews are poor tools for hypothesis generation: for instance, to determine what interventions have been used to increase the awareness and acceptability of a vaccine or to investigate the ways that predictive analytics have been used in health care management. In the first case, we don't know what interventions to search for and so have to screen all the articles about awareness and acceptability. In the second, there is no agreed on set of methods that make up predictive analytics, and health care management is far too broad. The search will necessarily be incomplete, vague and very large all at the same time. In most cases, reviews without clearly and exactly specified populations, interventions, exposures, and outcomes will produce results sets that quickly outstrip the resources of a small team and offer no consistent way to assess and synthesize findings from the studies that are identified.

If not a systematic review, then what?

You might consider performing a scoping review . This framework allows iterative searching over a reduced number of data sources and no requirement to assess individual studies for risk of bias. The framework includes built-in mechanisms to adjust the analysis as the work progresses and more is learned about the topic. A scoping review won't help you limit the number of records you'll need to screen (broad questions lead to large results sets) but may give you means of dealing with a large set of results.

This tool can help you decide what kind of review is right for your question.

Can my student complete a systematic review during her summer project?

Probably not. Systematic reviews are a lot of work. Including creating the protocol, building and running a quality search, collecting all the papers, evaluating the studies that meet the inclusion criteria and extracting and analyzing the summary data, a well done review can require dozens to hundreds of hours of work that can span several months. Moreover, a systematic review requires subject expertise, statistical support and a librarian to help design and run the search. Be aware that librarians sometimes have queues for their search time. It may take several weeks to complete and run a search. Moreover, all guidelines for carrying out systematic reviews recommend that at least two subject experts screen the studies identified in the search. The first round of screening can consume 1 hour per screener for every 100-200 records. A systematic review is a labor-intensive team effort.

How can I know if my topic has been been reviewed already?

Before starting out on a systematic review, check to see if someone has done it already. In PubMed you can use the systematic review subset to limit to a broad group of papers that is enriched for systematic reviews. You can invoke the subset by selecting if from the Article Types filters to the left of your PubMed results, or you can append AND systematic[sb] to your search. For example:

"neoadjuvant chemotherapy" AND systematic[sb]

The systematic review subset is very noisy, however. To quickly focus on systematic reviews (knowing that you may be missing some), simply search for the word systematic in the title:

"neoadjuvant chemotherapy" AND systematic[ti]

Any PRISMA-compliant systematic review will be captured by this method since including the words "systematic review" in the title is a requirement of the PRISMA checklist. Cochrane systematic reviews do not include 'systematic' in the title, however. It's worth checking the Cochrane Database of Systematic Reviews independently.

You can also search for protocols that will indicate that another group has set out on a similar project. Many investigators will register their protocols in PROSPERO , a registry of review protocols. Other published protocols as well as Cochrane Review protocols appear in the Cochrane Methodology Register, a part of the Cochrane Library .

• Next: Guides and Standards >>
• Last Updated: Sep 4, 2024 4:04 PM
• URL: https://guides.library.harvard.edu/meta-analysis

• Locations and Hours
• UCLA Library
• Research Guides
• Biomedical Library Guides

Systematic Reviews

• Types of Literature Reviews

What Makes a Systematic Review Different from Other Types of Reviews?

• Planning Your Systematic Review
• Database Searching
• Creating the Search
• Search Filters and Hedges
• Grey Literature
• Managing and Appraising Results
• Further Resources

Reproduced from Grant, M. J. and Booth, A. (2009), A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26: 91–108. doi:10.1111/j.1471-1842.2009.00848.x

• << Previous: Home
• Next: Planning Your Systematic Review >>
• Last Updated: Jul 23, 2024 3:40 PM
• URL: https://guides.library.ucla.edu/systematicreviews

Assessing Scientific Inquiry: A Systematic Literature Review of Tasks, Tools and Techniques

• Theoretical Studies
• Open access
• Published: 04 September 2024

Cite this article

You have full access to this open access article

• De Van Vo   ORCID: orcid.org/0000-0002-8515-0221 1 &
• Geraldine Mooney Simmie   ORCID: orcid.org/0000-0002-5026-4261 1  

While national curricula in science education highlight the importance of inquiry-based learning, assessing students’ capabilities in scientific inquiry remains a subject of debate. Our study explored the construction, developmental trends and validation techniques in relation to assessing scientific inquiry using a systematic literature review from 2000 to 2024. We used PRISMA guidelines in combination with bibliometric and Epistemic Network Analyses. Sixty-three studies were selected, across all education sectors and with a majority of studies in secondary education. Results showed that assessing scientific inquiry has been considered around the world, with a growing number (37.0%) involving global researcher networks focusing on novel modelling approaches and simulation performance in digital-based environments. Although there was modest variation between the frameworks, studies were mainly concerned with cognitive processes and psychological characteristics and were reified from wider ethical, affective, intersectional and socio-cultural considerations. Four core categories (formulating questions/hypotheses, designing experiments, analysing data, and drawing conclusions) were most often used with nine specific components (formulate questions formulate prediction/hypotheses, set experiment, vary independent variable, measure dependent variable, control confounding variables, describe data, interpret data, reach reasonable conclusion). There was evidence of transitioning from traditional to online modes, facilitated by interactive simulations, but the independent tests and performance assessments, in both multiple-choice and open-ended formats remained the most frequently used approach with a greater emphasis on context than heretofore. The findings will be especially useful for science teachers, researchers and policy decision makers with an active interest in assessing capabilities in scientific inquiry.

Similar content being viewed by others

The role of structured inquiry, open inquiry, and epistemological beliefs in developing secondary students’ scientific and mathematical literacies

Methodological Ethics Considerations in Science Education Research: Symmetric, Authentic, Material, Adaptive and Multidisciplinary

Secondary School Students’ Understanding of Science and Their Socioscientific Reasoning

Explore related subjects.

• Artificial Intelligence

Avoid common mistakes on your manuscript.

Introduction

In contemporary times as more information and knowledge are created in a shorter timeline, the need for scientific literacy and inquiry-based capabilities beyond nature of science is increasing, especially in relation to the pressing needs of the wider world (Erduran, 2014 ). This is a growing concern, in relation to the future survival of humanity and sustainability of the planet for the reconceptualization of science education for epistemic justice and the foregrounding of intersectionality (Wallace et al., 2022 ). At the same time, policymakers and employers demand 21st century skills and inquiry-oriented approaches that include creativity, critical thinking, collaboration, communication and digital competencies (Binkley et al., 2012 ; Chu et al., 2017 ; Voogt & Roblin, 2012 ). Rather than teaching extensive content knowledge, there is a policy imperative to teach skills, dispositions, literacies and inquiry-oriented competencies. Mastery of capabilities, such as inquiry-oriented learning has therefore become a core outcome of national science education curricula globally (Baur et al., 2022 ).

Inquiry orientations are continuously emphasized in science education by the Organisation for Economic Cooperation and Development (OECD) operating in more than forty countries globally (OECD, 2015 , 2017 ) in the US (National Research Council [NRC], 2000 ), in Europe (European Commission and Directorate-General for Research and Innovation, 2015 ), and in nation states, such as in Ireland with the National Council for Curriculum and Assessment (NCCA, 2015 ).

The policy imperative for inquiry-oriented activities in science classrooms prompts a growing interest in assessing students’ scientific inquiry capabilities. While scientific inquiry is a well-established research area in science education (Fukuda et al., 2022 ), assessing students’ scientific inquiry capabilities is a growing topic of research, innovation and consideration.

There is a growing demand for innovative assessments that aim to either enhance or replace traditional summative methods. These assessments should focus on creating customized, student-centered formative tasks, tools, and techniques that capture both the final products and the processes used to achieve them (Hattie & Timperley, 2007 ). Many researchers argue that traditional models, originally designed to measure content knowledge, are no longer adequate for assessing competencies. Griffin et al. ( 2012 ) argued that traditional methods lack the ability to measure the higher-order skills, dispositions, and knowledge requirements of collaborative learning. Instead, it is asserted that modes of formative assessment can provide teachers and students with diagnostic information in order to continually adapt instruction and to foster a pedagogical cycle of learning (Kruit et al., 2018 ; Voogt & Roblin, 2012 ).

In this study, we systematically examined the construction, developmental trends and validation tasks, tools and techniques used in assessing students’ scientific inquiry capabilities in educational settings. We combined a systematic literature review from 2000 to 2024, using Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines with Bibliometric (Diodato & Gellatly, 2013 ) and Epistemic Network Analyses (ENA) (Shaffer et al., 2016 ). Our aim was to illuminate current global trends, possibilities and challenges in relation to the assessment of scientific inquiry and to suggest potential spaces for future research. Our study was guided by the following three research questions:

RQ1: To what extent is research on assessment of scientific inquiry in educational contexts found in the international literature?

RQ2: What are the predominant components, tasks, tools, and techniques used to assess scientific inquiry?

RQ3: What are the trends and developments in the assessment of scientific inquiry?

We structured the paper as follows. First, we briefly interrogate current conceptualisations of inquiry-based learning and scientific inquiry as an important background to the study. Second, we justify our selected methodology, the use of a systematic literature review with bibliometric and ENA analyses. Third, we present the results from each research question in turn. Finally, we discuss the changing shape of this research domain and the implications for the future of science education.

Conceptualizations of Scientific Inquiry

Here we first explore the construct of inquiry-based learning in science education before considering something of the global policy imperatives underway in this regard.

Inquiry-based Approach in Science Education

In science education, two visions of scientific literacy are discussed: Vision I emphasizes scientific content and propositional knowledge, while Vision II focuses on engaging students with real-world applications of science knowledge (Roberts, 2007 ; Roberts & Bybee, 2014 ). Achieving the scientific literacy mentioned in Vision II literacy is a key challenge for 21st-century science education, shifting towards enabling individuals to apply scientific concepts in everyday life rather than solely producing ‘mini-scientists’ (Roberts & Bybee, 2014 ). Balancing these visions is crucial to meeting diverse student needs and enhancing understanding science-in-context in today’s highly scientific world (Roberts & Bybee, 2014 ). Scientific inquiry is considered fundamental to scientific literacy, encompassing practices and epistemology, with a growing focus on the meaning, application and contexts of real world inquiry (Schwartz et al., 2023 ).

An inquiry-orientation therefore provides a pedagogical approach in which students learn by actively using scientific methods to reason and generate explanations in relation to design, data and evidence (Anderson, 2002 ; Stender et al., 2018 ). Neumann et al. ( 2011 ) considered the Nature of Science and Scientific Inquiry as separate domains for inquiry-orientations including for analysing data, identifying and controlling variables, and forming logical cause-and‐effect relationships. Wenning ( 2007 ) proposed a detailed rubric for developing proficiency in scientific inquiry, that included identifying a problem to be investigated, doing background research, using induction, formulating a hypothesis, incorporating logic and evidence, using deduction, generating a prediction, designing experimental procedures to test the prediction, conducting a scientific experiment, observing or simulating a test or model, collecting data, organizing, and analysing data accurately and precisely, applying statistical methods to support conclusions and communicating results. Moreover, Turner et al. ( 2018 ) grouped sixteen of the activities into three components of inquiry for secondary school students in science and math classrooms, namely working with hypotheses (i.e., generation of hypotheses/predictions, designing procedures), communication in inquiry (i.e., interpreting outcomes, asking questions), hands-on inquiry (i.e., recording data, visualising data).

Pedaste et al. ( 2015 ) conceptualised an inquiry-based learning framework of four phases based on their review of thirty-two studies: orientation , conceptualization , investigation , and conclusion . The orientation phase stimulates interest and curiosity, involves background research and results in the writing of a problem statement or topic by the teacher and/or students. Conceptualization involves formulating theory-based questions as predictions or hypotheses. The investigation phase turns curiosity into action through exploration, experimentation, data gathering and interpretation. In the conclusion phase, learners address their original research questions and consider whether these questions are answered, supported or refuted.

The studies showed that the inquiry-orientation enhanced comprehension (Marshall et al., 2017 ), fostered an appreciation of the nature of scientific knowledge (Dogan et al., 2024 ), improved students’ achievement in both scientific practices and conceptual knowledge (Marshall et al., 2017 ). Inquiry-based approach was found to positively impact student engagement and motivation while the hands-on experimental skills made learning science more enjoyable (Ramnarain, 2014 ). Inquiry activities make learning visible and help to integrate scientific reasoning skills for the social construction of knowledge (Stender et al., 2018 ).

Global Policy Imperatives in Relation to Scientific Inquiry

The US National Science Education Standards presented by the National Research Council (NRC, 1996 ) defined inquiry is “a multifaceted activity that involves making observations; posing questions; examining books and other sources of information to see what is already known; planning investigations; reviewing what is already known in light of experimental evidence; using tools to gather, analyze, and interpret data; proposing answers, explanations, and predictions; and communicating the results” (p. 23). Scientific inquiry encompasses the various methods scientists use to investigate the natural world and formulate explanations grounded in evidence from their research. It also involves students’ activities where they gain knowledge and understanding of scientific concepts and learn about the processes which scientists use to explore the natural world.

Later NRC standards (2000, 2006) elaborated such proficiency as identifying a scientific question, designing and conducting an investigation, using appropriate tools to collect and analyse data, and developing evidence-based explanations. The US framework for K-12 science education (NRC, 2012 ) focused on a few core ideas and concepts, integrating them with the practices needed for scientific inquiry and engineering design. The emphasis appeared to have shifted from “inquiry” to “scientific practices” as a basis of the framework (Rönnebeck et al., 2016 ). It listed eight components of scientific and engineering practices, including asking questions, developing and using models, planning and carrying out investigations, analyzing and interpreting data, using mathematics and computational thinking, constructing explanations, engaging in argument from evidence, obtaining, evaluating, and communicating information (NRC, 2012 ). The eight practices intentionally intersect and connect with others rather than stand-alone (NRC, 2012 ; Rönnebeck et al., 2016 ).

The Twenty First Century Science program (2006) in England emphasized a broad qualitative understanding of significant “whole explanations” and placed a strong focus on Ideas about Science . It also prioritized developing the understanding and skills needed to critically evaluate scientific information encountered in everyday life. This initiative focuses on a foundational course aimed at fostering scientific literacy among all students. It emphasized equipping students with the knowledge and skills needed to critically evaluate scientific information encountered in daily life​. This connects science to real-world contexts and applications, and the big ideas of science rather than isolated facts​ (Millar, 2006 ).

The 2015 Programme for International Student Assessment (PISA) specified a number of essential science inquiry competencies in three key areas: explaining phenomena scientifically, interpreting data and evidence scientifically, and evaluating and designing scientific inquiry (OECD, 2017 ). The explaining phenomena dimension involves students being able to identify, provide, and assess explanations for a variety of natural and technological phenomena. The interpreting dimension means that students can describe and evaluate scientific investigations and suggest methods for scientifically addressing questions. The designing dimension refers to students who can analyse and assess claims and arguments presented in various forms and draw accurate scientific conclusions (OECD, 2017 ).

In the 21st-century vision for science education in Europe, involving citizens as active participants in inquiry-oriented learning was essential (European Commission and Directorate-General for Research and Innovation, 2015 ). The scientific inquiry involves students identifying research problems and finding solutions that apply science to everyday life. Inquiry-based science education engages students in problem-based learning, hands-on experiments, self-regulated learning, and collaborative discussion, fostering a deep understanding of science and awareness of the practical applications of scientific concepts.

In summary, global policy imperatives focus on enhancing the cognitive processes and psychological characteristics of scientific inquiry and its application in real-world contexts. This approach consistently emphasizes inquiry as fundamental to teaching and learning science, although the focus has varied over time between Vision I and Vision II in relation to scientific literacy and science education.

Methodology

For the systematic literature review, we used the PRISMA methodology (Moher et al., 2009 ) in order to assemble an evidence base of relevant studies. This was further supported by Bibliometric analysis (Diodato & Gellatly, 2013 ) and ENA analysis (Shaffer et al., 2016 ). Bibliometric analysis is a quantitative method used to evaluate various aspects of academic publications within a specified field of study. It involves the application of mathematical and statistical tools to analyse patterns, and impact within a defined body of literature. It is a powerful tool for analysing the knowledge framework and structure in a specific research area (Diodato & Gellatly, 2013 ). Meanwhile, ENA is an analytical method to describe individual (group) framework patterns through quantitative analysis of data by examining the structure of the co-occurrence or connections in coded data (Shaffer et al., 2016 ). ENA can be used to compare units of analysis in terms of their plotted point positions, individual networks, mean plotted point positions, and mean networks, which average the connection weights across individual networks. This approach has been applied in several fields, including educational research (Ruis & Lee, 2021 ).

A comprehensive examination of extant literature was undertaken using the PRISMA-framework stages, with a specific focus on empirical research. The criterion for article selection was predicated on the utilization of a testing instrument for assessment of scientific inquiry. The inclusion criteria were threefold. Firstly, empirical studies that assessed the information retrieval abilities of students - qualitative, quantitative, or mixed methods - were considered. Secondly, the selected studies were required to incorporate scientific inquiry assessment tasks for K-12 science education. Thirdly, the chosen articles were limited to those originally published in the English language and within a timeline from 2000 to 2024 (09/06/2024).

We conducted a systematic search for academic papers in electronic databases as presented in Fig. 1 , employing specific search terms in the title, keywords, and abstract sections: (“inquiry” OR “scientific inquiry” OR “science inquiry” OR “investigation skill”) AND (“assessment” OR “testing” OR “measurement” OR “computer-based assessment”) AND NOT (“review”). The review used two scientific databases: Scopus and Web of Science (WoS). The results in Scopus and WoS suggested 2228 and 1532 references respectively through the first search strategy. After merging the two datasets based on articles’ DOIs indices, as well as following the removal of duplicate entries, we reached 589 articles. We continued to check the titles and abstracts of the remaining articles for pre-selection purposes based on our predefined inclusion criteria. The process led to the identification of 263 papers for further consideration. In this stage, the authors further discussed and agreed on the inclusion criteria, content relevance, methodological quality, and methodological relevance for the selection of papers. We also facilitated discussions among raters to build consensus on ambiguous cases. Finally, we ended up with sixty-three articles selected for our dataset. Then, the data were manually entered one by one, coded and documented for final selection.

Flowchart of the inclusion and exclusion process following PRISMA guidelines

To server our research questions, we collected information from the selected articles as a dataset for thematic analysis in the PRISMA framework. This information included: (1) year of publication, (2) age groups of the participants (categorized into four age groups: 5–10 years, 11–15 ages and 16–18 ages, (3) study context, (4) components of scientific inquiry, (5) instruments/tests, and (6) technique/validation approaches. (Readers can access full raw data at https://osf.io/5bt82 ).

For bibliometric analysis, the data of the selected articles was exported from the Scopus platform. It involved common bibliographical information such authors, title, year, DOI, affiliation, abstract, keyword and reference. We used bibliometric analysis via R software version 4.2.3 (R Core Team, 2023 ) with shiny (Chang et al., 2023 ) and bibliometrix package (Aria & Cuccurullo, 2017 ).

To facilitate for ENA analysis, we coded the data regarding components of scientific inquiry, based on existing frameworks (Table 1 ). The analyses were employed via ENA Web Tool (Marquart et al., 2018 ).

The results are presented here in relation to the key research questions. First, we present surface characteristics that provide a general overview of empirical studies on assessing scientific inquiry worldwide. Then, we explore the components, constructs, and techniques most often used in these assessments across the empirical studies with specific illustrative examples highlighted. Finally, we review the results to identify trends and developments in the assessment of scientific inquiry over time.

Studies on Measuring Scientific Inquiry in School Contexts Worldwide

The 63 selected articles comprised a total of 189 authors, with only four single-author articles (Kind, 2013 ; Mutlu, 2020 ; Sarıoğlu, 2023 ; Teig, 2024 ). Bibliometric analysis showed 3194 references cited, while international co-author index and co-author per article was 17.46% and 3.62, respectively. There were 21 papers published from 2000 to 2012. This number more than doubled to 42 articles from 2013 to 2024. The articles were published in 29 journals, with the core source recognized for the International Journal of Science Education ( IJSE) (11 articles) and the Journal of Research in Science Teaching (JRST) (10 articles), followed by the International Journal of Science and Mathematics Education ( IJSME) (7 articles), and the Research in Science and Technological Education (RSTE) (3 articles). Figure 2 depicts the cumulative articles of the core sources’ production during the period from 2000 to 2024. The graph shows the major journals contributing to this field of study (IJSE, JRST and IJSME), and the noticeable growth curve in the last decade.

The cumulative occurrence of articles in key journals published over time

The findings showed that the 63 articles have a global reach, with study contexts spanning 19 different countries and territories. Notably, a high proportion of studies (23 articles, 36.5%) come from the United States, followed by Taiwan (9 articles, 14.3%), Turkey (5 articles, 7.9%), and Germany (4 articles, 6.3%), while Israel and China each contributed 3 studies (4.8%). The distribution indicates that assessing scientific inquiry is a relatively attractive area of research in science education at a global level.

Regarding affiliation contribution, Fig. 3 shows that five universities emerge as the significant contributors to this collection of publications. Among these institutions, two are located in the US: The University of California (UC) and the Caltech Precollege Science Initiative (CAPSI). UC has remained consistently active in the field since 2002, while CAPSI’s involvement has stagnated since 2005. Humboldt University in Berlin (HU-Berlin) began contributing in 2012. Meanwhile, the National Taiwan Normal University (NTNU) has been actively contributing since 2013, with a sharp increase in activity. Beijing Normal University (BNU) entered the research landscape later, but has shown a steady increase in contributions recently. It is noted that the contributions refer to the frequency distribution of affiliations of all co-authors for each paper (Aria & Cuccurullo, 2017 ).

Top 5 of the research institution contribution over time

With respect to collaboration network in the research field, Fig. 4 represents collaborative patterns among researchers in selected articles, covering author and country levels. Based on the studies selected, the analysis identified 11 distinct research networks, illustrated in Fig. 4 a, that present as networks with a significant number of researchers. For instance, in the networks, we can find research groups such as the ones led by Wu, Linn, and Gobert. Furthermore, Fig. 4 b shows that the United States play a pivotal role in leading out international collaborations within the field of scientific inquiry assessment.

Collaboration networks of researchers identified in the articles selected

The cumulative participant count involved in all the studies totalled 50,470 individuals, encompassing educational levels from primary to high schools. Participant categorization was contingent upon respective age group, with a predominant focus on students at age range of 11–15 years. Notably, more than half of the studies (36 studies, accounting for 57.1%) were centred on participants in this age range. Following closely, another significant portion, comprising 23 studies (36.5%), targeted students in the 16-18-year students. It was noted that there are seven studies assessing students, covering two age range groups.

Task, Tests and Techniques of Assessing Scientific Inquiry

Components (facets) for assessing scientific inquiry.

In empirical studies selected, various assessment frameworks were introduced to evaluate scientific inquiry, each incorporating a diverse range of specific components. Zachos et al. ( 2000 ) considered scientific inquiry as multi-aspects of competence related to human cognitive characteristics. They employed hands-on performance assessment tasks, Floating and Sinking and the Period of Oscillation of a Pendulum, to assess students’ inquiry abilities within specific components: linking theory with evidence, formulating hypotheses, maintaining records, using appropriate or innovative laboratory materials, identifying cause-and-effect relationships, controlling experiments, and applying parsimony in drawing conclusions.

Cuevas and colleages ( 2005 ) developed contextual problem tasks to assess inquiry in five components: questioning, planning, implementing, concluding, and reporting. Their assessment task described a story about a child named Marie, who was trying to determine if the size of a container’s opening would influence the rate at which water evaporated. Students were asked to formulate a question reflecting the problem Marie was trying to solve, develop a hypothesis, design an investigation, list the materials needed, describe how to record results, and explain how to draw a conclusion. The framework were referred in a study by Turkan and Liu ( 2012 ) and later utilized in a study by Yang et al. ( 2016 ), where science inquiry was defined as comprising seven aspects of identifying a research question, formulating a hypothesis, designing an experimental procedure, planning necessary equipment and materials, collecting data and evidence, drawing evidence-based conclusions, and constructing conceptual understanding.

Other studies described inquiry as process skills (Kipnis & Hofstein, 2008 ), science process skills (Feyzíoglu, 2012 ; Temiz et al., 2006 ) and scientific process skills (Tosun, 2019 ). For example, Temiz et al. ( 2006 ) developed an instrument aimed to measure the development of 12 science process skills: formulating hypotheses, observing, manipulating materials, measuring, identifying and controlling variables, recording the data, demonstrating the ability to use numbers in space and time relationships, classifying, using the data to create models, predicting, interpreting data, and inferring information or solutions to problems. Meanwhile, an inquiry process framework of Kipnis and Hofstein ( 2008 ) included identifying problems, formulating hypotheses, designing an experiment, gathering and analysing data, and drawing conclusions about scientific problems and phenomena.

Furthermore, based on the previous studies (Gobert et al., 2013 ; Liu et al., 2008 ; Pine et al., 2006 ; Quellmalz et al., 2012 ; Zachos et al., 2000 ), Kuo et al., ( 2015 ) defined an inquiry proficiency framework to integrate cognitive skills with scientific knowledge during student participation in activities akin to scientific discovery. The framework emphasized four fundamental abilities as core components including questioning (e.g., asking and identifying questions), experimenting (e.g., identifying variables and planning experimental procedures), analysing (e.g., identifying relevant data and transforming data), and explaining (e.g., making a claim and using evidence). Their scenario-based tasks were created within a web-based application, covering four content areas (Physics, Chemistry, Biology, and Earth Science) across four inquiry abilities (Wu et al., 2015 ). Chi et al. ( 2019 ) defined scientific inquiry as the ability to integrate science knowledge and skills to identify scientific questions design and conduct investigation, analyse and interpret information and generate evidence-based explanations. A hands-on performance assessment instrument for measuring student scientific inquiry competences in the science lab was developed based on this framework (see a sample task in Fig. 5 a).

PISA 2015 developed the framework to assess 15-year-old students’ scientific inquiry competency of explaining phenomena, designing inquiry, interpreting data (OECD, 2017 ). Some empirical studies (e.g., Intasoi et al., 2020 ; Lin & Shie, 2024 ) developed assessment framework based on the framework to assess scientific inquiry competence of students. For example, Lin and Shie ( 2024 ) developed a PISA-type test to assess Grade 9 students’ scientific competence and knowledge related to curriculum and daily-life contexts (e.g., trolley motion, camping, household electricity, driving speed, etc.).

In the line, Arnold et al. ( 2018 ) referred to scientific inquiry as the competence to emphasize the cognitive aspects of the ability to use problem-solving procedures. Scientific competence was defined as the ability to understand, conduct, and critically evaluate scientific experiments on causal relationships, addressing problems and phenomena in the natural world. Three key sub-competences of experimentation were identified: generating hypotheses, designing experiments, and analysing data. Each sub-competence included five specific components. For instance, the sub-competence of generating hypotheses covered the ability to define the investigative problem, identify the relationship between dependent and independent variables to generate testable hypotheses or predictions and justify them, as well as propose different independent variables or alternative predictions. Zheng et al. ( 2022 ) categorized inquiry into eight components, highlighting information processing and reflective evaluation, echoed in study by Mutlu ( 2020 ).

In other approaches, Nowak et al. ( 2013 ) developed a model for assessing students’ inquiry ability, which had two dimensions: scientific reasoning (including question and hypothesis, plan and performance, and analysis and reflection) and inquiry methods (comprising modelling, experimenting, observing, comparing, and arranging). Together, these dimensions form a 9-cell matrix. Based on the theoretical structure, they developed a test instrument to assess students’ scientific inquiry (see sample item in Fig. 5 b). Meanwhile, Pedaste and colleages ( 2021 ) developed a science inquiry test for primary students based on the four-stage inquiry-based learning framework by Pedaste et al. ( 2015 ). The test encompassed the essential skills aligned with the four stages of the inquiry-based learning framework. These included analytical skills, which are primarily required in the Orientation, Conceptualization, and Investigation phases; planning skills, mainly needed in the Investigation phase; and interpretation skills, primarily needed in the Conclusion and Discussion phases.

Samples of the item/task for assessing scientific inquiry

A virtual experimentation environment developed by McElhaney and Linn ( 2011 ) simulated the experimentation activities of Airbags. These activities illustrated the interaction between the airbag and the driver during a head-on collision, using the steering wheel as a point of reference. Referred the existing studies (e.g., Kind, 2013 ; Liu et al., 2008 ; Pine et al., 2006 ), a simulation-based test developed by Wu et al. ( 2014 ) focused on two types of abilities: experimental and explaining. Experimental ability involved three sub-abilities: identifying and choosing variables, planning an experiment and selecting appropriate measurements, while explaining ability covered three sub-abilities: making a claim, using evidence, and evaluating alternative explanations. They designed four simulation tasks, namely Camera, Viscosity, Buoyancy and Flypaper. For example, the Flypaper task simulated a farm context in which students investigated which colour of flypaper could catch the most fruit flies. They were asked to propose hypotheses related to the decrease in flies according to the given chart, conduct appropriate experiments to measure the effect of the flypaper colour, investigate which colour of flypaper is best for catching fruit flies, and decide on alternative explanations based on the data evidence.

In the vein, Sui et al. ( 2024 ) designed an animation-based web application allow students conduct a scientific inquiry on atmospheric chemistry with animation experiments to understand the climate change and atmospheric chemistry. The scientific inquiry was defined with three core abilities: data analytic, control of variables and scientific reasoning. The digital game-based inquiry, BioScientist (Bónus et al., 2024 ) involved series of tasks, which focused on inquiry skills focusing on design of experiment, identification and control of variables, interpretation of data, and conclusion. For instance, a simulation provided some relevant variables, students need to manipulate the first one and then second variables to generate the data set. Based on the data-based evidence, they selected the answer and draw reasonable conclusions.

In summary, what becomes clear is that the mainstream framing of the construct of scientific inquiry was categorised as lists of specific components of competence. The frameworks for assessing scientific inquiry in technology-rich environments share many similarities with those used in traditional settings. In this view, it may summarise scientific competence into four main sub-competencies and their respective components (facets) based on the existing frameworks, as shown in Table 1 .

The Frequent Usage of the Components in Assessing Scientific Inquiry

In this section, we employed ENA to quantitatively visualize the usage frequency of yed ENA to quantitatively visualize the usage frequency of individual components and their co-usage with others in the selected empirical studies. Figure 6  illustrates the frequency of usage (represented by the size of the nodes) and the degree of co-usage of the components (represented by the width of the lines) across the reviewed studies.

In general, it appears that the nine facets were most often used to assess scientific inquiry, including formulate prediction or hypotheses (FP), formulate questions (FQ), set experiment (DS), vary independent variable (DV), measure dependent variable (DM), control confounding variables (DC), describe data (AD), interpret data (AI), and reach reasonable conclusion (CR). Other components were frequently used in inquiry tasks, including identify independent variable (FI), Identify dependent variable (FD), using appropriate method (AU) and evaluate methods (CE).

The pattern of components of scientific inquiry competence in selected studies simulated in the ENA model

Foundation Frameworks for Scientific Inquiry Assessment

To explore foundational frameworks for scientific inquiry assessment, we employed the Bibliometric analyses via the co-citation networks prevalent in the studies selected. The findings as depicted in Fig. 7 showed that US science education standards (NRC, 1996 ) stood out as the most frequently cited, followed by NRC texts A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas” (2012) and “Inquiry and the National Science Education Standards: A Guide for Teaching and Learning” (NRC, 2000 ). Other texts were often cited such as: “The development of scientific thinking skills in elementary and middle school” (Zimmerman, 2007 ) and “Next Generation Science Standards: For States, By States” (2013). It is clear that the 1996 NRC standards were prominently featured in the discussion, while the 2012 framework was referred to more frequently than the actual standards, particularly in terms of citations in the reviewed studies.

The co-citation networks found in the studies reviewed

Constructs, Formats and Techniques Approaches in Assessing Scientific Inquiry

Generally, three types of tests emerged within the realm of scientific inquiry assessment: hands-on performance assessment, a battery of independent tests (paper battery), and digital-based battery tests (online battery) and simulation performance assessment. The analysis revealed that paper battery (41.1%) and on-line battery tests (39.7%) were the most widely applied construct, followed by and simulation performance assessment (37.0%). Hands-on performance (17.6%) still continues to hold its place in the field. The findings also suggest that, regardless of the mode of assessment, multiple-choice (71.4%) and open-ended (69.8%) formats are consistently prevalent. Notably, several studies (44.5%) used a combination of multiple-choice and open-ended formats.

Assessment of Scientific Inquiry in Traditional Environment

Performance assessments represent a groundwork approach to measuring students’ capabilities in scientific investigation, conceptualization, and problem-solving within authentic contexts. Researchers explored various dimensions of hands-on performance assessments, designing tasks that authentically mirror the scientific process. For example, Zachos et al. ( 2000 ) developed performance tasks mirroring scientific inquiry processes, assessing concepts, data collection, and conclusion drawing. Pine et al. ( 2006 ) emphasized inquiry skills like planning and data interpretation. Emden and Sumfleth ( 2016 ) assessed students’ ability in generating ideas, planning experiments, and drawing conclusions through hands-on inquiry tasks. They used video analysis in combined with paper-pencil free response reports to measure performance.

Traditional assessments tend to rely on standardized tests, featuring multiple-choice items aligned with policy-led standards. These tests, often administered in a paper-and-pencil format, measure students’ proficiency levels in comparison with peers. Without the need for advanced technology, they covered a wide range of content and question types, including multiple-choice, short answer, and essays (Fig. 8 ). The majority of studies employed such a battery of independent tests to assess one or more components of scientific inquiry (e.g., Arnold et al., 2018 ; Kaberman & Dori, 2009 ; Kazeni et al., 2018 ; Lin & Shie, 2024 ; Nowak et al., 2013 ; Schwichow et al., 2016 ; Vo et al., 2023 ; Van Vo & Csapó, 2021 ). There were a significant positive correlations between the paper-and-pencil tests and performance assessment tasks (e.g., Kruit et al., 2018 ). Table 2 presents an excerpt from the summarised table of the studies selected (See more Supplemental material at https://osf.io/5bt82 ).

Samples of item/task assessing scientific inquiry in paper-based modality. a A sample item in requiring interpretation [source from Kruit et al. ( 2018 )]. b A sample of a task for assessing inquiry [source from Temiz et al. ( 2006 )]

Assessment of Scientific Inquiry in Digital Based Environments

From 2012 onwards, studies started to increasingly use advanced technologies in digital-based environments in their assessment of scientific inquiry. Studies (e.g., Gobert et al., 2013 ; Kuo et al., 2015 ; Quellmalz et al., 2012 ; Sui et al., 2024 ) started to use innovative tools and methodologies to construct assessment platforms that more accurately captured the nuanced complexities of scientific inquiry. These include dynamic simulations with web-based applications like (Quellmalz et al., 2012 , 2013 ), Inquiry Intelligent Tutoring System (Inq-ITS) (Gobert et al., 2013 ), 3D-game simulation (Hickey et al., 2009 ; Ketelhut et al., 2013 ), PISA 2015 (e.g., OECD, 2017 ; Teig et al., 2020 ) (see Fig. 9 ) and scenario-based tasks integrating multimedia elements (Kuo et al., 2015 ). For example, Inq-ITS is an online intelligent tutoring and assessment platform designed for physics, life science, and earth science. It aims to automatically evaluate scientific inquiry skills in real-time through interactive microworld simulations.

Simulation-based tools like Simulation-based assessment of scientific inquiry abilities (Wu et al., 2014 ; Wu & Wu, 2020 ) can effectively assess abilities in explaining and other relevant components. Immersive virtual settings and automated content scoring engines offered efficient evaluation methods (Baker et al., 2016 ; Liu et al., 2016 ; Scalise & Clarke-Midura, 2018 ; Sui et al., 2024 ) and were potential for formative assessment (Hickey et al., 2009 ). The digital game-based inquiry, i.e., BioScientist (Bónus et al., 2024 ), Quest Atlantis (Hickey et al., 2009 ), allowed students to engage with a series of tasks, which focused on inquiry skills using simulation in which students interacted with suitable elements during the inquiry process. Table 3 illustrates an excerpt regarding components, tools and techniques in digital-based scientific inquiry assessment (See Supplemental material at https://osf.io/5bt82 ).

A screenshot of item 3 of Task 1 from the PISA 2015 item from the Running in Hot Weather unit [Source from OECD ( 2015 )]

Techniques for Developing and Validating Scientific Inquiry Assessment

Most studies referred to the American Education Research Association (AERA, 1999 ) for developing and validating scientific inquiry assessment tasks. This included defining the assessment framework, designing tasks and items, scoring rubrics, and conducting a pilot test (Arnold et al., 2018 ; Kuo et al., 2015 ; Lin & Shie, 2024 ; Lin et al., 2016 ; Nowak et al., 2013 ; Schwichow et al., 2016 ; Vo & Csapó, 2023 ).

Numerous methods and techniques were employed for scoring proficiency in assessing scientific inquiry. Full credit was applied to correct answers in multiple-choice tests and partial credit to score open-ended questions (Arnold et al., 2018 ; Kaberman & Dori, 2009 ; OECD, 2017 ; Sui et al., 2024 ; Teig et al., 2020 ). Interestingly, a high percentage of studies, as much as 36.8%, utilized a 3-point scale rubric in their assessments or evaluations (Intasoi et al., 2020 ). Log-file techniques were increasingly popular for assessing scientific inquiry in recent studies (Baker et al., 2016 ; McElhaney & Linn, 2011 ; Teig, 2024 ; Teig et al., 2020 ). Data-mining algorithms enhanced assessment accuracy (Gobert et al., 2015 ). Virtual Performance Assessments allowed to record a log data (Baker et al., 2016 ), containing students’ actions (e.g., clicks, double clicks, slider movements, drag and drop, changes in the text area) along with the timestamp for each action. Different actions and their timings were combined to reveal behavioural indicators, such as number of actions, number of trials, time before the first action, response time for each item, and total time for each unit. The process of assessment development and validation was found to be based on a construct modelling approach (Brown & Wilson, 2011 ; Kuo et al., 2015 ).

For validation approaches, the face validity of the test instrument was established based on faculty and student feedback (Kuo et al., 2015 ) or expert judgments (Šmida et al., 2024 ; Vo & Csapó, 2023 ; Wu et al., 2014 ). Construct validity focused on the test score as a measure of the psychological properties of the instrument. For validity analysis, most studies applied Rasch measurement model (Arnold et al., 2018 ; Chi et al., 2019 ; Intasoi et al., 2020 ; Kuo et al., 2015 ; Lin & Shie, 2024 ; Liu et al., 2008 ; Nowak et al., 2013 ; Pedaste et al., 2021 ; Quellmalz et al., 2013 ; Scalise & Clarke-Midura, 2018 ; Schwichow et al., 2016 ; Vo & Csapó, 2023 ; Wu et al., 2015 ), followed by factor analyses (Feyzíoglu, 2012 ; Lou et al., 2015 ; Pedaste et al., 2021 ; Samarapungavan et al., 2009 ; Šmida et al., 2024 ; Tosun, 2019 ). Predictive or criterion-related validity was used to demonstrate that the test scores are dependent on other variables, tests, or outcome criteria. In assessment of scientific inquiry, predictive validity referred to some standard tests, such as Lawson’s Classroom Test of Scientific Inquiry (e.g., Kuo et al., 2015 ; Wu et al., 2014 ), Louisiana Educational Assessment Program (e.g., Lou et al., 2015 ), General cognitive ability (e.g., Dori et al., 2018 ; Kruit et al., 2018 ) and science grades in school (Pedaste et al., 2021 ).

Most popular software employed for data analysis including the R (Sui et al., 2024 ; Van Vo & Csapó, 2021 ), ConQuest (Kuo et al., 2015 ; Lin & Shie, 2024 ; Nowak et al., 2013 ; Seeratan et al., 2020 ; Vo & Csapó, 2021 ), SPSS (Bónus et al., 2024 ; Temiz et al., 2006 ) and Winsteps (Arnold et al., 2018 ; Chi et al., 2019 ; Pedaste et al., 2021 ), and LISREL (Tosun, 2019 ).

Developmental Trend in Assessing Scientific Inquiry

The objective here was to investigate the evolving trends and patterns of scientific inquiry employed within the studies over time. The articles were sub-divided into two distinct temporal spans − 2000–2012 and 2013–2024. Figure 10 visualizes patterns of components of scientific inquiry competence which were used the studies in the 2000–2012 period (Fig. 10 a), the 2013–2024 period (Fig. 10 b) and a comparison of that between the two periods (Fig. 10 c). The graph of comparison was calculated by subtracting the weight of each connection in one network from the corresponding connections in another.

The results revealed that some main components, i.e., measure dependent variable (DM), reach reasonable conclusion (CR), identify independent variable (FI), set experiment (DS), control confounding variables (DC), vary independent variable (DV), identify dependent variable (FD), and formulate prediction (FP), were often used consistently over time. However, components such as using appropriate method (AU), evaluate methods (CE), defining task time (DT), defining replication (DR), and recognizing limitations (CL) demonstrated a heightened prevalence in the later period, indicating a heightened emphasis on these aspects of assessing scientific inquiry. Conversely, when examining the earlier period (2000–2012), components like identify independent variable (FI) and justify question / hypothesis (FJ) exhibited a more noticeable frequency of application.

Patterns of facets of scientific inquiry competence in selected studies simulated in the ENA model

To streamline the understanding of these tests in the scientific inquiry tasks, we employed co-occurrence networks adapted in Bibliometric analysis. The analysis revealed that battery independent tests and performance assessment are most frequently used with multiple-choice and open-ended constructs. However, the trend is toward the online and simulation ones with new techniques of log-file tracking and scaffolding (Figure 11 a).

When it comes to emphasizing vision in science education, empirical evidence has shown that the design of inquiry tests incorporated both the content of pure science, vision I scientific literacy, and the science-in-context applications related to science, vision II scientific literacy. This ensures a balanced evaluation that covers fundamental scientific principles as well as their real-world applications. However, it is noteworthy that recent studies have shown a growing preference for assessing scientific inquiry within science-in-context (Figure 11 b).

Trend of types and formats in assessing scientific inquiry. a Co-occurrence networks depicting types, formats and “vision” emphasis. b Types, formats and “vision” emphasis over time

Discussion and Conclusions

The paper utilized the PRISMA guideline for systematic review in combination with bibliometric analyses for reviewing scientific research literature to draw together a detailed overview of research on assessing scientific inquiry abilities in global educational settings.

Our review of the problem of assessing scientific inquiry allowed us illuminate this rapidly changing area of research. In the last two decades, while research on curriculum reforms in science inquiry-orientations have proceeded apace, research on digital modes of assessing scientific inquiry have only recently started to make an impact. Our analysis of sixty-three studies showed that scientific inquiry has been emphasized, integrated, and assessed in the settings of science education around the world. The bulk of this research, started in the US, was brought to global significance through the influence of transnational policy decision-makers, such as the OECD and mainly US-led networks of researchers. The US researchers published several academic papers in the earliest part of the timeline studied, and their findings remain today as foundational citations. This research was quickly followed by new networks forming from Germany, Turkey, Taiwan and China. Co-citation networks revealed that the US National Science Education Standards (NRC, 1996 ) remains as a foundational reference, even though the 2012 document should have had nearly equal significance. Surprisingly, the American Association for the Advancement of Science (AAAS) benchmarks were not cited as frequently in the case.

Over two decades, performance assessments and batteries of independent tests, employing both multiple-choice and open-ended formats, continue to be widely used for assessing scientific inquiry. Hands-on performance assessment remains one of the main modes of assessing competence in scientific inquiry. Moreover, a traditional written test can be easily administered, reliably scored, and is familiar to students, but falls short in effectively capturing the dynamics of real-life inquiry and may be significantly influenced by reading proficiency (Kruit et al., 2018 ). Besides, hands-on performance assessment is not efficient for large-scale assessments (Kuo et al., 2015 ). Therefore, there is a growing emphasis on developing authentic tests. These tests, which may include manipulatives, are considered to provide a more comprehensive assessment of students’ capability in conducting scientific inquiry through multiple formats (e.g., open-constructed, multiple-choice, multiple-true-false, short closed-constructed).

Our analysis showed that original components like formulating questions or hypotheses, designing experiments, analysing data, and drawing conclusions were consistently used for assessing scientific inquiry capabilities over time. However, certain sub-components, such as formulating prediction or hypotheses , formulating questions , setting experiment , varying independent variable , measuring dependent variable , controlling confounding variables , describing data , interpreting data , and reaching reasonable conclusions , were the most frequently used competences in the selected studies. Meanwhile, facets like specifying test time , defining replication , and recognizing limitations were shown to have an increasing prevalence in the last decade. This trend signals a possible enhanced emphasis on these facets or sub-components of scientific inquiry, particularly in digital-based environments. The growing focus on these areas may reflect the advancements in technology that allow for more precise measurement and analysis, thereby promoting a more rigorous approach to scientific inquiry.

In the last decade, online battery tests and simulation performance assessments have gained increasing popularity. These studies reflect the design and enactment of innovative assessments using advanced technology, such as Web-based Inquiry Science Environments (McElhaney & Linn, 2011 ), SimScientists (Quellmalz et al., 2012 , 2013 ), iSA–Earth Science (Lou et al., 2015 ), Multimedia-based assessment of scientific inquiry abilities (Kuo et al., 2015 ; Wu et al., 2015 ), Inq-ITS system (Inquiry Intelligent Tutoring System (Gobert et al., 2013 , 2015 ), Virtual Performance Assessments (Baker et al., 2016 ), Dynamic visualization to design animation-based activities (Sui et al., 2024 ).

In terms of emphasizing vision in science education, empirical evidence demonstrated that the design of inquiry tests included pure science content (vision I) and science-in-context considerations (vision II). However, recent studies increasingly preferred assessing scientific inquiry within real-world contexts. This trend reflects an understanding of the importance of students being able to apply scientific concepts to real-world problems, thus preparing them for the complex, interdisciplinary challenges they are likely to face in their futures. By focusing on context, these studies aim to enhance students’ ability to think critically and engage with science in a way that is relevant to their everyday lives and broader community issues. These are also partly reflected in alignment with national and international frameworks.

Implications

The paper not only identifies various aspects of studies and research within a specific field of assessing inquiry competence, but also provides systematic rationales related to the construction of the tools, tasks and techniques used to assess scientific inquiry capabilities in educational settings. This is valuable for science teachers as they create inquiry-oriented tasks in their classrooms. Additionally, new researchers can gain an overview of the research teams working in this area.

The foreseeable trend may be that the move towards dynamic and interactive inquiry assessments enables researchers to examine not just the accuracy of students’ responses (product data) but also the procedures and actions they employ to arrive at responses (process data) (Teig, 2024 ). Multi-faceted aspects of scientific inquiry can be observed during assessment tasks. Beside traditional components in formulating questions or hypotheses , designing experiments , analysing data , and drawing conclusions , some new aspects like task time , replication and recognizing limitations seem to more consider as they become measurable in technology-rich environment. Therefore, log-file analysis will be more popular approach in the field.

The development of scientific inquiry assessments should be considered as a multifaceted process of construct modelling. The combination of multiple validity approaches is encouraged in development of the assessment of scientific inquiry. Psychometric analysis through Rasch model is often employed in validating and scaling student performance. Alternative approaches to deal with log-file records are still in the early pioneering stages of development (e.g., Baker et al., 2016 ; McElhaney & Linn, 2011 ; Teig, 2024 ; Teig et al., 2020 ). An automated scoring engine demonstrated a promising approach to scoring constructed-response in assessment of inquiry ability (Liu et al., 2016 ). This opens a potential space for upcoming new research in this field with application of artificial intelligence.

The review illuminates the evolving landscape of scientific inquiry assessment development and validation, emphasizing the importance of a comprehensive and flexible approach to meet the diverse needs of educational and research settings. However, tackling such novel tasks necessitated not only an understanding of scientific inquiry assessment but also sophisticated technology and its corresponding infrastructures. For example, simulation tasks addressing complex real-world problems, such as climate change, water shortages, and global food security, necessitate the collaboration of various relevant stakeholders. It is crucial for research and educational technology institutions to play supportive roles for science teachers. More robust and published research on scientist-led K-12 outreach is essential for enhancing comprehension among scientists and K-12 stakeholders regarding the optimal practices and challenges associated with outreach initiatives (Abramowitz et al., 2024 ).

Science teachers were encouraged to integrate both pure science content and science-in-context applications into their teaching and assessment (Roberts & Bybee, 2014 ). This will involve teachers’ designing inquiry-based activities that apply scientific principles to real-world problems, helping students develop higher-order critical thinking skills and preparing them for future interdisciplinary challenges. Emphasizing real-world applications of scientific inquiry can help to make science education more relevant and engaging for students.

Moreover, the adoption of combined approaches to the literature review, integrating bibliometric and ENA analyses with systematic review PRISMA guidelines, demonstrates a meticulous and systematic approach to data synthesis. Beyond its immediate application here, this research design may serve as a model for future research endeavours, contributing to the advancement of novel methodologies.

Limitation of the Review

The review conducted here was limited to 63 empirical studies published in SCOPUS/WoS data between 2000 and 2024 and in English. It may not cover the full range of academic documents that are made available in other academic databases, potentially missing significant studies published in different languages or within other research repositories.

The nature of psychological issues is often controversial, and our suggested framework for assessing scientific inquiry competence is merely one of several approaches presented in the literature. Different scholars proposed various models, each with its own strengths and limitations, reflecting the ongoing debate and complexity within this field. Furthermore, the selection of articles was conducted and scored by the authors, which introduces the possibility of certain biases. These biases may stem from subjective interpretations, or unintentional preferences, potentially influencing the overall findings.

The application of advanced technology is sophisticated and diverse; we have highlighted only a few features without covering all aspects of digital-based assessment. Therefore, generalizations from the study need to be approached with caution. However, the study provides valuable insights into the fast-globalizing landscape of assessing scientific inquiry and will be of interest to researchers, educators, teachers in science education and those with an interest in grappling with similar problems of assessment.

Abramowitz, B., Ennes, M., Kester, B., & Antonenko, P. (2024). Scientist-school STEM partnerships through outreach in the USA: A systematic review. International Journal of Science and Mathematics Education . https://doi.org/10.1007/s10763-024-10445-7

American Education Research Association, American Psychological Association, and National Council on Measurement in Education. (1999). Standards for educational and psychological testing . American Psychological Association. https://www.aera.net/Portals/38/1999Standards_revised.pdf

Anderson, R. D. (2002). Reforming science teaching: What research says about inquiry. Journal of Science Teacher Education, 13 (1), 1–12. https://doi.org/10.1023/A:1015171124982

Article   Google Scholar  

Aria, M., & Cuccurullo, C. (2017). Bibliometrix: An R-tool for comprehensive science mapping analysis. Journal of Informetrics , 11 (4), 959–975. https://doi.org/10.1016/j.joi.2017.08.007

Arnold, J. C., Boone, W. J., Kremer, K., & Mayer, J. (2018). Assessment of competencies in scientific inquiry through the application of Rasch measurement techniques. Education Sciences , 8 (4), 184. https://doi.org/10.3390/educsci8040184

Baker, R. S., Clarke-Midura, J., & Ocumpaugh, J. (2016). Towards general models of effective science inquiry in virtual performance assessments. Journal of Computer Assisted Learning , 32 (3), 267–280. https://doi.org/10.1111/jcal.12128

Baur, A., Baumgartner-hirscher, N., Lehtinen, A., Neudecker, C., Nieminen, P., Papaevripidou, M., Rohrmann, S., Schiffl, I., Schuknecht, M., Virtbauer, L., & Xenofontos, N. (Eds.). (2022). Differentiation in inquiry-based learning: A differentiation tool with a focus on experimentation . Heidelberg, University of Education.

Google Scholar  

Beaumont-Walters, Y., & Soyibo, K. (2001). An analysis of high school students’ performance on five integrated science process skills. Research in Science & Technological Education , 19 (2), 133–145. https://doi.org/10.1080/02635140120087687

Binkley, M., Erstad, O., Herman, J., Raizen, S., Ripley, M., Miller-ricci, M., & Rumble, M. (2012). Defining twenty-first century skills. In P. Griffin, B. McGaw, & E. Care (Eds.),  Assessment and Teaching of 21st Century skills (pp.17–66). Springer Science + Business Media. https://doi.org/10.1007/978-94-007-2324-5

Chapter   Google Scholar  

Bónus, L., Antal, E., & Korom, E. (2024). Digital game-based inquiry learning to improve eighth graders’ inquiry skills in Biology. Journal of Science Education and Technology , 33 , 1–17. https://doi.org/10.1007/s10956-024-10096-x

Brown, N. J. S., & Wilson, M. (2011). A model of cognition: The missing cornerstone of assessment. Educational Psychology Review, 23 (2), 221–234. https://doi.org/10.1007/s10648-011-9161-z

Chang, W., Cheng, J., Allaire, J., Sievert, C., Schloerke, B., Xie, Y., Allen, J., McPherson, J., Dipert, A., & Borges, B. (2023). shiny: Web application framework for R (R package version 1.8.0). https://cran.r-project.org/package=shiny

Chi, S., Wang, Z., & Liu, X. (2019). Investigating disciplinary context effect on student scientific inquiry competence. International Journal of Science Education , 41 (18), 2736–2764. https://doi.org/10.1080/09500693.2019.1697837

Chu, S. K. W., Reynolds, R. B., Tavares, N. J., Notari, M., & Lee, C. W. Y. (2017). 21st century skills development through inquiry-based learning: From theory to practice . Springer. https://doi.org/10.1007/978-981-10-2481-8

Clark, D., & Linn, M. C. (2009). Designing for knowledge integration: The impact of instructional time. Journal of Education , 189 (1–2), 139–158. https://doi.org/10.1177/0022057409189001-210

Cuevas, P., Lee, O., Hart, J., & Deaktor, R. (2005). Improving science inquiry with elementary students of diverse backgrounds. Journal of Research in Science Teaching , 42 (3), 337–357. https://doi.org/10.1002/tea.20053

Diodato, V. P., & Gellatly, P. (2013). Dictionary of bibliometrics . Routledge.

Book   Google Scholar  

Dogan, O. K., Han-Tosunoglu, C., Arslan, N., Cakir, M., & Irez, S. (2024). Middle school graduates’ understandings of scientific inquiry and its relation to academic achievement. International Journal of Science and Mathematics Education, 22 (1), 143–166. https://doi.org/10.1007/s10763-023-10365-y

Dori, Y. J., Zohar, A., Fischer-Shachor, D., Kohan-Mass, J., & Carmi, M. (2018). Gender-fair assessment of young gifted students’ scientific thinking skills. International Journal of Science Education , 40 (6), 595–620. https://doi.org/10.1080/09500693.2018.1431419

Emden, M., & Sumfleth, E. (2016). Assessing students’ experimentation processes in guided inquiry. International Journal of Science and Mathematics Education , 14 (1), 29–54. https://doi.org/10.1007/s10763-014-9564-7

Erduran, S. (2014). Beyond nature of science: The case for reconceptualising Science for science education. Science Education International , 25 (1), 95–111. https://eric.ed.gov/?id=EJ1022972

European Commission and Directorate-General for Research and Innovation. (2015). Science education for responsible citizenship: Report to the European Commission of the expert group on science education . Publications Office. https://doi.org/10.2777/12626

Feyzíoglu, B. (2012). Developing a science process skills test for secondary students: Validity and reliability study. Educational Sciences: Theory & Practices , 12 , 1899–1906. https://xlink.rsc.org/?DOI=C8RP00071A

Fukuda, M., Hajian, S., Jain, M., Liu, A. L., Obaid, T., Nesbit, J. C., & Winne, P. H. (2022). Scientific inquiry learning with a simulation: Providing within-task guidance tailored to learners’ understanding and inquiry skill. International Journal of Science Education , 44 (6), 1021–1043. https://doi.org/10.1080/09500693.2022.2062799

Gobert, J. D., Pedro, S., Raziuddin, M., J., & Baker, R. S. (2013). From log files to assessment metrics: Measuring students’ science inquiry skills using educational data mining. Journal of the Learning Sciences , 22 (4), 521–563. https://doi.org/10.1080/10508406.2013.837391

Gobert, J. D., Kim, Y. J., Sao Pedro, M. A., Kennedy, M., & Betts, C. G. (2015). Using educational data mining to assess students’ skills at designing and conducting experiments within a complex systems microworld. Thinking Skills and Creativity , 18 , 81–90. https://doi.org/10.1016/j.tsc.2015.04.008

Griffin, P., McGaw, B., & Care, E. (Eds.). (2012).  Assessment and teaching of 21st century skills . Springer Science + Business Media. https://doi.org/10.1007/978-94-007-2324-5

Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research , 77 (1), 81–112. https://doi.org/10.3102/003465430298487

Hickey, D. T., Ingram-Goble, A. A., & Jameson, E. M. (2009). Designing assessments and assessing designs in virtual educational environments. Journal of Science Education and Technology , 18 (2), 187–208. https://doi.org/10.1007/s10956-008-9143-1

Intasoi, S., Junpeng, P., Tang, K. N., Ketchatturat, J., Zhang, Y., & Wilson, M. (2020). Developing an assessment framework of multidimensional scientific competencies. International Journal of Evaluation and Research in Education (IJERE) , 9 (4), 963–970. https://doi.org/10.11591/ijere.v9i4.20542

Kaberman, Z., & Dori, Y. J. (2009). Question posing, inquiry, and modeling skills of Chemistry students in the case-based computerized laboratory environment. International Journal of Science and Mathematics Education , 7 (3), 597–625. https://doi.org/10.1007/s10763-007-9118-3

Kazeni, M., Baloyi, E., & Gaigher, E. (2018). Effectiveness of individual and group investigations in developing integrated science inquiry skills. South African Journal of Education , 38 (3), 1–12. https://doi.org/10.15700/saje.v38n3a1549

Ketelhut, D., Nelson, B., Schifter, C., & Kim, Y. (2013). Improving science assessments by situating them in a virtual environment. Education Sciences , 3 (2), 172–192. https://doi.org/10.3390/educsci3020172

Kind, P. M. (2013). Establishing assessment scales using a novel disciplinary rationale for scientific reasoning. Journal of Research in Science Teaching , 50 (5), 530–560. https://doi.org/10.1002/tea.21086

Kipnis, M., & Hofstein, A. (2008). The inquiry laboratory as a source for development of metacognitive skills. International Journal of Science and Mathematics Education , 6 (3), 601–627. https://doi.org/10.1007/s10763-007-9066-y

Kruit, P. M., Oostdam, R. J., van den Berg, E., & Schuitema, J. A. (2018). Assessing students’ ability in performing scientific inquiry: Instruments for measuring science skills in primary education. Research in Science and Technological Education , 36 (4), 413–439. https://doi.org/10.1080/02635143.2017.1421530

Kuo, C. Y., Wu, H. K., Jen, T. H., & Hsu, Y. S. (2015). Development and validation of a multimedia-based assessment of scientific inquiry abilities. International Journal of Science Education , 37 (14), 2326–2357. https://doi.org/10.1080/09500693.2015.1078521

Lin, S. F., & Shie, W. C. (2024). A cooperative model of development and validation of a curriculum-based scientific competence test. International Journal of Science and Mathematics Education , 22 (3), 491–514. https://doi.org/10.1007/s10763-023-10366-x

Lin, S. W., Liu, Y., Chen, S. F., Wang, J. R., & Kao, H. L. (2015). Development of a computer-based measure of listening comprehension of science talk. International Journal of Science and Mathematics Education , 13 (6), 1469–1486. https://doi.org/10.1007/s10763-014-9559-4

Lin, S. W., Liu, Y., Chen, S. F., Wang, J. R., & Kao, H. L. (2016). Elementary school students’ science talk ability in inquiry-oriented settings in Taiwan: Test development, verification, and performance benchmarks. International Journal of Science and Mathematics Education , 14 (7), 1199–1214. https://doi.org/10.1007/s10763-015-9663-0

Liu, O. L., Lee, H. S., Hofstetter, C., & Linn, M. (2008). Assessing knowledge integration in science: Construct, measures, and evidence. Educational Assessment , 13 (1), 33–55. https://doi.org/10.1080/10627190801968224

Liu, O. L., Rios, J. A., Heilman, M., Gerard, L., & Linn, M. C. (2016). Validation of automated scoring of science assessments. Journal of Research in Science Teaching , 53 (2), 215–233. https://doi.org/10.1002/tea.21299

Lou, Y., Blanchard, P., & Kennedy, E. (2015). Development and validation of a science inquiry skills assessment. Journal of Geoscience Education , 63 (1), 73–85. https://doi.org/10.5408/14-028.1

Marquart, C. L., Hinojosa, C., Swiecki, Z., Eagan, B., & Shaffer, D. W. (2018). Epistemic network analysis (Version 1.7.0) [Software] .

Marshall, J. C., Smart, J. B., & Alston, D. M. (2017). Inquiry-based instruction: A possible solution to improving student learning of both science concepts and scientific practices. International Journal of Science and Mathematics Education , 15 (5), 777–796. https://doi.org/10.1007/s10763-016-9718-x

McElhaney, K. W., & Linn, M. C. (2011). Investigations of a complex, realistic task: Intentional, unsystematic, and exhaustive experimenters. Journal of Research in Science Teaching , 48 (7), 745–770. https://doi.org/10.1002/tea.20423

Millar, R. (2006). Twenty First Century Science: Insights from the design and implementation of a scientific literacy approach in school science. International Journal of Science Education , 28 (13), 1499–1521. https://doi.org/10.1080/09500690600718344

Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Journal of Clinical Epidemiology , 62 (10), 1006–1012. https://doi.org/10.1016/j.jclinepi.2009.06.005

Mutlu, A. (2020). Evaluation of students’ scientific process skills through reflective worksheets in the inquiry-based learning environments. Reflective Practice , 21 (2), 271–286. https://doi.org/10.1080/14623943.2020.1736999

National Council for Curriculum and Assessment [NCCA]. (2015). Junior cycle science: Curriculum specification. Author.

National Research Council [NRC]. (1996). National Science education standards. National committee for science education standards and assessment . National Academies.

National Research Council [NRC]. (2000). Inquiry and the national science education standards: A guide for teaching and learning . National Academies.

National Research Council [NRC]. (2012). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas . National Academies. https://smile.oregonstate.edu/sites/smile.oregonstate.edu/files/a_framework_for_k-12_science_education.pdf

Neumann, I., Neumann, K., & Nehm, R. (2011). Evaluating instrument quality in science education: Rasch-based analyses of a nature of science test. International Journal of Science Education , 33 (10), 1373–1405. https://doi.org/10.1080/09500693.2010.511297

Nowak, K. H., Nehring, A., Tiemann, R., & Upmeier, A. (2013). Assessing students’ abilities in processes of scientific inquiry in biology using a paper-and-pencil test. Journal of Biological Education , 47 (3), 182–188. https://doi.org/10.1080/00219266.2013.822747

Organisation for Economic Cooperation and Development [OECD]. (2017). PISA 2015 assessment and analytical framework . Author. https://doi.org/10.1787/9789264281820-en

Organisation for Economic Cooperation and Development [OECD]. (2015). Try PISA 2015 test questions . Author. https://www.oecd.org/pisa/test/pisa2015/#d.en.537240

Pedaste, M., Mäeots, M., Siiman, L. A., de Jong, T., van Riesen, S. A. N., Kamp, E. T., Manoli, C. C., Zacharia, Z. C., & Tsourlidaki, E. (2015). Phases of inquiry-based learning: Definitions and the inquiry cycle. Educational Research Review , 14 , 47–61. https://doi.org/10.1016/j.edurev.2015.02.003

Pedaste, M., Baucal, A., & Reisenbuk, E. (2021). Towards a science inquiry test in primary education: Development of items and scales. International Journal of STEM Education , 8 (1), Article 19. https://doi.org/10.1186/s40594-021-00278-z

Pine, J., Aschbacher, P., Roth, E., Jones, M., McPhee, C., Martin, C., Phelps, S., Kyle, T., & Foley, B. (2006). Fifth graders’ science inquiry abilities: A comparative study of students in hands-on and textbook curricula. Journal of Research in Science Teaching , 43 (5), 467–484. https://doi.org/10.1002/tea.20140

Quellmalz, E. S., Timms, M. J., Silberglitt, M. D., & Buckley, B. C. (2012). Science assessments for all: Integrating science simulations into balanced state science assessment systems. Journal of Research in Science Teaching , 49 (3), 363–393. https://doi.org/10.1002/tea.21005

Quellmalz, E. S., Davenport, J. L., Timms, M. J., DeBoer, G. E., Jordan, K. A., Huang, C. W., & Buckley, B. C. (2013). Next-generation environments for assessing and promoting complex science learning. Journal of Educational Psychology , 105 (4), 1100–1114. https://doi.org/10.1037/a0032220

R Core Team. (2023). R: A language and environment for statistical computing . R Foundation for Statistical Computing. https://www.r-project.org/

Ramnarain, U. D. (2014). Teachers ’ perceptions of inquiry-based learning in urban, suburban, township and rural high schools : The context-speci fi city of science curriculum implementation in South Africa. Teaching and Teacher Education , 38 , 65–75. https://doi.org/10.1016/j.tate.2013.11.003

Roberts, D. A. (2007). Scientific literacy/science literacy. In S. K. Abell, & N. G. Lederman (Eds.), Handbook of research on science education (pp. 729–780). Routledge.

Roberts, D. A., & Bybee, R. W. (2014). Scientific literacy, science literacy, and science education. In N. G. Lederman & S. K. Abell (Eds.), Handbook of research on science education (Vol. 2, pp. 559–572). Routledge.

Rönnebeck, S., Bernholt, S., & Ropohl, M. (2016). Searching for a common ground – A literature review of empirical research on scientific inquiry activities. Studies in Science Education , 52 (2), 161–197. https://doi.org/10.1080/03057267.2016.1206351

Ruis, A. R., & Lee, S. B. (Eds.). (2021). Advances in quantitative ethnography (Vol. 1312). Springer International Publishing. https://doi.org/10.1007/978-3-030-67788-6

Ruiz-Primo, M. A., Li, M., Tsai, S. P., & Schneider, J. (2010). Testing one premise of scientific inquiry in science classrooms: Examining students’ scientific explanations and student learning. Journal of Research in Science Teaching , 47 (5), 583–608. https://doi.org/10.1002/tea.20356

Samarapungavan, A., Mantzicopoulos, P., Patrick, H., & French, B. (2009). The development and validation of the science learning assessment (SLA): A measure of kindergarten science learning. Journal of Advanced Academics , 20 (3), 502–535. https://doi.org/10.1177/1932202X0902000306

Sarıoğlu, S. (2023). Development of Online science process skills test for 8th grade pupils. Journal of Turkish Science Education , 20 (3), 418–432. https://doi.org/10.36681/tused.2023.024

Scalise, K., & Clarke-Midura, J. (2018). The many faces of scientific inquiry: Effectively measuring what students do and not only what they say. Journal of Research in Science Teaching , 55 (10), 1469–1496. https://doi.org/10.1002/tea.21464

Schwartz, R. S., Lederman, J. S., & Enderle, P. J. (2023). Scientific inquiry literacy: The missing link on the continuum from science literacy to scientific literacy. In N. G. Lederman, D. L. Zeidler, & J. S. Lederman (Eds.), Handbook of research on science education (pp. 749–782). Routledge.

Schwichow, M., Christoph, S., Boone, W. J., & Härtig, H. (2016). The impact of sub-skills and item content on students’ skills with regard to the control-of-variables strategy. International Journal of Science Education , 38 (2), 216–237. https://doi.org/10.1080/09500693.2015.1137651

Seeratan, K. L., McElhaney, K. W., Mislevy, J., McGhee, R., Conger, D., & Long, M. C. (2020). Measuring students’ ability to engage in scientific inquiry: A new instrument to assess data analysis, explanation, and argumentation. Educational Assessment , 25 (2), 112–135. https://doi.org/10.1080/10627197.2020.1756253

Shaffer, D., williamson, Collier, W., & Ruis, A. R. (2016). A tutorial on epistemic network analysis: Analyzing the structure of connections in cognitive, cocial, and interaction data. Journal of Learning Analytics , 3 (3), 9–45. https://doi.org/10.18608/jla.2016.33.3

Shahali, E. H. M., & Halim, L. (2010). Development and validation of a test of integrated science process skills. Procedia - Social and Behavioral Sciences , 9 , 142–146. https://doi.org/10.1016/j.sbspro.2010.12.127

Šmida, D., Čipková, E., & Fuchs, M. (2024). Developing the test of inquiry skills: Measuring the level of inquiry skills among pupils in Slovakia. International Journal of Science Education , 46 (1), 73–108. https://doi.org/10.1080/09500693.2023.2219811

Stender, A., Schwichow, M., Zimmerman, C., & Härtig, H. (2018). Making inquiry-based science learning visible: The influence of CVS and cognitive skills on content knowledge learning in guided inquiry. International Journal of Science Education , 40 (15), 1812–1831. https://doi.org/10.1080/09500693.2018.1504346

Sui, C., Hsiao, S., Yeh, S., Zhao, P., Chang, C., & Lin, J. (2024). Do they have inquiry skill profiles? Exploring high school students’ scientific inquiry in an animation-based activity. Science Education, 108 (2), 467–494. https://doi.org/10.1002/sce.21844

Teig, N. (2024). Uncovering student strategies for solving scientific snquiry tasks: Insights from student process data in PISA. Research in Science Education , 54 , 205–224. https://doi.org/10.1007/s11165-023-10134-5

Teig, N., Scherer, R., & Kjærnsli, M. (2020). Identifying patterns of students’ performance on simulated inquiry tasks using PISA 2015 log-file data. Journal of Research in Science Teaching , 57 (9), 1400–1429. https://doi.org/10.1002/tea.21657

Temiz, B. K., Taşar, M. F., & Tan, M. (2006). Development and validation of a test of integrated science process skills. International Education Journal , 7 (7), 1007–1027. https://doi.org/10.1016/j.sbspro.2010.12.127

Tosun, C. (2019). Scientific process skills test development within the topic Matter and its nature and the predictive effect of different variables on 7th and 8th grade students’ scientific process skill levels. Chemistry Education Research and Practice , 20 (1), 160–174. https://doi.org/10.1039/C8RP00071A

Toth, E. E., Suthers, D. D., & Lesgold, A. M. (2002). Mapping to know: The effects of representational guidance and reflective assessment on scientific inquiry. Science Education , 86 (2), 264–286. https://doi.org/10.1002/sce.10004

Turkan, S., & Liu, O. L. (2012). Differential performance by English language learners on an inquiry-based science assessment. International Journal of Science Education , 34 (15), 2343–2369. https://doi.org/10.1080/09500693.2012.705046

Turner, R. C., Keiffer, E. A., & Salamo, G. J. (2018). Observing inquiry-based learning environments using the scholastic inquiry observation instrument. International Journal of Science and Mathematics Education , 16 (8), 1455–1478. https://doi.org/10.1007/s10763-017-9843-1

Van Vo, D., & Csapó, B. (2021). Development of scientific reasoning test measuring control of variables strategy in physics for high school students: evidence of validity and latent predictors of item difficulty. International Journal of Science Education, 43 (13), 2185–2205. https://doi.org/10.1080/09500693.2021.1957515

Vo, D. V., & Csapó, B. (2023). Exploring inductive reasoning, scientific reasoning and science motivation, and their role in predicting STEM achievement across grade levels. International Journal of Science and Mathematics Education, 21 (8), 2375–2398. https://doi.org/10.1007/s10763-022-10349-4

Vo, D. V., Csapó, B., & Greiff, S. (2023). Development of the control of variables strategy in physics among secondary school students. Thinking Skills and Creativity, 49, 101371. https://doi.org/10.1016/J.TSC.2023.101371

Voogt, J., & Roblin, N. P. (2012). A comparative analysis of international frameworks for 21stcentury competences: Implications for national curriculum policies. Journal of Curriculum Studies , 44 (3), 299–321. https://doi.org/10.1080/00220272.2012.668938

Wallace, M. F. G., Bazzul, J., Higgins, M., & Tolbert, S. (Eds.). (2022). Reimagining science education in the Anthropocene . Palgrave Macmillan.

Wenning, C. J. (2007). Assessing inquiry skills as a component of scientific literacy. Journal of Physics Teacher Education Online , 4 (2), 21–24.

Wu, P. H., & Wu, H. K. (2020). Constructing a model of engagement in scientific inquiry: Investigating relationships between inquiry-related curiosity, dimensions of engagement, and inquiry abilities. Instructional Science , 48 (1), 79–113. https://doi.org/10.1007/s11251-020-09503-8

Wu, P. H., Wu, H. K., & Hsu, Y. S. (2014). Establishing the criterion-related, construct, and content validities of a simulation-based sssessment of inquiry abilities. International Journal of Science Education , 36 (10), 1630–1650. https://doi.org/10.1080/09500693.2013.871660

Wu, H. K., Kuo, C. Y., Jen, T. H., & Hsu, Y. S. (2015). What makes an item more difficult? Effects of modality and type of visual information in a computer-based assessment of scientific inquiry abilities. Computers & Education , 85 , 35–48. https://doi.org/10.1016/j.compedu.2015.01.007

Yang, K. K., Lin, S. F., Hong, Z. R., & Lin, H. (2016). Exploring the assessment of and relationship between elementary students’ scientific creativity and science inquiry. Creativity Research Journal , 28 (1), 16–23. https://doi.org/10.1080/10400419.2016.1125270

Zachos, P., Hick, T. L., Doane, W. E. J., & Sargent, C. (2000). Setting theoretical and empirical foundations for assessing scientific inquiry and discovery in educational programs. Journal of Research in Science Teaching , 37 (9), 938–962. https://doi.org/10.1002/1098-2736(200011)37:9<938::AID-TEA5>3.0.CO;2-S

Zheng, Y., Yu, S., Zhang, M., Wang, J., Yang, X., Zheng, S., & Ping, X. (2022). Research on performance assessment of students’ inquiry skills in China’s elementary schools: A video analysis of Beijing discovering science around us. Research in Science & Technological Education . https://doi.org/10.1080/02635143.2022.2126973

Zimmerman, C. (2007). The development of scientific thinking skills in elementary and middle school. Developmental Review, 27 (2), 172–223. https://doi.org/10.1016/j.dr.2006.12.001

Download references

Open Access funding provided by the IReL Consortium. This research was supported by the Irish Research Council under grant number: GOIPD/2023/148.

Author information

Authors and affiliations.

EPI•STEM National Research Centre for STEM Education, School of Education, University of Limerick, Castletroy, Limerick, V94 T9PX, Ireland

De Van Vo & Geraldine Mooney Simmie

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to De Van Vo .

Ethics declarations

Declarations of originality.

This manuscript has not been published elsewhere and is not under consideration by another journal.

Conflict of interest

The authors have stated no potential conflict of interest.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Vo, D.V., Simmie, G. Assessing Scientific Inquiry: A Systematic Literature Review of Tasks, Tools and Techniques. Int J of Sci and Math Educ (2024). https://doi.org/10.1007/s10763-024-10498-8

Download citation

Received : 28 February 2024

Accepted : 17 August 2024

Published : 04 September 2024

DOI : https://doi.org/10.1007/s10763-024-10498-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Scientific Inquiry
• Science Education
• Systematic Literature Review
• Epistemic Network Analysis
• Bibliometrix
• Find a journal
• Publish with us
• Track your research

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• For authors
• Browse by collection
• BMJ Journals

You are here

• Volume 14, Issue 9
• Meta-analysis and systematic review of the diagnostic value of contrast-enhanced spectral mammography for the detection of breast cancer
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• Jiulin Liu 1 , 2 ,
• Ran Xiao 3 ,
• Huijia Yin 1 ,
• Ying Hu 1 ,
• Siyu Zhen 1 ,
• Shihao Zhou 1 , 2 ,
• http://orcid.org/0000-0001-8516-1396 Dongming Han 1
• 1 Department of Magnetic Resonance Imaging (MRI) , The First Affiliated Hospital of Xinxiang Medical University , Weihui , Henan , China
• 2 Department of Radiology , Luoyang Orthopedic-Traumatological Hospital of Henan Province (Henan Provincial Orthopedic Hospital) , Zhengzhou , Henan , China
• 3 Department of Respiratory Medicine , The First Affiliated Hospital of Xinxiang Medical University , Weihui , Henan , China
• Correspondence to Dr Dongming Han; 625492590{at}qq.com

Objective The objective is to evaluate the diagnostic effectiveness of contrast-enhanced spectral mammography (CESM) in the diagnosis of breast cancer.

Data sources PubMed, Embase and Cochrane libraries up to 18 June 2022.

Eligibility criteria for selecting studies We included trials studies, compared the results of different researchers on CESM in the diagnosis of breast cancer, and calculated the diagnostic value of CESM for breast cancer.

Data extraction and synthesis Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2) evaluated the methodological quality of all the included studies. The study followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses specification. In addition to sensitivity and specificity, other important parameters were explored in an analysis of CESM accuracy for breast cancer diagnosis. For overall accuracy estimation, summary receiver operating characteristic curves were calculated. STATA V.14.0 was used for all analyses.

Results This meta-analysis included a total of 12 studies. According to the summary estimates for CESM in the diagnosis of breast cancer, the pooled sensitivity and specificity were 0.97 (95% CI 0.92 to 0.98) and 0.76 (95% CI 0.64 to 0.85), respectively. Positive likelihood ratio was 4.03 (95% CI 2.65 to 6.11), negative likelihood ratio was 0.05 (95% CI 0.02 to 0.09) and the diagnostic odds ratio was 89.49 (95% CI 45.78 to 174.92). Moreover, there was a 0.95 area under the curve.

Conclusions The CESM has high sensitivity and good specificity when it comes to evaluating breast cancer, particularly in women with dense breasts. Thus, provide more information for clinical diagnosis and treatment.

• breast imaging
• breast tumours
• diagnostic radiology

Data availability statement

Data sharing not applicable as no datasets generated and/or analysed for this study.

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See:  http://creativecommons.org/licenses/by-nc/4.0/ .

https://doi.org/10.1136/bmjopen-2022-069788

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

STRENGTHS AND LIMITATIONS OF THIS STUDY

This systematic review was a comprehensive search of experimental and observational studies on contrast-enhanced spectral mammography (CESM) in the diagnosis of breast cancer.

We included only prospective studies. Prospective studies were of higher quality with less bias, and our study screening criteria were developed prior to the meta-analysis.

The study was conducted by two people and was strictly based on inclusion criteria.

The data in this study were summarised using sound statistical methods.

A recent literature was added, and a literature from the same institution included only the most recent or the largest sample size.

We summarised the sensitivity and specificity of CESM in the diagnosis of breast cancer.

Introduction

Globally, female breast cancer has overtaken lung cancer as the leading cause of cancer death, making it the fifth most common cause of death. 1 From the mid-20th century, the incidence of breast cancer in women has been increasing slowly by about 0.5% per year. 2 At present, the diagnostic methods of breast cancer include MRI, full field digital mammography (FFDM) and ultrasound (US). MRI is the most sensitive examination in the diagnosis of breast cancer at present. 3 However, it has some disadvantages such as no claustrophobic and high price. In addition, although FFDM is an effective diagnostic method for breast cancer, it also has the hazard of recall and needs further testing. 4 Ultrasonography has good diagnostic efficacy for breast cancer, especially in women with dense breasts; however, it has a relatively low positive predictive value. 5 Contrast-enhanced spectral mammography (CESM), which visualises breast neovascularisation in a manner similar to MRI, is an emerging technology that uses iodine contrast agent. 6 CESM has the advantages of patient friendliness and low cost. Previous studies have shown that CESM has obvious advantages in displaying lesions compared with US. The advantage of CESM is that it can show changes in anatomy and local blood perfusion, which may be caused by tumour angiogenesis. 7 Moreover, CESM is useful in detecting the suspicious findings in routine breast imaging 7 and the sensitivity and specificity of CESM are different in different studies.

It has been reported that several meta-analyses have been conducted regarding the diagnostic performance of CESM for breast cancer; however, their pooled results were different and had several limitations. 8–11 On the one hand, the sensitivity and specificity differed across the above-mentioned meta-analyses. 8 10 11 On the other hand, the numbers of included studies were limited. In addition, partial meta-analyses included none-English studies and overlapped studies, which might affect their pooled results. In the past few years, several studies evaluating the diagnostic value of CESM in breast cancer have been published. Therefore, we conducted this meta-analysis using available evidence to comprehensively determine whether CESM is effective in detecting breast cancer in women.

Material and methods

As recommended by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), we conducted our study followed the PRISMA specification, 12 which met the requirements of diagnostic systematic review.

Search strategy

To evaluate the accuracy of CESM in diagnosing breast cancer, we retrieved the following databases: PubMed, Embase and Cochrane library. Two reviewers, JL and RX, independently searched the above databases up to the date of 18 June 2022. Our searching terms included ‘contrast-enhanced spectral mammography’, ‘Dual-Energy Contrast-Enhanced Spectral Mammography’, ‘CESM’, ‘contrast-enhanced digital mammography’, ‘CEDM’, ‘Breast Neoplasms’, ‘Breast Neoplasm’, ‘Breast Tumor’, ‘Breast Tumors’, ‘Breast Cancer’, ‘Malignant Neoplasm of Breast’, ‘Breast Malignant Neoplasm’, ‘Breast Carcinomas’, ‘Breast Carcinoma’, ‘breast mass’, ‘breast lesion’, ‘breast lesions’, ‘breast diseases’. In addition, the references of all the included studies were also reviewed.

Inclusion and exclusion criteria

Following is the list of inclusion criteria: (1) studies diagnosing breast cancer, (2) studies provided data on the sensitivity and specificity, (3) studies involving ≥10 patients or case, (4) English language and(5) prospective studies. Following is the list of exclusion criteria: (1) overlapped research, (2) commentaries, letters, editorials or abstracts or (3) studies referencing artificial intelligence and radiomics.

Study screening

The titles and abstracts of the literature in the electronic databases were initially screened by two authors, following the above criteria for inclusion and exclusion. Each of the two researchers screened two times to avoid omission. If there is any disagreement, the third author was consulted to decide. Eligibly downloaded full texts and further screened. First, if the authors and institutions of the study are the same, we will include the most recently published studies with the largest sample size. If the research institutions are the same, but the authors are different, we will send an email to the corresponding authors to ask. If we do not receive a reply, we will include the most recently published studies having the largest sample size.

Data abstraction

Two reviewers extracted data. If necessary, the difference shall be solved by the third reviewer. Each study was analysed for the following information: first author name, publication year, country, the numbers of patients and lesions, median age, the results of true positive (TP), false positive (FP), false negative (FN) and true negative (TN).

Quality assessment

The quality of the methodology included in the publication was assessed by the Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2). 13 QUADAS-2 were mainly focused on the following four domains: patient selection, index test, reference standard and flow and timing, with minimal overlapping, which present the main quality of the diagnostic study. Each domain is assessed according to risk of bias, with the three domains assessed according to applicability. The risk of bias was considered low if the study met the above criteria and high otherwise. Disagreements between the two reviewers on quality assessment were resolved by consensus.

Statistical analysis

STATA V.14.0 was used for all analyses. I 2 measure was used to quantify the heterogeneity between studies. If there is no statistical heterogeneity, the fixed effect model is used to consolidate the data. On the contrary, the random effect model is used to summarise the data. The sensitivity was shown in the form TP/(TP+FN), where TP represents the number of true-positive results and FN represent the number of FN results. The specificity was shown in the form TN/(TN+FP), where TN represent the number of TN results and FP represent the number of FN results. 14 We also computed other significant measures on the evaluation of diagnostic experiments such as positive likelihood ratio (PLR) and negative likelihood ratio (NLR) and diagnostic OR (DOR). The summary receiver operating characteristic curve ROC (SROC) curve and the area under the curve (AUC) of the SROC curve were also computed.

Study characteristics

After a systematic search, we included 12 studies. 15–26 The complete selection process is in detail in PRISMA flowchart ( figure 1 ). From 544 screened studies, 85 studies were subjected to full text reading. The characteristics of all the 12 included studies are shown in table 1 . These 12 studies are all prospective studies published between 2014 and 2022. Most patients had US, mammography and related examinations before CESM examination. The dense breast we collected account for approximately two-thirds. In addition, the methodological quality assessment of all included studies was shown in online supplemental table 1 .

Supplemental material

• View inline

Study characteristics of each included study

• Download figure
• Open in new tab
• Download powerpoint

The figure shows the workflow for study screening and selection. CESM, contrast-enhanced spectral mammography.

Diagnostic accuracy of CESM

The sensitivity and specificity values were shown in Forest plots ( figure 2 ). A very high pooled test sensitivity of 0.97 (95% CI 0.92 to 0.98) was estimated. The pooled specificity was 0.76 (95% CI 0.64 to 0.85). The PLR was 4.03 (95% CI 2.65 to 6.11), NLR was 0.05 (95% CI 0.02 to 0.09) ( figure 3 ) and DOR was 89.49 (95% CI 45.78 to 174.92) ( online supplemental figure 1 ). I 2 values of sensitivity, specificity, PLR, NLR and DOR were 76.60%, 87.95%, 86.25%, 65.73% and 99.78%, respectively.

Forest plot of estimates of sensitivity and specificity for contrast-enhanced spectral mammography in the diagnosis of breast cancer.

Forest plot of estimates of positive likelihood ratio and negative likelihood ratio for contrast-enhanced spectral mammography in the diagnosis of breast cancer.

As shown in figure 4 , the SROC curve shows an AUC of 0.95 (0.93 to 0.97). CI is an interval estimation based on the average point estimation. The prediction interval is the interval estimation based on the individual value point estimation.

The plot shows the summary bivariate ROC curve for CESM diagnostic accuracy. AUC, area under the curve; CESM, contrast-enhanced spectral mammography; ROC, receiver operating characteristic curve; SENS, sensitivity; SPEC, specificity; SROC, summary receiver operating characteristic curve.

A confidence contour and a prediction contour were shown in the figure.

Fagan plots were drawn to understand the prior probability (current incidence) and the posterior probability (incidence estimated from this diagnostic experiment). In our sample, the pretest probability of malignancy was 50%, with a positive finding at CESM a post-test probability of 80% while a negative finding a post-test probability of 4% ( online supplemental figure 2 ).

Regression analysis

We analysed some covariates (number of lesions, number of patients, being dense breast or not, year of publication) possible influence on the diagnostic accuracy of CESM. The regression analysis showed that the sensitivity of the studies that only included dense breast was different from that of other studies, but both were high ( online supplemental figure 3 ). In addition, a limited number of studies were included, which reduced the reliability of the regression analysis.

Publication bias

A funnel plot drawn with Stata V.14.0 software was used to analyse the publication bias of the included studies ( online supplemental figure 4 ). The included studies were evenly distributed on both sides of the regression line, showing that the included literatures had no obvious publication bias (p=0.78).

CESM is emerging as a valuable tool for the diagnosis and staging of breast cancer. CESM combines the contrast enhancement effect caused by tumour neovascularisation with the information of anatomical changes. The lesions were highlighted by reciprocal subtraction of the images, which further increased the sensitivity of CESM for the diagnosis of breast cancer. It improves the accuracy in diagnosing breast cancer, providing more accurate tumour size and identification of multifocal disease, especially in patients with the dense type of breast. 27

Results showed that the pooled sensitivity (0.97, 95% CI 0.92 to 0.98) was higher and the pooled specificity (0.76, 95% CI 0.64 to 0.85) was slightly lower than a previous meta-analysis 9 which indicated a pooled sensitivity of 0.89 (95% CI 0.88 to 0.91) and a pooled specificity of 0.84 (95% CI 0.82 to 0.85). The reason for the high sensitivity may be that our study went through more rigorous study screening, included the latest literature, and CESM has been increasingly used in clinical practice in recent years. Another point is that all the studies we included are prospective studies, which are less susceptible to bias than retrospective studies. Another previous meta-analysis 8 has obtained that CESM has high sensitivity for the diagnosis of breast cancer, but it has low specificity. This may be due to the following reasons: three studies included by the meta-analysis were similar and written by the same first author; the meta-analysis only included eight studies and the pooled specificity were obtained by six literatures. All the reasons may result in some bias. However, during our screening, there are five studies from the same authors 15 28–31 and with similar results, we only included one in which the study type was prospective and with large sample size and longest time span.

In addition, compared with other studies, this study included the latest studies in recent years, and conducted a more rigorous article screening, with each of the two researchers screening two times.

The DOR is a common statistic in epidemiology that expresses the strength of the association between exposure and disease. 32 The diagnostic DOR for a test is the ratio of the odds of being positive in the disease to the odds of being positive in the non-disease. In our meta-analysis, the DOR was 89.49 (95% CI45.78 to 174.92), which was high. It indicated that if CESM showed a positive result, the probability of a true breast cancer being correctly diagnosed was 89.49 to 1. DOR offers considerable advantages in a meta-analysis of diagnostic studies by combining results from different studies into a more precise pooled estimate. The I 2 statistic, also known as the inconsistency index, is a measure of heterogeneity or variability across studies in a meta-analysis. It quantifies the proportion of total variation in effect estimates that is due to heterogeneity rather than chance. Differences in study populations: the studies included in the meta-analysis may have varied in terms of patient characteristics, such as age, mammary gland type, disease severity or comorbidities. These differences can contribute to heterogeneity in the estimated DOR. Clinical and contextual factors: heterogeneity in DOR can also arise from differences in the clinical context, such as variations in disease prevalence, healthcare settings or geographic locations.

The SROC curve method takes into account the possible heterogeneity of thresholds. 33 The SROC indicates the relationship between the TP rate and FP rate at different diagnostic thresholds. 34 In general, the AUC of a diagnostic method between 0.5 and 0.7 means low accuracy, 0.7 and 0.9 means good accuracy, above 0.9 high accuracy. The SROC curve shows an AUC of 0.95, indicating high accuracy.

The study of Hobbs et al 35 reminds of that patients’ preferences for CESM will provide further evidence supporting the adoption of CESM as an alternative to ce-MRI in selected clinical indications, if diagnostic non-inferiority of CESM is confirmed. Ferranti et al 25 suggested that CESM may provide compensation for MRI through a slight FN tendency. Furthermore, Clauser et al 36 thought the specificity of CESM is higher than that of MRI. CEM determines breast cancer based on tumour angiogenesis assessment. 24 Growth factors secreted by cancer cells promote the formation of new blood vessels during division and proliferate to tumour cells. It is because of the increased vascular endothelial cell gap and permeability that the contrast in the tumour area is enhanced. CESM may combine the high sensitivity of MRI with the low cost and availability of FFDM. 37

However, there are some limitations in the study. First, primary source participants were all patients with lesions diagnosed by breast US or mammography. This may induce a selection bias. Second, the majority of the main participants were with dense breast. This point, while highlighting the superiority of CESM over dense breast examination, may still be subject to some bias. Third, due to the excessive number of retrieved literatures, we only included prospective studies and studies writing in English. In this way, some reliable studies and results may be missed.

The CESM has high sensitivity and good specificity when it comes to evaluating breast cancer, particularly in women with dense breasts. Thus, provide more information for clinical diagnosis and treatment.

Ethics statements

Patient consent for publication.

Not applicable.

Ethics approval

• Siegel RL , et al
• Siegel RL ,
• Miller KD ,
• Fuchs HE , et al
• Poortmans P ,
• Morrow M , et al
• Lobbes MBI ,
• Jochelson MS ,
• Tagliafico AS ,
• Bignotti B ,
• Rossi F , et al
• Huang J-M ,
• Zhang K , et al
• Pesapane F ,
• Agazzi GM , et al
• McInnes MDF ,
• Thombs BD , et al
• Whiting PF ,
• Rutjes AWS ,
• Westwood ME , et al
• Psoter KJ ,
• Roudsari BS ,
• Dighe MK , et al
• Luczyńska E ,
• Heinze-Paluchowska S ,
• Dyczek S , et al
• Mokhtar O ,
• Wang L , et al
• Akashi-Tanaka S ,
• Suzuki S , et al
• Sun B , et al
• El Ghany EA
• Pan Y , et al
• Petrillo A ,
• Vallone P , et al
• Mohamed SAS ,
• Moftah SG ,
• Chalabi NAEM , et al
• Sannapareddy K ,
• Potlapalli A , et al
• Ferranti FR ,
• Vasselli F ,
• Barba M , et al
• Abu Samra MF ,
• Ibraheem MA , et al
• Adamczyk A , et al
• Łuczyńska E ,
• Hendrick E , et al
• Luczynska E ,
• Niemiec J ,
• Ambicka A , et al
• Lijmer JG ,
• Prins MH , et al
• Takwoingi Y ,
• Riley RD , et al
• Taylor DB ,
• Buzynski S , et al
• Clauser P ,
• Baltzer PAT ,
• Kapetas P , et al
• Bicchierai G ,
• Tonelli P ,
• Piacenti A , et al

Contributors JL and RX designed the study. SZou and YH gathered data. JL and SZhen performed the analysis. HY and DH revised it critically for important intellectual content. DH acted as guarantor. All authors contributed to the article and approved the submitted version.

Funding This work has received funding by the Henan Medical Science and Technology Research Program (LHGJ20210498,LHGJ20230528).

Competing interests None declared.

Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

Provenance and peer review Not commissioned; externally peer reviewed.

Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Read the full text or download the PDF:

We Trust in Human Precision

20,000+ Professional Language Experts Ready to Help. Expertise in a variety of Niches.

API Solutions

• API Pricing
• Cost estimate
• Customer loyalty program
• Educational Discount
• Non-Profit Discount
• Green Initiative Discount1

Value-Driven Pricing

Unmatched expertise at affordable rates tailored for your needs. Our services empower you to boost your productivity.

• Special Discounts
• Enterprise transcription solutions
• Enterprise translation solutions
• Transcription/Caption API
• AI Transcription Proofreading API

Trusted by Global Leaders

GoTranscript is the chosen service for top media organizations, universities, and Fortune 50 companies.

GoTranscript

One of the Largest Online Transcription and Translation Agencies in the World. Founded in 2005.

Speaker 1: The first step of doing a systematic literature review is coming up with a review question, like what do you actually want to know about the world and how can you phrase that as a simple question. You can write down all of the questions you want and then choose from the best one or a combination but I like to go to ChatGPT and use them as like a sounding board and a research assistant so that they can help me really sort of refine what I actually want to do a systematic literature review on. So here we are, we head over and we say, help me define a systematic literature review research question about beards and their smell. Maybe that's what I was interested in. My beard smells lovely. It smells like Australian sandalwood at the moment. Beautiful. It says a systematic literature review research question should be specific blah blah blah. And then it comes up with one. How do microbial communities in beards influence blah blah blah. And it gives me kind of a first start. The one thing I found about any AI that you're asking, it makes a lot of assumptions about what you want to know. So I highly recommend that you go in and you sort of like re-prompt it and you say, I like this bit, but I don't like this bit, or this bit's good, but you're a little bit off on this area. That is how you kind of use this as a research assistant as like a sounding board for all of your ideas. Then once you've got a research question and you need to spend probably most of the time of the first bit of searching on this because it's so very important. Come up with a definitive but broad, and I know that is so contradictory, but you need to come up with something that is focused enough that it will give you sort of like a good outcome but not too broad that all of a sudden, you know, like you're dealing with thousands and thousands of papers. So that is the challenge, and use ChatGPT to get that balance. Now, you can also use frameworks. There's different frameworks that you can use which will help you with this first sort of like step. And I just asked ChatGPT. I'm familiar with some of these, but some of these were new to me as well. I said, what frameworks for a systematic literature review can be used for this question? And it says Prisma, it used Cochrane Handbook for systematic reviews, it's got the Joanna Briggs Institute Methodology, Spyder and Pico. One of the most famous ones arguably is Pico where you say, okay, I've got this P, population, I've got this I, intervention that I'm looking at, I've got this C, comparison of all of the things that I found and O, outcome. Then what happened when they did these things? And quite often the C stands for comparison because it's a quantitative measurement of comparing it to say like a placebo if you're doing a lot of health stuff or another sort of intervention. So that's how we use frameworks to start thinking about our research question. What population are we gonna look at? What intervention are we looking at? What comparison, if any, are we gonna look at? And we're gonna look for the outcomes within those systems and structures that we set in place. So that's step one. Step two, actually, is what defines a literature review from a systematic literature review? Let's get into that. This is so very important for a systematic literature review because we need to know what methods we are going to use to filter all of the different stuff that we're gonna come across. We wanna know stuff like what procedure are we gonna go through to find the literature. We wanna know what keywords we're gonna use, what semantic search terms we're gonna use in certain databases to find the literature. Now, I like to head over to something like Search Smart. This will give you sort of like the best databases to search for your systematic literature review. And so all you need to do is look for scholarly records or clinical trials if you want, put in the subjects or the keywords and then sort of like define whether or not you want systemic keyword searching, backwards citation, forwards, all of that sort of stuff and also non-paywall databases and you click Start Comparison and it will go off and give you all of the different databases that you can look at. Then, keywords. Keywords are so very important because we often find research based on how they're described like in the abstract or the title. So be very specific with your keywords. By the way, I have another video, go check it out here, where I talk about how to find all of the literature that you'll ever need using different approaches, AI, Boolean searches, old school keyword searches, and that video will allow you to find everything you need in your systematic review. But databases are very important. Where are you gonna search? what keywords are you gonna search for, what semantic search questions, and that's new for this sort of like era of AI because it allows us to actually just put our research question into a database and have it sort of understand that question and give us results back. So now we're on to the exciting part which is finding the research papers. The one thing I like to do first and foremost, and that's only possible now because of AI's semantic search. I love it so much. Let's head over to the three tools that I think you would wanna use. The first one is Elicit. Ask a research question. Beards and, ooh, not bears, and smells. Let's see, that's not really a research question, but let's see what it comes up with. But it's that sort of stuff that you need to sort of like thinking about. Like, is that a keyword combination that you want to put in all of the databases or not? Whatever you decide using your meat brain. So, here we go. Here's all of the different papers that I could talk about. Brilliant. The next one is consensus. Beards and smell. Then we can go off and find all of the papers here using that sort of semantic keyword search as well. And we've also got size space. I can go here, beards and smell. And this is where I like to find all of my stuff using keywords and semantic search. So making sense, oh, this hasn't really done too well with beards, beards and issues, blah, blah, blah. So overall, you can see that we've got a little bit of discrepancy between what these pick up. So it's very important, I think, that you try a few to see what works best for you. And then finally, we gotta head over to something like Google Scholar, and we wanna say, okay, what keywords are we gonna put in? This isn't semantic search, this is just putting in beards and smell. And we can use Boolean operators to make sure that we're actually gonna get the papers that are relevant for us. So we can go beards, and then and, because we want and, smell. There we are. So then we're gonna come up with all of the smell and beard articles that it's going to come up with. The smell report, shame and glory. Only the beards, even after beards became merely rather than daring, the rather radical, oh my God, I don't like this one. The British Journal of Sociology, come on now, you can do better than that. But that is where you can go and actually find all of this information. And so semantic, keywords, databases, and Boolean operators to have a look at what you're excluding and including in your search is very, very important. So that is the step three. Yeah, step three, that is searching for the paper. And now we need to filter and screen and read. Once we've ended up with a load of papers from our searching based on the criteria and the methods we set out in step two, we've now got like an exclusion and inclusion protocol where we need to say, okay, we've got all of these studies, Which ones are we going to include and which ones are we going to exclude? And it's a really sort of like simple process of just filtering. This is why you need a load of papers at the top. Put loads of papers at the top and then they have to filter down to the useful papers down the bottom. And it may only be a small fraction of all of the papers you found, but this is what a systematic review is all about. It's about making sure that we include the papers that are relevant for your research question and not just like general themes, which is like a normal literature review where we just sort of say, oh yeah, there's this theme and this theme and this theme. No, this one's much more focused, so we need to filter it. I like to use the Prisma flowchart to work out which ones I'm getting rid of and keep track of the ones I've got rid of and how much I've filtered it down. So a Prisma flowchart looks like this. We've got identification in the top here and then we've got records identified through database searching. In this case, they had 96. and then we've got other additional identified through other sources, and this was none in this bit. Then they removed duplicates, so there was two that were the same, so they removed one of them, and then they said, okay, we've got this many in screen, 95, and eligibility, full text articles assessed for eligibility, there was only five, and all of these were actually excluded because it didn't meet their criteria that they'd set out in part of their exclusion or inclusion criteria. So you can see we've got like examines treatment, not prevention. So this was like obviously like a health study where they were looking at treatment and not the prevention or something. So that was most of them, that was 52. Then one was pediatric, one was irrelevant. Oh no, loads were irrelevant, 37 were irrelevant. So you can see we've gone from 96 all the way down to five at this point. And then full text articles not included. Well, there was none there, which is great. but here we've got four which studies included in quantitative synthesis or a meta-analysis was only four, they got rid of 92 of them because they didn't meet the specific search and exclusion and inclusion criteria that they set. That is so important and that is very, very typical of a systematic review. So now it's about taking those special studies that you found and getting all of the important stuff out of them. you should read them, especially if there's only four. You should read them from end to beginning. No, don't read them like that. Read them however you want, normally with abstract, then to conclusions, then to introduction, then to method, anyway, you get the idea. Do you know what, actually, I've got another video on how to read like a PhD. Go check out that one there. It's much better than what I just said. But now you need to read them and you need to start thinking about how these studies are influencing your research question sort of response. Are they for it? Are they against it? Do they give you a new insight? Is there something sneaky in there when you look at them all together that is surprising? It's those sort of things that really should be sort of milling around in your head. We're not looking for any sort of definitive stuff just yet, but we just need to read, analyze, refine, understand, all of those stuff. Those words are very important, put them there. But now, we've got a couple of new ways that we can actually talk to all of our documents. So one place I really like is docanalyzer.ai and what you can do is upload your documents and tag them as, in this case I've got literature review, you can see I've got one, two, three, four, five, six here. So then we can go to labels and we can go chat with these six documents. And the one thing I love about docanalyzer is that it doesn't like try to make stuff up. If it doesn't understand what you're asking or it can't identify it in the documents that you've given it, it will just say, hey, I don't really know, can you give me a bit more information? It doesn't sort of like BS its way into chat, which I really like. So, for example here, it says to identify the important parts of the document, I would need more specific keywords or topics of interest. That's what I want from an AI, something that isn't just gonna make stuff up. Another thing you can do is head back over to size space, And in SciSpace, you can actually get results from my library. So if you put those very specific studies that you've filtered and found into your library, you can then ask it questions across that library, which I think is really, really fantastic. So not only do you read it all, if you can, if it's a sensible amount of papers, but then you can start chatting to all of the documents together in something like DocAnalyzer and SciSpace, and then you can get sort of further connections, further deeper inquiry into things that maybe you have missed. Or maybe there's just a question, you've read them all, and there's a question sort of in your mind. You're like, actually, does this apply to all of the papers or not? Put it into something like this and it will search across all of your documents. I absolutely love, I'm doing this today, Chef's Kiss, it's my new favorite thing. Chef's Kiss, yum, yum, yum, yum, yum. But doing that means that you're not gonna miss out on anything because you're going to use old school tactics by just reading, read, read, read, read, read, and new school tactics by using AI, AI, AI, AI. Together, they are the perfect combination, yes. And then it's all about writing it up, making sure that you actually talk about what your research question is, the methods you've used, the filtration criteria, and the exclusion and inclusion criteria, the keywords you search for, then what you've found, how they all sort of like relate together, and the outcome. What is the outcome of this literature review? Does it support your research questions? Does it give you a new insight? That is how you write this. That is the structure. It is so very sort of systematic. A systematic literature review has to be systematic, otherwise you'll just end up being completely lost in all of the papers. Oh, so many papers, so many papers. Filter them out, find the good ones, write it out. Brilliant. All right, if you like this video, Go check out this one where I talk about how to write an exceptional literature review with AI. It's going to be a great sort of addition to what you've learned in here. Go check it out.

• Download PDF
• Share X Facebook Email LinkedIn
• Permissions

Adverse Events in Studies of Classic Psychedelics : A Systematic Review and Meta-Analysis

• 1 Center for Psychedelic and Consciousness Research, Department of Psychiatry and Behavioral Sciences, Johns Hopkins University School of Medicine, Baltimore, Maryland
• 2 Department of Clinical Psychology, Hofstra University, Hempstead, New York

Question   What is the nature, frequency, and severity of adverse events (AEs) reported in studies of classic psychedelic administration in monitored clinical or research settings?

Findings   Reports of serious AEs (SAEs) and nonserious AEs (NSAEs) requiring medical or psychiatric attention in classic psychedelic research were rare. In this systematic review and meta-analysis of 3504 participants from 114 studies, SAEs were reported for no healthy participants and approximately 4% of participants with preexisting neuropsychiatric disorders; however, for most studies, there was concern for underdetection or incomplete AE reporting.

Meaning   Classic psychedelics were generally well tolerated in clinical or research environments according to existing literature, although SAEs and medically significant NSAEs did occur, which demonstrates the importance of improved pharmacovigilance to understand and quantify the risk and benefit profiles of classic psychedelic substances.

Importance   A clear and comprehensive understanding of risks associated with psychedelic-assisted therapy is necessary as investigators extend its application to new populations and indications.

Objective   To assess adverse events (AEs) associated with classic psychedelics, particularly serious AEs (SAEs) and nonserious AEs (NSAEs) requiring medical or psychiatric evaluation.

Data Sources   The search for potentially eligible studies was conducted in the Scopus, MEDLINE, PsycINFO, and Web of Science databases from inception through February 8, 2024.

Study Selection   Two independent reviewers screened articles of classic psychedelics (lysergic acid diethylamide [LSD], psilocybin, dimethyltryptamine [DMT], and 5-methoxy-N,N-dimethyltryptamine [5-MeO-DMT]) involving administration in clinical or research contexts.

Data Extraction and Synthesis   AE data were extracted and synthesized by 2 reviewers and were used for random-effects meta-analysis of AE frequency and heterogeneity. Risk of bias assessment focused on AE ascertainment (eg, systematic assessment and quality of follow-up).

Main Outcomes and Measures   A hybrid approach was used for capture of all reported AEs following high-dose classic psychedelic exposure and confirmatory capture of AEs of special interest, including suicidality, psychotic disorder, manic symptoms, cardiovascular events, and hallucinogen persisting perception disorder. AEs were stratified by timescale and study population type. Forest plots of common AEs were generated, and the proportions of participants affected by SAEs or NSAEs requiring medical intervention were summarized descriptively.

Results   A total of 214 unique studies were included, of which 114 (53.3%) reported analyzable AE data for 3504 total participants. SAEs were reported for no healthy participants and for approximately 4% of participants with preexisting neuropsychiatric disorders; among these SAEs were worsening depression, suicidal behavior, psychosis, and convulsive episodes. NSAEs requiring medical intervention (eg, paranoia, headache) were similarly rare. In contemporary research settings, there were no reports of deaths by suicide, persistent psychotic disorders, or hallucinogen persisting perception disorders following administration of high-dose classic psychedelics. However, there was significant heterogeneity in the quality of AE monitoring and reporting. Of 68 analyzed studies published since 2005, only 16 (23.5%) described systematic approaches to AE assessment, and 20 studies (29.4%) reported all AEs, as opposed to only adverse drug reactions. Meta-analyses of prevalence for common AEs (eg, headache, anxiety, nausea, fatigue, and dizziness) yielded comparable results for psilocybin and LSD.

Conclusions and Relevance   In this systematic review and meta-analysis, classic psychedelics were generally well tolerated in clinical or research settings according to the existing literature, although SAEs did occur. These results provide estimates of common AE frequencies and indicate that certain catastrophic events reported in recreational or nonclinical contexts have yet to be reported in contemporary trial participants. Careful, ongoing, and improved pharmacovigilance is required to understand the risk and benefit profiles of these substances and to communicate such risks to prospective study participants and the public.

Read More About

Hinkle JT , Graziosi M , Nayak SM , Yaden DB. Adverse Events in Studies of Classic Psychedelics : A Systematic Review and Meta-Analysis . JAMA Psychiatry. Published online September 04, 2024. doi:10.1001/jamapsychiatry.2024.2546

Manage citations:

© 2024

Artificial Intelligence Resource Center

Psychiatry in JAMA : Read the Latest

Browse and subscribe to JAMA Network podcasts!

Others Also Liked

Select your interests.

Customize your JAMA Network experience by selecting one or more topics from the list below.

• Academic Medicine
• Acid Base, Electrolytes, Fluids
• Allergy and Clinical Immunology
• American Indian or Alaska Natives
• Anesthesiology
• Anticoagulation
• Art and Images in Psychiatry
• Artificial Intelligence
• Assisted Reproduction
• Bleeding and Transfusion
• Caring for the Critically Ill Patient
• Challenges in Clinical Electrocardiography
• Climate and Health
• Climate Change
• Clinical Challenge
• Clinical Decision Support
• Clinical Implications of Basic Neuroscience
• Clinical Pharmacy and Pharmacology
• Complementary and Alternative Medicine
• Consensus Statements
• Coronavirus (COVID-19)
• Critical Care Medicine
• Cultural Competency
• Dental Medicine
• Dermatology
• Diabetes and Endocrinology
• Diagnostic Test Interpretation
• Drug Development
• Electronic Health Records
• Emergency Medicine
• End of Life, Hospice, Palliative Care
• Environmental Health
• Equity, Diversity, and Inclusion
• Facial Plastic Surgery
• Gastroenterology and Hepatology
• Genetics and Genomics
• Genomics and Precision Health
• Global Health
• Guide to Statistics and Methods
• Hair Disorders
• Health Care Delivery Models
• Health Care Economics, Insurance, Payment
• Health Care Quality
• Health Care Reform
• Health Care Safety
• Health Care Workforce
• Health Disparities
• Health Inequities
• Health Policy
• Health Systems Science
• History of Medicine
• Hypertension
• Images in Neurology
• Implementation Science
• Infectious Diseases
• Innovations in Health Care Delivery
• JAMA Infographic
• Law and Medicine
• Leading Change
• Less is More
• LGBTQIA Medicine
• Lifestyle Behaviors
• Medical Coding
• Medical Devices and Equipment
• Medical Education
• Medical Education and Training
• Medical Journals and Publishing
• Mobile Health and Telemedicine
• Narrative Medicine
• Neuroscience and Psychiatry
• Notable Notes
• Nutrition, Obesity, Exercise
• Obstetrics and Gynecology
• Occupational Health
• Ophthalmology
• Orthopedics
• Otolaryngology
• Pain Medicine
• Palliative Care
• Pathology and Laboratory Medicine
• Patient Care
• Patient Information
• Performance Improvement
• Performance Measures
• Perioperative Care and Consultation
• Pharmacoeconomics
• Pharmacoepidemiology
• Pharmacogenetics
• Pharmacy and Clinical Pharmacology
• Physical Medicine and Rehabilitation
• Physical Therapy
• Physician Leadership
• Population Health
• Primary Care
• Professional Well-being
• Professionalism
• Psychiatry and Behavioral Health
• Public Health
• Pulmonary Medicine
• Regulatory Agencies
• Reproductive Health
• Research, Methods, Statistics
• Resuscitation
• Rheumatology
• Risk Management
• Scientific Discovery and the Future of Medicine
• Shared Decision Making and Communication
• Sleep Medicine
• Sports Medicine
• Stem Cell Transplantation
• Substance Use and Addiction Medicine
• Surgical Innovation
• Surgical Pearls
• Teachable Moment
• Technology and Finance
• The Art of JAMA
• The Arts and Medicine
• The Rational Clinical Examination
• Tobacco and e-Cigarettes
• Translational Medicine
• Trauma and Injury
• Treatment Adherence
• Ultrasonography
• Users' Guide to the Medical Literature
• Vaccination
• Venous Thromboembolism
• Veterans Health
• Women's Health
• Workflow and Process
• Wound Care, Infection, Healing
• Register for email alerts with links to free full-text articles
• Access PDFs of free articles
• Manage your interests
• Save searches and receive search alerts

This week: the arXiv Accessibility Forum

Help | Advanced Search

Computer Science > Machine Learning

Title: multimodal methods for analyzing learning and training environments: a systematic literature review.

Abstract: Recent technological advancements have enhanced our ability to collect and analyze rich multimodal data (e.g., speech, video, and eye gaze) to better inform learning and training experiences. While previous reviews have focused on parts of the multimodal pipeline (e.g., conceptual models and data fusion), a comprehensive literature review on the methods informing multimodal learning and training environments has not been conducted. This literature review provides an in-depth analysis of research methods in these environments, proposing a taxonomy and framework that encapsulates recent methodological advances in this field and characterizes the multimodal domain in terms of five modality groups: Natural Language, Video, Sensors, Human-Centered, and Environment Logs. We introduce a novel data fusion category -- mid fusion -- and a graph-based technique for refining literature reviews, termed citation graph pruning. Our analysis reveals that leveraging multiple modalities offers a more holistic understanding of the behaviors and outcomes of learners and trainees. Even when multimodality does not enhance predictive accuracy, it often uncovers patterns that contextualize and elucidate unimodal data, revealing subtleties that a single modality may miss. However, there remains a need for further research to bridge the divide between multimodal learning and training studies and foundational AI research.

Submission history

Access paper:.

• HTML (experimental)
• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Author Biographies
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Methodological quality of user-centered usability evaluation of digital applications to promote citizens’ engagement and participation in public governance: a systematic literature review.

1. Introduction

2. materials and methods, 2.1. research question, 2.2. search strategies, 2.3. inclusion and exclusion criteria, 2.4. screening procedures, 2.5. synthesis and reporting, 2.6. methodological quality, 3.1. study selection, 3.2. demographics of the included studies, 3.3. purpose of the reported applications, 3.3.1. participatory reporting of urban issues, 3.3.2. environmental sustainability, 3.3.3. civic participation, 3.3.4. urban planning, 3.3.5. promotion of democratic values, 3.3.6. electronic voting, 3.3.7. chatbots, 3.4. security and privacy mechanisms and strategies to incentivize citizen participation, 3.5. usability evaluation procedures, methods, and instruments, 3.6. methodological quality assessment, 3.7. implications of the reviewed usability evaluation studies on future applications, 4. discussion, 5. limitations, 6. conclusions, author contributions, institutional review board statement, informed consent statement, data availability statement, conflicts of interest.

• Moon, M.J. The evolution of e-government among municipalities: Rhetoric or reality? Public Adm. Rev. 2002 , 62 , 424–433. [ Google Scholar ] [ CrossRef ]
• Webster, C.; Leleux, C. Smart governance: Opportunities for technologically-mediated citizen co-production. Inf. Polity 2018 , 23 , 95–110. [ Google Scholar ] [ CrossRef ]
• Barcevičius, E.; Cibaitė, G.; Codagnone, C.; Gineikytė, V.; Klimavičiūtė, L.; Liva, G.; Matulevič, L.; Misuraca, G.; Vanini, I. Exploring Digital Government Transformation in the EU ; Publications Office of the European Union: Luxembourg, 2019. [ Google Scholar ] [ CrossRef ]
• Cornips, L.; Voorberg, W.; van Meerkerk, I.; Kramer, R. Co-production as a strategy for enhanced equal representation in public service delivery: The case of Rotterdam. Cities 2023 , 141 , 104480. [ Google Scholar ] [ CrossRef ]
• Gerrard, J.; Savage, G.C. Policy translations of citizen participation and new public governance: The case of school governing bodies. Crit. Policy Stud. 2023 , 17 , 484–501. [ Google Scholar ] [ CrossRef ]
• Cao, H.; Kang, C.I. A citizen participation model for co-creation of public value in a smart city. J. Urban Aff. 2024 , 46 , 905–924. [ Google Scholar ] [ CrossRef ]
• Gil-García, J.R.; Pardo, T.A. E-government success factors: Mapping practical tools to theoretical foundations. Gov. Inf. Q. 2005 , 22 , 187–216. [ Google Scholar ] [ CrossRef ]
• Ravšelj, D.; Umek, L.; Todorovski, L.; Aristovnik, A. A review of digital era governance research in the first two decades: A bibliometric study. Future Internet 2022 , 14 , 126. [ Google Scholar ] [ CrossRef ]
• Resolution Adopted by the General Assembly on 26 November 2018 (A/73/L.20 and A/73/L.20/Add.1) ; United Nations: New York, NY, USA, 2018.
• Janowski, T. Digital government evolution: From transformation to contextualization. Gov. Inf. Q. 2015 , 32 , 221–236. [ Google Scholar ] [ CrossRef ]
• Charalabidis, Y.; Sarantis, D.; Askounis, D. Knowledge-driven project management in government transformation. In Handbook of Research on ICT-Enabled Transformational Government: A Global Perspective ; IGI Global: Dauphin, PA, USA, 2009. [ Google Scholar ] [ CrossRef ]
• Layne, K.; Lee, J. Developing fully functional E-government: A four stage model. Gov. Inf. Q. 2001 , 18 , 122–136. [ Google Scholar ] [ CrossRef ]
• Reddick, C.G. A two-stage model of e-government growth: Theories and empirical evidence for US cities. Gov. Inf. Q. 2004 , 21 , 51–64. [ Google Scholar ] [ CrossRef ]
• Wendler, R. The maturity of maturity model research: A systematic mapping study. Inf. Softw. Technol. 2012 , 54 , 1317–1339. [ Google Scholar ] [ CrossRef ]
• Fath-Allah, A.; Cheikhi, L.; Al-Qutaish, R.E.; Idri, A. E-government maturity models: A comparative study. Int. J. Softw. Eng. Appl. 2014 , 5 , 71. [ Google Scholar ] [ CrossRef ]
• Joshi, P.R.; Islam, S. E-government maturity model for sustainable E-government services from the perspective of developing countries. Sustainability 2018 , 10 , 1882. [ Google Scholar ] [ CrossRef ]
• Hochstetter, J.; Diaz, J.; Dieguez, M.; Espinosa, R.; Arango-López, J.; Cares, C. Assessing transparency in eGovernment electronic processes. IEEE Access 2021 , 10 , 3074–3087. [ Google Scholar ] [ CrossRef ]
• Aldrees, A.; Gračanin, D. UX in E-government Services for Citizens: A Systematic Literature Review. J. User Exp. 2023 , 18 , 133–169. [ Google Scholar ]
• Bindu, N.; Sankar, C.P.; Kumar, K.S. From conventional governance to e-democracy: Tracing the evolution of e-governance research trends using network analysis tools. Gov. Inf. Q. 2019 , 36 , 385–399. [ Google Scholar ] [ CrossRef ]
• Wittels, A.S. Participatory governance and responsiveness: Do motivational interventions increase engagement with citizen input? Political Res. Q. 2023 , 76 , 741–756. [ Google Scholar ] [ CrossRef ]
• Vrabie, C. E-Government 3.0: An AI model to use for enhanced local democracies. Sustainability 2023 , 15 , 9572. [ Google Scholar ] [ CrossRef ]
• Al-Nidawi, W.J.A.; Al-Wassiti, S.K.J.; Maan, M.A.; Othman, M. A review in E-government service quality measurement. Indones. J. Electr. Eng. Comput. Sci. 2018 , 10 , 1257–1265. [ Google Scholar ] [ CrossRef ]
• Adam, I.; Fazekas, M. Are emerging technologies helping win the fight against corruption? A review of the state of evidence. Inf. Econ. Policy 2021 , 57 , 100950. [ Google Scholar ] [ CrossRef ]
• Alruwaie, M.; El-Haddadeh, R.; Weerakkody, V. Citizens’ continuous use of eGovernment services: The role of self-efficacy, outcome expectations and satisfaction. Gov. Inf. Q. 2020 , 37 , 101485. [ Google Scholar ] [ CrossRef ]
• Buntaine, M.T.; Nielson, D.L.; Skaggs, J.T. Escaping the disengagement dilemma: Two field experiments on motivating citizens to report on public services. Br. J. Political Sci. 2021 , 51 , 685–705. [ Google Scholar ] [ CrossRef ]
• Bricout, J.; Baker, P.M.; Moon, N.W.; Sharma, B. Exploring the smart future of participation: Community, inclusivity, and people with disabilities. Int. J. E-Plan. Res. (IJEPR) 2021 , 10 , 94–108. [ Google Scholar ] [ CrossRef ]
• Kim, N.Y.; Kim, H.J.; Kim, S.H. Are satisfied citizens willing to participate more? An analysis of citizens’ life satisfaction in the public service domain and public participation. Int. Rev. Public Adm. 2022 , 27 , 211–227. [ Google Scholar ] [ CrossRef ]
• ISO 9241-210 ; Human-Centred Design for Interactive Systems. International Organization for Standardization: Geneve, Switzerland, 2019.
• Nielsen, J. Usability Engineering ; Morgan Kaufmann: San Francisco, CA, USA, 1994. [ Google Scholar ]
• ISO 9241-11 ; Guidance on Usability. International Organization for Standardization: Geneve, Switzerland, 1999.
• Venkatesh, V.; Ramesh, V.; Massey, A.P. Understanding usability in mobile commerce. Commun. ACM 2003 , 46 , 53–56. [ Google Scholar ] [ CrossRef ]
• Srivastava, P.; Mostafavi, A. Challenges and opportunities of crowdsourcing and participatory planning in developing infrastructure systems of smart cities. Infrastructures 2018 , 3 , 51. [ Google Scholar ] [ CrossRef ]
• Hyvärinen, T.; Kaikkonen, A.; Hiltunen, M. Placing links in mobile banking application. In Proceedings of the 7th International Conference on Human Computer Interaction with Mobile Devices & Services, Salzburg, Austria, 19–22 September 2005; pp. 63–68. [ Google Scholar ] [ CrossRef ]
• Chanana, L.; Agrawal, R.; Punia, D.K. Service quality parameters for mobile government services in India. Glob. Bus. Rev. 2016 , 17 , 136–146. [ Google Scholar ] [ CrossRef ]
• Al-Hubaishi, H.S.; Ahmad, S.Z.; Hussain, M. Exploring mobile government from the service quality perspective. J. Enterp. Inf. Manag. 2017 , 30 , 4–16. [ Google Scholar ] [ CrossRef ]
• Saadi, M.R.; Ahmad, S.Z.; Hussain, M. Prioritization of citizens’ preferences for using mobile government services: The analytic hierarchy process (AHP) approach. Transform. Gov. People Process Policy 2017 , 11 , 476–503. [ Google Scholar ] [ CrossRef ]
• Singh, H.; Grover, P.; Kar, A.K.; Ilavarasan, P.V. Review of performance assessment frameworks of e-government projects. Transform. Gov. People Process Policy 2020 , 14 , 31–64. [ Google Scholar ] [ CrossRef ]
• Desmal, A.J.; Hamid, S.; Othman, M.K.; Zolait, A. Exploration of the usability quality attributes of mobile government services: A literature review. PeerJ Comput. Sci. 2022 , 8 , e1026. [ Google Scholar ] [ CrossRef ]
• Sheoran, S.; Vij, S. A Review of E-Government Assessment Frameworks: E-Readiness, Adoption, Citizen Engagement and Quality: E-Readiness, Adoption, Citizen Engagement and Quality. JeDEM-Ejournal Edemocracy Open Gov. 2022 , 14 , 197–213. [ Google Scholar ] [ CrossRef ]
• Venkatesh, V.; Hoehle, H.; Aljafari, R. A usability study of the obamacare website: Evaluation and recommendations. Gov. Inf. Q. 2017 , 34 , 199–210. [ Google Scholar ] [ CrossRef ]
• Ojo, A.; Mellouli, S. Deploying governance networks for societal challenges. Gov. Inf. Q. 2018 , 35 , S106–S112. [ Google Scholar ] [ CrossRef ]
• Ababneh, R.; Alrefaie, L. Evaluating the quality of public administration institutes’ websites in the Arab world. In Global Knowledge, Memory and Communication ; Emerald: Bingley, UK, 2022. [ Google Scholar ] [ CrossRef ]
• Agrawal, G.; Kumar, D.; Singh, M. Assessing the usability, accessibility, and mobile readiness of e-government websites: A case study in India. Univers. Access Inf. Soc. 2022 , 21 , 737–748. [ Google Scholar ] [ CrossRef ]
• Rivas-Delgado, O.; Libaque-Saenz, C.F. The impact of usability on e-government usage in the Peruvian context. Issues Inf. Syst. 2022 , 23 , 1–14. [ Google Scholar ] [ CrossRef ]
• Liu, Q.; Kim, K. Research on the Usability Test of Interface Design in e-Government-Focused on Qingdao e-Government Website. Arch. Des. Res. 2023 , 36 , 59–72. [ Google Scholar ] [ CrossRef ]
• Qonita, F.; Budiman, M.F.; Sari, V.M.; Limantara, N. Analysis of User Experience on The Government Application of Indonesian Higher Education Institutional Information Systems Using Usability Method. In Proceedings of the 4th International Conference on Innovative Trends in Information Technology (ICITIIT), Kerala, India, 11–12 February 2023; pp. 1–6. [ Google Scholar ] [ CrossRef ]
• Sheoran, S.; Mohanasundaram, S.; Kasilingam, R.; Vij, S. Usability and Accessibility of Open Government Data Portals of Countries Worldwide: An Application of TOPSIS and Entropy Weight Method. Int. J. Electron. Gov. Res. (IJEGR) 2023 , 19 , 1–25. [ Google Scholar ] [ CrossRef ]
• Desmal, A.J.; Hamid, S.; Othman, M.K.; Zolait, A. A user satisfaction model for mobile government services: A literature review. PeerJ Comput. Sci. 2022 , 8 , e1074. [ Google Scholar ] [ CrossRef ]
• Menezes, V.G.D.; Pedrosa, G.V.; Silva, M.P.D.; Figueiredo, R.M.D. Evaluation of public services considering the expectations of users-A systematic literature review. Information 2022 , 13 , 162. [ Google Scholar ] [ CrossRef ]
• Lyzara, R.; Purwandari, B.; Zulfikar, M.F.; Santoso, H.B.; Solichah, I. E-government usability evaluation: Insights from a systematic literature review. In Proceedings of the 2nd International Conference on Software Engineering and Information Management, Bali, Indonesia, 10–12 January 2019; pp. 249–253. [ Google Scholar ] [ CrossRef ]
• Monzón, F.H.; Tupia, M.; Bruzza, M. Security versus usability in e-government: Insights from the literature. In Proceedings of the MICRADS 2020, Quito, Ecuador, 29–31 July 2020; pp. 29–42. [ Google Scholar ] [ CrossRef ]
• Alshamsi, A.; Williams, N.; Andras, P. The Trade-off between Usability and Security in the Context of eGovernment: A Mapping Study. In Proceedings of the 30th International BCS Human Computer Interaction Conference, Fern Barrow, UK, 11–15 July 2016; pp. 1–13. [ Google Scholar ] [ CrossRef ]
• Yerlikaya, Z.; Onay Durdu, P. Usability of university websites: A systematic review. In Proceedings of the 11th International Conference of Universal Access in Human-Computer Interaction, Vancouver, BC, Canada, 9–14 July 2017; pp. 277–287. [ Google Scholar ] [ CrossRef ]
• Cisneros, D.; Huamán Monzón, F.; Paz, F. Accessibility evaluation of E-Government web applications: A systematic review. In Proceedings of the International Conference on Human-Computer Interaction, Virtual Event, 24–29 July 2021; pp. 210–223. [ Google Scholar ] [ CrossRef ]
• Zhang, J.; Chang, D.; Zhang, Z. Review on the Application of Eye-tracking Technology in Usability Evaluation of E-government Apps. In Proceedings of the IEEE International Conference on Industrial Engineering and Engineering Management (IEEM), Singapore, 13–16 December 2021; pp. 1646–1650. [ Google Scholar ] [ CrossRef ]
• Moher, D.; Liberati, A.; Tetzlaff, J.; Altman, D.G.; Prisma Group. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Int. J. Surg. 2010 , 8 , 336–341. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Booth, A.; Sutton, A.; Clowes, M.; Martyn-St James, M. Systematic Approaches to a Successful Literature Review ; Sage: Newcastle upon Tyne, UK, 2021. [ Google Scholar ]
• Silva, A.G.; Simões, P.; Santos, R.; Queirós, A.; Rocha, N.P.; Rodrigues, M. A scale to assess the methodological quality of studies assessing usability of electronic health products and services: Delphi study followed by validity and reliability testing. J. Med. Internet Res. 2019 , 21 , e14829. [ Google Scholar ] [ CrossRef ]
• Madyatmadja, E.D.; Prabowo, H. Participation to public e-service development: A systematic literature review. J. Telecommun. Electron. Comput. Eng. (JTEC) 2016 , 8 , 139–143. [ Google Scholar ]
• Delgado, M.; Paz, F.; Tupia, M. Sistemas de lógica difusa para la evaluación de usabilidad de sitios web de gobierno electrónico: Una revisión sistemática. Rev. Ibérica De Sist. E Tecnol. De Informação 2021 , E41 , 141–154. [ Google Scholar ]
• Mahou-Lago, X.M.; Varela-Álvarez, E.J. Innovation and opportunities for citizen participation in Spanish smart cities. In Smarter as the New Urban Agenda ; Springer: Cham, Switzerland, 2016. [ Google Scholar ] [ CrossRef ]
• Li, E.; Chen, Q.; Zhang, X.; Zhang, C. Digital government development, local governments’ attention distribution and enterprise total factor productivity: Evidence from China. Sustainability 2023 , 15 , 2472. [ Google Scholar ] [ CrossRef ]
• Castilla, R.; Pacheco, A.; Franco, J. Digital government: Mobile applications and their impact on access to public information. SoftwareX 2023 , 22 , 101382. [ Google Scholar ] [ CrossRef ]
• De Róiste, M. Bringing in the users: The role for usability evaluation in eGovernment. Gov. Inf. Q. 2013 , 30 , 441–449. [ Google Scholar ] [ CrossRef ]
• Sachs, M.; Schossböck, J. Acceptance of tools for electronic citizen participation. In Proceedings of the International Conference on Electronic Participation, San Benedetto Del Tronto, Italy, 2–4 September 2019; pp. 35–46. [ Google Scholar ] [ CrossRef ]
• Haustein, E.; Lorson, P.C.; Oulasvirta, L.O.; Sinervo, L.M. Perceived usability of local government (LG) financial statements by local councillors: Comparative study of Finland and Germany. Int. J. Public Sect. Manag. 2021 , 34 , 441–458. [ Google Scholar ] [ CrossRef ]
• Rey, W.P. Assessing MABIS Mobile App Based on People at the Center of Mobile Application Development (PACMAD) Usability Model: Empirical Investigation. In Proceedings of the 13th International Conference on Software Technology and Engineering (ICSTE), Osaka, Japan, 27–29 October 2023; pp. 37–43. [ Google Scholar ] [ CrossRef ]
• Ouaadi, I.; Haddad, M.E. Fuzzy expert system and information systems auditing: An approach for risk assessment in audit pre-planning. Int. J. Bus. Contin. Risk Manag. 2023 , 13 , 204–228. [ Google Scholar ] [ CrossRef ]
• Therias, A.; Rafiee, A. City digital twins for urban resilience. Int. J. Digit. Earth 2023 , 16 , 4164–4190. [ Google Scholar ] [ CrossRef ]
• Dwivedi, V.; Iqbal, M.; Norta, A.; Matulevičius, R. Evaluation of a Legally Binding Smart-Contract Language for Blockchain Applications. J. Univers. Comput. Sci. 2023 , 29 , 691–717. [ Google Scholar ] [ CrossRef ]
• Chang Lee, K.; Kirlidog, M.; Lee, S.; Gun Lim, G. User evaluations of tax filing web sites: A comparative study of South Korea and Turkey. Online Inf. Rev. 2008 , 32 , 842–859. [ Google Scholar ] [ CrossRef ]
• Awan, M.A. Dubai e-government: An evaluation of G2B websites. J. Internet Commer. 2008 , 6 , 115–129. [ Google Scholar ] [ CrossRef ]
• Choudrie, J.; Wisal, J.; Ghinea, G. Evaluating the usability of developing countries’ e-government sites: A user perspective. Electron. Gov. Int. J. 2009 , 6 , 265–281. [ Google Scholar ] [ CrossRef ]
• Yuan, L.; Zhongling, L. Experimental evaluation on government portal website’s usability to 11 government websites of Zhejiang province. In Proceedings of the 2nd International Conference on Information Science and Engineering, Kerbala, Iraq, 26–29 March 2010; pp. 2076–2078. [ Google Scholar ] [ CrossRef ]
• Israel, D.; Tiwari, R. Empirical study of factors influencing acceptance of e-government services in India. In Proceedings of the 5th International Conference on Theory and Practice of Electronic Governance, Tallinn, Estonia, 26–28 October 2011; pp. 141–146. [ Google Scholar ] [ CrossRef ]
• González Martínez, S.; Luna-Reyes, L.F.; Luna, D.E.; Gil-Garcia, J.R.; Sandoval-Almazán, R. Comparing usability of government web portals during governor change of terms. In Proceedings of the 12th Annual International Digital Government Research Conference: Digital Government Innovation in Challenging Times, College Park, MD, USA, 12–15 June 2011; pp. 327–328. [ Google Scholar ] [ CrossRef ]
• Bouzas-Lorenzo, R.; Mahou-Lago, X.M. An evaluation of citizen service web portals in Latin America. Acad. Rev. Latinoam. De Adm. 2015 , 28 , 99–114. [ Google Scholar ] [ CrossRef ]
• Baguma, R. Usability evaluation of the etax portal for Uganda. In Proceedings of the 11th International Conference on Theory and Practice of Electronic Governance, Galway, Ireland, 4–6 April 2018; pp. 449–458. [ Google Scholar ] [ CrossRef ]
• Madariaga, L.; Nussbaum, M.; Marañón, F.; Alarcón, C.; Naranjo, M.A. User experience of government documents: A framework for informing design decisions. Gov. Inf. Q. 2019 , 36 , 179–195. [ Google Scholar ] [ CrossRef ]
• Hussain, A.; Mkpojiogu, E.O.; Ishak, N.; Mokhtar, N.; Ani, Z.C. An Interview Report on Users’ Perception about the Usability Performance of a Mobile E-Government Application. Int. J. Interact. Mob. Technol. 2019 , 13 , 169–178. [ Google Scholar ] [ CrossRef ]
• Patsoulis, G.; Promikyridis, R.; Tambouris, E. Integration of chatbots with Knowledge Graphs in eGovernment: The case of Getting a Passport. In Proceedings of the 25th Pan-Hellenic Conference on Informatics, Volos, Greece, 26–28 November 2021; pp. 425–429. [ Google Scholar ] [ CrossRef ]
• Butt, M.A.; Li, S.; Javed, N. Towards Co-PPGIS-a collaborative public participatory GIS-based measure for transparency in housing schemes: A case of Lahore, Pakistan. Appl. Geomat. 2016 , 8 , 27–40. [ Google Scholar ] [ CrossRef ]
• Birghan, F.; Hettenhausen, R.; Meschede, C.; Siebenlist, T. Informing Citizens via Council Information Systems. In Proceedings of the 20th Annual International Conference on Digital Government Research, Dubai, United Arab Emirates, 18–20 June 2019; pp. 280–286. [ Google Scholar ] [ CrossRef ]
• Sofyan, Z. User requirements elicitation in web-based Participatory Geographic Information System interface design. In Proceedings of the 9th Annual International Conference on Sciences & Engineering, Banda Aceh, Indonesia, 18–20 September 2019; p. 012028. [ Google Scholar ] [ CrossRef ]
• Magoutas, B.; Schmidt, K.U.; Mentzas, G.; Stojanovic, L. An adaptive e-questionnaire for measuring user perceived portal quality. Int. J. Hum. Comput. Stud. 2010 , 68 , 729–745. [ Google Scholar ] [ CrossRef ]
• Byun, D.H.; Finnie, G. An AHP method for evaluating usability of electronic government portals. Electron. Gov. Int. J. 2011 , 8 , 343–362. [ Google Scholar ] [ CrossRef ]
• Faisal, M.; Al-Qouz, H.; Husain, F. A direct method for measuring user experience in E-government portals. In Proceedings of the 15th International Conference on Information Technology Based Higher Education and Training (ITHET), Istanbul, Turkey, 8–10 September 2016; pp. 1–4. [ Google Scholar ] [ CrossRef ]
• Schneider, H.; Frison, K.; Wagner, J.; Butz, A. CrowdUX: A case for using widespread and lightweight tools in the quest for UX. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems, Brisbane, Australia, 4–8 June 2016; pp. 415–426. [ Google Scholar ] [ CrossRef ]
• Burian, M.F.; Fauvety, P.; Aquino, N.; González, M.; Romero, D.; Cernuzzi, L.; Paniagua, J.; Chenú-Abente, R. Design of SmartMoving, an application for pedestrians with reduced mobility. In Proceedings of the XLVI Latin American Computing Conference (CLEI), Loja, Ecuador, 19–23 October 2020; pp. 367–376. [ Google Scholar ] [ CrossRef ]
• Callegari, L.S.; Nelson, K.M.; Arterburn, D.E.; Dehlendorf, C.; Magnusson, S.L.; Benson, S.K.; Schwarz, E.B.; Borrero, S. Development and pilot testing of a patient-centered web-based reproductive decision support tool for primary care. J. Gen. Intern. Med. 2021 , 36 , 2989–2999. [ Google Scholar ] [ CrossRef ]
• Remelhe, E.; Cerqueira, M.; Faria, P.M.; Paiva, S. Sustainable smart parking solution in a campus environment. EAI Endorsed Trans. Energy Web 2022 , 9 , e2. [ Google Scholar ] [ CrossRef ]
• Wang, G.; Qin, Z.; Wang, S.; Sun, H.; Dong, Z.; Zhang, D. Towards accessible shared autonomous electric mobility with dynamic deadlines. IEEE Trans. Mob. Comput. 2022 , 23 , 925–940. [ Google Scholar ] [ CrossRef ]
• Henderson, H.; Grace, K.; Gulbransen-Diaz, N.; Klaassens, B.; Leong, T.W.; Tomitsch, M. From parking meters to vending machines: A study of usability issues in self-service technologies. Int. J. Hum. -Comput. Interact. 2023 , 40 , 4365–4379. [ Google Scholar ] [ CrossRef ]
• Armand, T.P.T.; Mozumder, M.A.I.; Ali, S.; Amaechi, A.O.; Kim, H.C. Developing a Low-Cost IoT-Based Remote Cardiovascular Patient Monitoring System in Cameroon. Healthcare 2023 , 11 , 199. [ Google Scholar ] [ CrossRef ]
• Moriuchi, E.; Berbary, C.; Easton, C. Looking through the lenses of a patient: An empirical study on the factors affecting patients’ intention to use avatar-assisted therapy. J. Technol. Behav. Sci. 2023 , 8 , 100–112. [ Google Scholar ] [ CrossRef ]
• Medich, M.; Cannedy, S.L.; Hoffmann, L.C.; Chinchilla, M.Y.; Pila, J.M.; Chassman, S.A.; Calderon, R.A.; Young, A.S. Clinician and Patient Perspectives on the Use of Passive Mobile Monitoring and Self-Tracking for Patients with Serious Mental Illness: User-Centered Approach. JMIR Hum. Factors 2023 , 10 , e46909. [ Google Scholar ] [ CrossRef ]
• Jabbar, W.A.; Tiew, L.Y.; Shah, N.Y.A. Internet of things enabled parking management system using long range wide area network for smart city. Internet Things Cyber-Phys. Syst. 2024 , 4 , 82–98. [ Google Scholar ] [ CrossRef ]
• Bohman, S.; Hansson, H.; Mobini, P. Online participation in higher education decision-making. JeDEM-Ejournal Edemocracy Open Gov. 2014 , 6 , 267–285. [ Google Scholar ] [ CrossRef ]
• Bahadori, H.; Vahdat-Nejad, H.; Moradi, H. CrowdBIG: Crowd-based system for information gathering from the earthquake environment. Nat. Hazards 2022 , 114 , 3719–3741. [ Google Scholar ] [ CrossRef ]
• Fuglerud, K.S.; Røssvoll, T.H. An evaluation of web-based voting usability and accessibility. Univ. Access Inf. Soc. 2012 , 11 , 359–373. [ Google Scholar ] [ CrossRef ]
• Yang, D.; Zhang, D.; Frank, K.; Robertson, P.; Jennings, E.; Roddy, M.; Lichtenstern, M. Providing real-time assistance in disaster relief by leveraging crowdsourcing power. Pers. Ubiquitous Comput. 2014 , 18 , 2025–2034. [ Google Scholar ] [ CrossRef ]
• Brito, J.; Vieira, V.; Duran, A. Towards a framework for gamification design on crowdsourcing systems: The GAME approach. In Proceedings of the 12th International Conference on Information Technology-New Generations, Las Vegas, NV, USA, 13–15 April 2015; pp. 445–450. [ Google Scholar ] [ CrossRef ]
• Pistolato, A.C.; Brandão, W.C. ConnectCity: A collaborative e-government approach to report city incidents. In Proceedings of the 15th International Conference WWW/Internet 2016, Mannheim, Germany, 28–30 October 2016; pp. 233–237. [ Google Scholar ]
• Winckler, M.; Bernhaupt, R.; Bach, C. Identification of UX dimensions for incident reporting systems with mobile applications in urban contexts: A longitudinal study. Cogn. Technol. Work 2016 , 18 , 673–694. [ Google Scholar ] [ CrossRef ]
• Idris, N.H.; Osman, M.J.; Kanniah, K.D.; Idris, N.H.; Ishak, M.H.I. Engaging indigenous people as geo-crowdsourcing sensors for ecotourism mapping via mobile data collection: A case study of the Royal Belum State Park. Cartogr. Geogr. Inf. Sci. 2017 , 44 , 113–127. [ Google Scholar ] [ CrossRef ]
• Bousios, A.; Gavalas, D.; Lambrinos, L. CityCare: Crowdsourcing daily life issue reports in smart cities. In Proceedings of the 2017 IEEE Symposium on Computers and Communications (ISCC), Heraklion, Greece, 3–6 July 2017; pp. 266–271. [ Google Scholar ] [ CrossRef ]
• Koroleva, M.B.K.; Vitorino, K.D.D.; Novak, J. Developing a collective awareness platform for urban sustainability challenges: Case study of the POWER project. Eur. J. Sustain. Dev. 2019 , 8 , 214. [ Google Scholar ] [ CrossRef ]
• Falcão, A.G.R.; Wanderley, P.F.; da Silva Leite, T.H.; de Souza Baptista, C.; de Queiroz, J.E.R.; de Oliveira, M.G.; Rocha, J.H. Crowdsourcing urban issues in smart cities: A usability assessment of the Crowd4City system. In Proceedings of the 8th International Conference on Electronic Government and the Information Systems Perspective, Linz, Austria, 26–29 August 2019; pp. 147–159. [ Google Scholar ] [ CrossRef ]
• Hasim, W.; Wibirama, S.; Nugroho, H.A. Redesign of E-participation using user-centered design approach for improving user experience. In Proceedings of the 2019 International Conference on Information and Communications Technology (ICOIACT), Yogyakarta, Indonesia, 24–25 July 2019; pp. 857–861. [ Google Scholar ] [ CrossRef ]
• Knecht, K.; Stefanescu, D.A.; Koenig, R. Citizen Engagement through Design Space Exploration Integrating citizen knowledge and expert design in computational urban planning. In Proceedings of the 37th Education and Research in Computer Aided Architectural Design in Europe and XXIII Iberoamerican Society of Digital Graphics, Joint Conference, Porto, Portugal, 11–13 September 2019; pp. 785–794. [ Google Scholar ] [ CrossRef ]
• Nguyen, Q.N.; Frisiello, A.; Rossi, C. The Design of a Mobile Application for Crowdsourcing in Disaster Risk Reduction. In Proceedings of the 16th International Conference on Information Systems for Crisis Response and Management, València, Spain, 19–22 May 2019; pp. 607–618. [ Google Scholar ]
• Zabaleta, K.; Lopez-Novoa, U.; Pretel, I.; López-de-Ipiña, D.; Cartelli, V.; Di Modica, G.; Tomarchio, O. Designing a Human Computation Framework to Enhance Citizen-Government Interaction. J. Univers. Comput. Sci. 2019 , 25 , 122–153. [ Google Scholar ] [ CrossRef ]
• Dioquino, J.; Cac, A.R.; Tandingan, D.R. Development of Material Recovery Trading System: An Innovative Tool for Community Waste Management Support. In Proceedings of the International Conference on Information Technology and Digital Applications 2019, Yogyakarta, Indonesia, 15 November 2019; p. 012013. [ Google Scholar ] [ CrossRef ]
• Aditya, T.; Sugianto, A.; Sanjaya, A.; Susilo, A.; Zawani, H.; Widyawati, Y.S.; Amin, S. Channelling participation into useful representation: Combining digital survey app and collaborative mapping for national slum-upgrading programme. Appl. Geomat. 2020 , 12 , 133–148. [ Google Scholar ] [ CrossRef ]
• Asfarian, A.; Putra, R.P.; Panatagama, A.P.; Nurhadryani, Y.; Ramadhan, D.A. E-Initiative for Food Security: Design of Mobile Crowdfunding Platform to Reduce Food Insecurity in Indonesia. In Proceedings of the 8th International Conference on Information and Communication Technology (ICoICT), Yogyakarta, Indonesia, 24 June 2020; pp. 1–5. [ Google Scholar ] [ CrossRef ]
• Liu, H.K.; Hung, M.J.; Tse, L.H.; Saggau, D. Strengthening urban community governance through geographical information systems and participation: An evaluation of my Google Map and service coordination. Aust. J. Soc. Issues 2020 , 55 , 182–200. [ Google Scholar ] [ CrossRef ]
• Biswas, R.; Arya, K.; Fernandes, V.; Shah, T. Find A Loo: An app for sanitation governance. Inf. Commun. Soc. 2021 , 24 , 1586–1602. [ Google Scholar ] [ CrossRef ]
• Janoskova, P.; Stofkova, K.R.; Kovacikova, M.; Stofkova, J.; Kovacikova, K. The concept of a smart city communication in the form of an urban mobile application. Sustainability 2021 , 13 , 9703. [ Google Scholar ] [ CrossRef ]
• Jindal, A.; Chowdhury, A. Designing a donation portal to help underprivileged Indians. In Proceedings of the International Conference on Research into Design, Mumbai, India, 7–10 January 2021; pp. 399–411. [ Google Scholar ] [ CrossRef ]
• Takenouchi, K.; Choh, I. Development of a support system for creating disaster prevention maps focusing on road networks and hazardous elements. Vis. Comput. Ind. Biomed. Art 2021 , 4 , 22. [ Google Scholar ] [ CrossRef ]
• Thoneick, R. Integrating online and onsite participation in urban planning: Assessment of a digital participation system. Int. J. E-Plan. Res. (IJEPR) 2021 , 10 , 1–20. [ Google Scholar ] [ CrossRef ]
• Wernbacher, T.; Pfeiffer, A.; Gebetsroither-Geringer, E.; Goels, M.; Worster, J.; Meißner, E.; Graf, A.; Stollnberger, R.; Geyer, R.; Schmidt, R.R.; et al. HotCity-A gamified token system for reporting waste heat sources. In Proceedings of the International Conference on Intelligent Systems & Networks, Hanoi, Vietnam, 19–19 March 2021; pp. 15–27. [ Google Scholar ] [ CrossRef ]
• Görgü, L.; O’Grady, M.; Mangina, E.; O’Hare, G.M. Participatory risk management in the smart city. In Proceedings of the 2022 IEEE International Smart Cities Conference (ISC2), Paphos, Cyprus, 26–29 September 2022; pp. 1–6. [ Google Scholar ] [ CrossRef ]
• Matsuda, Y.; Kawanaka, S.; Suwa, H.; Arakawa, Y.; Yasumoto, K. ParmoSense: Scenario-based Participatory Mobile Urban Sensing Platform with User Motivation Engine. Sens. Mater. 2022 , 34 , 3063. [ Google Scholar ] [ CrossRef ]
• Ong, A.K.S.; Prasetyo, Y.T.; Kusonwattana, P.; Mariñas, K.A.; Yuduang, N.; Chuenyindee, T.; Robas, K.P.E.; Persada, S.F.; Nadlifatin, R. Determining factors affecting the perceived usability of air pollution detection mobile application “AirVisual” in Thailand: A structural equation model forest classifier approach. Heliyon 2022 , 8 , e12538. [ Google Scholar ] [ CrossRef ]
• Gonzales, A.L.R.; Ingalla, E.J.M.; Javier, N.A.F.; Serrano, E.A.; Rodriguez, R.L. CharitAble: A Software Application for Charity Donation. In Proceedings of the 14th International Conference on Humanoid, Nanotechnology, Information Technology, Communication and Control, Environment, and Management, Boracay, Philippines, 1–4 December 2022; pp. 1–5. [ Google Scholar ] [ CrossRef ]
• Ananta, M.T.; Rohidah, S.; Brata, K.C.; Abidin, Z. Mobile Crowdsourcing App Design: Managing Waste Through Waste Bank in Rural Area of Indonesia. In Proceedings of the 8th International Conference on Sustainable Information Engineering and Technology, Bali, Indonesia, 24–25 October 2023; pp. 664–672. [ Google Scholar ] [ CrossRef ]
• Cortés-Cediel, M.E.; Segura-Tinoco, A.; Cantador, I.; Bolívar, M.P.R. Trends and challenges of e-government chatbots: Advances in exploring open government data and citizen participation content. Gov. Inf. Q. 2023 , 40 , 101877. [ Google Scholar ] [ CrossRef ]
• Junqueira, L.; Freire, A.P.; Grützmann, A.; Zitkus, E. Challenges and barriers faced by older adults to access legislative e-participation platforms in Brazil. Electron. J. Inf. Syst. Dev. Ctries. 2023 , 89 , e12274. [ Google Scholar ] [ CrossRef ]
• Lam, J.; Kop, N.; Houtman, C. The impact of technological support on citizen searches for missing persons. Justice Eval. J. 2023 , 6 , 62–80. [ Google Scholar ] [ CrossRef ]
• Leoputra, C.D.; Satya, D.P.; Al-Ghazali, M.R. Application of User-Centered Design Approach in Developing Interaction Design of In-Kind Donation Feature on a Crowdfunding Platform. In Proceedings of the 2023 International Conference on Electrical Engineering and Informatics (ICEEI), Bandung, Indonesia, 10–11 October 2023; pp. 1–6. [ Google Scholar ] [ CrossRef ]
• Manik, L.P.; Albasri, H.; Puspasari, R.; Yaman, A.; Al Hakim, S.; Kushadiani, S.K.; Riyanto, S.; Setiawan, F.A.; Thesiana, L.; Jabbar, M.A.; et al. Usability and acceptance of crowd-based early warning of harmful algal blooms. PeerJ 2023 , 11 , e14923. [ Google Scholar ] [ CrossRef ]
• Sapraz, M.; Han, S. Users’ evaluation of a digital government collaborative platform (DGCP) in Sri Lanka. Transform. Gov. People Process Policy 2024 , 18 , 131–144. [ Google Scholar ] [ CrossRef ]
• Bastos, D.; Fernández-Caballero, A.; Pereira, A.; Rocha, N.P. Smart city applications to promote citizen participation in city management and governance: A systematic review. Informatics 2022 , 9 , 89. [ Google Scholar ] [ CrossRef ]
• Sapraz, M.; Han, S. Implicating human values for designing a digital government collaborative platform for environmental issues: A value sensitive design approach. Sustainability 2021 , 13 , 6240. [ Google Scholar ] [ CrossRef ]
• Bajdor, P.; Starostka-Patyk, M. Smart City: A bibliometric analysis of conceptual dimensions and areas. Energies 2021 , 14 , 4288. [ Google Scholar ] [ CrossRef ]
• Sharif, A.; Allam, Z.; Feizizadeh, B.; Ghamari, H. Three decades of research on smart cities: Mapping knowledge structure and trends. Sustainability 2021 , 13 , 7140. [ Google Scholar ] [ CrossRef ]
• Vento, I. Hands-off or hands-on governance for public innovation? A comparative case study in the EU cohesion policy implementation in Finland. Int. J. Public Adm. 2020 , 43 , 989–999. [ Google Scholar ] [ CrossRef ]
• Almeida, A.F.; Rocha, N.P.; Silva, A.G. Methodological quality of manuscripts reporting on the usability of mobile applications for pain assessment and management: A systematic review. Int. J. Environ. Res. Public Health 2020 , 17 , 785. [ Google Scholar ] [ CrossRef ]
• Silva, A.G.; Caravau, H.; Martins, A.; Almeida, A.M.; Silva, T.; Ribeiro, Ó.; Santinha, G.; Rocha, N.P. Procedures of User-Centered Usability Assessment for Digital Solutions: Scoping Review of Reviews Reporting on Digital Solutions Relevant for Older Adults. JMIR Hum. Factors 2021 , 8 , e22774. [ Google Scholar ] [ CrossRef ]

Click here to enlarge figure

Share and Cite

Bastardo, R.; Pavão, J.; Rocha, N.P. Methodological Quality of User-Centered Usability Evaluation of Digital Applications to Promote Citizens’ Engagement and Participation in Public Governance: A Systematic Literature Review. Digital 2024 , 4 , 740-761. https://doi.org/10.3390/digital4030038

Bastardo R, Pavão J, Rocha NP. Methodological Quality of User-Centered Usability Evaluation of Digital Applications to Promote Citizens’ Engagement and Participation in Public Governance: A Systematic Literature Review. Digital . 2024; 4(3):740-761. https://doi.org/10.3390/digital4030038

Bastardo, Rute, João Pavão, and Nelson Pacheco Rocha. 2024. "Methodological Quality of User-Centered Usability Evaluation of Digital Applications to Promote Citizens’ Engagement and Participation in Public Governance: A Systematic Literature Review" Digital 4, no. 3: 740-761. https://doi.org/10.3390/digital4030038

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

• Open access
• Published: 05 September 2024

Exploring instructional design in K-12 STEM education: a systematic literature review

• Suarman Halawa 1 ,
• Tzu-Chiang Lin 2 , 3 &
• Ying-Shao Hsu   ORCID: orcid.org/0000-0002-1635-8213 4  

International Journal of STEM Education volume  11 , Article number:  43 ( 2024 ) Cite this article

Metrics details

This study aimed to analyze articles published in the Web of Science database from 2012 to 2021 to examine the educational goals and instructional designs for STEM education. We selected articles based on the following criteria: (a) empirical research; (b) incorporating instructional design and strategies into STEM teaching; (c) including intervention; (d) focusing on K-12 education and on assessment of learning outcomes; and (e) excluding higher education and STEAM education. Based on the criteria, 229 articles were selected for coding educational goals and instructional designs for STEM education. The aspects of STEM educational goals were coded including engagement and career choice, STEM literacy, and twenty-first century competencies. The categories of instructional designs for STEM education were examined including design-based learning, inquiry-based learning, project-based learning, and problem-based learning. The results showed that engagement and career choices and STEM literacy were mainly emphasized in STEM education. Design-based learning was adopted more than inquiry-based, project-based, or problem-based learning, and this instructional design was mainly used to achieve STEM literacy. It is suggested that studies on twenty-first century competencies may require more research efforts in future STEM education research.

Introduction

Emphasizing STEM (science, technology, engineering, and mathematics) has been the main focus of policy makers in many countries (English, 2016 ; National Academy of Engineering & National Research Council, 2014 ; National Research Council, 2012 , 2013 ) to meet economic challenges (Kelley & Knowles, 2016 ). Educational systems are accordingly prioritizing STEM to prepare students’ capability for the workplace to face the sophisticated technologies and competitive economy (Kayan-Fadlelmula et al., 2022 ). Hence, students are expected to be interested in STEM so that they will engage in and pursue careers in STEM-related fields (Lie et al., 2019 ; Struyf et al., 2019 ). Besides, we need a new generation that has the abilities to develop proficient knowledge, to apply such knowledge to solve problems, and to face existing and upcoming issues of the twenty-first century (Bybee, 2010 ).

Although STEM education has been proved to benefit students, there is a lack of understanding of instructional design for STEM education, despite the fact that such understanding is critical to research and to classroom practices. Limited understanding of relevant instructional design may lead to problems in implementing STEM education in the classroom. There is hence a need to examine educational goals, specific designs, and features of the instructional designs consistently and specifically documented in the STEM education literature. Therefore, this current study conducted systematic analysis of the literature to understand the educational goals and instructional designs for STEM education. Based on the analysis, we present a thorough picture of how researchers have developed instructional designs for STEM education.

Despite the fact that many researchers have promoted STEM education, the definition of STEM education has not reached a consensus in the literature, and there is a certain degree of disagreement in the scientific community. Lamb et al. ( 2015 ) defined STEM as a broad area encompassing many disciplines and epistemological practices. Other researchers, such as Breiner et al. ( 2012 ), defined STEM as applying transdisciplinary knowledge and skills in solving real-world problems. A similar definition established by Shaughnessy ( 2013 ) regarding STEM education is problem solving based on science and mathematics concepts that incorporate engineering strategies and technology. Another study defined STEM education as teaching approaches based on technology and engineering design that integrate the concepts and practices of science and mathematics (Sanders & Wells, 2006 ). In this study, we clarify STEM education as an approach that utilizes integrations of knowledge and skills from science, technology, engineering, and/or mathematics to solve real-world problems that help students to succeed in school learning, future careers, and/or society.

The definition of STEM as an integrated approach involving science, technology, engineering, and mathematics raises several pertinent questions about its composition and expectations. First, the requirement for all four disciplines to be present in order to qualify an educational program or project as “STEM” is debatable. Conceptually, integrating any two or more fields helps foster the interdisciplinary learning that is the hallmark of STEM education. This flexibility allows educators to tailor their programs to match the available resources and specific learning outcomes without necessarily incorporating all four disciplines in every instance. Regarding the classification of “science” within STEM, it is more a conglomerate of disciplines—such as biology, chemistry, physics, and earth sciences—than a single field. This diversity within science enriches STEM education, providing a broader knowledge base and problem-solving skills. Each scientific discipline brings a unique perspective and set of tools to the interdisciplinary mix, enhancing the complexity and richness of STEM learning experiences.

Furthermore, previous studies have identified several challenges to the implementation of STEM education in the classroom including poor motivation of students, weak connection with individual learners, little support from the school system, poor content without integration across disciplines, lack of quality assessments, poor facilities, and lack of hands-on experience (Ejiwale, 2013 ; Hsu & Fang, 2019 ; Margot & Kettler, 2019 ). To help teachers face challenges in the advancement of STEM education, Hsu and Fang ( 2019 ) proposed a 5-step STEM curriculum designs framework and provided examples of how to apply it to a lesson plan to help teachers design their instruction. This previous study also suggested that researchers conduct more investigations related to instructional design to enrich our understanding of various aspects of STEM education. Teachers of STEM require more opportunities to construct their perspective and a vision of STEM education as well as to conduct appropriate instructional designs. Moreover, from review articles published from 2000 to 2016, Margot and Kettler ( 2019 ) found that in multiple studies concerning similar challenges and supports, teachers believed that the availability of a quality curriculum would enhance the success of STEM education. Teachers need to provide and use an appropriate instructional design for STEM education and understand the educational goals. Therefore, we see the need to conduct research related to STEM education, especially exploring the instructional design because identifying and using a quality instructional design could increase the effectivess of STEM education.

According to the previous literature review, educational goals for instructional design were highlighted in STEM education. First, engagement and career choice need to be emphasized in STEM learning to improve students’ interest and self-efficacy (Vongkulluksn et al., 2018 ). Students need to engage in STEM education to raise their interest and engagement in STEM and to increase and develop a STEM-capable workforce (Honey et al., 2014 ; Hsu & Fang, 2019 ; Schütte & Köller, 2015 ). Engaging students in STEM education could improve their attitudes (Vossen et al., 2018 ) and their interest in STEM fields, and encourage them to pursue STEM careers (Means et al., 2017 ).

Second, STEM literacy needs to be promoted in K-12 schools (Falloon et al., 2020 ; Jackson et al., 2021 ) to develop students’ ability to encounter global challenges (Bybee, 2010 ). Students need to have the ability to apply concepts from science, technology, engineering, and mathematics, and skills to solve problems related to social, personal, and global issues in society (Bybee, 2010 ; Jackson et al., 2021 ). Besides, improving students’ STEM literacy is needed for their decision-making, participation in civic and cultural affairs, and economic productivity (National Academy of Engineering & National Research Council, 2014 ; National Research Council, 2011 ).

Last, regarding the twenty-first century competencies, students are anticipated to have abilities of creativity and innovation, problem solving, critical thinking, collaboration and communication (Boon, 2019 ) as citizens, workers, and leaders in the twenty-first century (Bryan et al., 2015 ; National Academy of Engineering & National Research Council, 2014 ; Stehle & Peters-Burton, 2019 ). These abilities are critical for students to adapt and thrive in a changing world (National Research Council, 2013 ). Also, students need to have the abilities to adapt to the twenty-first century in order to succeed in the new workforce (Bybee, 2013 ).

Considering the achievement of students’ engagement, motivation, STEM literacy, as well as twenty-first century competencies, many countries have significantly enlarged the funding for research and education relevant to STEM (Sanders, 2009 ). One of the strands of the existing research is to help teachers know how to implement STEM education in schools (Aranda, 2020 ; Barak & Assal, 2018 ; English, 2017 ). Researchers have proposed instructional designs for STEM education including design-based learning (Kelley & Knowles, 2016 ; Yata et al., 2020 ), inquiry-based learning (Bybee, 2010 ), project-based learning (Capraro et al., 2013 ), and problem-based learning (Carpraro & Slough, 2013 ).

Design-based learning focuses on technological and engineering design. This instructional design engages students in learning about engineering design practices (Fan et al., 2021 ; Guzey et al., 2016 ; Hernandez et al., 2014 ) through the steps of designing, building, and testing (Yata et al., 2020 ). Design-based learning promotes problem solving, design, building, testing, and communication skills (Johnson et al., 2015 ) and improves students’ interest in STEM activities (Vongkulluksn et al., 2018 ). Also, design-based learning improves students’ engineering abilities and twenty-first century competencies (Wu et al., 2019 ) and attitudes (Vossen et al., 2018 ), and engages them in understanding core disciplinary ideas (Guzey et al., 2016 ).

Inquiry-based learning focuses on engaging students in hands-on activities to investigate scientific phenomena (Lederman & Lederman, 2012 ) and to construct their new knowledge (Bybee, 2010 ; Halawa et al., 2020 ). Students are encouraged to plan and design their experiments, analyze and interpret data, argue, and communicate their findings (Halawa et al., 2023 ; National Research Council, 2012 , 2013 ). Inquiry-based learning is also deemed to improve students’ knowledge, interest, engagement (Sinatra et al., 2017 ) and creativity (Smyrnaiou et al., 2020 ). Besides, researchers have noticed the importance of inquiry-based learning for improving students’ attitudes toward science-related careers (Kim, 2016 ). Although inquiry-based learning mainly focuses on science education to engage students in authentic learning (Halawa et al., 2024 ), it has been known to share common goals and characteristics with mathematics, technology, and engineering (Grangeat et al., 2021 ; Lin et al., 2020 ). Common elements in STEM education are engaging students in asking questions and testing their ideas in a systematic and interactive way (Grangeat et al., 2021 ).

Project-based learning and problem-based learning, both instructional designs, engage students in experiential and authentic learning with open-ended and real-world problems (English, 2017 ). Yet, project-based learning tends to be of longer duration and occurs over an extended period of time (Wilson, 2021 ), while problem-based learning is usually embedded in multiple problems (Carpraro & Slough, 2013 ). STEM project-based learning focuses on engaging students in an ill-defined task within a well-defined outcome situated with a contextually rich task, requiring them to solve certain problems (Capraro et al., 2013 ). Project-based learning and problem-based learning are both used to develop students’ problem solving, creativity, collaboration skills (Barak & Assal, 2018 ), and attitude (Preininger, 2017 ).

According to previous studies, researchers have adopted STEM instructional designs to achieve certain educational goals. For instance, in the aspects of engagement and career choice, Sullivan and Bers ( 2019 ) used design-based learning to improve students’ interest in engineering and students’ performance in elementary school. Kang et al. ( 2021 ) adopted inquiry-based learning for secondary school by embedding careers education to foster the students’ interest in science. Vallera and Bodzin ( 2020 ) adopted project-based learning at primary school in the northeastern United States to improve students’ STEM literacy and attitude. Preininger ( 2017 ) used problem-based learning to influence students’ attitudes toward mathematics and careers involving mathematics. In the aspect of STEM literacy, King and English ( 2016 ) adopted design-based learning to enable students to apply STEM concepts to the model of the construction of an optical instrument. Han et al. ( 2015 ) adopted STEM project-based learning to improve the performance of low-performing students in mathematics. Lastly, regarding the twenty-first century competencies, English et al. ( 2017 ) adopted design-based learning to improve students’ capabilities of handling the complexity of the task (English et al., 2017 ).

In conclusion, studies have grown to explore educational goals related to instructional designs for STEM education. However, consistent and systematic reviews related to instructional designs in K-12 STEM education are comparatively scarce. Although there are some reviews of the STEM education literature (Andrews et al., 2022 ; Gladstone & Cimpian, 2021 ; Kaya-Fadlelmula et al., 2022 ; López et al., 2022 ; Margot & Kettler, 2019 ; Martín-Páez et al., 2019 ; Nguyen et al., 2021 ), it is noteworthy that previous studies only explored undergraduate instruction in STEM education (Andrews et al., 2022 ; Henderson et al., 2011 ; Nguyen et al., 2021 ). Therefore, to fill the research gap, this current study conducted a systematic analysis of literature to understand the educational goals and instructional designs for K-12 STEM education from articles published between 2012 and 2021. The research questions of this study were formulated as follows:

What STEM education goals were more focused on in the reviewed articles? What was the trend of educational goals in the reviewed articles?

What instructional designs were more focused on in the reviewed articles? What was the trend of the instructional design in the review articles?

What instructional designs were more focused on to achieve certain educational goals in the reviewed articles?

What features of instructional designs were more focused on in the reviewed articles?

Data collection

To identify the target literature for further analysis, this study conducted several rounds of searching the Web of Science (WOS) database for articles (Gough et al., 2012 ; Møller & Myles, 2016 ). A systematic literature review using the PRISMA guidelines was used for article selection (Møller & Myles, 2016 ). First, we searched for articles using the keyword “STEM Education” along with “learning”, “teaching”, “curriculum”, and “professional development”, to refine the search results. The search identified a total of 1,531 articles published in the Web of Science from 2012 to 2021 (Fig.  1 ). We initially excluded duplicated articles; the search retrieved a total of 1,513 articles. We then screened the titles, abstract, and keywords of the articles based on the following criteria: (a) empirical research; (b) incorporating instructional design and strategies into STEM teaching; (c) including intervention; (d) focusing on K-12 education and on assessment of learning outcomes; and (e) excluding higher education and STEAM education. During this screening, we discussed which articles met the criteria through round-table discussions, and determined the preliminary target candidates composed of 394 articles. A full-text examination was then conducted. In this round of examination, we removed the articles without clear information about the educational goals and instructional designs related to STEM education. Finally, a corpus of literature comprising 229 articles was formed for further analysis.

PRISMA flow diagram of articles selection

Data analysis

According to the research questions, for this study, we developed a coding framework to conduct content analysis and to categorize the target literature. We first selected paradigmatic references of STEM education and instructional design from high quality publications. These articles provided sets of core concepts and terms to shape the provisional coding categories. We then constantly reviewed the paradigmatic references and discussed them to improve the coding scheme. The final analytic framework with coding categories was developed as follows. The first category, STEM educational goals, includes engagement and career choice (Honey et al., 2014 ; Hsu & Fang, 2019 ), STEM literacy (Falloon et al., 2020 ; Jackson et al., 2021 ), and twenty-first century competencies (Boon, 2019 ) (see Appendix 1). The second category, instructional design, includes design-based learning (Yata et al., 2020 ), inquiry-based learning (Bybee, 2010 ; Halawa et al., 2020 ), project-based learning (Capraro & Slough, 2013 ), and problem-based learning (Priemer et al., 2020 ). From the review articles, we found that 6E - oriented STEM (engage, explore, explain, engineer, enrich, and evaluate) and game-based learning were used for STEM education. These two instructional designs were added to our coding scheme. Articles that did not specify the instructional design were coded as “others”. We then analyzed the outcomes to see whether instructional design successfully improved STEM educational goals. We analyzed design-based, inquiry-based, and project-based learning to achieve engagement and career choice, STEM literacy, and a combination of engagement and career choice and STEM literacy because the selected articles mainly concentrated on them. We categorized the outcomes as positively improved, partially improved, and none (Amador et al., 2021 ). Instructional design that successfully increased STEM educational goals was categorized as positively improved. Instructional design that only increased a part of STEM educational goals was categorized as partially improved. If the instructional design did not increase STEM educational goals, we categorized it as none.

We then extended our coding scheme to identify the features of design-based, inquiry-based, and project-based learning. We focused on these three instructional designs because the selected articles mainly adopted them. Yata et al. ( 2020 ) proposed designing, building, and testing as the features of design-based learning. Other features of instructional designs including questioning or identifying problems, experimenting, analyzing, explaining, collaborating, communicating, and reflecting were proposed as features of inquiry-based learning (Bybee, 2010 ; Halawa et al., 2020 ) and project-based learning (Capraro et al., 2013 ). From the review articles, we found that redesigning was one of the features of instructional design and so added it to the coding scheme. These features of instructional designs were adopted for our coding scheme including questioning or identifying problems, designing, building, testing, experimenting, analyzing, collaborating, reflecting, communicating, and redesigning (Appendix 2). We then calculated the number of articles that adopted these features of instructional designs. We further summarized the features of instructional designs that were frequently used in the selected articles.

In order to make sure the coding process was reliable, we conducted a trial coding by randomly selecting 40 articles and individually categorizing the articles into the aforementioned categories: (a) STEM education goal, and (b) instructional design. Interrater reliability was calculated using a percent agreement metric reaching an acceptable level of 0.85 (McHugh, 2012 ). The discrepancies between authors were negotiated and solved through discussions. The NVivo 11 software was utilized to complete coding works on the remaining articles. We then calculated and reported descriptive statistics of the coded data as the analytic results.

Engagement and career choice as the main focused STEM educational goals

Table 1 shows that more articles focused on engagement and career choice (64 articles) and STEM literacy (61 articles) than twenty-first century competencies (16 articles). The articles also mainly focused on a combination of engagement and career choice and STEM literacy (47 articles) and a combination of engagement and career choice and twenty-first century competencies (18 articles). Nine articles were found that focused on the three learning goals of engagement and career choice, STEM literacy, and twenty-first century competencies.

Table 1 shows the numbers of articles regarding educational goals for STEM education for each 2 years in the review papers. The number of articles per 2 years increased from 2012 to 2021. The trend analysis indicated that engagement and career choice and STEM literacy increased greatly from 2014 to 2021. The numbers of articles focused on the combination of two educational goals (STEM literacy and twenty-first competencies) and three learning goals (engagement and career choice, STEM literacy, and twenty-first competencies) from 2016 to 2021 are also presented.

Design-based and inquiry-based learning as the main instructional designs for STEM

Table 2 reveals the numbers of articles that used instructional design for STEM education. The instructional designs of design-based, inquiry-based, project-based, and problem-based learning were mainly used and continued to be used over the study period. The trend analysis indicated a big jump in design-based, inquiry-based, and project-based learning from 2018 to 2021.

Table 2 also shows the instructional designs and educational goals for STEM from review papers. Most articles adopted design-based (80 articles), inquiry-based (46 articles), project-based (42 articles), and problem-based (27 articles) learning.

Design-based learning mainly used to achieve STEM literacy

The findings shown in Table  3 identified that STEM instructional designs were used differently to achieve engagement and career choice, STEM literacy, and the combination of engagement and career choice and STEM literacy. We found that design-based learning was mainly adopted to achieve STEM literacy (28 articles), while inquiry-based learning was mainly used to achieve engagement and career choice (14 articles) and the combination of engagement and career choice and STEM literacy (14 articles). Also, more articles (15 articles) adopted project-based learning to achieve engagement and career choice. Furthermore, more design-based learning (7 articles) and problem-based learning (4 articles) than inquiry-based learning (2 articles) and project-based learning (1) articles were adopted to achieve twenty-first century competencies.

As we identified that a major portion of the articles adopted design-based learning, inquiry-based learning, and project-based learning focused on engagement and career choice, STEM literacy, and a combination of engagement and career choice and STEM literacy (see Table  3 ), we focused further analysis on the outcomes of STEM educational goals in the articles. The total number of selected articles was 124, of which 54 adopted design-based learning, 37 adopted inquiry-based learning, and 33 adopted project-based learning (Table  4 ).

We categorized the outcomes of STEM education goals into three categories (positively improved, partially improved, and none) (Amador et al., 2021 ). Table 4 shows that the majority of selected articles adopted design-based, inquiry-based, and project-based learning, improving STEM educational goals positively. Most selected articles found that design-based learning positively improved engagement and career choice (10 articles), STEM literacy (26 articles), and a combination of engagement and career choice and STEM literacy (15 articles). Also, most of the selected articles indicated that inquiry learning has a positive impact on engagement and career choice (14 articles), STEM literacy (7 articles), and a combination of engagement and career choice and STEM literacy (13 articles). Project-based learning has demonstrated a beneficial impact on various outcomes, as reported across the selected literature. Specifically, 12 articles documented the enhancement of engagement and career decisions, nine indicated the advancement of STEM literacy, and six discussed a combined effect on engagement, career choice, and STEM literacy.

Frequently used features of STEM instructional designs

To identify the frequently used features of STEM instructional design, we further explored the activities in the selected articles. As previous results show that the major part of articles adopted design-based learning, inquiry-based learning, and project-based learning, we further analyzed the frequently used features of these STEM instructional designs that focused on engagement and career choice, STEM literacy, and combination of engagement and career choice and STEM literacy (see Table  3 ). We selected 54 articles that adopted design-based learning, 37 adopted inquiry-based learning, and 33 adopted project-based learning (Table  5 ).

Frequently used features of design-based learning

Based on the findings, a large portion of the selected articles adopted design-based learning for STEM education (54 articles). Table 5 shows the features that were adopted to implement instructional design for design-based learning. More than half of the selected articles adopted designing, building, testing, collaborating, experimenting, and reflecting. Building (88.9%), designing (87.0%), and testing (70.4%) were used to engage students in engineering (Yata et al., 2020 ). Besides, engaging students in these activities required students to use their knowledge and skills (Kelley & Knowles, 2016 ). For example, Aranda et al. ( 2020 ) and Lie et al. ( 2019 ) implemented design-based learning by asking students to design a process to both prevent and test for cross-pollination of non-GMO from GMO fields. In these selected articles, the curriculums were focused on helping students with designing, building, and testing.

Collaborating, which engages students in working with their classmates in the process of design-based learning, was also mainly emphasized in the selected articles (64.8%). For instance, English and King ( 2019 ) asked students to work with their groups to discuss the possible design of the bridge. Researchers also emphasized experimenting (53.7%) to engage students in design-based learning. English ( 2019 ) engaged students in investigating their feet and shoes. Students collected, represented, analyzed data, and drew conclusions from their findings. Lie et al. ( 2019 ) helped students conduct an investigation to prevent cross-contamination of non-GMO from GMO corn fields. The last critical feature of design-based learning is reflecting (51.9%). In this activity, students engaged in assessing their solutions against a set of criteria and constraints, generating, and evaluating solutions (Cunningham et al., 2019 ). By engaging students in reflecting, students have an opportunity to improve their design and choose their best strategy (Aranda et al., 2020 ; Lie et al., 2019 ).

Frequently used features of inquiry-based learning

As shown in Table  5 , the inquiry-based learning approach was frequently adopted by researchers for STEM education. The features of this approach applied to achieve specific STEM education goals (e.g., engagement and career choice, and STEM literacy) included experimenting (91.9%), collaborating (83.8%), reflecting (62.2%), and communicating (51.4%) (see Table  5 ). This finding indicated that the top three frequently used features of inquiry-based learning in STEM were experimenting, collaborating, and reflecting, which play an essential role when learners try out their ideas about a real-world problem related to STEM. For example, a four-phase inquiry (clarifying the situation, hands-on experiments, representing, analyzing the produced data, and reporting/whole-class discussions) for authentic modeling tasks guided students to develop their credibility of the tasks and to acquire STEM knowledge (Carreira & Baioa, 2018 ).

Frequently used features of project-based learning

As previously mentioned, project-based learning is one of the major approaches to support instructional design in the reviewed STEM education studies. The results shown in Table  5 further indicate the features that researchers tended to integrate into instructional design for project-based learning. More than half (51.5%) of the selected articles reported “reflecting” as a pivotal part of teaching that triggered students’ project-based learning. Reflecting is deemed to depict learners’ active perceptions and deliberation of what they encounter and what they are doing. This may contribute to their competence to retrieve appropriate information, to provide feedback, and to revise the project underlying their learning. For example, in Dasgupta et al.’s ( 2019 ) study, a design journal was utilized to help students’ reflection on what they knew, what is necessary to know, as well as their learning outcomes. Vallera and Bodzin ( 2020 ) also addressed the critical design features of their curriculum to help students achieve information obtaining, evaluating, and communicating in the learning project based on real-world contexts.

Besides, researchers focused on project-based learning regarding STEM have a tendency to foster students’ learning via “identifying problems” (48.5%). These studies can be differentiated into two types based on whether the researchers provided a driving question for the learning project. In Vallera and Bodzin’s ( 2020 ) study, the instructional design arranged a clear-cut driving question to guide students’ thinking about helping farmers to prepare products for sale in a farmers’ market. This led students to extend their thinking and identify further problems while solving the driving question. As for Barak and Assal’s ( 2018 ) study, their instructional design provided open-ended tasks and ill-defined problems. Such arrangements were deemed to afford students’ learning through problem defining and learning objective setting.

It is also noteworthy to mention that the percentages of “experimenting” and “collaborating” in studies involved with project-based learning design were lower than those of studies with design-based learning or inquiry-based learning. However, researchers who were interested in STEM project-based learning would still to some extent agree with instructional design that may provide opportunities to students to access authentic scientific activities and social communications.

This study focused on analyzing the STEM educational goals and instructional designs adopted in the 2012–2021 articles. The findings of this study present knowledge and understanding of the educational goals that need to be considered in STEM education, and how these goals could be achieved by adopting various STEM instructional designs.

Educational goals for STEM education

The majority of reviewed articles adopted instructional designs to achieve the goals of engagement, career choice and STEM literacy. In contrast, few articles focused on twenty-first century competencies. It is not surprising because many recent studies in nature emphasized economic viewpoints and workplace-readiness outcomes in the STEM education field (Cheng et al., 2021 ; Kelley & Knowles, 2016 ). The aspects of engagement and career choice were frequently considered in many previous studies on STEM education (Struyf et al., 2019 ; Vongkulluksn et al., 2018 ; Vossen et al., 2018 ). It indicated that engagement and career choice are important goals for STEM education (Honey et al., 2014 ; Hsu & Fang, 2019 ; Kelley & Knowles, 2016 ). Engaging and motivating students in STEM education are necessary to enhance their understanding of their future careers (Fleer, 2021 ) and to cultivate them to continue STEM learning (Maltese et al., 2014 ). Students who were motivated and interested in STEM education would pursue STEM careers (Maltese & Tai, 2011 ). Furthermore, the aspects of STEM literacy are also addressed in the reviewed articles. The aspects of STEM literacy (e.g., knowledge and capabilities) are deemed important for students’ productive engagement with STEM studies, issues, and practices (Falloon et al., 2020 ). The focus of STEM literacy encourages students to apply their knowledge to life situations and solve problems (Bybee, 2010 ). The importance of STEM literacy has been highlighted in several national documents (e.g., Committee on STEM Education of the National Science & Technology Council, 2018 ; National Research Council, 2011 ; U.S. Department of Education, 2016 ). These findings provide insights into what teaching goals have been focused on in STEM education. For instance, engagement and career choice have been mainly focused on in STEM education because the STEM teaching was designed to connect to the students’ real-world experiences or future professional situations (Strobel et al., 2013 ). The authentic and meaningful experience could engage and motivate students in the activity, and later they should pursue their future careers related to what they have learned.

However, there are few selected articles focused on twenty-first century competencies, although many previous studies considered the twenty-first century competencies as important goals for students. Some studies have advocated that students should be engaged in interdisciplinary sets of complex problems and encourage them to use critical thinking and develop their creativity and innovation as well as collaboration (Finegold & Notabartolo, 2010 ; Jang, 2016 ). Engaging students in STEM education focused on twenty-first century competencies could prepare them for the workplace and help them become successful in STEM-related fields (Jang, 2016 ). Future researchers should consider integrating twenty-first century competencies into STEM education to complement the existing focus on engagement, career choice, and STEM literacy, preparing students for a broader range of skills necessary for the modern workforce.

Instructional design for STEM education

Although the reviewed articles adopted various instructional designs for STEM education, the articles mostly adopted design-based rather than inquiry-based, project-based, or problem-based learning. The findings are in accordance with the existing literature on STEM education. Notably, these results corroborate the conclusions drawn from a comprehensive systematic review conducted by Mclure et al. ( 2022 ). Design-based learning was adopted to achieve the goals of STEM literacy, engagement and career choice, and this instructional design tended to be used more often according to the trend analysis. This indicated that design-based learning was considered as a main instructional design for STEM education. This instructional design has become an essential approach to engaging K-12 students in STEM education (Bybee, 2013 ; National Academy of Engineering & National Research Council, 2014 ; National Research Council, 2013 ). Some researchers claimed that students who participate in design-based learning could make meaningful connections between knowledge and skills by solving problems (English & King, 2019 ; Kelley et al., 2010 ). Design-based learning engages students in authentic problems and challenges that increase their level of engagement (Sadler et al., 2000 ), help students learn fundamental scientific principles (Mehalik et al., 2008 ), and build students’ natural and intuitive experience (Fortus et al., 2004 ). In the process of design, students learn the concepts of science, technology, and mathematics in the process of designing, building, or testing products (Yata et al., 2020 ). For instance, students have to learn the concept of energy to design a house that produces more renewable energy than it consumes over a period of 1 year (Zheng et al., 2020 ). It was also found that the majority of selected articles which adopted design-based learning successfully improved learners’ engagement, career choice, and STEM literacy (Table  4 ). The results align with the findings of a previous meta-analysis focusing on STEM education at the middle school level (Thomas & Larwin, 2023 ). K-12 students’ STEM learning successfully improved because the selected articles reported studies conducting design-based learning in K-12 education. For example, Cunningham et al. ( 2019 ) successfully implemented design-based learning to improve elementary students’ learning outcomes, while Fan et al. ( 2018 ) found that design-based learning positively improved secondary students’ conceptual knowledge and attitude.

However, the selected articles have not equally used the features of design-based learning such as collaborating, reflecting, and redesigning. We identified that the selected articles mainly used designing, building, and testing to engage students in engineering activities. One of the explanations for this finding is that researchers may face challenges in implementing a full cycle of design-based learning because of the time limit of instruction, so they only focus on the process of designing, building, and testing. Collaborating, reflecting, and redesigning should be emphasized while adopting effective design-based learning because students could solve complex problems by collaborating with others. With collaboration, the students can learn/solve problems through discussion within the group. This activity allows students to share new ideas and debate with others to generate solutions. Reflecting on the data and experience allows students to make improvements to their model and leads them to redesign it to produce a better model. This process could also grow students’ science knowledge (Fortus et al., 2004 ). This finding hence suggests future studies, and educators emphasize more collaborating, reflecting, and redesigning for design-based learning for STEM instruction.

Moreover, inquiry-based learning, project-based learning, and problem-based learning were adopted in some selected articles. Inquiry-based learning was considered to enable and to promote connections within and across curriculum disciplines and improve students’ engagement in STEM education (Attard et al., 2021 ). Project-based and problem-based learning can be used to engage students in authentic problems (Blumenfeld et al., 1991 ) and to improve their engagement in STEM education (Beckett et al., 2016 ). Furthermore, we identified that inquiry-based learning mainly engages students in experimenting, collaborating, and reflecting (Kim, 2016 ), and project-based learning (Han et al., 2015 ) mainly engages students in identifying problems and reflecting. This finding reveals the frequently used features of inquiry-based learning and project-based learning. Teachers could use these components of instructional design for preparing their instruction for teaching STEM. Given these findings, it is advisable to explore the integration of inquiry-based, project-based, and problem-based learning alongside design-based learning in STEM education. Such an approach may enhance the effectiveness of STEM education by providing a more comprehensive strategy to improve STEM literacy, engagement, and career choice among K-12 students.

However, we identified that some essentials of these instructional designs have not been included in selected articles. For instance, studies adopting inquiry-based learning rarely asked students to propose their questions, although questioning is one of the frequently used features of inquiry (National Research Council, 2012 , 2013 ). One of the possible explanations for this finding is that students may have a lack of experience with inquiry learning and not know how to formulate meaningful questions, and they may tend to propose low-level factual questions related to their personal interests (Krajcik et al., 1998 ). Besides, STEM education requires students to engage in complex real-world problems, which requires sufficient ability to propose meaningful questions. Yet, we expect that future studies and teachers should encourage students to propose their own questions because questioning improves students’ creativity, critical thinking, and problem solving skills (Hofstein et al., 2005 ). Teachers could start asking students to propose their own questions once they have experience and ability to propose good questions. Krajcik et al. ( 1998 ) suggested providing situations in which students can receive informative and critical feedback from teachers, classmates, and others so as to propose their own significant questions.

Conclusions

From an instructional design perspective, this study provides crucial insights into practical STEM education approaches. The findings underscore the importance of aligning instructional designs with specific STEM educational goals. The trend analysis revealed a significant increase in focus on engagement, career choice, and STEM literacy from 2014 to 2021, with a particularly sharp rise observed between 2018 and 2021. Each instructional design approach demonstrated unique strengths: design-based learning fosters STEM literacy. In contrast, inquiry-based and project-based learning effectively enhanced engagement and career choice. The study delineates specific features of these instructional designs that contribute to their success, such as building and testing in design-based learning, experimenting and collaborating in inquiry-based learning, and reflecting and problem identification in project-based learning.

Furthermore, this study advocates for a deliberate and systematic application of inquiry-based and project-based learning alongside design-based learning. Such integration is likely to cultivate a more dynamic and interactive learning environment that encourages critical thinking, problem-solving, and collaborative skills among students. The integration of twenty-first century competencies in the instructional design of STEM, though less presented, suggests a potential research space for further exploration of STEM teaching. This study recommends an expanded focus on incorporating these competencies to ensure a holistic educational approach that addresses immediate educational goals and equips students with essential skills for future challenges.

Teachers’ limited understanding of STEM instructional design also presents a significant challenge, necessitating targeted professional development initiatives. Educators must comprehend and implement a comprehensive approach that aligns educational goals with appropriate instructional designs to optimize STEM learning outcomes. This approach involves clearly defining learning objectives, such as STEM literacy, selecting suitable instructional designs, and effectively guiding students through the chosen learning process.

The findings in this study furnish instructional designers and educators with a clear framework for developing targeted STEM curricula. The research accentuates the importance of aligning instructional design features with specific educational goals, suggesting that a nuanced, goal-oriented approach to STEM instruction can significantly enhance student outcomes in literacy, engagement, and career readiness. These insights offer a robust foundation for refining and optimizing instructional design strategies in STEM education.

Availability of data and materials

No applicable.

Amador, J. M., Bragelman, J., & Superfine, A. C. (2021). Prospective teachers’ noticing: A literature review of methodological approaches to support and analyze noticing. Teaching and Teacher Education . https://doi.org/10.1016/j.tate.2020.103256

Article   Google Scholar  

Andrews, T. C., Speer, N. M., & Shultz, G. V. (2022). Building bridges: A review and synthesis of research on teaching knowledge for undergraduate instruction in science, engineering, and mathematics. International Journal of STEM Education, 9 (1), 66. https://doi.org/10.1186/s40594-022-00380-w

Aranda, M. L., Guzey, S. S., & Moore, T. J. (2020). Multidisciplinary discourses in an engineering design-based science curricular unit. International Journal of Technology and Design Education, 30 (3), 507–529. https://doi.org/10.1007/s10798-019-09517-5

Attard, C., Berger, N., & Mackenzie, E. (2021). The positive influence of inquiry-based learning teacher professional learning and industry partnerships on student engagement with STEM. Frontiers in Education . https://doi.org/10.3389/feduc.2021.693221

Barak, M., & Assal, M. (2018). Robotics and STEM learning: Students’ achievements in assignments according to the P3 Task Taxonomy-practice, problem solving, and projects. International Journal of Technology and Design Education, 28 (1), 121–144. https://doi.org/10.1007/s10798-016-9385-9

Beckett, G. H., Hemmings, A., Maltbie, C., Wright, K., Sherman, M., & Sersion, B. (2016). Urban high school student engagement through cincySTEM iTEST projects. Journal of Science Education and Technology, 25 (6), 995–1007. https://doi.org/10.1007/s10956-016-9640-6

Berland, L., Steingut, R., & Ko, P. (2014). High school student perceptions of the utility of the engineering design process: Creating opportunities to engage in engineering practices and apply math and science content. Journal of Science Education and Technology, 23 (6), 705–720. https://doi.org/10.1007/s10956-014-9498-4

Blumenfeld, P. C., Soloway, E., Marx, R. W., Krajcik, J. S., Guzdial, M., & Palincsar, A. (1991). Motivating project-based learning: Sustaining the doing, supporting the learning. Educational Psychologist, 26 (3–4), 369–398. https://doi.org/10.1080/00461520.1991.9653139

Boon, N. S. (2019). Exploring STEM competences for the 21st century . Current and Critical Issues in Curriculum, Learning and Assessment 30. Geneva: IBEUNESCO. Retrieved June 19, 2022, from https://unesdoc.unesco.org/ark:/48223/pf0000368485.locale=en

Breiner, J. M., Harkness, S. S., Johnson, C. C., & Koehler, C. M. (2012). What is STEM? A discussion about conceptions of STEM in education and partnerships. School Science and Mathematics, 112 (1), 3–11. https://doi.org/10.1111/j.1949-8594.2011.00109.x

Bryan, L. A., Moore, T. J., Johnson, C. C., & Roehrig, G. H. (2015). Integrated STEM education. In C. C. Johnson, E. E. Peters-Burton, & T. J. Moore (Eds.), STEM roadmap: A framework for integration (pp. 23–37). Taylor & Francis.

Chapter   Google Scholar  

Bybee, R. W. (2010). Advancing STEM education: A 2020 vision. Technology and Engineering Teacher, 70 (1), 30–35.

Google Scholar  

Bybee, R. W. (2013). The case for STEM education . NSTA press.

Capraro, R. M., Capraro, M. M., & Morgan, J. (Eds.). (2013). Project-based learning: An integrated science, technology, engineering, and mathematics (STEM) approach (2nd ed.). Sense.

Capraro, R. M., & Slough, S. W. (2013). Why PBL? Why STEM? Why now? an introduction to STEM project-based learning. In R. M. Capraro, M. M. Capraro, & J. R. Morgan (Eds.), STEM Project-based learning: An integrated Science, Technology, Engineering, and Mathematics (STEM) approach (pp. 1–5). Sense Publishers.

Carreira, S., & Baioa, A. M. (2018). Mathematical modelling with hands-on experimental tasks: On the student’s sense of credibility. ZDM-Mathematics Education, 50 (1–2), 201–215. https://doi.org/10.1007/s11858-017-0905-1

Chen, C. S., & Lin, J. W. (2019). A practical action research study of the impact of maker-centered STEM-PjBL on a rural middle school in Taiwan. International Journal of Science and Mathematics Education, 17 , S85–S108. https://doi.org/10.1007/s10763-019-09961-8

Cheng, L., Antonenko, P., Ritzhaupt, A. D., & MacFadden, B. (2021). Exploring the role of 3D printing and STEM integration levels in students’ STEM career interest. British Journal of Educational Technology, 52 (3), 1262–1278. https://doi.org/10.1111/bjet.13077

Committee on STEM Education of the National Science & Technology Council. (2018). Charting a course for success: America’s strategy for STEM education . Executive Office of the President National Science and Technology Council.

Cunningham, C. M., Lachapelle, C. P., Brennan, R. T., Kelly, G. J., Tunis, C. S. A., & Gentry, C. A. (2019). The impact of engineering curriculum design principles on elementary students’ engineering and science learning. Journal of Research in Science Teaching, 57 (3), 423–453. https://doi.org/10.1002/tea.21601

Dasgupta, C., Magana, A. J., & Vieira, C. (2019). Investigating the affordances of a CAD enabled learning environment for promoting integrated STEM learning. Computers & Education, 129 , 122–142. https://doi.org/10.1016/j.compedu.2018.10.014

Ejiwale, J. (2013). Barriers to successful implementation of STEM education. Journal of Education and Learning, 7 (2), 63–74.

English, L. D. (2016). STEM education K-12: Perspectives on integration. International Journal of STEM Education, 3 (1), 3. https://doi.org/10.1186/s40594-016-0036-1

English, L. D. (2017). Advancing elementary and middle school STEM education. International Journal of Science and Mathematics Education, 15 (1), 5–24. https://doi.org/10.1007/s10763-017-9802-x

English, L. D. (2019). Learning while designing in a fourth-grade integrated STEM problem. International Journal of Technology and Design Education, 29 (5), 1011–1032. https://doi.org/10.1007/s10798-018-9482-z

English, L. D., & King, D. (2019). STEM integration in sixth grade: Designing and constructing paper bridges. International Journal of Science and Mathematics Education, 17 (5), 863–884. https://doi.org/10.1007/s10763-018-9912-0

English, L. D., King, D., & Smeed, J. (2017). Advancing integrated STEM learning through engineering design: Sixth-grade students’ design and construction of earthquake resistant buildings. Journal of Educational Research, 110 (3), 255–271. https://doi.org/10.1080/00220671.2016.1264053

Falloon, G., Hatzigianni, M., Bower, M., Forbes, A., & Stevenson, M. (2020). Understanding K-12 STEM education: A framework for developing STEM literacy. Journal of Science Education and Technology, 29 (3), 369–385. https://doi.org/10.1007/s10956-020-09823-x

Fan, S.-C., Yu, K.-C., & Lin, K.-Y. (2021). A framework for implementing an engineering-focused stem curriculum. International Journal of Science and Mathematics Education, 19 (8), 1523–1541. https://doi.org/10.1007/s10763-020-10129-y

Fan, S.-C., Yu, K.-C., & Lou, S.-J. (2018). Why do students present different design objectives in engineering design projects? International Journal of Technology and Design Education . https://doi.org/10.1007/s10798-017-9420-5

Finegold, D., & Notabartolo, A. S. (2010). 21st century competencies and their impact: an interdisciplinary literature review. In D. Finegold, M. Gatta, H. Salzman, & S. J. Schurman (Eds.), Transforming the US workforce development system (pp. 19–56). Labor and Employment Relations Association.

Fleer, M. (2021). When preschool girls engineer: Future imaginings of being and becoming an engineer. Learning Culture and Social Interaction, 30 , 100372. https://doi.org/10.1016/j.lcsi.2019.100372

Fortus, D., Dershimer, R. C., Krajcik, J., Marx, R. W., & Mamlok-Naaman, R. (2004). Design-based science and student learning. Journal of Research in Science Teaching, 41 (10), 1081–1110. https://doi.org/10.1002/tea.2004

Gladstone, J. R., & Cimpian, A. (2021). Which role models are effective for which students? A systematic review and four recommendations for maximizing the effectiveness of role models in STEM. International Journal of STEM Education, 8 (1), 59. https://doi.org/10.1186/s40594-021-00315-x

Gough, D., Oliver, S., & Thomas, J. (2012). Introducing systematic reviews. In D. Gough, S. Oliver, & J. Thomas (Eds.), An introduction to systematic reviews (pp. 1–16). Sage.

Grangeat, M., Harrison, C., & Dolin, J. (2021). Exploring assessment in STEM inquiry learning classrooms. International Journal of Science Education, 43 (3), 345–361. https://doi.org/10.1080/09500693.2021.1903617

Guzey, S. S., Moore, T. J., Harwell, M., & Moreno, M. (2016). STEM integration in middle school life science: Student learning and attitudes. Journal of Science Education and Technology, 25 (4), 550–560. https://doi.org/10.1007/s10956-016-9612-x

Halawa, S., Hsu, Y.-S., & Zhang, W.-X. (2023). Analysis of physics textbooks through the lens of inquiry practices. The Asia-Pacific Education Researcher, 32 (4), 497–506. https://doi.org/10.1007/s40299-022-00671-4

Halawa, S., Hsu, Y.-S., & Zhang, W.-X. (2024). Inquiry activity design from Singaporean and indonesian physics textbooks. Science & Education, 33 (3), 581–607. https://doi.org/10.1007/s11191-022-00396-2

Halawa, S., Hsu, Y.-S., Zhang, W.-X., Kuo, Y.-R., & Wu, J.-Y. (2020). Features and trends of teaching strategies for scientific practices from a review of 2008–2017 articles. International Journal of Science Education, 42 (7), 1183–1206. https://doi.org/10.1080/09500693.2020.1752415

Han, S., Capraro, R., & Capraro, M. M. (2015). How science, technology, engineering, and mathematics (stem) project-based learning (pbl) affects high, middle, and low achievers differently: The impact of student factors on achievement. International Journal of Science and Mathematics Education, 13 (5), 1089–1113. https://doi.org/10.1007/s10763-014-9526-0

Henderson, C., Beach, A., & Finkelstein, N. (2011). Facilitating change in undergraduate STEM instructional practices: An analytic review of the literature. Journal of Research in Science Teaching, 48 (8), 952–984. https://doi.org/10.1002/tea.20439

Hernandez, P. R., Bodin, R., Elliott, J. W., Ibrahim, B., Rambo-Hernandez, K. E., Chen, T. W., & de Miranda, M. A. (2014). Connecting the STEM dots: Measuring the effect of an integrated engineering design intervention. International Journal of Technology and Design Education, 24 (1), 107–120. https://doi.org/10.1007/s10798-013-9241-0

Hofstein, A., Navon, O., Kipnis, M., & Mamlok-Naaman, R. (2005). Developing students’ ability to ask more and better questions resulting from inquiry-type chemistry laboratories. Journal of Research in Science Teaching, 42 (7), 791–806. https://doi.org/10.1002/tea.20072

Honey, M., Pearson, G., & Schweingruber, H. (Eds.). (2014). STEM integration in K-12 education: Status, prospects and an agenda for research . National Academies Press.

Hsu, Y.-S., & Fang, S.-C. (2019). Opportunities and challenges of STEM education. In Y.-S. Hsu & Y.-F. Yeh (Eds.), Asia-Pacific STEM teaching practices: From theoretical frameworks to practices (pp. 1–16). Springer Singapore.

Jackson, C., Mohr-Schroeder, M. J., Bush, S. B., Maiorca, C., Roberts, T., Yost, C., & Fowler, A. (2021). Equity-oriented conceptual framework for k-12 STEM literacy. International Journal of STEM Education, 8 (1), 38. https://doi.org/10.1186/s40594-021-00294-z

Jang, H. (2016). Identifying 21st century STEM competencies using workplace data. Journal of Science Education and Technology, 25 (2), 284–301. https://doi.org/10.1007/s10956-015-9593-1

Johnson, C. C., Peters-Burton, E. E., & Moore, T. J. (2015). STEM road map: A framework for integrated STEM Education . Routledge.

Book   Google Scholar  

Kang, J., Salonen, A., Tolppanen, S., Scheersoi, A., Hense, J., Rannikmae, M., & Keinonen, T. (2021). Effect of embedded careers education in science lessons on students’ interest, awareness, and aspirations. International Journal of Science and Mathematics Education . https://doi.org/10.1007/s10763-021-10238-2

Kayan-Fadlelmula, F., Sellami, A., Abdelkader, N., & Umer, S. (2022). A systematic review of STEM education research in the GCC countries: Trends, gaps and barriers. International Journal of STEM Education, 9 (1), 2. https://doi.org/10.1186/s40594-021-00319-7

Kelley, T. R., Brenner, D. C., & Pieper, J. T. (2010). Two approaches to engineering design: Observations in STEM education. Journal of STEM Teacher Education, 47 (2), 5–40.

Kelley, T. R., & Knowles, J. G. (2016). A conceptual framework for integrated STEM education. International Journal of STEM Education, 3 (1), 11. https://doi.org/10.1186/s40594-016-0046-z

Kim, H. (2016). Inquiry-based science and technology enrichment program for middle school-aged female students. Journal of Science Education and Technology, 25 (2), 174–186. https://doi.org/10.1007/s10956-015-9584-2

Krajcik, J., Blumenfeld, P. C., Marx, R. W., Bass, K. M., Fredricks, J., & Soloway, E. (1998). Inquiry in project-based science classrooms: Initial attempts by middle school students. Journal of the Learning Sciences, 7 (3–4), 313–350. https://doi.org/10.1080/10508406.1998.9672057

Lamb, R., Akmal, T., & Petrie, K. (2015). Development of a cognition-priming model describing learning in a STEM classroom. Journal of Research in Science Teaching, 52 (3), 410–437. https://doi.org/10.1002/tea.21200

Lederman, N. G., & Lederman, J. S., et al. (2012). Nature of scientific knowledge and scientific inquiry: Building instructional capacity through professional development. In B. J. Fraser (Ed.), Second international handbook of science education (pp. 335–359). Springer.

Lie, R., Guzey, S. S., & Moore, T. J. (2019). Implementing engineering in diverse upper elementary and middle school science classrooms: Student learning and attitudes. Journal of Science Education and Technology, 28 (2), 104–117. https://doi.org/10.1007/s10956-018-9751-3

Lin, K.-Y., Hsiao, H.-S., Williams, P. J., & Chen, Y.-H. (2020). Effects of 6E-oriented STEM practical activities in cultivating middle school students’ attitudes toward technology and technological inquiry ability. Research in Science & Technological Education, 38 (1), 1–18. https://doi.org/10.1080/02635143.2018.1561432

López, N., Morgan, D. L., Hutchings, Q. R., & Davis, K. (2022). Revisiting critical STEM interventions: A literature review of STEM organizational learning. International Journal of STEM Education, 9 (1), 39. https://doi.org/10.1186/s40594-022-00357-9

Maltese, A. V., Melki, C. S., & Wiebke, H. L. (2014). The nature of experiences responsible for the generation and maintenance of interest in STEM. Science Education, 98 (6), 937–962. https://doi.org/10.1002/sce.21132

Maltese, A. V., & Tai, R. H. (2011). Pipeline persistence: Examining the association of educational experiences with earned degrees in STEM among US students. Science Education, 95 (5), 877–907. https://doi.org/10.1002/sce.20441

Margot, K. C., & Kettler, T. (2019). Teachers’ perception of STEM integration and education: A systematic literature review. International Journal of STEM Education, 6 (1), 2. https://doi.org/10.1186/s40594-018-0151-2

Martín-Páez, T., Aguilera, D., Perales-Palacios, F. J., & Vílchez-González, J. M. (2019). What are we talking about when we talk about STEM education? A Review of Literature. Science Education, 103 (4), 799–822. https://doi.org/10.1002/sce.21522

McHugh, M. L. (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22 , 276–282.

McLure, F. I., Tang, K.-S., & Williams, P. J. (2022). What do integrated STEM projects look like in middle school and high school classrooms? A systematic literature review of empirical studies of iSTEM projects. International Journal of STEM Education, 9 (1), 73. https://doi.org/10.1186/s40594-022-00390-8

Means, B., Wang, H., Wei, X., Lynch, S., Peters, V., Young, V., & Allen, C. (2017). Expanding STEM opportunities through inclusive STEM-focused high schools. Science Education, 101 (5), 681–715. https://doi.org/10.1002/sce.21281

Mehalik, M. M., Doppelt, Y., & Schuun, C. D. (2008). Middle-school science through design-based learning versus scripted inquiry: Better overall science concept learning and equity gap reduction. Journal of Engineering Education, 97 (1), 71–85. https://doi.org/10.1002/j.2168-9830.2008.tb00955.x

Møller, A. M., & Myles, P. S. (2016). What makes a good systematic review and meta-analysis? BJA British Journal of Anaesthesia, 117 (4), 428–430. https://doi.org/10.1093/bja/aew264

National Academy of Engineering and National Research Council. (2014). STEM integration in K-12 education: Status, prospects, and an agenda for research . National Academies Press.

National Research Council. (2011). Successful K-12 STEM education: Identifying effective approaches in science, technology, engineering, and mathematics . National Academies Press.

National Research Council. (2012). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas . The National Academies Press.

National Research Council. (2013). Monitoring progress toward successful K-12 STEM education: A nation advancing? National Academies Press. https://doi.org/10.17226/13509

Nguyen, K. A., Borrego, M., Finelli, C. J., DeMonbrun, M., Crockett, C., Tharayil, S., Shekhar, P., Waters, C., & Rosenberg, R. (2021). Instructor strategies to aid implementation of active learning: A systematic literature review. International Journal of STEM Education, 8 (1), 9. https://doi.org/10.1186/s40594-021-00270-7

Preininger, A. M. (2017). Embedded mathematics in chemistry: A case study of students’ attitudes and mastery. Journal of Science Education and Technology, 26 (1), 58–69. https://doi.org/10.1007/s10956-016-9651-3

Priemer, B., Eilerts, K., Filler, A., Pinkwart, N., Rösken-Winter, B., Tiemann, R., & Zu Belzen, A. U. (2020). A framework to foster problem-solving in STEM and computing education. Research in Science & Technological Education, 38 (1), 105–130. https://doi.org/10.1080/02635143.2019.1600490

Sadler, P. M., Coyle, H. P., & Schwartz, M. (2000). Engineering competitions in the middle school classroom: Key elements in developing effective design challenges. Journal of the Learning Sciences, 9 (3), 299–327. https://doi.org/10.1207/S15327809JLS0903_3

Sanders, M. (2009). STEM, STEM education, STEMmania. The Technology Teacher, 68 (4), 20–26.

Sanders, M. E., & Wells, J. (2006). Integrative STEM education course syllabi & instructional materials: STEM education foundations . In STEM Education Trends & Issues, STEM Education Seminar.

Schütte, K., & Köller, O. (2015). ‘Discover, understand, implement, and transfer’: Effectiveness of an intervention programme to motivate students for science. International Journal of Science Education, 37 (14), 2306–2325. https://doi.org/10.1080/09500693.2015.1077537

Shaughnessy, J. M. (2013). Mathematics in a STEM context. Mathematics Teaching in the Middle School, 18 (6), 324. https://doi.org/10.5951/mathteacmiddscho.18.6.0324

Sinatra, G. M., Mukhopadhyay, A., Allbright, T. N., Marsh, J. A., & Polikoff, M. S. (2017). Speedometry: A vehicle for promoting interest and engagement through integrated STEM instruction. Journal of Educational Research, 110 (3), 308–316. https://doi.org/10.1080/00220671.2016.1273178

Smyrnaiou, Z., Georgakopoulou, E., & Sotiriou, S. (2020). Promoting a mixed-design model of scientific creativity through digital storytelling—the CCQ model for creativity. International Journal of STEM Education, 7 (1), 25. https://doi.org/10.1186/s40594-020-00223-6

Stehle, S. M., & Peters-Burton, E. E. (2019). Developing student 21st Century skills in selected exemplary inclusive STEM high schools. International Journal of STEM Education, 6 (1), 39. https://doi.org/10.1186/s40594-019-0192-1

Strobel, J., Wang, J., Weber, N. R., & Dyehouse, M. (2013). The role of authenticity in design-based learning environments: The case of engineering education. Computers & Education, 64 , 143–152. https://doi.org/10.1016/j.compedu.2012.11.026

Struyf, A., De Loof, H., Boeve-de Pauw, J., & Van Petegem, P. (2019). Students’ engagement in different STEM learning environments: Integrated STEM education as promising practice? International Journal of Science Education, 41 (10), 1387–1407. https://doi.org/10.1080/09500693.2019.1607983

Sullivan, A., & Bers, M. U. (2019). Investigating the use of robotics to increase girls’ interest in engineering during early elementary school. International Journal of Technology and Design Education, 29 (5), 1033–1051. https://doi.org/10.1007/s10798-018-9483-y

Thomas, D. R., & Larwin, K. H. (2023). A meta-analytic investigation of the impact of middle school STEM education: Where are all the students of color? International Journal of STEM Education, 10 (1), 43. https://doi.org/10.1186/s40594-023-00425-8

U.S. Department of Education. (2016). STEM 2026: A vision for innovation in STEM education . U.S. Department of Education.

Vallera, F. L., & Bodzin, A. M. (2020). Integrating STEM with AgLIT (Agricultural literacy through innovative technology): The efficacy of a project-based curriculum for upper-primary students. International Journal of Science and Mathematics Education, 18 (3), 419–439. https://doi.org/10.1007/s10763-019-09979-y

Vongkulluksn, V. W., Matewos, A. M., Sinatra, G. M., & Marsh, J. A. (2018). Motivational factors in makerspaces: A mixed methods study of elementary school students’ situational interest, self-efficacy, and achievement emotions. International Journal of STEM Education, 5 (1), 43. https://doi.org/10.1186/s40594-018-0129-0

Vossen, T. E., Henze, I., Rippe, R. C. A., Van Driel, J. H., & De Vries, M. J. (2018). Attitudes of secondary school students towards doing research and design activities. International Journal of Science Education, 40 (13), 1629–1652. https://doi.org/10.1080/09500693.2018.1494395

Wilson, K. (2021). Exploring the challenges and enablers of implementing a STEM project-based learning programme in a diverse junior secondary context. International Journal of Science and Mathematics Education, 19 (5), 881–897. https://doi.org/10.1007/s10763-020-10103-8

Wu, Y., Guo, S., & Zhu, L. (2019). Design and implementation of data collection mechanism for 3D design course based on xAPI standard. Interactive Learning Environments, 28 (5), 602–619. https://doi.org/10.1080/10494820.2019.1696842

Yata, C., Ohtani, T., & Isobe, M. (2020). Conceptual framework of STEM based on Japanese subject principles. International Journal of STEM Education, 7 (1), 12. https://doi.org/10.1186/s40594-020-00205-8

Zheng, J., Xing, W., Zhu, G., Chen, G., Zhao, H., & Xie, C. (2020). Profiling self-regulation behaviors in STEM learning of engineering design. Computers & Education . https://doi.org/10.1016/j.compedu.2019.103669

Download references

Acknowledgements

The authors express their sincere gratitude to the editors and reviewers for their invaluable inputs and suggestions, which have significantly enhanced the quality of this work.

This work was financially supported by the Institute for Research Excellence in Learning Sciences of National Taiwan Normal University (NTNU) from The Featured Areas Research Center Program within the framework of the Higher Education Sprout Project by the Ministry of Education (MOE) in Taiwan.

Author information

Authors and affiliations.

Research Center for Education, National Research and Innovation Agency, Jakarta, Indonesia

Suarman Halawa

Center for Teacher Education, National Kaohsiung University of Science and Technology, Kaohsiung, Taiwan

Tzu-Chiang Lin

Center for the Liberal Arts, National Kaohsiung University of Science and Technology, Kaohsiung, Taiwan

Graduate Institute of Science Education, National Taiwan Normal University, Taipei, Taiwan

Ying-Shao Hsu

You can also search for this author in PubMed   Google Scholar

Contributions

SH contributed to the conception of the study, research question, methods, analysis, and interpretation of the data. TC contributed to the data collection, analysis and interpretation of data, and editing of the manuscript. YS contributed to the conception of the study, data analysis and interpretation, and editing of the manuscript. All authors equally contributed to writing, reading, and approving the manuscript.

Corresponding author

Correspondence to Ying-Shao Hsu .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The author declare that they have no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Description of STEM education goals

Description of the elements of instructional design for STEM education

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Halawa, S., Lin, TC. & Hsu, YS. Exploring instructional design in K-12 STEM education: a systematic literature review. IJ STEM Ed 11 , 43 (2024). https://doi.org/10.1186/s40594-024-00503-5

Download citation

Received : 15 April 2024

Accepted : 24 August 2024

Published : 05 September 2024

DOI : https://doi.org/10.1186/s40594-024-00503-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• STEM education
• Engagement and career choice
• STEM literacy

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List

How-to conduct a systematic literature review: A quick guide for computer science research

Angela carrera-rivera.

a Faculty of Engineering, Mondragon University

William Ochoa

Felix larrinaga.

b Design Innovation Center(DBZ), Mondragon University

Associated Data

• No data was used for the research described in the article.

Performing a literature review is a critical first step in research to understanding the state-of-the-art and identifying gaps and challenges in the field. A systematic literature review is a method which sets out a series of steps to methodically organize the review. In this paper, we present a guide designed for researchers and in particular early-stage researchers in the computer-science field. The contribution of the article is the following:

• • Clearly defined strategies to follow for a systematic literature review in computer science research, and
• • Algorithmic method to tackle a systematic literature review.

Graphical abstract

Specifications table

Method details

A Systematic Literature Review (SLR) is a research methodology to collect, identify, and critically analyze the available research studies (e.g., articles, conference proceedings, books, dissertations) through a systematic procedure [12] . An SLR updates the reader with current literature about a subject [6] . The goal is to review critical points of current knowledge on a topic about research questions to suggest areas for further examination [5] . Defining an “Initial Idea” or interest in a subject to be studied is the first step before starting the SLR. An early search of the relevant literature can help determine whether the topic is too broad to adequately cover in the time frame and whether it is necessary to narrow the focus. Reading some articles can assist in setting the direction for a formal review., and formulating a potential research question (e.g., how is semantics involved in Industry 4.0?) can further facilitate this process. Once the focus has been established, an SLR can be undertaken to find more specific studies related to the variables in this question. Although there are multiple approaches for performing an SLR ( [5] , [26] , [27] ), this work aims to provide a step-by-step and practical guide while citing useful examples for computer-science research. The methodology presented in this paper comprises two main phases: “Planning” described in section 2, and “Conducting” described in section 3, following the depiction of the graphical abstract.

Defining the protocol is the first step of an SLR since it describes the procedures involved in the review and acts as a log of the activities to be performed. Obtaining opinions from peers while developing the protocol, is encouraged to ensure the review's consistency and validity, and helps identify when modifications are necessary [20] . One final goal of the protocol is to ensure the replicability of the review.

Define PICOC and synonyms

The PICOC (Population, Intervention, Comparison, Outcome, and Context) criteria break down the SLR's objectives into searchable keywords and help formulate research questions [ 27 ]. PICOC is widely used in the medical and social sciences fields to encourage researchers to consider the components of the research questions [14] . Kitchenham & Charters [6] compiled the list of PICOC elements and their corresponding terms in computer science, as presented in Table 1 , which includes keywords derived from the PICOC elements. From that point on, it is essential to think of synonyms or “alike” terms that later can be used for building queries in the selected digital libraries. For instance, the keyword “context awareness” can also be linked to “context-aware”.

Planning Step 1 “Defining PICOC keywords and synonyms”.

Formulate research questions

Clearly defined research question(s) are the key elements which set the focus for study identification and data extraction [21] . These questions are formulated based on the PICOC criteria as presented in the example in Table 2 (PICOC keywords are underlined).

Research questions examples.

Select digital library sources

The validity of a study will depend on the proper selection of a database since it must adequately cover the area under investigation [19] . The Web of Science (WoS) is an international and multidisciplinary tool for accessing literature in science, technology, biomedicine, and other disciplines. Scopus is a database that today indexes 40,562 peer-reviewed journals, compared to 24,831 for WoS. Thus, Scopus is currently the largest existing multidisciplinary database. However, it may also be necessary to include sources relevant to computer science, such as EI Compendex, IEEE Xplore, and ACM. Table 3 compares the area of expertise of a selection of databases.

Planning Step 3 “Select digital libraries”. Description of digital libraries in computer science and software engineering.

Define inclusion and exclusion criteria

Authors should define the inclusion and exclusion criteria before conducting the review to prevent bias, although these can be adjusted later, if necessary. The selection of primary studies will depend on these criteria. Articles are included or excluded in this first selection based on abstract and primary bibliographic data. When unsure, the article is skimmed to further decide the relevance for the review. Table 4 sets out some criteria types with descriptions and examples.

Planning Step 4 “Define inclusion and exclusion criteria”. Examples of criteria type.

Define the Quality Assessment (QA) checklist

Assessing the quality of an article requires an artifact which describes how to perform a detailed assessment. A typical quality assessment is a checklist that contains multiple factors to evaluate. A numerical scale is used to assess the criteria and quantify the QA [22] . Zhou et al. [25] presented a detailed description of assessment criteria in software engineering, classified into four main aspects of study quality: Reporting, Rigor, Credibility, and Relevance. Each of these criteria can be evaluated using, for instance, a Likert-type scale [17] , as shown in Table 5 . It is essential to select the same scale for all criteria established on the quality assessment.

Planning Step 5 “Define QA assessment checklist”. Examples of QA scales and questions.

Define the “Data Extraction” form

The data extraction form represents the information necessary to answer the research questions established for the review. Synthesizing the articles is a crucial step when conducting research. Ramesh et al. [15] presented a classification scheme for computer science research, based on topics, research methods, and levels of analysis that can be used to categorize the articles selected. Classification methods and fields to consider when conducting a review are presented in Table 6 .

Planning Step 6 “Define data extraction form”. Examples of fields.

The data extraction must be relevant to the research questions, and the relationship to each of the questions should be included in the form. Kitchenham & Charters [6] presented more pertinent data that can be captured, such as conclusions, recommendations, strengths, and weaknesses. Although the data extraction form can be updated if more information is needed, this should be treated with caution since it can be time-consuming. It can therefore be helpful to first have a general background in the research topic to determine better data extraction criteria.

After defining the protocol, conducting the review requires following each of the steps previously described. Using tools can help simplify the performance of this task. Standard tools such as Excel or Google sheets allow multiple researchers to work collaboratively. Another online tool specifically designed for performing SLRs is Parsif.al 1 . This tool allows researchers, especially in the context of software engineering, to define goals and objectives, import articles using BibTeX files, eliminate duplicates, define selection criteria, and generate reports.

Build digital library search strings

Search strings are built considering the PICOC elements and synonyms to execute the search in each database library. A search string should separate the synonyms with the boolean operator OR. In comparison, the PICOC elements are separated with parentheses and the boolean operator AND. An example is presented next:

(“Smart Manufacturing” OR “Digital Manufacturing” OR “Smart Factory”) AND (“Business Process Management” OR “BPEL” OR “BPM” OR “BPMN”) AND (“Semantic Web” OR “Ontology” OR “Semantic” OR “Semantic Web Service”) AND (“Framework” OR “Extension” OR “Plugin” OR “Tool”

Gather studies

Databases that feature advanced searches enable researchers to perform search queries based on titles, abstracts, and keywords, as well as for years or areas of research. Fig. 1 presents the example of an advanced search in Scopus, using titles, abstracts, and keywords (TITLE-ABS-KEY). Most of the databases allow the use of logical operators (i.e., AND, OR). In the example, the search is for “BIG DATA” and “USER EXPERIENCE” or “UX” as a synonym.

Example of Advanced search on Scopus.

In general, bibliometric data of articles can be exported from the databases as a comma-separated-value file (CSV) or BibTeX file, which is helpful for data extraction and quantitative and qualitative analysis. In addition, researchers should take advantage of reference-management software such as Zotero, Mendeley, Endnote, or Jabref, which import bibliographic information onto the software easily.

Study Selection and Refinement

The first step in this stage is to identify any duplicates that appear in the different searches in the selected databases. Some automatic procedures, tools like Excel formulas, or programming languages (i.e., Python) can be convenient here.

In the second step, articles are included or excluded according to the selection criteria, mainly by reading titles and abstracts. Finally, the quality is assessed using the predefined scale. Fig. 2 shows an example of an article QA evaluation in Parsif.al, using a simple scale. In this scenario, the scoring procedure is the following YES= 1, PARTIALLY= 0.5, and NO or UNKNOWN = 0 . A cut-off score should be defined to filter those articles that do not pass the QA. The QA will require a light review of the full text of the article.

Performing quality assessment (QA) in Parsif.al.

Data extraction

Those articles that pass the study selection are then thoroughly and critically read. Next, the researcher completes the information required using the “data extraction” form, as illustrated in Fig. 3 , in this scenario using Parsif.al tool.

Example of data extraction form using Parsif.al.

The information required (study characteristics and findings) from each included study must be acquired and documented through careful reading. Data extraction is valuable, especially if the data requires manipulation or assumptions and inferences. Thus, information can be synthesized from the extracted data for qualitative or quantitative analysis [16] . This documentation supports clarity, precise reporting, and the ability to scrutinize and replicate the examination.

Analysis and Report

The analysis phase examines the synthesized data and extracts meaningful information from the selected articles [10] . There are two main goals in this phase.

The first goal is to analyze the literature in terms of leading authors, journals, countries, and organizations. Furthermore, it helps identify correlations among topic s . Even when not mandatory, this activity can be constructive for researchers to position their work, find trends, and find collaboration opportunities. Next, data from the selected articles can be analyzed using bibliometric analysis (BA). BA summarizes large amounts of bibliometric data to present the state of intellectual structure and emerging trends in a topic or field of research [4] . Table 7 sets out some of the most common bibliometric analysis representations.

Techniques for bibliometric analysis and examples.

Several tools can perform this type of analysis, such as Excel and Google Sheets for statistical graphs or using programming languages such as Python that has available multiple  data visualization libraries (i.e. Matplotlib, Seaborn). Cluster maps based on bibliographic data(i.e keywords, authors) can be developed in VosViewer which makes it easy to identify clusters of related items [18] . In Fig. 4 , node size is representative of the number of papers related to the keyword, and lines represent the links among keyword terms.

[1] Keyword co-relationship analysis using clusterization in vos viewer.

This second and most important goal is to answer the formulated research questions, which should include a quantitative and qualitative analysis. The quantitative analysis can make use of data categorized, labelled, or coded in the extraction form (see Section 1.6). This data can be transformed into numerical values to perform statistical analysis. One of the most widely employed method is frequency analysis, which shows the recurrence of an event, and can also represent the percental distribution of the population (i.e., percentage by technology type, frequency of use of different frameworks, etc.). Q ualitative analysis includes the narration of the results, the discussion indicating the way forward in future research work, and inferring a conclusion.

Finally, the literature review report should state the protocol to ensure others researchers can replicate the process and understand how the analysis was performed. In the protocol, it is essential to present the inclusion and exclusion criteria, quality assessment, and rationality beyond these aspects.

The presentation and reporting of results will depend on the structure of the review given by the researchers conducting the SLR, there is no one answer. This structure should tie the studies together into key themes, characteristics, or subgroups [ 28 ].

SLR can be an extensive and demanding task, however the results are beneficial in providing a comprehensive overview of the available evidence on a given topic. For this reason, researchers should keep in mind that the entire process of the SLR is tailored to answer the research question(s). This article has detailed a practical guide with the essential steps to conducting an SLR in the context of computer science and software engineering while citing multiple helpful examples and tools. It is envisaged that this method will assist researchers, and particularly early-stage researchers, in following an algorithmic approach to fulfill this task. Finally, a quick checklist is presented in Appendix A as a companion of this article.

CRediT author statement

Angela Carrera-Rivera: Conceptualization, Methodology, Writing-Original. William Ochoa-Agurto : Methodology, Writing-Original. Felix Larrinaga : Reviewing and Supervision Ganix Lasa: Reviewing and Supervision.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

Funding : This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie Grant No. 814078.

Carrera-Rivera, A., Larrinaga, F., & Lasa, G. (2022). Context-awareness for the design of Smart-product service systems: Literature review. Computers in Industry, 142, 103730.

1 https://parsif.al/

Data Availability