how to write methodology in literature review

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Writing a Literature Review

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

A literature review is a document or section of a document that collects key sources on a topic and discusses those sources in conversation with each other (also called synthesis ). The lit review is an important genre in many disciplines, not just literature (i.e., the study of works of literature such as novels and plays). When we say “literature review” or refer to “the literature,” we are talking about the research ( scholarship ) in a given field. You will often see the terms “the research,” “the scholarship,” and “the literature” used mostly interchangeably.

Where, when, and why would I write a lit review?

There are a number of different situations where you might write a literature review, each with slightly different expectations; different disciplines, too, have field-specific expectations for what a literature review is and does. For instance, in the humanities, authors might include more overt argumentation and interpretation of source material in their literature reviews, whereas in the sciences, authors are more likely to report study designs and results in their literature reviews; these differences reflect these disciplines’ purposes and conventions in scholarship. You should always look at examples from your own discipline and talk to professors or mentors in your field to be sure you understand your discipline’s conventions, for literature reviews as well as for any other genre.

A literature review can be a part of a research paper or scholarly article, usually falling after the introduction and before the research methods sections. In these cases, the lit review just needs to cover scholarship that is important to the issue you are writing about; sometimes it will also cover key sources that informed your research methodology.

Lit reviews can also be standalone pieces, either as assignments in a class or as publications. In a class, a lit review may be assigned to help students familiarize themselves with a topic and with scholarship in their field, get an idea of the other researchers working on the topic they’re interested in, find gaps in existing research in order to propose new projects, and/or develop a theoretical framework and methodology for later research. As a publication, a lit review usually is meant to help make other scholars’ lives easier by collecting and summarizing, synthesizing, and analyzing existing research on a topic. This can be especially helpful for students or scholars getting into a new research area, or for directing an entire community of scholars toward questions that have not yet been answered.

What are the parts of a lit review?

Most lit reviews use a basic introduction-body-conclusion structure; if your lit review is part of a larger paper, the introduction and conclusion pieces may be just a few sentences while you focus most of your attention on the body. If your lit review is a standalone piece, the introduction and conclusion take up more space and give you a place to discuss your goals, research methods, and conclusions separately from where you discuss the literature itself.

Introduction:

• An introductory paragraph that explains what your working topic and thesis is
• A forecast of key topics or texts that will appear in the review
• Potentially, a description of how you found sources and how you analyzed them for inclusion and discussion in the review (more often found in published, standalone literature reviews than in lit review sections in an article or research paper)
• Summarize and synthesize: Give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: Don’t just paraphrase other researchers – add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically Evaluate: Mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: Use transition words and topic sentence to draw connections, comparisons, and contrasts.

Conclusion:

• Summarize the key findings you have taken from the literature and emphasize their significance
• Connect it back to your primary research question

How should I organize my lit review?

Lit reviews can take many different organizational patterns depending on what you are trying to accomplish with the review. Here are some examples:

• Chronological : The simplest approach is to trace the development of the topic over time, which helps familiarize the audience with the topic (for instance if you are introducing something that is not commonly known in your field). If you choose this strategy, be careful to avoid simply listing and summarizing sources in order. Try to analyze the patterns, turning points, and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred (as mentioned previously, this may not be appropriate in your discipline — check with a teacher or mentor if you’re unsure).
• Thematic : If you have found some recurring central themes that you will continue working with throughout your piece, you can organize your literature review into subsections that address different aspects of the topic. For example, if you are reviewing literature about women and religion, key themes can include the role of women in churches and the religious attitude towards women.
• Qualitative versus quantitative research
• Empirical versus theoretical scholarship
• Divide the research by sociological, historical, or cultural sources
• Theoretical : In many humanities articles, the literature review is the foundation for the theoretical framework. You can use it to discuss various theories, models, and definitions of key concepts. You can argue for the relevance of a specific theoretical approach or combine various theorical concepts to create a framework for your research.

What are some strategies or tips I can use while writing my lit review?

Any lit review is only as good as the research it discusses; make sure your sources are well-chosen and your research is thorough. Don’t be afraid to do more research if you discover a new thread as you’re writing. More info on the research process is available in our "Conducting Research" resources .

As you’re doing your research, create an annotated bibliography ( see our page on the this type of document ). Much of the information used in an annotated bibliography can be used also in a literature review, so you’ll be not only partially drafting your lit review as you research, but also developing your sense of the larger conversation going on among scholars, professionals, and any other stakeholders in your topic.

Usually you will need to synthesize research rather than just summarizing it. This means drawing connections between sources to create a picture of the scholarly conversation on a topic over time. Many student writers struggle to synthesize because they feel they don’t have anything to add to the scholars they are citing; here are some strategies to help you:

• It often helps to remember that the point of these kinds of syntheses is to show your readers how you understand your research, to help them read the rest of your paper.
• Writing teachers often say synthesis is like hosting a dinner party: imagine all your sources are together in a room, discussing your topic. What are they saying to each other?
• Look at the in-text citations in each paragraph. Are you citing just one source for each paragraph? This usually indicates summary only. When you have multiple sources cited in a paragraph, you are more likely to be synthesizing them (not always, but often
• Read more about synthesis here.

The most interesting literature reviews are often written as arguments (again, as mentioned at the beginning of the page, this is discipline-specific and doesn’t work for all situations). Often, the literature review is where you can establish your research as filling a particular gap or as relevant in a particular way. You have some chance to do this in your introduction in an article, but the literature review section gives a more extended opportunity to establish the conversation in the way you would like your readers to see it. You can choose the intellectual lineage you would like to be part of and whose definitions matter most to your thinking (mostly humanities-specific, but this goes for sciences as well). In addressing these points, you argue for your place in the conversation, which tends to make the lit review more compelling than a simple reporting of other sources.

Methodological Approaches to Literature Review

• Living reference work entry
• First Online: 09 May 2023
• Cite this living reference work entry

• Dennis Thomas 2 ,
• Elida Zairina 3 &
• Johnson George 4  

566 Accesses

1 Citations

The literature review can serve various functions in the contexts of education and research. It aids in identifying knowledge gaps, informing research methodology, and developing a theoretical framework during the planning stages of a research study or project, as well as reporting of review findings in the context of the existing literature. This chapter discusses the methodological approaches to conducting a literature review and offers an overview of different types of reviews. There are various types of reviews, including narrative reviews, scoping reviews, and systematic reviews with reporting strategies such as meta-analysis and meta-synthesis. Review authors should consider the scope of the literature review when selecting a type and method. Being focused is essential for a successful review; however, this must be balanced against the relevance of the review to a broad audience.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Institutional subscriptions

Akobeng AK. Principles of evidence based medicine. Arch Dis Child. 2005;90(8):837–40.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Alharbi A, Stevenson M. Refining Boolean queries to identify relevant studies for systematic review updates. J Am Med Inform Assoc. 2020;27(11):1658–66.

Article   PubMed   PubMed Central   Google Scholar  

Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32.

Article   Google Scholar  

Aromataris E MZE. JBI manual for evidence synthesis. 2020.

Google Scholar  

Aromataris E, Pearson A. The systematic review: an overview. Am J Nurs. 2014;114(3):53–8.

Article   PubMed   Google Scholar  

Aromataris E, Riitano D. Constructing a search strategy and searching for evidence. A guide to the literature search for a systematic review. Am J Nurs. 2014;114(5):49–56.

Babineau J. Product review: covidence (systematic review software). J Canad Health Libr Assoc Canada. 2014;35(2):68–71.

Baker JD. The purpose, process, and methods of writing a literature review. AORN J. 2016;103(3):265–9.

Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7(9):e1000326.

Bramer WM, Rethlefsen ML, Kleijnen J, Franco OH. Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study. Syst Rev. 2017;6(1):1–12.

Brown D. A review of the PubMed PICO tool: using evidence-based practice in health education. Health Promot Pract. 2020;21(4):496–8.

Cargo M, Harris J, Pantoja T, et al. Cochrane qualitative and implementation methods group guidance series – paper 4: methods for assessing evidence on intervention implementation. J Clin Epidemiol. 2018;97:59–69.

Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med. 1997;126(5):376–80.

Article   CAS   PubMed   Google Scholar  

Counsell C. Formulating questions and locating primary studies for inclusion in systematic reviews. Ann Intern Med. 1997;127(5):380–7.

Cummings SR, Browner WS, Hulley SB. Conceiving the research question and developing the study plan. In: Cummings SR, Browner WS, Hulley SB, editors. Designing Clinical Research: An Epidemiological Approach. 4th ed. Philadelphia (PA): P Lippincott Williams & Wilkins; 2007. p. 14–22.

Eriksen MB, Frandsen TF. The impact of patient, intervention, comparison, outcome (PICO) as a search strategy tool on literature search quality: a systematic review. JMLA. 2018;106(4):420.

Ferrari R. Writing narrative style literature reviews. Medical Writing. 2015;24(4):230–5.

Flemming K, Booth A, Hannes K, Cargo M, Noyes J. Cochrane qualitative and implementation methods group guidance series – paper 6: reporting guidelines for qualitative, implementation, and process evaluation evidence syntheses. J Clin Epidemiol. 2018;97:79–85.

Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Inf Libr J. 2009;26(2):91–108.

Green BN, Johnson CD, Adams A. Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. J Chiropr Med. 2006;5(3):101–17.

Gregory AT, Denniss AR. An introduction to writing narrative and systematic reviews; tasks, tips and traps for aspiring authors. Heart Lung Circ. 2018;27(7):893–8.

Harden A, Thomas J, Cargo M, et al. Cochrane qualitative and implementation methods group guidance series – paper 5: methods for integrating qualitative and implementation evidence within intervention effectiveness reviews. J Clin Epidemiol. 2018;97:70–8.

Harris JL, Booth A, Cargo M, et al. Cochrane qualitative and implementation methods group guidance series – paper 2: methods for question formulation, searching, and protocol development for qualitative evidence synthesis. J Clin Epidemiol. 2018;97:39–48.

Higgins J, Thomas J. In: Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.3, updated February 2022). Available from www.training.cochrane.org/handbook.: Cochrane; 2022.

International prospective register of systematic reviews (PROSPERO). Available from https://www.crd.york.ac.uk/prospero/ .

Khan KS, Kunz R, Kleijnen J, Antes G. Five steps to conducting a systematic review. J R Soc Med. 2003;96(3):118–21.

Landhuis E. Scientific literature: information overload. Nature. 2016;535(7612):457–8.

Lockwood C, Porritt K, Munn Z, Rittenmeyer L, Salmond S, Bjerrum M, Loveday H, Carrier J, Stannard D. Chapter 2: Systematic reviews of qualitative evidence. In: Aromataris E, Munn Z, editors. JBI Manual for Evidence Synthesis. JBI; 2020. Available from https://synthesismanual.jbi.global . https://doi.org/10.46658/JBIMES-20-03 .

Chapter   Google Scholar  

Lorenzetti DL, Topfer L-A, Dennett L, Clement F. Value of databases other than medline for rapid health technology assessments. Int J Technol Assess Health Care. 2014;30(2):173–8.

Moher D, Liberati A, Tetzlaff J, Altman DG, the PRISMA Group. Preferred reporting items for (SR) and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;6:264–9.

Mulrow CD. Systematic reviews: rationale for systematic reviews. BMJ. 1994;309(6954):597–9.

Munn Z, Peters MDJ, Stern C, Tufanaru C, McArthur A, Aromataris E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med Res Methodol. 2018;18(1):143.

Munthe-Kaas HM, Glenton C, Booth A, Noyes J, Lewin S. Systematic mapping of existing tools to appraise methodological strengths and limitations of qualitative research: first stage in the development of the CAMELOT tool. BMC Med Res Methodol. 2019;19(1):1–13.

Murphy CM. Writing an effective review article. J Med Toxicol. 2012;8(2):89–90.

NHMRC. Guidelines for guidelines: assessing risk of bias. Available at https://nhmrc.gov.au/guidelinesforguidelines/develop/assessing-risk-bias . Last published 29 August 2019. Accessed 29 Aug 2022.

Noyes J, Booth A, Cargo M, et al. Cochrane qualitative and implementation methods group guidance series – paper 1: introduction. J Clin Epidemiol. 2018b;97:35–8.

Noyes J, Booth A, Flemming K, et al. Cochrane qualitative and implementation methods group guidance series – paper 3: methods for assessing methodological limitations, data extraction and synthesis, and confidence in synthesized qualitative findings. J Clin Epidemiol. 2018a;97:49–58.

Noyes J, Booth A, Moore G, Flemming K, Tunçalp Ö, Shakibazadeh E. Synthesising quantitative and qualitative evidence to inform guidelines on complex interventions: clarifying the purposes, designs and outlining some methods. BMJ Glob Health. 2019;4(Suppl 1):e000893.

Peters MD, Godfrey CM, Khalil H, McInerney P, Parker D, Soares CB. Guidance for conducting systematic scoping reviews. Int J Evid Healthcare. 2015;13(3):141–6.

Polanin JR, Pigott TD, Espelage DL, Grotpeter JK. Best practice guidelines for abstract screening large-evidence systematic reviews and meta-analyses. Res Synth Methods. 2019;10(3):330–42.

Article   PubMed Central   Google Scholar  

Shea BJ, Grimshaw JM, Wells GA, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7(1):1–7.

Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. Brit Med J. 2017;358

Sterne JA, Hernán MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. Br Med J. 2016;355

Stroup DF, Berlin JA, Morton SC, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. JAMA. 2000;283(15):2008–12.

Tawfik GM, Dila KAS, Mohamed MYF, et al. A step by step guide for conducting a systematic review and meta-analysis with simulation data. Trop Med Health. 2019;47(1):1–9.

The Critical Appraisal Program. Critical appraisal skills program. Available at https://casp-uk.net/ . 2022. Accessed 29 Aug 2022.

The University of Melbourne. Writing a literature review in Research Techniques 2022. Available at https://students.unimelb.edu.au/academic-skills/explore-our-resources/research-techniques/reviewing-the-literature . Accessed 29 Aug 2022.

The Writing Center University of Winconsin-Madison. Learn how to write a literature review in The Writer’s Handbook – Academic Professional Writing. 2022. Available at https://writing.wisc.edu/handbook/assignments/reviewofliterature/ . Accessed 29 Aug 2022.

Thompson SG, Sharp SJ. Explaining heterogeneity in meta-analysis: a comparison of methods. Stat Med. 1999;18(20):2693–708.

Tricco AC, Lillie E, Zarin W, et al. A scoping review on the conduct and reporting of scoping reviews. BMC Med Res Methodol. 2016;16(1):15.

Tricco AC, Lillie E, Zarin W, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467–73.

Yoneoka D, Henmi M. Clinical heterogeneity in random-effect meta-analysis: between-study boundary estimate problem. Stat Med. 2019;38(21):4131–45.

Yuan Y, Hunt RH. Systematic reviews: the good, the bad, and the ugly. Am J Gastroenterol. 2009;104(5):1086–92.

Download references

Author information

Authors and affiliations.

Centre of Excellence in Treatable Traits, College of Health, Medicine and Wellbeing, University of Newcastle, Hunter Medical Research Institute Asthma and Breathing Programme, Newcastle, NSW, Australia

Dennis Thomas

Department of Pharmacy Practice, Faculty of Pharmacy, Universitas Airlangga, Surabaya, Indonesia

Elida Zairina

Centre for Medicine Use and Safety, Monash Institute of Pharmaceutical Sciences, Faculty of Pharmacy and Pharmaceutical Sciences, Monash University, Parkville, VIC, Australia

Johnson George

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Johnson George .

Section Editor information

College of Pharmacy, Qatar University, Doha, Qatar

Derek Charles Stewart

Department of Pharmacy, University of Huddersfield, Huddersfield, United Kingdom

Zaheer-Ud-Din Babar

Rights and permissions

Reprints and permissions

Copyright information

© 2023 Springer Nature Switzerland AG

About this entry

Cite this entry.

Thomas, D., Zairina, E., George, J. (2023). Methodological Approaches to Literature Review. In: Encyclopedia of Evidence in Pharmaceutical Public Health and Health Services Research in Pharmacy. Springer, Cham. https://doi.org/10.1007/978-3-030-50247-8_57-1

Download citation

DOI : https://doi.org/10.1007/978-3-030-50247-8_57-1

Received : 22 February 2023

Accepted : 22 February 2023

Published : 09 May 2023

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-50247-8

Online ISBN : 978-3-030-50247-8

eBook Packages : Springer Reference Biomedicine and Life Sciences Reference Module Biomedical and Life Sciences

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Harvey Cushing/John Hay Whitney Medical Library

• Collections
• Research Help

YSN Doctoral Programs: Steps in Conducting a Literature Review

• Biomedical Databases
• Global (Public Health) Databases
• Soc. Sci., History, and Law Databases
• Grey Literature
• Trials Registers
• Data and Statistics
• Public Policy
• Google Tips
• Recommended Books
• Steps in Conducting a Literature Review

What is a literature review?

A literature review is an integrated analysis -- not just a summary-- of scholarly writings and other relevant evidence related directly to your research question.  That is, it represents a synthesis of the evidence that provides background information on your topic and shows a association between the evidence and your research question.

A literature review may be a stand alone work or the introduction to a larger research paper, depending on the assignment.  Rely heavily on the guidelines your instructor has given you.

Why is it important?

A literature review is important because it:

• Explains the background of research on a topic.
• Demonstrates why a topic is significant to a subject area.
• Discovers relationships between research studies/ideas.
• Identifies major themes, concepts, and researchers on a topic.
• Identifies critical gaps and points of disagreement.
• Discusses further research questions that logically come out of the previous studies.

APA7 Style resources

APA Style Blog - for those harder to find answers

1. Choose a topic. Define your research question.

Your literature review should be guided by your central research question.  The literature represents background and research developments related to a specific research question, interpreted and analyzed by you in a synthesized way.

• Make sure your research question is not too broad or too narrow.  Is it manageable?
• Begin writing down terms that are related to your question. These will be useful for searches later.
• If you have the opportunity, discuss your topic with your professor and your class mates.

2. Decide on the scope of your review

How many studies do you need to look at? How comprehensive should it be? How many years should it cover? 

• This may depend on your assignment.  How many sources does the assignment require?

3. Select the databases you will use to conduct your searches.

Make a list of the databases you will search. 

Where to find databases:

• use the tabs on this guide
• Find other databases in the Nursing Information Resources web page
• More on the Medical Library web page
• ... and more on the Yale University Library web page

4. Conduct your searches to find the evidence. Keep track of your searches.

• Use the key words in your question, as well as synonyms for those words, as terms in your search. Use the database tutorials for help.
• Save the searches in the databases. This saves time when you want to redo, or modify, the searches. It is also helpful to use as a guide is the searches are not finding any useful results.
• Review the abstracts of research studies carefully. This will save you time.
• Use the bibliographies and references of research studies you find to locate others.
• Check with your professor, or a subject expert in the field, if you are missing any key works in the field.
• Ask your librarian for help at any time.
• Use a citation manager, such as EndNote as the repository for your citations. See the EndNote tutorials for help.

Review the literature

Some questions to help you analyze the research:

• What was the research question of the study you are reviewing? What were the authors trying to discover?
• Was the research funded by a source that could influence the findings?
• What were the research methodologies? Analyze its literature review, the samples and variables used, the results, and the conclusions.
• Does the research seem to be complete? Could it have been conducted more soundly? What further questions does it raise?
• If there are conflicting studies, why do you think that is?
• How are the authors viewed in the field? Has this study been cited? If so, how has it been analyzed?

Tips: 

• Review the abstracts carefully.  
• Keep careful notes so that you may track your thought processes during the research process.
• Create a matrix of the studies for easy analysis, and synthesis, across all of the studies.
• << Previous: Recommended Books
• Last Updated: Jan 4, 2024 10:52 AM
• URL: https://guides.library.yale.edu/YSNDoctoral

• UConn Library
• Literature Review: The What, Why and How-to Guide
• Introduction

Literature Review: The What, Why and How-to Guide — Introduction

• Getting Started
• How to Pick a Topic
• Strategies to Find Sources
• Evaluating Sources & Lit. Reviews
• Tips for Writing Literature Reviews
• Writing Literature Review: Useful Sites
• Citation Resources
• Other Academic Writings

What are Literature Reviews?

So, what is a literature review? "A literature review is an account of what has been published on a topic by accredited scholars and researchers. In writing the literature review, your purpose is to convey to your reader what knowledge and ideas have been established on a topic, and what their strengths and weaknesses are. As a piece of writing, the literature review must be defined by a guiding concept (e.g., your research objective, the problem or issue you are discussing, or your argumentative thesis). It is not just a descriptive list of the material available, or a set of summaries." Taylor, D.  The literature review: A few tips on conducting it . University of Toronto Health Sciences Writing Centre.

Goals of Literature Reviews

What are the goals of creating a Literature Review?  A literature could be written to accomplish different aims:

• To develop a theory or evaluate an existing theory
• To summarize the historical or existing state of a research topic
• Identify a problem in a field of research 

Baumeister, R. F., & Leary, M. R. (1997). Writing narrative literature reviews .  Review of General Psychology , 1 (3), 311-320.

What kinds of sources require a Literature Review?

• A research paper assigned in a course
• A thesis or dissertation
• A grant proposal
• An article intended for publication in a journal

All these instances require you to collect what has been written about your research topic so that you can demonstrate how your own research sheds new light on the topic.

Types of Literature Reviews

What kinds of literature reviews are written?

Narrative review: The purpose of this type of review is to describe the current state of the research on a specific topic/research and to offer a critical analysis of the literature reviewed. Studies are grouped by research/theoretical categories, and themes and trends, strengths and weakness, and gaps are identified. The review ends with a conclusion section which summarizes the findings regarding the state of the research of the specific study, the gaps identify and if applicable, explains how the author's research will address gaps identify in the review and expand the knowledge on the topic reviewed.

• Example : Predictors and Outcomes of U.S. Quality Maternity Leave: A Review and Conceptual Framework:  10.1177/08948453211037398  

Systematic review : "The authors of a systematic review use a specific procedure to search the research literature, select the studies to include in their review, and critically evaluate the studies they find." (p. 139). Nelson, L. K. (2013). Research in Communication Sciences and Disorders . Plural Publishing.

• Example : The effect of leave policies on increasing fertility: a systematic review:  10.1057/s41599-022-01270-w

Meta-analysis : "Meta-analysis is a method of reviewing research findings in a quantitative fashion by transforming the data from individual studies into what is called an effect size and then pooling and analyzing this information. The basic goal in meta-analysis is to explain why different outcomes have occurred in different studies." (p. 197). Roberts, M. C., & Ilardi, S. S. (2003). Handbook of Research Methods in Clinical Psychology . Blackwell Publishing.

• Example : Employment Instability and Fertility in Europe: A Meta-Analysis:  10.1215/00703370-9164737

Meta-synthesis : "Qualitative meta-synthesis is a type of qualitative study that uses as data the findings from other qualitative studies linked by the same or related topic." (p.312). Zimmer, L. (2006). Qualitative meta-synthesis: A question of dialoguing with texts .  Journal of Advanced Nursing , 53 (3), 311-318.

• Example : Women’s perspectives on career successes and barriers: A qualitative meta-synthesis:  10.1177/05390184221113735

Literature Reviews in the Health Sciences

• UConn Health subject guide on systematic reviews Explanation of the different review types used in health sciences literature as well as tools to help you find the right review type
• << Previous: Getting Started
• Next: How to Pick a Topic >>
• Last Updated: Sep 21, 2022 2:16 PM
• URL: https://guides.lib.uconn.edu/literaturereview

Teaching and Research guides

Literature reviews.

• Introduction
• Plan your search
• Where to search
• Refine and update your search
• Finding grey literature
• Writing the review
• Referencing

Research methods overview

Finding literature on research methodologies, sage research methods online.

• Get material not at RMIT
• Further help

What are research methods?

Research methodology is the specific strategies, processes, or techniques utilised in the collection of information that is created and analysed.

The methodology section of a research paper, or thesis, enables the reader to critically evaluate the study’s validity and reliability by addressing how the data was collected or generated, and how it was analysed.

Types of research methods

There are three main types of research methods which use different designs for data collection.  

(1) Qualitative research

Qualitative research gathers data about lived experiences, emotions or behaviours, and the meanings individuals attach to them. It assists in enabling researchers to gain a better understanding of complex concepts, social interactions or cultural phenomena. This type of research is useful in the exploration of how or why things have occurred, interpreting events and describing actions.

Examples of qualitative research designs include:

• focus groups
• observations
• document analysis
• oral history or life stories  

(2) Quantitative research

Quantitative research gathers numerical data which can be ranked, measured or categorised through statistical analysis. It assists with uncovering patterns or relationships, and for making generalisations. This type of research is useful for finding out how many, how much, how often, or to what extent.

Examples of quantitative research designs include:

• surveys or questionnaires
• observation
• document screening
• experiments  

(3) Mixed method research

Mixed Methods research integrates both Qualitative research and Quantitative research. It provides a holistic approach combining and analysing the statistical data with deeper contextualised insights. Using Mixed Methods also enables triangulation, or verification, of the data from two or more sources.

Sometimes in your literature review, you might need to discuss and evaluate relevant research methodologies in order to justify your own choice of research methodology.

When searching for literature on research methodologies it is important to search across a range of sources. No single information source will supply all that you need. Selecting appropriate sources will depend upon your research topic.

Developing a robust search strategy will help reduce irrelevant results. It is good practice to plan a strategy before you start to search.

Search tips

(1) free text keywords.

Free text searching is the use of natural language words to conduct your search. Use selective free text keywords such as: phenomenological, "lived experience", "grounded theory", "life experiences", "focus groups", interview, quantitative, survey, validity, variance, correlation and statistical.

To locate books on your desired methodology, try LibrarySearch . Remember to use  refine  options such as books, ebooks, subject, and publication date.  

(2) Subject headings in Databases

Databases categorise their records using subject terms, or a controlled vocabulary (thesaurus). These subject headings may be useful to use, in addition to utilising free text keywords in a database search.

Subject headings will differ across databases, for example, the PubMed database uses 'Qualitative Research' whilst the CINHAL database uses 'Qualitative Studies.'  

(3) Limiting search results

Databases enable sets of results to be limited or filtered by specific fields, look for options such as Publication Type, Article Type, etc. and apply them to your search.  

(4) Browse the Library shelves

To find books on  research methods  browse the Library shelves at call number  001.42

• SAGE Research Methods Online SAGE Research Methods Online (SRMO) is a research tool supported by a newly devised taxonomy that links content and methods terms. It provides the most comprehensive picture available today of research methods (quantitative, qualitative and mixed methods) across the social and behavioural sciences.

SAGE Research Methods Overview  (2:07 min) by SAGE Publishing  ( YouTube ) 

• << Previous: Referencing
• Next: Get material not at RMIT >>

• Last Updated: May 28, 2024 11:17 AM
• URL: https://rmit.libguides.com/literature-review

Research Methods: Literature Reviews

• Annotated Bibliographies
• Literature Reviews
• Scoping Reviews
• Systematic Reviews
• Scholarship of Teaching and Learning
• Persuasive Arguments
• Subject Specific Methodology

A literature review involves researching, reading, analyzing, evaluating, and summarizing scholarly literature (typically journals and articles) about a specific topic. The results of a literature review may be an entire report or article OR may be part of a article, thesis, dissertation, or grant proposal. A literature review helps the author learn about the history and nature of their topic, and identify research gaps and problems.

Steps & Elements

Problem formulation

• Determine your topic and its components by asking a question
• Research: locate literature related to your topic to identify the gap(s) that can be addressed
• Read: read the articles or other sources of information
• Analyze: assess the findings for relevancy
• Evaluating: determine how the article are relevant to your research and what are the key findings
• Synthesis: write about the key findings and how it is relevant to your research

Elements of a Literature Review

• Summarize subject, issue or theory under consideration, along with objectives of the review
• Divide works under review into categories (e.g. those in support of a particular position, those against, those offering alternative theories entirely)
• Explain how each work is similar to and how it varies from the others
• Conclude which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of an area of research

Writing a Literature Review Resources

• How to Write a Literature Review From the Wesleyan University Library
• Write a Literature Review From the University of California Santa Cruz Library. A Brief overview of a literature review, includes a list of stages for writing a lit review.
• Literature Reviews From the University of North Carolina Writing Center. Detailed information about writing a literature review.
• Undertaking a literature review: a step-by-step approach Cronin, P., Ryan, F., & Coughan, M. (2008). Undertaking a literature review: A step-by-step approach. British Journal of Nursing, 17(1), p.38-43

Literature Review Tutorial

• << Previous: Annotated Bibliographies
• Next: Scoping Reviews >>
• Last Updated: Feb 29, 2024 12:00 PM
• URL: https://guides.auraria.edu/researchmethods

1100 Lawrence Street Denver, CO 80204 303-315-7700 Ask Us Directions

Covidence website will be inaccessible as we upgrading our platform on Monday 23rd August at 10am AEST, / 2am CEST/1am BST (Sunday, 15th August 8pm EDT/5pm PDT) 

How to write the methods section of a systematic review

Home | Blog | How To | How to write the methods section of a systematic review

Covidence breaks down how to write a methods section

The methods section of your systematic review describes what you did, how you did it, and why. Readers need this information to interpret the results and conclusions of the review. Often, a lot of information needs to be distilled into just a few paragraphs. This can be a challenging task, but good preparation and the right tools will help you to set off in the right direction 🗺️🧭.

Systematic reviews are so-called because they are conducted in a way that is rigorous and replicable. So it’s important that these methods are reported in a way that is thorough, clear, and easy to navigate for the reader – whether that’s a patient, a healthcare worker, or a researcher. 

Like most things in a systematic review, the methods should be planned upfront and ideally described in detail in a project plan or protocol. Reviews of healthcare interventions follow the PRISMA guidelines for the minimum set of items to report in the methods section. But what else should be included? It’s a good idea to consider what readers will want to know about the review methods and whether the journal you’re planning to submit the work to has expectations on the reporting of methods. Finding out in advance will help you to plan what to include.

Describe what happened

While the research plan sets out what you intend to do, the methods section is a write-up of what actually happened. It’s not a simple case of rewriting the plan in the past tense – you will also need to discuss and justify deviations from the plan and describe the handling of issues that were unforeseen at the time the plan was written. For this reason, it is useful to make detailed notes before, during, and after the review is completed. Relying on memory alone risks losing valuable information and trawling through emails when the deadline is looming can be frustrating and time consuming! 

Keep it brief

The methods section should be succinct but include all the noteworthy information. This can be a difficult balance to achieve. A useful strategy is to aim for a brief description that signposts the reader to a separate section or sections of supporting information. This could include datasets, a flowchart to show what happened to the excluded studies, a collection of search strategies, and tables containing detailed information about the studies.This separation keeps the review short and simple while enabling the reader to drill down to the detail as needed. And if the methods follow a well-known or standard process, it might suffice to say so and give a reference, rather than describe the process at length. 

Follow a structure

A clear structure provides focus. Use of descriptive headings keeps the writing on track and helps the reader get to key information quickly. What should the structure of the methods section look like? As always, a lot depends on the type of review but it will certainly contain information relating to the following areas:

• Selection criteria ⭕
• Data collection and analysis 👩‍💻
• Study quality and risk of bias ⚖️

Let’s look at each of these in turn.

1. Selection criteria ⭕

The criteria for including and excluding studies are listed here. This includes detail about the types of studies, the types of participants, the types of interventions and the types of outcomes and how they were measured. 

2. Search 🕵🏾‍♀️

Comprehensive reporting of the search is important because this means it can be evaluated and replicated. The search strategies are included in the review, along with details of the databases searched. It’s also important to list any restrictions on the search (for example, language), describe how resources other than electronic databases were searched (for example,  non-indexed journals), and give the date that the searches were run. The PRISMA-S extension provides guidance on reporting literature searches. 

Systematic reviewer pro-tip:

 Copy and paste the search strategy to avoid introducing typos

3. Data collection and analysis 👩‍💻

This section describes:

• how studies were selected for inclusion in the review
• how study data were extracted from the study reports
• how study data were combined for analysis and synthesis

To describe how studies were selected for inclusion , review teams outline the screening process. Covidence uses reviewers’ decision data to automatically populate a PRISMA flow diagram for this purpose. Covidence can also calculate Cohen’s kappa to enable review teams to report the level of agreement among individual reviewers during screening.

To describe how study data were extracted from the study reports , reviewers outline the form that was used, any pilot-testing that was done, and the items that were extracted from the included studies. An important piece of information to include here is the process used to resolve conflict among the reviewers. Covidence’s data extraction tool saves reviewers’ comments and notes in the system as they work. This keeps the information in one place for easy retrieval ⚡.

To describe how study data were combined for analysis and synthesis, reviewers outline the type of synthesis (narrative or quantitative, for example), the methods for grouping data, the challenges that came up, and how these were dealt with. If the review includes a meta-analysis, it will detail how this was performed and how the treatment effects were measured.

4. Study quality and risk of bias ⚖️

Because the results of systematic reviews can be affected by many types of bias, reviewers make every effort to minimise it and to show the reader that the methods they used were appropriate. This section describes the methods used to assess study quality and an assessment of the risk of bias across a range of domains. 

Steps to assess the risk of bias in studies include looking at how study participants were assigned to treatment groups and whether patients and/or study assessors were blinded to the treatment given. Reviewers also report their assessment of the risk of bias due to missing outcome data, whether that is due to participant drop-out or non-reporting of the outcomes by the study authors.

Covidence’s default template for assessing study quality is Cochrane’s risk of bias tool but it is also possible to start from scratch and build a tool with a set of custom domains if you prefer.

Careful planning, clear writing, and a structured approach are key to a good methods section. A methodologist will be able to refer review teams to examples of good methods reporting in the literature. Covidence helps reviewers to screen references, extract data and complete risk of bias tables quickly and efficiently. Sign up for a free trial today!

Laura Mellor. Portsmouth, UK

Perhaps you'd also like....

Top 5 Tips for High-Quality Systematic Review Data Extraction

Data extraction can be a complex step in the systematic review process. Here are 5 top tips from our experts to help prepare and achieve high quality data extraction.

How to get through study quality assessment Systematic Review

Find out 5 tops tips to conducting quality assessment and why it’s an important step in the systematic review process.

How to extract study data for your systematic review

Learn the basic process and some tips to build data extraction forms for your systematic review with Covidence.

Better systematic review management

Head office, working for an institution or organisation.

Find out why over 350 of the world’s leading institutions are seeing a surge in publications since using Covidence!

Request a consultation with one of our team members and start empowering your researchers:

By using our site you consent to our use of cookies to measure and improve our site’s performance. Please see our Privacy Policy for more information. 

New change to library operations

All Main Library and Weaver Library doors lock 15 minutes before closing.

Conduct a literature review

What is a literature review.

A literature review is a summary of the published work in a field of study. This can be a section of a larger paper or article, or can be the focus of an entire paper. Literature reviews show that you have examined the breadth of knowledge and can justify your thesis or research questions. They are also valuable tools for other researchers who need to find a summary of that field of knowledge.

Unlike an annotated bibliography, which is a list of sources with short descriptions, a literature review synthesizes sources into a summary that has a thesis or statement of purpose—stated or implied—at its core.

How do I write a literature review?

Step 1: define your research scope.

• What is the specific research question that your literature review helps to define?
• Are there a maximum or minimum number of sources that your review should include?

Ask us if you have questions about refining your topic, search methods, writing tips, or citation management.

Step 2: Identify the literature

Start by searching broadly. Literature for your review will typically be acquired through scholarly books, journal articles, and/or dissertations. Develop an understanding of what is out there, what terms are accurate and helpful, etc., and keep track of all of it with citation management tools . If you need help figuring out key terms and where to search, ask us .

Use citation searching to track how scholars interact with, and build upon, previous research:

• Mine the references cited section of each relevant source for additional key sources
• Use Google Scholar or Scopus to find other sources that have cited a particular work

Step 3: Critically analyze the literature

Key to your literature review is a critical analysis of the literature collected around your topic. The analysis will explore relationships, major themes, and any critical gaps in the research expressed in the work. Read and summarize each source with an eye toward analyzing authority, currency, coverage, methodology, and relationship to other works. The University of Toronto's Writing Center provides a comprehensive list of questions you can use to analyze your sources.

Step 4: Categorize your resources

Divide the available resources that pertain to your research into categories reflecting their roles in addressing your research question. Possible ways to categorize resources include organization by:

• methodology
• theoretical/philosophical approach

Regardless of the division, each category should be accompanied by thorough discussions and explanations of strengths and weaknesses, value to the overall survey, and comparisons with similar sources. You may have enough resources when:

• You've used multiple databases and other resources (web portals, repositories, etc.) to get a variety of perspectives on the research topic.
• The same citations are showing up in a variety of databases.

Additional resources

Undergraduate student resources.

• Literature Review Handout (University of North Carolina at Chapel Hill)
• Learn how to write a review of literature (University of Wisconsin-Madison)

Graduate student and faculty resources

• Information Research Strategies (University of Arizona)
• Literature Reviews: An Overview for Graduate Students (NC State University)
• Oliver, P. (2012). Succeeding with Your Literature Review: A Handbook for Students [ebook]
• Machi, L. A. & McEvoy, B. T. (2016). The Literature Review: Six Steps to Success [ebook]
• Graustein, J. S. (2012). How to Write an Exceptional Thesis or Dissertation: A Step-by-Step Guide from Proposal to Successful Defense [ebook]
• Thomas, R. M. & Brubaker, D. L. (2008). Theses and Dissertations: A Guide to Planning, Research, and Writing

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.

Handbook of eHealth Evaluation: An Evidence-based Approach [Internet].

Chapter 9 methods for literature reviews.

Guy Paré and Spyros Kitsiou .

9.1. Introduction

Literature reviews play a critical role in scholarship because science remains, first and foremost, a cumulative endeavour ( vom Brocke et al., 2009 ). As in any academic discipline, rigorous knowledge syntheses are becoming indispensable in keeping up with an exponentially growing eHealth literature, assisting practitioners, academics, and graduate students in finding, evaluating, and synthesizing the contents of many empirical and conceptual papers. Among other methods, literature reviews are essential for: (a) identifying what has been written on a subject or topic; (b) determining the extent to which a specific research area reveals any interpretable trends or patterns; (c) aggregating empirical findings related to a narrow research question to support evidence-based practice; (d) generating new frameworks and theories; and (e) identifying topics or questions requiring more investigation ( Paré, Trudel, Jaana, & Kitsiou, 2015 ).

Literature reviews can take two major forms. The most prevalent one is the “literature review” or “background” section within a journal paper or a chapter in a graduate thesis. This section synthesizes the extant literature and usually identifies the gaps in knowledge that the empirical study addresses ( Sylvester, Tate, & Johnstone, 2013 ). It may also provide a theoretical foundation for the proposed study, substantiate the presence of the research problem, justify the research as one that contributes something new to the cumulated knowledge, or validate the methods and approaches for the proposed study ( Hart, 1998 ; Levy & Ellis, 2006 ).

The second form of literature review, which is the focus of this chapter, constitutes an original and valuable work of research in and of itself ( Paré et al., 2015 ). Rather than providing a base for a researcher’s own work, it creates a solid starting point for all members of the community interested in a particular area or topic ( Mulrow, 1987 ). The so-called “review article” is a journal-length paper which has an overarching purpose to synthesize the literature in a field, without collecting or analyzing any primary data ( Green, Johnson, & Adams, 2006 ).

When appropriately conducted, review articles represent powerful information sources for practitioners looking for state-of-the art evidence to guide their decision-making and work practices ( Paré et al., 2015 ). Further, high-quality reviews become frequently cited pieces of work which researchers seek out as a first clear outline of the literature when undertaking empirical studies ( Cooper, 1988 ; Rowe, 2014 ). Scholars who track and gauge the impact of articles have found that review papers are cited and downloaded more often than any other type of published article ( Cronin, Ryan, & Coughlan, 2008 ; Montori, Wilczynski, Morgan, Haynes, & Hedges, 2003 ; Patsopoulos, Analatos, & Ioannidis, 2005 ). The reason for their popularity may be the fact that reading the review enables one to have an overview, if not a detailed knowledge of the area in question, as well as references to the most useful primary sources ( Cronin et al., 2008 ). Although they are not easy to conduct, the commitment to complete a review article provides a tremendous service to one’s academic community ( Paré et al., 2015 ; Petticrew & Roberts, 2006 ). Most, if not all, peer-reviewed journals in the fields of medical informatics publish review articles of some type.

The main objectives of this chapter are fourfold: (a) to provide an overview of the major steps and activities involved in conducting a stand-alone literature review; (b) to describe and contrast the different types of review articles that can contribute to the eHealth knowledge base; (c) to illustrate each review type with one or two examples from the eHealth literature; and (d) to provide a series of recommendations for prospective authors of review articles in this domain.

9.2. Overview of the Literature Review Process and Steps

As explained in Templier and Paré (2015) , there are six generic steps involved in conducting a review article:

• formulating the research question(s) and objective(s),
• searching the extant literature,
• screening for inclusion,
• assessing the quality of primary studies,
• extracting data, and
• analyzing data.

Although these steps are presented here in sequential order, one must keep in mind that the review process can be iterative and that many activities can be initiated during the planning stage and later refined during subsequent phases ( Finfgeld-Connett & Johnson, 2013 ; Kitchenham & Charters, 2007 ).

Formulating the research question(s) and objective(s): As a first step, members of the review team must appropriately justify the need for the review itself ( Petticrew & Roberts, 2006 ), identify the review’s main objective(s) ( Okoli & Schabram, 2010 ), and define the concepts or variables at the heart of their synthesis ( Cooper & Hedges, 2009 ; Webster & Watson, 2002 ). Importantly, they also need to articulate the research question(s) they propose to investigate ( Kitchenham & Charters, 2007 ). In this regard, we concur with Jesson, Matheson, and Lacey (2011) that clearly articulated research questions are key ingredients that guide the entire review methodology; they underscore the type of information that is needed, inform the search for and selection of relevant literature, and guide or orient the subsequent analysis. Searching the extant literature: The next step consists of searching the literature and making decisions about the suitability of material to be considered in the review ( Cooper, 1988 ). There exist three main coverage strategies. First, exhaustive coverage means an effort is made to be as comprehensive as possible in order to ensure that all relevant studies, published and unpublished, are included in the review and, thus, conclusions are based on this all-inclusive knowledge base. The second type of coverage consists of presenting materials that are representative of most other works in a given field or area. Often authors who adopt this strategy will search for relevant articles in a small number of top-tier journals in a field ( Paré et al., 2015 ). In the third strategy, the review team concentrates on prior works that have been central or pivotal to a particular topic. This may include empirical studies or conceptual papers that initiated a line of investigation, changed how problems or questions were framed, introduced new methods or concepts, or engendered important debate ( Cooper, 1988 ). Screening for inclusion: The following step consists of evaluating the applicability of the material identified in the preceding step ( Levy & Ellis, 2006 ; vom Brocke et al., 2009 ). Once a group of potential studies has been identified, members of the review team must screen them to determine their relevance ( Petticrew & Roberts, 2006 ). A set of predetermined rules provides a basis for including or excluding certain studies. This exercise requires a significant investment on the part of researchers, who must ensure enhanced objectivity and avoid biases or mistakes. As discussed later in this chapter, for certain types of reviews there must be at least two independent reviewers involved in the screening process and a procedure to resolve disagreements must also be in place ( Liberati et al., 2009 ; Shea et al., 2009 ). Assessing the quality of primary studies: In addition to screening material for inclusion, members of the review team may need to assess the scientific quality of the selected studies, that is, appraise the rigour of the research design and methods. Such formal assessment, which is usually conducted independently by at least two coders, helps members of the review team refine which studies to include in the final sample, determine whether or not the differences in quality may affect their conclusions, or guide how they analyze the data and interpret the findings ( Petticrew & Roberts, 2006 ). Ascribing quality scores to each primary study or considering through domain-based evaluations which study components have or have not been designed and executed appropriately makes it possible to reflect on the extent to which the selected study addresses possible biases and maximizes validity ( Shea et al., 2009 ). Extracting data: The following step involves gathering or extracting applicable information from each primary study included in the sample and deciding what is relevant to the problem of interest ( Cooper & Hedges, 2009 ). Indeed, the type of data that should be recorded mainly depends on the initial research questions ( Okoli & Schabram, 2010 ). However, important information may also be gathered about how, when, where and by whom the primary study was conducted, the research design and methods, or qualitative/quantitative results ( Cooper & Hedges, 2009 ). Analyzing and synthesizing data : As a final step, members of the review team must collate, summarize, aggregate, organize, and compare the evidence extracted from the included studies. The extracted data must be presented in a meaningful way that suggests a new contribution to the extant literature ( Jesson et al., 2011 ). Webster and Watson (2002) warn researchers that literature reviews should be much more than lists of papers and should provide a coherent lens to make sense of extant knowledge on a given topic. There exist several methods and techniques for synthesizing quantitative (e.g., frequency analysis, meta-analysis) and qualitative (e.g., grounded theory, narrative analysis, meta-ethnography) evidence ( Dixon-Woods, Agarwal, Jones, Young, & Sutton, 2005 ; Thomas & Harden, 2008 ).

9.3. Types of Review Articles and Brief Illustrations

EHealth researchers have at their disposal a number of approaches and methods for making sense out of existing literature, all with the purpose of casting current research findings into historical contexts or explaining contradictions that might exist among a set of primary research studies conducted on a particular topic. Our classification scheme is largely inspired from Paré and colleagues’ (2015) typology. Below we present and illustrate those review types that we feel are central to the growth and development of the eHealth domain.

9.3.1. Narrative Reviews

The narrative review is the “traditional” way of reviewing the extant literature and is skewed towards a qualitative interpretation of prior knowledge ( Sylvester et al., 2013 ). Put simply, a narrative review attempts to summarize or synthesize what has been written on a particular topic but does not seek generalization or cumulative knowledge from what is reviewed ( Davies, 2000 ; Green et al., 2006 ). Instead, the review team often undertakes the task of accumulating and synthesizing the literature to demonstrate the value of a particular point of view ( Baumeister & Leary, 1997 ). As such, reviewers may selectively ignore or limit the attention paid to certain studies in order to make a point. In this rather unsystematic approach, the selection of information from primary articles is subjective, lacks explicit criteria for inclusion and can lead to biased interpretations or inferences ( Green et al., 2006 ). There are several narrative reviews in the particular eHealth domain, as in all fields, which follow such an unstructured approach ( Silva et al., 2015 ; Paul et al., 2015 ).

Despite these criticisms, this type of review can be very useful in gathering together a volume of literature in a specific subject area and synthesizing it. As mentioned above, its primary purpose is to provide the reader with a comprehensive background for understanding current knowledge and highlighting the significance of new research ( Cronin et al., 2008 ). Faculty like to use narrative reviews in the classroom because they are often more up to date than textbooks, provide a single source for students to reference, and expose students to peer-reviewed literature ( Green et al., 2006 ). For researchers, narrative reviews can inspire research ideas by identifying gaps or inconsistencies in a body of knowledge, thus helping researchers to determine research questions or formulate hypotheses. Importantly, narrative reviews can also be used as educational articles to bring practitioners up to date with certain topics of issues ( Green et al., 2006 ).

Recently, there have been several efforts to introduce more rigour in narrative reviews that will elucidate common pitfalls and bring changes into their publication standards. Information systems researchers, among others, have contributed to advancing knowledge on how to structure a “traditional” review. For instance, Levy and Ellis (2006) proposed a generic framework for conducting such reviews. Their model follows the systematic data processing approach comprised of three steps, namely: (a) literature search and screening; (b) data extraction and analysis; and (c) writing the literature review. They provide detailed and very helpful instructions on how to conduct each step of the review process. As another methodological contribution, vom Brocke et al. (2009) offered a series of guidelines for conducting literature reviews, with a particular focus on how to search and extract the relevant body of knowledge. Last, Bandara, Miskon, and Fielt (2011) proposed a structured, predefined and tool-supported method to identify primary studies within a feasible scope, extract relevant content from identified articles, synthesize and analyze the findings, and effectively write and present the results of the literature review. We highly recommend that prospective authors of narrative reviews consult these useful sources before embarking on their work.

Darlow and Wen (2015) provide a good example of a highly structured narrative review in the eHealth field. These authors synthesized published articles that describe the development process of mobile health ( m-health ) interventions for patients’ cancer care self-management. As in most narrative reviews, the scope of the research questions being investigated is broad: (a) how development of these systems are carried out; (b) which methods are used to investigate these systems; and (c) what conclusions can be drawn as a result of the development of these systems. To provide clear answers to these questions, a literature search was conducted on six electronic databases and Google Scholar . The search was performed using several terms and free text words, combining them in an appropriate manner. Four inclusion and three exclusion criteria were utilized during the screening process. Both authors independently reviewed each of the identified articles to determine eligibility and extract study information. A flow diagram shows the number of studies identified, screened, and included or excluded at each stage of study selection. In terms of contributions, this review provides a series of practical recommendations for m-health intervention development.

9.3.2. Descriptive or Mapping Reviews

The primary goal of a descriptive review is to determine the extent to which a body of knowledge in a particular research topic reveals any interpretable pattern or trend with respect to pre-existing propositions, theories, methodologies or findings ( King & He, 2005 ; Paré et al., 2015 ). In contrast with narrative reviews, descriptive reviews follow a systematic and transparent procedure, including searching, screening and classifying studies ( Petersen, Vakkalanka, & Kuzniarz, 2015 ). Indeed, structured search methods are used to form a representative sample of a larger group of published works ( Paré et al., 2015 ). Further, authors of descriptive reviews extract from each study certain characteristics of interest, such as publication year, research methods, data collection techniques, and direction or strength of research outcomes (e.g., positive, negative, or non-significant) in the form of frequency analysis to produce quantitative results ( Sylvester et al., 2013 ). In essence, each study included in a descriptive review is treated as the unit of analysis and the published literature as a whole provides a database from which the authors attempt to identify any interpretable trends or draw overall conclusions about the merits of existing conceptualizations, propositions, methods or findings ( Paré et al., 2015 ). In doing so, a descriptive review may claim that its findings represent the state of the art in a particular domain ( King & He, 2005 ).

In the fields of health sciences and medical informatics, reviews that focus on examining the range, nature and evolution of a topic area are described by Anderson, Allen, Peckham, and Goodwin (2008) as mapping reviews . Like descriptive reviews, the research questions are generic and usually relate to publication patterns and trends. There is no preconceived plan to systematically review all of the literature although this can be done. Instead, researchers often present studies that are representative of most works published in a particular area and they consider a specific time frame to be mapped.

An example of this approach in the eHealth domain is offered by DeShazo, Lavallie, and Wolf (2009). The purpose of this descriptive or mapping review was to characterize publication trends in the medical informatics literature over a 20-year period (1987 to 2006). To achieve this ambitious objective, the authors performed a bibliometric analysis of medical informatics citations indexed in medline using publication trends, journal frequencies, impact factors, Medical Subject Headings (MeSH) term frequencies, and characteristics of citations. Findings revealed that there were over 77,000 medical informatics articles published during the covered period in numerous journals and that the average annual growth rate was 12%. The MeSH term analysis also suggested a strong interdisciplinary trend. Finally, average impact scores increased over time with two notable growth periods. Overall, patterns in research outputs that seem to characterize the historic trends and current components of the field of medical informatics suggest it may be a maturing discipline (DeShazo et al., 2009).

9.3.3. Scoping Reviews

Scoping reviews attempt to provide an initial indication of the potential size and nature of the extant literature on an emergent topic (Arksey & O’Malley, 2005; Daudt, van Mossel, & Scott, 2013 ; Levac, Colquhoun, & O’Brien, 2010). A scoping review may be conducted to examine the extent, range and nature of research activities in a particular area, determine the value of undertaking a full systematic review (discussed next), or identify research gaps in the extant literature ( Paré et al., 2015 ). In line with their main objective, scoping reviews usually conclude with the presentation of a detailed research agenda for future works along with potential implications for both practice and research.

Unlike narrative and descriptive reviews, the whole point of scoping the field is to be as comprehensive as possible, including grey literature (Arksey & O’Malley, 2005). Inclusion and exclusion criteria must be established to help researchers eliminate studies that are not aligned with the research questions. It is also recommended that at least two independent coders review abstracts yielded from the search strategy and then the full articles for study selection ( Daudt et al., 2013 ). The synthesized evidence from content or thematic analysis is relatively easy to present in tabular form (Arksey & O’Malley, 2005; Thomas & Harden, 2008 ).

One of the most highly cited scoping reviews in the eHealth domain was published by Archer, Fevrier-Thomas, Lokker, McKibbon, and Straus (2011) . These authors reviewed the existing literature on personal health record ( phr ) systems including design, functionality, implementation, applications, outcomes, and benefits. Seven databases were searched from 1985 to March 2010. Several search terms relating to phr s were used during this process. Two authors independently screened titles and abstracts to determine inclusion status. A second screen of full-text articles, again by two independent members of the research team, ensured that the studies described phr s. All in all, 130 articles met the criteria and their data were extracted manually into a database. The authors concluded that although there is a large amount of survey, observational, cohort/panel, and anecdotal evidence of phr benefits and satisfaction for patients, more research is needed to evaluate the results of phr implementations. Their in-depth analysis of the literature signalled that there is little solid evidence from randomized controlled trials or other studies through the use of phr s. Hence, they suggested that more research is needed that addresses the current lack of understanding of optimal functionality and usability of these systems, and how they can play a beneficial role in supporting patient self-management ( Archer et al., 2011 ).

9.3.4. Forms of Aggregative Reviews

Healthcare providers, practitioners, and policy-makers are nowadays overwhelmed with large volumes of information, including research-based evidence from numerous clinical trials and evaluation studies, assessing the effectiveness of health information technologies and interventions ( Ammenwerth & de Keizer, 2004 ; Deshazo et al., 2009 ). It is unrealistic to expect that all these disparate actors will have the time, skills, and necessary resources to identify the available evidence in the area of their expertise and consider it when making decisions. Systematic reviews that involve the rigorous application of scientific strategies aimed at limiting subjectivity and bias (i.e., systematic and random errors) can respond to this challenge.

Systematic reviews attempt to aggregate, appraise, and synthesize in a single source all empirical evidence that meet a set of previously specified eligibility criteria in order to answer a clearly formulated and often narrow research question on a particular topic of interest to support evidence-based practice ( Liberati et al., 2009 ). They adhere closely to explicit scientific principles ( Liberati et al., 2009 ) and rigorous methodological guidelines (Higgins & Green, 2008) aimed at reducing random and systematic errors that can lead to deviations from the truth in results or inferences. The use of explicit methods allows systematic reviews to aggregate a large body of research evidence, assess whether effects or relationships are in the same direction and of the same general magnitude, explain possible inconsistencies between study results, and determine the strength of the overall evidence for every outcome of interest based on the quality of included studies and the general consistency among them ( Cook, Mulrow, & Haynes, 1997 ). The main procedures of a systematic review involve:

• Formulating a review question and developing a search strategy based on explicit inclusion criteria for the identification of eligible studies (usually described in the context of a detailed review protocol).
• Searching for eligible studies using multiple databases and information sources, including grey literature sources, without any language restrictions.
• Selecting studies, extracting data, and assessing risk of bias in a duplicate manner using two independent reviewers to avoid random or systematic errors in the process.
• Analyzing data using quantitative or qualitative methods.
• Presenting results in summary of findings tables.
• Interpreting results and drawing conclusions.

Many systematic reviews, but not all, use statistical methods to combine the results of independent studies into a single quantitative estimate or summary effect size. Known as meta-analyses , these reviews use specific data extraction and statistical techniques (e.g., network, frequentist, or Bayesian meta-analyses) to calculate from each study by outcome of interest an effect size along with a confidence interval that reflects the degree of uncertainty behind the point estimate of effect ( Borenstein, Hedges, Higgins, & Rothstein, 2009 ; Deeks, Higgins, & Altman, 2008 ). Subsequently, they use fixed or random-effects analysis models to combine the results of the included studies, assess statistical heterogeneity, and calculate a weighted average of the effect estimates from the different studies, taking into account their sample sizes. The summary effect size is a value that reflects the average magnitude of the intervention effect for a particular outcome of interest or, more generally, the strength of a relationship between two variables across all studies included in the systematic review. By statistically combining data from multiple studies, meta-analyses can create more precise and reliable estimates of intervention effects than those derived from individual studies alone, when these are examined independently as discrete sources of information.

The review by Gurol-Urganci, de Jongh, Vodopivec-Jamsek, Atun, and Car (2013) on the effects of mobile phone messaging reminders for attendance at healthcare appointments is an illustrative example of a high-quality systematic review with meta-analysis. Missed appointments are a major cause of inefficiency in healthcare delivery with substantial monetary costs to health systems. These authors sought to assess whether mobile phone-based appointment reminders delivered through Short Message Service ( sms ) or Multimedia Messaging Service ( mms ) are effective in improving rates of patient attendance and reducing overall costs. To this end, they conducted a comprehensive search on multiple databases using highly sensitive search strategies without language or publication-type restrictions to identify all rct s that are eligible for inclusion. In order to minimize the risk of omitting eligible studies not captured by the original search, they supplemented all electronic searches with manual screening of trial registers and references contained in the included studies. Study selection, data extraction, and risk of bias assessments were performed inde­­pen­dently by two coders using standardized methods to ensure consistency and to eliminate potential errors. Findings from eight rct s involving 6,615 participants were pooled into meta-analyses to calculate the magnitude of effects that mobile text message reminders have on the rate of attendance at healthcare appointments compared to no reminders and phone call reminders.

Meta-analyses are regarded as powerful tools for deriving meaningful conclusions. However, there are situations in which it is neither reasonable nor appropriate to pool studies together using meta-analytic methods simply because there is extensive clinical heterogeneity between the included studies or variation in measurement tools, comparisons, or outcomes of interest. In these cases, systematic reviews can use qualitative synthesis methods such as vote counting, content analysis, classification schemes and tabulations, as an alternative approach to narratively synthesize the results of the independent studies included in the review. This form of review is known as qualitative systematic review.

A rigorous example of one such review in the eHealth domain is presented by Mickan, Atherton, Roberts, Heneghan, and Tilson (2014) on the use of handheld computers by healthcare professionals and their impact on access to information and clinical decision-making. In line with the methodological guide­lines for systematic reviews, these authors: (a) developed and registered with prospero ( www.crd.york.ac.uk/ prospero / ) an a priori review protocol; (b) conducted comprehensive searches for eligible studies using multiple databases and other supplementary strategies (e.g., forward searches); and (c) subsequently carried out study selection, data extraction, and risk of bias assessments in a duplicate manner to eliminate potential errors in the review process. Heterogeneity between the included studies in terms of reported outcomes and measures precluded the use of meta-analytic methods. To this end, the authors resorted to using narrative analysis and synthesis to describe the effectiveness of handheld computers on accessing information for clinical knowledge, adherence to safety and clinical quality guidelines, and diagnostic decision-making.

In recent years, the number of systematic reviews in the field of health informatics has increased considerably. Systematic reviews with discordant findings can cause great confusion and make it difficult for decision-makers to interpret the review-level evidence ( Moher, 2013 ). Therefore, there is a growing need for appraisal and synthesis of prior systematic reviews to ensure that decision-making is constantly informed by the best available accumulated evidence. Umbrella reviews , also known as overviews of systematic reviews, are tertiary types of evidence synthesis that aim to accomplish this; that is, they aim to compare and contrast findings from multiple systematic reviews and meta-analyses ( Becker & Oxman, 2008 ). Umbrella reviews generally adhere to the same principles and rigorous methodological guidelines used in systematic reviews. However, the unit of analysis in umbrella reviews is the systematic review rather than the primary study ( Becker & Oxman, 2008 ). Unlike systematic reviews that have a narrow focus of inquiry, umbrella reviews focus on broader research topics for which there are several potential interventions ( Smith, Devane, Begley, & Clarke, 2011 ). A recent umbrella review on the effects of home telemonitoring interventions for patients with heart failure critically appraised, compared, and synthesized evidence from 15 systematic reviews to investigate which types of home telemonitoring technologies and forms of interventions are more effective in reducing mortality and hospital admissions ( Kitsiou, Paré, & Jaana, 2015 ).

9.3.5. Realist Reviews

Realist reviews are theory-driven interpretative reviews developed to inform, enhance, or supplement conventional systematic reviews by making sense of heterogeneous evidence about complex interventions applied in diverse contexts in a way that informs policy decision-making ( Greenhalgh, Wong, Westhorp, & Pawson, 2011 ). They originated from criticisms of positivist systematic reviews which centre on their “simplistic” underlying assumptions ( Oates, 2011 ). As explained above, systematic reviews seek to identify causation. Such logic is appropriate for fields like medicine and education where findings of randomized controlled trials can be aggregated to see whether a new treatment or intervention does improve outcomes. However, many argue that it is not possible to establish such direct causal links between interventions and outcomes in fields such as social policy, management, and information systems where for any intervention there is unlikely to be a regular or consistent outcome ( Oates, 2011 ; Pawson, 2006 ; Rousseau, Manning, & Denyer, 2008 ).

To circumvent these limitations, Pawson, Greenhalgh, Harvey, and Walshe (2005) have proposed a new approach for synthesizing knowledge that seeks to unpack the mechanism of how “complex interventions” work in particular contexts. The basic research question — what works? — which is usually associated with systematic reviews changes to: what is it about this intervention that works, for whom, in what circumstances, in what respects and why? Realist reviews have no particular preference for either quantitative or qualitative evidence. As a theory-building approach, a realist review usually starts by articulating likely underlying mechanisms and then scrutinizes available evidence to find out whether and where these mechanisms are applicable ( Shepperd et al., 2009 ). Primary studies found in the extant literature are viewed as case studies which can test and modify the initial theories ( Rousseau et al., 2008 ).

The main objective pursued in the realist review conducted by Otte-Trojel, de Bont, Rundall, and van de Klundert (2014) was to examine how patient portals contribute to health service delivery and patient outcomes. The specific goals were to investigate how outcomes are produced and, most importantly, how variations in outcomes can be explained. The research team started with an exploratory review of background documents and research studies to identify ways in which patient portals may contribute to health service delivery and patient outcomes. The authors identified six main ways which represent “educated guesses” to be tested against the data in the evaluation studies. These studies were identified through a formal and systematic search in four databases between 2003 and 2013. Two members of the research team selected the articles using a pre-established list of inclusion and exclusion criteria and following a two-step procedure. The authors then extracted data from the selected articles and created several tables, one for each outcome category. They organized information to bring forward those mechanisms where patient portals contribute to outcomes and the variation in outcomes across different contexts.

9.3.6. Critical Reviews

Lastly, critical reviews aim to provide a critical evaluation and interpretive analysis of existing literature on a particular topic of interest to reveal strengths, weaknesses, contradictions, controversies, inconsistencies, and/or other important issues with respect to theories, hypotheses, research methods or results ( Baumeister & Leary, 1997 ; Kirkevold, 1997 ). Unlike other review types, critical reviews attempt to take a reflective account of the research that has been done in a particular area of interest, and assess its credibility by using appraisal instruments or critical interpretive methods. In this way, critical reviews attempt to constructively inform other scholars about the weaknesses of prior research and strengthen knowledge development by giving focus and direction to studies for further improvement ( Kirkevold, 1997 ).

Kitsiou, Paré, and Jaana (2013) provide an example of a critical review that assessed the methodological quality of prior systematic reviews of home telemonitoring studies for chronic patients. The authors conducted a comprehensive search on multiple databases to identify eligible reviews and subsequently used a validated instrument to conduct an in-depth quality appraisal. Results indicate that the majority of systematic reviews in this particular area suffer from important methodological flaws and biases that impair their internal validity and limit their usefulness for clinical and decision-making purposes. To this end, they provide a number of recommendations to strengthen knowledge development towards improving the design and execution of future reviews on home telemonitoring.

9.4. Summary

Table 9.1 outlines the main types of literature reviews that were described in the previous sub-sections and summarizes the main characteristics that distinguish one review type from another. It also includes key references to methodological guidelines and useful sources that can be used by eHealth scholars and researchers for planning and developing reviews.

Typology of Literature Reviews (adapted from Paré et al., 2015).

As shown in Table 9.1 , each review type addresses different kinds of research questions or objectives, which subsequently define and dictate the methods and approaches that need to be used to achieve the overarching goal(s) of the review. For example, in the case of narrative reviews, there is greater flexibility in searching and synthesizing articles ( Green et al., 2006 ). Researchers are often relatively free to use a diversity of approaches to search, identify, and select relevant scientific articles, describe their operational characteristics, present how the individual studies fit together, and formulate conclusions. On the other hand, systematic reviews are characterized by their high level of systematicity, rigour, and use of explicit methods, based on an “a priori” review plan that aims to minimize bias in the analysis and synthesis process (Higgins & Green, 2008). Some reviews are exploratory in nature (e.g., scoping/mapping reviews), whereas others may be conducted to discover patterns (e.g., descriptive reviews) or involve a synthesis approach that may include the critical analysis of prior research ( Paré et al., 2015 ). Hence, in order to select the most appropriate type of review, it is critical to know before embarking on a review project, why the research synthesis is conducted and what type of methods are best aligned with the pursued goals.

9.5. Concluding Remarks

In light of the increased use of evidence-based practice and research generating stronger evidence ( Grady et al., 2011 ; Lyden et al., 2013 ), review articles have become essential tools for summarizing, synthesizing, integrating or critically appraising prior knowledge in the eHealth field. As mentioned earlier, when rigorously conducted review articles represent powerful information sources for eHealth scholars and practitioners looking for state-of-the-art evidence. The typology of literature reviews we used herein will allow eHealth researchers, graduate students and practitioners to gain a better understanding of the similarities and differences between review types.

We must stress that this classification scheme does not privilege any specific type of review as being of higher quality than another ( Paré et al., 2015 ). As explained above, each type of review has its own strengths and limitations. Having said that, we realize that the methodological rigour of any review — be it qualitative, quantitative or mixed — is a critical aspect that should be considered seriously by prospective authors. In the present context, the notion of rigour refers to the reliability and validity of the review process described in section 9.2. For one thing, reliability is related to the reproducibility of the review process and steps, which is facilitated by a comprehensive documentation of the literature search process, extraction, coding and analysis performed in the review. Whether the search is comprehensive or not, whether it involves a methodical approach for data extraction and synthesis or not, it is important that the review documents in an explicit and transparent manner the steps and approach that were used in the process of its development. Next, validity characterizes the degree to which the review process was conducted appropriately. It goes beyond documentation and reflects decisions related to the selection of the sources, the search terms used, the period of time covered, the articles selected in the search, and the application of backward and forward searches ( vom Brocke et al., 2009 ). In short, the rigour of any review article is reflected by the explicitness of its methods (i.e., transparency) and the soundness of the approach used. We refer those interested in the concepts of rigour and quality to the work of Templier and Paré (2015) which offers a detailed set of methodological guidelines for conducting and evaluating various types of review articles.

To conclude, our main objective in this chapter was to demystify the various types of literature reviews that are central to the continuous development of the eHealth field. It is our hope that our descriptive account will serve as a valuable source for those conducting, evaluating or using reviews in this important and growing domain.

• Ammenwerth E., de Keizer N. An inventory of evaluation studies of information technology in health care. Trends in evaluation research, 1982-2002. International Journal of Medical Informatics. 2004; 44 (1):44–56. [ PubMed : 15778794 ]
• Anderson S., Allen P., Peckham S., Goodwin N. Asking the right questions: scoping studies in the commissioning of research on the organisation and delivery of health services. Health Research Policy and Systems. 2008; 6 (7):1–12. [ PMC free article : PMC2500008 ] [ PubMed : 18613961 ] [ CrossRef ]
• Archer N., Fevrier-Thomas U., Lokker C., McKibbon K. A., Straus S.E. Personal health records: a scoping review. Journal of American Medical Informatics Association. 2011; 18 (4):515–522. [ PMC free article : PMC3128401 ] [ PubMed : 21672914 ]
• Arksey H., O’Malley L. Scoping studies: towards a methodological framework. International Journal of Social Research Methodology. 2005; 8 (1):19–32.
• A systematic, tool-supported method for conducting literature reviews in information systems. Paper presented at the Proceedings of the 19th European Conference on Information Systems ( ecis 2011); June 9 to 11; Helsinki, Finland. 2011.
• Baumeister R. F., Leary M.R. Writing narrative literature reviews. Review of General Psychology. 1997; 1 (3):311–320.
• Becker L. A., Oxman A.D. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Overviews of reviews; pp. 607–631.
• Borenstein M., Hedges L., Higgins J., Rothstein H. Introduction to meta-analysis. Hoboken, nj : John Wiley & Sons Inc; 2009.
• Cook D. J., Mulrow C. D., Haynes B. Systematic reviews: Synthesis of best evidence for clinical decisions. Annals of Internal Medicine. 1997; 126 (5):376–380. [ PubMed : 9054282 ]
• Cooper H., Hedges L.V. In: The handbook of research synthesis and meta-analysis. 2nd ed. Cooper H., Hedges L. V., Valentine J. C., editors. New York: Russell Sage Foundation; 2009. Research synthesis as a scientific process; pp. 3–17.
• Cooper H. M. Organizing knowledge syntheses: A taxonomy of literature reviews. Knowledge in Society. 1988; 1 (1):104–126.
• Cronin P., Ryan F., Coughlan M. Undertaking a literature review: a step-by-step approach. British Journal of Nursing. 2008; 17 (1):38–43. [ PubMed : 18399395 ]
• Darlow S., Wen K.Y. Development testing of mobile health interventions for cancer patient self-management: A review. Health Informatics Journal. 2015 (online before print). [ PubMed : 25916831 ] [ CrossRef ]
• Daudt H. M., van Mossel C., Scott S.J. Enhancing the scoping study methodology: a large, inter-professional team’s experience with Arksey and O’Malley’s framework. bmc Medical Research Methodology. 2013; 13 :48. [ PMC free article : PMC3614526 ] [ PubMed : 23522333 ] [ CrossRef ]
• Davies P. The relevance of systematic reviews to educational policy and practice. Oxford Review of Education. 2000; 26 (3-4):365–378.
• Deeks J. J., Higgins J. P. T., Altman D.G. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Analysing data and undertaking meta-analyses; pp. 243–296.
• Deshazo J. P., Lavallie D. L., Wolf F.M. Publication trends in the medical informatics literature: 20 years of “Medical Informatics” in mesh . bmc Medical Informatics and Decision Making. 2009; 9 :7. [ PMC free article : PMC2652453 ] [ PubMed : 19159472 ] [ CrossRef ]
• Dixon-Woods M., Agarwal S., Jones D., Young B., Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods. Journal of Health Services Research and Policy. 2005; 10 (1):45–53. [ PubMed : 15667704 ]
• Finfgeld-Connett D., Johnson E.D. Literature search strategies for conducting knowledge-building and theory-generating qualitative systematic reviews. Journal of Advanced Nursing. 2013; 69 (1):194–204. [ PMC free article : PMC3424349 ] [ PubMed : 22591030 ]
• Grady B., Myers K. M., Nelson E. L., Belz N., Bennett L., Carnahan L. … Guidelines Working Group. Evidence-based practice for telemental health. Telemedicine Journal and E Health. 2011; 17 (2):131–148. [ PubMed : 21385026 ]
• Green B. N., Johnson C. D., Adams A. Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. Journal of Chiropractic Medicine. 2006; 5 (3):101–117. [ PMC free article : PMC2647067 ] [ PubMed : 19674681 ]
• Greenhalgh T., Wong G., Westhorp G., Pawson R. Protocol–realist and meta-narrative evidence synthesis: evolving standards ( rameses ). bmc Medical Research Methodology. 2011; 11 :115. [ PMC free article : PMC3173389 ] [ PubMed : 21843376 ]
• Gurol-Urganci I., de Jongh T., Vodopivec-Jamsek V., Atun R., Car J. Mobile phone messaging reminders for attendance at healthcare appointments. Cochrane Database System Review. 2013; 12 cd 007458. [ PMC free article : PMC6485985 ] [ PubMed : 24310741 ] [ CrossRef ]
• Hart C. Doing a literature review: Releasing the social science research imagination. London: SAGE Publications; 1998.
• Higgins J. P. T., Green S., editors. Cochrane handbook for systematic reviews of interventions: Cochrane book series. Hoboken, nj : Wiley-Blackwell; 2008.
• Jesson J., Matheson L., Lacey F.M. Doing your literature review: traditional and systematic techniques. Los Angeles & London: SAGE Publications; 2011.
• King W. R., He J. Understanding the role and methods of meta-analysis in IS research. Communications of the Association for Information Systems. 2005; 16 :1.
• Kirkevold M. Integrative nursing research — an important strategy to further the development of nursing science and nursing practice. Journal of Advanced Nursing. 1997; 25 (5):977–984. [ PubMed : 9147203 ]
• Kitchenham B., Charters S. ebse Technical Report Version 2.3. Keele & Durham. uk : Keele University & University of Durham; 2007. Guidelines for performing systematic literature reviews in software engineering.
• Kitsiou S., Paré G., Jaana M. Systematic reviews and meta-analyses of home telemonitoring interventions for patients with chronic diseases: a critical assessment of their methodological quality. Journal of Medical Internet Research. 2013; 15 (7):e150. [ PMC free article : PMC3785977 ] [ PubMed : 23880072 ]
• Kitsiou S., Paré G., Jaana M. Effects of home telemonitoring interventions on patients with chronic heart failure: an overview of systematic reviews. Journal of Medical Internet Research. 2015; 17 (3):e63. [ PMC free article : PMC4376138 ] [ PubMed : 25768664 ]
• Levac D., Colquhoun H., O’Brien K. K. Scoping studies: advancing the methodology. Implementation Science. 2010; 5 (1):69. [ PMC free article : PMC2954944 ] [ PubMed : 20854677 ]
• Levy Y., Ellis T.J. A systems approach to conduct an effective literature review in support of information systems research. Informing Science. 2006; 9 :181–211.
• Liberati A., Altman D. G., Tetzlaff J., Mulrow C., Gøtzsche P. C., Ioannidis J. P. A. et al. Moher D. The prisma statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. Annals of Internal Medicine. 2009; 151 (4):W-65. [ PubMed : 19622512 ]
• Lyden J. R., Zickmund S. L., Bhargava T. D., Bryce C. L., Conroy M. B., Fischer G. S. et al. McTigue K. M. Implementing health information technology in a patient-centered manner: Patient experiences with an online evidence-based lifestyle intervention. Journal for Healthcare Quality. 2013; 35 (5):47–57. [ PubMed : 24004039 ]
• Mickan S., Atherton H., Roberts N. W., Heneghan C., Tilson J.K. Use of handheld computers in clinical practice: a systematic review. bmc Medical Informatics and Decision Making. 2014; 14 :56. [ PMC free article : PMC4099138 ] [ PubMed : 24998515 ]
• Moher D. The problem of duplicate systematic reviews. British Medical Journal. 2013; 347 (5040) [ PubMed : 23945367 ] [ CrossRef ]
• Montori V. M., Wilczynski N. L., Morgan D., Haynes R. B., Hedges T. Systematic reviews: a cross-sectional study of location and citation counts. bmc Medicine. 2003; 1 :2. [ PMC free article : PMC281591 ] [ PubMed : 14633274 ]
• Mulrow C. D. The medical review article: state of the science. Annals of Internal Medicine. 1987; 106 (3):485–488. [ PubMed : 3813259 ] [ CrossRef ]
• Evidence-based information systems: A decade later. Proceedings of the European Conference on Information Systems ; 2011. Retrieved from http://aisel ​.aisnet.org/cgi/viewcontent ​.cgi?article ​=1221&context ​=ecis2011 .
• Okoli C., Schabram K. A guide to conducting a systematic literature review of information systems research. ssrn Electronic Journal. 2010
• Otte-Trojel T., de Bont A., Rundall T. G., van de Klundert J. How outcomes are achieved through patient portals: a realist review. Journal of American Medical Informatics Association. 2014; 21 (4):751–757. [ PMC free article : PMC4078283 ] [ PubMed : 24503882 ]
• Paré G., Trudel M.-C., Jaana M., Kitsiou S. Synthesizing information systems knowledge: A typology of literature reviews. Information & Management. 2015; 52 (2):183–199.
• Patsopoulos N. A., Analatos A. A., Ioannidis J.P. A. Relative citation impact of various study designs in the health sciences. Journal of the American Medical Association. 2005; 293 (19):2362–2366. [ PubMed : 15900006 ]
• Paul M. M., Greene C. M., Newton-Dame R., Thorpe L. E., Perlman S. E., McVeigh K. H., Gourevitch M.N. The state of population health surveillance using electronic health records: A narrative review. Population Health Management. 2015; 18 (3):209–216. [ PubMed : 25608033 ]
• Pawson R. Evidence-based policy: a realist perspective. London: SAGE Publications; 2006.
• Pawson R., Greenhalgh T., Harvey G., Walshe K. Realist review—a new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy. 2005; 10 (Suppl 1):21–34. [ PubMed : 16053581 ]
• Petersen K., Vakkalanka S., Kuzniarz L. Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology. 2015; 64 :1–18.
• Petticrew M., Roberts H. Systematic reviews in the social sciences: A practical guide. Malden, ma : Blackwell Publishing Co; 2006.
• Rousseau D. M., Manning J., Denyer D. Evidence in management and organizational science: Assembling the field’s full weight of scientific knowledge through syntheses. The Academy of Management Annals. 2008; 2 (1):475–515.
• Rowe F. What literature review is not: diversity, boundaries and recommendations. European Journal of Information Systems. 2014; 23 (3):241–255.
• Shea B. J., Hamel C., Wells G. A., Bouter L. M., Kristjansson E., Grimshaw J. et al. Boers M. amstar is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology. 2009; 62 (10):1013–1020. [ PubMed : 19230606 ]
• Shepperd S., Lewin S., Straus S., Clarke M., Eccles M. P., Fitzpatrick R. et al. Sheikh A. Can we systematically review studies that evaluate complex interventions? PLoS Medicine. 2009; 6 (8):e1000086. [ PMC free article : PMC2717209 ] [ PubMed : 19668360 ]
• Silva B. M., Rodrigues J. J., de la Torre Díez I., López-Coronado M., Saleem K. Mobile-health: A review of current state in 2015. Journal of Biomedical Informatics. 2015; 56 :265–272. [ PubMed : 26071682 ]
• Smith V., Devane D., Begley C., Clarke M. Methodology in conducting a systematic review of systematic reviews of healthcare interventions. bmc Medical Research Methodology. 2011; 11 (1):15. [ PMC free article : PMC3039637 ] [ PubMed : 21291558 ]
• Sylvester A., Tate M., Johnstone D. Beyond synthesis: re-presenting heterogeneous research literature. Behaviour & Information Technology. 2013; 32 (12):1199–1215.
• Templier M., Paré G. A framework for guiding and evaluating literature reviews. Communications of the Association for Information Systems. 2015; 37 (6):112–137.
• Thomas J., Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. bmc Medical Research Methodology. 2008; 8 (1):45. [ PMC free article : PMC2478656 ] [ PubMed : 18616818 ]
• Reconstructing the giant: on the importance of rigour in documenting the literature search process. Paper presented at the Proceedings of the 17th European Conference on Information Systems ( ecis 2009); Verona, Italy. 2009.
• Webster J., Watson R.T. Analyzing the past to prepare for the future: Writing a literature review. Management Information Systems Quarterly. 2002; 26 (2):11.
• Whitlock E. P., Lin J. S., Chou R., Shekelle P., Robinson K.A. Using existing systematic reviews in complex systematic reviews. Annals of Internal Medicine. 2008; 148 (10):776–782. [ PubMed : 18490690 ]

This publication is licensed under a Creative Commons License, Attribution-Noncommercial 4.0 International License (CC BY-NC 4.0): see https://creativecommons.org/licenses/by-nc/4.0/

• Cite this Page Paré G, Kitsiou S. Chapter 9 Methods for Literature Reviews. In: Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.
• PDF version of this title (4.5M)
• Disable Glossary Links

In this Page

• Introduction
• Overview of the Literature Review Process and Steps
• Types of Review Articles and Brief Illustrations
• Concluding Remarks

Related information

• PMC PubMed Central citations
• PubMed Links to PubMed

Recent Activity

• Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Ev... Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Evidence-based Approach

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

Which review is that? A guide to review types

• Which review is that?
• Review Comparison Chart
• Decision Tool
• Critical Review
• Integrative Review
• Narrative Review
• State of the Art Review
• Narrative Summary
• Systematic Review
• Meta-analysis
• Comparative Effectiveness Review
• Diagnostic Systematic Review
• Network Meta-analysis
• Prognostic Review
• Psychometric Review
• Review of Economic Evaluations
• Systematic Review of Epidemiology Studies
• Living Systematic Reviews
• Umbrella Review
• Review of Reviews
• Rapid Review
• Rapid Evidence Assessment
• Rapid Realist Review
• Qualitative Evidence Synthesis
• Qualitative Interpretive Meta-synthesis
• Qualitative Meta-synthesis
• Qualitative Research Synthesis
• Framework Synthesis - Best-fit Framework Synthesis
• Meta-aggregation
• Meta-ethnography
• Meta-interpretation
• Meta-narrative Review
• Meta-summary
• Thematic Synthesis
• Mixed Methods Synthesis
• Narrative Synthesis
• Bayesian Meta-analysis
• EPPI-Centre Review
• Critical Interpretive Synthesis
• Realist Synthesis - Realist Review
• Scoping Review
• Mapping Review
• Systematised Review
• Concept Synthesis
• Expert Opinion - Policy Review
• Technology Assessment Review

Methodological Review

• Systematic Search and Review

A methodological review is a type of systematic secondary research (i.e., research synthesis) which focuses on summarising the state-of-the-art methodological practices of research in a substantive field or topic" (Chong et al, 2021).

Methodological reviews "can be performed to examine any methodological issues relating to the design, conduct and review of research studies and also evidence syntheses". Munn et al, 2018)

Further Reading/Resources

Clarke, M., Oxman, A. D., Paulsen, E., Higgins, J. P. T., & Green, S. (2011). Appendix A: Guide to the contents of a Cochrane Methodology protocol and review. Cochrane Handbook for systematic reviews of interventions . Full Text PDF

Aguinis, H., Ramani, R. S., & Alabduljader, N. (2023). Best-Practice Recommendations for Producers, Evaluators, and Users of Methodological Literature Reviews. Organizational Research Methods, 26(1), 46-76. https://doi.org/10.1177/1094428120943281 Full Text

Jha, C. K., & Kolekar, M. H. (2021). Electrocardiogram data compression techniques for cardiac healthcare systems: A methodological review. IRBM . Full Text

References Munn, Z., Stern, C., Aromataris, E., Lockwood, C., & Jordan, Z. (2018). What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences. BMC medical research methodology , 18 (1), 1-9. Full Text Chong, S. W., & Reinders, H. (2021). A methodological review of qualitative research syntheses in CALL: The state-of-the-art. System , 103 , 102646. Full Text

• << Previous: Technology Assessment Review
• Next: Systematic Search and Review >>
• Last Updated: Jun 5, 2024 10:35 AM
• URL: https://unimelb.libguides.com/whichreview

• Open access
• Published: 03 June 2024

Continue nursing education: an action research study on the implementation of a nursing training program using the Holton Learning Transfer System Inventory

• MingYan Shen 1 , 2 &
• ZhiXian Feng 1 , 2  

BMC Medical Education volume  24 , Article number:  610 ( 2024 ) Cite this article

Metrics details

To address the gap in effective nursing training for quality management, this study aims to implement and assess a nursing training program based on the Holton Learning Transfer System Inventory, utilizing action research to enhance the practicality and effectiveness of training outcomes.

The study involved the formation of a dedicated training team, with program development informed by an extensive situation analysis and literature review. Key focus areas included motivation to transfer, learning environment, and transfer design. The program was implemented in a structured four-step process: plan, action, observation, reflection.

Over a 11-month period, 22 nurses completed 14 h of theoretical training and 18 h of practical training with a 100% attendance rate and 97.75% satisfaction rate. The nursing team successfully led and completed 22 quality improvement projects, attaining a practical level of application. Quality management implementation difficulties, literature review, current situation analysis, cause analysis, formulation of plans, implementation plans, and report writing showed significant improvement and statistical significance after training.

The study confirms the efficacy of action research guided by Holton’s model in significantly enhancing the capabilities of nursing staff in executing quality improvement projects, thereby improving the overall quality of nursing training. Future research should focus on refining the training program through long-term observation, developing a multidimensional evaluation index system, exploring training experiences qualitatively, and investigating the personality characteristics of nurses to enhance training transfer effects.

Peer Review reports

Introduction

The “Medical Quality Management Measures“ [ 1 ] and “Accreditation Standards for Tertiary Hospitals (2020 Edition)” [ 2 ] both emphasize the importance of using quality management tools in medical institutions to carry out effective quality management [ 3 ]. However, there is a notable gap in translating theoretical training into effective, practical application in clinical settings [ 4 ]. This gap is further highlighted in the context of healthcare quality management, as evidenced in studies [ 5 ] which demonstrate the universality of these challenges across healthcare systems worldwide.

Addressing this issue, contemporary literature calls for innovative and effective training methods that transition from passive knowledge acquisition to active skill application [ 6 ]. The Holton Learning Transfer System Inventory [ 7 ] provides a framework focusing on key factors such as motivation, learning environment, and transfer design [ 7 , 8 , 9 ]. This study aims to implement a nursing training program based on the Holton model, using an action research methodology to bridge the theoretical-practical gap in nursing education.

Quality management training for clinical nurses has predominantly been characterized by short-term theoretical lectures, a format that often fails to foster deep engagement and lasting awareness among nursing personnel [ 10 ]. The Quality Indicator Project in Taiwan’s nursing sector, operational for over a decade, demonstrates the effective use of collective intelligence and scientific methodologies to address these challenges [ 11 ]. The proposed study responds to the need for training programs that not only impart knowledge but also ensure the practical application of skills in real-world nursing settings, thereby contributing to transformative changes within the healthcare system [ 12 ].

In April 2021, the Nursing Education Department of our hospital launched a quality improvement project training program for nurses. The initiation of this study is underpinned by the evident disconnect between theoretical training and the practical challenges nurses face in implementing quality management initiatives, a gap also identified in the work [ 13 ]. By exploring the efficacy of the Holton Learning Transfer System Inventory, this study seeks to enhance the practical application of training and significantly contribute to the field of nursing education and quality management in healthcare.

Developing a nursing training program with the Holton Learning Transfer System Inventory

Establishing a research team and assigning roles.

There are 10 members in the group who serve as both researchers and participants, aiming to investigate training process issues and solutions. The roles within the group are as follows: the deputy dean in charge of nursing is responsible for program review and organizational support, integrating learning transfer principles in different settings [ 14 ]; the deputy director of the Nursing Education Department handles the design and implementation of the training program, utilizing double-loop learning for training transfer [ 15 ]; the deputy director of the Nursing Department oversees quality control and project evaluation, ensuring integration of evidence-based practices and technology [ 16 ] and the deputy director of the Quality Management Office provides methodological guidance. The remaining members consist of 4 faculty members possessing significant university teaching experience and practical expertise in quality control projects, and 2 additional members who are jointly responsible for educational affairs, data collection, and analysis. Additionally, to ensure comprehensive pedagogical guidance in this training, professors specializing in nursing pedagogy have been specifically invited to provide expertise on educational methodology.

Current situation survey

Based on the Holton Learning Transfer System Inventory (refer to Fig.  1 ), the appropriate levels of Motivation to Improve Work Through Learning (MTIWL), learning environment, and transfer design are crucial in facilitating changes in individual performance, thereby influencing organizational outcomes [ 17 , 18 ]. Motivation to Improve Work Through Learning (MTIWL) is closely linked to expectation theory, fairness theory, and goal-setting theory, significantly impacting the positive transfer of training [ 19 ]. Learning environment encompasses environmental factors that either hinder or promote the application of learned knowledge in actual work settings [ 20 ]. Transfer design, as a pivotal component, includes training program design and organizational planning.

To conduct the survey, the research team retrieved 26 quality improvement reports from the nursing quality information management system, which were generated by nursing units in 2020. A checklist was formulated, and a retrospective evaluation was conducted across eight aspects, namely, team participation, topic selection feasibility, method accuracy, indicator scientificity, program implementation rate, effect maintenance, and promotion and application. Methods employed in the evaluation process included report analysis, on-site tracking, personnel interviews, and data review within the quality information management system [ 21 ]. From the perspective of motivation [ 22 ], learning environment [ 23 ], and transfer design, a total of 14 influencing factors were identified. These factors serve as a reference for designing the training plan and encompass the following aspects: lack of awareness regarding importance, low willingness to participate in training, unclear understanding of individual career development, absence of incentive mechanisms, absence of a scientific training organization model, lack of a training quality management model, inadequate literature retrieval skills and support, insufficient availability of practical training materials and resources, incomplete mastery of post-training methods, lack of cultural construction plans, suboptimal communication methods and venues, weak internal organizational atmosphere, inadequate leadership support, and absence of platforms and mechanisms for promoting and applying learned knowledge.

Learning Transfer System Inventory

Development of the training program using the 4W1H approach

Drawing upon Holton’s Learning Transfer System Inventory and the hospital training transfer model diagram, a comprehensive training outline was formulated for the training program [ 24 , 25 ]. The following components were considered:

(1) Training Participants (Who): The training is open for voluntary registration to individuals with an undergraduate degree or above, specifically targeting head nurses, responsible team leaders, and core members of the hospital-level nursing quality control team. Former members who have participated in quality improvement projects such as Plan-Do-Check-Act Circle (PDCA) or Quality control circle (QCC) are also eligible.

(2) Training Objectives (Why): At the individual level, the objectives include enhancing the understanding of quality management concepts, improving the cognitive level and application abilities of project improvement methods, and acquiring the necessary skills for nursing quality improvement project. At the team level, the aim is to enhance effective communication among team members and elevate the overall quality of communication. Moreover, the training seeks to facilitate collaborative efforts in improving the existing nursing quality management system and processes. At the operational level, participants are expected to gain the competence to design, implement, and manage nursing quality improvement project initiatives. Following the training, participants will lead and successfully complete a nursing quality improvement project, which will undergo a rigorous audit.

(3) Training Duration (When): The training program spans a duration of 11 months.

(4) Training Content (What): The program consists of 14 h of theoretical courses and 18 h of practical training sessions, as detailed in Table  1 .

(5) Quality Management Approach (How): To ensure quality throughout the training process, two team members are assigned to monitor the entire training journey. This encompasses evaluating whether quality awareness education, quality management knowledge, and professional skills training are adequately covered. Additionally, attention is given to participants’ learning motivation, the emphasis placed on active participation in training methods, support from hospital management and relevant departments, as well as participants’ satisfaction and assessment results. Please refer to Fig.  2 for a visual representation.

In-house training model from Holton Learning Transfer

Implementation of the nursing project training program using the action research method

The first cycle (april 2021).

In the initial cycle, a total of 22 nurses were included as training participants after a self-registration process and qualification review. The criteria used to select these participants, elaborated in Section Development of the training program using the 4W1H approach, ‘Development of the Training Program,’ were meticulously crafted to capture a broad spectrum of experience, expertise, and functional roles within our hospital’s nursing staff. The primary focus was to investigate their learning motivation. The cycle comprised the following key activities:

(1) Training Objectives: The focus was on understanding the learning motivation of the participating nurses.

(2) Theoretical Training Sessions: A total of 7 theoretical training sessions, spanning 14 class hours, were completed. The contents covered various aspects, including an overview of nursing quality improvement projects, methods for selecting project topics, common tools used in nursing quality improvement projects, effective leadership strategies to promote project practices, literature retrieval and evaluation methods, formulation and promotion of project plans, and writing project reports. Detailed course information, including the title, content, and class hours, is listed in Table  1 . At the end of each training session, a course satisfaction survey was conducted.

(3) Assessment and Reporting: Following the completion of the 7 training sessions, a theoretical assessment on quality management knowledge was conducted. Additionally, nurses were organized to present their plans for special projects to be carried out during the training. Several issues were identified during this cycle:

Incomplete Literature Review Skills: Compared to other quality control tools, nursing quality improvement project places more emphasis on the scientific construction of project plans. The theoretical evaluation and interviews with nurses highlighted the incomplete and challenging nature of their literature review skills.

Insufficient Leadership: Among the participants, 6 individuals were not head nurses, which resulted in a lack of adequate leadership for their respective projects.

Learning environment and Support: The learning environment, as well as the support from hospital management and relevant departments, needed to be strengthened.

Second cycle (may-october 2021)

In response to the issues identified during the first cycle, our approach in the second cycle was both corrective and adaptive, focusing on immediate issues while also setting the stage for addressing any emerging challenges. The team members actively implemented improvements during the second cycle. The key actions taken were as follows:

(1) Establishing an Enabling Organizational Environment: The quality management department took the lead, and multiple departments collaborated in conducting the “Hospital Safety and Quality Red May” activity. This initiative aimed to enhance the overall quality improvement atmosphere within the hospital. Themed articles were also shared through the hospital’s WeChat public account.

(2) Salon-style Training Format: The training sessions were conducted in the form of salons, held in a meeting room specifically prepared for this purpose. The room was arranged with a round table, warm yellow lighting, green plants, and a coffee bar, creating a conducive environment for free, democratic, and equal communication among the participants. The salon topics included revising project topic selection, conducting current situation investigations, facilitating communication and guidance for literature reviews, formulating improvement plans, implementing those plans, and writing project reports. After the projects were presented, quality management experts provided comments and analysis, promoting the transformation of training outcomes from mere memory and understanding to higher-level abilities such as application, analysis, and creativity.

(3) Continuous Support Services: Various support services were provided to ensure ongoing assistance. This included assigning nursing postgraduates to aid in literature retrieval and evaluation. Project team members also provided on-site guidance and support, actively engaging in the project improvement process to facilitate training transfer.

(4) Emphasis on Spiritual Encouragement: The Vice President of Nursing Department actively participated in the salons and provided feedback on each occasion. Moreover, the President of the hospital consistently commended the training efforts during the weekly hospital meetings.

Issues identified in this cycle

(1) Inconsistent Ability to Write Project Documents: The proficiency in writing project documents for project improvement varied among participants, and there was a lack of standardized evaluation criteria. This issue had the potential to impact the quality of project dissemination.

(2) Lack of Clarity Regarding the Platform and Mechanism for Training Result Transfer: The platform and mechanisms for transferring training results were not clearly defined, posing a challenge in effectively sharing and disseminating the outcomes of the training.

The third cycle (November 2021-march 2022)

During the third cycle, the following initiatives were undertaken.

(1) Utilizing the “Reporting Standards for Quality Improvement Research (SQUIRE)”, as issued by the US Health Care Promotion Research, to provide guidance for students in writing nursing project improvement reports.

(2) Organizing a hospital-level nursing quality improvement project report meeting to acknowledge and commend outstanding projects.

(3) Compiling the “Compilation of Nursing Quality Improvement Projects” for dissemination and exchange among nurses both within and outside the hospital.

(4) Addressing the issue of inadequate management of indicator monitoring data, a hospital-level quality index management platform was developed. The main evaluation data from the 22 projects were entered into this platform, allowing for continuous monitoring and timely intervention.

Effect evaluation

To assess the efficacy of the training, a diverse set of evaluation metrics, encompassing both outcome and process measures [ 26 ]. These measures can be structured around the four-level training evaluation framework proposed by Donald Kirkpatrick [ 27 ].

Process evaluation

Evaluation method.

To assess the commitment and support within the organization, the process evaluation involved recording the proportion of nurses’ classroom participation time and the presence of leaders during each training session. Additionally, a satisfaction survey was conducted after the training to assess various aspects such as venue layout, time arrangement, training methods, lecturer professionalism, content practicality, and interaction. On-site recycling statistics were also collected for project evaluation purposes.

Evaluation results the results of the process evaluation are as follows

Nurse training participation rate: 100%.

Training satisfaction rate (average): 97.75%.

Proportion of nurses’ participation time in theoretical training sessions (average): 36.88%.

Proportion of nurses’ participation time in salon training sessions (average): 74.23%.

Attendance rate of school-level leaders: 100%.

Results evaluation

Assessment of theoretical knowledge of quality management.

To evaluate the effectiveness in enhancing the trainees’ theoretical knowledge of quality management, the research team conducted assessments before the training, after the first round of implementation, and after the third round of implementation. Assessments to evaluate the effectiveness of the training program were conducted immediately following the first round of implementation, and after the third round of implementation. This dual-timing approach was designed to evaluate both the immediate impact of the training and its sustained effects over time, addressing potential influences of memory decay on the study results. The assessment consisted of a 60-minute examination with different question types, including 30 multiple-choice questions (2 points each), 2 short-answer questions (10 points each), and 1 comprehensive analysis question (20 points). The maximum score achievable was 100 points.

The assessment results are as follows:

Before training (average): 75.05 points.

After the first round of implementation (average): 82.18 points.

After the third round of implementation (average): 90.82 points.

Assessment of difficulty in quality management project implementation

To assess the difficulty of implementing quality management projects, the trainees completed the “Quality Management Project Implementation Difficulty Assessment Form” before and after the training. They self-evaluated 10 aspects using a 5-point scale, with 5 indicating the most difficult and 1 indicating no difficulty. The evaluation results before and after implementation are presented in Table  2 .

Statistically significant differences were found in the following items: literature review, current situation analysis, cause analysis, plan formulation, implementation plan, and report writing. This indicates that the training significantly enhanced the nurses’ confidence and ability to tackle practical challenges.

Evaluation of transfer effect

To assess how effectively the training translated into practical applications. The implementation of the 22 quality improvement projects was evaluated using the application hierarchy analysis table. The specific results are presented in Table  3 .

In addition, the “Nursing Project Guidance Manual” and “Compilation of Nursing Project Improvement Projects” were compiled and distributed to the hospital’s management staff, nurses, and four collaborating hospitals, receiving positive feedback. The lecture titled “Improving Nurses’ Project Improvement Ability Based on the Training Transfer Theory Model” shared experiences with colleagues both within and outside the province in national and provincial teaching sessions in 2022. Furthermore, four papers were published on the subject.

The effectiveness of the training program based on the Holton Learning transfer System Inventory

The level of refined management in hospitals is closely tied to the quality management awareness and skills of frontline medical staff. Quality management training plays a crucial role in improving patient safety management and fostering a culture of quality and safety. Continuous quality improvement is an integral part of nursing management, ensuring that patients receive high-quality and safe nursing care. Compared to the focus of existing literature on the individual performance improvements following nursing training programs [ 28 , 29 , 30 ], our study expands the evaluation framework to include organizational performance metrics. Our research underscores a significantly higher level of organizational engagement as evidenced by the 100% attendance rate of school-level leaders. The publication of four papers related to this study highlights not only individual performance achievements but also significantly broadens the hospital organization’s impact on quality management, leading to meaningful organizational outcomes.

Moreover, our initiative to incorporate indicators of quality projects into a hospital-level evaluation index system post-training signifies a pivotal move towards integrating quality improvement practices into the very fabric of organizational operations. In training programs, it is essential not only to achieve near-transfer, but also to ensure that nurses continuously apply the acquired management skills to their clinical work, thereby enhancing quality, developing their professional value, and improving organizational performance. The Holton learning Transfer System Inventory provides valuable guidance on how to implement training programs and evaluate their training effect.

This study adopts the training transfer model as a framework to explore the mechanisms of “how training works” rather than simply assessing “whether training works [ 31 ].” By examining factors such as Motivation to Improve Work Through Learning (MTIWL), learning environment, and transfer design, the current situation is analyzed, underlying reasons are identified, and relevant literature is reviewed to develop and implement training programs based on the results of a needs survey. While individual transfer motivation originates from within the individual, it is influenced by the transfer atmosphere and design. By revising the nurse promotion system and performance management system and aligning them with career development, nurses’ motivation to participate and engage in active learning has significantly increased [ 32 ]. At the learning environment level, enhancing the training effect involves improving factors such as stimulation and response that correspond to the actual work environment [ 33 ]. This project has garnered attention and support from hospital-level leaders, particularly the nursing dean who regularly visits the training site to provide guidance, which serves as invaluable recognition. Timely publicity and recognition of exemplary project improvement initiatives have also increased awareness and understanding of project knowledge among doctors and nurses, fostering a stronger quality improvement atmosphere within the team.

Transfer design, the most critical component for systematic learning and mastery of quality management tools, is achieved through theoretical lectures, salon exchanges, and project-based training. These approaches allow nurses to gain hands-on experience in project improvement under the guidance of instructors. Throughout the project, nurses connect project management knowledge and skills with practical application, enabling personal growth and organizational development through problem-solving in real work scenarios. Finally, a comprehensive evaluation of the training program was conducted, including assessments of theoretical knowledge, perception of management challenges, and project quality. The results showed high satisfaction among nurses, with a satisfaction rate of 97.75%. The proportion of nurses’ participation time in theoretical and practical training classes was 36.88% and 74.23%, respectively. The average score for theoretical knowledge of quality management increased from 75.05 to 90.82. There was also a significant improvement in the evaluation of the implementation difficulties of quality management projects. Moreover, 22 nurses successfully led the completion of one project improvement project, with six projects focusing on preventing the COVID-19 pandemic, demonstrating valuable crisis response practices.

Action research helps to ensure the quality of organizational management of training

Well-organized training is the basis for ensuring the scientific and standardized development of nursing project improvement activities. According to the survey results of the current situation, there is a lot of room for improvement in the training quality; since it is the first time to apply the Holton training transfer model to the improvement training process of nurses in the hospital, in order to allow the nurses to have sufficient time to implement and evaluate the improvement project, the total training time Set at 11 months, a strong methodology is required to ensure training management during this period. Action research is a research method that closely combines research with solving practical problems in work. It is a research method aimed at solving practical problems through self-reflective exploration in realistic situations, emphasizing the participation of researchers and researchees. Practice, find problems in practice, and adjust the plan in a timely manner. According to the implementation of the first round, it was found that nurses had insufficient literature review skills, insufficient leadership, and lack of support from hospital management and related departments [ 32 ]. In the second round, the training courses were carried out in the form of salons. The project team members went deep into the project to improve on-site guidance, arranged graduate students to assist in document retrieval and evaluation, and promoted the transfer of training; the “Hospital Safety and Quality Red May” activity was carried out, and the vice president of nursing Regularly participate in the salon and make comments. The problems exposed after this round of implementation are the low quality of the project improvement project document, and the unclear platform and mechanism for the transfer of training results. In the third round, the “Reporting Standards for Quality Improvement Research (SQUIRE)” was used to standardize the writing of the report [ 33 ], and the “Compilation of Nursing Project Improvement Projects” was completed, and the main evaluation data of 22 projects were entered into the hospital-level quality index management platform for continuous monitoring and intervention. As of May 2022, the effect maintenance data of each project has reached the target value. It can not only produce useful improvement projects, but also help to promote the dissemination and penetration of quality awareness.

Future research directions

Drawing on the Holton training evaluation model, this study implemented nurse quality improvement project training using action research methodology, resulting in a successful exploration practice, and achieving positive transfer effects. To further advance this research area, the following future research directions are recommended:

Summarize the experiences gained from this action research training and continue to refine and enhance the training program. Through ongoing practice, reflection, and refinement in subsequent training sessions, long-term observation of the transfer effects can be conducted to establish an effective experiential model that can serve as a reference for future initiatives.

Develop a multidimensional evaluation index system for assessing transfer effects. A comprehensive framework that captures various dimensions of transfer, such as knowledge application, skill utilization, and behavior change, should be established. This will enable a more holistic and accurate assessment of the training program’s impact on the participants and the organization.

Conduct qualitative research to explore the training experiences of nurses. By gathering in-depth insights through interviews or focus group discussions, a deeper understanding of the nurses’ perceptions, challenges, and facilitators of training transfer can be obtained. This qualitative exploration will provide valuable information to further refine and tailor the training program to meet the specific needs and preferences of the nurses.

Investigate the personality characteristics of nurses who actively engage in training transfer and consider developing them as internal trainers. By identifying and cultivating nurses with a proactive attitude and a strong inclination towards knowledge application and skill development, the organization can enhance employee participation and initiative. These internal trainers can play a crucial role in motivating their colleagues and driving the transfer of training outcomes into daily practice.

By pursuing these future research directions, the field of healthcare and nursing care can continue to advance in optimizing training programs, enhancing transfer effects, and ultimately improving the quality of care and patient outcomes.

Limitations

The research was conducted with a cohort of 22 nurses and a 10-member research team from Grade 3, Class A hospitals in China Southeast. This specific composition and the relatively small sample size may affect the generalizability of our findings. The experiences and outcomes observed in this study might not fully encapsulate the diverse challenges and environments encountered by nursing professionals in varying healthcare settings. The significant improvements noted in the capabilities of the participating nursing staff underscore the potential impact of the training program. However, the study’s focus on a specific demographic—nurses from high-grade hospitals in a developed urban center—may limit the external validity of the findings.

Conclusions

This study affirms the efficacy of the Holton Learning Transfer System Inventory-based training program, coupled with action research, in significantly advancing nursing quality management practices. The strategic incorporation of motivation to improve work through learning, an enriched learning environment, and thoughtful transfer design significantly boosted the nurses’ engagement, knowledge acquisition, and practical application of quality management tools in their clinical work.

It highlights the importance of continuous learning, organizational support, and methodological flexibility in achieving sustainable improvements in healthcare quality and safety. Future endeavors should aim to expand the scope of this training model to diverse nursing contexts and evaluate its long-term impact on organizational performance and patient care outcomes.

Data availability

The datasets generated and/or analyzed during the current study are not publicly available due to hospital policy but are available from the corresponding author on reasonable request.

National Health and Family Planning Commission. Medical Quality Management Measures. Published 2016. Accessed September 19, 2018. http://www.nhc.gov.cn/fzs/s3576/201610/ae125f28eef24ca7aac57c8ec530c6d2.shtml .

National Health Commission of the People’s Republic of China. Notice of the National Health Commission on Printing and Distributing the Standards for the Review of Third-level Hospitals (2020 Edition) [EB/OL]. (2021-10-21) [2022-07-17].

Grossu-Leibovica D, Kalkis H. Total quality management tools and techniques for improving service quality and client satisfaction in the healthcare environment: a qualitative systematic review. Manage Sci Lett. 2023;13(2):118–23. https://doi.org/10.1051/shsconf/202213102009 .

Article   Google Scholar  

Shade L, Reeves K, Rees J, Hendrickson L, Halladay J, Dolor RJ, Bray P, Tapp H. Research nurses as practice facilitators to disseminate an asthma shared decision making intervention. BMC Nurs. 2020;19:40. https://doi.org/10.1186/s12912-020-00414-0 .

Ali J, Jusoh A, Abbas A, Nor K. Global trends of Service Quality in Healthcare: a bibliometric analysis of Scopus Database. J Contemp Issues Bus Government. 2021;27:2917.

Google Scholar  

Ahmed FA, Choudhary RA, Khan H, Ayub F, Hassan SSU, Munir T, Asif F, Ajani K, Jaffer M, Tharani Z, Aboumatar HJ, Haider A, Latif A. Incorporating Patient Safety and Quality Course into the nursing curriculum: an Assessment of Student gains. J Patient Saf. 2023;19(6):408–14. https://doi.org/10.1097/PTS.0000000000001146 .

Holton EF. Holton’s evaluation model: new evidence and construct elaborations. Adv Developing Hum Resour. 2005;7(1):37–54. https://doi.org/10.1177/1523422304272080 .

Rigot SK, DiGiovine KM, Boninger ML, Hibbs R, Smith I, Worobey LA, Connerton C, Mason J, Bonhotal S. (2023). Peer-Led Functional Mobility and Transfer Training. Nurse Educator, 48(5), 286.

Wang C, Liu A, Xu J, et al. Work Method and Effect of Hospital Quality Management Circle Stage Management model [J]. Chin Hosp. 2016;20(12):23–5.

Elshama S. Quality Management in Medical Education between Theory and Application: paradigm shift or falsification of. J Clin Case Rep Stud. 2022;3:1–5. https://doi.org/10.31579/2690-8808/109 .

Chang S-J, Huang HH-C, Li-Hua, Chang H. Taiwan quality indicator project and hospital productivity growth. Omega. 2011;39(1):14–22.

Arnold AP, Laurene Finley [email protected], Roberta G, Sands, Joretha Bourjolly & Victoria Stanhope. (2012) Training Mental Health Providers in Cultural Competence: A Transformative Learning Process, American Journal of Psychiatric Rehabilitation, 15:4, 334–356, https://doi.org/10.1080/15487768.2012.733287 .

Chang S-J, Hsiao H-C, Huang L-H, Chang H. 2011. Taiwan quality indicator project and hospital productivity growth, Omega, Elsevier, vol. 39(1), pages 14–22, January. 2010.01.006.

Finn F, Chesser-Smyth P. Promoting learning transfer in Preceptor Preparation. J Nurses Prof Dev. 2013;29:309–15. https://doi.org/10.1097/NND.0000000000000014 .

Guzman G, Fitzgerald J, Fulop L, Hayes K, Poropat A, Avery M, Campbell S, Fisher R, Gapp R, Herington C, McPhail R, Vecchio N. How best practices are copied, transferred, or translated between health care facilities: a conceptual framework. Health Care Manage Rev. 2015;40:193–202. https://doi.org/10.1097/HMR.0000000000000023 .

Billings D, Connors H, Skiba D. Benchmarking Best practices in web-based nursing courses. Adv Nurs Sci. 2001;23:41–52. https://doi.org/10.1097/00012272-200103000-00005 .

Devos C, Dumay X, Bonami M, Bates R, Holton E. The learning transfer system inventory (LTSI) translated into French: Internal structure and predictive validity. Eur Economics: Labor Social Conditions eJournal. 2007. https://doi.org/10.1111/j.1468-2419.2007.00280.x .

Mongkolsirikiet K, Akaraborworn C, Research Model. (2019). A Revisit of Holton’s HRD Evaluation and (2005) for Learning Transfer., 12, 15–34. https://doi.org/10.14456/JCDR-HS.2019.12 .

Yaqub Y, Singh A. Impact of training design on trainees’ motivation: an empirical study. Industrial Commercial Train. 2021. https://doi.org/10.1108/ict-05-2021-0038 .

Dewettinck K, Dijk H. Linking Belgian employee performance management system characteristics with performance management system effectiveness: exploring the mediating role of fairness. Int J Hum Resource Manage. 2013;24:806–25. https://doi.org/10.1080/09585192.2012.700169 .

Duprez V, Vandecasteele T, Verhaeghe S, Beeckman D, Hecke A. The effectiveness of interventions to enhance self-management support competencies in the nursing profession: a systematic review. J Adv Nurs. 2017;73:1807–24. https://doi.org/10.1111/jan.13249 .

Norouzi S, Mogadam F. Experiences of nursing Student\‘s clinical evaluation: a qualitative content analysis. J Med Educ Dev. 2016;11:134–45.

Niskala J, Kanste O, Tomietto M, Miettunen J, Tuomikoski A, Kyngäs H, Mikkonen K. Interventions to improve nurses’ job satisfaction: a systematic review and meta-analysis. J Adv Nurs. 2020. https://doi.org/10.1111/jan.14342 .

Gkioka M, Schneider J, Kruse A, Tsolaki M, Moraitou D, Teichmann B. Evaluation and Effectiveness of Dementia Staff Training Programs in General Hospital settings: a narrative synthesis with Holton’s three-level Model Applied. J Alzheimers Dis. 2020;78:1089–108. https://doi.org/10.3233/JAD-200741 .

Ghazvini A, Shukur Z. Awareness training transfer and Information Security Content Development for Healthcare Industry. Int J Adv Comput Sci Appl. 2016. https://doi.org/10.14569/IJACSA.2016.070549 . 7.

Ragsdale J, Berry A, Gibson J, Herber-Valdez C, Germain L, Engle D. Evaluating the effectiveness of undergraduate clinical education programs. Med Educ Online. 2020;25. https://doi.org/10.1080/10872981.2020.1757883 .

Kirkpatrick DL. Techniques for evaluation training programs. J Am Soc Train Dir. 1959;13:21–6.

Kirkman TR. High Fidelity Simulation Effectiveness in Nursing Students’ Transfer of Learning International Journal of Nursing Education Scholarship, vol. 10, no. 1, 2013, pp. 171–176. https://doi.org/10.1515/ijnes-2012-0009 .

Chen SL, Huang TW, Liao IC, Liu C. Development and validation of the simulation learning effectiveness inventory. J Adv Nurs. 2015;71(10):2444–53. https://doi.org/10.1111/jan.12707 .

Ayed A, Khalaf I, Fashafsheh I, Saleh A, Bawadi H, Abuidhail J, Thultheen I, Joudallah H. Effect of High-Fidelity Simulation on Clinical Judgment among nursing students. Inquiry: J Med Care Organ Provis Financing. 2022;59. https://doi.org/10.1177/00469580221081997 .

Yun J, Kim D, Park Y. The influence of informal learning and learning transfer on nurses’ clinical performance: a descriptive cross-sectional study. Nurse Educ Today. 2019. https://doi.org/10.1016/J.NEDT.2019.05.027 .

Bhatti M, Ali S, Isa M, Battour M. Training transfer and transfer motivation: the influence of individual, environmental, situational, Training Design, and affective reaction factors. Perform Improv Q. 2014;27:51–82. https://doi.org/10.1002/PIQ.21165 .

Kontoghiorghes C. Factors affecting training effectiveness in the context of the introduction of a New Technology–A U.S. Case Study. Int J Train Dev. 2001;5:248–60. https://doi.org/10.1111/1468-2419.00137 .

Download references

This study was funded by Department of Education of Zhejiang Province, Grant Number jg20220475.

Author information

Authors and affiliations.

School of Nursing, Zhejiang Shuren University, 8 Shuren Road, 310015, Hangzhou, ZheJiang, China

MingYan Shen & ZhiXian Feng

Department of Nursing, Shulan (Hangzhou) Hospital, Shulan International Medical College, Zhejiang Shuren University, 310022, Hangzhou, China

You can also search for this author in PubMed   Google Scholar

Contributions

The following statements specify the individual contributions of each author to the manuscript titled “Continue Nursing Education: An Action Research Study on the Implementation of a Nursing Training Program Using the Holton Learning Transfer System Inventory”:ZhiXian Feng conceived and designed the analysis; led the research team and coordinated the project; critically reviewed and revised the manuscript for important intellectual content; oversaw the implementation of the training program; MingYan Shen conducted the research; collected and organized the data; analyzed and interpreted the data; contributed to the statistical analysis; wrote the initial draft of the manuscript; managed logistics and operational aspects of the study.

Corresponding author

Correspondence to ZhiXian Feng .

Ethics declarations

Ethics approval and consent to participate.

Approval of this study was granted by the Research Ethics Committee of Shulan Hospital (Approval no. KY2021042).

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Prior to participation, all subjects (or their legal guardians) were informed about the nature, objectives, potential benefits, and risks of the study. Written informed consent was obtained from all subjects and/or their legal guardians. All data were collected and processed maintaining strict confidentiality and anonymity, safeguarding the privacy and rights of all participants.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Shen, M., Feng, Z. Continue nursing education: an action research study on the implementation of a nursing training program using the Holton Learning Transfer System Inventory. BMC Med Educ 24 , 610 (2024). https://doi.org/10.1186/s12909-024-05552-6

Download citation

Received : 16 December 2023

Accepted : 13 May 2024

Published : 03 June 2024

DOI : https://doi.org/10.1186/s12909-024-05552-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Nursing education
• Quality improvement
• Action research
• Holton Learning Transfer System Inventory

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]