systematic literature review search terms

Systematic Reviews

• Introduction
• Review Process: Step by Step
• 1. Planning a Review
• 2. Defining Your Question & Criteria
• 3. Standards & Protocols

Designing Your Search Strategy

Search strategy checklists, pre-search tips, search strategies: filters & hedges, search terms, search strategies: and/or, phrase searching & truncation.

• 5. Locating Published Research
• 6. Locating Grey Literature
• 7. Managing & Documenting Results
• 8. Selecting & Appraising Studies
• 9. Extracting Data
• 10. Writing a Systematic Review
• Tools & Software
• Guides & Tutorials
• Accessing Resources
• Research Assistance

A well designed search strategy is essential to the success of your systematic review. Your strategy should be specific, unbiased, reproducible and will typically include subject headings along with a range of keywords/phrases for each of your concepts.  

Your searches should be designed to capture as many studies as possible that meet your criteria.

Chapter 4 of the  Cochrane Handbook for Systematic Reviews of Interventions  provides detailed guidance for searching and study selection; see  Supplement 3.8 Adapting search strategies across databases / sources  for translating your search across databases.

Systematic Reviews: Constructing a Search Strategy and Searching for Evidence  from the Joanna Briggs Institute provides step-by-step guidance using PubMed as an example database. 

General Steps:

• Locate previous/ relevant searches
• Identify your databases
• Develop your search terms and design search
• Evaluate and modify your search
• Document your search ( PRISMA-S  Checklist)
• Translate your search for other databases
• Step by Step Systematic Review Search Checklist from MD Anderson Center Library
• PRESS Peer Review Checklist for Search Strategies

Conduct a preliminary set of scoping searches in various databases to test out your search terms (keywords and subject headings) and locate additional terms for your concepts.

Try building a "gold set" of relevant references to help you identify search terms. Sources for this gold set may include:

• Recommended key papers
• Papers by known authors in the field
• Results of preliminary searches from key databases such 
• Reviewing references and "cited by" articles lists for key papers
• Articles that have been published in authoritative journals 

Hedges/ Filters

• PubMed Special Queries

Hedges are search strings created by experts to help you retrieve specific types of studies or topics; a hedge will filter your results by adding specific search terms, or specific combinations of search terms, to your search.  

Hedges can be good starting points but you may need to modify the search string to fit your research. Resources for hedges:

• University of Texas, School of Public Health  (study type)
• McMaster University Health Information Research Unit
• The InterTASC Information Specialists' Sub-Group Search Filter Resource
• Pubmed Search Strategies blog
• PubMed Special Queries Topic-Specific PubMed Queries; includes keyword and search strategy examples.

Example:  Health Disparities & Minority Health Search Strategies

• Subject Headings
• Keywords Vs. Subject Headings
• Locating Subject Headings
• Medical Subject Headings (MeSH)
• Keyword & Subject Headings Logic Grid

You can use your PICOTS concepts as preliminary search terms. The important terms in this question:

In adults , is screening for depression and feedback of results to providers more effective than no screening and feedback in improving outcomes of major depression in primary care settings?

...might include:

Major depression

Primary Care

(From Lackey, M. (2013). Systematic reviews: Searching the literature [PowerPoint slides]. Retrieved from http://guides.lib.unc.edu/ld.php?content_id=258919 )

Your search will include both keywords and subject headings. Controlled vocabulary systems, such as the Medical Subject Headings (MeSH) or Library of Congress Subject Headings (LCSH) , use pre-set terms that are used to tag resources on similar subjects. See boxes below for more information on finding and using subject terms.

Not all databases will have subject heading searching and for those that do, the subject heading categories may differ between databases. This is because databases classify articles using different criteria.

Using the keywords from our example, here are some MeSH terms for:

Adults : Adult (A person having attained full growth or maturity. Adults are of 19 through 44 years of age. For a person between 19 and 24 years of age, YOUNG ADULT is available.)

Screening : Mass Screening (Organized periodic procedures performed on large groups of people for the purpose of detecting disease.)

Major depression : Depressive Disorder, Major (Marked depression appearing in the involution period and characterized by hallucinations, delusions, paranoia, and agitation.)

Here is a LCSH subject term for:

Depression : Depression, mental (Dejection ; Depression, Unipolar ; Depressive disorder ; Depressive psychoses ; Melancholia ; Mental depression ; Unipolar depression)

• Most EBSCO databases have a tool to help you discover subject terms . See Academic Search Complete > Subject Terms and Academic Search Complete > Subject Terms: Thesaurus
• Most ProQuest databases have a tool to help you discover subject terms: See PsycInfo > Thesaurus
• When you find a useful article, look at the article's Subject Headings (or Subject or Subject Terms) , and record them as possible terms to use in a subject term search.

Here is an example of the subject terms listed for a systematic review found in PsycINFO, " Primary care screening for and treatment of depression in pregnant and postpartum women: Evidence report and systematic review for the US Preventive Services Task Force " (2016).

MeSH are standardized terms that describe the main concepts of PubMed/MedLine articles. Searching with MeSH can increase the precision of your search by providing a consistent way to retrieve articles that may use different terminology or spelling variations. 

Note: new articles will not have MeSH terms; the indexing process may take up to a few weeks for newly ingested articles. 

Use the  MeSH  database  to locate and build a search using MeSH.

To search the MeSH database:

• Search for 1 concept at a time.
• If you do not see a relevant MeSH in the results, search again with a synonym or related term.
• Click on the MeSH term to view to the complete record​, subheadings, broader and narrower terms. 

Build a search from the results list or from the MeSH term record to specify subheadings.

• Select the box next to the MeSH term or subheadings that you wish to search and click Add to Search Builder.
• ​You may need to switch  AND to OR , depending on how you would like to combine terms.
• Repeat the above steps to add additional MeSH terms. When your search is ready, click  Search PubMed.

Logic Grid with Keywords and Index Terms or Subject Headings from Systematic Reviews: Constructing a Search Strategy and Searching for Evidence.

 Bhuiyan, M. U., Stiboy, E., Hassan, M. Z., Chan, M., Islam, M. S., Haider, N., Jaffe, A., & Homaira, N. (2021). Epidemiology of COVID-19 infection in young children under five years: A systematic review and meta-analysis.   Vaccine ,  39 (4), 667–677. https://doi.org/10.1016/j.vaccine.2020.11.078 

• Boolean Logic: AND, OR, NOT
• Phrase Searching " "
• Truncation *
• Proximity Searching

AND, OR, NOT

Join together search terms in a logical manner.

AND - narrows searches, used to join dissimilar terms OR - broadens searches, used to join similar terms

NOT -  removes results containing specified keywords

#1 "major depression" AND "primary care"

#2 screen* OR feedback

#3 (screen* OR feedback)

AND “major depression”

AND “primary care”

"major depression" NOT suicide

" "  To search for specific phrases, enclose them in quotation marks . The database will search for those words together in that order.

“ primary care ”

“ major depression ”

Truncate a word in order to search for different forms of the same word. Many databases use the asterisk * as the truncation symbol.

Add the truncation symbol to the word screen * to search for screen, screens, screening, etc.

You do have to be careful with truncation. If you add the truncation symbol to the word minor* , the database will search for minor, minors, minority, minorities, etc.

Not all databases support proximity searching. You can use these strategies in ProQuest databases such as  Sociological Abstracts .

pre/#  is used to search for terms in proximity to each other in a  specific order;  # is replaced with the number of words permitted between the search terms.

Sample Search: parent*  pre/2  educational (within 2 words &   in order )

• This would retrieve articles with no more than two words between parent* and educational (in this order) e.g. " Parent  practices and  educational  achievement" OR " Parents  on  Educational  Attainment" OR " Parental  Values,  Educational  Attainment" etc.

w/#  is used to search for terms in proximity to each other in  any order ; # is replaced with the number of words permitted between the search terms.

Sample Search: parent*  w/3  educational (within 3 words & in  any order )

• This would retrieve articles with no more than three words between parent* and educational (in any order)   e.g. "Educational practices of parents" OR "Parents value motivation and education" OR "Educational attainments of Latino parents"
• << Previous: 3. Standards & Protocols
• Next: 5. Locating Published Research >>
• Last Updated: Jun 18, 2024 4:31 PM
• URL: https://libguides.ucmerced.edu/systematic-reviews

Last updated 20/06/24: Online ordering is currently unavailable due to technical issues. We apologise for any delays responding to customers while we resolve this. For further updates please visit our website: https://www.cambridge.org/news-and-insights/technical-incident

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings .

Login Alert

• > Journals
• > BJPsych Advances
• > Volume 24 Issue 2
• > How to carry out a literature search for a systematic...

Article contents

• LEARNING OBJECTIVES
• DECLARATION OF INTEREST

Defining the clinical question

Scoping search, search strategy, sources to search, developing a search strategy, searching electronic databases, supplementary search techniques, obtaining unpublished literature, conclusions, how to carry out a literature search for a systematic review: a practical guide.

Published online by Cambridge University Press:  01 March 2018

Performing an effective literature search to obtain the best available evidence is the basis of any evidence-based discipline, in particular evidence-based medicine. However, with a vast and growing volume of published research available, searching the literature can be challenging. Even when journals are indexed in electronic databases, it can be difficult to identify all relevant studies without an effective search strategy. It is also important to search unpublished literature to reduce publication bias, which occurs from a tendency for authors and journals to preferentially publish statistically significant studies. This article is intended for clinicians and researchers who are approaching the field of evidence synthesis and would like to perform a literature search. It aims to provide advice on how to develop the search protocol and the strategy to identify the most relevant evidence for a given research or clinical question. It will also focus on how to search not only the published but also the unpublished literature using a number of online resources.

• Understand the purpose of conducting a literature search and its integral part of the literature review process

• Become aware of the range of sources that are available, including electronic databases of published data and trial registries to identify unpublished data

• Understand how to develop a search strategy and apply appropriate search terms to interrogate electronic databases or trial registries

A literature search is distinguished from, but integral to, a literature review. Literature reviews are conducted for the purpose of (a) locating information on a topic or identifying gaps in the literature for areas of future study, (b) synthesising conclusions in an area of ambiguity and (c) helping clinicians and researchers inform decision-making and practice guidelines. Literature reviews can be narrative or systematic, with narrative reviews aiming to provide a descriptive overview of selected literature, without undertaking a systematic literature search. By contrast, systematic reviews use explicit and replicable methods in order to retrieve all available literature pertaining to a specific topic to answer a defined question (Higgins Reference Higgins and Green 2011 ). Systematic reviews therefore require a priori strategies to search the literature, with predefined criteria for included and excluded studies that should be reported in full detail in a review protocol.

Performing an effective literature search to obtain the best available evidence is the basis of any evidence-based discipline, in particular evidence-based medicine (Sackett Reference Sackett 1997 ; McKeever Reference McKeever, Nguyen and Peterson 2015 ). However, with a vast and growing volume of published research available, searching the literature can be challenging. Even when journals are indexed in electronic databases, it can be difficult to identify all relevant studies without an effective search strategy (Hopewell Reference Hopewell, Clarke and Lefebvre 2007 ). In addition, unpublished data and ‘grey’ literature (informally published material such as conference abstracts) are now becoming more accessible to the public. It is important to search unpublished literature to reduce publication bias, which occurs because of a tendency for authors and journals to preferentially publish statistically significant studies (Dickersin Reference Dickersin and Min 1993 ). Efforts to locate unpublished and grey literature during the search process can help to reduce bias in the results of systematic reviews (Song Reference Song, Parekh and Hooper 2010 ). A paradigmatic example demonstrating the importance of capturing unpublished data is that of Turner et al ( Reference Turner, Matthews and Linardatos 2008 ), who showed that using only published data in their meta-analysis led to effect sizes for antidepressants that were one-third (32%) larger than effect sizes derived from combining both published and unpublished data. Such differences in findings from published and unpublished data can have real-life implications in clinical decision-making and treatment recommendation. In another relevant publication, Whittington et al ( Reference Whittington, Kendall and Fonagy 2004 ) compared the risks and benefits of selective serotonin reuptake inhibitors (SSRIs) in the treatment of depression in children. They found that published data suggested favourable risk–benefit profiles for SSRIs in this population, but the addition of unpublished data indicated that risk outweighed treatment benefits. The relative weight of drug efficacy to side-effects can be skewed if there has been a failure to search for, or include, unpublished data.

In this guide for clinicians and researchers on how to perform a literature search we use a working example about efficacy of an intervention for bipolar disorder to demonstrate the search techniques outlined. However, the overarching methods described are purposefully broad to make them accessible to all clinicians and researchers, regardless of their research or clinical question.

The review question will guide not only the search strategy, but also the conclusions that can be drawn from the review, as these will depend on which studies or other forms of evidence are included and excluded from the literature review. A narrow question will produce a narrow and precise search, perhaps resulting in too few studies on which to base a review, or be so focused that the results are not useful in wider clinical settings. Using an overly narrow search also increases the chances of missing important studies. A broad question may produce an imprecise search, with many false-positive search results. These search results may be too heterogeneous to evaluate in one review. Therefore from the outset, choices should be made about the remit of the review, which will in turn affect the search.

A number of frameworks can be used to break the review question into concepts. One such is the PICO (population, intervention, comparator and outcome) framework, developed to answer clinical questions such as the effectiveness of a clinical intervention (Richardson Reference Richardson, Wilson and Nishikawa 1995 ). It is noteworthy that ‘outcome’ concepts of the PICO framework are less often used in a search strategy as they are less well defined in the titles and abstracts of available literature (Higgins Reference Higgins and Green 2011 ). Although PICO is widely used, it is not a suitable framework for identifying key elements of all questions in the medical field, and minor adaptations are necessary to enable the structuring of different questions. Other frameworks exist that may be more appropriate for questions about health policy and management, such as ECLIPSE (expectation, client group, location, impact, professionals, service) (Wildridge Reference Wildridge and Bell 2002 ) or SPICE (setting, perspective, intervention, comparison, evaluation) for service evaluation (Booth Reference Booth 2006 ). A detailed overview of frameworks is provided in Davies ( Reference Davies 2011 ).

Before conducting a comprehensive literature search, a scoping search of the literature using just one or two databases (such as PubMed or MEDLINE) can provide valuable information as to how much literature for a given review question already exists. A scoping search may reveal whether systematic reviews have already been undertaken for a review question. Caution should be taken, however, as systematic reviews that may appear to ask the same question may have differing inclusion and exclusion criteria for studies included in the review. In addition, not all systematic reviews are of the same quality. If the original search strategy is of poor quality methodologically, original data are likely to have been missed and the search should not simply be updated (compare, for example, Naughton et al ( Reference Naughton, Clarke and O'Leary 2014 ) and Caddy et al ( Reference Caddy, Amit and McCloud 2015 ) on ketamine for treatment-resistant depression).

The first step in conducting a literature search should be to develop a search strategy. The search strategy should define how relevant literature will be identified. It should identify sources to be searched (list of databases and trial registries) and keywords used in the literature (list of keywords). The search strategy should be documented as an integral part of the systematic review protocol. Just as the rest of a well-conducted systematic review, the search strategy used needs to be explicit and detailed such that it could reproduced using the same methodology, with exactly the same results, or updated at a later time. This not only improves the reliability and accuracy of the review, but also means that if the review is replicated, the difference in reviewers should have little effect, as they will use an identical search strategy. The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) statement was developed to standardise the reporting of systematic reviews (Moher Reference Moher, Liberati and Tetzlaff 2009 ). The PRISMA statement consists of a 27-item checklist to assess the quality of each element of a systematic review (items 6, 7 and 8 relate to the quality of literature searching) and also to guide authors when reporting their findings.

There are a number of databases that can be searched for literature, but the identification of relevant sources is dependent on the clinical or research question (different databases have different focuses, from more biology to more social science oriented) and the type of evidence that is sought (i.e. some databases report only randomised controlled trials).

• MEDLINE and Embase are the two main biomedical literature databases. MEDLINE contains more than 22 million references from more than 5600 journals worldwide. In addition, the MEDLINE In-Process & Other Non-Indexed Citations database holds references before they are published on MEDLINE. Embase has a strong coverage of drug and pharmaceutical research and provides over 30 million references from more than 8500 currently published journals, 2900 of which are not in MEDLINE. These two databases, however, are only available to either individual subscribers or through institutional access such as universities and hospitals. PubMed, developed by the National Center for Biotechnology Information of the US National Library of Medicine, provides access to a free version of MEDLINE and is accessible to researchers, clinicians and the public. PubMed comprises medical and biomedical literature indexed in MEDLINE, but provides additional access to life science journals and e-books.

In addition, there are a number of subject- and discipline-specific databases.

• PsycINFO covers a range of psychological, behavioural, social and health sciences research.

• The Cochrane Central Register of Controlled Trials (CENTRAL) hosts the most comprehensive source of randomised and quasi-randomised controlled trials. Although some of the evidence on this register is also included in Embase and MEDLINE, there are over 150 000 reports indexed from other sources, such as conference proceedings and trial registers, that would otherwise be less accessible (Dickersin Reference Dickersin, Manheimer and Wieland 2002 ).

• The Cumulative Index to Nursing and Allied Health Literature (CINAHL), British Nursing Index (BNI) and the British Nursing Database (formerly BNI with Full Text) are databases relevant to nursing, but they span literature across medical, allied health, community and health management journals.

• The Allied and Complementary Medicine Database (AMED) is a database specifically for alternative treatments in medicine.

The examples of specific databases given here are by no means exhaustive, but they are popular and likely to be used for literature searching in medicine, psychiatry and psychology. Website links for these databases are given in Box 1 , along with links to resources not mentioned above. Box 1 also provides a website link to a couple of video tutorials for searching electronic databases. Box 2 shows an example of the search sources chosen for a review of a pharmacological intervention of calcium channel antagonists in bipolar disorder, taken from a recent systematic review (Cipriani Reference Cipriani, Saunders and Attenburrow 2016a ).

BOX 1 Website links of search sources to obtain published and unpublished literature

Electronic databases

• MEDLINE/PubMed: www.ncbi.nlm.nih.gov/pubmed

• Embase: www.embase.com

• PsycINFO: www.apa.org/psycinfo

• Cochrane Central Register of Controlled Trials (CENTRAL): www.cochranelibrary.com

• Cumulative Index of Nursing and Allied Health Literature (CINAHL): www.cinahl.com

• British Nursing Index: www.bniplus.co.uk

• Allied and Complementary Medicine Database: https://www.ebsco.com/products/research-databases/amed-the-allied-and-complementary-medicine-database

Grey literature databases

• BIOSIS Previews (part of Thomson Reuters Web of Science): https://apps.webofknowledge.com

Trial registries

• ClinicalTrials.gov: www.clinicaltrials.gov

• Drugs@FDA: www.accessdata.fda.gov/scripts/cder/daf

• European Medicines Agency (EMA): www.ema.europa.eu

• World Health Organization International Clinical Trials Registry Platform (WHO ICTRP): www.who.int/ictrp

• GlaxoSmithKline Study Register: www.gsk-clinicalstudyregister.com

• Eli-Lilly clinical trial results: https://www.lilly.com/clinical-study-report-csr-synopses

Guides to further resources

• King's College London Library Services: http://libguides.kcl.ac.uk/ld.php?content_id=17678464

• Georgetown University Medical Center Dahlgren Memorial Library: https://dml.georgetown.edu/core

• University of Minnesota Biomedical Library: https://hsl.lib.umn.edu/biomed/help/nursing

Tutorial videos

• Searches in electronic databases: http://library.buffalo.edu/hsl/services/instruction/tutorials.html

• Using the Yale MeSH Analyzer tool: http://library.medicine.yale.edu/tutorials/1559

BOX 2 Example of search sources chosen for a review of calcium channel antagonists in bipolar disorder (Cipriani Reference Cipriani, Saunders and Attenburrow 2016a )

Electronic databases searched:

• MEDLINE In-Process and Other Non-Indexed Citations

For a comprehensive search of the literature it has been suggested that two or more electronic databases should be used (Suarez-Almazor Reference Suarez-Almazor, Belseck and Homik 2000 ). Suarez-Almazor and colleagues demonstrated that, in a search for controlled clinical trials (CCTs) for rheumatoid arthritis, osteoporosis and lower back pain, only 67% of available citations were found by both Embase and MEDLINE. Searching MEDLINE alone would have resulted in 25% of available CCTs being missed and searching Embase alone would have resulted in 15% of CCTs being missed. However, a balance between the sensitivity of a search (an attempt to retrieve all relevant literature in an extensive search) and the specificity of a search (an attempt to retrieve a more manageable number of relevant citations) is optimal. In addition, supplementing electronic database searches with unpublished literature searches (see ‘Obtaining unpublished literature’ below) is likely to reduce publication bias. The capacity of the individuals or review team is likely largely to determine the number of sources searched. In all cases, a clear rationale should be outlined in the review protocol for the sources chosen (the expertise of an information scientist is valuable in this process).

Important methodological considerations (such as study design) may also be included in the search strategy. Dependent on the databases and supplementary sources chosen, filters can be used to search the literature by study design (see ‘Searching electronic databases’). For instance, if the search strategy is confined to one study design term only (e.g. randomised controlled trial, RCT), only the articles labelled in this way will be selected. However, it is possible that in the database some RCTs are not labelled as such, so they will not be picked up by the filtered search. Filters can help reduce the number of references retrieved by the search, but using just one term is not 100% sensitive, especially if only one database is used (i.e. MEDLINE). It is important for systematic reviewers to know how reliable such a strategy can be and treat the results with caution.

Identifying search terms

Standardised search terms are thesaurus and indexing terms that are used by electronic databases as a convenient way to categorise articles, allowing for efficient searching. Individual database records may be assigned several different standardised search terms that describe the same or similar concepts (e.g. bipolar disorder, bipolar depression, manic–depressive psychosis, mania). This has the advantage that even if the original article did not use the standardised term, when the article is catalogued in a database it is allocated that term (Guaiana Reference Guaiana, Barbui and Cipriani 2010 ). For example, an older paper might refer to ‘manic depression’, but would be categorised under the term ‘bipolar disorder’ when catalogued in MEDLINE. These standardised search terms are called MeSH (medical subject headings) in MEDLINE and PubMed, and Emtree in Embase, and are organised in a hierarchal structure ( Fig. 1 ). In both MEDLINE and Embase an ‘explode’ command enables the database to search for a requested term, as well as specific related terms. Both narrow and broader search terms can be viewed and selected to be included in the search if appropriate to a topic. The Yale MeSH Analyzer tool ( mesh.med.yale.edu ) can be used to help identify potential terms and phrases to include in a search. It is also useful to understand why relevant articles may be missing from an initial search, as it produces a comparison grid of MeSH terms used to index each article (see Box 1 for a tutorial video link).

FIG 1 Search terms and hierarchical structure of MeSH (medical subject heading) in MEDLINE and PubMed.

In addition, MEDLINE also distinguishes between MeSH headings (MH) and publication type (PT) terms. Publication terms are less about the content of an article than about its type, specifying for example a review article, meta-analysis or RCT.

Both MeSH and Emtree have their own peculiarities, with variations in thesaurus and indexing terms. In addition, not all concepts are assigned standardised search terms, and not all databases use this method of indexing the literature. It is advisable to check the guidelines of selected databases before undertaking a search. In the absence of a MeSH heading for a particular term, free-text terms could be used.

Free-text terms are used in natural language and are not part of a database’s controlled vocabulary. Free-text terms can be used in addition to standardised search terms in order to identify as many relevant records as possible (Higgins Reference Higgins and Green 2011 ). Using free-text terms allows the reviewer to search using variations in language or spelling (e.g. hypomani* or mania* or manic* – see truncation and wildcard functions below and Fig. 2 ). A disadvantage of free-text terms is that they are only searched for in the title and abstracts of database records, and not in the full texts, meaning that when a free-text word is used only in the body of an article, it will not be retrieved in the search. Additionally, a number of specific considerations should be taken into account when selecting and using free-text terms:

• synonyms, related terms and alternative phrases (e.g. mood instability, affective instability, mood lability or emotion dysregulation)

• abbreviations or acronyms in medical and scientific research (e.g. magnetic resonance imaging or MRI)

• lay and medical terminology (e.g. high blood pressure or hypertension)

• brand and generic drug names (e.g. Prozac or fluoxetine)

• variants in spelling (e.g. UK English and American English: behaviour or behavior; paediatric or pediatric).

FIG 2 Example of a search strategy about bipolar disorder using MEDLINE (Cipriani Reference Cipriani, Saunders and Attenburrow 2016a ). The strategy follows the PICO framework and includes MeSH terms, free-text keywords and a number of other techniques, such as truncation, that have been outlined in this article. Numbers in bold give the number of citations retrieved by each search.

Truncation and wildcard functions can be used in most databases to capture variations in language:

• truncation allows the stem of a word that may have variant endings to be searched: for example, a search for depress* uses truncation to retrieve articles that mention both depression and depressive; truncation symbols may vary by database, but common symbols include: *, ! and #

• wild cards substitute one letter within a word to retrieve alternative spellings: for example, ‘wom?n’ would retrieve the terms ‘woman’ and ‘women’.

Combining search terms

Search terms should be combined in the search strategy using Boolean operators. Boolean operators allow standardised search terms and free-text terms to be combined. There are three main Boolean operators – AND, OR and NOT ( Fig. 3 ).

• OR – this operator is used to broaden a search, finding articles that contain at least one of the search terms within a concept. Sets of terms can be created for each concept, for example the population of interest: (bipolar disorder OR bipolar depression). Parentheses are used to build up search terms, with words within parentheses treated as a unit.

• AND – this can be used to join sets of concepts together, narrowing the retrieved literature to articles that contain all concepts, for example the population or condition of interest and the intervention to be evaluated: (bipolar disorder OR bipolar depression) AND calcium channel blockers. However, if at least one term from each set of concepts is not identified from the title or abstract of an article, this article will not be identified by the search strategy. It is worth mentioning here that some databases can run the search also across the full texts. For example, ScienceDirect and most publishing houses allow this kind of search, which is much more comprehensive than abstract or title searches only.

• NOT – this operator, used less often, can focus a search strategy so that it does not retrieve specific literature, for example human studies NOT animal studies. However, in certain cases the NOT operator can be too restrictive, for example if excluding male gender from a population, using ‘NOT male’ would also mean that any articles about both males and females are not obtained by the search.

FIG 3 Example of Boolean operator concepts (the resulting search is the light red shaded area).

The conventions of each database should be checked before undertaking a literature search, as functions and operators may differ slightly between them (Cipriani Reference Cipriani, Saunders and Attenburrow 2016b ). This is particularly relevant when using limits and filters. Figure 2 shows an example search strategy incorporating many of the concepts described above. The search strategy is taken from Cipriani et al ( Reference Cipriani, Zhou and Del Giovane 2016a ), but simplified to include only one intervention.

Search filters

A number of filters exist to focus a search, including language, date and study design or study focus filters. Language filters can restrict retrieval of articles to the English language, although if language is not an inclusion criterion it should not be restricted, to avoid language bias. Date filters can be used to restrict the search to literature from a specified period, for example if an intervention was only made available after a certain date. In addition, if good systematic reviews exist that are likely to capture all relevant literature (as advised by an information specialist), date restrictions can be used to search additional literature published after the date of that included in the systematic review. In the same way, date filters can be used to update a literature search since the last time it was conducted. Reviewing the literature should be a timely process (new and potentially relevant evidence is produced constantly) and updating the search is an important step, especially if collecting evidence to inform clinical decision-making, as publications in the field of medicine are increasing at an impressive rate (Barber Reference Barber, Corsi and Furukawa 2016 ). The filters chosen will depend on the research question and nature of evidence that is sought through the literature search and the guidelines of the individual database that is used.

• Google Scholar

Google Scholar allows basic Boolean operators to be used in strings of search terms. However, the search engine does not use standardised search terms that have been tagged as in traditional databases and therefore variations of keywords should always be searched. There are advantages and disadvantages to using a web search engine such as Google Scholar. Google Scholar searches the full text of an article for keywords and also searches a wider range of sources, such as conference proceedings and books, that are not found in traditional databases, making it a good resource to search for grey literature (Haddaway Reference Haddaway, Collins and Coughlin 2015 ). In addition, Google Scholar finds articles cited by other relevant articles produced in the search. However, variable retrieval of content (due to regular updating of Google algorithms and the individual's search history and location) means that search results are not necessarily reproducible and are therefore not in keeping with replicable search methods required by systematic reviews. Google Scholar alone has not been shown to retrieve more literature than other traditional databases discussed in this article and therefore should be used in addition to other sources (Bramer Reference Bramer, Giustini and Kramer 2016 ).

Citation searching

Once the search strategy has identified relevant literature, the reference lists in these sources can be searched. This is called citation searching or backward searching, and it can be used to see where particular research topics led others. This method is particularly useful if the search identifies systematic reviews or meta-analyses of a similar topic.

Conference abstracts

Conference abstracts are considered ‘grey literature’, i.e. literature that is not formally published in journals or books (Alberani Reference Alberani, De Castro Pietrangeli and Mazza 1990 ). Scherer and colleagues found that only 52.6% of all conference abstracts go on to full publication of results, and factors associated with publication were studies that had RCT designs and the reporting of positive or significant results (Scherer Reference Scherer, Langenberg and von Elm 2007 ). Therefore, failure to search relevant grey literature might miss certain data and bias the results of a review. Although conference abstracts are not indexed in most major electronic databases, they are available in databases such as BIOSIS Previews ( Box 1 ). However, as with many unpublished studies, these data did not undergo the peer review process that is often a tool for assessing and possibly improving the quality of the publication.

Searching trial registers and pharmaceutical websites

For reviews of trial interventions, a number of trial registers exist. ClinicalTrials.gov ( clinicaltrials.gov ) provides access to information on public and privately conducted clinical trials in humans. Results for both published and unpublished studies can be found for many trials on the register, in addition to information about studies that are ongoing. Searching each trial register requires a slightly different search strategy, but many of the basic principles described above still apply. Basic searches on ClinicialTrials.gov include searching by condition, specific drugs or interventions and these can be linked using Boolean operators: for example, (bipolar disorder OR manic depressive disorder) AND lithium. As mentioned above, parentheses can be used to build up search terms. More advanced searches allow one to specify further search fields such as the status of studies, study type and age of participants. The US Food and Drug Administration (FDA) hosts a database providing information about FDA-approved drugs, therapeutic products and devices ( www.fda.gov ). The database (with open access to anyone, not only in the USA) can be searched by the drug name, its active ingredient or its approval application number and, for most drugs approved in the past 20 years or so, a review of clinical trial results (some of which remain unpublished) used as evidence in the approval process is available. The European Medicines Agency (EMA) hosts a similar register for medicines developed for use in the European Union ( www.ema.europa.eu ). An internet search will show that many other national and international trial registers exist that, depending on the review question, may be relevant search sources. The World Health Organization International Clinical Trials Registry Platform (WHO ICTRP; www.who.int/ictrp ) provides access to a central database bringing a number of these national and international trial registers together. It can be searched in much the same way as ClinicalTrials.gov.

A number of pharmaceutical companies now share data from company-sponsored clinical trials. GlaxoSmithKline (GSK) is transparent in the sharing of its data from clinical studies and hosts its own clinical study register ( www.gsk-clinicalstudyregister.com ). Eli-Lilly provides clinical trial results both on its website ( www.lillytrialguide.com ) and in external registries. However, other pharmaceutical companies, such as Wyeth and Roche, divert users to clinical trial results in external registries. These registries include both published and previously unpublished studies. Searching techniques differ for each company and hand-searching through documents is often required to identify studies.

Communication with authors

Direct communication with authors of published papers could produce both additional data omitted from published studies and other unpublished studies. Contact details are usually available for the corresponding author of each paper. Although high-quality reviews do make efforts to obtain and include unpublished data, this does have potential disadvantages: the data may be incomplete and are likely not to have been peer-reviewed. It is also important to note that, although reviewers should make every effort to find unpublished data in an effort to minimise publication bias, there is still likely to remain a degree of this bias in the studies selected for a systematic review.

Developing a literature search strategy is a key part of the systematic review process, and the conclusions reached in a systematic review will depend on the quality of the evidence retrieved by the literature search. Sources should therefore be selected to minimise the possibility of bias, and supplementary search techniques should be used in addition to electronic database searching to ensure that an extensive review of the literature has been carried out. It is worth reminding that developing a search strategy should be an iterative and flexible process (Higgins Reference Higgins and Green 2011 ), and only by conducting a search oneself will one learn about the vast literature available and how best to capture it.

Acknowledgements

We thank Sarah Stockton for her help in drafting this article. Andrea Cipriani is supported by the NIHR Oxford cognitive health Clinical Research Facility.

Select the single best option for each question stem

a an explicit and replicable method used to retrieve all available literature pertaining to a specific topic to answer a defined question

b a descriptive overview of selected literature

c an initial impression of a topic which is understood more fully as a research study is conducted

d a method of gathering opinions of all clinicians or researchers in a given field

e a step-by-step process of identifying the earliest published literature through to the latest published literature.

a does not need to be specified in advance of a literature search

b does not need to be reported in a systematic literature review

c defines which sources of literature are to be searched, but not how a search is to be carried out

d defines how relevant literature will be identified and provides a basis for the search strategy

e provides a timeline for searching each electronic database or unpublished literature source.

a the Cochrane Central Register of Controlled Trials (CENTRAL)

d the Cumulative Index to Nursing and Allied Health Literature (CINAHL)

e the British Nursing Index.

a bipolar disorder OR treatment

b bipolar* OR treatment

c bipolar disorder AND treatment

d bipolar disorder NOT treatment

e (bipolar disorder) OR (treatment).

a publication bias

b funding bias

c language bias

d outcome reporting bias

e selection bias.

MCQ answers

1 a 2 d 3 b 4 c 5 a

FIG 2 Example of a search strategy about bipolar disorder using MEDLINE (Cipriani 2016a). The strategy follows the PICO framework and includes MeSH terms, free-text keywords and a number of other techniques, such as truncation, that have been outlined in this article. Numbers in bold give the number of citations retrieved by each search.

This article has been cited by the following publications. This list is generated based on data provided by Crossref .

View all Google Scholar citations for this article.

Save article to Kindle

To save this article to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle .

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

• Volume 24, Issue 2
• Lauren Z. Atkinson and Andrea Cipriani
• DOI: https://doi.org/10.1192/bja.2017.3

Save article to Dropbox

To save this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Dropbox account. Find out more about saving content to Dropbox .

Save article to Google Drive

To save this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Google Drive account. Find out more about saving content to Google Drive .

Reply to: Submit a response

- No HTML tags allowed - Web page URLs will display as text only - Lines and paragraphs break automatically - Attachments, images or tables are not permitted

Your details

Your email address will be used in order to notify you when your comment has been reviewed by the moderator and in case the author(s) of the article or the moderator need to contact you directly.

You have entered the maximum number of contributors

Conflicting interests.

Please list any fees and grants from, employment by, consultancy for, shared ownership in or any close relationship with, at any time over the preceding 36 months, any organisation whose interests may be affected by the publication of the response. Please also list any non-financial associations or interests (personal, professional, political, institutional, religious or other) that a reasonable reader would want to know about in relation to the submitted work. This pertains to all the authors of the piece, their spouses or partners.

• UNC Libraries
• HSL Academic Process
• Systematic Reviews
• Step 3: Conduct Literature Searches

Systematic Reviews: Step 3: Conduct Literature Searches

Created by health science librarians.

• Step 1: Complete Pre-Review Tasks
• Step 2: Develop a Protocol

About Step 3: Conduct Literature Searches

Partner with a librarian, systematic searching process, choose a few databases, search with controlled vocabulary and keywords, acknowledge outdated or offensive terminology, helpful tip - building your search, use nesting, boolean operators, and field tags, build your search, translate to other databases and other searching methods, document the search, updating your review.

• Searching FAQs
• Step 4: Manage Citations
• Step 5: Screen Citations
• Step 6: Assess Quality of Included Studies
• Step 7: Extract Data from Included Studies
• Step 8: Write the Review

  Check our FAQ's

   Email us

   Call (919) 962-0800

   Make an appointment with a librarian

  Request a systematic or scoping review consultation

Search the FAQs

In Step 3, you will design a search strategy to find all of the articles related to your research question. You will:

• Define the main concepts of your topic
• Choose which databases you want to search
• List terms to describe each concept
• Add terms from controlled vocabulary like MeSH
• Use field tags to tell the database where to search for terms
• Combine terms and concepts with Boolean operators AND and OR
• Translate your search strategy to match the format standards for each database
• Save a copy of your search strategy and details about your search

There are many factors to think about when building a strong search strategy for systematic reviews. Librarians are available to provide support with this step of the process.

Click an item below to see how it applies to Step 3: Conduct Literature Searches.

Reporting your review with PRISMA

For PRISMA, there are specific items you will want to report from your search.  For this step, review the PRISMA-S checklist.

• PRISMA-S for Searching
• Specify all databases, registers, websites, organizations, reference lists, and other sources searched or consulted to identify studies. Specify the date when each source was last searched or consulted. Present the full search strategies for all databases, registers and websites, including any filters and limits used.
• For information on how to document database searches and other search methods on your PRISMA flow diagram, visit our FAQs "How do I document database searches on my PRISMA flow diagram?" and "How do I document a grey literature search for my PRISMA flow diagram?"

Managing your review with Covidence

For this step of the review, in Covidence you can:

• Document searches in Covidence review settings so all team members can view
• Add keywords from your search to be highlighted in green or red while your team screens articles in your review settings

How a librarian can help with Step 3

When designing and conducting literature searches, a librarian can advise you on :

• How to create a search strategy with Boolean operators, database-specific syntax, subject headings, and appropriate keywords 
• How to apply previously published systematic review search strategies to your current search
• How to test your search strategy's performance 
• How to translate a search strategy from one database's preferred structure to another

The goal of a systematic retrieve is to find all results that are relevant to your topic. Because systematic review searches can be quite extensive and retrieve large numbers of results, an important aspect of systematic searching is limiting the number of irrelevant results that need to be screened. Librarians are experts trained in literature searching and systematic review methodology. Ask us a question or partner with a librarian to save time and improve the quality of your review. Our comparison chart detailing two tiers of partnership provides more information on how librarians can collaborate with and contribute to systematic review teams.

Search Process

• Use controlled vocabulary, if applicable
• Include synonyms/keyword terms
• Choose databases, websites, and/or registries to search
• Translate to other databases
• Search using other methods (e.g. hand searching)
• Validate and peer review the search

Databases can be multidisciplinary or subject specific. Choose the best databases for your research question. Databases index various journals, so in order to be comprehensive, it is important to search multiple databases when conducting a systematic review. Consider searching databases with more diverse or global coverage (i.e., Global Index Medicus) when appropriate. A list of frequently used databases is provided below. You can access UNC Libraries' full listing of databases on the HSL website (arranged alphabetically or by subject ).

Generally speaking, when literature searching, you are not searching the full-text article. Instead, you are searching certain citation data fields, like title, abstract, keyword, controlled vocabulary terms, and more. When developing a literature search, a good place to start is to identify searchable concepts of the research question, and then expand by adding other terms to describe those concepts. Read below for more information and examples on how to develop a literature search, as well as find tips and tricks for developing more comprehensive searches.

Identify search concepts and terms for each

Start by identifying the main concepts of your research question. If unsure, try using a question framework to help identify the main searchable concepts. PICO is one example of a question framework and is used specifically for clinical questions. If your research question doesn't fit into the PICO model well, view other examples of question frameworks and try another!

View our example in PICO format

Question: for patients 65 years and older, does an influenza vaccine reduce the future risk of pneumonia.

Controlled Vocabulary

Controlled vocabulary is a set of terminology assigned to citations to describe the content of each reference. Searching with controlled vocabulary can improve the relevancy of search results. Many databases assign controlled vocabulary terms to citations, but their naming schema is often specific to each database. For example, the controlled vocabulary system searchable via PubMed is MeSH, or Medical Subject Headings. More information on searching MeSH can be found on the HSL PubMed Ten Tips Legacy Guide .

Note: Controlled vocabulary may be outdated, and some databases allow users to submit requests to update terminology.

View Controlled Vocabulary for our example PICO

As mentioned above, databases with controlled vocabulary often use their own unique system. A listing of controlled vocabulary systems by database is shown below.

Keyword Terms

Not all citations are indexed with controlled vocabulary terms, however, so it is important to combine controlled vocabulary searches with keyword, or text word, searches. 

Authors often write about the same topic in varied ways and it is important to add these terms to your search in order to capture most of the literature. For example, consider these elements when developing a list of keyword terms for each concept:

• American versus British spelling
• hyphenated terms
• quality of life
• satisfaction
• vaccination
• influenza vaccination

There are several resources to consider when searching for synonyms. Scan the results of preliminary searches to identify additional terms. Look for synonyms, word variations, and other possibilities in Wikipedia, other encyclopedias or dictionaries, and databases. For example, PubChem lists additional drug names and chemical compounds.

Display Controlled Vocabulary and Keywords for our example PICO

Combining controlled vocabulary and text words in PubMed would look like this:

"Influenza Vaccines"[Mesh] OR "influenza vaccine" OR "influenza vaccines" OR "flu vaccine" OR "flu vaccines" OR "flu shot" OR "flu shots" OR "influenza virus vaccine" OR "influenza virus vaccines"

Social and cultural norms have been rapidly changing around the world. This has led to changes in the vocabulary used, such as when describing people or populations. Library and research terminology changes more slowly, and therefore can be considered outdated, unacceptable, or overly clinical for use in conversation or writing.

For our example with people 65 years and older, APA Style Guidelines recommend that researchers use terms like “older adults” and “older persons” and forgo terms like “senior citizens” and “elderly” that connote stereotypes. While these are current recommendations, researchers will recognize that terms like “elderly” have previously been used in the literature. Therefore, removing these terms from the search strategy may result in missed relevant articles. 

Research teams need to discuss current and outdated terminology and decide which terms to include in the search to be as comprehensive as possible. The research team or a librarian can search for currently preferred terms in glossaries, dictionaries, published guidelines, and governmental or organizational websites. The University of Michigan Library provides suggested wording to use in the methods section when antiquated, non-standard, exclusionary, or potentially offensive terms are included in the search.

Check the methods sections or supplementary materials of published systematic reviews for search strategies to see what terminology they used. This can help inform your search strategy by using MeSH terms or keywords you may not have thought of. However, be aware that search strategies will differ in their comprehensiveness.

You can also run a preliminary search for your topic, sort the results by Relevance or Best Match, and skim through titles and abstracts to identify terminology from relevant articles that you should include in your search strategy.

Nesting is a term that describes organizing search terms inside parentheses. This is important because, just like their function in math, commands inside a set of parentheses occur first. Parentheses let the database know in which order terms should be combined. 

Always combine terms for a single concept inside a parentheses set. For example: 

( "Influenza Vaccines"[Mesh] OR "influenza vaccine" OR "influenza vaccines" OR "flu vaccine" OR "flu vaccines" OR "flu shot" OR "flu shots" OR "influenza virus vaccine" OR "influenza virus vaccines" )

Additionally, you may nest a subset of terms for a concept inside a larger parentheses set, as seen below. Pay careful attention to the number of parenthesis sets and ensure they are matched, meaning for every open parentheses you also have a closed one.

( "Influenza Vaccines"[Mesh] OR "influenza vaccine" OR "influenza vaccines" OR "flu vaccine" OR "flu vaccines" OR "flu shot" OR "flu shots" OR "influenza virus vaccine" OR "influenza virus vaccines" OR   (( flu OR influenza ) AND ( vaccine OR vaccines OR vaccination OR immunization )))

Boolean operators

Boolean operators are used to combine terms in literature searches. Searches are typically organized using the Boolean operators OR or AND. OR is used to combine search terms for the same concept (i.e., influenza vaccine). AND is used to combine different concepts (i.e., influenza vaccine AND older adults AND pneumonia). An example of how Boolean operators can affect search retrieval is shown below. Using AND to combine the three concepts will only retrieve results where all are present. Using OR to combine the concepts will retrieve results that use all separately or together. It is important to note that, generally speaking, when you are performing a literature search you are only searching the title, abstract, keywords and other citation data. You are not searching the full-text of the articles.

The last major element to consider when building systematic literature searches are field tags. Field tags tell the database exactly where to search. For example, you can use a field tag to tell a database to search for a term in just the title, the title and abstract, and more. Just like with controlled vocabulary, field tag commands are different for every database.

If you do not manually apply field tags to your search, most databases will automatically search in a set of citation data points. Databases may also overwrite your search with algorithms if you do not apply field tags. For systematic review searching, best practice is to apply field tags to each term for reproducibility.

For example:

("Influenza Vaccines"[Mesh] OR "influenza vaccine"[tw] OR "influenza vaccines"[tw] OR "flu vaccine"[tw] OR "flu vaccines"[tw] OR "flu shot"[tw] OR "flu shots"[tw] OR "influenza virus vaccine"[tw] OR "influenza virus vaccines"[tw] OR ((flu[tw] OR influenza[tw]) AND (vaccine[tw] OR vaccines[tw] OR vaccination[tw] OR immunization[tw])))

View field tags for several health databases

For more information about how to use a variety of databases, check out our guides on searching.

• Searching PubMed guide Guide to searching Medline via the PubMed database
• Searching Embase guide Guide to searching Embase via embase.com
• Searching Scopus guide Guide to searching Scopus via scopus.com
• Searching EBSCO Databases guide Guide to searching CINAHL, PsycInfo, Global Health, & other databases via EBSCO

Combining search elements together

Organizational structure of literature searches is very important. Specifically, how terms are grouped (or nested) and combined with Boolean operators will drastically impact search results. These commands tell databases exactly how to combine terms together, and if done incorrectly or inefficiently, search results returned may be too broad or irrelevant.

For example, in PubMed:

(influenza OR flu) AND vaccine is a properly combined search and it produces around 50,000 results.

influenza OR flu AND vaccine is not properly combined.  Databases may read it as everything about influenza OR everything about (flu AND vaccine), which would produce more results than needed.

We recommend one or more of the following:

• put all your synonyms together inside a set of parentheses, then put AND between the closing parenthesis of one set and the opening parenthesis of the next set
• use a separate search box for each set of synonyms
• run each set of synonyms as a separate search, and then combine all your searches
• ask a librarian if your search produces too many or too few results

View the proper way to combine MeSH terms and Keywords for our example PICO

Question: for patients 65 years and older, does an influenza vaccine reduce the future risk of pneumonia .

Translating search strategies to other databases

Databases often use their own set of terminology and syntax. When searching multiple databases, you need to adjust the search slightly to retrieve comparable results. Our sections on Controlled Vocabulary and Field Tags have information on how to build searches in different databases.  Resources to help with this process are listed below.

• Polyglot search A tool to translate a PubMed or Ovid search to other databases
• Search Translation Resources (Cornell) A listing of resources for search translation from Cornell University
• Advanced Searching Techniques (King's College London) A collection of advanced searching techniques from King's College London

Other searching methods

Hand searching.

Literature searches can be supplemented by hand searching. One of the most popular ways this is done with systematic reviews is by searching the reference list and citing articles of studies included in the review. Another method is manually browsing key journals in your field to make sure no relevant articles were missed. Other sources that may be considered for hand searching include: clinical trial registries, white papers and other reports, pharmaceutical or other corporate reports, conference proceedings, theses and dissertations, or professional association guidelines.

Searching grey literature

Grey literature typically refers to literature not published in a traditional manner and often not retrievable through large databases and other popular resources. Grey literature should be searched for inclusion in systematic reviews in order to reduce bias and increase thoroughness. There are several databases specific to grey literature that can be searched.

• Open Grey Grey literature for Europe
• OAIster A union catalog of millions of records representing open access resources from collections worldwide
• Grey Matters: a practical tool for searching health-related grey literature (CADTH) From CADTH, the Canadian Agency for Drugs and Technologies in Health, Grey Matters is a practical tool for searching health-related grey literature. The MS Word document covers a grey literature checklist, including national and international health technology assessment (HTA) web sites, drug and device regulatory agencies, clinical trial registries, health economics resources, Canadian health prevalence or incidence databases, and drug formulary web sites.
• Duke Medical Center Library: Searching for Grey Literature A good online compilation of resources by the Duke Medical Center Library.

Systematic review quality is highly dependent on the literature search(es) used to identify studies. To follow best practices for reporting search strategies, as well as increase reproducibility and transparency, document various elements of the literature search for your review. To make this process more clear, a statement and checklist for reporting literature searches has been developed and and can be found below.

• PRISMA-S: Reporting Literature Searches in Systematic Reviews
• Section 4.5 Cochrane Handbook - Documenting and reporting the search process

At a minimum, document and report certain elements, such as databases searched, including name (i.e., Scopus) and platform (i.e. Elsevier), websites, registries, and grey literature searched. In addition, this also may include citation searching and reaching out to experts in the field. Search strategies used in each database or source should be documented, along with any filters or limits, and dates searched. If a search has been updated or was built upon previous work, that should be noted as well. It is also helpful to document which search terms have been tested and decisions made for term inclusion or exclusion by the team. Last, any peer review process should be stated as well as the total number of records identified from each source and how deduplication was handled. 

If you have a librarian on your team who is creating and running the searches, they will handle the search documentation.

You can document search strategies in word processing software you are familiar with like Microsoft Word or Excel, or Google Docs or Sheets. A template, and separate example file, is provided below for convenience. 

• Search Strategy Documentation Template
• Search Strategy Documentation Example

*Some databases like PubMed are being continually updated with new technology and algorithms. This means that searches may retrieve different results than when originally run, even with the same filters, date limits, etc.

When you decide to update a systematic review search, there are two ways of identifying new articles:  

1. rerun the original search strategy without any changes. .

Rerun the original search strategy without making any changes.  Import the results into your citation manager, and remove all articles duplicated from the original set of search results.

2. Rerun the original search strategy and add an entry date filter.

Rerun the original search strategy and add a date filter for when the article was added to the database ( not the publication date).  An entry date filter will find any articles added to the results since you last ran the search, unlike a publication date filter, which would only find more recent articles.

Some examples of entry date filters for articles entered since December 31, 2021 are:

• PubMed:   AND ("2021/12/31"[EDAT] : "3000"[EDAT])
• Embase: AND [31-12-2021]/sd
• CINAHL:   AND EM 20211231-20231231
• PsycInfo: AND RD 20211231-20231231
• Scopus:   AND LOAD-DATE AFT 20211231  

Your PRISMA flow diagram

For more information about updating the PRISMA flow diagram for your systematic review, see the information on filling out a PRISMA flow diagram for review updates on the Step 8: Write the Review page of the guide.

• << Previous: Step 2: Develop a Protocol
• Next: Step 4: Manage Citations >>
• Last Updated: May 16, 2024 3:24 PM
• URL: https://guides.lib.unc.edu/systematic-reviews

• Locations and Hours
• UCLA Library
• Research Guides
• Biomedical Library Guides

Systematic Reviews

• Creating the Search
• Types of Literature Reviews
• Planning Your Systematic Review
• Database Searching

Before You Start

Step 1: structure your concepts, step 2: brainstorm keywords for each concept, step 3: determine appropriate controlled vocabulary terms, step 4: put it all together, step 5: refine your strategy.

• Search Filters and Hedges
• Grey Literature
• Managing and Appraising Results
• Further Resources

Explicitly state your research question, determine which databases you will search, and determine your inclusion/exclusion criteria for studies that you find.  Here is some information on writing a protocol for your systematic review study . You might want to  search PROSPERO , a database of protocols, to make sure that no one else is currently working on a review on the same topic. You can also submit your protocol to PROSPERO.

• Break down your research question into smaller concepts in order to make the next few steps manageable.

• Now we want to fill out the other columns on this table. The next step is to figure out any potential synonyms or conceptually equivalent ideas for each of our concepts. This will help us perform an exhaustive search using all possible terminology. If you have already identified any papers that you plan to include, it's helpful to scan their abstracts for any obvious keywords that should be included. Don't forget that there may be spelling variants (estrogen vs. oestrogen)!
• You can truncate words to retrieve multiple word endings. The truncation symbol is usually an asterisk (*) but can sometimes be another character - check the database's Help documentation to determine what it is for any particular database. For example, inflam* will retrieve the following keywords in the results: inflammation, inflammatory, inflamed, inflaming...etc. This can be helpful when searching for plurals.
• You can phrase search for exact phrases. For example, searching "post menopausal" in quotes will retrieve different results than searching post menopausal . Without quotes, most databases will look for the word post anywhere in the record and the word menopausal anywhere in the record, which may result in unwanted results. With quotes, you'll only see the exact phrase "post menopausal."

• At this point you'll need to decide which database you are going to search first and complete your search strategy using only that database's syntax. It can be a little confusing to try to switch back and forth between databases while still trying to finalize your search. For our example we will proceed using PubMed, but the techniques are the same in any database that offers controlled vocabulary searching.
• In PubMed, the controlled vocabulary is called MeSH. Here is a tutorial that explains how powerful and functional MeSH is . I recommend completing this tutorial to have a good understanding of the next few steps.
• Now that you've been introduced to MeSH, let's put it to work for our sample search. We're going to search for each of our concepts in the MeSH database to determine the appropriate MeSH heading for each of them. Now is also a good time to go back to any papers that you've already identified as relevant and see if there are any MeSH terms on those papers that you might want to use (to find the MeSH terms, search the papers in PubMed and look at the bottom of each article record).
• In the MeSH database, search for your concepts. Click on any relevant terms. From the MeSH database record, you can click "Add to Search Builder" to add the term into the PubMed search box. Then you can copy and paste it into your table.

• Note that this behavior of including narrower terms is usually called "explosion." MeSH explodes by default, but other controlled vocabularies might not. Look for the option to explode in other databases.
• For databases that do not use controlled vocabularies (such as Web of Science), you can search with only the keyword column.
• Now we are ready to construct a search using all the terms we've listed in the table. Since we put each concept on its own row, all the terms in the same row are conceptually equivalent, so we will combine them with the Boolean operator OR. You can do this in the PubMed Advanced search builder (or any database search builder if you're not in PubMed), or you can type it out. Even if the database allows you to save search strategies, it's good to document the searches in a file separately from the database.
• At this point you will want to determine which field you want to use to search the words you've listed in the Keywords column. This will differ depending on the database. For PubMed, you will probably want to use either the Title/Abstract field (which searches those two fields) or the Text Word field (which searches title, abstract, MeSH terms and subheadings, and chemical substance names). We'll proceed using Text Word for our keywords.

• ("post menopause"[Text Word] OR "post menopausal"[Text Word] OR postmenopaus*[Text Word] OR "Postmenopause"[Mesh]) AND ("hormone therapy"[Text Word] OR "hormone replacement therapy"[Text Word] OR estrogen[Text Word] OR oestrogen[Text Word] OR progesterone[Text Word] OR "Hormone Replacement Therapy"[Mesh]) AND (cardiovascular[Text Word] OR atherosclerosis[Text Word] OR hypertension[Text Word] OR "heart failure"[Text Word] OR arrythmia[Text Word] OR stroke[Text Word] OR "myocardial infarction"[Text Word] OR "heart attack"[Text Word] OR "Cardiovascular Diseases"[Mesh])

Here are some techniques you can use to improve your search results

• Go back to your protocol. Can you apply any of your inclusion and exclusion criteria to your search, for instance, English-only papers or a particular date limit?

• Sort your results by relevance and try to identify any particular relevant papers in the first few pages of results. If you find any good ones, scan their abstracts and MeSH terms to make sure you didn't miss anything important from your own search strategy.
• Sort your results by date of publication and try to identify any particularly irrelevant papers. If you are seeing thousands of results that are utterly off topic, click into them and see if you can figure out why they are coming up. Some common issues include: a broad MeSH term exploding to include very irrelevant terms, keywords that are too broad and used too commonly by authors, or unexpected results of truncation. In PubMed, you can always look at the "search details" box on the results page to see how PubMed translated your search commands.
• << Previous: Database Searching
• Next: Search Filters and Hedges >>
• Last Updated: Apr 17, 2024 2:02 PM
• URL: https://guides.library.ucla.edu/systematicreviews

• Duke NetID Login
• 919.660.1100
• Duke Health Badge: 24-hour access
• Accounts & Access
• Databases, Journals & Books
• Request & Reserve
• Training & Consulting
• Request Articles & Books
• Renew Online
• Reserve Spaces
• Reserve a Locker
• Study & Meeting Rooms
• Course Reserves
• Pay Fines/Fees
• Recommend a Purchase
• Access From Off Campus
• Building Access
• Computers & Equipment
• Wifi Access
• My Accounts
• Mobile Apps
• Known Access Issues
• Report an Access Issue
• All Databases
• Article Databases
• Basic Sciences
• Clinical Sciences
• Dissertations & Theses
• Drugs, Chemicals & Toxicology
• Grants & Funding
• Interprofessional Education
• Non-Medical Databases
• Search for E-Journals
• Search for Print & E-Journals
• Search for E-Books
• Search for Print & E-Books
• E-Book Collections
• Biostatistics
• Global Health
• MBS Program
• Medical Students
• MMCi Program
• Occupational Therapy
• Path Asst Program
• Physical Therapy
• Researchers
• Community Partners

Conducting Research

• Archival & Historical Research
• Black History at Duke Health
• Data Analytics & Viz Software
• Data: Find and Share
• Evidence-Based Practice
• NIH Public Access Policy Compliance
• Publication Metrics
• Qualitative Research
• Searching Animal Alternatives

Systematic Reviews

• Test Instruments

Using Databases

• JCR Impact Factors
• Web of Science

Finding & Accessing

• COVID-19: Core Clinical Resources
• Health Literacy
• Health Statistics & Data
• Library Orientation

Writing & Citing

• Creating Links
• Getting Published
• Reference Mgmt
• Scientific Writing

Meet a Librarian

• Request a Consultation
• Find Your Liaisons
• Register for a Class
• Request a Class
• Self-Paced Learning

Search Services

• Literature Search
• Systematic Review
• Animal Alternatives (IACUC)
• Research Impact

Citation Mgmt

• Other Software

Scholarly Communications

• About Scholarly Communications
• Publish Your Work
• Measure Your Research Impact
• Engage in Open Science
• Libraries and Publishers
• Directions & Maps
• Floor Plans

Library Updates

• Annual Snapshot
• Conference Presentations
• Contact Information
• Gifts & Donations
• What is a Systematic Review?
• Types of Reviews
• Manuals and Reporting Guidelines
• Our Service
• 1. Assemble Your Team
• 2. Develop a Research Question
• 3. Write and Register a Protocol

4. Search the Evidence

• 5. Screen Results
• 6. Assess for Quality and Bias
• 7. Extract the Data
• 8. Write the Review
• Additional Resources
• Finding Full-Text Articles

The goal of systematic review searches is to identify all relevant studies on a topic. Therefore, systematic review searches are typically quite extensive. It is necessary, however, to strike a balance between striving for comprehensiveness and maintaining relevance when developing a search strategy. Additionally, the searches should be well-documented and reproducible. Click through the tabs below to review core tenets from both the Cochrane Handbook for Systematic Reviews and the National Academies Press / IOM. 

Issues to consider when creating a systematic review search:   

• If possible, work with a librarian experienced in conducting comprehensive literature searches.
• All major concepts are included in the strategy; typically, we do not search on the outcomes.
• Use of all appropriate controlled vocabulary terms across each database.
• Appropriate use of explosion, subheadings, and floating subheadings.
• Use of natural language (text words) in addition to controlled vocabulary terms.
• Use of appropriate synonyms, acronyms, etc.
• Truncation and spelling variation as appropriate.
• Appropriate use of limits and filters.
• Boolean operators used appropriately.
• Check indexing of exemplar articles.
• Search strategy adapted as needed for multiple databases (three minimum).
• Inclusion of grey literature sources - see below for more information. 

Search Standards

• Cochrane Handbook
• National Academies Press / IOM
• PRISMA-S Literature Search Extension
• Review authors should work closely, from the start of the protocol, with an experienced medical librarian.
• Searches should aim for high sensitivity, which may result in relatively low precision.
• Search strategies should avoid using too many different search concepts but a wide variety of search terms should be combined with OR within each included concept.
• Both free-text and subject headings (e.g. Medical Subject Headings (MeSH) and Emtree) should be used.

Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf M-I, Noel-Storr A, Rader T, Shokraneh F, Thomas J, Wieland LS. Chapter 4: Searching for and selecting studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). 

• Chapter 4: Searching for and selecting studies
• 3.1.1 Work with a librarian trained in performing systematic reviews (SRs) to plan the search strategy
• 3.1.2 Design the search strategy to address each key research question
• 3.1.3 [summarized] independent peer review of search
• 3.1.4, 3.15 Search bibliographic databases + indexes
• 3.1.6 Search literature cited by eligible studies
• 3.1.7 Update the search at intervals appropriate to the pace of generation of new information for the research question being addressed

Institute of Medicine 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press. https://doi.org/10.17226/13059.

• Standard 3.1: Conduct a comprehensive systematic search for evidence Download the free PDF; navigate to Chapter 3.

PRISMA-S is a  16-item checklist  that covers multiple aspects of the search process for systematic reviews. It is intended to guide reporting, not conduct, of the search. The checklist should be read in conjunction with the Explanation and Elaboration (Part 3), which provides more detail about each item.

Rethlefsen ML, Kirtley S, Waffenschmidt S, Ayala AP, Moher D, Page MJ, Koffel JB; PRISMA-S Group. PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews. Syst Rev. 2021 Jan 26;10(1):39. doi: 10.1186/s13643-020-01542-z. PMID: 33499930; PMCID: PMC7839230.

• Explanation and Elaboration
• Literature Search Reporting Checklist

Grey Literature: Definition and How to Search

• Importance in systematic reviews
• Searching the grey literature

" Grey literature stands for manifold document types produced on all levels of government, academics, business and industry  in print and electronic formats that are protected by intellectual property rights, of sufficient quality to be collected and preserved by libraries and institutional repositories, but not controlled by commercial publishers; i.e.  where publishing is not the primary activity of the producing body ."

-  The Twelfth International Conference on Grey Literature in Prague in 2010. 

Examples of grey literature include: conference abstracts, presentations, proceedings; regulatory data; unpublished trial data; government publications; reports (such as white papers, working papers, internal documentation); dissertations/theses; patents;  and policies & procedures.

Inclusion of grey literature into a systematic review is recommended in order to help minimize publication bias. The inclusion of grey literature in systematic reviews is widely recognized as important and international organizations have incorporated this information in their guidelines and manuals for working on reviews and meta-analyses.

• AHRQ: Finding Grey Literature Evidence and Assessing for Outcome and Analysis Reporting Biases When Comparing Medical Interventions This is a chapter from AHRQ's "Methods Guide for Effectiveness and Comparative Effectiveness Reviews."
• Cochrane Handbook: Unpublished and Onging Studies From the Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]
• Cochrane Handbook: Including unpublished studies in systematic reviews From the Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]
• The Joanna Briggs Institute Reviewer's Manual Aromataris E, Munn Z (Editors). Joanna Briggs Institute Reviewer's Manual. The Joanna Briggs Institute, 2017. Available from https://reviewersmanual.joannabriggs.org/
• PRISMA for systematic review protocols (PRISMA-P) Item 9: "Describe all intended information sources (such as electronic databases, contact with study authors, trial registers or other grey literature sources) with planned dates of coverage."

Searching the grey literature can be a daunting task. You should search those resources that make the most sense for your research question. At a minimum, consider searching unpublished trial data through trial registries as well as conference abstracts. Also check out the papers and reports of relevant stakeholder organizations.

The links below may shed some additional light on the process.

• SRC Grey Literature - Tips and Checklist
• Grey Matters! Find Grey literature Grey matters: a practical search tool for evidence-based medicine. The Canadian Agency for Drugs and Technologies in Health (CADTH) is a national body that provides Canada’s federal, provincial and territorial health care decision makers with credible, impartial advice and evidence-based information about the effectiveness and efficiency of drugs and other health technologies.
• Finding Grey Literature Evidence and Assessing for OUtcome and Analysis Reporting Biases When Comparing Medical Interventions AHRQ and the Effective Health Care Program

Sources of Grey Literature

• Abstracts & Conferences
• Pharma Data
• Clinical Trial Registries
• Repositories & Reports
• International
• PapersFirst
• Scopus Includes Embase and MEDLINE content, in addition to other journals. Also features citation reporting. However, does not include MeSH and Emtree index terms, which makes it of limited value for systematic review searching.
• Web of Science Through its unique and powerful combination of citation indexing and Web technology, the ISI Citation Databases allow current and retrospective searching of quality, peer-reviewed journals, providing complete bibliographic data, full-length author abstracts, and cited references.

The approval process for new drugs and devices involves submission of data that might not be published elsewhere. These documents can help identify publication bias even when complete methodological details of unpublished trials are unavailable. This information is not available prior to a drug’s approval and may be heavily redacted; however, when available, reviewers can compare results of published and unpublished trials.

• Drugs@FDA Presented by the US Food and Drug Administration (FDA), this site can be searched by drug name or active ingredient. Select an individual drug to view “Approval History, Letters, Reviews, and Related Documents.” Trial data are usually found in the Reviews.
• Devices@FDA This FDA site includes information about approved medical devices. It contains documents regarding approval, summary of safety and effectiveness, and relevant clinical trials, from 1995 to the present.
• Health Canada Drug Product Database (DPD) The DPD contains product-specific information on drugs approved for use in Canada. Entries may link to a drug monograph containing summary information and trial information.
• European Public Assessment Reports The European Medicines Agency publishes a full scientific assessment report called a European public assessment report (EPAR) for every medicine granted a central marketing authorisation by the European Commission.

If further information is requested, these are additional resources to consider:

• FDA: Dockets Management This website serves as the official repository for the administrative proceedings and rule-making documents for the FDA. It also provides presentations, usually in PowerPoint format, and background information on clinical trials. Transcripts may include discussion of unpublished data that was presented during hearings.
• FDA: MedWatch The FDA gateway for clinically important safety information and reporting serious problems with human medical products.
• Drug Information Portal From the National Library of Medicine
• RePORTER RePORTER is an electronic tool that allows users to search a repository of NIH-funded research projects and access publications and patents resulting from NIH funding.
• NLM: HSRProj HSRProj (Health Services Research Projects in Progress). Database of information about ongoing health services research and public health projects.
• AHRQ - Grants On Line Database Grants On-Line Database is divided into various searches. Each search includes Portfolio/Program, Priority Population and indication of whether it is a Recovery Act project.
• HSRR (Health Services and Sciences Research Resources) Information about research datasets and instruments/indices employed in Health Services Research, Behavioral and Social Sciences and Public Health with links to PubMed.
• NTIS US government-funded scientific, technical, and engineering reports.
• AstraZeneca
• Bristol-Myers Squibb Clinical Trial Registry
• Eli Lilly Company Clinical Trial Registry This site contains results and links to results from individual trials/studies only.
• GSK Clinical Trial Register The GlaxoSmithKline (GSK) Clinical Study Register provides a repository of data from GSK-Sponsored Clinical Studies, supplementing communication in journals, at scientific meetings, in letters to healthcare professionals, and in approved prescribing information.
• NovartisClinicalTrials
• Pfizer Oncology Clinical trials Note: not clear if this will lead to detailed Pfizer clinical trial data.
• Roche Clinical Trial Protocol Registry
• List of Major Pharmaceutical Companies

Online trial registries may include results of completed but unpublished clinical trials. These resources can be helpful in identifying otherwise unreachable trials and provide additional details of published trials. Many individual drug companies also have posted trial registries on their websites, although their quality varies. Even without results, knowledge that the trial exists can be helpful for reviewers, because they can contact the principal investigator for more information. 

• ClinicalTrials.gov This National Institutes of Health–sponsored site offers an advanced search screen that allows one to limit to closed trials, as well as to search by condition.
• World Health Organization (WHO) International Clinical Trials Registry Platform (ICTRP)
• CenterWatch More than 875 clinical research centers and 600 companies offering a variety of services to the clinical trials community are profiled on www.centerwatch.com. Locate research centers by medical expertise or geographic region and industry providers by type of services provided. Both profiles offer detailed information about the services they provide.
• ClinicalResearch.com
• Google Scholar Subset of Google that covers journal articles, theses, books and abstracts. NOTE: It is not clear how much of the literature is included in Google Scholar.
• Grey Literature Report New York Academy of Medicine. A quarterly list of gray literature documents in the field of public health.
• The Joanna Briggs Institute The Joanna Briggs Institute (JBI) is an international not-for-profit, membership based, research and development organisation based within the Faculty of Health Sciences at the University of Adelaide, South Australia. The Institute collaborates internationally with over 70 entities across the world. The Institute and its collaborating entities promote and support the synthesis, transfer and utilisation of evidence through identifying feasible, appropriate, meaningful and effective healthcare practices to assist in the improvement of healthcare outcomes globally.
• PROSPERO - International Register of prospective Systematic reviews PROSPERO is an international database of prospectively registered systematic reviews in health and social care. Key features from the review protocol are recorded and maintained as a permanent record in PROSPERO. This will provide a comprehensive listing of systematic reviews registered at inception, and enable comparison of reported review findings with what was planned in the protocol.
• Registry of Open Access Repositories Find potentially relevant repositories through this registry.
• The Sigma Repository (formerly the Virginia Henderson Global Nursing e-Repository) Sigma repository is dedicated to sharing works created by nurses around the world. It is an open digital academic and clinical focused service that freely collects, preserves, and disseminates full-text nursing research, educational, and evidence-based practice materials in a variety of formats and item types. The repository is a resource of Sigma Theta Tau International (Sigma).
• Centre for Reviews and Dissemination (UK) We provide research-based information about the effects of health and social care interventions via our databases and undertake systematic reviews evaluating the research evidence on health and public health questions of national and international importance.
• INAHTA (International Network of Agencies for Health Technology Assessment) INAHTA (International Network of Agencies for Health Technology Assessment) a non-profit organization was established in 1993 and has now grown to 53 member agencies from 29 countries including North and Latin America, Europe, Africa, Asia, Australia, and New Zealand.
• Institute for Scientific and Technical Information (INstitut de l’Information Scientifique et Technique - Inist) of the French National Center for Scientific Research (Cnrs) For over a decade, the Institute for Scientific and Technical Information (INstitut de l’Information Scientifique et Technique - Inist) of the French National Center for Scientific Research (Cnrs) has been designing and developing services to facilitate access to and valorization of scientific information produced by international basic and applied research communities.
• International Clinical Trials Registry Platform (ICTRP) The mission of the WHO International Clinical Trials Registry Platform is to ensure that a complete view of research is accessible to all those involved in health care decision making.
• Lenus (The Irish Health Repository)
• NARCIS - National Academic Research and Collaborations Information System (Netherlands) NARCIS is thé national portal for those who are in search of information about researchers and their work.
• RIAN (Pathways to Irish Research)
• UK Clinical Research Network Study Portfolio The UKCRN Portfolio comprises four smaller portfolios of studies belonging to the four nations of the United Kingdom.
• Virtual Health Library
• WHO WHO Publications
• << Previous: 3. Write and Register a Protocol
• Next: 5. Screen Results >>
• Last Updated: Jun 18, 2024 9:41 AM
• URL: https://guides.mclibrary.duke.edu/sysreview
• Duke Health
• Duke University
• Duke Libraries
• Medical Center Archives
• Duke Directory
• Seeley G. Mudd Building
• 10 Searle Drive
• [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses

Affiliations.

• 1 Behavioural Science Centre, Stirling Management School, University of Stirling, Stirling FK9 4LA, United Kingdom; email: [email protected].
• 2 Department of Psychological and Behavioural Science, London School of Economics and Political Science, London WC2A 2AE, United Kingdom.
• 3 Department of Statistics, Northwestern University, Evanston, Illinois 60208, USA; email: [email protected].
• PMID: 30089228
• DOI: 10.1146/annurev-psych-010418-102803

Systematic reviews are characterized by a methodical and replicable methodology and presentation. They involve a comprehensive search to locate all relevant published and unpublished work on a subject; a systematic integration of search results; and a critique of the extent, nature, and quality of evidence in relation to a particular research question. The best reviews synthesize studies to draw broad theoretical conclusions about what a literature means, linking theory to evidence and evidence to theory. This guide describes how to plan, conduct, organize, and present a systematic review of quantitative (meta-analysis) or qualitative (narrative review, meta-synthesis) information. We outline core standards and principles and describe commonly encountered problems. Although this guide targets psychological scientists, its high level of abstraction makes it potentially relevant to any subject area or discipline. We argue that systematic reviews are a key methodology for clarifying whether and how research findings replicate and for explaining possible inconsistencies, and we call for researchers to conduct systematic reviews to help elucidate whether there is a replication crisis.

Keywords: evidence; guide; meta-analysis; meta-synthesis; narrative; systematic review; theory.

PubMed Disclaimer

Similar articles

• The future of Cochrane Neonatal. Soll RF, Ovelman C, McGuire W. Soll RF, et al. Early Hum Dev. 2020 Nov;150:105191. doi: 10.1016/j.earlhumdev.2020.105191. Epub 2020 Sep 12. Early Hum Dev. 2020. PMID: 33036834
• A Primer on Systematic Reviews and Meta-Analyses. Nguyen NH, Singh S. Nguyen NH, et al. Semin Liver Dis. 2018 May;38(2):103-111. doi: 10.1055/s-0038-1655776. Epub 2018 Jun 5. Semin Liver Dis. 2018. PMID: 29871017 Review.
• Publication Bias and Nonreporting Found in Majority of Systematic Reviews and Meta-analyses in Anesthesiology Journals. Hedin RJ, Umberham BA, Detweiler BN, Kollmorgen L, Vassar M. Hedin RJ, et al. Anesth Analg. 2016 Oct;123(4):1018-25. doi: 10.1213/ANE.0000000000001452. Anesth Analg. 2016. PMID: 27537925 Review.
• Summarizing systematic reviews: methodological development, conduct and reporting of an umbrella review approach. Aromataris E, Fernandez R, Godfrey CM, Holly C, Khalil H, Tungpunkom P. Aromataris E, et al. Int J Evid Based Healthc. 2015 Sep;13(3):132-40. doi: 10.1097/XEB.0000000000000055. Int J Evid Based Healthc. 2015. PMID: 26360830
• RAMESES publication standards: meta-narrative reviews. Wong G, Greenhalgh T, Westhorp G, Buckingham J, Pawson R. Wong G, et al. BMC Med. 2013 Jan 29;11:20. doi: 10.1186/1741-7015-11-20. BMC Med. 2013. PMID: 23360661 Free PMC article.
• Surveillance of Occupational Exposure to Volatile Organic Compounds at Gas Stations: A Scoping Review Protocol. Mendes TMC, Soares JP, Salvador PTCO, Castro JL. Mendes TMC, et al. Int J Environ Res Public Health. 2024 Apr 23;21(5):518. doi: 10.3390/ijerph21050518. Int J Environ Res Public Health. 2024. PMID: 38791733 Free PMC article. Review.
• Association between poor sleep and mental health issues in Indigenous communities across the globe: a systematic review. Fernandez DR, Lee R, Tran N, Jabran DS, King S, McDaid L. Fernandez DR, et al. Sleep Adv. 2024 May 2;5(1):zpae028. doi: 10.1093/sleepadvances/zpae028. eCollection 2024. Sleep Adv. 2024. PMID: 38721053 Free PMC article.
• Barriers to ethical treatment of patients in clinical environments: A systematic narrative review. Dehkordi FG, Torabizadeh C, Rakhshan M, Vizeshfar F. Dehkordi FG, et al. Health Sci Rep. 2024 May 1;7(5):e2008. doi: 10.1002/hsr2.2008. eCollection 2024 May. Health Sci Rep. 2024. PMID: 38698790 Free PMC article.
• Studying Adherence to Reporting Standards in Kinesiology: A Post-publication Peer Review Brief Report. Watson NM, Thomas JD. Watson NM, et al. Int J Exerc Sci. 2024 Jan 1;17(7):25-37. eCollection 2024. Int J Exerc Sci. 2024. PMID: 38666001 Free PMC article.
• Evidence for Infant-directed Speech Preference Is Consistent Across Large-scale, Multi-site Replication and Meta-analysis. Zettersten M, Cox C, Bergmann C, Tsui ASM, Soderstrom M, Mayor J, Lundwall RA, Lewis M, Kosie JE, Kartushina N, Fusaroli R, Frank MC, Byers-Heinlein K, Black AK, Mathur MB. Zettersten M, et al. Open Mind (Camb). 2024 Apr 3;8:439-461. doi: 10.1162/opmi_a_00134. eCollection 2024. Open Mind (Camb). 2024. PMID: 38665547 Free PMC article.
• Search in MeSH

LinkOut - more resources

Full text sources.

• Ingenta plc
• Ovid Technologies, Inc.

Other Literature Sources

• scite Smart Citations

Miscellaneous

• NCI CPTAC Assay Portal
• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

• University of Texas Libraries
• UT Libraries

Systematic Reviews & Evidence Synthesis Methods

• Types of Reviews
• Formulate Question
• Find Existing Reviews & Protocols
• Register a Protocol
• Searching Systematically
• Supplementary Searching
• Managing Results
• Deduplication
• Critical Appraisal
• Glossary of terms
• Librarian Support
• Video tutorials This link opens in a new window
• Systematic Review & Evidence Synthesis Boot Camp

Where to Search

In order to cover your bases, you'll want to have (and document) a thorough search strategy that includes:

• Databases and indexes
• Reference lists of existing review articles
• Reference lists and "cited by" articles of the studies chosen for inclusion in your evidence synthesis project
• Key journals
• Contacting authors
• Publications of relevant organizations
• Grey literature sources, which may include preprints, dissertations, registered trials, conference proceedings, conference abstracts, white papers, and other types of unpublished literature.

Search Process Videos

• Determine Search Needs
• Identify Databases & Journals, Pt. 1
• Identify Databases & Journals, Pt. 2
• Identify Databases & Journals, Pt. 3

Three Main Types of Searches

• Exploratory
• Hand, or Supplementary

Generally speaking, exploratory searching is where you start . This includes searching for existing reviews and getting a rough idea of the scope of the literature. It is also a way for you to test your search strategy and individual terms before conducting the systematic search. Exploratory searching should guide you in developing your research question and eligibility criteria as well as developing your search strategy. We highly recommend reaching out to a librarian for help with exploratory searching.

The systematic search is what you will document in the methods section of your paper. Once you have finalized your search strategy, you will implement that search in each of your chosen databases individually. The more rigorous guidelines expect you to document your search, the number of search results per database, and the date on which the search was conducted.

Hand or supplementary searching picks up articles that may be missed in your systematic search and will occur concurrently with exploratory and systematic searching as well as through the screening stage. This type of search includes:

• directly searching highly relevant journals that may not be indexed in your chosen databases
• looking at the recent edition of journals, whose articles may not be indexed yet in your chosen database
• looking at research that does not appear in academic journals, like clinical trials, dissertations, websites of relevant organizations, conference abstracts and proceedings, and other gray literature. 
• Contact relevant authors/experts on the topic for additional recommendations

Once you have determined the articles you will use in your review, you should

• comb through their reference lists 
• find the "cited by" articles  (i.e. articles that cite the articles you are using in your review) 
• through these two acts, you may pick up additional articles that your own searches may have missed.

Recommended Databases

• Multi-disciplinary
• Biological Sciences
• Business/Economics
• Communication & Speech Disorders
• Engineering
• Health Sciences
• Social Sciences

Best practice for systematic searching recommends searching at least 3 databases including one that is multi-disciplinary.

To help you determine which databases will be best bets for your research questions, you can ask a librarian, but you'll also want to look at existing reviews that ask similar questions to your research question and likely you will want to use a similar list of databases to the ones they report.

Identify and test your search terms in your identified databases before conducting your final structured search.

The Web of Science platform currently also provides temporary access to several databases that are not part of the Core Collection, including Biosis Citation Index, Data Citation Index, and Zoological Record.

• Zoological Record Select "Zoological Record" from the drop down menu next to "Search In:"
• BIOSIS Citation Index Select "BIOSIS Citation Index" from the drop down menu next to "Search In:"

Searching Broadly Across the Engineering Literature

• NHS Economic Evaluation Database (NHS EED)
• SocINDEX with Full Text This link opens in a new window 1895 - present. Unlimited. more... less... In addition to full-text journals, SocINDEX with Full Text contains informative abstracts for core coverage journals dating as far back as 1895. Includes indexing for books, monographs, conference papers and other non-periodical content sources.
• Identifying the Studies, Part I, II, III From Assembling the Pieces of a Systematic Review, by Margaret Foster and Sarah Jewell. 2017. Part 1: Database Searching Part 2: Beyond Database Searching Part 3: Case Studies
• Searching the Literature In How to Perform a Systematic Literature Review, by Edward Purssell and Niall McCrae. 2020
• Last Updated: Jun 26, 2024 3:15 PM
• URL: https://guides.lib.utexas.edu/systematicreviews

Systematic Reviews and Meta-Analyses: How to Search

• Get Started
• Exploratory Search
• Where to Search
• How to Search
• Grey Literature
• What about errata and retractions?
• Eligibility Screening
• Critical Appraisal
• Data Extraction
• Synthesis & Discussion
• Assess Certainty
• Share & Archive

This page provides more information about how  to design a search strategy.

The short answer: Start by creating a base search strategy, then translate across all your sources

Designing a Comprehensive Search

Overview of how to search.

In a systematic review and/or meta-analysis, the search should collect all potentially relevant material that is available. In other words, the search must be  comprehensive .

Though  exact strategies  will vary according to  where  you search, you can start by designing a base search strategy by following these steps:

• Identify important concepts
• Create exhaustive list of synonyms 
• Test out natural language terms to confirm usefulness and find controlled vocabulary 
• Join synonyms within a concept using the OR operator
• Join concepts using the AND operator

Check out the content below for more about designing a comprehensive search strategy.

Bramer, W. M., De Jonge, G. B., Rethlefsen, M. L., Mast, F., & Kleijnen, J. (2018).  A systematic approach to searching: An efficient and complete method to develop literature searches.   Journal of the Medical Library Association ,  106 (4).  https://doi.org/10.5195/JMLA.2018.283

More detail

• Controlled vocabulary (CV)
• Build the Search Statement
• Database Limits, Filters, Hedges
• Translate Across Databases

Download All Results

Choose concepts to search for.

Concepts come directly from your research question  and  eligibility criteria  - they are the basic ideas underlying your scope. However, not all concepts will be appropriate to include in the search strategy.

Bramer, et. al., (2018) suggest mapping concepts on a scale of specificity and importance ( see image to the left ). Ideally, a search will only include specific, important terms . In some cases, it may be necessary to use more general terms. However, unimportant terms should never be included in the search strategy.

Guiding Consideration

When considering whether a concept should be included or not, ask yourself: I s it possible for an article to be relevant and  not  contain terms related to this concept? If yes, the concept should not be included. If no, the concept would be appropriate to include in the search. This logic can also be used when determining which  terms  should be included in your search strategy.

Exhaustive Synonyms 

Some concepts are well-defined, meaning a fairly uniform set of terms related to that concept are used within a field. But this is not always the case, and identifying an exhaustive list of synonyms can be difficult. 

Start by identifying synonyms based on your team's preexisting knowledge - but never stop here!

Look at the titles and abstracts of seminal articles   and other relevant reviews  to find terms, spellings, etc. that you may not have considered. You may also look to collaborations and experts in your field who may be able to provide feedback on the comprehensiveness of your terms. LitsearchR is a tool that can partially automate this process.

Grames, EM, AN Stillman, MW Tingley, and CS Elphick (2019). An automated approach to identifying search terms for systematic reviews using keyword co-occurrence networks .  Methods in Ecology and Evolution  10: 1645-1654.  https://doi.org/10.1111/2041-210X.13268

Finally, controlled vocabulary   descriptions often include a list of relevant terms to consider adding to your list of natural language terms. See the next tab for more details about controlled vocabulary.  Hedges can also be helpful in identifying synonyms!

Considerations

Depending on your scope  (e.g., geographic region, timeframe) be sure to consider other languages , alternate spellings (e.g., US v. UK English), and changes in terminology over time . 

Video:  Developing Keywords

If you're new to developing keywords, this video is a great place to start:

Controlled Vocabulary

Controlled vocabulary, also known as  subject headings, thesaurus terms,  or indexed terms  describes  predefined terms established by the host of a database that are manually applied to articles related to that term. This is a normalizing tool used to curate all related material, despite the unique terms authors use. In effect, searching for a controlled vocabulary term will result in any article  categorized   under that term, even if the original authors didn't use that exact term. This way, you are less likely to miss relevant results that don't use your exact terminology.

Unfortunately, because controlled vocabulary is manually applied to articles , if you rely solely on controlled vocabulary, you may miss material that has not yet been processed (e.g., new publications, older backlogged material). Therefore, always use controlled vocabulary  with natural language terms .

If you're searching a database(s) that uses Medical Subject Heading (MeSH) terms, quickly identify terms relevant to your topic using the  MeSH On Demand tool from the National Library of Medicine (NLM).

Build the Search Statement 

Once you've identified all of the relevant terms in both natural language and controlled vocabulary*, it's time to structure the search string. This is done using  boolean operators ,  parentheses , quotations,  and potentially many more syntax operators. Below, we describe some important details related to search string development.

*remember, controlled vocabulary varies across databases

Video:  From Keywords to Database

If this is your first time building a search string, check out this video.

Limiting & Using Hedges

The built-in filters ( like those highlighted in this image from PubMed ) are manually applied to the content in a database. Therefore, they are inappropriate to use for systematic reviews, as older and newer material that has not yet been processed will be missed.

What is a hedge?

Hedges are validated sets of terms that aim to retrieve specific content, often including natural language terms and controlled vocabulary. These search strings are tailored to a specific database and intended for use beyond a single project.

Hedges, which can also be called filters, are a more systematic, comprehensive counterpart to the filters/limiters built into databases often found on the side of a results page, as seen in the example from PubMed. For an example of a hedge, check out the Canadian Health Libraries Association hedge for studies focused on adolescent and young adult populations . 

Both built-in filters and hedges can also be helpful when identifying synonyms!

Finding Hedges

Though there won't be hedges available for every discipline or topic, you can search in a web browser for each of your concepts accompanied by the term "hedge" or "filter". Here are some (mostly health) collections of to get you started!

Collections of Hedges

• InterTASC Information Specialists'' Sub-Group (ISSG) - ISSG Search Filter Resource
• CADTH - Search Filters Database
• MD Anderson Cancer Center Library Guide - Literature Search Hedges & Filters Guide
• University of Alberta - Health Science Search Filters Library Guide
• American University of Beirut - Systematic Reviews: Health Sciences - Search Filters/Hedges
• Library of Search Strategy Resources (LSSR) from the European Association for Health Information and Libraries
• Health Improvement Scotland - SIGN Search Filters
• McMaster - Health Knowledge Refinery

Translating across databases

Translating in this case is mostly about syntax,  or how the database interprets your search query. Databases may share common search syntax features like boolean operators , parentheses, and quotations, however many features vary in terms of availability and syntax.

For example, you may be able to truncate  in most academic journal databases. In some databases the symbol for truncation may be an asterisk (*) and in others, a question mark (?) . In another example,  Proximity operators , which allow you to search for terms that are within a certain number of words apart, are only available in some databases , but not all. 

Controlled vocabulary will be unique to both the scope of your review and each database .  

Using Word Macros for Translation

As you might imagine, making these adjustments for long search strings across several databases can be time consuming and tedious.  Wichor Bramer, et al., (2018) outlined the differences in syntax across the top 5 most used databases for biomedical literature ( Table 1 ) and describe a translation process (Item 14) aided by macros in MS Word. If you will be using these databases, you can set up macros by following these instructions !

Using Polyglot from the SR-Accelerator for Translation across Health Sciences Databases

Polyglot is a semi-automated tool available through the SR-Accelerator  that allows you to translate your search strategy to databases commonly used in health science systematic reviews. However, there are more general purpose databases Scopus and platforms like Web of Science options that you could use for topics outside of health sciences!

After executing the search in each database, you'll download  all  of the results into a citation manager . Note, there may be limits to how many results can be downloaded at once. Downloading all of the results instead of cherry picking seemingly relevant articles from the database is an important mechanism for reducing bias . However, this step is also helpful in terms of project management, as it is much easier to keep track of all results in one central library or even a single RIS file. 

In the next stages of the systematic review, you'll  systematically process  all of these articles through the  eligibility screening , critical appraisal , and data extraction .

More about the Process

• Peer Review of Search
• Update the Search

Documenting the Search

Peer review.

According to  Cochrane, Part 2: Chapter 4, Section 4.8 :

Peer review of search strategies is increasingly recognized as a necessary step in designing and executing high-quality search strategies to identify studies for possible inclusion in systematic reviews.

Get your search peer-reviewed!

Library and information science professionals are a great resource for peer-reviewing a completed search strategy. Peer reviewers may rely on guidelines such as the PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement . Check out the Evidence Synthesis Services (ESS) current support offerings .

Updates 

According to the Cochrane Handbook 4.4.10 Timing of Searches:

Searches for all the relevant databases should be rerun prior to publication, if the initial search date is more than 12 months (preferably six months) from the intended publication date (see  MECIR Box 4.4.g ). This is also good practice for searches of non-database sources. The results should also be screened to identify potentially eligible studies. Ideally, the studies should be incorporated fully in the review.

Additional Guidance

Bramer, W., & Bain, P. (2017). Updating search strategies for systematic reviews using EndNot e. Journal of the Medical Library Association , 105 (3). https://doi.org/10.5195/JMLA.2017.183

Documenting the search with enough detail that another team could feasibly replicate it is a requirement for systematic reviews and/or meta-analyses. PRISMA-S  outlines information that is necessary to report in the final manuscript. It also helps to keep thorough documentation  throughout  the search development and execution.

Tools for documenting search development

While developing the search strategy, you should also keep a search journal to track sources searched, when a search is executed, the exact search statement used, limits, results, etc. You can make a copy of this search strategy journal (template)  from the Evidence Synthesis Services (ESS) team to track when, how, and where you searched! This is a great way to increase replicability and transparency of the final manuscript ( e.g., by including as supplemental material ), as well as keeping track of your team's progress mid-review.

Once you've finalized your search approach, you can share it publicly . For example the searchrxiv , hosted by CABI, is available as a repository to share search strings from any a review in any discipline.

• << Previous: Where to Search
• Next: Grey Literature >>
• Last Updated: Jun 26, 2024 2:52 PM
• URL: https://guides.lib.vt.edu/SRMA

• University of Michigan Library
• Research Guides

Systematic Reviews

• Search Strategy
• Work with a Search Expert
• Covidence Review Software
• Types of Reviews
• Evidence in a Systematic Review
• Information Sources

Developing an Answerable Question

Creating a search strategy, identifying synonyms & related terms, keywords vs. index terms, combining search terms using boolean operators, a sr search strategy, search limits.

• Managing Records
• Selection Process
• Data Collection Process
• Study Risk of Bias Assessment
• Reporting Results
• For Search Professionals

Validated Search Filters

Depending on your topic, you may be able to save time in constructing your search by using specific search filters (also called "hedges") developed & validated by researchers in the Health Information Research Unit (HiRU) of McMaster University, under contract from the National Library of Medicine.  These filters can be found on

• PubMed’s Clinical Queries &  Health Services Research Queries pages
• Ovid Medline’s Clinical Queries  filters or here
• Embase  & PsycINFO
• EBSCOhost’s main search page for CINAHL (Clinical Queries category)
• HiRU’s Nephrology Filters page
• American U of Beirut, esp. for " humans" filters .
• Countway Library of Medicine methodology filters
• InterTASC Information Specialists' Sub-Group Search Filter Resource
• SIGN (Scottish Intercollegiate Guidelines Network) filters page

Why Create a Sensitive Search?

In many literature reviews, you try to balance the sensitivity of the search (how many potentially relevant articles you find) &  specificit y (how many definitely relevant articles  you find ), realizing that you will miss some.  In a systematic review, you want a very sensitive search:  you are trying to find any potentially relevant article.  A systematic review search will:

• contain many synonyms & variants of search terms
• use care in adding search filters
• search multiple resources, databases & grey literature, such as reports & clinical trials

PICO is a good framework to help clarify your systematic review question.

P -   Patient, Population or Problem: What are the important characteristics of the patients &/or problem?

I -  Intervention:  What you plan to do for the patient or problem?

C -  Comparison: What, if anything, is the alternative to the intervention?

O -  Outcome:  What is the outcome that you would like to measure?

Beyond PICO: the SPIDER tool for qualitative evidence synthesis.

5-SPICE: the application of an original framework for community health worker program design, quality improvement and research agenda setting.

A well constructed search strategy is the core of your systematic review and will be reported on in the methods section of your paper. The search strategy retrieves the majority of the studies you will assess for eligibility & inclusion. The quality of the search strategy also affects what items may have been missed.  Informationists can be partners in this process.

For a systematic review, it is important to broaden your search to maximize the retrieval of relevant results.

Use keywords:  How other people might describe a topic?

Identify the appropriate index terms (subject headings) for your topic.

• Index terms differ by database (MeSH, or  Medical Subject Headings ,   Emtree terms , Subject headings) are assigned by experts based on the article's content.
• Check the indexing of sentinel articles (3-6 articles that are fundamental to your topic).  Sentinel articles can also be used to  test your search results.

Include spelling variations (e.g., behavior, behaviour ).  

Both types of  search terms are useful & both should be used in your search.

Keywords help to broaden your results.  They will be searched for at least in journal titles, author names, article titles, & article abstracts.  They can also be tagged to search all text.

Index/subject terms  help to focus your search appropriately, looking for items that have had a specific term applied by an indexer.

Boolean operators let you combine search terms in specific ways to broaden or narrow your results.

An example of a search string for one concept in a systematic review.

In this example from a PubMed search, [mh] = MeSH &  [tiab] = Title/Abstract, a more focused version of a keyword search.

A typical database search limit allows you to narrow results so that you retrieve articles that are most relevant to your research question. Limit types vary by database & include:

• Article/publication type
• Publication dates

In a systematic review search, you should use care when applying limits, as you may lose articles inadvertently.  For more information, see, particularly regarding language & format limits.     Cochrane 2008 6.4.9

Systematic Review

• Library Help
• What is a Systematic Review (SR)?
• Steps of a Systematic Review
• Framing a Research Question

Developing a Search Strategy

• Searching the Literature
• Managing the Process
• Meta-analysis
• Publishing your Systematic Review

Workshop materials

• PICO Worksheet
• Search Strategy Example
• Search Strategy Presentation Slides
• Search strings for demo

Find step-by-step instructions on how to develop a search strategy on p. 44

Errors in search strategies

Salvador-Oliván, J. A., Marco-Cuenca, G., & Arquero-Avilés, R. (2019).  Errors in search strategies used in systematic reviews and their effects on information retrieval .  Journal of the Medical Library Association : JMLA ,  107 (2), 210–221.  https://doi.org/10.5195/jmla.2019.567 . 

• Search Term Harvesting
• Text Mining Tools
• Search Filters / Hedges
• Documenting
• Blogs & Discussion Lists

Translating search strategies across databases

• ChatGPT Ask ChatGPT using this prompt, "Covert this search into terms appropriate for the [name] database." Further reading: Wang, S., Scells, H., Koopman, B., & Zuccon, G. (2023). Can ChatGPT write a good boolean query for systematic review literature search?. arXiv preprint arXiv:2302.03495.
• LitSonar Use the Help section for further guidance on how to use this tool (https://litsonar.com/help). Capable of searching eight different databases simultaneously
• Polyglot Use the Polyglot tool to translate search strings from PubMed across multiple databases. Access the tool's tutorial for more information (https://sr-accelerator.com/#/help/polyglot). 
• MEDLINE Transpose Use this tool to translate your MEDLINE (PubMed) search to MEDLINE (Ovid) format or vice versa.
• Database Syntax Guide ​Guide to translating syntax for multiple databases. From Cochrane.

_____________________________________________________________

Take control of your search and turn off Pubmed's  Automatic Term Mapping (ATM) ! It will not include all variant terminology and automatically explodes MeSH terms. Not using ATM allows for clearer documentation of the search method.

For more information on Automatic Term Mapping, watch the video below .

Further readings

• Burns, C. S., Nix, T., Shapiro, R. M., & Huber, J. T. (2021). MEDLINE search retrieval issues: A longitudinal query analysis of five vendor platforms. PLoS ONE , 16 (5), e0234221. https://doi.org/10.1371/journal.pone.0234221

• PubMed Pub ReMiner  - Text mining for PubMed to look at commonalities between MeSH terms and keywords
• Go PubMed  - Text mining tool for PubMed or MeSH terms. This article  explains the features of this text mining tool.
• PubVenn - This tool enables you to explore PubMed using venn diagrams. Also, try Search Workbench .
• Yale MeSH Analyzer  - Watch this tutorial (7 min.). This tool allows users to enter up to 20 PubMed ID numbers, which it uses to aggregate the metadata from the associated articles into a spreadsheet. For systematic reviews, it is useful in search strategy development to quickly aggregate the Medical Subject Heading (MeSH) terms associated with relevant articles. While it only works for PubMed, it can be useful for developing searches in medical-adjacent fields, such as psychology, nutrition, and animal health. 
• NCBI MeSH on Demand  identifies MeSH® terms in your submitted text (abstract or manuscript). MeSH on Demand also lists PubMed similar articles relevant to your submitted text.
• HelioBLAST - This tool finds text records that are similar to the submitted query. Your query is searched against the citations (abstract and titles) in Medline/PubMed and the top matching articles are returned in the results.
• Coremine - It is ideal for those seeking an overview of a complex subject while allowing the possibility to "drill down" to specific details. Instructions
• Carrot2 - This tool can automatically organize search results into topics. It can query PubMed and allows boolean searching.
• SWIFT-Review - Desktop text mining tool specific to systematic reviews. To obtain your free license for SWIFT Review, simply browse to the  Sciome Software  web page to login and/or create your SWIFT-Review account.  
• Voyant  - General text mining (this is the download). For the web version go to http://voyant-tools.org
• TerMine  - General text mining
• JSTOR Text Analyzer - Recommends journal articles in JSTOR relevant to text.
• CREBP-SRA Word Frequency Analyser (WFA) - This tool helps determine which words you should use to construct and refine a search strategy
• Medline Ranker  requires a set of known relevant records with PubMed identifiers and a test set of records (e.g. search results from a highly sensitive search). Medline Ranker sorts the records in the test set and presents those that were most similar to the relevant records first. Medline Ranker also provides a list of discriminating terms which discriminate relevant records from non-relevant records.

_________________________________________________________________________

For more information on text mining tools - review and comparison, read the following article:

Paynter, R., Bañez, L. L., Berliner, E., Erinoff, E., Lege-Matsuura, J., Potter, S., & Uhl, S. (2016). EPC methods: an exploration of the use of text-mining software in systematic reviews .

You might limit to a particular publication type in Pubmed. See a full list of Pubmed publication types .

• Cochrane Handbook Part 2, Section 6.4.11 provides search filters to limit to randomized controlled trials in Medline/PubMed, Medline/Ovid, and Embase
• McMaster - Filters by the Hedges team

• PubMed Systematic Review Filter Search Strategy
• Search Filters from Univ. of Texas School of Public Health

Hedges by Topic (in alphabetical order )

• ​ Prady, S. L., Uphoff, E. P., Power, M., & Golder, S. (2018). Development and validation of a search filter to identify equity-focused studies: Reducing the number needed to screen. BMC Medical Research Methodology, 18 (1), 106. https://doi.org/10.1186/s12874-018-0567-x
• Health Risk Assessment by Vicky Tessier at the INSPQ
• Effectiveness of Interventions
• van der Mierden, S., Hooijmans, C. R., Tillema, A. H., Rehn, S., Bleich, A., & Leenaars, C. H. (2022). Laboratory animals search filter for different literature databases: PubMed, Embase, Web of Science and PsycINFO.  Laboratory animals ,  56 (3), 279–286. https://doi.org/10.1177/00236772211045485

• Updated Press Checklist (2015) See page 41-42
• IOM Standards for Systematic Reviews
• PRESS Checklist

McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C.  PRESS Peer Review of Electronic Search Strategies: 2015 guideline statement .  Journal of Clinical  Epidemiology, 75 , 40-46. 

Systematic Literature Review Worksheet

Use the Database Search Log to record your search terms, search strategy and databases searched.

Guidance on Reporting Systematic Reviews

Cochrane strongly encourages that review authors include a study flow diagram as recommended by the PRISMA statement.

• PRISMA Flow Diagram
• PRISMA Flow Diagram Generator
• PRISMA Checklist

Other checklists include:

• ARRIVE and DSPC for animal studies
• MOOSE - meta-analysis of observational studies in epidemiology
• STARLITE - general health policy and clinical practice
• TIDier-PHP - population health and policy interventions

Examples of documented search methodologies:

• Full search strategies for all database searches provided in the Appendices:

Bath, P. & Krishnan, K. (2014). Interventions for deliberately altering blood pressure in acute stroke .  Cochrane Database of Systematic Reviews, 10.

• A summary of sources searched and keywords used in the Sources section:

McIntyre, S, Taitz, D, Keogh, J, Goldsmith, S, Badawi, N & Blair, E. (2013). A systematic review of risk factors for cerebral palsy in children born at term in developed countries . Developmental Medicine & Child Neurology, 55( 6), 499-508.

• ACRL Systematic Reviews & Related Methods Interest Group [email protected]
• Cindy Schmidt's Blog: PubMed Search Strategies This blog has been created to share PubMed search strategies. Search strategies posted here are not perfect. They are posted in the hope that others will benefit from the work already put into their creation and/or will offer suggestions for improvements.
• MedTerm Search Assist from the University of Pittsburgh By librarians for librarians - A database to share biomedical terminology and strategies for comprehensive searches.
• MLA expert searching discussion list [email protected] - This discussion list often discusses subject strategies and sometimes search filters.
• PRESS Forum This closed wiki-based forum is a place for librarians to request reviews of systematic review search strategies, and to review the searches of others.
• << Previous: Framing a Research Question
• Next: Searching the Literature >>
• Last Updated: May 8, 2024 1:44 PM
• URL: https://lib.guides.umd.edu/SR

What is a Systematic Literature Review?

A systematic literature review (SLR) is an independent academic method that aims to identify and evaluate all relevant literature on a topic in order to derive conclusions about the question under consideration. "Systematic reviews are undertaken to clarify the state of existing research and the implications that should be drawn from this." (Feak & Swales, 2009, p. 3) An SLR can demonstrate the current state of research on a topic, while identifying gaps and areas requiring further research with regard to a given research question. A formal methodological approach is pursued in order to reduce distortions caused by an overly restrictive selection of the available literature and to increase the reliability of the literature selected (Tranfield, Denyer & Smart, 2003). A special aspect in this regard is the fact that a research objective is defined for the search itself and the criteria for determining what is to be included and excluded are defined prior to conducting the search. The search is mainly performed in electronic literature databases (such as Business Source Complete or Web of Science), but also includes manual searches (reviews of reference lists in relevant sources) and the identification of literature not yet published in order to obtain a comprehensive overview of a research topic.

An SLR protocol documents all the information gathered and the steps taken as part of an SLR in order to make the selection process transparent and reproducible. The PRISMA flow-diagram support you in making the selection process visible.

In an ideal scenario, experts from the respective research discipline, as well as experts working in the relevant field and in libraries, should be involved in setting the search terms . As a rule, the literature is selected by two or more reviewers working independently of one another. Both measures serve the purpose of increasing the objectivity of the literature selection. An SLR must, then, be more than merely a summary of a topic (Briner & Denyer, 2012). As such, it also distinguishes itself from “ordinary” surveys of the available literature. The following table shows the differences between an SLR and an “ordinary” literature review.

• Charts of BSWL workshop (pdf, 2.88 MB)
• Listen to the interview (mp4, 12.35 MB)

Differences to "common" literature reviews

What are the objectives of SLRs?

• Avoidance of research redundancies despite a growing amount of publications
• Identification of research areas, gaps and methods
• Input for evidence-based management, which allows to base management decisions on scientific methods and findings
• Identification of links between different areas of researc

Process steps of an SLR

A SLR has several process steps which are defined differently in the literature (Fink 2014, p. 4; Guba 2008, Transfield et al. 2003). We distinguish the following steps which are adapted to the economics and management research area:

1. Defining research questions

Briner & Denyer (2009, p. 347ff.) have developed the CIMO scheme to establish clearly formulated and answerable research questions in the field of economic sciences:

C – CONTEXT:  Which individuals, relationships, institutional frameworks and systems are being investigated?

I – Intervention:  The effects of which event, action or activity are being investigated?

M – Mechanisms:  Which mechanisms can explain the relationship between interventions and results? Under what conditions do these mechanisms take effect?

O – Outcomes:  What are the effects of the intervention? How are the results measured? What are intended and unintended effects?

The objective of the systematic literature review is used to formulate research questions such as “How can a project team be led effectively?”. Since there are numerous interpretations and constructs for “effective”, “leadership” and “project team”, these terms must be particularized.

With the aid of the scheme, the following concrete research questions can be derived with regard to this example:

Under what conditions (C) does leadership style (I) influence the performance of project teams (O)?

Which constructs have an effect upon the influence of leadership style (I) on a project team’s performance (O)?          

Research questions do not necessarily need to follow the CIMO scheme, but they should:

• ... be formulated in a clear, focused and comprehensible manner and be answerable;
• ... have been determined prior to carrying out the SLR;
• ... consist of general and specific questions.

As early as this stage, the criteria for inclusion and exclusion are also defined. The selection of the criteria must be well-grounded. This may include conceptual factors such as a geographical or temporal restrictions, congruent definitions of constructs, as well as quality criteria (journal impact factor > x).

2. Selecting databases and other research sources

The selection of sources must be described and explained in detail. The aim is to find a balance between the relevance of the sources (content-related fit) and the scope of the sources.

In the field of economic sciences, there are a number of literature databases that can be searched as part of an SLR. Some examples in this regard are:

• Business Source Complete
• ProQuest One Business
• EconBiz        

Our video " Selecting the right databases " explains how to find relevant databases for your topic.

Literature databases are an important source of research for SLRs, as they can minimize distortions caused by an individual literature selection (selection bias), while offering advantages for a systematic search due to their data structure. The aim is to find all database entries on a topic and thus keep the retrieval bias low (tutorial on retrieval bias ).  Besides articles from scientific journals, it is important to inlcude working papers, conference proceedings, etc to reduce the publication bias ( tutorial on publication bias ).

Our online self-study course " Searching economic databases " explains step 2 und 3.

3. Defining search terms

Once the literature databases and other research sources have been selected, search terms are defined. For this purpose, the research topic/questions is/are divided into blocks of terms of equal ranking. This approach is called the block-building method (Guba 2008, p. 63). The so-called document-term matrix, which lists topic blocks and search terms according to a scheme, is helpful in this regard. The aim is to identify as many different synonyms as possible for the partial terms. A precisely formulated research question facilitates the identification of relevant search terms. In addition, keywords from particularly relevant articles support the formulation of search terms.

A document-term matrix for the topic “The influence of management style on the performance of project teams” is shown in this example .

Identification of headwords and keywords

When setting search terms, a distinction must be made between subject headings and keywords, both of which are described below:

• appear in the title, abstract and/or text
• sometimes specified by the author, but in most cases automatically generated
• non-standardized
• different spellings and forms (singular/plural) must be searched separately

Subject headings

• describe the content
• are generated by an editorial team
• are listed in a standardized list (thesaurus)
• may comprise various keywords
• include different spellings
• database-specific

Subject headings are a standardized list of words that are generated by the specialists in charge of some databases. This so-called index of subject headings (thesaurus) helps searchers find relevant articles, since the headwords indicate the content of a publication. By contrast, an ordinary keyword search does not necessarily result in a content-related fit, since the database also displays articles in which, for example, a word appears once in the abstract, even though the article’s content does not cover the topic.

Nevertheless, searches using both headwords and keywords should be conducted, since some articles may not yet have been assigned headwords, or errors may have occurred during the assignment of headwords. 

To add headwords to your search in the Business Source Complete database, please select the Thesaurus tab at the top. Here you can find headwords in a new search field and integrate them into your search query. In the search history, headwords are marked with the addition DE (descriptor).

The EconBiz database of the German National Library of Economics (ZBW – Leibniz Information Centre for Economics), which also contains German-language literature, has created its own index of subject headings with the STW Thesaurus for Economics . Headwords are integrated into the search by being used in the search query.

Since the indexes of subject headings divide terms into synonyms, generic terms and sub-aspects, they facilitate the creation of a document-term matrix. For this purpose it is advisable to specify in the document-term matrix the origin of the search terms (STW Thesaurus for Economics, Business Source Complete, etc.).

Searching in literature databases

Once the document-term matrix has been defined, the search in literature databases begins. It is recommended to enter each word of the document-term matrix individually into the database in order to obtain a good overview of the number of hits per word. Finally, all the words contained in a block of terms are linked with the Boolean operator OR and thereby a union of all the words is formed. The latter are then linked with each other using the Boolean operator AND. In doing so, each block should be added individually in order to see to what degree the number of hits decreases.

Since the search query must be set up separately for each database, tools such as  LitSonar  have been developed to enable a systematic search across different databases. LitSonar was created by  Professor Dr. Ali Sunyaev (Institute of Applied Informatics and Formal Description Methods – AIFB) at the Karlsruhe Institute of Technology.

Advanced search

Certain database-specific commands can be used to refine a search, for example, by taking variable word endings into account (*) or specifying the distance between two words, etc. Our overview shows the most important search commands for our top databases.

Additional searches in sources other than literature databases

In addition to literature databases, other sources should also be searched. Fink (2014, p. 27) lists the following reasons for this:

• the topic is new and not yet included in indexes of subject headings;
• search terms are not used congruently in articles because uniform definitions do not exist;
• some studies are still in the process of being published, or have been completed, but not published.

Therefore, further search strategies are manual search, bibliographic analysis, personal contacts and academic networks (Briner & Denyer, p. 349). Manual search means that you go through the source information of relevant articles and supplement your hit list accordingly. In addition, you should conduct a targeted search for so-called gray literature, that is, literature not distributed via the book trade, such as working papers from specialist areas and conference reports. By including different types of publications, the so-called publication bias (DBWM video “Understanding publication bias” ) – that is, distortions due to exclusive use of articles from peer-reviewed journals – should be kept to a minimum.

The PRESS-Checklist can support you to check the correctness of your search terms.

4. Merging hits from different databases

In principle, large amounts of data can be easily collected, structured and sorted with data processing programs such as Excel. Another option is to use reference management programs such as EndNote, Citavi or Zotero. The Saxon State and University Library Dresden (SLUB Dresden) provides an  overview of current reference management programs  . Software for qualitative data analysis such as NVivo is equally suited for data processing. A comprehensive overview of the features of different tools that support the SLR process can be found in Bandara et al. (2015).

Our online-self study course "Managing literature with Citavi" shows you how to use the reference management software Citavi.

When conducting an SLR, you should specify for each hit the database from which it originates and the date on which the query was made. In addition, you should always indicate how many hits you have identified in the various databases or, for example, by manual search.

Exporting data from literature databases

Exporting from literature databases is very easy. In  Business Source Complete  , you must first click on the “Share” button in the hit list, then “Email a link to download exported results” at the very bottom and then select the appropriate format for the respective literature program.

Exporting data from the literature database  EconBiz  is somewhat more complex. Here you must first create a marked list and then select each hit individually and add it to the marked list. Afterwards, articles on the list can be exported.

After merging all hits from the various databases, duplicate entries (duplicates) are deleted.

5. Applying inclusion and exclusion criteria

All publications are evaluated in the literature management program applying the previously defined criteria for inclusion and exclusion. Only those sources that survive this selection process will subsequently be analyzed. The review process and inclusion criteria should be tested with a small sample and adjustments made if necessary before applying it to all articles. In the ideal case, even this selection would be carried out by more than one person, with each working independently of one another. It needs to be made clear how discrepancies between reviewers are dealt with. 

The review of the criteria for inclusion and exclusion is primarily based on the title, abstract and subject headings in the databases, as well as on the keywords provided by the authors of a publication in the first step. In a second step the whole article / source will be read.

You can create tag words for the inclusion and exclusion in your literature management tool to keep an overview.

In addition to the common literature management tools, you can also use software tools that have been developed to support SLRs. The central library of the university in Zurich has published an overview and evaluation of different tools based on a survey among researchers. --> View SLR tools

The selection process needs to be made transparent. The PRISMA flow diagram supports the visualization of the number of included / excluded studies.

Forward and backward search

Should it become apparent that the number of sources found is relatively small, or if you wish to proceed with particular thoroughness, a forward-and-backward search based on the sources found is recommendable (Webster & Watson 2002, p. xvi). A backward search means going through the bibliographies of the sources found. A forward search, by contrast, identifies articles that have cited the relevant publications. The Web of Science and Scopus databases can be used to perform citation analyses.

6. Perform the review

As the next step, the remaining titles are analyzed as to their content by reading them several times in full. Information is extracted according to defined criteria and the quality of the publications is evaluated. If the data extraction is carried out by more than one person, a training ensures that there will be no differences between the reviewers.

Depending on the research questions there exist diffent methods for data abstraction (content analysis, concept matrix etc.). A so-called concept matrix can be used to structure the content of information (Webster & Watson 2002, p. xvii). The image to the right gives an example of a concept matrix according to Becker (2014).

Particularly in the field of economic sciences, the evaluation of a study’s quality cannot be performed according to a generally valid scheme, such as those existing in the field of medicine, for instance. Quality assessment therefore depends largely on the research questions.

Based on the findings of individual studies, a meta-level is then applied to try to understand what similarities and differences exist between the publications, what research gaps exist, etc. This may also result in the development of a theoretical model or reference framework.

Example concept matrix (Becker 2013) on the topic Business Process Management

7. Synthesizing results

Once the review has been conducted, the results must be compiled and, on the basis of these, conclusions derived with regard to the research question (Fink 2014, p. 199ff.). This includes, for example, the following aspects:

• historical development of topics (histogram, time series: when, and how frequently, did publications on the research topic appear?);
• overview of journals, authors or specialist disciplines dealing with the topic;
• comparison of applied statistical methods;
• topics covered by research;
• identifying research gaps;
• developing a reference framework;
• developing constructs;
• performing a meta-analysis: comparison of the correlations of the results of different empirical studies (see for example Fink 2014, p. 203 on conducting meta-analyses)

Publications about the method

Bandara, W., Furtmueller, E., Miskon, S., Gorbacheva, E., & Beekhuyzen, J. (2015). Achieving Rigor in Literature Reviews: Insights from Qualitative Data Analysis and Tool-Support.  Communications of the Association for Information Systems . 34(8), 154-204.

Booth, A., Papaioannou, D., and Sutton, A. (2012)  Systematic approaches to a successful literature review.  London: Sage.

Briner, R. B., & Denyer, D. (2012). Systematic Review and Evidence Synthesis as a Practice and Scholarship Tool. In Rousseau, D. M. (Hrsg.),  The Oxford Handbook of Evidenence Based Management . (S. 112-129). Oxford: Oxford University Press.

Durach, C. F., Wieland, A., & Machuca, Jose A. D. (2015). Antecedents and dimensions of supply chain robustness: a systematic literature review . International Journal of Physical Distribution & Logistic Management , 46 (1/2), 118-137. doi:  https://doi.org/10.1108/IJPDLM-05-2013-0133

Feak, C. B., & Swales, J. M. (2009). Telling a Research Story: Writing a Literature Review.  English in Today's Research World 2.  Ann Arbor: University of Michigan Press. doi:  10.3998/mpub.309338

Fink, A. (2014).  Conducting Research Literature Reviews: From the Internet to Paper  (4. Aufl.). Los Angeles, London, New Delhi, Singapore, Washington DC: Sage Publication.

Fisch, C., & Block, J. (2018). Six tips for your (systematic) literature review in business and management research.  Management Review Quarterly,  68, 103–106 (2018).  doi.org/10.1007/s11301-018-0142-x

Guba, B. (2008). Systematische Literaturrecherche.  Wiener Medizinische Wochenschrift , 158 (1-2), S. 62-69. doi:  doi.org/10.1007/s10354-007-0500-0  Hart, C.  Doing a literature review: releasing the social science research imagination.  London: Sage.

Jesson, J. K., Metheson, L. & Lacey, F. (2011).  Doing your Literature Review - traditional and Systematic Techniques . Los Angeles, London, New Delhi, Singapore, Washington DC: Sage Publication.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71. doi: 10.1136/bmj.n71.

Petticrew, M. and Roberts, H. (2006).  Systematic Reviews in the Social Sciences: A Practical Guide . Oxford:Blackwell. Ridley, D. (2012).  The literature review: A step-by-step guide . 2nd edn. London: Sage. 

Chang, W. and Taylor, S.A. (2016), The Effectiveness of Customer Participation in New Product Development: A Meta-Analysis,  Journal of Marketing , American Marketing Association, Los Angeles, CA, Vol. 80 No. 1, pp. 47–64.

Tranfield, D., Denyer, D. & Smart, P. (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review.  British Journal of Management , 14 (3), S. 207-222. doi:  https://doi.org/10.1111/1467-8551.00375

Webster, J., & Watson, R. T. (2002). Analyzing the Past to Prepare for the Future: Writing a Literature Review.  Management Information Systems Quarterly , 26(2), xiii-xxiii.  http://www.jstor.org/stable/4132319

Durach, C. F., Wieland, A. & Machuca, Jose. A. D. (2015). Antecedents and dimensions of supply chain robustness: a systematic literature review. International Journal of Physical Distribution & Logistics Management, 45(1/2), 118 – 137.

What is particularly good about this example is that search terms were defined by a number of experts and the review was conducted by three researchers working independently of one another. Furthermore, the search terms used have been very well extracted and the procedure of the literature selection very well described.

On the downside, the restriction to English-language literature brings the language bias into play, even though the authors consider it to be insignificant for the subject area.

Bos-Nehles, A., Renkema, M. & Janssen, M. (2017). HRM and innovative work behaviour: a systematic literature review. Personnel Review, 46(7), pp. 1228-1253

• Only very specific keywords used
• No precise information on how the review process was carried out (who reviewed articles?)
• Only journals with impact factor (publication bias)

Jia, F., Orzes, G., Sartor, M. & Nassimbeni, G. (2017). Global sourcing strategy and structure: towards a conceptual framework. International Journal of Operations & Production Management, 37(7), 840-864

• Research questions are explicitly presented
• Search string very detailed
• Exact description of the review process
• 2 persons conducted the review independently of each other

Franziska Klatt

[email protected]

+49 30 314-29778

Privacy notice: The TU Berlin offers a chat information service. If you enable it, your IP address and chat messages will be transmitted to external EU servers. more information

The chat is currently unavailable.

Please use our alternative contact options.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• BMC Med Res Methodol

Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies

Chris cooper.

1 Institute of Health Research, University of Exeter Medical School, Exeter, UK

Andrew Booth

2 HEDS, School of Health and Related Research (ScHARR), University of Sheffield, Sheffield, UK

Jo Varley-Campbell

Nicky britten.

3 Institute of Health Research, University of Exeter Medical School, Exeter, UK

Ruth Garside

4 European Centre for Environment and Human Health, University of Exeter Medical School, Truro, UK

Associated Data

Systematic literature searching is recognised as a critical component of the systematic review process. It involves a systematic search for studies and aims for a transparent report of study identification, leaving readers clear about what was done to identify studies, and how the findings of the review are situated in the relevant evidence.

Information specialists and review teams appear to work from a shared and tacit model of the literature search process. How this tacit model has developed and evolved is unclear, and it has not been explicitly examined before.

The purpose of this review is to determine if a shared model of the literature searching process can be detected across systematic review guidance documents and, if so, how this process is reported in the guidance and supported by published studies.

A literature review.

Two types of literature were reviewed: guidance and published studies. Nine guidance documents were identified, including: The Cochrane and Campbell Handbooks. Published studies were identified through ‘pearl growing’, citation chasing, a search of PubMed using the systematic review methods filter, and the authors’ topic knowledge.

The relevant sections within each guidance document were then read and re-read, with the aim of determining key methodological stages. Methodological stages were identified and defined. This data was reviewed to identify agreements and areas of unique guidance between guidance documents. Consensus across multiple guidance documents was used to inform selection of ‘key stages’ in the process of literature searching.

Eight key stages were determined relating specifically to literature searching in systematic reviews. They were: who should literature search, aims and purpose of literature searching, preparation, the search strategy, searching databases, supplementary searching, managing references and reporting the search process.

Conclusions

Eight key stages to the process of literature searching in systematic reviews were identified. These key stages are consistently reported in the nine guidance documents, suggesting consensus on the key stages of literature searching, and therefore the process of literature searching as a whole, in systematic reviews. Further research to determine the suitability of using the same process of literature searching for all types of systematic review is indicated.

Electronic supplementary material

The online version of this article (10.1186/s12874-018-0545-3) contains supplementary material, which is available to authorized users.

Systematic literature searching is recognised as a critical component of the systematic review process. It involves a systematic search for studies and aims for a transparent report of study identification, leaving review stakeholders clear about what was done to identify studies, and how the findings of the review are situated in the relevant evidence.

Information specialists and review teams appear to work from a shared and tacit model of the literature search process. How this tacit model has developed and evolved is unclear, and it has not been explicitly examined before. This is in contrast to the information science literature, which has developed information processing models as an explicit basis for dialogue and empirical testing. Without an explicit model, research in the process of systematic literature searching will remain immature and potentially uneven, and the development of shared information models will be assumed but never articulated.

One way of developing such a conceptual model is by formally examining the implicit “programme theory” as embodied in key methodological texts. The aim of this review is therefore to determine if a shared model of the literature searching process in systematic reviews can be detected across guidance documents and, if so, how this process is reported and supported.

Identifying guidance

Key texts (henceforth referred to as “guidance”) were identified based upon their accessibility to, and prominence within, United Kingdom systematic reviewing practice. The United Kingdom occupies a prominent position in the science of health information retrieval, as quantified by such objective measures as the authorship of papers, the number of Cochrane groups based in the UK, membership and leadership of groups such as the Cochrane Information Retrieval Methods Group, the HTA-I Information Specialists’ Group and historic association with such centres as the UK Cochrane Centre, the NHS Centre for Reviews and Dissemination, the Centre for Evidence Based Medicine and the National Institute for Clinical Excellence (NICE). Coupled with the linguistic dominance of English within medical and health science and the science of systematic reviews more generally, this offers a justification for a purposive sample that favours UK, European and Australian guidance documents.

Nine guidance documents were identified. These documents provide guidance for different types of reviews, namely: reviews of interventions, reviews of health technologies, reviews of qualitative research studies, reviews of social science topics, and reviews to inform guidance.

Whilst these guidance documents occasionally offer additional guidance on other types of systematic reviews, we have focused on the core and stated aims of these documents as they relate to literature searching. Table  1 sets out: the guidance document, the version audited, their core stated focus, and a bibliographical pointer to the main guidance relating to literature searching.

Guidance documents audited for this literature review

Once a list of key guidance documents was determined, it was checked by six senior information professionals based in the UK for relevance to current literature searching in systematic reviews.

Identifying supporting studies

In addition to identifying guidance, the authors sought to populate an evidence base of supporting studies (henceforth referred to as “studies”) that contribute to existing search practice. Studies were first identified by the authors from their knowledge on this topic area and, subsequently, through systematic citation chasing key studies (‘pearls’ [ 1 ]) located within each key stage of the search process. These studies are identified in Additional file  1 : Appendix Table 1. Citation chasing was conducted by analysing the bibliography of references for each study (backwards citation chasing) and through Google Scholar (forward citation chasing). A search of PubMed using the systematic review methods filter was undertaken in August 2017 (see Additional file 1 ). The search terms used were: (literature search*[Title/Abstract]) AND sysrev_methods[sb] and 586 results were returned. These results were sifted for relevance to the key stages in Fig.  1 by CC.

The key stages of literature search guidance as identified from nine key texts

Extracting the data

To reveal the implicit process of literature searching within each guidance document, the relevant sections (chapters) on literature searching were read and re-read, with the aim of determining key methodological stages. We defined a key methodological stage as a distinct step in the overall process for which specific guidance is reported, and action is taken, that collectively would result in a completed literature search.

The chapter or section sub-heading for each methodological stage was extracted into a table using the exact language as reported in each guidance document. The lead author (CC) then read and re-read these data, and the paragraphs of the document to which the headings referred, summarising section details. This table was then reviewed, using comparison and contrast to identify agreements and areas of unique guidance. Consensus across multiple guidelines was used to inform selection of ‘key stages’ in the process of literature searching.

Having determined the key stages to literature searching, we then read and re-read the sections relating to literature searching again, extracting specific detail relating to the methodological process of literature searching within each key stage. Again, the guidance was then read and re-read, first on a document-by-document-basis and, secondly, across all the documents above, to identify both commonalities and areas of unique guidance.

Results and discussion

Our findings.

We were able to identify consensus across the guidance on literature searching for systematic reviews suggesting a shared implicit model within the information retrieval community. Whilst the structure of the guidance varies between documents, the same key stages are reported, even where the core focus of each document is different. We were able to identify specific areas of unique guidance, where a document reported guidance not summarised in other documents, together with areas of consensus across guidance.

Unique guidance

Only one document provided guidance on the topic of when to stop searching [ 2 ]. This guidance from 2005 anticipates a topic of increasing importance with the current interest in time-limited (i.e. “rapid”) reviews. Quality assurance (or peer review) of literature searches was only covered in two guidance documents [ 3 , 4 ]. This topic has emerged as increasingly important as indicated by the development of the PRESS instrument [ 5 ]. Text mining was discussed in four guidance documents [ 4 , 6 – 8 ] where the automation of some manual review work may offer efficiencies in literature searching [ 8 ].

Agreement between guidance: Defining the key stages of literature searching

Where there was agreement on the process, we determined that this constituted a key stage in the process of literature searching to inform systematic reviews.

From the guidance, we determined eight key stages that relate specifically to literature searching in systematic reviews. These are summarised at Fig. ​ Fig.1. 1 . The data extraction table to inform Fig. ​ Fig.1 1 is reported in Table  2 . Table ​ Table2 2 reports the areas of common agreement and it demonstrates that the language used to describe key stages and processes varies significantly between guidance documents.

The order of literature search methods as presented in the guidance documents

For each key stage, we set out the specific guidance, followed by discussion on how this guidance is situated within the wider literature.

Key stage one: Deciding who should undertake the literature search

The guidance.

Eight documents provided guidance on who should undertake literature searching in systematic reviews [ 2 , 4 , 6 – 11 ]. The guidance affirms that people with relevant expertise of literature searching should ‘ideally’ be included within the review team [ 6 ]. Information specialists (or information scientists), librarians or trial search co-ordinators (TSCs) are indicated as appropriate researchers in six guidance documents [ 2 , 7 – 11 ].

How the guidance corresponds to the published studies

The guidance is consistent with studies that call for the involvement of information specialists and librarians in systematic reviews [ 12 – 26 ] and which demonstrate how their training as ‘expert searchers’ and ‘analysers and organisers of data’ can be put to good use [ 13 ] in a variety of roles [ 12 , 16 , 20 , 21 , 24 – 26 ]. These arguments make sense in the context of the aims and purposes of literature searching in systematic reviews, explored below. The need for ‘thorough’ and ‘replicable’ literature searches was fundamental to the guidance and recurs in key stage two. Studies have found poor reporting, and a lack of replicable literature searches, to be a weakness in systematic reviews [ 17 , 18 , 27 , 28 ] and they argue that involvement of information specialists/ librarians would be associated with better reporting and better quality literature searching. Indeed, Meert et al. [ 29 ] demonstrated that involving a librarian as a co-author to a systematic review correlated with a higher score in the literature searching component of a systematic review [ 29 ]. As ‘new styles’ of rapid and scoping reviews emerge, where decisions on how to search are more iterative and creative, a clear role is made here too [ 30 ].

Knowing where to search for studies was noted as important in the guidance, with no agreement as to the appropriate number of databases to be searched [ 2 , 6 ]. Database (and resource selection more broadly) is acknowledged as a relevant key skill of information specialists and librarians [ 12 , 15 , 16 , 31 ].

Whilst arguments for including information specialists and librarians in the process of systematic review might be considered self-evident, Koffel and Rethlefsen [ 31 ] have questioned if the necessary involvement is actually happening [ 31 ].

Key stage two: Determining the aim and purpose of a literature search

The aim: Five of the nine guidance documents use adjectives such as ‘thorough’, ‘comprehensive’, ‘transparent’ and ‘reproducible’ to define the aim of literature searching [ 6 – 10 ]. Analogous phrases were present in a further three guidance documents, namely: ‘to identify the best available evidence’ [ 4 ] or ‘the aim of the literature search is not to retrieve everything. It is to retrieve everything of relevance’ [ 2 ] or ‘A systematic literature search aims to identify all publications relevant to the particular research question’ [ 3 ]. The Joanna Briggs Institute reviewers’ manual was the only guidance document where a clear statement on the aim of literature searching could not be identified. The purpose of literature searching was defined in three guidance documents, namely to minimise bias in the resultant review [ 6 , 8 , 10 ]. Accordingly, eight of nine documents clearly asserted that thorough and comprehensive literature searches are required as a potential mechanism for minimising bias.

The need for thorough and comprehensive literature searches appears as uniform within the eight guidance documents that describe approaches to literature searching in systematic reviews of effectiveness. Reviews of effectiveness (of intervention or cost), accuracy and prognosis, require thorough and comprehensive literature searches to transparently produce a reliable estimate of intervention effect. The belief that all relevant studies have been ‘comprehensively’ identified, and that this process has been ‘transparently’ reported, increases confidence in the estimate of effect and the conclusions that can be drawn [ 32 ]. The supporting literature exploring the need for comprehensive literature searches focuses almost exclusively on reviews of intervention effectiveness and meta-analysis. Different ‘styles’ of review may have different standards however; the alternative, offered by purposive sampling, has been suggested in the specific context of qualitative evidence syntheses [ 33 ].

What is a comprehensive literature search?

Whilst the guidance calls for thorough and comprehensive literature searches, it lacks clarity on what constitutes a thorough and comprehensive literature search, beyond the implication that all of the literature search methods in Table ​ Table2 2 should be used to identify studies. Egger et al. [ 34 ], in an empirical study evaluating the importance of comprehensive literature searches for trials in systematic reviews, defined a comprehensive search for trials as:

• a search not restricted to English language;
• where Cochrane CENTRAL or at least two other electronic databases had been searched (such as MEDLINE or EMBASE); and
• at least one of the following search methods has been used to identify unpublished trials: searches for (I) conference abstracts, (ii) theses, (iii) trials registers; and (iv) contacts with experts in the field [ 34 ].

Tricco et al. (2008) used a similar threshold of bibliographic database searching AND a supplementary search method in a review when examining the risk of bias in systematic reviews. Their criteria were: one database (limited using the Cochrane Highly Sensitive Search Strategy (HSSS)) and handsearching [ 35 ].

Together with the guidance, this would suggest that comprehensive literature searching requires the use of BOTH bibliographic database searching AND supplementary search methods.

Comprehensiveness in literature searching, in the sense of how much searching should be undertaken, remains unclear. Egger et al. recommend that ‘investigators should consider the type of literature search and degree of comprehension that is appropriate for the review in question, taking into account budget and time constraints’ [ 34 ]. This view tallies with the Cochrane Handbook, which stipulates clearly, that study identification should be undertaken ‘within resource limits’ [ 9 ]. This would suggest that the limitations to comprehension are recognised but it raises questions on how this is decided and reported [ 36 ].

What is the point of comprehensive literature searching?

The purpose of thorough and comprehensive literature searches is to avoid missing key studies and to minimize bias [ 6 , 8 , 10 , 34 , 37 – 39 ] since a systematic review based only on published (or easily accessible) studies may have an exaggerated effect size [ 35 ]. Felson (1992) sets out potential biases that could affect the estimate of effect in a meta-analysis [ 40 ] and Tricco et al. summarize the evidence concerning bias and confounding in systematic reviews [ 35 ]. Egger et al. point to non-publication of studies, publication bias, language bias and MEDLINE bias, as key biases [ 34 , 35 , 40 – 46 ]. Comprehensive searches are not the sole factor to mitigate these biases but their contribution is thought to be significant [ 2 , 32 , 34 ]. Fehrmann (2011) suggests that ‘the search process being described in detail’ and that, where standard comprehensive search techniques have been applied, increases confidence in the search results [ 32 ].

Does comprehensive literature searching work?

Egger et al., and other study authors, have demonstrated a change in the estimate of intervention effectiveness where relevant studies were excluded from meta-analysis [ 34 , 47 ]. This would suggest that missing studies in literature searching alters the reliability of effectiveness estimates. This is an argument for comprehensive literature searching. Conversely, Egger et al. found that ‘comprehensive’ searches still missed studies and that comprehensive searches could, in fact, introduce bias into a review rather than preventing it, through the identification of low quality studies then being included in the meta-analysis [ 34 ]. Studies query if identifying and including low quality or grey literature studies changes the estimate of effect [ 43 , 48 ] and question if time is better invested updating systematic reviews rather than searching for unpublished studies [ 49 ], or mapping studies for review as opposed to aiming for high sensitivity in literature searching [ 50 ].

Aim and purpose beyond reviews of effectiveness

The need for comprehensive literature searches is less certain in reviews of qualitative studies, and for reviews where a comprehensive identification of studies is difficult to achieve (for example, in Public health) [ 33 , 51 – 55 ]. Literature searching for qualitative studies, and in public health topics, typically generates a greater number of studies to sift than in reviews of effectiveness [ 39 ] and demonstrating the ‘value’ of studies identified or missed is harder [ 56 ], since the study data do not typically support meta-analysis. Nussbaumer-Streit et al. (2016) have registered a review protocol to assess whether abbreviated literature searches (as opposed to comprehensive literature searches) has an impact on conclusions across multiple bodies of evidence, not only on effect estimates [ 57 ] which may develop this understanding. It may be that decision makers and users of systematic reviews are willing to trade the certainty from a comprehensive literature search and systematic review in exchange for different approaches to evidence synthesis [ 58 ], and that comprehensive literature searches are not necessarily a marker of literature search quality, as previously thought [ 36 ]. Different approaches to literature searching [ 37 , 38 , 59 – 62 ] and developing the concept of when to stop searching are important areas for further study [ 36 , 59 ].

The study by Nussbaumer-Streit et al. has been published since the submission of this literature review [ 63 ]. Nussbaumer-Streit et al. (2018) conclude that abbreviated literature searches are viable options for rapid evidence syntheses, if decision-makers are willing to trade the certainty from a comprehensive literature search and systematic review, but that decision-making which demands detailed scrutiny should still be based on comprehensive literature searches [ 63 ].

Key stage three: Preparing for the literature search

Six documents provided guidance on preparing for a literature search [ 2 , 3 , 6 , 7 , 9 , 10 ]. The Cochrane Handbook clearly stated that Cochrane authors (i.e. researchers) should seek advice from a trial search co-ordinator (i.e. a person with specific skills in literature searching) ‘before’ starting a literature search [ 9 ].

Two key tasks were perceptible in preparing for a literature searching [ 2 , 6 , 7 , 10 , 11 ]. First, to determine if there are any existing or on-going reviews, or if a new review is justified [ 6 , 11 ]; and, secondly, to develop an initial literature search strategy to estimate the volume of relevant literature (and quality of a small sample of relevant studies [ 10 ]) and indicate the resources required for literature searching and the review of the studies that follows [ 7 , 10 ].

Three documents summarised guidance on where to search to determine if a new review was justified [ 2 , 6 , 11 ]. These focused on searching databases of systematic reviews (The Cochrane Database of Systematic Reviews (CDSR) and the Database of Abstracts of Reviews of Effects (DARE)), institutional registries (including PROSPERO), and MEDLINE [ 6 , 11 ]. It is worth noting, however, that as of 2015, DARE (and NHS EEDs) are no longer being updated and so the relevance of this (these) resource(s) will diminish over-time [ 64 ]. One guidance document, ‘Systematic reviews in the Social Sciences’, noted, however, that databases are not the only source of information and unpublished reports, conference proceeding and grey literature may also be required, depending on the nature of the review question [ 2 ].

Two documents reported clearly that this preparation (or ‘scoping’) exercise should be undertaken before the actual search strategy is developed [ 7 , 10 ]).

The guidance offers the best available source on preparing the literature search with the published studies not typically reporting how their scoping informed the development of their search strategies nor how their search approaches were developed. Text mining has been proposed as a technique to develop search strategies in the scoping stages of a review although this work is still exploratory [ 65 ]. ‘Clustering documents’ and word frequency analysis have also been tested to identify search terms and studies for review [ 66 , 67 ]. Preparing for literature searches and scoping constitutes an area for future research.

Key stage four: Designing the search strategy

The Population, Intervention, Comparator, Outcome (PICO) structure was the commonly reported structure promoted to design a literature search strategy. Five documents suggested that the eligibility criteria or review question will determine which concepts of PICO will be populated to develop the search strategy [ 1 , 4 , 7 – 9 ]. The NICE handbook promoted multiple structures, namely PICO, SPICE (Setting, Perspective, Intervention, Comparison, Evaluation) and multi-stranded approaches [ 4 ].

With the exclusion of The Joanna Briggs Institute reviewers’ manual, the guidance offered detail on selecting key search terms, synonyms, Boolean language, selecting database indexing terms and combining search terms. The CEE handbook suggested that ‘search terms may be compiled with the help of the commissioning organisation and stakeholders’ [ 10 ].

The use of limits, such as language or date limits, were discussed in all documents [ 2 – 4 , 6 – 11 ].

Search strategy structure

The guidance typically relates to reviews of intervention effectiveness so PICO – with its focus on intervention and comparator - is the dominant model used to structure literature search strategies [ 68 ]. PICOs – where the S denotes study design - is also commonly used in effectiveness reviews [ 6 , 68 ]. As the NICE handbook notes, alternative models to structure literature search strategies have been developed and tested. Booth provides an overview on formulating questions for evidence based practice [ 69 ] and has developed a number of alternatives to the PICO structure, namely: BeHEMoTh (Behaviour of interest; Health context; Exclusions; Models or Theories) for use when systematically identifying theory [ 55 ]; SPICE (Setting, Perspective, Intervention, Comparison, Evaluation) for identification of social science and evaluation studies [ 69 ] and, working with Cooke and colleagues, SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type) [ 70 ]. SPIDER has been compared to PICO and PICOs in a study by Methley et al. [ 68 ].

The NICE handbook also suggests the use of multi-stranded approaches to developing literature search strategies [ 4 ]. Glanville developed this idea in a study by Whitting et al. [ 71 ] and a worked example of this approach is included in the development of a search filter by Cooper et al. [ 72 ].

Writing search strategies: Conceptual and objective approaches

Hausner et al. [ 73 ] provide guidance on writing literature search strategies, delineating between conceptually and objectively derived approaches. The conceptual approach, advocated by and explained in the guidance documents, relies on the expertise of the literature searcher to identify key search terms and then develop key terms to include synonyms and controlled syntax. Hausner and colleagues set out the objective approach [ 73 ] and describe what may be done to validate it [ 74 ].

The use of limits

The guidance documents offer direction on the use of limits within a literature search. Limits can be used to focus literature searching to specific study designs or by other markers (such as by date) which limits the number of studies returned by a literature search. The use of limits should be described and the implications explored [ 34 ] since limiting literature searching can introduce bias (explored above). Craven et al. have suggested the use of a supporting narrative to explain decisions made in the process of developing literature searches and this advice would usefully capture decisions on the use of search limits [ 75 ].

Key stage five: Determining the process of literature searching and deciding where to search (bibliographic database searching)

Table ​ Table2 2 summarises the process of literature searching as reported in each guidance document. Searching bibliographic databases was consistently reported as the ‘first step’ to literature searching in all nine guidance documents.

Three documents reported specific guidance on where to search, in each case specific to the type of review their guidance informed, and as a minimum requirement [ 4 , 9 , 11 ]. Seven of the key guidance documents suggest that the selection of bibliographic databases depends on the topic of review [ 2 – 4 , 6 – 8 , 10 ], with two documents noting the absence of an agreed standard on what constitutes an acceptable number of databases searched [ 2 , 6 ].

The guidance documents summarise ‘how to’ search bibliographic databases in detail and this guidance is further contextualised above in terms of developing the search strategy. The documents provide guidance of selecting bibliographic databases, in some cases stating acceptable minima (i.e. The Cochrane Handbook states Cochrane CENTRAL, MEDLINE and EMBASE), and in other cases simply listing bibliographic database available to search. Studies have explored the value in searching specific bibliographic databases, with Wright et al. (2015) noting the contribution of CINAHL in identifying qualitative studies [ 76 ], Beckles et al. (2013) questioning the contribution of CINAHL to identifying clinical studies for guideline development [ 77 ], and Cooper et al. (2015) exploring the role of UK-focused bibliographic databases to identify UK-relevant studies [ 78 ]. The host of the database (e.g. OVID or ProQuest) has been shown to alter the search returns offered. Younger and Boddy [ 79 ] report differing search returns from the same database (AMED) but where the ‘host’ was different [ 79 ].

The average number of bibliographic database searched in systematic reviews has risen in the period 1994–2014 (from 1 to 4) [ 80 ] but there remains (as attested to by the guidance) no consensus on what constitutes an acceptable number of databases searched [ 48 ]. This is perhaps because thinking about the number of databases searched is the wrong question, researchers should be focused on which databases were searched and why, and which databases were not searched and why. The discussion should re-orientate to the differential value of sources but researchers need to think about how to report this in studies to allow findings to be generalised. Bethel (2017) has proposed ‘search summaries’, completed by the literature searcher, to record where included studies were identified, whether from database (and which databases specifically) or supplementary search methods [ 81 ]. Search summaries document both yield and accuracy of searches, which could prospectively inform resource use and decisions to search or not to search specific databases in topic areas. The prospective use of such data presupposes, however, that past searches are a potential predictor of future search performance (i.e. that each topic is to be considered representative and not unique). In offering a body of practice, this data would be of greater practicable use than current studies which are considered as little more than individual case studies [ 82 – 90 ].

When to database search is another question posed in the literature. Beyer et al. [ 91 ] report that databases can be prioritised for literature searching which, whilst not addressing the question of which databases to search, may at least bring clarity as to which databases to search first [ 91 ]. Paradoxically, this links to studies that suggest PubMed should be searched in addition to MEDLINE (OVID interface) since this improves the currency of systematic reviews [ 92 , 93 ]. Cooper et al. (2017) have tested the idea of database searching not as a primary search method (as suggested in the guidance) but as a supplementary search method in order to manage the volume of studies identified for an environmental effectiveness systematic review. Their case study compared the effectiveness of database searching versus a protocol using supplementary search methods and found that the latter identified more relevant studies for review than searching bibliographic databases [ 94 ].

Key stage six: Determining the process of literature searching and deciding where to search (supplementary search methods)

Table ​ Table2 2 also summaries the process of literature searching which follows bibliographic database searching. As Table ​ Table2 2 sets out, guidance that supplementary literature search methods should be used in systematic reviews recurs across documents, but the order in which these methods are used, and the extent to which they are used, varies. We noted inconsistency in the labelling of supplementary search methods between guidance documents.

Rather than focus on the guidance on how to use the methods (which has been summarised in a recent review [ 95 ]), we focus on the aim or purpose of supplementary search methods.

The Cochrane Handbook reported that ‘efforts’ to identify unpublished studies should be made [ 9 ]. Four guidance documents [ 2 , 3 , 6 , 9 ] acknowledged that searching beyond bibliographic databases was necessary since ‘databases are not the only source of literature’ [ 2 ]. Only one document reported any guidance on determining when to use supplementary methods. The IQWiG handbook reported that the use of handsearching (in their example) could be determined on a ‘case-by-case basis’ which implies that the use of these methods is optional rather than mandatory. This is in contrast to the guidance (above) on bibliographic database searching.

The issue for supplementary search methods is similar in many ways to the issue of searching bibliographic databases: demonstrating value. The purpose and contribution of supplementary search methods in systematic reviews is increasingly acknowledged [ 37 , 61 , 62 , 96 – 101 ] but understanding the value of the search methods to identify studies and data is unclear. In a recently published review, Cooper et al. (2017) reviewed the literature on supplementary search methods looking to determine the advantages, disadvantages and resource implications of using supplementary search methods [ 95 ]. This review also summarises the key guidance and empirical studies and seeks to address the question on when to use these search methods and when not to [ 95 ]. The guidance is limited in this regard and, as Table ​ Table2 2 demonstrates, offers conflicting advice on the order of searching, and the extent to which these search methods should be used in systematic reviews.

Key stage seven: Managing the references

Five of the documents provided guidance on managing references, for example downloading, de-duplicating and managing the output of literature searches [ 2 , 4 , 6 , 8 , 10 ]. This guidance typically itemised available bibliographic management tools rather than offering guidance on how to use them specifically [ 2 , 4 , 6 , 8 ]. The CEE handbook provided guidance on importing data where no direct export option is available (e.g. web-searching) [ 10 ].

The literature on using bibliographic management tools is not large relative to the number of ‘how to’ videos on platforms such as YouTube (see for example [ 102 ]). These YouTube videos confirm the overall lack of ‘how to’ guidance identified in this study and offer useful instruction on managing references. Bramer et al. set out methods for de-duplicating data and reviewing references in Endnote [ 103 , 104 ] and Gall tests the direct search function within Endnote to access databases such as PubMed, finding a number of limitations [ 105 ]. Coar et al. and Ahmed et al. consider the role of the free-source tool, Zotero [ 106 , 107 ]. Managing references is a key administrative function in the process of review particularly for documenting searches in PRISMA guidance.

Key stage eight: Documenting the search

The Cochrane Handbook was the only guidance document to recommend a specific reporting guideline: Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [ 9 ]. Six documents provided guidance on reporting the process of literature searching with specific criteria to report [ 3 , 4 , 6 , 8 – 10 ]. There was consensus on reporting: the databases searched (and the host searched by), the search strategies used, and any use of limits (e.g. date, language, search filters (The CRD handbook called for these limits to be justified [ 6 ])). Three guidance documents reported that the number of studies identified should be recorded [ 3 , 6 , 10 ]. The number of duplicates identified [ 10 ], the screening decisions [ 3 ], a comprehensive list of grey literature sources searched (and full detail for other supplementary search methods) [ 8 ], and an annotation of search terms tested but not used [ 4 ] were identified as unique items in four documents.

The Cochrane Handbook was the only guidance document to note that the full search strategies for each database should be included in the Additional file 1 of the review [ 9 ].

All guidance documents should ultimately deliver completed systematic reviews that fulfil the requirements of the PRISMA reporting guidelines [ 108 ]. The guidance broadly requires the reporting of data that corresponds with the requirements of the PRISMA statement although documents typically ask for diverse and additional items [ 108 ]. In 2008, Sampson et al. observed a lack of consensus on reporting search methods in systematic reviews [ 109 ] and this remains the case as of 2017, as evidenced in the guidance documents, and in spite of the publication of the PRISMA guidelines in 2009 [ 110 ]. It is unclear why the collective guidance does not more explicitly endorse adherence to the PRISMA guidance.

Reporting of literature searching is a key area in systematic reviews since it sets out clearly what was done and how the conclusions of the review can be believed [ 52 , 109 ]. Despite strong endorsement in the guidance documents, specifically supported in PRISMA guidance, and other related reporting standards too (such as ENTREQ for qualitative evidence synthesis, STROBE for reviews of observational studies), authors still highlight the prevalence of poor standards of literature search reporting [ 31 , 110 – 119 ]. To explore issues experienced by authors in reporting literature searches, and look at uptake of PRISMA, Radar et al. [ 120 ] surveyed over 260 review authors to determine common problems and their work summaries the practical aspects of reporting literature searching [ 120 ]. Atkinson et al. [ 121 ] have also analysed reporting standards for literature searching, summarising recommendations and gaps for reporting search strategies [ 121 ].

One area that is less well covered by the guidance, but nevertheless appears in this literature, is the quality appraisal or peer review of literature search strategies. The PRESS checklist is the most prominent and it aims to develop evidence-based guidelines to peer review of electronic search strategies [ 5 , 122 , 123 ]. A corresponding guideline for documentation of supplementary search methods does not yet exist although this idea is currently being explored.

How the reporting of the literature searching process corresponds to critical appraisal tools is an area for further research. In the survey undertaken by Radar et al. (2014), 86% of survey respondents (153/178) identified a need for further guidance on what aspects of the literature search process to report [ 120 ]. The PRISMA statement offers a brief summary of what to report but little practical guidance on how to report it [ 108 ]. Critical appraisal tools for systematic reviews, such as AMSTAR 2 (Shea et al. [ 124 ]) and ROBIS (Whiting et al. [ 125 ]), can usefully be read alongside PRISMA guidance, since they offer greater detail on how the reporting of the literature search will be appraised and, therefore, they offer a proxy on what to report [ 124 , 125 ]. Further research in the form of a study which undertakes a comparison between PRISMA and quality appraisal checklists for systematic reviews would seem to begin addressing the call, identified by Radar et al., for further guidance on what to report [ 120 ].

Limitations

Other handbooks exist.

A potential limitation of this literature review is the focus on guidance produced in Europe (the UK specifically) and Australia. We justify the decision for our selection of the nine guidance documents reviewed in this literature review in section “ Identifying guidance ”. In brief, these nine guidance documents were selected as the most relevant health care guidance that inform UK systematic reviewing practice, given that the UK occupies a prominent position in the science of health information retrieval. We acknowledge the existence of other guidance documents, such as those from North America (e.g. the Agency for Healthcare Research and Quality (AHRQ) [ 126 ], The Institute of Medicine [ 127 ] and the guidance and resources produced by the Canadian Agency for Drugs and Technologies in Health (CADTH) [ 128 ]). We comment further on this directly below.

The handbooks are potentially linked to one another

What is not clear is the extent to which the guidance documents inter-relate or provide guidance uniquely. The Cochrane Handbook, first published in 1994, is notably a key source of reference in guidance and systematic reviews beyond Cochrane reviews. It is not clear to what extent broadening the sample of guidance handbooks to include North American handbooks, and guidance handbooks from other relevant countries too, would alter the findings of this literature review or develop further support for the process model. Since we cannot be clear, we raise this as a potential limitation of this literature review. On our initial review of a sample of North American, and other, guidance documents (before selecting the guidance documents considered in this review), however, we do not consider that the inclusion of these further handbooks would alter significantly the findings of this literature review.

This is a literature review

A further limitation of this review was that the review of published studies is not a systematic review of the evidence for each key stage. It is possible that other relevant studies could help contribute to the exploration and development of the key stages identified in this review.

This literature review would appear to demonstrate the existence of a shared model of the literature searching process in systematic reviews. We call this model ‘the conventional approach’, since it appears to be common convention in nine different guidance documents.

The findings reported above reveal eight key stages in the process of literature searching for systematic reviews. These key stages are consistently reported in the nine guidance documents which suggests consensus on the key stages of literature searching, and therefore the process of literature searching as a whole, in systematic reviews.

In Table ​ Table2, 2 , we demonstrate consensus regarding the application of literature search methods. All guidance documents distinguish between primary and supplementary search methods. Bibliographic database searching is consistently the first method of literature searching referenced in each guidance document. Whilst the guidance uniformly supports the use of supplementary search methods, there is little evidence for a consistent process with diverse guidance across documents. This may reflect differences in the core focus across each document, linked to differences in identifying effectiveness studies or qualitative studies, for instance.

Eight of the nine guidance documents reported on the aims of literature searching. The shared understanding was that literature searching should be thorough and comprehensive in its aim and that this process should be reported transparently so that that it could be reproduced. Whilst only three documents explicitly link this understanding to minimising bias, it is clear that comprehensive literature searching is implicitly linked to ‘not missing relevant studies’ which is approximately the same point.

Defining the key stages in this review helps categorise the scholarship available, and it prioritises areas for development or further study. The supporting studies on preparing for literature searching (key stage three, ‘preparation’) were, for example, comparatively few, and yet this key stage represents a decisive moment in literature searching for systematic reviews. It is where search strategy structure is determined, search terms are chosen or discarded, and the resources to be searched are selected. Information specialists, librarians and researchers, are well placed to develop these and other areas within the key stages we identify.

This review calls for further research to determine the suitability of using the conventional approach. The publication dates of the guidance documents which underpin the conventional approach may raise questions as to whether the process which they each report remains valid for current systematic literature searching. In addition, it may be useful to test whether it is desirable to use the same process model of literature searching for qualitative evidence synthesis as that for reviews of intervention effectiveness, which this literature review demonstrates is presently recommended best practice.

Additional file

Appendix tables and PubMed search strategy. Key studies used for pearl growing per key stage, working data extraction tables and the PubMed search strategy. (DOCX 30 kb)

Acknowledgements

CC acknowledges the supervision offered by Professor Chris Hyde.

This publication forms a part of CC’s PhD. CC’s PhD was funded through the National Institute for Health Research (NIHR) Health Technology Assessment (HTA) Programme (Project Number 16/54/11). The open access fee for this publication was paid for by Exeter Medical School.

RG and NB were partially supported by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care South West Peninsula.

The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.

Abbreviations

Authors’ contributions

CC conceived the idea for this study and wrote the first draft of the manuscript. CC discussed this publication in PhD supervision with AB and separately with JVC. CC revised the publication with input and comments from AB, JVC, RG and NB. All authors revised the manuscript prior to submission. All authors read and approved the final manuscript.

Ethics approval and consent to participate

Consent for publication, competing interests.

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Contributor Information

Chris Cooper, Email: [email protected] .

Andrew Booth, Email: [email protected] .

Jo Varley-Campbell, Email: [email protected] .

Nicky Britten, Email: [email protected] .

Ruth Garside, Email: [email protected] .

• Open access
• Published: 14 August 2018

Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies

• Chris Cooper   ORCID: orcid.org/0000-0003-0864-5607 1 ,
• Andrew Booth 2 ,
• Jo Varley-Campbell 1 ,
• Nicky Britten 3 &
• Ruth Garside 4  

BMC Medical Research Methodology volume  18 , Article number:  85 ( 2018 ) Cite this article

206k Accesses

209 Citations

116 Altmetric

Metrics details

Systematic literature searching is recognised as a critical component of the systematic review process. It involves a systematic search for studies and aims for a transparent report of study identification, leaving readers clear about what was done to identify studies, and how the findings of the review are situated in the relevant evidence.

Information specialists and review teams appear to work from a shared and tacit model of the literature search process. How this tacit model has developed and evolved is unclear, and it has not been explicitly examined before.

The purpose of this review is to determine if a shared model of the literature searching process can be detected across systematic review guidance documents and, if so, how this process is reported in the guidance and supported by published studies.

A literature review.

Two types of literature were reviewed: guidance and published studies. Nine guidance documents were identified, including: The Cochrane and Campbell Handbooks. Published studies were identified through ‘pearl growing’, citation chasing, a search of PubMed using the systematic review methods filter, and the authors’ topic knowledge.

The relevant sections within each guidance document were then read and re-read, with the aim of determining key methodological stages. Methodological stages were identified and defined. This data was reviewed to identify agreements and areas of unique guidance between guidance documents. Consensus across multiple guidance documents was used to inform selection of ‘key stages’ in the process of literature searching.

Eight key stages were determined relating specifically to literature searching in systematic reviews. They were: who should literature search, aims and purpose of literature searching, preparation, the search strategy, searching databases, supplementary searching, managing references and reporting the search process.

Conclusions

Eight key stages to the process of literature searching in systematic reviews were identified. These key stages are consistently reported in the nine guidance documents, suggesting consensus on the key stages of literature searching, and therefore the process of literature searching as a whole, in systematic reviews. Further research to determine the suitability of using the same process of literature searching for all types of systematic review is indicated.

Peer Review reports

Systematic literature searching is recognised as a critical component of the systematic review process. It involves a systematic search for studies and aims for a transparent report of study identification, leaving review stakeholders clear about what was done to identify studies, and how the findings of the review are situated in the relevant evidence.

Information specialists and review teams appear to work from a shared and tacit model of the literature search process. How this tacit model has developed and evolved is unclear, and it has not been explicitly examined before. This is in contrast to the information science literature, which has developed information processing models as an explicit basis for dialogue and empirical testing. Without an explicit model, research in the process of systematic literature searching will remain immature and potentially uneven, and the development of shared information models will be assumed but never articulated.

One way of developing such a conceptual model is by formally examining the implicit “programme theory” as embodied in key methodological texts. The aim of this review is therefore to determine if a shared model of the literature searching process in systematic reviews can be detected across guidance documents and, if so, how this process is reported and supported.

Identifying guidance

Key texts (henceforth referred to as “guidance”) were identified based upon their accessibility to, and prominence within, United Kingdom systematic reviewing practice. The United Kingdom occupies a prominent position in the science of health information retrieval, as quantified by such objective measures as the authorship of papers, the number of Cochrane groups based in the UK, membership and leadership of groups such as the Cochrane Information Retrieval Methods Group, the HTA-I Information Specialists’ Group and historic association with such centres as the UK Cochrane Centre, the NHS Centre for Reviews and Dissemination, the Centre for Evidence Based Medicine and the National Institute for Clinical Excellence (NICE). Coupled with the linguistic dominance of English within medical and health science and the science of systematic reviews more generally, this offers a justification for a purposive sample that favours UK, European and Australian guidance documents.

Nine guidance documents were identified. These documents provide guidance for different types of reviews, namely: reviews of interventions, reviews of health technologies, reviews of qualitative research studies, reviews of social science topics, and reviews to inform guidance.

Whilst these guidance documents occasionally offer additional guidance on other types of systematic reviews, we have focused on the core and stated aims of these documents as they relate to literature searching. Table  1 sets out: the guidance document, the version audited, their core stated focus, and a bibliographical pointer to the main guidance relating to literature searching.

Once a list of key guidance documents was determined, it was checked by six senior information professionals based in the UK for relevance to current literature searching in systematic reviews.

Identifying supporting studies

In addition to identifying guidance, the authors sought to populate an evidence base of supporting studies (henceforth referred to as “studies”) that contribute to existing search practice. Studies were first identified by the authors from their knowledge on this topic area and, subsequently, through systematic citation chasing key studies (‘pearls’ [ 1 ]) located within each key stage of the search process. These studies are identified in Additional file  1 : Appendix Table 1. Citation chasing was conducted by analysing the bibliography of references for each study (backwards citation chasing) and through Google Scholar (forward citation chasing). A search of PubMed using the systematic review methods filter was undertaken in August 2017 (see Additional file 1 ). The search terms used were: (literature search*[Title/Abstract]) AND sysrev_methods[sb] and 586 results were returned. These results were sifted for relevance to the key stages in Fig.  1 by CC.

The key stages of literature search guidance as identified from nine key texts

Extracting the data

To reveal the implicit process of literature searching within each guidance document, the relevant sections (chapters) on literature searching were read and re-read, with the aim of determining key methodological stages. We defined a key methodological stage as a distinct step in the overall process for which specific guidance is reported, and action is taken, that collectively would result in a completed literature search.

The chapter or section sub-heading for each methodological stage was extracted into a table using the exact language as reported in each guidance document. The lead author (CC) then read and re-read these data, and the paragraphs of the document to which the headings referred, summarising section details. This table was then reviewed, using comparison and contrast to identify agreements and areas of unique guidance. Consensus across multiple guidelines was used to inform selection of ‘key stages’ in the process of literature searching.

Having determined the key stages to literature searching, we then read and re-read the sections relating to literature searching again, extracting specific detail relating to the methodological process of literature searching within each key stage. Again, the guidance was then read and re-read, first on a document-by-document-basis and, secondly, across all the documents above, to identify both commonalities and areas of unique guidance.

Results and discussion

Our findings.

We were able to identify consensus across the guidance on literature searching for systematic reviews suggesting a shared implicit model within the information retrieval community. Whilst the structure of the guidance varies between documents, the same key stages are reported, even where the core focus of each document is different. We were able to identify specific areas of unique guidance, where a document reported guidance not summarised in other documents, together with areas of consensus across guidance.

Unique guidance

Only one document provided guidance on the topic of when to stop searching [ 2 ]. This guidance from 2005 anticipates a topic of increasing importance with the current interest in time-limited (i.e. “rapid”) reviews. Quality assurance (or peer review) of literature searches was only covered in two guidance documents [ 3 , 4 ]. This topic has emerged as increasingly important as indicated by the development of the PRESS instrument [ 5 ]. Text mining was discussed in four guidance documents [ 4 , 6 , 7 , 8 ] where the automation of some manual review work may offer efficiencies in literature searching [ 8 ].

Agreement between guidance: Defining the key stages of literature searching

Where there was agreement on the process, we determined that this constituted a key stage in the process of literature searching to inform systematic reviews.

From the guidance, we determined eight key stages that relate specifically to literature searching in systematic reviews. These are summarised at Fig. 1 . The data extraction table to inform Fig. 1 is reported in Table  2 . Table 2 reports the areas of common agreement and it demonstrates that the language used to describe key stages and processes varies significantly between guidance documents.

For each key stage, we set out the specific guidance, followed by discussion on how this guidance is situated within the wider literature.

Key stage one: Deciding who should undertake the literature search

The guidance.

Eight documents provided guidance on who should undertake literature searching in systematic reviews [ 2 , 4 , 6 , 7 , 8 , 9 , 10 , 11 ]. The guidance affirms that people with relevant expertise of literature searching should ‘ideally’ be included within the review team [ 6 ]. Information specialists (or information scientists), librarians or trial search co-ordinators (TSCs) are indicated as appropriate researchers in six guidance documents [ 2 , 7 , 8 , 9 , 10 , 11 ].

How the guidance corresponds to the published studies

The guidance is consistent with studies that call for the involvement of information specialists and librarians in systematic reviews [ 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 ] and which demonstrate how their training as ‘expert searchers’ and ‘analysers and organisers of data’ can be put to good use [ 13 ] in a variety of roles [ 12 , 16 , 20 , 21 , 24 , 25 , 26 ]. These arguments make sense in the context of the aims and purposes of literature searching in systematic reviews, explored below. The need for ‘thorough’ and ‘replicable’ literature searches was fundamental to the guidance and recurs in key stage two. Studies have found poor reporting, and a lack of replicable literature searches, to be a weakness in systematic reviews [ 17 , 18 , 27 , 28 ] and they argue that involvement of information specialists/ librarians would be associated with better reporting and better quality literature searching. Indeed, Meert et al. [ 29 ] demonstrated that involving a librarian as a co-author to a systematic review correlated with a higher score in the literature searching component of a systematic review [ 29 ]. As ‘new styles’ of rapid and scoping reviews emerge, where decisions on how to search are more iterative and creative, a clear role is made here too [ 30 ].

Knowing where to search for studies was noted as important in the guidance, with no agreement as to the appropriate number of databases to be searched [ 2 , 6 ]. Database (and resource selection more broadly) is acknowledged as a relevant key skill of information specialists and librarians [ 12 , 15 , 16 , 31 ].

Whilst arguments for including information specialists and librarians in the process of systematic review might be considered self-evident, Koffel and Rethlefsen [ 31 ] have questioned if the necessary involvement is actually happening [ 31 ].

Key stage two: Determining the aim and purpose of a literature search

The aim: Five of the nine guidance documents use adjectives such as ‘thorough’, ‘comprehensive’, ‘transparent’ and ‘reproducible’ to define the aim of literature searching [ 6 , 7 , 8 , 9 , 10 ]. Analogous phrases were present in a further three guidance documents, namely: ‘to identify the best available evidence’ [ 4 ] or ‘the aim of the literature search is not to retrieve everything. It is to retrieve everything of relevance’ [ 2 ] or ‘A systematic literature search aims to identify all publications relevant to the particular research question’ [ 3 ]. The Joanna Briggs Institute reviewers’ manual was the only guidance document where a clear statement on the aim of literature searching could not be identified. The purpose of literature searching was defined in three guidance documents, namely to minimise bias in the resultant review [ 6 , 8 , 10 ]. Accordingly, eight of nine documents clearly asserted that thorough and comprehensive literature searches are required as a potential mechanism for minimising bias.

The need for thorough and comprehensive literature searches appears as uniform within the eight guidance documents that describe approaches to literature searching in systematic reviews of effectiveness. Reviews of effectiveness (of intervention or cost), accuracy and prognosis, require thorough and comprehensive literature searches to transparently produce a reliable estimate of intervention effect. The belief that all relevant studies have been ‘comprehensively’ identified, and that this process has been ‘transparently’ reported, increases confidence in the estimate of effect and the conclusions that can be drawn [ 32 ]. The supporting literature exploring the need for comprehensive literature searches focuses almost exclusively on reviews of intervention effectiveness and meta-analysis. Different ‘styles’ of review may have different standards however; the alternative, offered by purposive sampling, has been suggested in the specific context of qualitative evidence syntheses [ 33 ].

What is a comprehensive literature search?

Whilst the guidance calls for thorough and comprehensive literature searches, it lacks clarity on what constitutes a thorough and comprehensive literature search, beyond the implication that all of the literature search methods in Table 2 should be used to identify studies. Egger et al. [ 34 ], in an empirical study evaluating the importance of comprehensive literature searches for trials in systematic reviews, defined a comprehensive search for trials as:

a search not restricted to English language;

where Cochrane CENTRAL or at least two other electronic databases had been searched (such as MEDLINE or EMBASE); and

at least one of the following search methods has been used to identify unpublished trials: searches for (I) conference abstracts, (ii) theses, (iii) trials registers; and (iv) contacts with experts in the field [ 34 ].

Tricco et al. (2008) used a similar threshold of bibliographic database searching AND a supplementary search method in a review when examining the risk of bias in systematic reviews. Their criteria were: one database (limited using the Cochrane Highly Sensitive Search Strategy (HSSS)) and handsearching [ 35 ].

Together with the guidance, this would suggest that comprehensive literature searching requires the use of BOTH bibliographic database searching AND supplementary search methods.

Comprehensiveness in literature searching, in the sense of how much searching should be undertaken, remains unclear. Egger et al. recommend that ‘investigators should consider the type of literature search and degree of comprehension that is appropriate for the review in question, taking into account budget and time constraints’ [ 34 ]. This view tallies with the Cochrane Handbook, which stipulates clearly, that study identification should be undertaken ‘within resource limits’ [ 9 ]. This would suggest that the limitations to comprehension are recognised but it raises questions on how this is decided and reported [ 36 ].

What is the point of comprehensive literature searching?

The purpose of thorough and comprehensive literature searches is to avoid missing key studies and to minimize bias [ 6 , 8 , 10 , 34 , 37 , 38 , 39 ] since a systematic review based only on published (or easily accessible) studies may have an exaggerated effect size [ 35 ]. Felson (1992) sets out potential biases that could affect the estimate of effect in a meta-analysis [ 40 ] and Tricco et al. summarize the evidence concerning bias and confounding in systematic reviews [ 35 ]. Egger et al. point to non-publication of studies, publication bias, language bias and MEDLINE bias, as key biases [ 34 , 35 , 40 , 41 , 42 , 43 , 44 , 45 , 46 ]. Comprehensive searches are not the sole factor to mitigate these biases but their contribution is thought to be significant [ 2 , 32 , 34 ]. Fehrmann (2011) suggests that ‘the search process being described in detail’ and that, where standard comprehensive search techniques have been applied, increases confidence in the search results [ 32 ].

Does comprehensive literature searching work?

Egger et al., and other study authors, have demonstrated a change in the estimate of intervention effectiveness where relevant studies were excluded from meta-analysis [ 34 , 47 ]. This would suggest that missing studies in literature searching alters the reliability of effectiveness estimates. This is an argument for comprehensive literature searching. Conversely, Egger et al. found that ‘comprehensive’ searches still missed studies and that comprehensive searches could, in fact, introduce bias into a review rather than preventing it, through the identification of low quality studies then being included in the meta-analysis [ 34 ]. Studies query if identifying and including low quality or grey literature studies changes the estimate of effect [ 43 , 48 ] and question if time is better invested updating systematic reviews rather than searching for unpublished studies [ 49 ], or mapping studies for review as opposed to aiming for high sensitivity in literature searching [ 50 ].

Aim and purpose beyond reviews of effectiveness

The need for comprehensive literature searches is less certain in reviews of qualitative studies, and for reviews where a comprehensive identification of studies is difficult to achieve (for example, in Public health) [ 33 , 51 , 52 , 53 , 54 , 55 ]. Literature searching for qualitative studies, and in public health topics, typically generates a greater number of studies to sift than in reviews of effectiveness [ 39 ] and demonstrating the ‘value’ of studies identified or missed is harder [ 56 ], since the study data do not typically support meta-analysis. Nussbaumer-Streit et al. (2016) have registered a review protocol to assess whether abbreviated literature searches (as opposed to comprehensive literature searches) has an impact on conclusions across multiple bodies of evidence, not only on effect estimates [ 57 ] which may develop this understanding. It may be that decision makers and users of systematic reviews are willing to trade the certainty from a comprehensive literature search and systematic review in exchange for different approaches to evidence synthesis [ 58 ], and that comprehensive literature searches are not necessarily a marker of literature search quality, as previously thought [ 36 ]. Different approaches to literature searching [ 37 , 38 , 59 , 60 , 61 , 62 ] and developing the concept of when to stop searching are important areas for further study [ 36 , 59 ].

The study by Nussbaumer-Streit et al. has been published since the submission of this literature review [ 63 ]. Nussbaumer-Streit et al. (2018) conclude that abbreviated literature searches are viable options for rapid evidence syntheses, if decision-makers are willing to trade the certainty from a comprehensive literature search and systematic review, but that decision-making which demands detailed scrutiny should still be based on comprehensive literature searches [ 63 ].

Key stage three: Preparing for the literature search

Six documents provided guidance on preparing for a literature search [ 2 , 3 , 6 , 7 , 9 , 10 ]. The Cochrane Handbook clearly stated that Cochrane authors (i.e. researchers) should seek advice from a trial search co-ordinator (i.e. a person with specific skills in literature searching) ‘before’ starting a literature search [ 9 ].

Two key tasks were perceptible in preparing for a literature searching [ 2 , 6 , 7 , 10 , 11 ]. First, to determine if there are any existing or on-going reviews, or if a new review is justified [ 6 , 11 ]; and, secondly, to develop an initial literature search strategy to estimate the volume of relevant literature (and quality of a small sample of relevant studies [ 10 ]) and indicate the resources required for literature searching and the review of the studies that follows [ 7 , 10 ].

Three documents summarised guidance on where to search to determine if a new review was justified [ 2 , 6 , 11 ]. These focused on searching databases of systematic reviews (The Cochrane Database of Systematic Reviews (CDSR) and the Database of Abstracts of Reviews of Effects (DARE)), institutional registries (including PROSPERO), and MEDLINE [ 6 , 11 ]. It is worth noting, however, that as of 2015, DARE (and NHS EEDs) are no longer being updated and so the relevance of this (these) resource(s) will diminish over-time [ 64 ]. One guidance document, ‘Systematic reviews in the Social Sciences’, noted, however, that databases are not the only source of information and unpublished reports, conference proceeding and grey literature may also be required, depending on the nature of the review question [ 2 ].

Two documents reported clearly that this preparation (or ‘scoping’) exercise should be undertaken before the actual search strategy is developed [ 7 , 10 ]).

The guidance offers the best available source on preparing the literature search with the published studies not typically reporting how their scoping informed the development of their search strategies nor how their search approaches were developed. Text mining has been proposed as a technique to develop search strategies in the scoping stages of a review although this work is still exploratory [ 65 ]. ‘Clustering documents’ and word frequency analysis have also been tested to identify search terms and studies for review [ 66 , 67 ]. Preparing for literature searches and scoping constitutes an area for future research.

Key stage four: Designing the search strategy

The Population, Intervention, Comparator, Outcome (PICO) structure was the commonly reported structure promoted to design a literature search strategy. Five documents suggested that the eligibility criteria or review question will determine which concepts of PICO will be populated to develop the search strategy [ 1 , 4 , 7 , 8 , 9 ]. The NICE handbook promoted multiple structures, namely PICO, SPICE (Setting, Perspective, Intervention, Comparison, Evaluation) and multi-stranded approaches [ 4 ].

With the exclusion of The Joanna Briggs Institute reviewers’ manual, the guidance offered detail on selecting key search terms, synonyms, Boolean language, selecting database indexing terms and combining search terms. The CEE handbook suggested that ‘search terms may be compiled with the help of the commissioning organisation and stakeholders’ [ 10 ].

The use of limits, such as language or date limits, were discussed in all documents [ 2 , 3 , 4 , 6 , 7 , 8 , 9 , 10 , 11 ].

Search strategy structure

The guidance typically relates to reviews of intervention effectiveness so PICO – with its focus on intervention and comparator - is the dominant model used to structure literature search strategies [ 68 ]. PICOs – where the S denotes study design - is also commonly used in effectiveness reviews [ 6 , 68 ]. As the NICE handbook notes, alternative models to structure literature search strategies have been developed and tested. Booth provides an overview on formulating questions for evidence based practice [ 69 ] and has developed a number of alternatives to the PICO structure, namely: BeHEMoTh (Behaviour of interest; Health context; Exclusions; Models or Theories) for use when systematically identifying theory [ 55 ]; SPICE (Setting, Perspective, Intervention, Comparison, Evaluation) for identification of social science and evaluation studies [ 69 ] and, working with Cooke and colleagues, SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type) [ 70 ]. SPIDER has been compared to PICO and PICOs in a study by Methley et al. [ 68 ].

The NICE handbook also suggests the use of multi-stranded approaches to developing literature search strategies [ 4 ]. Glanville developed this idea in a study by Whitting et al. [ 71 ] and a worked example of this approach is included in the development of a search filter by Cooper et al. [ 72 ].

Writing search strategies: Conceptual and objective approaches

Hausner et al. [ 73 ] provide guidance on writing literature search strategies, delineating between conceptually and objectively derived approaches. The conceptual approach, advocated by and explained in the guidance documents, relies on the expertise of the literature searcher to identify key search terms and then develop key terms to include synonyms and controlled syntax. Hausner and colleagues set out the objective approach [ 73 ] and describe what may be done to validate it [ 74 ].

The use of limits

The guidance documents offer direction on the use of limits within a literature search. Limits can be used to focus literature searching to specific study designs or by other markers (such as by date) which limits the number of studies returned by a literature search. The use of limits should be described and the implications explored [ 34 ] since limiting literature searching can introduce bias (explored above). Craven et al. have suggested the use of a supporting narrative to explain decisions made in the process of developing literature searches and this advice would usefully capture decisions on the use of search limits [ 75 ].

Key stage five: Determining the process of literature searching and deciding where to search (bibliographic database searching)

Table 2 summarises the process of literature searching as reported in each guidance document. Searching bibliographic databases was consistently reported as the ‘first step’ to literature searching in all nine guidance documents.

Three documents reported specific guidance on where to search, in each case specific to the type of review their guidance informed, and as a minimum requirement [ 4 , 9 , 11 ]. Seven of the key guidance documents suggest that the selection of bibliographic databases depends on the topic of review [ 2 , 3 , 4 , 6 , 7 , 8 , 10 ], with two documents noting the absence of an agreed standard on what constitutes an acceptable number of databases searched [ 2 , 6 ].

The guidance documents summarise ‘how to’ search bibliographic databases in detail and this guidance is further contextualised above in terms of developing the search strategy. The documents provide guidance of selecting bibliographic databases, in some cases stating acceptable minima (i.e. The Cochrane Handbook states Cochrane CENTRAL, MEDLINE and EMBASE), and in other cases simply listing bibliographic database available to search. Studies have explored the value in searching specific bibliographic databases, with Wright et al. (2015) noting the contribution of CINAHL in identifying qualitative studies [ 76 ], Beckles et al. (2013) questioning the contribution of CINAHL to identifying clinical studies for guideline development [ 77 ], and Cooper et al. (2015) exploring the role of UK-focused bibliographic databases to identify UK-relevant studies [ 78 ]. The host of the database (e.g. OVID or ProQuest) has been shown to alter the search returns offered. Younger and Boddy [ 79 ] report differing search returns from the same database (AMED) but where the ‘host’ was different [ 79 ].

The average number of bibliographic database searched in systematic reviews has risen in the period 1994–2014 (from 1 to 4) [ 80 ] but there remains (as attested to by the guidance) no consensus on what constitutes an acceptable number of databases searched [ 48 ]. This is perhaps because thinking about the number of databases searched is the wrong question, researchers should be focused on which databases were searched and why, and which databases were not searched and why. The discussion should re-orientate to the differential value of sources but researchers need to think about how to report this in studies to allow findings to be generalised. Bethel (2017) has proposed ‘search summaries’, completed by the literature searcher, to record where included studies were identified, whether from database (and which databases specifically) or supplementary search methods [ 81 ]. Search summaries document both yield and accuracy of searches, which could prospectively inform resource use and decisions to search or not to search specific databases in topic areas. The prospective use of such data presupposes, however, that past searches are a potential predictor of future search performance (i.e. that each topic is to be considered representative and not unique). In offering a body of practice, this data would be of greater practicable use than current studies which are considered as little more than individual case studies [ 82 , 83 , 84 , 85 , 86 , 87 , 88 , 89 , 90 ].

When to database search is another question posed in the literature. Beyer et al. [ 91 ] report that databases can be prioritised for literature searching which, whilst not addressing the question of which databases to search, may at least bring clarity as to which databases to search first [ 91 ]. Paradoxically, this links to studies that suggest PubMed should be searched in addition to MEDLINE (OVID interface) since this improves the currency of systematic reviews [ 92 , 93 ]. Cooper et al. (2017) have tested the idea of database searching not as a primary search method (as suggested in the guidance) but as a supplementary search method in order to manage the volume of studies identified for an environmental effectiveness systematic review. Their case study compared the effectiveness of database searching versus a protocol using supplementary search methods and found that the latter identified more relevant studies for review than searching bibliographic databases [ 94 ].

Key stage six: Determining the process of literature searching and deciding where to search (supplementary search methods)

Table 2 also summaries the process of literature searching which follows bibliographic database searching. As Table 2 sets out, guidance that supplementary literature search methods should be used in systematic reviews recurs across documents, but the order in which these methods are used, and the extent to which they are used, varies. We noted inconsistency in the labelling of supplementary search methods between guidance documents.

Rather than focus on the guidance on how to use the methods (which has been summarised in a recent review [ 95 ]), we focus on the aim or purpose of supplementary search methods.

The Cochrane Handbook reported that ‘efforts’ to identify unpublished studies should be made [ 9 ]. Four guidance documents [ 2 , 3 , 6 , 9 ] acknowledged that searching beyond bibliographic databases was necessary since ‘databases are not the only source of literature’ [ 2 ]. Only one document reported any guidance on determining when to use supplementary methods. The IQWiG handbook reported that the use of handsearching (in their example) could be determined on a ‘case-by-case basis’ which implies that the use of these methods is optional rather than mandatory. This is in contrast to the guidance (above) on bibliographic database searching.

The issue for supplementary search methods is similar in many ways to the issue of searching bibliographic databases: demonstrating value. The purpose and contribution of supplementary search methods in systematic reviews is increasingly acknowledged [ 37 , 61 , 62 , 96 , 97 , 98 , 99 , 100 , 101 ] but understanding the value of the search methods to identify studies and data is unclear. In a recently published review, Cooper et al. (2017) reviewed the literature on supplementary search methods looking to determine the advantages, disadvantages and resource implications of using supplementary search methods [ 95 ]. This review also summarises the key guidance and empirical studies and seeks to address the question on when to use these search methods and when not to [ 95 ]. The guidance is limited in this regard and, as Table 2 demonstrates, offers conflicting advice on the order of searching, and the extent to which these search methods should be used in systematic reviews.

Key stage seven: Managing the references

Five of the documents provided guidance on managing references, for example downloading, de-duplicating and managing the output of literature searches [ 2 , 4 , 6 , 8 , 10 ]. This guidance typically itemised available bibliographic management tools rather than offering guidance on how to use them specifically [ 2 , 4 , 6 , 8 ]. The CEE handbook provided guidance on importing data where no direct export option is available (e.g. web-searching) [ 10 ].

The literature on using bibliographic management tools is not large relative to the number of ‘how to’ videos on platforms such as YouTube (see for example [ 102 ]). These YouTube videos confirm the overall lack of ‘how to’ guidance identified in this study and offer useful instruction on managing references. Bramer et al. set out methods for de-duplicating data and reviewing references in Endnote [ 103 , 104 ] and Gall tests the direct search function within Endnote to access databases such as PubMed, finding a number of limitations [ 105 ]. Coar et al. and Ahmed et al. consider the role of the free-source tool, Zotero [ 106 , 107 ]. Managing references is a key administrative function in the process of review particularly for documenting searches in PRISMA guidance.

Key stage eight: Documenting the search

The Cochrane Handbook was the only guidance document to recommend a specific reporting guideline: Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [ 9 ]. Six documents provided guidance on reporting the process of literature searching with specific criteria to report [ 3 , 4 , 6 , 8 , 9 , 10 ]. There was consensus on reporting: the databases searched (and the host searched by), the search strategies used, and any use of limits (e.g. date, language, search filters (The CRD handbook called for these limits to be justified [ 6 ])). Three guidance documents reported that the number of studies identified should be recorded [ 3 , 6 , 10 ]. The number of duplicates identified [ 10 ], the screening decisions [ 3 ], a comprehensive list of grey literature sources searched (and full detail for other supplementary search methods) [ 8 ], and an annotation of search terms tested but not used [ 4 ] were identified as unique items in four documents.

The Cochrane Handbook was the only guidance document to note that the full search strategies for each database should be included in the Additional file 1 of the review [ 9 ].

All guidance documents should ultimately deliver completed systematic reviews that fulfil the requirements of the PRISMA reporting guidelines [ 108 ]. The guidance broadly requires the reporting of data that corresponds with the requirements of the PRISMA statement although documents typically ask for diverse and additional items [ 108 ]. In 2008, Sampson et al. observed a lack of consensus on reporting search methods in systematic reviews [ 109 ] and this remains the case as of 2017, as evidenced in the guidance documents, and in spite of the publication of the PRISMA guidelines in 2009 [ 110 ]. It is unclear why the collective guidance does not more explicitly endorse adherence to the PRISMA guidance.

Reporting of literature searching is a key area in systematic reviews since it sets out clearly what was done and how the conclusions of the review can be believed [ 52 , 109 ]. Despite strong endorsement in the guidance documents, specifically supported in PRISMA guidance, and other related reporting standards too (such as ENTREQ for qualitative evidence synthesis, STROBE for reviews of observational studies), authors still highlight the prevalence of poor standards of literature search reporting [ 31 , 110 , 111 , 112 , 113 , 114 , 115 , 116 , 117 , 118 , 119 ]. To explore issues experienced by authors in reporting literature searches, and look at uptake of PRISMA, Radar et al. [ 120 ] surveyed over 260 review authors to determine common problems and their work summaries the practical aspects of reporting literature searching [ 120 ]. Atkinson et al. [ 121 ] have also analysed reporting standards for literature searching, summarising recommendations and gaps for reporting search strategies [ 121 ].

One area that is less well covered by the guidance, but nevertheless appears in this literature, is the quality appraisal or peer review of literature search strategies. The PRESS checklist is the most prominent and it aims to develop evidence-based guidelines to peer review of electronic search strategies [ 5 , 122 , 123 ]. A corresponding guideline for documentation of supplementary search methods does not yet exist although this idea is currently being explored.

How the reporting of the literature searching process corresponds to critical appraisal tools is an area for further research. In the survey undertaken by Radar et al. (2014), 86% of survey respondents (153/178) identified a need for further guidance on what aspects of the literature search process to report [ 120 ]. The PRISMA statement offers a brief summary of what to report but little practical guidance on how to report it [ 108 ]. Critical appraisal tools for systematic reviews, such as AMSTAR 2 (Shea et al. [ 124 ]) and ROBIS (Whiting et al. [ 125 ]), can usefully be read alongside PRISMA guidance, since they offer greater detail on how the reporting of the literature search will be appraised and, therefore, they offer a proxy on what to report [ 124 , 125 ]. Further research in the form of a study which undertakes a comparison between PRISMA and quality appraisal checklists for systematic reviews would seem to begin addressing the call, identified by Radar et al., for further guidance on what to report [ 120 ].

Limitations

Other handbooks exist.

A potential limitation of this literature review is the focus on guidance produced in Europe (the UK specifically) and Australia. We justify the decision for our selection of the nine guidance documents reviewed in this literature review in section “ Identifying guidance ”. In brief, these nine guidance documents were selected as the most relevant health care guidance that inform UK systematic reviewing practice, given that the UK occupies a prominent position in the science of health information retrieval. We acknowledge the existence of other guidance documents, such as those from North America (e.g. the Agency for Healthcare Research and Quality (AHRQ) [ 126 ], The Institute of Medicine [ 127 ] and the guidance and resources produced by the Canadian Agency for Drugs and Technologies in Health (CADTH) [ 128 ]). We comment further on this directly below.

The handbooks are potentially linked to one another

What is not clear is the extent to which the guidance documents inter-relate or provide guidance uniquely. The Cochrane Handbook, first published in 1994, is notably a key source of reference in guidance and systematic reviews beyond Cochrane reviews. It is not clear to what extent broadening the sample of guidance handbooks to include North American handbooks, and guidance handbooks from other relevant countries too, would alter the findings of this literature review or develop further support for the process model. Since we cannot be clear, we raise this as a potential limitation of this literature review. On our initial review of a sample of North American, and other, guidance documents (before selecting the guidance documents considered in this review), however, we do not consider that the inclusion of these further handbooks would alter significantly the findings of this literature review.

This is a literature review

A further limitation of this review was that the review of published studies is not a systematic review of the evidence for each key stage. It is possible that other relevant studies could help contribute to the exploration and development of the key stages identified in this review.

This literature review would appear to demonstrate the existence of a shared model of the literature searching process in systematic reviews. We call this model ‘the conventional approach’, since it appears to be common convention in nine different guidance documents.

The findings reported above reveal eight key stages in the process of literature searching for systematic reviews. These key stages are consistently reported in the nine guidance documents which suggests consensus on the key stages of literature searching, and therefore the process of literature searching as a whole, in systematic reviews.

In Table 2 , we demonstrate consensus regarding the application of literature search methods. All guidance documents distinguish between primary and supplementary search methods. Bibliographic database searching is consistently the first method of literature searching referenced in each guidance document. Whilst the guidance uniformly supports the use of supplementary search methods, there is little evidence for a consistent process with diverse guidance across documents. This may reflect differences in the core focus across each document, linked to differences in identifying effectiveness studies or qualitative studies, for instance.

Eight of the nine guidance documents reported on the aims of literature searching. The shared understanding was that literature searching should be thorough and comprehensive in its aim and that this process should be reported transparently so that that it could be reproduced. Whilst only three documents explicitly link this understanding to minimising bias, it is clear that comprehensive literature searching is implicitly linked to ‘not missing relevant studies’ which is approximately the same point.

Defining the key stages in this review helps categorise the scholarship available, and it prioritises areas for development or further study. The supporting studies on preparing for literature searching (key stage three, ‘preparation’) were, for example, comparatively few, and yet this key stage represents a decisive moment in literature searching for systematic reviews. It is where search strategy structure is determined, search terms are chosen or discarded, and the resources to be searched are selected. Information specialists, librarians and researchers, are well placed to develop these and other areas within the key stages we identify.

This review calls for further research to determine the suitability of using the conventional approach. The publication dates of the guidance documents which underpin the conventional approach may raise questions as to whether the process which they each report remains valid for current systematic literature searching. In addition, it may be useful to test whether it is desirable to use the same process model of literature searching for qualitative evidence synthesis as that for reviews of intervention effectiveness, which this literature review demonstrates is presently recommended best practice.

Abbreviations

Behaviour of interest; Health context; Exclusions; Models or Theories

Cochrane Database of Systematic Reviews

The Cochrane Central Register of Controlled Trials

Database of Abstracts of Reviews of Effects

Enhancing transparency in reporting the synthesis of qualitative research

Institute for Quality and Efficiency in Healthcare

National Institute for Clinical Excellence

Population, Intervention, Comparator, Outcome

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

Setting, Perspective, Intervention, Comparison, Evaluation

Sample, Phenomenon of Interest, Design, Evaluation, Research type

STrengthening the Reporting of OBservational studies in Epidemiology

Trial Search Co-ordinators

Booth A. Unpacking your literature search toolbox: on search styles and tactics. Health Information & Libraries Journal. 2008;25(4):313–7.

Article   Google Scholar  

Petticrew M, Roberts H. Systematic reviews in the social sciences: a practical guide. Oxford: Blackwell Publishing Ltd; 2006.

Book   Google Scholar  

Institute for Quality and Efficiency in Health Care (IQWiG). IQWiG Methods Resources. 7 Information retrieval 2014 [Available from: https://www.ncbi.nlm.nih.gov/books/NBK385787/ .

NICE: National Institute for Health and Care Excellence. Developing NICE guidelines: the manual 2014. Available from: https://www.nice.org.uk/media/default/about/what-we-do/our-programmes/developing-nice-guidelines-the-manual.pdf .

Sampson M. MJ, Lefebvre C, Moher D, Grimshaw J. Peer Review of Electronic Search Strategies: PRESS; 2008.

Google Scholar  

Centre for Reviews & Dissemination. Systematic reviews – CRD’s guidance for undertaking reviews in healthcare. York: Centre for Reviews and Dissemination, University of York; 2009.

eunetha: European Network for Health Technology Assesment Process of information retrieval for systematic reviews and health technology assessments on clinical effectiveness 2016. Available from: http://www.eunethta.eu/sites/default/files/Guideline_Information_Retrieval_V1-1.pdf .

Kugley SWA, Thomas J, Mahood Q, Jørgensen AMK, Hammerstrøm K, Sathe N. Searching for studies: a guide to information retrieval for Campbell systematic reviews. Oslo: Campbell Collaboration. 2017; Available from: https://www.campbellcollaboration.org/library/searching-for-studies-information-retrieval-guide-campbell-reviews.html

Lefebvre C, Manheimer E, Glanville J. Chapter 6: searching for studies. In: JPT H, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions; 2011.

Collaboration for Environmental Evidence. Guidelines for Systematic Review and Evidence Synthesis in Environmental Management.: Environmental Evidence:; 2013. Available from: http://www.environmentalevidence.org/wp-content/uploads/2017/01/Review-guidelines-version-4.2-final-update.pdf .

The Joanna Briggs Institute. Joanna Briggs institute reviewers’ manual. 2014th ed: the Joanna Briggs institute; 2014. Available from: https://joannabriggs.org/assets/docs/sumari/ReviewersManual-2014.pdf

Beverley CA, Booth A, Bath PA. The role of the information specialist in the systematic review process: a health information case study. Health Inf Libr J. 2003;20(2):65–74.

Article   CAS   Google Scholar  

Harris MR. The librarian's roles in the systematic review process: a case study. Journal of the Medical Library Association. 2005;93(1):81–7.

PubMed   PubMed Central   Google Scholar  

Egger JB. Use of recommended search strategies in systematic reviews and the impact of librarian involvement: a cross-sectional survey of recent authors. PLoS One. 2015;10(5):e0125931.

Li L, Tian J, Tian H, Moher D, Liang F, Jiang T, et al. Network meta-analyses could be improved by searching more sources and by involving a librarian. J Clin Epidemiol. 2014;67(9):1001–7.

Article   PubMed   Google Scholar  

McGowan J, Sampson M. Systematic reviews need systematic searchers. J Med Libr Assoc. 2005;93(1):74–80.

Rethlefsen ML, Farrell AM, Osterhaus Trzasko LC, Brigham TJ. Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. J Clin Epidemiol. 2015;68(6):617–26.

Weller AC. Mounting evidence that librarians are essential for comprehensive literature searches for meta-analyses and Cochrane reports. J Med Libr Assoc. 2004;92(2):163–4.

Swinkels A, Briddon J, Hall J. Two physiotherapists, one librarian and a systematic literature review: collaboration in action. Health Info Libr J. 2006;23(4):248–56.

Foster M. An overview of the role of librarians in systematic reviews: from expert search to project manager. EAHIL. 2015;11(3):3–7.

Lawson L. OPERATING OUTSIDE LIBRARY WALLS 2004.

Vassar M, Yerokhin V, Sinnett PM, Weiher M, Muckelrath H, Carr B, et al. Database selection in systematic reviews: an insight through clinical neurology. Health Inf Libr J. 2017;34(2):156–64.

Townsend WA, Anderson PF, Ginier EC, MacEachern MP, Saylor KM, Shipman BL, et al. A competency framework for librarians involved in systematic reviews. Journal of the Medical Library Association : JMLA. 2017;105(3):268–75.

Cooper ID, Crum JA. New activities and changing roles of health sciences librarians: a systematic review, 1990-2012. Journal of the Medical Library Association : JMLA. 2013;101(4):268–77.

Crum JA, Cooper ID. Emerging roles for biomedical librarians: a survey of current practice, challenges, and changes. Journal of the Medical Library Association : JMLA. 2013;101(4):278–86.

Dudden RF, Protzko SL. The systematic review team: contributions of the health sciences librarian. Med Ref Serv Q. 2011;30(3):301–15.

Golder S, Loke Y, McIntosh HM. Poor reporting and inadequate searches were apparent in systematic reviews of adverse effects. J Clin Epidemiol. 2008;61(5):440–8.

Maggio LA, Tannery NH, Kanter SL. Reproducibility of literature search reporting in medical education reviews. Academic medicine : journal of the Association of American Medical Colleges. 2011;86(8):1049–54.

Meert D, Torabi N, Costella J. Impact of librarians on reporting of the literature searching component of pediatric systematic reviews. Journal of the Medical Library Association : JMLA. 2016;104(4):267–77.

Morris M, Boruff JT, Gore GC. Scoping reviews: establishing the role of the librarian. Journal of the Medical Library Association : JMLA. 2016;104(4):346–54.

Koffel JB, Rethlefsen ML. Reproducibility of search strategies is poor in systematic reviews published in high-impact pediatrics, cardiology and surgery journals: a cross-sectional study. PLoS One. 2016;11(9):e0163309.

Article   PubMed   PubMed Central   CAS   Google Scholar  

Fehrmann P, Thomas J. Comprehensive computer searches and reporting in systematic reviews. Research Synthesis Methods. 2011;2(1):15–32.

Booth A. Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Systematic Reviews. 2016;5(1):74.

Article   PubMed   PubMed Central   Google Scholar  

Egger M, Juni P, Bartlett C, Holenstein F, Sterne J. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health technology assessment (Winchester, England). 2003;7(1):1–76.

Tricco AC, Tetzlaff J, Sampson M, Fergusson D, Cogo E, Horsley T, et al. Few systematic reviews exist documenting the extent of bias: a systematic review. J Clin Epidemiol. 2008;61(5):422–34.

Booth A. How much searching is enough? Comprehensive versus optimal retrieval for technology assessments. Int J Technol Assess Health Care. 2010;26(4):431–5.

Papaioannou D, Sutton A, Carroll C, Booth A, Wong R. Literature searching for social science systematic reviews: consideration of a range of search techniques. Health Inf Libr J. 2010;27(2):114–22.

Petticrew M. Time to rethink the systematic review catechism? Moving from ‘what works’ to ‘what happens’. Systematic Reviews. 2015;4(1):36.

Betrán AP, Say L, Gülmezoglu AM, Allen T, Hampson L. Effectiveness of different databases in identifying studies for systematic reviews: experience from the WHO systematic review of maternal morbidity and mortality. BMC Med Res Methodol. 2005;5

Felson DT. Bias in meta-analytic research. J Clin Epidemiol. 1992;45(8):885–92.

Article   PubMed   CAS   Google Scholar  

Franco A, Malhotra N, Simonovits G. Publication bias in the social sciences: unlocking the file drawer. Science. 2014;345(6203):1502–5.

Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews. BMC Med Res Methodol. 2017;17(1):64.

Schmucker CM, Blümle A, Schell LK, Schwarzer G, Oeller P, Cabrera L, et al. Systematic review finds that study data not published in full text articles have unclear impact on meta-analyses results in medical research. PLoS One. 2017;12(4):e0176210.

Egger M, Zellweger-Zahner T, Schneider M, Junker C, Lengeler C, Antes G. Language bias in randomised controlled trials published in English and German. Lancet (London, England). 1997;350(9074):326–9.

Moher D, Pham B, Lawson ML, Klassen TP. The inclusion of reports of randomised trials published in languages other than English in systematic reviews. Health technology assessment (Winchester, England). 2003;7(41):1–90.

Pham B, Klassen TP, Lawson ML, Moher D. Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary. J Clin Epidemiol. 2005;58(8):769–76.

Mills EJ, Kanters S, Thorlund K, Chaimani A, Veroniki A-A, Ioannidis JPA. The effects of excluding treatments from network meta-analyses: survey. BMJ : British Medical Journal. 2013;347

Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. The contribution of databases to the results of systematic reviews: a cross-sectional study. BMC Med Res Methodol. 2016;16(1):127.

van Driel ML, De Sutter A, De Maeseneer J, Christiaens T. Searching for unpublished trials in Cochrane reviews may not be worth the effort. J Clin Epidemiol. 2009;62(8):838–44.e3.

Buchberger B, Krabbe L, Lux B, Mattivi JT. Evidence mapping for decision making: feasibility versus accuracy - when to abandon high sensitivity in electronic searches. German medical science : GMS e-journal. 2016;14:Doc09.

Lorenc T, Pearson M, Jamal F, Cooper C, Garside R. The role of systematic reviews of qualitative evidence in evaluating interventions: a case study. Research Synthesis Methods. 2012;3(1):1–10.

Gough D. Weight of evidence: a framework for the appraisal of the quality and relevance of evidence. Res Pap Educ. 2007;22(2):213–28.

Barroso J, Gollop CJ, Sandelowski M, Meynell J, Pearce PF, Collins LJ. The challenges of searching for and retrieving qualitative studies. West J Nurs Res. 2003;25(2):153–78.

Britten N, Garside R, Pope C, Frost J, Cooper C. Asking more of qualitative synthesis: a response to Sally Thorne. Qual Health Res. 2017;27(9):1370–6.

Booth A, Carroll C. Systematic searching for theory to inform systematic reviews: is it feasible? Is it desirable? Health Info Libr J. 2015;32(3):220–35.

Kwon Y, Powelson SE, Wong H, Ghali WA, Conly JM. An assessment of the efficacy of searching in biomedical databases beyond MEDLINE in identifying studies for a systematic review on ward closures as an infection control intervention to control outbreaks. Syst Rev. 2014;3:135.

Nussbaumer-Streit B, Klerings I, Wagner G, Titscher V, Gartlehner G. Assessing the validity of abbreviated literature searches for rapid reviews: protocol of a non-inferiority and meta-epidemiologic study. Systematic Reviews. 2016;5:197.

Wagner G, Nussbaumer-Streit B, Greimel J, Ciapponi A, Gartlehner G. Trading certainty for speed - how much uncertainty are decisionmakers and guideline developers willing to accept when using rapid reviews: an international survey. BMC Med Res Methodol. 2017;17(1):121.

Ogilvie D, Hamilton V, Egan M, Petticrew M. Systematic reviews of health effects of social interventions: 1. Finding the evidence: how far should you go? J Epidemiol Community Health. 2005;59(9):804–8.

Royle P, Milne R. Literature searching for randomized controlled trials used in Cochrane reviews: rapid versus exhaustive searches. Int J Technol Assess Health Care. 2003;19(4):591–603.

Pearson M, Moxham T, Ashton K. Effectiveness of search strategies for qualitative research about barriers and facilitators of program delivery. Eval Health Prof. 2011;34(3):297–308.

Levay P, Raynor M, Tuvey D. The Contributions of MEDLINE, Other Bibliographic Databases and Various Search Techniques to NICE Public Health Guidance. 2015. 2015;10(1):19.

Nussbaumer-Streit B, Klerings I, Wagner G, Heise TL, Dobrescu AI, Armijo-Olivo S, et al. Abbreviated literature searches were viable alternatives to comprehensive searches: a meta-epidemiological study. J Clin Epidemiol. 2018;102:1–11.

Briscoe S, Cooper C, Glanville J, Lefebvre C. The loss of the NHS EED and DARE databases and the effect on evidence synthesis and evaluation. Res Synth Methods. 2017;8(3):256–7.

Stansfield C, O'Mara-Eves A, Thomas J. Text mining for search term development in systematic reviewing: A discussion of some methods and challenges. Research Synthesis Methods.n/a-n/a.

Petrova M, Sutcliffe P, Fulford KW, Dale J. Search terms and a validated brief search filter to retrieve publications on health-related values in Medline: a word frequency analysis study. Journal of the American Medical Informatics Association : JAMIA. 2012;19(3):479–88.

Stansfield C, Thomas J, Kavanagh J. 'Clustering' documents automatically to support scoping reviews of research: a case study. Res Synth Methods. 2013;4(3):230–41.

PubMed   Google Scholar  

Methley AM, Campbell S, Chew-Graham C, McNally R, Cheraghi-Sohi S. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res. 2014;14:579.

Andrew B. Clear and present questions: formulating questions for evidence based practice. Library Hi Tech. 2006;24(3):355–68.

Cooke A, Smith D, Booth A. Beyond PICO: the SPIDER tool for qualitative evidence synthesis. Qual Health Res. 2012;22(10):1435–43.

Whiting P, Westwood M, Bojke L, Palmer S, Richardson G, Cooper J, et al. Clinical effectiveness and cost-effectiveness of tests for the diagnosis and investigation of urinary tract infection in children: a systematic review and economic model. Health technology assessment (Winchester, England). 2006;10(36):iii-iv, xi-xiii, 1–154.

Cooper C, Levay P, Lorenc T, Craig GM. A population search filter for hard-to-reach populations increased search efficiency for a systematic review. J Clin Epidemiol. 2014;67(5):554–9.

Hausner E, Waffenschmidt S, Kaiser T, Simon M. Routine development of objectively derived search strategies. Systematic Reviews. 2012;1(1):19.

Hausner E, Guddat C, Hermanns T, Lampert U, Waffenschmidt S. Prospective comparison of search strategies for systematic reviews: an objective approach yielded higher sensitivity than a conceptual one. J Clin Epidemiol. 2016;77:118–24.

Craven J, Levay P. Recording database searches for systematic reviews - what is the value of adding a narrative to peer-review checklists? A case study of nice interventional procedures guidance. Evid Based Libr Inf Pract. 2011;6(4):72–87.

Wright K, Golder S, Lewis-Light K. What value is the CINAHL database when searching for systematic reviews of qualitative studies? Syst Rev. 2015;4:104.

Beckles Z, Glover S, Ashe J, Stockton S, Boynton J, Lai R, et al. Searching CINAHL did not add value to clinical questions posed in NICE guidelines. J Clin Epidemiol. 2013;66(9):1051–7.

Cooper C, Rogers M, Bethel A, Briscoe S, Lowe J. A mapping review of the literature on UK-focused health and social care databases. Health Inf Libr J. 2015;32(1):5–22.

Younger P, Boddy K. When is a search not a search? A comparison of searching the AMED complementary health database via EBSCOhost, OVID and DIALOG. Health Inf Libr J. 2009;26(2):126–35.

Lam MT, McDiarmid M. Increasing number of databases searched in systematic reviews and meta-analyses between 1994 and 2014. Journal of the Medical Library Association : JMLA. 2016;104(4):284–9.

Bethel A, editor Search summary tables for systematic reviews: results and findings. HLC Conference 2017a.

Aagaard T, Lund H, Juhl C. Optimizing literature search in systematic reviews - are MEDLINE, EMBASE and CENTRAL enough for identifying effect studies within the area of musculoskeletal disorders? BMC Med Res Methodol. 2016;16(1):161.

Adams CE, Frederick K. An investigation of the adequacy of MEDLINE searches for randomized controlled trials (RCTs) of the effects of mental health care. Psychol Med. 1994;24(3):741–8.

Kelly L, St Pierre-Hansen N. So many databases, such little clarity: searching the literature for the topic aboriginal. Canadian family physician Medecin de famille canadien. 2008;54(11):1572–3.

Lawrence DW. What is lost when searching only one literature database for articles relevant to injury prevention and safety promotion? Injury Prevention. 2008;14(6):401–4.

Lemeshow AR, Blum RE, Berlin JA, Stoto MA, Colditz GA. Searching one or two databases was insufficient for meta-analysis of observational studies. J Clin Epidemiol. 2005;58(9):867–73.

Sampson M, Barrowman NJ, Moher D, Klassen TP, Pham B, Platt R, et al. Should meta-analysts search Embase in addition to Medline? J Clin Epidemiol. 2003;56(10):943–55.

Stevinson C, Lawlor DA. Searching multiple databases for systematic reviews: added value or diminishing returns? Complementary Therapies in Medicine. 2004;12(4):228–32.

Suarez-Almazor ME, Belseck E, Homik J, Dorgan M, Ramos-Remus C. Identifying clinical trials in the medical literature with electronic databases: MEDLINE alone is not enough. Control Clin Trials. 2000;21(5):476–87.

Taylor B, Wylie E, Dempster M, Donnelly M. Systematically retrieving research: a case study evaluating seven databases. Res Soc Work Pract. 2007;17(6):697–706.

Beyer FR, Wright K. Can we prioritise which databases to search? A case study using a systematic review of frozen shoulder management. Health Info Libr J. 2013;30(1):49–58.

Duffy S, de Kock S, Misso K, Noake C, Ross J, Stirk L. Supplementary searches of PubMed to improve currency of MEDLINE and MEDLINE in-process searches via Ovid. Journal of the Medical Library Association : JMLA. 2016;104(4):309–12.

Katchamart W, Faulkner A, Feldman B, Tomlinson G, Bombardier C. PubMed had a higher sensitivity than Ovid-MEDLINE in the search for systematic reviews. J Clin Epidemiol. 2011;64(7):805–7.

Cooper C, Lovell R, Husk K, Booth A, Garside R. Supplementary search methods were more effective and offered better value than bibliographic database searching: a case study from public health and environmental enhancement (in Press). Research Synthesis Methods. 2017;

Cooper C, Booth, A., Britten, N., Garside, R. A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: A methodological review. (In Press). BMC Systematic Reviews. 2017.

Greenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ (Clinical research ed). 2005;331(7524):1064–5.

Article   PubMed Central   Google Scholar  

Hinde S, Spackman E. Bidirectional citation searching to completion: an exploration of literature searching methods. PharmacoEconomics. 2015;33(1):5–11.

Levay P, Ainsworth N, Kettle R, Morgan A. Identifying evidence for public health guidance: a comparison of citation searching with web of science and Google scholar. Res Synth Methods. 2016;7(1):34–45.

McManus RJ, Wilson S, Delaney BC, Fitzmaurice DA, Hyde CJ, Tobias RS, et al. Review of the usefulness of contacting other experts when conducting a literature search for systematic reviews. BMJ (Clinical research ed). 1998;317(7172):1562–3.

Westphal A, Kriston L, Holzel LP, Harter M, von Wolff A. Efficiency and contribution of strategies for finding randomized controlled trials: a case study from a systematic review on therapeutic interventions of chronic depression. Journal of public health research. 2014;3(2):177.

Matthews EJ, Edwards AG, Barker J, Bloor M, Covey J, Hood K, et al. Efficient literature searching in diffuse topics: lessons from a systematic review of research on communicating risk to patients in primary care. Health Libr Rev. 1999;16(2):112–20.

Bethel A. Endnote Training (YouTube Videos) 2017b [Available from: http://medicine.exeter.ac.uk/esmi/workstreams/informationscience/is_resources,_guidance_&_advice/ .

Bramer WM, Giustini D, de Jonge GB, Holland L, Bekhuis T. De-duplication of database search results for systematic reviews in EndNote. Journal of the Medical Library Association : JMLA. 2016;104(3):240–3.

Bramer WM, Milic J, Mast F. Reviewing retrieved references for inclusion in systematic reviews using EndNote. Journal of the Medical Library Association : JMLA. 2017;105(1):84–7.

Gall C, Brahmi FA. Retrieval comparison of EndNote to search MEDLINE (Ovid and PubMed) versus searching them directly. Medical reference services quarterly. 2004;23(3):25–32.

Ahmed KK, Al Dhubaib BE. Zotero: a bibliographic assistant to researcher. J Pharmacol Pharmacother. 2011;2(4):303–5.

Coar JT, Sewell JP. Zotero: harnessing the power of a personal bibliographic manager. Nurse Educ. 2010;35(5):205–7.

Moher D, Liberati A, Tetzlaff J, Altman DG, The PG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097.

Sampson M, McGowan J, Tetzlaff J, Cogo E, Moher D. No consensus exists on search reporting methods for systematic reviews. J Clin Epidemiol. 2008;61(8):748–54.

Toews LC. Compliance of systematic reviews in veterinary journals with preferred reporting items for systematic reviews and meta-analysis (PRISMA) literature search reporting guidelines. Journal of the Medical Library Association : JMLA. 2017;105(3):233–9.

Booth A. "brimful of STARLITE": toward standards for reporting literature searches. Journal of the Medical Library Association : JMLA. 2006;94(4):421–9. e205

Faggion CM Jr, Wu YC, Tu YK, Wasiak J. Quality of search strategies reported in systematic reviews published in stereotactic radiosurgery. Br J Radiol. 2016;89(1062):20150878.

Mullins MM, DeLuca JB, Crepaz N, Lyles CM. Reporting quality of search methods in systematic reviews of HIV behavioral interventions (2000–2010): are the searches clearly explained, systematic and reproducible? Research Synthesis Methods. 2014;5(2):116–30.

Yoshii A, Plaut DA, McGraw KA, Anderson MJ, Wellik KE. Analysis of the reporting of search strategies in Cochrane systematic reviews. Journal of the Medical Library Association : JMLA. 2009;97(1):21–9.

Bigna JJ, Um LN, Nansseu JR. A comparison of quality of abstracts of systematic reviews including meta-analysis of randomized controlled trials in high-impact general medicine journals before and after the publication of PRISMA extension for abstracts: a systematic review and meta-analysis. Syst Rev. 2016;5(1):174.

Akhigbe T, Zolnourian A, Bulters D. Compliance of systematic reviews articles in brain arteriovenous malformation with PRISMA statement guidelines: review of literature. Journal of clinical neuroscience : official journal of the Neurosurgical Society of Australasia. 2017;39:45–8.

Tao KM, Li XQ, Zhou QH, Moher D, Ling CQ, Yu WF. From QUOROM to PRISMA: a survey of high-impact medical journals' instructions to authors and a review of systematic reviews in anesthesia literature. PLoS One. 2011;6(11):e27611.

Wasiak J, Tyack Z, Ware R. Goodwin N. Jr. Poor methodological quality and reporting standards of systematic reviews in burn care management. International wound journal: Faggion CM; 2016.

Tam WW, Lo KK, Khalechelvam P. Endorsement of PRISMA statement and quality of systematic reviews and meta-analyses published in nursing journals: a cross-sectional study. BMJ Open. 2017;7(2):e013905.

Rader T, Mann M, Stansfield C, Cooper C, Sampson M. Methods for documenting systematic review searches: a discussion of common issues. Res Synth Methods. 2014;5(2):98–115.

Atkinson KM, Koenka AC, Sanchez CE, Moshontz H, Cooper H. Reporting standards for literature searches and report inclusion criteria: making research syntheses more transparent and easy to replicate. Res Synth Methods. 2015;6(1):87–95.

McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS peer review of electronic search strategies: 2015 guideline statement. J Clin Epidemiol. 2016;75:40–6.

Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C. An evidence-based practice guideline for the peer review of electronic search strategies. J Clin Epidemiol. 2009;62(9):944–52.

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ (Clinical research ed). 2017;358.

Whiting P, Savović J, Higgins JPT, Caldwell DM, Reeves BC, Shea B, et al. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016;69:225–34.

Relevo R, Balshem H. Finding evidence for comparing medical interventions: AHRQ and the effective health care program. J Clin Epidemiol. 2011;64(11):1168–77.

Medicine Io. Standards for Systematic Reviews 2011 [Available from: http://www.nationalacademies.org/hmd/Reports/2011/Finding-What-Works-in-Health-Care-Standards-for-Systematic-Reviews/Standards.aspx .

CADTH: Resources 2018.

Download references

Acknowledgements

CC acknowledges the supervision offered by Professor Chris Hyde.

This publication forms a part of CC’s PhD. CC’s PhD was funded through the National Institute for Health Research (NIHR) Health Technology Assessment (HTA) Programme (Project Number 16/54/11). The open access fee for this publication was paid for by Exeter Medical School.

RG and NB were partially supported by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care South West Peninsula.

The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.

Author information

Authors and affiliations.

Institute of Health Research, University of Exeter Medical School, Exeter, UK

Chris Cooper & Jo Varley-Campbell

HEDS, School of Health and Related Research (ScHARR), University of Sheffield, Sheffield, UK

Andrew Booth

Nicky Britten

European Centre for Environment and Human Health, University of Exeter Medical School, Truro, UK

Ruth Garside

You can also search for this author in PubMed   Google Scholar

Contributions

CC conceived the idea for this study and wrote the first draft of the manuscript. CC discussed this publication in PhD supervision with AB and separately with JVC. CC revised the publication with input and comments from AB, JVC, RG and NB. All authors revised the manuscript prior to submission. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Chris Cooper .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional file

Additional file 1:.

Appendix tables and PubMed search strategy. Key studies used for pearl growing per key stage, working data extraction tables and the PubMed search strategy. (DOCX 30 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Cooper, C., Booth, A., Varley-Campbell, J. et al. Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies. BMC Med Res Methodol 18 , 85 (2018). https://doi.org/10.1186/s12874-018-0545-3

Download citation

Received : 20 September 2017

Accepted : 06 August 2018

Published : 14 August 2018

DOI : https://doi.org/10.1186/s12874-018-0545-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Literature Search Process
• Citation Chasing
• Tacit Models
• Unique Guidance
• Information Specialists

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]

University of Tasmania, Australia

Systematic reviews for health: 4. develop search terms - controlled vocabulary.

• Handbooks / Guidelines for Systematic Reviews
• Standards for Reporting
• Registering a Protocol
• Tools for Systematic Review
• Online Tutorials & Courses
• Books and Articles about Systematic Reviews
• Finding Systematic Reviews
• Critical Appraisal
• Library Help
• Bibliographic Databases
• Grey Literature
• Handsearching
• Citation Searching
• 1. Formulate the Research Question
• 2. Identify the Key Concepts
• 3. Develop Search Terms - Free-Text
• 4. Develop Search Terms - Controlled Vocabulary
• 5. Search Fields
• 6. Phrase Searching, Wildcards and Proximity Operators
• 7. Boolean Operators
• 8. Search Limits
• 9. Pilot Search Strategy & Monitor Its Development
• 10. Final Search Strategy
• 11. Adapt Search Syntax
• Documenting Search Strategies
• Handling Results & Storing Papers

Develop Search Terms

The  Cochrane Handbook, 4.4.4   suggests searches should comprise a combination of subject terms selected from the controlled vocabulary or thesaurus (‘exploded' where appropriate) with a wide range of free-text terms   (see  Step 3 ) in order to identify as many relevant records as possible searches.

• If you use keywords only, you could miss articles that do not use your precise terms.
• If you use controlled vocabulary only, you could miss articles that have not been indexed yet or have older indexing.

Search Strategy Template

• Template for Systematic Review Search
• Template for Systematic Review search

Step 4. Controlled Vocabulary Terms

Many major bibliographic databases can be searched using subject headings (also known as subject terms, index terms, descriptors). Subject headings are used by databases to describe the content of each article they index using a "controlled vocabulary". These subject headings differ between the databases, hence, you need to establish the controlled vocabulary terms for each database that you search.  

Examples of controlled vocabulary options for UTAS databases are:

• MeSH - Medical Subject Headings ( PubMed , Medline via Ovid , Cochrane )
• Emtree ( Embase via Ovid )
• CINAHL Headings ( CINAHL )
• PsycINFO Thesaurus ( PsycInfo via Ovid )
• Index Terms ( Scopus )

Including subject headings is your search is useful because they provide a way of retrieving articles that may use different words to describe the same concept and because they can provide information beyond that which is simply contained in the words of the title and abstract.

NOTE!   Please note that Web of Science does not provide subject headings to control a search for a topic.

Identify Relevant Subject Headings

• It is a good idea to carry out a simple search using a few key concepts in the different databases, to find some relevant articles and see how they were indexed using controlled vocabulary. Repeat this for a number of different articles to see what subject headings have been used and decide whether you want to use them for your own search too.
• TIP!  It's best to use truncation as otherwise MeSH terms may not be found (e.g. search for body temperature*)
• In Ovid databases ( Medline , Embase and PsycInfo ): select the tab  Search Tools and enter each term in the Map Term search box
• In CINAHL : select CINAHL  Headings and search each term in the search box
• You may like to check for existing search strategies, e.g. subject search filters  ( see  Step 3 ) .
• Use text mining tool PubMed PubReMiner : a simple PubMed search reveals the most frequently used MeSH terms.
• Make a note of the exact subject headings, including commas. Copy and paste is best!
• More than one subject heading might be relevant to your topic.
• One subject heading may combine two concepts. Add the subject heading to both concepts.
• Sometimes a concept can only be captured by combining two separate subject headings.
• Consider See Also / Related terms and Previous indexing
• It is worth noting the Entry terms, Used for terms, Related terms mentioned for the relevant subject headings as they could give you ideas for alternative free-text terms for  Step 3 .

You need to identify controlled vocabulary terms for all your concepts. This needs to be done for each database that is to be searched. You will find that the terms are different but similar for each database.

• Medline via Ovid
• Medline via PubMed

MeSH (Medical Subject Headings)  is the controlled vocabulary used in Medline. T his Concept Table lists relevant MeSH terms  using the Ovid syntax.

​ See source of example

MeSH (Medical Subject Headings)  is the controlled vocabulary used in Medline. T his Concept Table lists relevant MeSH terms  using the PubMed syntax.

*  The MeSH term Alzheimer disease is not specifically included as it is covered by exploding the MeSH term Dementia . Alzheimer disease is a more specific term below Dementia in the MeSH tree. 

Explode / Not Explode

When you search for relevant subject headings, select the most relevant term. Examine where in the tree/hierarchy it sits. There may also be more specific terms below your term. You need to decide whether you only want to

• search on the exact subject heading only (not explode it),
• include all narrower/more specific terms below it in the search (explode it), or
• pick and choose from above (broader terms) or below the term (more specific terms).

NOTE !    The broadest relevant subject heading for your topic should always be included.

The default of whether subject headings are exploded or not varies between the databases:

For a subject heading that is the most specific/narrow term in a tree, it does not matter whether the term is exploded or not, as the same search is performed.

Need More Help? Book a consultation with a  Learning and Research Librarian  or contact  [email protected] .

• << Previous: 3. Develop Search Terms - Free-Text
• Next: 5. Search Fields >>
• Last Updated: Jun 7, 2024 2:00 PM
• URL: https://utas.libguides.com/SystematicReviews

• Subject guides
• Systematic Review
• Identify search terms

Systematic Review: Identify search terms

• Getting started
• Manuals, documentation & PRISMA
• Develop question & key concepts
• Look for existing reviews
• Scoping searches & gold set
• Select databases & grey literature sources
• Develop criteria & protocol
• Run your search
• Limits & filters
• Review & test your search
• Save & manage your search results
• Database search translation
• Screening process steps
• Assess quality of your included studies
• Request a consultation

Identify the search terms that you will use

Developing a search strategy is the process of converting your research question into a format that the database can interpret. Databases work by matching the search terms that you enter to the titles, abstracts, and subject headings in the records of items in the database. In order to locate the relevant literature you need to use the same terms that are used in relevant sources.

For each identified concept in your research question, make a list of relevant keywords and subject headings.

• Keywords are simply the terms used within an article. A database will generally search for keywords in the title and abstract fields, and may also search other fields of the database record.
• It is important to include alternative spellings and synonyms for your keywords to retrieve all articles on your topic.
• Author assigned keywords may also be searchable depending on the database being used.

Subject headings

• Subject headings are internationally recognised index terms that can be used to describe what an article is about. They are assigned after an article is published, and are based on the topic of the article (they may not be words used in the title or abstract).
• A database will only search for subject headings in the subject field of database records. You can also browse subject headings and see broader or narrower terms in the "Tree".
• Subject heading searching increases relevance and recall, and when combined with keyword searching is considered best practice for search strategies when reviewing the literature. 
• Subject headings are often unique to a particular database, so you will need to look for appropriate subject headings in each database you intend to use. They are not available for every topic so only use them if specifically relevant.
• Not all databases will have subject heading searching so ensure that your keywords are robust enough to stand alone in databases that don't index by subject headings.

As part of the process of developing a search strategy, it is recommended that you keep a master list of search terms for each key concept. This will make it easier when it comes to translating your search strategy across multiple database platforms. 

Text mining for search terms

Once you have identified your key concepts, identify the words most likely to have been used in the published literature on this topic. It is important to develop a comprehensive range of terms for each discrete concept entailing a combination of subject headings and a wide range of keywords/phrases for each concept. 

You can scan or  ‘text mine' the papers in your gold set for useful search terms.

• Check if the author has provided a list of key terms that describe the content of the article, these may appear below the abstract or in the database record. Also look for relevant keywords or phrases that have been used in the title and abstract fields of the article.
• Use a title search to locate the gold set papers in a key database such as Ovid Medline or Embase and then use the database record to scan for subject headings and useful keywords or phrases.

Text mining tools you may want to try out are:

• PubReMiner  Enter a simple search string or PMIDS (PubMed IDentifier) from gold set to determine high frequency words and subject headings
• Termine  Upload a pdf, URL, or copy and paste an abstract to highlight frequency of words and phrases
• MeSH on Demand  Enter abstract or text to identify subject headings.

Documenting your search terms

As part of documenting the SR process, it is recommended that you keep a master list of all search terms for each key concept. Create a concept map to list the subject headings and key terms that you have identified. This list can be quite extensive and it can be helpful to use an Excel spreadsheet. This will make it easier when it comes to translating your search strategy across multiple database platforms.

• Concept Map for search terms A simple template for documenting search terms for each key concept
• << Previous: Scoping searches & gold set
• Next: Select databases & grey literature sources >>

Systematic Review

• Systematic reviews

Being systematic

Search terms, choosing databases, finding additional resources.

• Search techniques
• Systematically search databases
• Appraisal & synthesis
• Reporting findings
• Systematic review tools

Searching literature systematically is useful for all types of literature reviews!

However, if you are writing a systematic literature review the search needs to be particularly well planned and structured to ensure it is:

• comprehensive
• transparent

These help ensure bias is eliminated and the review is methodologically sound.

To achieve the above goals, you will need to:

• create a search strategy and ensure it is reviewed by your research group
• document each stage of your literature searching
• report each stage of quality appraisal 

Identify the key concepts in your research question

The first step in developing your search strategy is identifying the key concepts your research question covers.

• A preliminary search is often done to understand the topic and to refine your research question. 

Identify search terms

Use an iterative process to identify useful search terms for conducting your search. 

• Brainstorm keywords and phrases that can describe each concept you have identified in your research question.
• Create a table to record these keywords
• Select your keywords carefully
• Check against inclusion/exclusion criteria
• Repeated testing   is required to create a robust search strategy for a systematic review
• Run your search on your primary database and evaluate the first page of records to see how suitable your search is
• Identify reasons for irrelevant results and adjust your keywords accordingly 
• Consider whether it would be useful to use broader or narrower terms for your concepts
• Identify keywords in relevant results that you could add to your search to retrieve more relevant resources

Using a concept map or a mind map may help you clarify concepts and the relationships between or within concepts. Watch these YouTube videos for some ideas: 

• How to make a concept map  (by Lucidchart)
• Make sense of this mess world - mind maps  (by Sheng Huang)

Example keywords table:

Research question: What is the relationship between adverse childhood experiences and depression in mothers during the perinatal period? 

Revise your strategy/search terms until :

• the results match your research question
• you are confident you will find all the relevant literature on your topic

See Creating search strings for information on how to enter your search terms into databases. 

Example search string (using Scopus's Advanced search option) for the terms in the above table:

(TITLE-ABS-KEY("advserse childhood experienc*" OR ACE OR "childhood trauma") AND TITLE-ABS-KEY("perinatal depress*" OR "postpartum depress*" OR "postnatal depress*" OR "maternal mental health" OR "maternal psychological distress") AND TITLE-ABS-KEY(mother* OR women*))

See Subject headings  for information on including these database specific terms to your search terms.

Systematic reviewers usually use several databases to search for literature. This ensures that the searching is comprehensive and biases are minimised. 

Use both subject-specific and multidisciplinary databases to find resources relevant to your research question:

• Subject-specific databases: in-depth coverage of literature specific to a research field.
• Multi-disciplinary databases: literature from many research fields - help you find resources from disciplines you may not have considered.

Check for databases in your subject area via the Databases tab > Find by subject on the library homepage .

Find the  key databases that are often used for systematic reviews in this guide. 

Test searches to determine database usefulness. You can consult your Liaison Librarians to finalise the list of databases for your review.

Recommendations:

For all systematic reviews we recommend using Scopus , a high-quality, multidisciplinary database:

• Scopus is an abstract and citation database with links to full text on publisher websites or in other databases.
• Scopus indexes a curated collection of high quality journals along with books and conference proceedings.
• Research outputs are across a range of fields - science, technology, medicine, social science, arts and humanities.

For systematic reviews within the health/biomedical field, we recommend including Medline as one of the databases for your review:

MEDLINE  (via Ebsco, via Ovid, via PubMed)

• Medline is the National Library of Medicine’s (NLM) article citation database.
• Medline is hosted individually on a variety of platforms (EBSCO, OVID) and comprises the majority of PubMed.
• Articles in Medline are indexed using MeSH headings. See Subject headings for more information on MeSH.

Note: PubMed contains all of Medline and additional citations, e.g. books, manuscripts, citations that predate Medline.

To ensure your search is comprehensive you may need to search beyond academic databases when conducting a systematic review, particularly to find grey literature  (literature not published commercially and outside traditional academic sources such as journals).

Google Scholar

Google Scholar contains academic resources across disciplines and sources types. These come from academic publishers, professional societies, online repositories, universities and web sites.

Use Google Scholar

• as an additional tool to locate relevant publications not included in high-level academic databases
• for finding grey literature such as postgraduate theses and conference proceedings

You can limit your search to the type of websites by using site:ac . nz; site:edu

Note that Google Scholar searches are not as replicable or transparent as academic database searches, and may find large numbers of results.

Other sources of grey literature

• Grey literature checklist  (health related grey literature)
• OpenGrey  
• Public health Ontario guide to appraising grey literature
• Institutional Repository for Information Sharing (IRIS)
• Google search: use it for finding government reports, policies, theses, etc. You can limit your search to a particular type of websites by including site : govt.nz, site: . gov, site: . ac . nz, site: . edu, in your search

Watch our Finding grey literature  video (3.49 mins) online.

• << Previous: Planning
• Next: Search techniques >>
• Last Updated: May 29, 2024 9:20 AM
• URL: https://aut.ac.nz.libguides.com/systematic_reviews

How To Do Secondary Research or a Literature Review

• Secondary Research
• Literature Review
• Step 1: Develop topic
• Step 2: Develop your search strategy
• Step 3. Document search strategy and organize results

Before You Begin: Develop Search Terms

Where do i search, pubmed tips, psycinfo tips, using search history to formulate advanced searches, what about google scholar, use a citation manager, other places to search.

• Guide License
• More Information

Paul V. Galvin Library

email: [email protected]

Chat with us:

Make a research appointment:, search our faq:.

Developing search terms starts with developing a research question. There are many ways to develop a research question, and your assignment may dictate which format to use, but the PICO framework is a very common format used in the health field.

• About PICO Format Information about forming a health-based medical question in the PICO format ( P opulation, I ntervention, C omparison/Control, O utcome)

An example research question using the PICO criteria:

Does group therapy [ I ntervention] lower the number of panic attacks per year [ O utcome] when compared to drug therapy [ C omparison/ C ontrol] for teenagers diagnosed with panic disorders [ P opulation]?

After you establish a question, you can begin developing keywords for the four PICO criteria (or if you're not using PICO, the main themes of the question), including synonyms you can think of. It can help to break this up into a chart, like the example below. In this case, it's best to break up any compound criteria such as "teenagers diagnosed with panic disorders," into distinct concepts like simply "teenagers" and "panic disorder."

For a psychology literature review, searching both PsycINFO and PubMed are your best bets. Both of these databases are very comprehensive. There will be some overlap between the two databases and some articles will appear during both searches, but you can filter for duplicates if you use a citation management program like Zotero (see later box about citation management). Searching tips for both databases follow.

How to search PubMed in a systematic way 

1. create a saved search for all of the terms that represent a concept.

The most comprehensive way to search PubMed is to create a separate but comprehensive search for each of the terms related to the concepts (step 1), then combining all of  those searches in a logical way (step 2). To do this, I would want to make one search string for all the potential terms used for each concept. I'll use the Intervention concept as an example. First, I will search for the first keyword I thought of to describe the concept, "group therapy."

Even though I searched just for "group therapy," the database interpreted my search in a different way. This is due to PubMed's algorithms. You can find the search details on the right side of the page: 

Because this is how the database interpreted my search, I'll want to copy and paste that into the upper search box. We'll look at why in a bit.

From here, I would continue like this, searching for each phrase of word that describes my concept separately. When done, go to the Advanced Search to see your history:

As you can see, the simple searches I entered have the same number of results as the more complicated, database code that I copied and pasted. The reason I did that is to preserve the actual details as run by the database. From here, I can combine the detailed searches by applying Boolean logic.  Because I want all possible terms that cover the same concept, the searches will be combined with OR. You can do this by clicking on the Add link next to searches 11 and 10, and separating by OR. Click search again and return to the search menu. There, you will have and you have one comprehensive list of all the terms related for one of your concepts (as number 12 below). 

Make sure to sign in and save this history to preserve it in case you need to make changes in the future. Click on the number of the search and choose Save to NCBI (if you don't have a free NCBI account, you will need to create one first):

Repeat this step for all of your concepts, combining each separate search into one comprehensive search for that concept and saving it.

2. Combine the concept searches logically

After you have created a comprehensive search for each concept, think about how the searches should be combined. Often, this will be simply combining all the concept searches with AND (so you get the overlap between all the terms), but not always.

For example, some researchers don't include textual searches for an age group as key terms; instead, they rely on the database filters for that. This is up to personal preference but you will likely have slightly different results depending on what you choose. 

For my sample question, I'm really researching two separate issues because I want to see both how group therapy works for panic attacks and also how drugs work, so that I can compare them. The reason I wouldn't want to search for ALL the concepts together in my example is that it's unlikely there are many articles explaining my exact issue (in other words, articles that compare the two therapies). 

To search for relevant articles about the intervention of group therapy question, I'd want to combine my comprehensive searches made for the Population, Intervention, and the Outcome concepts with AND by using the advanced search:

I click Search and end up with 184 relevant results. You should of course save this final search as well!

3. Modify as needed

If you think of other terms to include in your searches, you should modify each concept search individually and then re-combine them.

Other tips:

• Sign up for a free NCBI account. This will allow you to customize your search criteria, save searches and search history, and organize articles into folders.
• PubMed's search algorithm uses something called Automatic Term Mapping (ATM), which automatically groups phrases and searches different fields for a thorough search. This is useful for simpler searches or when you don't need to document search strategy, but for literature search documentation, it's best to look at the Search Details box, modify the search as needed, and record that as your actual search method.
• use Medical Subject Headings (MeSH) to help find targeted results. You can use MeSH terms and the corresponding subheadings to find targeted results.
• PubMed Search Strategies Blog A good place to start to find starting point strategies on a topic similar to yours.

• In addition to using keywords, take advantage of the Thesaurus, which uses indexed terms. See more info about the index terms on the PsycINFO guide .
• Create an EBSCO account. This is separate from your MyIIT login and allows you to save search history, set up search alerts, and organize research.
• Avoid using quotation marks when searching. By leaving them off, the database will automatically search for slight variations of your keywords, such as plural versions or alternate spellings. 
• Using the "Peer Reviewed" filter will limit results to only peer-reviewed journal articles. This will make up the majority of your literature, but if you'd like to also find items like books or conference proceedings, consider leaving that filter off.
• Note that your search results will vary by using Limiters, such as Age or Population Group, will provide you with different results than by using keywords for the same concept.

How to search PsycINFO in a systematic way

Similar to the PubMed strategy above, create a separate but comprehensive search for each of the terms related to the concepts (step 1), then combining all of  those searches in a logical way (step 2). To do this, I would want to make one search string for all the potential terms used for each concept. I'll use the Intervention concept as an example. First, I will search for the first keyword I thought of to describe the concept, "group therapy." PsycINFO differs from PubMed because the search you enter is the search you get (except the database will apply slight variants such as British spelling, plurals, etc.). You can choose whether to search all fields or a specific field such as the Title (See the More Tips section if you'd like to use the database subject terms).

Repeat for all of your terms:

When you are done searching for each individual term separately, click on the Search History link underneath the search boxes. From there, first clear any terms left in the boxes above and then combine your searches using the checkboxes and the "Search with OR" link.

This will create a combined search, but unlike PubMed PsycINFO does not list the details of the words used, so it can help to immediately save and rename the search to something more descriptive. To save your searches, click on the search(es) you'd like to save and click on Save Searches/Alerts. You will need to make a separate EBSCO account to do this.

Once you have created separate, comprehensive searches for each concept, then combine them following the same strategy listed above in the PubMed box but using the "Search with AND" box on the search history page.

As explained above, when conducting a comprehensive literature review, it's very important to use a systematic approach. This is especially important when submitting an article for publication, because you're often required to submit the search strategy you used. Instead of combining words/phrases into one search on the home page of the database, it's a good idea to use your Search History instead. This will help you be sure that your search terms are being combined properly and in the right order. See the links below for more background and alternative explanations: 

• PubMed Search Strategy For another explanation of searching PubMed, see this tutorial. The tutorial references "systematic reviews," but the search concept is similar to literature reviews.
• Example Search Methodology in an APA Paper Scroll to see a sample search method described in the Methodology section.
• Conducting a winning literature search Helpful article for finding health-based literature

In general, PsycINFO and PubMed are better bets to use than Google Scholar when conducting literature searches, because the search fields and algorithms are much more advancec. PsycINFO and PubMed also employ actual human indexers that review and categorize articles, whereas Google Scholar relies on keyword searching alone, so it's easier to get more complete and relevant results in PsycINFO or PubMed. It's also easier to keep track of your search strategy when using PubMed or PsycINFO.

• Zotero by IIT Galvin Library Last Updated Jun 20, 2024 2444 views this year

Using a citation manager requires a bit of a learning and adjustment period, but has a great payoff. Invest a bit of time to learn how to use one and you will benefit for the rest of your educational/professional career! Citation managers help you capture and organize references that you've found online, including the full text if available, and then help you to draft in-text citations and bibliographies. There are several available, but the library recommends Zotero if you aren't yet using a citation manager, because it is free, open-source, and very easy to use.

If you have a novel topic or one that has not yet been empirically studied extensively via research articles, you may need to supplement with dissertations, theses, or books.

• I-Share Catalog Find books from IIT and other universities here.

• << Previous: Step 3. Document search strategy and organize results
• Next: More Information >>
• Last Updated: Jun 18, 2024 4:20 PM
• URL: https://guides.library.iit.edu/litreview

University Libraries      University of Nevada, Reno

• Skill Guides
• Subject Guides

Systematic, Scoping, and Other Literature Reviews: Overview

• Project Planning

What Is a Systematic Review?

Regular literature reviews are simply summaries of the literature on a particular topic. A systematic review, however, is a comprehensive literature review conducted to answer a specific research question. Authors of a systematic review aim to find, code, appraise, and synthesize all of the previous research on their question in an unbiased and well-documented manner. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) outline the minimum amount of information that needs to be reported at the conclusion of a systematic review project. 

Other types of what are known as "evidence syntheses," such as scoping, rapid, and integrative reviews, have varying methodologies. While systematic reviews originated with and continue to be a popular publication type in medicine and other health sciences fields, more and more researchers in other disciplines are choosing to conduct evidence syntheses. 

This guide will walk you through the major steps of a systematic review and point you to key resources including Covidence, a systematic review project management tool. For help with systematic reviews and other major literature review projects, please send us an email at  [email protected] .

Getting Help with Reviews

Organization such as the Institute of Medicine recommend that you consult a librarian when conducting a systematic review. Librarians at the University of Nevada, Reno can help you:

• Understand best practices for conducting systematic reviews and other evidence syntheses in your discipline
• Choose and formulate a research question
• Decide which review type (e.g., systematic, scoping, rapid, etc.) is the best fit for your project
• Determine what to include and where to register a systematic review protocol
• Select search terms and develop a search strategy
• Identify databases and platforms to search
• Find the full text of articles and other sources
• Become familiar with free citation management (e.g., EndNote, Zotero)
• Get access to you and help using Covidence, a systematic review project management tool

Doing a Systematic Review

• Plan - This is the project planning stage. You and your team will need to develop a good research question, determine the type of review you will conduct (systematic, scoping, rapid, etc.), and establish the inclusion and exclusion criteria (e.g., you're only going to look at studies that use a certain methodology). All of this information needs to be included in your protocol. You'll also need to ensure that the project is viable - has someone already done a systematic review on this topic? Do some searches and check the various protocol registries to find out. 
• Identify - Next, a comprehensive search of the literature is undertaken to ensure all studies that meet the predetermined criteria are identified. Each research question is different, so the number and types of databases you'll search - as well as other online publication venues - will vary. Some standards and guidelines specify that certain databases (e.g., MEDLINE, EMBASE) should be searched regardless. Your subject librarian can help you select appropriate databases to search and develop search strings for each of those databases.  
• Evaluate - In this step, retrieved articles are screened and sorted using the predetermined inclusion and exclusion criteria. The risk of bias for each included study is also assessed around this time. It's best if you import search results into a citation management tool (see below) to clean up the citations and remove any duplicates. You can then use a tool like Rayyan (see below) to screen the results. You should begin by screening titles and abstracts only, and then you'll examine the full text of any remaining articles. Each study should be reviewed by a minimum of two people on the project team. 
• Collect - Each included study is coded and the quantitative or qualitative data contained in these studies is then synthesized. You'll have to either find or develop a coding strategy or form that meets your needs. 
• Explain - The synthesized results are articulated and contextualized. What do the results mean? How have they answered your research question?
• Summarize - The final report provides a complete description of the methods and results in a clear, transparent fashion. 

Adapted from

Types of reviews, systematic review.

These types of studies employ a systematic method to analyze and synthesize the results of numerous studies. "Systematic" in this case means following a strict set of steps - as outlined by entities like PRISMA and the Institute of Medicine - so as to make the review more reproducible and less biased. Consistent, thorough documentation is also key. Reviews of this type are not meant to be conducted by an individual but rather a (small) team of researchers. Systematic reviews are widely used in the health sciences, often to find a generalized conclusion from multiple evidence-based studies. 

Meta-Analysis

A systematic method that uses statistics to analyze the data from numerous studies. The researchers combine the data from studies with similar data types and analyze them as a single, expanded dataset. Meta-analyses are a type of systematic review.

Scoping Review

A scoping review employs the systematic review methodology to explore a broader topic or question rather than a specific and answerable one, as is generally the case with a systematic review. Authors of these types of reviews seek to collect and categorize the existing literature so as to identify any gaps.

Rapid Review

Rapid reviews are systematic reviews conducted under a time constraint. Researchers make use of workarounds to complete the review quickly (e.g., only looking at English-language publications), which can lead to a less thorough and more biased review. 

Narrative Review

A traditional literature review that summarizes and synthesizes the findings of numerous original research articles. The purpose and scope of narrative literature reviews vary widely and do not follow a set protocol. Most literature reviews are narrative reviews. 

Umbrella Review

Umbrella reviews are, essentially, systematic reviews of systematic reviews. These compile evidence from multiple review studies into one usable document. 

Grant, Maria J., and Andrew Booth. “A Typology of Reviews: An Analysis of 14 Review Types and Associated Methodologies.” Health Information & Libraries Journal , vol. 26, no. 2, 2009, pp. 91-108. doi: 10.1111/j.1471-1842.2009.00848.x .

• Next: Project Planning >>

Communicative Sciences and Disorders

• Online Learners: Quick Links
• ASHA Journals
• Research Tip 1: Define the Research Question
• Reference Resources
• Evidence Summaries & Clinical Guidelines
• Drug Information
• Health Data & Statistics
• Patient/Consumer Facing Materials
• Images/Streaming Video
• Database Tutorials
• Crafting a Search
• Cited Reference Searching
• Research Tip 4: Find Grey Literature
• Research Tip 5: Save Your Work
• Cite and Manage Your Sources
• Critical Appraisal
• What are Literature Reviews?
• Conducting & Reporting Systematic Reviews
• Finding Systematic Reviews
• Tutorials & Tools for Literature Reviews
• Point of Care Tools (Mobile Apps)

Choosing a Review Type

For guidance related to choosing a review type, see:

• "What Type of Review is Right for You?" - Decision Tree (PDF) This decision tree, from Cornell University Library, highlights key difference between narrative, systematic, umbrella, scoping and rapid reviews.
• Reviewing the literature: choosing a review design Noble, H., & Smith, J. (2018). Reviewing the literature: Choosing a review design. Evidence Based Nursing, 21(2), 39–41. https://doi.org/10.1136/eb-2018-102895
• What synthesis methodology should I use? A review and analysis of approaches to research synthesis Schick-Makaroff, K., MacDonald, M., Plummer, M., Burgess, J., & Neander, W. (2016). What synthesis methodology should I use? A review and analysis of approaches to research synthesis. AIMS Public Health, 3 (1), 172-215. doi:10.3934/publichealth.2016.1.172 More information less... ABSTRACT: Our purpose is to present a comprehensive overview and assessment of the main approaches to research synthesis. We use "research synthesis" as a broad overarching term to describe various approaches to combining, integrating, and synthesizing research findings.
• Right Review - Decision Support Tool Not sure of the most suitable review method? Answer a few questions and be guided to suitable knowledge synthesis methods. Updated in 2022 and featured in the Journal of Clinical Epidemiology 10.1016/j.jclinepi.2022.03.004

Types of Evidence Synthesis / Literature Reviews

Literature reviews are comprehensive summaries and syntheses of the previous research on a given topic.  While narrative reviews are common across all academic disciplines, reviews that focus on appraising and synthesizing research evidence are increasingly important in the health and social sciences.  

Most evidence synthesis methods use formal and explicit methods to identify, select and combine results from multiple studies, making evidence synthesis a form of meta-research.  

The review purpose, methods used and the results produced vary among different kinds of literature reviews; some of the common types of literature review are detailed below.

Common Types of Literature Reviews 1

Narrative (literature) review.

• A broad term referring to reviews with a wide scope and non-standardized methodology
• Search strategies, comprehensiveness of literature search, time range covered and method of synthesis will vary and do not follow an established protocol

Integrative Review

• A type of literature review based on a systematic, structured literature search
• Often has a broadly defined purpose or review question
• Seeks to generate or refine and theory or hypothesis and/or develop a holistic understanding of a topic of interest
• Relies on diverse sources of data (e.g. empirical, theoretical or methodological literature; qualitative or quantitative studies)

Systematic Review

• Systematically and transparently collects and categorize existing evidence on a question of scientific, policy or management importance
• Follows a research protocol that is established a priori
• Some sub-types of systematic reviews include: SRs of intervention effectiveness, diagnosis, prognosis, etiology, qualitative evidence, economic evidence, and more.
• Time-intensive and often takes months to a year or more to complete 
• The most commonly referred to type of evidence synthesis; sometimes confused as a blanket term for other types of reviews

Meta-Analysis

• Statistical technique for combining the findings from disparate quantitative studies
• Uses statistical methods to objectively evaluate, synthesize, and summarize results
• Often conducted as part of a systematic review

Scoping Review

• Systematically and transparently collects and categorizes existing evidence on a broad question of scientific, policy or management importance
• Seeks to identify research gaps, identify key concepts and characteristics of the literature and/or examine how research is conducted on a topic of interest
• Useful when the complexity or heterogeneity of the body of literature does not lend itself to a precise systematic review
• Useful if authors do not have a single, precise review question
• May critically evaluate existing evidence, but does not attempt to synthesize the results in the way a systematic review would 
• May take longer than a systematic review

Rapid Review

• Applies a systematic review methodology within a time-constrained setting
• Employs methodological "shortcuts" (e.g., limiting search terms and the scope of the literature search), at the risk of introducing bias
• Useful for addressing issues requiring quick decisions, such as developing policy recommendations

Umbrella Review

• Reviews other systematic reviews on a topic
• Often defines a broader question than is typical of a traditional systematic review
• Most useful when there are competing interventions to consider

1. Adapted from:

Eldermire, E. (2021, November 15). A guide to evidence synthesis: Types of evidence synthesis. Cornell University LibGuides. https://guides.library.cornell.edu/evidence-synthesis/types

Nolfi, D. (2021, October 6). Integrative Review: Systematic vs. Scoping vs. Integrative. Duquesne University LibGuides. https://guides.library.duq.edu/c.php?g=1055475&p=7725920

Delaney, L. (2021, November 24). Systematic reviews: Other review types. UniSA LibGuides. https://guides.library.unisa.edu.au/SystematicReviews/OtherReviewTypes

Further Reading: Exploring Different Types of Literature Reviews

• A typology of reviews: An analysis of 14 review types and associated methodologies Grant, M. J., & Booth, A. (2009). A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information and Libraries Journal, 26 (2), 91-108. doi:10.1111/j.1471-1842.2009.00848.x More information less... ABSTRACT: The expansion of evidence-based practice across sectors has lead to an increasing variety of review types. However, the diversity of terminology used means that the full potential of these review types may be lost amongst a confusion of indistinct and misapplied terms. The objective of this study is to provide descriptive insight into the most common types of reviews, with illustrative examples from health and health information domains.
• Clarifying differences between review designs and methods Gough, D., Thomas, J., & Oliver, S. (2012). Clarifying differences between review designs and methods. Systematic Reviews, 1 , 28. doi:10.1186/2046-4053-1-28 More information less... ABSTRACT: This paper argues that the current proliferation of types of systematic reviews creates challenges for the terminology for describing such reviews....It is therefore proposed that the most useful strategy for the field is to develop terminology for the main dimensions of variation.
• Are we talking the same paradigm? Considering methodological choices in health education systematic review Gordon, M. (2016). Are we talking the same paradigm? Considering methodological choices in health education systematic review. Medical Teacher, 38 (7), 746-750. doi:10.3109/0142159X.2016.1147536 More information less... ABSTRACT: Key items discussed are the positivist synthesis methods meta-analysis and content analysis to address questions in the form of "whether and what" education is effective. These can be juxtaposed with the constructivist aligned thematic analysis and meta-ethnography to address questions in the form of "why." The concept of the realist review is also considered. It is proposed that authors of such work should describe their research alignment and the link between question, alignment and evidence synthesis method selected.
• Meeting the review family: Exploring review types and associated information retrieval requirements Sutton, A., Clowes, M., Preston, L., & Booth, A. (2019). Meeting the review family: Exploring review types and associated information retrieval requirements. Health Information & Libraries Journal, 36(3), 202–222. doi: 10.1111/hir.12276

Integrative Reviews

"The integrative review method is an approach that allows for the inclusion of diverse methodologies (i.e. experimental and non-experimental research)." (Whittemore & Knafl, 2005, p. 547).

• The integrative review: Updated methodology Whittemore, R., & Knafl, K. (2005). The integrative review: Updated methodology. Journal of Advanced Nursing, 52 (5), 546–553. doi:10.1111/j.1365-2648.2005.03621.x More information less... ABSTRACT: The aim of this paper is to distinguish the integrative review method from other review methods and to propose methodological strategies specific to the integrative review method to enhance the rigour of the process....An integrative review is a specific review method that summarizes past empirical or theoretical literature to provide a more comprehensive understanding of a particular phenomenon or healthcare problem....Well-done integrative reviews present the state of the science, contribute to theory development, and have direct applicability to practice and policy.

• Conducting integrative reviews: A guide for novice nursing researchers Dhollande, S., Taylor, A., Meyer, S., & Scott, M. (2021). Conducting integrative reviews: A guide for novice nursing researchers. Journal of Research in Nursing, 26(5), 427–438. https://doi.org/10.1177/1744987121997907
• Rigour in integrative reviews Whittemore, R. (2007). Rigour in integrative reviews. In C. Webb & B. Roe (Eds.), Reviewing Research Evidence for Nursing Practice (pp. 149–156). John Wiley & Sons, Ltd. https://doi.org/10.1002/9780470692127.ch11

Scoping Reviews

Scoping reviews are evidence syntheses that are conducted systematically, but begin with a broader scope of question than traditional systematic reviews, allowing the research to 'map' the relevant literature on a given topic.

• Scoping studies: Towards a methodological framework Arksey, H., & O'Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8 (1), 19-32. doi:10.1080/1364557032000119616 More information less... ABSTRACT: We distinguish between different types of scoping studies and indicate where these stand in relation to full systematic reviews. We outline a framework for conducting a scoping study based on our recent experiences of reviewing the literature on services for carers for people with mental health problems.
• Scoping studies: Advancing the methodology Levac, D., Colquhoun, H., & O'Brien, K. K. (2010). Scoping studies: Advancing the methodology. Implementation Science, 5 (1), 69. doi:10.1186/1748-5908-5-69 More information less... ABSTRACT: We build upon our experiences conducting three scoping studies using the Arksey and O'Malley methodology to propose recommendations that clarify and enhance each stage of the framework.
• Methodology for JBI scoping reviews Peters, M. D. J., Godfrey, C. M., McInerney, P., Baldini Soares, C., Khalil, H., & Parker, D. (2015). The Joanna Briggs Institute reviewers’ manual: Methodology for JBI scoping reviews [PDF]. Retrieved from The Joanna Briggs Institute website: http://joannabriggs.org/assets/docs/sumari/Reviewers-Manual_Methodology-for-JBI-Scoping-Reviews_2015_v2.pdf More information less... ABSTRACT: Unlike other reviews that address relatively precise questions, such as a systematic review of the effectiveness of a particular intervention based on a precise set of outcomes, scoping reviews can be used to map the key concepts underpinning a research area as well as to clarify working definitions, and/or the conceptual boundaries of a topic. A scoping review may focus on one of these aims or all of them as a set.

Systematic vs. Scoping Reviews: What's the Difference? 

YouTube Video 4 minutes, 45 seconds

Rapid Reviews

Rapid reviews are systematic reviews that are undertaken under a tighter timeframe than traditional systematic reviews. 

• Evidence summaries: The evolution of a rapid review approach Khangura, S., Konnyu, K., Cushman, R., Grimshaw, J., & Moher, D. (2012). Evidence summaries: The evolution of a rapid review approach. Systematic Reviews, 1 (1), 10. doi:10.1186/2046-4053-1-10 More information less... ABSTRACT: Rapid reviews have emerged as a streamlined approach to synthesizing evidence - typically for informing emergent decisions faced by decision makers in health care settings. Although there is growing use of rapid review "methods," and proliferation of rapid review products, there is a dearth of published literature on rapid review methodology. This paper outlines our experience with rapidly producing, publishing and disseminating evidence summaries in the context of our Knowledge to Action (KTA) research program.
• What is a rapid review? A methodological exploration of rapid reviews in Health Technology Assessments Harker, J., & Kleijnen, J. (2012). What is a rapid review? A methodological exploration of rapid reviews in Health Technology Assessments. International Journal of Evidence‐Based Healthcare, 10 (4), 397-410. doi:10.1111/j.1744-1609.2012.00290.x More information less... ABSTRACT: In recent years, there has been an emergence of "rapid reviews" within Health Technology Assessments; however, there is no known published guidance or agreed methodology within recognised systematic review or Health Technology Assessment guidelines. In order to answer the research question "What is a rapid review and is methodology consistent in rapid reviews of Health Technology Assessments?", a study was undertaken in a sample of rapid review Health Technology Assessments from the Health Technology Assessment database within the Cochrane Library and other specialised Health Technology Assessment databases to investigate similarities and/or differences in rapid review methodology utilised.
• Rapid Review Guidebook Dobbins, M. (2017). Rapid review guidebook. Hamilton, ON: National Collaborating Centre for Methods and Tools.
• NCCMT Summary and Tool for Dobbins' Rapid Review Guidebook National Collaborating Centre for Methods and Tools. (2017). Rapid review guidebook. Hamilton, ON: McMaster University. Retrieved from http://www.nccmt.ca/knowledge-repositories/search/308
• << Previous: Literature Reviews
• Next: Conducting & Reporting Systematic Reviews >>
• Last Updated: Jun 26, 2024 3:00 PM
• URL: https://guides.nyu.edu/speech

• Systematic Review
• Open access
• Published: 27 June 2024

A systematic review and narrative synthesis of health literacy interventions among Spanish speaking populations in the United States

• Joel Hernandez 4 ,
• Liliana Demiranda 1 ,
• Priyanka Perisetla 1 ,
• Lauren Andrews 1 ,
• Keer Zhang 1 , 5 ,
• Rebecca Henderson 6 ,
• Ajay Mittal 1 ,
• Hannah F. Norton 3 &
• Melanie G. Hagen 1 , 2  

BMC Public Health volume  24 , Article number:  1713 ( 2024 ) Cite this article

Metrics details

While many populations struggle with health literacy, those who speak Spanish preferentially or exclusively, including Hispanic, immigrant, or migrant populations, may face particular barriers, as they navigate a predominantly English-language healthcare system. This population also faces greater morbidity and mortality from treatable chronic diseases, such as hypertension and diabetes. The aim of this systematic review was to describe existing health literacy interventions for patients with a Spanish-language preference and present their effectiveness.

We carried out a systematic review where Web of Science, EMBASE, and PubMed were queried using MeSH terms to identify relevant literature. Included articles described patients with a Spanish-language preference participating in interventions to improve health literacy levels in the United States. Screening and data abstraction were conducted independently and in pairs. Risk of bias assessments were conducted using validated appraisal tools.

A total of 2823 studies were identified, of which 62 met our eligibility criteria. The studies took place in a variety of community and clinical settings and used varied tools for measuring health literacy. Of the interventions, 28 consisted of in-person education and 27 implemented multimedia education, with 89% of studies in each category finding significant results. The remaining seven studies featured multimodal interventions, all of which achieved significant results.

Successful strategies included the addition of liaison roles, such as promotores (Hispanic community health workers), and the use of multimedia fotonovelas (photo comics) with linguistic and cultural adaptations. In some cases, the external validity of the results was limited. Improving low health literacy in patients with a Spanish-language preference, a population with existing barriers to high quality of care, may help them better navigate health infrastructure and make informed decisions regarding their health.

Registration

PROSPERO (available at https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42021257655.t ).

Peer Review reports

Introduction

While health literacy (HL) is a multifaceted concept [ 1 , 2 ] almost all definitions relate HL to “the literacy and numeracy skills that enable individuals to obtain, understand, appraise, and use information to make decisions and take actions that will have an impact on health status” [ 3 ]. Low HL has been linked to poorer health outcomes, including increased mortality [ 4 , 5 ]. HL has increasingly been recognized as a potentially important factor mediating health disparities, especially those related to race and ethnicity [ 5 ], and has been suggested as an important mediator of the relationship between socioeconomic status and health [ 6 ]. This may be due to communication barriers with physicians and difficulty understanding and making use of medical resources [ 5 ].

As a concept, HL has sometimes been poorly defined. A recent systematic review which sought to clarify the concept found that scholars commonly characterized HL along three main domains: knowledge of health/healthcare systems, processing and using information related to health and healthcare, and the ability to maintain health through collaboration with health providers [ 7 ]. Other theoretical frameworks developed for HL understand the concept through its effects. For example Nutbeam established a useful framework for understanding the benefits of health literacy through a “health outcomes model” in which HL is comprised of functional HL, the basic skills necessary for everyday health functioning, communicative/interactive HL, the more advanced skills needed to act independently with “motivation and self-confidence,” and critical HL, the ability to analyze and use information to “exert greater control over life events and situations” allowing people to respond adversity and to advocate for themselves [ 8 , 9 ]. HL is sometimes understood as not only a skill, but an important social determinant of health, with community level and public health implications [ 10 ].

While many U.S. residents struggle with limited health literacy, there may be a particular barrier among those who speak Spanish preferentially or exclusively, including Hispanic, immigrant or migrant populations. In the United States, minority groups, immigrants, migrants, and nonnative English speakers have lower health literacy scores than White adults and are at higher risk of having poor HL, making them more susceptible to the adverse outcomes associated with low HL [ 11 ]. Hispanics are the largest group of nonnative English speakers and preferential Spanish speakers in the U.S. and have low rates of HL compared to other populations [ 5 ]. Limited English proficiency may be a factor that contributes to poorer health outcomes and reduced quality of care, especially in a predominantly English language-based health care system with a shortage of bilingual and culturally competent providers [ 12 ]. For example, one recent study found higher rates of obesity among Spanish speakers in the United States [ 13 ]. These factors, in combination with a lack of healthcare access and insurance coverage, may contribute to higher morbidity and mortality rates among Hispanics due to chronic diseases such as diabetes and obesity [ 14 ].

Methods to accommodate the HL needs of patients with a Spanish-language preference (SLP) may therefore be important in improving health equity [ 15 ]. While strides have been made in community-based educational efforts and the translation or cultural adaptation of health communication tools and processes [ 16 ], there are limited data on effective interventions to improve HL for patients with SLP in the United States [ 17 ]. The literature on interventions targeting HL in the United States has frequently grouped together populations of immigrants who do not share a common language [ 18 ] or, conversely, focused only on individuals from a single nationality [ 19 , 20 ]. Given the gap in the literature synthesizing research on HL interventions for patients with SLP in the United States and the important association between HL and health outcomes, we conducted a systematic review of the literature that summarizes and evaluates the effectiveness of HL intervention strategies for patients with SLP in the United States. The aim of this systematic review was to describe existing HL interventions for patients with SLP and present their reported effectiveness.

Protocol and registration

The protocol for this review was registered with PROSPERO (CRD42021257655 ) . The use of the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) enabled authors to follow best practices in conducting the review [ 21 ].

Search strategy and screening

Searches were conducted in the PubMed, MEDLINE, Web of Science, and Embase databases and data was extracted from these databases between January 20, 2020 and April 27, 2023 (Fig.  1 .). The keywords for each database included: “health literacy” and “intervention” or “Spanish”, “Hispanic” or “LEP,” or “limited English proficiency.” Databases were queried to include only articles published between January 1, 2011 and April 27, 2023. In 2010, the U.S. The Department of Health and Human Services unveiled the National Action Plan to Improve Health Literacy, bringing more attention to this matter and inspiring more research on HL. Our review also avoids redundancy with a 2011 comprehensive review [ 5 ], which found no interventions focused on HL in Spanish-speaking populations, with only three mentioned measures of HL in this population.

PRISMA flow diagram for studies considered for the systematic review

After removing duplicates, two reviewers, P.P. and L.D., independently reviewed titles and abstracts to select potentially eligible articles based on the inclusion/exclusion criteria described below. Any disagreements regarding the inclusion of a study were resolved by a third reviewer, J.H. Bibliographies of included studies were subsequently hand searched.

Inclusion & exclusion criteria

Inclusion criteria for this literature review included articles that a) featured participants with SLP, b) described interventions that occurred in the United States, c) described interventions that were designed to mitigate the effects of low HL in participants with SLP and improve the use of health services or the health outcomes in these populations, d) were shared in an online format in indexed scientific journals, e) were written in English or Spanish, f) were published in 2011–2023, g) were randomized control trials (RCTs), pre/post (PP) studies, prospective cohort (PC) studies, cross-sectional (CC) studies, or mixed methods studies and h) measured effectiveness of intervention using HL assessment tools or health outcomes.

Exclusion criteria included studies of outcomes related to numeracy or literacy alone without reference to HL because such interventions were found to differ from those that dealt with these issues in the context of HL. We also excluded studies that did not report HL interventions targeting Spanish-speakers in the United States.

Assessment of methodology quality

We assessed the methodological quality of each included study using the Revised Tool to Assess Risk of Bias in Randomized Trials (RoB 2) [ 22 ] and the Risk Of Bias In Non-randomized Studies—of Interventions (ROBINS-I) tool for assessing risk of bias in the different interventions analyzed (RCTs, PP studies, PC studies, CC studies, and mixed methods studies) [ 23 ]. Two review authors (P.P. and L.D.) independently performed quality assessments. Disagreements regarding the overall assessments were resolved through discussion, with a third reviewer as the final arbitrator (J.H.). Bibliographies of included studies were subsequently hand searched.

Data synthesis

After piloting, four reviewers (J.H., L.A., P.P., L.D.) conducted data extraction using a standardized data extraction template (Appendix 1 ). Due to the heterogeneity of interventions, outcomes assessed, and varying durations of interventions, we did not pool the data and instead conducted a narrative analysis. We conducted a thematic analysis of identified studies and grouped studies for synthesis on the basis of identified categories. This process consisted of iterative discussions of the studies by all members of the study team and was based on published guidelines for Synthesis without Metanalysis (SWiM) [ 24 ]. Our data synthesis specifically grouped studies based on the categories of study characteristics, measures of effectiveness, reported effectiveness by intervention type, and quality assessment. We stratified the results by intervention type. While we did not focus on migrant status specifically, this could be estimated by one of our data extraction items, country of origin.

Study characteristics

After removal of duplicates, 2,823 titles and abstracts were screened for inclusion using the criteria described above. A manual search of bibliographies yielded eight additional articles for screening. A total of 121 potentially relevant articles were selected using the inclusion criteria described above. After a detailed full-text analysis of each study, 62 studies were included, and 59 were excluded, as indicated in Fig.  1 . This included 17 RCTs, 35 PP studies, 3 PC studies, 3 CC studies, and 4 mixed methods studies. A summary of the study characteristics can be found in Table  1 , 2 , and 3 . The studies encompassed mainly female, middle-aged adults (range: 30 to 50); only two studies included participants under the age of 18 [ 25 , 26 ] and no studies were focused solely on pediatric populations. Only a minority of participants had graduated from college. Sample sizes varied from 10 to 943. Interventions included in-person education ( n  = 28), multimedia education ( n  = 27) and other types of multimodal strategies ( n  = 7). Eighteen studies made use of lay health advisors and promotores .

Topics included prenatal care and parent education; breast, cervical, colorectal, and ovarian cancer; diet and healthy lifestyle choices; mental health literacy; diabetes; cardiovascular disease; end-stage renal disease; asthma; upper respiratory infections; inflammatory bowel disease; HIV/AIDS; skin care; hearing loss prevention; medication understanding; palliative care; family health history; chronic pain; healthcare navigation; and anesthesia education. Thirty-four studies employed a theoretical framework when designing and conducting research, and there was little heterogeneity in terms of frameworks employed. No framework was shared by more than four studies.

Studies were performed in a variety of settings, including clinics ( n  = 13), hospitals and health centers ( n  = 13), Federally Qualified Health Centers (FQHCs) or safety net clinics ( n  = 9) and community spaces ( n  = 18). Common community settings, which include community health centers and safety net clinics, frequently used curricular interventions embedded in educational curricula and educational workshops ( n  = 18). Larger hospital networks implemented organizational interventions, often updating their practices or replacing standard-of-care materials with language and culturally concordant materials ( n  = 8).

Measures of effectiveness

The measures of successful enhancement of HL used by the studies in our review were heterogenous, and were often unvalidated measures of knowledge or beliefs. Twenty-two studies had a questionnaire about beliefs, knowledge or practice that was developed by the researchers, limiting the validity of their results. Fifty-eight studies measured effectiveness quantitatively, and four were mixed methods. The two most common approaches to primary outcomes were either HL assessment tools [ 16 , 17 , 25 , 25 , 26 , 28 , 30 , 31 , 32 , 34 , 35 , 36 , 37 , 38 , 41 , 47 , 49 , 50 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 63 , 64 , 65 , 66 , 67 , 68 , 69 , 70 , 71 , 72 , 74 , 75 , 76 , 78 , 79 , 80 , 81 , 82 , 83 ] ( n  = 45) or health outcomes [ 27 , 29 , 39 , 42 , 48 , 51 , 52 ] ( n  = 7), with some studies using both [ 15 , 43 , 46 , 84 , 85 , 86 , 87 , 88 , 89 , 90 ] ( n  = 10). HL tools most commonly took the form of pretest/posttest questionnaires specifically developed by the researchers to assess knowledge gained over the course of a given intervention. A few studies ( n  = 10) utilized previously validated disease-specific assessments of HL, such as the High Blood Pressure-Health Literacy Scale for high blood pressure [ 32 ], or more standardized Test of Functional Health Literacy in Adults (TOFHLA) [ 31 , 37 , 38 , 49 , 50 , 52 , 55 , 60 , 68 , 73 ] ( n  = 10) and/or Newest Vital Sign (NVS) [ 28 , 43 , 51 , 52 , 68 , 76 ] ( n  = 6), to assess overall changes based on the participant’s ability to read and understand generic health-related materials.

Other outcome measures included patient satisfaction and patient attitude surveys, which were intended to predict not only knowledge of health conditions but also attitudes toward receiving treatment [ 84 ]. Higher satisfaction and improved attitude scores were thought to lead to a more positive and confident approach in obtaining healthcare. Some studies measured improvements in confidence and self-advocacy [ 31 , 34 , 70 ]. Medical health measurements and outcomes, such as blood pressure readings, were also commonly used as primary outcomes [ 27 , 46 , 48 , 74 ]. Secondary measures were also varied and included measures of patient confidence, perceived support, perceived barriers to care, level of comfort, and adherence to the intervention.

Studies also varied in how they measured the long-term changes associated with their interventions. Thirty-one studies had a follow up of at least a month, ranging from 1 to 24 months, with most studies doing a 1 month follow up ( n  = 7) or a 3 month follow up ( n  = 8).

Overview of health literacy interventions

In-person education.

In-person education health literacy programs varied in presentation of material but shared commonalities of repeated meetings in a class setting that encouraged practice and facilitated opportunities for enhanced participant engagement compared to other modalities (Table  1 ). A study by Cruz [ 30 ], found the use of 90 min training session conducted by promotores focusing on general knowledge for diabetes, risk factors, and prevention and control of diabetes provided significant improvement on diabetes knowledge for diabetic participants comparing pre- and posttest scores (13.7 vs. 18.6, p  < 0.001; Cohen’s d = 1.2), and for nondiabetic participants (12.9 vs. 18.2, p  < 0.001; Cohen’s d = 1.2).

Similarly, Buckley [ 27 ] assessed the implementation of social clubs hosted by navegantes (patient navigators) for 2 h every week over 5 weeks. The findings suggested 88.9% of 126 participants increased health literacy and over 60% decreased at least one risk factor associated with metabolic syndrome. Change for those that improved, [mean (SD)]: Weight [− 6.0 lbs (5.2)]; BMI [− 1.1 (1.0)]; Waist Circumference [− 2.2 inches (1.5)]; Blood Glucose [− 26.3 mg/dl (27.5)]; LDL Cholesterol [− 19.1 mg/dl (16.8)]; Systolic BP [− 11.1 mmHg (9.5)]; Health Literacy Test ( n  = 117) [+ 22.2% (19.7%)]. Castaneda [ 28 ] studied the implementation of 6-week, culturally tailored, promotora -based group for health prevention knowledge and found participants improved their self-reported cancer screening, breast cancer knowledge (Mpre = 2.64, Mpost = 3.02), daily fruit and vegetable intake, and ability to read a nutrition label ( p  < 0.05).

Across all the different in-person education there were common findings that repeated exposure to health education information in an engaging classroom setting provided meaningful improvements to health literacy in SLP populations that correlated with improvements in physical health and greater utilization of health screening services.

Multimedia education

Multimedia approaches to health literacy education varied from narrative films and fotonovelas to animated culturally sensitive videos and virtual workshops to assess applied knowledge (Table  2 ). The commonality shared with these interventions were that they could largely be independently navigated without need for transportation or cost to the participant as long as they had access to a computer and the internet.

A study related to health literacy in women’s health, Borrayo [ 53 ] found that through a 8-min narrative film to reinforce desired self-efficacy and behavioral intentions as precursors to engaging in mammography screening there was a significant increase in breast cancer knowledge ( Wilks’s Λ = 0.75, F(1, 39) = 13.15, p  < 0.001, η2 = 0.25) and mammography self-efficacy ( Wilks’s Λ = 0.76, F(1, 37) = 11.64, p  < 0.01, η2 = 0.24) compared to baseline and control group. Furthermore, Cabassa [ 54 ] assessed the use of a fotonovela centered around entertainment-education intervention toward mental health stigma finding a significant increase in depression treatment knowledge scores at posttest ( B = 1.22, p  < 0.001, Cohen’s d = 0.91) and 1-month follow-up ( B = 0.81, p  < 0.01, Cohen’s d = 0.53). Calderon [ 55 ] looked at the implementation of an animated, culturally sensitive, Spanish video to improve diabetes health literacy (DHL). The findings reported DHL survey scores improved significantly more in the experimental group than the control group (adjusted mean = 55% vs 53%, F = 4.7, df = 1, p  = 0.03). Additionally, Cheney [ 56 ] studied the application of tailor MyPlate recipes to local food sources and culture, virtual cooking demonstrations, and Spanish cookbook, on diabetes education finding there was an increased confidence in adherence to two of four components of the Mediterranean diet (b added sugar  = 0.24; 95%CI: 0.02, 0.46; b redmeat  = 0.5; 95% CI: 0.02, 0.98).

Other types of multimodal strategies

Multimodal strategies provided a crossover between in-person and multi-media focused health literacy approaches (Table  3 ). A study by Auger [ 15 ], found the use of fotonovelas as an educational tool along with health education facilitation by the teacher and lay health educator provided an increased knowledge of pregnancy, childbirth, and breastfeeding ( p  < 0.001) and confidence in navigating pregnancy, caring for oneself and the baby, and interacting with health professionals ( p  ≤ 0.05).

Additionally, Calderon [ 78 ] took a multimodal approach to mental health education via workshops including a short video on possible psychotic and depressive symptoms, La CLAve mnemonic device to describe the main symptoms of psychosis, and a narrative film to discuss its portrayal of symptoms. That study demonstrated a significant increase in psychotic symptoms reported as definition of serious mental illness (pre, M = 0.69, SD = 0.61; post, M = 1.23, SD = 0.90, t(80) = − 5.64; p  < 0.001; Cohen's d = 0.70) and ability to detect a serious mental illness in others (pretraining: M = 2.83, SD = 1.31; posttraining: M = 3.24, SD = 1.27, t(74) = − 2.76, p  < 0.05; Cohen's d = 0.32), and decrease in participants' recommendations for nonprofessional help-seeking (pre: 49.4%, post: 25.9%, N  = 81, p  = 0.001). There was no significant change in recommendations for professional help (pre: 64.2%, post: 72.8%, N  = 81, p  = 0.25).

Reported effectiveness by intervention type

Of the interventions, 89% of in-person educational interventions ( n  = 25) and 89% of multimedia educational interventions ( n  = 24) found improvements to HL. All multimodal interventions ( n  = 7) provided improvements in HL. The use of lay health advisors and promotores was correlated with increased effectiveness; all 18 studies that used this technique reported that their interventions had caused statistically significant changes in HL [ 27 , 28 , 34 , 46 , 70 ]. Similarly, all nine of the studies implementing fotonovela strategies reported statistically significant improvements in HL [ 16 , 25 , 53 , 54 , 61 , 63 , 74 ].

Quality assessment

The risk of bias assessment for RCTs evaluated risks due to randomization, outcomes, and result reporting (Table  4 ). Among RCTs ( n  = 17), one was assessed as having a high risk of bias, and eight were assessed as having some concerns. Non-RCTs were likewise evaluated for risk of bias due to problems with recruitment, confounding factors, missing data, and selective measurement of outcome or result reporting (Table  5 ). Among non-RCTs ( n  = 36), 14 studies had a serious risk of bias, while the remaining 22 studies had a moderate risk of bias.

To the best of our knowledge, this review is the first to systematically describe and evaluate the effectiveness of HL interventions among patients with SLP in the United States. Recent reviews have studied the impact of different intervention strategies for increasing the HL of the general population [ 85 ] and for immigrant communities [ 18 ] but have not focused on Spanish speakers – a community largely at risk for low HL and poor health outcomes [ 5 , 11 , 17 ].

Our review found that, as with other populations with a non-English language preference, including migrant populations [ 86 ], there is a lack of evidence-based specific interventions to raise HL tailored to U.S. patients with SLP. Further, our review found that the few existing studies may be at risk of bias. The high risk of bias we found especially in non-RCTs on this topic likely represents both the lack of attention to research addressing this need in SLP populations, as well as difficulties inherent in testing and measuring interventions aimed at improving HL more broadly. Our review of quality was in line with other reviews on this topic [ 18 , 87 ] which found that a risk of bias was introduced primarily due to difficulty blinding participants and moderators due to the nature of study designs. This made RCTs more difficult to conduct, and as a result, studies primarily used pretest/posttest and cross-sectional methodologies. This finding points not only to a need for high-quality studies of HL in this population, but also for the potential to critically rethink how to conduct research on HL in a high-quality, low risk of bias way. Additionally, studies reported sample sizes ranging from 10 to 943 participants, which made it difficult to compare effect sizes directly. This variation likely reflects the dissimilarity of study designs, sample populations and setting types, thus making it difficult to compare across studies, a challenge that has been previously acknowledged for reviews of HL.

There are also significant differences in patient populations across reviews, and many studies had a low number of participants. This small sample size was in some cases due to strict study inclusion criteria, and other cases were due to high rates of attrition. This could be partly due to primarily targeting participants already facing cultural, socioeconomic, and educational barriers, making them more difficult to recruit and retain in research. Many studies have indicated that their sample may not be representative due to sampling methodologies or that there may be limited generalizability of results. This was due in part to convenience sampling or small sample sizes, which made it difficult to determine whether findings represented a true effect due to limitations in statistical power.

Some studies were focused on only one research site and/or a highly specific Hispanic immigrant community with a SLP (i.e., Mexican immigrants [ 52 ]), limiting generalizability. At the same time, while we attempted to capture the difference between the broader category of Spanish speaking populations in the United States and specific migrant populations, most studies did not include this sort of information, indicating a potential need for studies that focus on specific SLP migrant communities. No studies addressed pediatric populations. Another factor limiting the generalizability of the reviewed studies was that the majority of study participants were women; this may be tied to a wider lack of healthcare utilization among Hispanic men, including those with a SLP [ 88 , 89 , 90 ]. The relative paucity of males in the sample population of the studies may indicate a need for research that focuses on men with a SLP. To date, only a few strategies have been developed to include males with a SLP in research, including the use of male community health workers and health outreach in workplaces and providing public transportation [ 41 ]. Finally, the studies reviewed included a predominantly adult to middle-aged population (aged 30–50) rather than older adults who are more at risk for serious medical problems. This suggests that several important populations (men, children, older adults) may be missed by most previous HL interventions in populations with SLP.

The varied, poorly standardized, heterogenous measures used to assess HL in reviewed studies demonstrate that HL as a concept is poorly defined by researchers, and the concept likely encompasses more than can be quantified by numeric scores on standardized assessments of knowledge. For example, in assessing HL, there may also be a need to address the ability of patients to advocate for themselves, ask questions, and feel empowered to change their health behaviors [ 8 , 9 , 16 , 74 , 85 ]. Existing measures of HL may not fully capture HL concepts, and thus may be a poor proxy of effectiveness. Studies in our review often used measures that were not validated and tested knowledge on a specific health topic or reported beliefs about health as proxies for HL, and relatively few measured direct behavioral changes, attempts at communication self-efficacy or advocacy, or effects on health outcomes. A key takeaway of this review is the need to critically reexamine definitions and measures of HL, and to develop and validate improved qualitative and quantitative measures of HL outcomes. Only about half of the studies used a theoretical framework to inform their intervention or research, and studies rarely employed the same frameworks, perhaps partially accounting for the variety of measures and the limitations in the ways that HL was framed by researchers.

We found that studies of HL among people with SLP in the United States therefore followed trends within the literature, in which HL is measured through knowledge of health/healthcare systems; to a lesser extent, studies included in our review also attempted to measure participants’ use of information related to health and healthcare, and their ability to maintain health through collaboration with health providers [ 7 ]. When framed in terms of Nutbeam’s health outcomes approach, the studies mostly attempted to measure functional HL, occasionally addressed communicative/interactive HL, and rarely attempted to address critical HL [ 8 , 9 ]. This focus on HL as knowledge rather than personal health advocacy has important ramifications in terms of the skills that HL interventions focus on building, and may help to explain the success and failure of HL interventions.

In addition to the importance of improving individual’s health literacy there is support in the literature to improve “organizational health literacy.” Organizational health literacy refers to the responsibility for health care systems to address populations with low health literacy [ 91 ]. Methods for organizations to address populations with low health literacy include “reducing the complexity of health care; increasing patient understanding of health information and enhancing supports for patients of all levels of health literacy” [ 91 ]. Because limited health literacy has been associated with increased cost of healthcare organizations have an incentive to address health literacy. However, few if any of the studies attempted to address organizational health literacy, and placed the onus for building HL on the individual patient and their family.

Specific recommendations

Successful interventions focused on HL interventions that targeted SLP populations through linguistically and culturally concordant techniques that utilized community member liaisons and culturally relevant storytelling. Successful interventions were also often well integrated within communities and organizations.

Our review found that interventions utilizing cultural and linguistic concordance (ie. Spanish-language, culturally salient concepts/terms), liaison roles ( promotores ), and narrative media were effective in achieving notable improvements in HL among patients with SLP. These interventions focused more on what Nutbeam frames as communicative/interactive HL [ 8 , 9 ]. The relative success of these interventions may be due to more effective communication with patients through a shared cultural background and deeper levels of trust. One particularly effective strategy is the use of narrative in media, as seen with fotonovela strategies [ 16 , 25 , 53 , 54 , 61 , 63 , 74 ]. Such strategies may involve a video or booklet presenting important health information in a story format. Narrative media appeared to activate study participants and lead to improvements in health knowledge and behavior change. Another important element of effective multimedia health interventions is cultural adaptation to address previously identified cultural concepts such as respeto , familismo , marianismo , and personalismo [ 16 , 25 , 55 , 62 , 65 , 68 , 73 , 76 ]. Realistic stories with Spanish-speaking characters and culturally tailored information were key components of these interventions [ 25 , 54 , 55 , 74 ]. Prior research has shown that identification with storytellers is an important prerequisite for patient engagement and is particularly useful in combating cultural stigma and eliciting behavioral health changes [ 92 ].

Liaison roles that employ educators and health promoters from similar cultural backgrounds as patients were also an important strategy used by reviewed studies. The lived experience and cultural understanding from these workers ( promotores , navegantes , community health workers) may help boost patient comprehension and overcome distrust of the healthcare system [ 93 ]. Linguistic and culturally concordant care, including cultural competency training for providers, has also repeatedly been identified as a successful strategy for increasing HL among immigrant populations generally [ 18 ]. Furthermore, successful interventions often consider the opinions of the target population when designing content to ensure that the experience is culturally relevant [ 16 , 28 , 29 , 34 , 35 , 40 , 44 , 45 , 55 , 62 , 65 , 73 , 76 ].

Our review also included a number of multimedia intervention strategies ( n  = 22) that might be utilized more often in the future following the increased acceptance of online options since the COVID-19 pandemic. Interestingly, of the 14 studies published since 2020, seven were multimedia interventions. Our search also revealed the importance of including nonmedical settings such as community gathering spaces, which may serve as a hub for creating a wider network of health promotion. The integration of health promotion interventions into communities may be complimentary to the long-term reinforcement of health education, serving as a means of achieving sustained outcomes.

Other elements of successful HL interventions may include finding a fit between factors such as intervention type, size and type of setting, duration of time available, and level of community integration (Fig.  2 ). As described above, HL may be framed as organizational as well as individual, and successful interventions better integrate organizational setting into the structure of the HL intervention. We refer to community integration as the level of incorporation of community resources, stakeholders such as promotores , and settings into interventions aimed at improving health literacy, concepts drawn from the literature [ 94 , 95 ]. These categories of community integration were inferred from the setting type since we expect large hospitals to be less involved in community initiatives than community clinics or community settings (i.e. local churches) themselves. Smaller, community-based settings and nonmedical centers such as churches and college campuses seemed to be more successful with implementing multiweek curricula interventions. This may be because these settings have the infrastructure in place for recruitment and retention of community members with a SLP. Larger hospital systems and clinics with less time and resources available may be better able to focus on culturally and linguistically concordant patient materials and replace standard of care materials written in English with multimedia health information. These recommendations are illustrated in Fig.  2 , which displays fit between intervention type and setting.

Recommendations for intervention models by common combinations of setting type, duration available, and community engagement

The findings of our review are also relevant to studies of HL in other populations with a non-English preference, including minority, migrant or immigrant populations. Previous reviews of HL interventions did not include studies measuring HL indirectly through variables such as health outcomes or behavioral change but only included those using standardized tools [ 85 ]. However, as these standardized assessment tools are available predominantly in English, this approach may limit the generalizability of past reviews to non-English speaking populations. A growing body of evidence suggests that a reframing of our understanding of HL, especially among marginalized communities, is necessary to improve health equity [ 2 ].

Finally, our review highlights a need for additional attention to the development and adaptation of HL interventions for patients with SLP in the United States. Policies promoting HL interventions may need to better address the needs of specific populations through research and the widespread promotion of effective strategies.

Limitations

A limitation of our review is that all studies were conducted in the United States, which limits the generalizability of our findings to healthcare systems in other countries. It should be noted that we did not explore gray literature. We also chose to limit our review to studies that took place after 2010, preventing a fuller historical examination of HL literature. Finally, we could not conduct a meta-analysis due to variability in design and measurement.

There is a small but growing body of literature that addresses the need for HL interventions among individuals in the United States with SLP. However, there is no consensus around strategies to improve or tools to assess HL, and studies vary greatly in quality and risk of bias. Important target populations, such as children, older adults and men, may be excluded from this research. Strategies that incorporate linguistic and cultural factors particular to this population, such as fotonovelas and health promoters from similar cultural backgrounds, may be of use in promoting HL. There is a need for improved research and policy on HL interventions specifically targeting this population.

Availability of data and materials

No datasets were generated or analysed during the current study.

Abbreviations

• Health literacy

Spanish-language preference

Preferred Reporting Items for Systematic Reviews and Meta-Analysis

Randomized control trials

Prospective cohort

Cross-sectional

Revised Tool to Assess Risk of Bias in Randomized Trials

Risk Of Bias In Non-randomized Studies - of Interventions

Synthesis without Metanalysis

Federally Qualified Health Centers

Test of Functional Health Literacy in Adults

Newest Vital Sign

Diabetes health literacy

Spanish-speaking

English-speaking

Intervention Care

Physical activity

Early Head Start

Knowledge, Attitudes, Perception

Pediatric Emergency Department

Empowering Change in Health Outcomes

Hypertension Self-Care Activity Level Effects

Analysis, Design, Develop, Implement, and Evaluate

Palliative Care Knowledge Scale

End-stage renal disease

Depression Literacy Questionnaire

Depression Stigma Scale

Attitudes Towards Seeking Professional Psychological Help Short Form

Extended Parallel Process Model

Hearing protection devices

Preventive health model

Rapid Estimate of Adult Literacy in Medicine—Short Form

9-item Patient Health Questionnaire

Depression Knowledge Measure

Visual analog scale

Plan, Do, Study, Act

Medication Understanding Questionnaire

Baker DW. The meaning and the measure of health literacy. J Gen Intern Med. 2006;21:878–83. https://doi.org/10.1111/j.1525-1497.2006.00540.x .

Article   PubMed   PubMed Central   Google Scholar  

Bahrambeygi F, Rakhshanderou S, Ramezankhani A, Ghaffari M. Hospital health literacy conceptual explanation: a qualitative content analysis based on experts and population perspectives. J Educ Health Promot. 2023;12:31. https://doi.org/10.4103/jehp.jehp_494_22 .

Paasche-Orlow MK, Wolf MS. Promoting health literacy research to reduce health disparities. J Health Commun. 2010;15(Suppl 2):34–41. https://doi.org/10.1080/10810730.2010.499994 .

Article   PubMed   Google Scholar  

Fan Z, Yang Y, Zhang F. Association between health literacy and mortality: a systematic review and meta-analysis. Arch Public Health. 2021;79:119. https://doi.org/10.1186/s13690-021-00648-7 .

Berkman ND, Sheridan SL, Donahue KE, Halpern DJ, Crotty K. Low health literacy and health outcomes: an updated systematic review. Ann Intern Med. 2011;155:97–107. https://doi.org/10.7326/0003-4819-155-2-201107190-00005 .

Stormacq C, Van den Broucke S, Wosinski J. Does health literacy mediate the relationship between socioeconomic status and health disparities? Integrative review Health Promot Int. 2019;34:e1–17. https://doi.org/10.1093/heapro/day062 .

Liu C, Wang D, Liu C, Jiang J, Wang X, Chen H, et al. What is the meaning of health literacy? A systematic review and qualitative synthesis. Fam Med Community Health. 2020;8:e000351. https://doi.org/10.1136/fmch-2020-000351 .

Nutbeam D. Health literacy as a public health goal: a challenge for contemporary health education and communication strategies into the 21st century. Health Promot Int. 2000;15:259–67. https://doi.org/10.1093/heapro/15.3.259 .

Article   Google Scholar  

Sørensen K, Van den Broucke S, Fullam J, Doyle G, Pelikan J, Slonska Z, et al. Health literacy and public health: a systematic review and integration of definitions and models. BMC Public Health. 2012;12:80. https://doi.org/10.1186/1471-2458-12-80 .

Nutbeam D, Lloyd JE. Understanding and responding to health literacy as a social determinant of health. Annu Rev Public Health. 2021;42:159–73. https://doi.org/10.1146/annurev-publhealth-090419-102529 .

Kunter M, Greenberg E, Jin Y, The PC, Literacy H, of America’s adults: results from the,. national assessment of adult literacy. US Dep Educ NCES. 2003;2006–483(2006):76.

Google Scholar  

Sentell T, Braun KL. Low health literacy, limited English proficiency, and health status in Asians, Latinos, and other racial/ethnic groups in California. J Health Commun. 2012;17(Suppl 3):82–99. https://doi.org/10.1080/10810730.2012.712621 .

Livingston G, Minushkin S, Cohn D. Hispanics and Health Care in the United States. Pew Research Center. 2008.

Alemán JO, Almandoz JP, Frias JP, Galindo RJ. Obesity among Latinx people in the United States: a review. Obes Silver Spring Md. 2023;31:329–37. https://doi.org/10.1002/oby.23638 .

Auger SJ, Verbiest S, Spickard JV, Simán FM, Colindres M. Participatory group prenatal education using photonovels: evaluation of a lay health educator model with low-income Latinas. J Particip Med. 2015;7:e13.

PubMed   Google Scholar  

Ochoa CY, Murphy ST, Frank LB, Baezconde-Garbanati LA. Using a culturally tailored narrative to increase cervical cancer detection among Spanish-speaking Mexican-American women. J Cancer Educ Off J Am Assoc Cancer Educ. 2020;35:736–42. https://doi.org/10.1007/s13187-019-01521-6 .

Mas FS, Ji M, Fuentes BO, Tinajero J. The health literacy and ESL study: a community-based intervention for Spanish-speaking adults. J Health Commun. 2015;20:369–76. https://doi.org/10.1080/10810730.2014.965368 .

Article   PubMed Central   Google Scholar  

Fernández-Gutiérrez M, Bas-Sarmiento P, Albar-Marín M j, Paloma-Castro O, Romero-Sánchez J m. Health literacy interventions for immigrant populations: a systematic review. Int Nurs Rev. 2018;65:54–64. https://doi.org/10.1111/inr.12373 .

Chen Y, Ran X, Chen Y, Jiang K. Effects of health literacy intervention on health literacy level and glucolipid metabolism of diabetic patients in Mainland China: a systematic review and meta-analysis. J Diabetes Res. 2021;2021:1503446. https://doi.org/10.1155/2021/1503446 .

Article   CAS   PubMed   PubMed Central   Google Scholar  

Flores BE, Acton G, Arevalo-Flechas L, Gill S, Mackert M. Health literacy and cervical cancer screening among Mexican-American Women. Health Lit Res Pract. 2019;3:e1–8. https://doi.org/10.3928/24748307-20181127-01 .

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372: n71. https://doi.org/10.1136/bmj.n71 .

Sterne JAC, Savović J, Page MJ, Elbers RG, Blencowe NS, Boutron I, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. 2019;366: l4898. https://doi.org/10.1136/bmj.l4898 .

Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355: i4919. https://doi.org/10.1136/bmj.i4919 .

Campbell M, McKenzie JE, Sowden A, Katikireddi SV, Brennan SE, Ellis S, et al. Synthesis without meta-analysis (SWiM) in systematic reviews: reporting guideline. BMJ. 2020;368: l6890. https://doi.org/10.1136/bmj.l6890 .

Unger JB, Cabassa LJ, Molina GB, Contreras S, Baron M. Evaluation of a fotonovela to increase depression knowledge and reduce stigma among hispanic adults. J Immigr Minor Health Cent Minor Public Health. 2013;15:398–406. https://doi.org/10.1007/s10903-012-9623-5 .

Schlumbrecht M, Yarian R, Salmon K, Niven C, Singh D. Targeted ovarian cancer education for Hispanic Women: a pilot program in Arizona. J Community Health. 2016;41:619–25. https://doi.org/10.1007/s10900-015-0137-7 .

Buckley J, Yekta S, Joseph V, Johnson H, Oliverio S, De Groot AS. Vida Sana: a lifestyle intervention for uninsured, predominantly Spanish-speaking immigrants improves metabolic syndrome indicators. J Community Health. 2015;40:116–23. https://doi.org/10.1007/s10900-014-9905-z .

Castañeda SF, Giacinto RE, Medeiros EA, Brongiel I, Cardona O, Perez P, et al. Academic-community partnership to develop a patient-centered breast cancer risk reduction program for latina primary care patients. J Racial Ethn Health Disparities. 2016;3:189–99. https://doi.org/10.1007/s40615-015-0125-8 .

Chen HW, Limmer EE, Joseph AK, Kinser K, Trevino A, Valencia A, et al. Efficacy of a lay community health worker (promotoras de salud) program to improve adherence to emollients in Spanish-speaking Latin American pediatric patients in the United States with atopic dermatitis: a randomized, controlled, evaluator-blinded study. Pediatr Dermatol. 2022;40:69–77. https://doi.org/10.1111/pde.15148 .

Cruz Y, Hernandez-Lane M-E, Cohello JI, Bautista CT. The effectiveness of a community health program in improving diabetes knowledge in the Hispanic population: Salud y Bienestar (Health and Wellness). J Community Health. 2013;38:1124–31. https://doi.org/10.1007/s10900-013-9722-9 .

Howie-Esquivel J, Bibbins-Domingo K, Clark R, Evangelista L, Dracup K. A culturally appropriate educational intervention can improve self-care in Hispanic patients with heart failure: a pilot randomized controlled trial. Cardiol Res. 2014;5:91–100. https://doi.org/10.14740/cr346w .

Han H-R, Delva S, Greeno RV, Negoita S, Cajita M, Will W. A Health literacy-focused intervention for Latinos with hypertension. Health Lit Res Pract. 2018;2:e21–5. https://doi.org/10.3928/24748307-20180108-02 .

Jandorf L, Ellison J, Shelton R, Thélémaque L, Castillo A, Mendez EI, et al. Esperanza y Vida: a culturally and linguistically customized breast and cervical education program for diverse latinas at three different United States sites. J Health Commun. 2012;17:160–76. https://doi.org/10.1080/10810730.2011.585695 .

Kaphingst KA, Lachance CR, Gepp A, Hoyt D’Anna L, Rios-Ellis B. Educating underserved Latino communities about family health history using lay health advisors. Public Health Genomics. 2011;14:211–21. https://doi.org/10.1159/000272456 .

Article   CAS   PubMed   Google Scholar  

Laughman AB, Boselli D, Love M, Steuerwald N, Symanowski J, Blackley K, et al. Outcomes of a structured education intervention for Latinas concerning breast cancer and mammography. Health Educ J. 2017;76:442–53. https://doi.org/10.1177/0017896917691789 .

Martin M, Garcia M, Christofferson M, Bensen R, Yeh A, Park K. Spanish and english language symposia to enhance activation in pediatric inflammatory bowel disease. J Pediatr Gastroenterol Nutr. 2016;63:508–11. https://doi.org/10.1097/MPG.0000000000001191 .

Soto Mas F, Jacobson HE, Olivárez A. Adult education and the health literacy of hispanic immigrants in the United States. J Lat Educ. 2017;16:314–22. https://doi.org/10.1080/15348431.2016.1247707 .

Mas FS, Schmitt CL, Jacobson HE, Myers OB. A cardiovascular health intervention for Spanish speakers: the health literacy and ESL curriculum. J Community Health. 2018;43:717–24. https://doi.org/10.1007/s10900-018-0475-3 .

Mojica CM, Morales-Campos DY, Carmona CM, Ouyang Y, Liang Y. Breast, cervical, and colorectal cancer education and navigation: results of a community health worker intervention. Health Promot Pract. 2016;17:353–63. https://doi.org/10.1177/1524839915603362 .

Moralez EA, Rao SP, Livaudais JC, Thompson B. Improving knowledge and screening for colorectal cancer among hispanics: overcoming barriers through a promotora-led home-based educational intervention. J Cancer Educ Off J Am Assoc Cancer Educ. 2012;27. https://doi.org/10.1007/s13187-012-0357–9 .

Nitsos A, Estrada RD, Hilfinger Messias DK. Tummy time for Latinos with limited english proficiency: evaluating the feasibility of a cultural and linguistically adapted parent education intervention. J Pediatr Nurs. 2017;36:31–6. https://doi.org/10.1016/j.pedn.2017.04.004 .

Ockene IS, Tellez TL, Rosal MC, Reed GW, Mordes J, Merriam PA, et al. Outcomes of a Latino Community-based intervention for the prevention of diabetes: the lawrence Latino diabetes prevention project. Am J Public Health. 2012;102:336–42. https://doi.org/10.2105/AJPH.2011.300357 .

Otilingam PG, Gatz M, Tello E, Escobar AJ, Goldstein A, Torres M, et al. Buenos hábitos alimenticios para una buena salud: evaluation of a nutrition education program to improve heart health and brain health in Latinas. J Aging Health. 2015;27:177–92. https://doi.org/10.1177/0898264314549660 .

Peña-Purcell NC, Boggess MM. An application of a diabetes knowledge scale for low-literate Hispanic/Latinos. Health Promot Pract. 2014;15:252–62. https://doi.org/10.1177/1524839912474006 .

Rascón MS, Garcia ML, Nguyen-Rodriguez ST, Galvez G, Gepp A, Carrillo E, et al. Comprando Rico y Sano: increasing latino nutrition knowledge, healthful diets, and food access through a national community-based intervention. Am J Health Promot. 2022;36:876–80. https://doi.org/10.1177/08901171211073956 .

Risica PM, McCarthy ML, Barry KL, Oliverio SP, Gans KM, De Groot AS. Clinical outcomes of a community clinic-based lifestyle change program for prevention and management of metabolic syndrome: results of the ‘Vida Sana/Healthy Life’ program. PLoS ONE. 2021;16: e0248473. https://doi.org/10.1371/journal.pone.0248473 .

Romero DC, Sauris A, Rodriguez F, Delgado D, Reddy A, Foody JM. Vivir Con Un Corazón Saludable: a community-based educational program aimed at increasing cardiovascular health knowledge in high-risk Hispanic Women. J Racial Ethn Health Disparities. 2016;3:99–107. https://doi.org/10.1007/s40615-015-0119-6 .

Sanchez JI, Briant KJ, Wu-Georges S, Gonzalez V, Galvan A, Cole S, et al. Eat healthy, be active community workshops implemented with rural hispanic women. BMC Womens Health. 2021;21:24. https://doi.org/10.1186/s12905-020-01157-5 .

Soto Mas F, Cordova C, Murrietta A, Jacobson HE, Ronquillo F, Helitzer D. A multisite community-based health literacy intervention for Spanish speakers. J Community Health. 2015;40:431–8. https://doi.org/10.1007/s10900-014-9953-4 .

Stockwell MS, Catallozzi M, Meyer D, Rodriguez C, Martinez E, Larson E. Improving care of upper respiratory infections among Latino early head start parents. J Immigr Minor Health. 2010;12:925–31. https://doi.org/10.1007/s10903-010-9326-8 .

Stockwell MS, Catallozzi M, Larson E, Rodriguez C, Subramony A, Andres Martinez R, et al. al. Effect of a URI-related educational intervention in early head start on ED visits. Pediatrics. 2014;133:e1233–40. https://doi.org/10.1542/peds.2013-2350 .

Warren-Findlow J, Coffman MJ, Thomas EV, Krinner LM. ECHO: a pilot health literacy intervention to improve hypertension self-care. HLRP Health Lit Res Pract. 2019;3:e259–67. https://doi.org/10.3928/24748307-20191028-01 .

Borrayo EA, Rosales M, Gonzalez P. Entertainment-education narrative versus nonnarrative interventions to educate and motivate latinas to engage in mammography screening. Health Educ Behav Off Publ Soc Public Health Educ. 2017;44:394–402. https://doi.org/10.1177/1090198116665624 .

Cabassa LJ, Oh H, Humensky JL, Unger JB, Molina GB, Baron M. Comparing the impact on latinos of a depression brochure and an entertainment-education depression fotonovela. Psychiatr Serv Wash DC. 2015;66:313–6. https://doi.org/10.1176/appi.ps.201400146 .

Calderón JL, Shaheen M, Hays RD, Fleming ES, Norris KC, Baker RS. Improving diabetes health literacy by animation. Diabetes Educ. 2014;40:361–72. https://doi.org/10.1177/0145721714527518 .

Cheney AM, McCarthy WJ, Pozar M, Reaves C, Ortiz G, Lopez D, et al. “Ancestral recipes”: a mixed-methods analysis of MyPlate-based recipe dissemination for Latinos in rural communities. BMC Public Health. 2023;23:216. https://doi.org/10.1186/s12889-022-14804-3 .

Enguidanos S, Storms AD, Lomeli S, van Zyl C. Improving palliative care knowledge among hospitalized hispanic patients: a pilot study. J Palliat Med. 2022;25:1179–85. https://doi.org/10.1089/jpm.2021.0559 .

Forster M, Allem J-P, Mendez N, Qazi Y, Unger JB. Evaluation of a telenovela designed to improve knowledge and behavioral intentions among Hispanic patients with end-stage renal disease in Southern California. Ethn Health. 2016;21:58–70. https://doi.org/10.1080/13557858.2015.1007119 .

Gonzalez F, Benuto LT. ¡Yo no Estoy Loca! A behavioral health telenovela style entertainment education video: increasing mental health literacy among Latinas. Community Ment Health J. 2022;58:850–61. https://doi.org/10.1007/s10597-021-00892-9 .

Gossey JT, Whitney SN, Crouch MA, Jibaja-Weiss ML, Zhang H, Volk RJ. Promoting knowledge of statins in patients with low health literacy using an audio booklet. Patient Prefer Adherence. 2011;5:397–403. https://doi.org/10.2147/PPA.S19995 .

Guiberson M, Wakefield E. A preliminary study of a Spanish graphic novella targeting hearing loss prevention. Am J Audiol. 2017;26:259–67. https://doi.org/10.1044/2017_AJA-16-0069 .

Gwede CK, Sutton SK, Chavarria EA, Gutierrez L, Abdulla R, Christy SM, et al. A culturally and linguistically salient pilot intervention to promote colorectal cancer screening among Latinos receiving care in a federally qualified health center. Health Educ Res. 2019;34:310–20. https://doi.org/10.1093/her/cyz010 .

Hernandez MY, Organista KC. Entertainment–Education? A Fotonovela? A new strategy to improve depression literacy and help-seeking behaviors in at-risk immigrant Latinas. Am J Community Psychol. 2013;52:224–35. https://doi.org/10.1007/s10464-013-9587-1 .

Merchant RC, Clark MA, Santelices CA, Liu T, Cortés DE. Efficacy of an HIV/AIDS and HIV testing video for Spanish-speaking Latinos in healthcare and non-healthcare settings. AIDS Behav. 2015;19:523. https://doi.org/10.1007/s10461-014-0889-6 .

Molokwu JC, Shokar N, Dwivedi A. Impact of targeted education on colorectal cancer screening knowledge and psychosocial attitudes in a predominantly hispanic population. Fam Community Health. 2017;40:298–305. https://doi.org/10.1097/FCH.0000000000000165 .

Pagán-Ortiz ME, Cortés DE. Feasibility of an online health intervention for Latinas with chronic pain. Rehabil Psychol. 2021;66:10–21. https://doi.org/10.1037/rep0000341 .

Payán DD, Maggard-Gibbons M, Flórez KR, Mejía N, Hemmelgarn M, Kanouse D, et al. Taking care of yourself and your risk for breast cancer (CUIDARSE): a randomized controlled trial of a health communication intervention for Latinas. Health Educ Behav. 2020;47:569–80. https://doi.org/10.1177/1090198120920529 .

Phipps MG, Venkatesh KK, Ware C, Lightfoot M, Raker C, Rodriguez P. Project ESCUCHE: a Spanish-language radio-based intervention to increase science literacy. Rhode Island Med J. 2018:41–5.

Ramos IN, Ramos KS, Boerner A, He Q, Tavera-Garcia MA. Culturally-tailored education programs to address health literacy deficits and pervasive health disparities among hispanics in rural shelbyville. Kentucky J Community Med Health Educ. 2013;3:20475. https://doi.org/10.4172/2161-0711.1000250 .

Reuland DS, Ko LK, Fernandez A, Braswell LC, Pignone M. Testing a Spanish-language colorectal cancer screening decision aid in Latinos with limited english proficiency: results from a pre-post trial and four month follow-up survey. BMC Med Inform Decis Mak. 2012;12:53. https://doi.org/10.1186/1472-6947-12-53 .

Lajonchere CM, Wheeler BY, Valente TW, Kreutzer C, Munson A, Narayanan S, et al. Strategies for disseminating information on biomedical research on autism to hispanic parents. J Autism Dev Disord. 2016;46:1038–50. https://doi.org/10.1007/s10803-015-2649-5 .

Riera A, Ocasio A, Tiyyagura G, Thomas A, Goncalves P, Krumeich L, et al. A web-based educational video to improve asthma knowledge for limited english proficiency Latino caregivers. J Asthma Off J Assoc Care Asthma. 2017;54:624. https://doi.org/10.1080/02770903.2016.1251597 .

Robinson JK, Friedewald JJ, Desai A, Gordon EJ. Response across the health-literacy spectrum of kidney transplant recipients to a sun-protection education program delivered on tablet computers: randomized controlled trial. JMIR Cancer. 2015;1: e8. https://doi.org/10.2196/cancer.4787 .

Sanchez K, Killian MO, Eghaneyan BH, Cabassa LJ, Trivedi MH. Culturally adapted depression education and engagement in treatment among Hispanics in primary care: outcomes from a pilot feasibility study. BMC Fam Pract. 2019;20:140. https://doi.org/10.1186/s12875-019-1031-7 .

Valdez A, Napoles AM, Stewart S, Garza A. A randomized controlled trial of a cervical cancer education intervention for Latinas delivered through interactive, multimedia kiosks. J Cancer Educ Off J Am Assoc Cancer Educ. 2018;33:222–30. https://doi.org/10.1007/s13187-016-1102-6 .

Valenzuela-Araujo D, Godage SK, Quintanilla K, Dominguez Cortez J, Polk S, DeCamp LR. Leaving paper behind: improving healthcare navigation by Latino immigrant parents through video-based education. J Immigr Minor Health. 2021;23:329–36. https://doi.org/10.1007/s10903-020-00969-9 .

West AM, Bittner EA, Ortiz VE. The effects of preoperative, video-assisted anesthesia education in Spanish on Spanish-speaking patients’ anxiety, knowledge, and satisfaction: a pilot study. J Clin Anesth. 2014;26:325–9. https://doi.org/10.1016/j.jclinane.2013.12.008 .

Calderon V, Cain R, Torres E, Lopez SR. Evaluating the message of an ongoing communication campaign to reduce the duration of untreated psychosis in a Latinx community in the United States. Early Interv Psychiatry. 2022;16:147–52. https://doi.org/10.1111/eip.13140 .

Chalela P, Muñoz E, Gallion KJ, Kaklamani V, Ramirez AG. Empowering Latina breast cancer patients to make informed decisions about clinical trials: a pilot study. Transl Behav Med. 2018;8:439–49. https://doi.org/10.1093/tbm/ibx083 .

Cullen SM, Osorio SN, Abramson EA, Kyvelos E. Improving caregiver understanding of liquid acetaminophen administration at primary care visits. Pediatrics. 2022;150: e2021054807. https://doi.org/10.1542/peds.2021-054807 .

Dunlap JL, Jaramillo JD, Koppolu R, Wright R, Mendoza F, Bruzoni M. The effects of language concordant care on patient satisfaction and clinical understanding for Hispanic pediatric surgery patients. J Pediatr Surg. 2015;50:1586–9. https://doi.org/10.1016/j.jpedsurg.2014.12.020 .

Arun Mohan MD, M. Brian Riley MA, Brian Schmotzer MS, Dane R. Boyington P, Sunil Kripalani MD. Improving medication understanding among Latinos through illustrated medication lists. Am J Manag Care 2015;20.

Vadaparampil ST, Moreno Botero L, Fuzzell L, Garcia J, Jandorf L, Hurtado-de-Mendoza A, et al. Development and pilot testing of a training for bilingual community education professionals about hereditary breast and ovarian cancer among Latinas: ÁRBOLES Familiares. Transl Behav Med. 2022;12:ibab093. https://doi.org/10.1093/tbm/ibab093 .

Anderson PM, Krallman R, Montgomery D, Kline-Rogers E, Bumpus SM. The relationship between patient satisfaction with hospitalization and outcomes up to 6 months post-discharge in cardiac patients. J Patient Exp. 2020;7:1685–92. https://doi.org/10.1177/2374373520948389 .

Berkman ND, Sheridan SL, Donahue KE, Halpern DJ, Viera A, Crotty K, et al. Health literacy interventions and outcomes: an updated systematic review. Agency for Healthcare Research and Quality (US). 2011.

Diaz E, Ortiz-Barreda G, Ben-Shlomo Y, Holdsworth M, Salami B, Rammohan A, et al. Interventions to improve immigrant health. A scoping review. Eur J Public Health. 2017;27:433–9.

Walters R, Leslie SJ, Polson R, Cusack T, Gorely T. Establishing the efficacy of interventions to improve health literacy and health behaviours: a systematic review. BMC Public Health. 2020;20:1040. https://doi.org/10.1186/s12889-020-08991-0 .

Funk C, Lopez MH. 2. Hispanic Americans’ experiences with health care. Pew Res Cent Sci Soc 2022. https://www.pewresearch.org/science/2022/06/14/hispanic-americans-experiences-with-health-care/ . Accessed 13 Oct 2022.

Ai AL, Noël LT, Appel HB, Huang B, Hefley WE. Overall Health and Health Care Utilization Among Latino American Men in the United States. Am J Mens Health. 2013;7:6–17. https://doi.org/10.1177/1557988312452752 .

Shafeek Amin N, Driver N. Health care utilization among Middle Eastern, Hispanic/Latino, and Asian immigrants in the United States: an application of Andersen’s behavioral model. Ethn Health. 2022;27:858–76. https://doi.org/10.1080/13557858.2020.1830034 .

Farmanova E, Bonneville L, Bouchard L. Organizational health literacy: review of theories, frameworks, guides, and implementation issues. Inq J Med Care Organ Provis Financ. 2018;55:0046958018757848. https://doi.org/10.1177/0046958018757848 .

Moran MB, Murphy ST, Frank L, Baezconde-Garbanati L. The ability of narrative communication to address health-related social norms. Int Rev Soc Res. 2013;3:131–49. https://doi.org/10.1515/irsr-2013-0014 .

Cupertino AP, Saint-Elin M, de los Rios JB, Engelman KK, Greiner KA, Ellerbeck EF, et al. Empowering promotores de salud as partners in cancer education and research in rural Southwest Kansas. Oncol Nurs Forum. 2015;42:15–22.

National academies of sciences, engineering, and medicine, health and medicine division, board on population health and public health practice, roundtable on health literacy. Community-based health literacy interventions: proceedings of a workshop. Washington (DC): National Academies Press (US); 2018.

Nutbeam D, McGill B, Premkumar P. Improving health literacy in community populations: a review of progress. Health Promot Int. 2018;33:901–11. https://doi.org/10.1093/heapro/dax015 .

Download references

Contributors

Not applicable.

Prior presentations

Primary Care Innovations Conference, University of Florida College of Medicine, June 10, 2022.

Author information

Authors and affiliations.

Equal Access Research, College of Medicine, University of Florida, 1549 Gale Lemerand Drive, 4 Floor, Suite 4592, Gainesville, FL, 32610-3008, USA

Liliana Demiranda, Priyanka Perisetla, Lauren Andrews, Keer Zhang, Ajay Mittal & Melanie G. Hagen

Department of Internal Medicine, College of Medicine, University of Florida, Gainesville, FL, USA

Melanie G. Hagen

University of Florida Health Science Center Libraries, Gainesville, FL, USA

Hannah F. Norton

University of Central Florida College of Medicine, University of Central Florida, Orlando, USA

Joel Hernandez

David Geffen School of Medicine, University of California Los Angeles, Los Angeles, USA

College of Medicine, University of Florida, Gainesville, Florida, USA

Rebecca Henderson

You can also search for this author in PubMed   Google Scholar

Contributions

All authors meet the criteria for authorship in the following way: JH, LD, PP, LA, KZ, and AM were responsible for the study design, data analysis, and writing and review of the manuscript. RH, HN, and MH were responsible for the study design and writing and review of the manuscript. The authors have no acknowledgments.

Corresponding author

Correspondence to Melanie G. Hagen .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., supplementary material 3., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Hernandez, J., Demiranda, L., Perisetla, P. et al. A systematic review and narrative synthesis of health literacy interventions among Spanish speaking populations in the United States. BMC Public Health 24 , 1713 (2024). https://doi.org/10.1186/s12889-024-19166-6

Download citation

Received : 29 November 2023

Accepted : 14 June 2024

Published : 27 June 2024

DOI : https://doi.org/10.1186/s12889-024-19166-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Limited English Proficiency (LEP)
• Spanish speaking
• Intervention
• Systematic review

BMC Public Health

ISSN: 1471-2458

• General enquiries: [email protected]

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Author Biographies
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

A preliminary scoping review of the impact of e-prescribing on pharmacists in community pharmacies.

1. Introduction

2. study rationale, 3. study objective, 4.1. identifying the research questions.

• What are the community pharmacists’ perceptions of the impact of electronic prescribing on medication errors and patient outcomes in community pharmacies?
• What are the community pharmacists’ perceptions of the impact of electronic prescribing on the pharmacist work and productivity in their pharmacy?

4.2. Identifying Relevant Studies

5. study selection, 5.1. charting the data, 5.2. collating, summarizing, and reporting the results, 6.1. study selection, 6.2. study characteristics, 6.3. characteristics of the subjects in the studies, 6.4. country and pharmacy characteristics, 6.5. data collection technique, 6.6. e-prescribing impact on medication errors, 6.7. e-prescribing impact on productivity, 7. discussion, new challenges with e-prescribing, 8. limitations, 9. conclusions, author contributions, institutional review board statement, informed consent statement, data availability statement, conflicts of interest.

• Campbell, C.; Morris, C.; McBain, L. Electronic transmission of prescriptions in primary care: Transformation, timing and teamwork. J. Prim. Health Care 2021 , 13 , 340–350. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Motulsky, A.; Liang, M.; Moreault, M.P.; Borycki, E.; Kushniruk, A.; Sicotte, C. Evaluation of a Nationwide e-Prescribing System. Stud. Health Technol. Inform. 2019 , 264 , 714–718. [ Google Scholar ] [ PubMed ]
• Phillips, J.L.; Shea, J.M.; Leung, V.; MacDonald, D. Impact of Early Electronic Prescribing on Pharmacists’ Clarification Calls in Four Community Pharmacies Located in St John’s, Newfoundland. JMIR Med. Inform. 2015 , 3 , e2. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Salmon, J.W.; Jiang, R. E-Prescribing: History, Issues, Potential. Online J. Public Health Inform. 2012 , 4 , e61049. Available online: https://journals.uic.edu/ojs/index.php/ojphi/article/view/4304 (accessed on 1 December 2018). [ CrossRef ] [ PubMed ]
• Elliott, R.; Lee, C.; Hussainy, S. Electronic prescribing and medication management at a residential aged care facility Uptake by general practitioners and qualitative evaluation of the benefits and barriers. Appl. Clin. Inform. 2016 , 7 , 116–127. [ Google Scholar ] [ PubMed ]
• Farghali, A.; Borycki, E.M.; Macdonald, S. Pharmacist’s perception of the impact of electronic prescribing on medication errors and productivity in community pharmacies. Knowl. Manag. E-Learn. Int. J. 2021 , 13 , 536–558. [ Google Scholar ]
• Au, Y.; Baker, L.; Hindmarsh, J. The Impact of an Electronic Prescribing Template with Decision Support upon the Prescribing of Subcutaneous Infusions at the End of Life in a Community Setting: A Future Vision for Community Palliative Care. Pharmacy 2022 , 10 , 112. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Aronson, J.K. Medication errors: Definitions and classification. Br. J. Clin. Pharmacol. 2009 , 67 , 599–604. [ Google Scholar ] [ CrossRef ]
• Garfield, S.; Hibberd, R.; Barber, N. English community pharmacists’ experiences of using electronic transmission of prescriptions: A qualitative study. BMC Health Serv. Res. 2013 , 13 , 435. [ Google Scholar ] [ CrossRef ]
• Porterfield, A.; Engelbert, K.; Coustasse, A. Electronic Prescribing: Improving the Efficiency and Accuracy of Prescribing in the Ambulatory Care Setting. Perspect. Health Inf. Manag. 2014 , 11 , 1g. [ Google Scholar ]
• National Statistics|NAPRA. 2018. Available online: https://napra.ca/national-statistics (accessed on 7 October 2018).
• MacLure, K.; Stewart, D. Digital literacy knowledge and needs of pharmacy staff: A systematic review. J. Innov. Health Inform. 2016 , 23 , 560–571. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Daudt, H.M.; van Mossel, C.; Scott, S.J. Enhancing the scoping study methodology: A large, inter-professional team’s experience with Arksey and O’Malley’s framework. BMC Med. Res. Methodol. 2013 , 13 , 48. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Peltoniemi, T.; Suomi, R.; Peura, S.; Lähteenoja, M.N.Y. Electronic prescription as a driver for digitalization in Finnish pharmacies. BMC Health Serv. Res. 2021 , 21 , 1017. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Panich, J.; Larson, N.; Sojka, L.; Wallace, Z.; Lokken, J. Assessing automated product selection success rates in transmissions between electronic prescribing and community pharmacy platforms. J. Am. Med. Inf. Assoc. 2021 , 28 , 113–118. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Shah, S.R.; Galt, K.A.; Fuji, K.T. Error types with use of medication-related technology: A mixed methods research study. Res. Soc. Adm. Pharm. 2019 , 15 , 1480–1483. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Kauppinen, H.; Ahonen, R.; Timonen, J. The impact of electronic prescriptions on medication safety in Finnish community pharmacies: A survey of pharmacists. Int. J. Med. Inform. 2017 , 100 , 56–62. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Lander, L.; Klepser, D.G.; Cochran, G.L.; Lomelin, D.E.; Morien, M. Barriers to electronic prescribing: Nebraska pharmacists’ perspective. J. Rural Health 2013 , 29 , 119–124. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Gilligan, A.M.; Miller, K.; Mohney, A.; Montenegro, C.; Schwarz, J.; Warholak, T.L. Analysis of pharmacists’ interventions on electronic versus traditional prescriptions in 2 community pharmacies. Res. Soc. Adm. Pharm. RSAP 2012 , 8 , 523–532. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Grossman, J.M.; Cross, D.A.; Boukus, E.R.; Cohen, G.R. Transmitting and processing electronic prescriptions: Experiences of physician practices and pharmacies. J. Am. Med. Inform. Assoc. JAMIA 2012 , 19 , 353–359. [ Google Scholar ] [ CrossRef ]
• Odukoya, O.K.; Chui, M.A. Relationship between e-prescriptions and community pharmacy workflow. J. Am. Pharm. Assoc. Japha 2012 , 52 , e168–e174. [ Google Scholar ] [ CrossRef ]
• Rahimi, B.; Timpka, T. Pharmacists’ views on integrated electronic prescribing systems: Associations between usefulness, pharmacological safety, and barriers to technology use. Eur. J. Clin. Pharmacol. 2011 , 67 , 179–184. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Munn, Z.; Peters, M.D.; Stern, C.; Tufanaru, C.; McArthur, A.; Aromataris, E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med. Res. Methodol. 2018 , 18 , 143. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Galt, K.A.; Fuji, K.T.; Kaufman, T.K.; Shah, S.R. Health Information Technology Use and Patient Safety: Study of Pharmacists in Nebraska. Pharmacy 2019 , 7 , 7. [ Google Scholar ] [ CrossRef ]
• Franklin, B.D.; Reynolds, M.J.; Hibberd, R.; Sadler, S.; Barber, N. Community pharmacists’ interventions with electronic prescriptions in England: An exploratory study. Int. J. Clin. Pharm. 2013 , 35 , 1030–1035. [ Google Scholar ] [ CrossRef ]
• Canadian Medical Association (CMA); Canadian Pharmacists Association (CPhA). Vision for e-Prescribing: A Joint Statement by the Canadian Medical Association and the Canadian Pharmacists Association. 2012. Available online: http://policybase.cma.ca/dbtw-wpd/Policypdf/PD13-02.pdf (accessed on 30 April 2020).
• Ekedahl, A. Problem prescriptions in Sweden necessitating contact with the prescriber before dispensing. Res. Soc. Adm. Pharm. 2010 , 6 , 174–184. [ Google Scholar ] [ CrossRef ]
• Warholak, T.L.; Rupp, M.T.; Leal, S.; Kurniawan, G.; Patel, N. Assessing the effect of providing a pharmacist with patient diagnosis on electronic prescription orders: A pilot study. Res. Soc. Adm. Pharm. 2014 , 10 , 246–251. [ Google Scholar ] [ CrossRef ]
• Hincapie, A.L.; Warholak, T.; Altyar, A.; Snead, R.; Modisett, T. Electronic prescribing problems reported to the Pharmacy and Provider ePrescribing Experience Reporting (PEER) portal. Res. Soc. Adm. Pharm. 2014 , 10 , 647–655. [ Google Scholar ] [ CrossRef ]
• McLeod, M.; Karampatakis, G.D.; Heyligen, L.; McGinley, A.; Franklin, B.D. The impact of implementing a hospital electronic prescribing and administration system on clinical pharmacists’ activities—A mixed methods study. BMC Health Serv. Res. 2019 , 19 , 156. [ Google Scholar ] [ CrossRef ]
• Smith, M.; Sprecher, B. Pharmacy communications with physician offices to clarify prescriptions. J. Am. Pharm. Assoc. Japha 2017 , 57 , 178–182. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Klepser, D.; Lanham, A.; Cochran, G. Electronic prescriptions: Opportunities and challenges for the patient and pharmacist. AHCT 2016 , 2 , 1–11. [ Google Scholar ] [ CrossRef ]
• Odukoya, O.K.; Stone, J.A.; Chui, M.A. How do community pharmacies recover from e-prescription errors? Res. Soc. Adm. Pharm. 2014 , 10 , 837–852. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Damiani, G.; Altamura, G.; Zedda, M.; Nurchis, M.C.; Aulino, G.; Alizadeh, A.H.; Cazzato, F.; Della Morte, G.; Caputo, M.; Grassi, S.; et al. Potentiality of algorithms and artificial intelligence adoption to improve medication management in primary care: A systematic review. BMJ Open 2023 , 13 , e065301. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Rieckert, A.; Reeves, D.; Altiner, A.; Drewelow, E.; Esmail, A.; Flamm, M.; Hann, M.; Johansson, T.; Klaassen-Mielke, R.; Kunnamo, I.; et al. Use of an electronic decision support tool to reduce polypharmacy in elderly people with chronic diseases: Cluster randomised controlled trial. BMJ 2020 , 369 , m1822. [ Google Scholar ] [ CrossRef ] [ PubMed ]

Click here to enlarge figure

Share and Cite

Farghali, A.A.; Borycki, E.M. A Preliminary Scoping Review of the Impact of e-Prescribing on Pharmacists in Community Pharmacies. Healthcare 2024 , 12 , 1280. https://doi.org/10.3390/healthcare12131280

Farghali AA, Borycki EM. A Preliminary Scoping Review of the Impact of e-Prescribing on Pharmacists in Community Pharmacies. Healthcare . 2024; 12(13):1280. https://doi.org/10.3390/healthcare12131280

Farghali, Amr A., and Elizabeth M. Borycki. 2024. "A Preliminary Scoping Review of the Impact of e-Prescribing on Pharmacists in Community Pharmacies" Healthcare 12, no. 13: 1280. https://doi.org/10.3390/healthcare12131280

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals