pico research question systematic review

Select your preferred language for Cochrane Reviews and other content. Sections without translation will be in English.

Select your preferred language for the Cochrane Library website.

Predefining unambiguous criteria for participants ( )

Predefined, unambiguous eligibility criteria are a fundamental prerequisite for a systematic review. The criteria for considering types of people included in studies in a review should be sufficiently broad to encompass the likely diversity of studies, but sufficiently narrow to ensure that a meaningful answer can be obtained when studies are considered in aggregate. Considerations when specifying participants include setting, diagnosis or definition of condition and demographic factors. Any restrictions to study populations must be based on a sound rationale, since it is important that Cochrane Reviews are widely relevant.

Predefining a strategy for studies with a subset of eligible participants ( )

Sometimes a study includes some ‘eligible’ participants and some ‘ineligible’ participants, for example when an age cut-off is used in the review’s eligibility criteria. If data from the eligible participants cannot be retrieved, a mechanism for dealing with this situation should be pre-specified.

Changing eligibility criteria ( )

Following pre-specified eligibility criteria is a fundamental attribute of a systematic review. However, unanticipated issues may arise. Review authors should make sensible post-hoc decisions about exclusion of studies, and these should be documented in the review, possibly accompanied by sensitivity analyses. Changes to the protocol must not be made on the basis of the findings of the studies or the synthesis, as this can introduce bias.

Intended recipient of intervention

Patient, carer, healthcare provider (general practitioners, nurses, allied health professionals), health system, policy maker, community

In a review of e-learning programmes for health professionals, a subgroup analysis was planned to examine if the effects were modified by the (doctors, nurses or physiotherapists). The authors hypothesized that e-learning programmes for doctors would be more effective than for other health professionals, but did not provide a rationale (Vaona et al 2018).

Disease/condition (to be treated or prevented)

Type and severity of a condition

In a review of platelet-rich therapies for musculoskeletal soft tissue injuries, a subgroup analysis was undertaken to examine if the effects of platelet-rich therapies were modified by the (e.g. rotator cuff tear, anterior cruciate ligament reconstruction, chronic Achilles tendinopathy) (Moraes et al 2014).

In planning a review of beta-blockers for heart failure, subgroup analyses were specified to examine if the effects of beta-blockers are modified by the (e.g. idiopathic dilated cardiomyopathy, ischaemic heart disease, valvular heart disease, hypertension) and the (‘reduced left ventricular ejection fraction (LVEF)’ ≤ 40%, ‘mid-range LVEF’ > 40% and < 50%, ‘preserved LVEF’ ≥ 50%, mixed, not specified). Studies have shown that patient characteristics and comorbidities differ by heart failure severity, and that therapies have been shown to reduce morbidity in ‘reduced LVEF’ patients, but the benefits in the other groups are uncertain (Safi et al 2017).

Participant characteristics

Age (neonate, child, adolescent, adult, older adult)

Race/ethnicity

Sex/gender

PROGRESS-Plus equity characteristics (e.g. place of residence, socio-economic status, education) (O’Neill et al 2014)

In a review of newer-generation antidepressants for depressive disorders in children and adolescents, a subgroup analysis was undertaken to examine if the effects of the antidepressants were modified by . The rationale was based on the findings of another review that suggested that children and adolescents may respond differently to antidepressants. The age groups were defined as ‘children’ (aged approximately 6 to 12 years), ‘adolescents’ (aged approximately 13 to 18 years), and ‘children and adolescents’ (when the study included both children and adolescents, and results could not be obtained separately by these subpopulations) (Hetrick et al 2012).

Setting

Setting of care (primary care, hospital, community)

Rurality (urban, rural, remote)

Socio-economic setting (low and middle-income countries, high-income countries)

Hospital ward (e.g. intensive care unit, general medical ward, outpatient)

In a review of hip protectors for preventing hip fractures in older people, separate comparisons were specified based on (institutional care or community-dwelling) for the critical outcome of hip fracture (Santesso et al 2014).

Predefining unambiguous criteria for interventions and comparators ( )

Predefined, unambiguous eligibility criteria are a fundamental prerequisite for a systematic review. Specification of comparator interventions requires particular clarity: are the experimental interventions to be compared with an inactive control intervention (e.g. placebo, no treatment, standard care, or a waiting list control), or with an active control intervention (e.g. a different variant of the same intervention, a different drug, a different kind of therapy)? Any restrictions on interventions and comparators, for example, regarding delivery, dose, duration, intensity, co-interventions and features of complex interventions should also be predefined and explained.

1. Identify intervention characteristics that may modify the effect of the intervention.

Consider whether differences in interventions characteristics might modify the size of the intervention effect importantly. Content-specific research literature and expertise should inform this step.

The TIDieR checklist – a tool for describing interventions – outlines the characteristics across which an intervention might differ (Hoffmann et al 2014). These include ‘what’ materials and procedures are used, ‘who’ provides the intervention, ‘when and how much’ intervention is delivered. The iCAT-SR tool provides equivalent guidance for complex interventions (Lewin et al 2017).

differ across multiple characteristics, which vary in importance depending on the review.

In a review of exercise for osteoporosis, whether the exercise is weight-bearing or non-weight-bearing may be a key characteristic, since the mechanism by which exercise is thought to work is by placing stress or mechanical load on bones (Howe et al 2011).

Different mechanisms apply in reviews of exercise for knee osteoarthritis (muscle strengthening), falls prevention (gait and balance), cognitive function (cardiovascular fitness).

The differing mechanisms might suggest different ways of grouping interventions (e.g. by intensity, mode of delivery) according to potential modifiers of the intervention effects.

2a. Label and define intervention groups to be considered in the synthesis.

For each intervention group, provide a short label (e.g. supportive psychotherapy) and describe the core characteristics (criteria) that will be used to assign each intervention from an included study to a group.

Groups are often defined by intervention content (especially the active components), such as materials, procedures or techniques (e.g. a specific drug, an information leaflet, a behaviour change technique). Other characteristics may also be used, although some are more commonly used to define subgroups (see ): the purpose or theoretical underpinning, mode of delivery, provider, dose or intensity, duration or timing of the intervention (Hoffmann et al 2014).

In specifying groups:

Logic models may help structure the synthesis (see and ).

In a review of psychological therapies for coronary heart disease, a single group was specified for meta-analysis that included all types of therapy. Subgroups were defined to examine whether intervention effects were modified by intervention components (e.g. cognitive techniques, stress management) or mode of delivery (e.g. individual, group) (Richards et al 2017).

In a review of psychological therapies for panic disorder (Pompoli et al 2016), eight types of therapy were specified:

1. psychoeducation;

2. supportive psychotherapy (with or without a psychoeducational component);

3. physiological therapies;

4. behaviour therapy;

5. cognitive therapy;

6. cognitive behaviour therapy (CBT);

7. third-wave CBT; and

8. psychodynamic therapies.

Groups were defined by the theoretical basis of each therapy (e.g. CBT aims to modify maladaptive thoughts through cognitive restructuring) and the component techniques used.

2b. Define levels for groups based on dose or intensity.

For groups based on ‘how much’ of an intervention is used (e.g. dose or intensity), criteria are needed to quantify each group. This may be straightforward for easy-to-quantify characteristics, but more complex for characteristics that are hard to quantify (e.g. duration or intensity of rehabilitation or psychological therapy).

The levels should be based on how the intervention is used in practice (e.g. cut-offs for low and high doses of a supplement based on recommended nutrient intake), or on a rationale for how the intervention might work.

In reviews of exercise, intensity may be defined by training time (session length, frequency, program duration), amount of work (e.g. repetitions), and effort/energy expenditure (exertion, heart rate) (Regnaux et al 2015).

In a review of organized inpatient care for stroke, acute stroke units were categorized as ‘intensive’, ‘semi-intensive’ or ‘non-intensive’ based on whether the unit had continuous monitoring, high nurse staffing, and life support facilities (Stroke Unit Trialists Collaboration 2013).

3. Determine whether there is an existing system for grouping interventions.

In some fields, intervention taxonomies and frameworks have been developed for labelling and describing interventions, and these can make it easier for those using a review to interpret and apply findings.

Using an agreed system is preferable to developing new groupings. Existing systems should be assessed for relevance and usefulness. The most useful systems:

Systems for grouping interventions may be generic, widely applicable across clinical areas, or specific to a condition or intervention type. Some Cochrane Groups recommend specific taxonomies.

The (BCT) (Michie et al 2013) categorizes intervention elements such as goal setting, self-monitoring and social support. A protocol for a review of social media interventions used this taxonomy to describe interventions and examine different BCTs as potential effect modifiers (Welch et al 2018).

The has been used to group interventions (or components) by function (e.g. to educate, persuade, enable) (Michie et al 2011). This system was used to describe the components of dietary advice interventions (Desroches et al 2013).

Multiple reviews have used the consensus-based taxonomy developed by the Prevention of Falls Network Europe (ProFaNE) (e.g. Verheyden et al 2013, Kendrick et al 2014). The taxonomy specifies broad groups (e.g. exercise, medication, environment/assistive technology) within which are more specific groups (e.g. exercise: gait, balance and functional training; flexibility; strength and resistance) (Lamb et al 2011).

4. Plan how the specified groups will be used in synthesis and reporting.

Decide whether it is useful to pool all interventions in a single meta-analysis (‘lumping’), within which specific characteristics can be explored as effect modifiers (e.g. in subgroups). Alternatively, if pooling all interventions is unlikely to address a useful question, separate synthesis of specific interventions may be more appropriate (‘splitting’).

Determining the right analytic approach is discussed further in .

In a review of exercise for knee osteoarthritis, the different categories of exercise were combined in a single meta-analysis, addressing the question ‘what is the effect of exercise on knee osteoarthritis?’. The categories were also analysed as subgroups within the meta-analysis to explore whether the effect size varied by type of exercise (Fransen et al 2015). Other subgroup analyses examined mode of delivery and dose.

5. Decide how to group interventions with multiple components or co-interventions.

Some interventions, especially those considered ‘complex’, include multiple components that could also be implemented independently (Guise et al 2014, Lewin et al 2017). These components might be eligible for inclusion in the review alone, or eligible only if used alongside an eligible intervention.

Options for considering multi-component interventions may include the following.

The first two approaches may be challenging but are likely to be most useful (Caldwell and Welton 2016).

See Section . for the special case of when a co-intervention is administered in both treatment arms.

In a review of psychological therapies for panic disorder, two of the eight eligible therapies (psychoeducation and supportive psychotherapy) could be used alone or as part of a multi-component therapy. When accompanied by another eligible therapy, the intervention was categorized as the other therapy (i.e. psychoeducation + cognitive behavioural therapy was categorized as cognitive behavioural therapy) (Pompoli et al 2016).

In a review of psychosocial interventions for smoking cessation in pregnancy, two approaches were used. All intervention types were included in a single meta-analysis with subgroups for multi-component, single and tailored interventions. Separate meta-analyses were also performed for each intervention type, with categorization of multi-component interventions based on the ‘main’ component (Chamberlain et al 2017).

6. Build in contingencies by specifying both specific and broader intervention groups.

Consider grouping interventions at more than one level, so that studies of a broader group of interventions can be synthesized if too few studies are identified for synthesis in more specific groups. This will provide flexibility where review authors anticipate few studies contributing to specific groups (e.g. in reviews with diverse interventions, additional diversity in other PICO elements, or few studies overall, see also ).

In a review of psychosocial interventions for smoking cessation, the authors planned to group any psychosocial intervention in a single comparison (addressing the higher level question of whether, on average, psychosocial interventions are effective). Given that sufficient data were available, they also presented separate meta-analyses to examine the effects of specific types of psychosocial interventions (e.g. counselling, health education, incentives, social support) (Chamberlain et al 2017).

Clarifying role of outcomes ( )

Outcome measures should not always form part of the criteria for including studies in a review. However, some reviews do legitimately restrict eligibility to specific outcomes. For example, the same intervention may be studied in the same population for different purposes (e.g. hormone replacement therapy, or aspirin); or a review may address specifically the adverse effects of an intervention used for several conditions. If authors do exclude studies on the basis of outcomes, care should be taken to ascertain that relevant outcomes are not available because they have not been measured rather than simply not reported.

Predefining outcome domains ( )

Full specification of the outcomes includes consideration of outcome domains (e.g. quality of life) and outcome measures (e.g. SF-36). Predefinition of outcome reduces the risk of selective outcome reporting. The should be as few as possible and should normally reflect at least one potential benefit and at least one potential area of harm. It is expected that the review should be able to synthesize these outcomes if eligible studies are identified, and that the conclusions of the review will be based largely on the effects of the interventions on these outcomes. Additional important outcomes may also be specified. Up to seven critical and important outcomes will form the basis of the GRADE assessment and summarized in the review’s abstract and other summary formats, although the review may measure more than seven outcomes.

Choosing outcomes ( )

Cochrane Reviews are intended to support clinical practice and policy, and should address outcomes that are critical or important to consumers. These should be specified at protocol stage. Where available, established sets of core outcomes should be used. Patient-reported outcomes should be included where possible. It is also important to judge whether evidence of resource use and costs might be an important component of decisions to adopt the intervention or alternative management strategies around the world. Large numbers of outcomes, while sometimes necessary, can make reviews unfocused, unmanageable for the user, and prone to selective outcome reporting bias. Biochemical, interim and process outcomes should be considered where they are important to decision makers. Any outcomes that would not be described as critical or important can be left out of the review.

Predefining outcome measures ( )

C17: Predefining choices from multiple outcome measures ( )

C18: Predefining time points of interest ( )

1. Fully specify outcome domains.

For each outcome domain, provide a short label (e.g. cognition, consumer evaluation of care) and describe the domain in sufficient detail to enable eligible outcomes from each included study to be categorized. The definition should be based on the concept (or construct) measured, that is ‘what’ is measured. ‘When’ and ‘how’ the outcome is measured will be considered in subsequent steps.

Outcomes can be defined hierarchically, starting with very broad groups (e.g. physiological/clinical outcomes, life impact, adverse events), then outcome domains (e.g. functioning and perceived health status are domains within ‘life impact’). Within these may be narrower domains (e.g. physical function, cognitive function), and then specific outcome measures (Dodd et al 2018). The level at which outcomes are grouped for synthesis alters the question addressed, and so decisions should be guided by the review objectives.

In specifying outcome domains:

In a review of computer-based interventions for sexual health promotion, three broad outcome domains were defined (cognitions, behaviours, biological) based on a conceptual model of how the intervention might work. Each domain comprised more specific domains and outcomes (e.g. condom use, seeking health services such as STI testing); listing these helped define the broad domains and guided categorization of the diverse outcomes reported in included studies (Bailey et al 2010).

In a protocol for a review of social media interventions for improving health, the rationale for synthesizing broad groupings of outcomes (e.g. health behaviours, physical health) was based on prediction of a common underlying mechanism by which the intervention would work, and the review objective, which focused on overall health rather than specific outcomes (Welch et al 2018).

2. Determine whether there is an existing system for identifying and grouping important outcomes.

Systems for categorizing outcomes include core outcome sets including the and initiatives, and outcome taxonomies (Dodd et al 2018). These systems define agreed outcomes that should be measured for specific conditions (Williamson et al 2017).These systems can be used to standardize the varied outcome labels used across studies and enable grouping and comparison (Kirkham et al 2013). Agreed terminology may help decision makers interpret review findings.

The COMET website provides a database of core outcome sets agreed or in development. Some Cochrane Groups have developed their own outcome sets. While the availability of outcome sets and taxonomies varies across clinical areas, several taxonomies exist for specifying broad outcome domains (e.g. Dodd et al 2018, ICHOM 2018).

In a review of combined diet and exercise for preventing gestational diabetes mellitus, a core outcome set agreed by the Cochrane Pregnancy and Childbirth group was used (Shepherd et al 2017).

In a review of decision aids for people facing health treatment or screening decisions (Stacey et al 2017), outcome domains were based on criteria for evaluating decision aids agreed in the (IPDAS). Doing so helped to assess the use of aids across diverse clinical decisions.

The Cochrane Consumers and Communication Group has an agreed taxonomy to guide specification of outcomes of importance in evaluating communication interventions (Cochrane Consumers & Communication Group).

3. Define the outcome time points.

A key attribute of defining an outcome is specifying the time of measurement. In reviews, time frames, and not specific time points, are often specified to handle the likely diversity in timing of outcome measurement across studies (e.g. a ‘medium-term’ time frame might be defined as including outcomes measured between 6 and 12 months).

In specifying outcome timing:

In a review of psychological therapies for panic disorder, the main outcomes were ‘short-term’ (≤6 months from treatment commencement). ‘Long-term’ outcomes (>6 months from treatment commencement) were considered important, but not specified as critical because of concerns of participant attrition (Pompoli et al 2018).

In contrast, in a review of antidepressants, a clinically meaningful time frame of 6 to 12 months might be specified for the critical outcome ‘depression’, since this is the recommended treatment duration. However, it may be anticipated that many studies will be of shorter duration with short-term follow-up, so an additional important outcome of ‘depression (<3 months)’ might also be specified.

4. Specify the measurement tool or measurement method.

For each outcome domain, specify:

Minimum criteria for inclusion of a measure may include:

Measures may be identified from core outcome sets (e.g. Williamson et al 2017, ICHOM 2018) or systematic reviews of instruments (see COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) initiative for a database of examples).

In a review of interventions to support women to stop smoking, objective (biochemically validated) and subjective (self-report) measures of smoking cessation were specified separately to examine bias due to the method used to measure the outcome (Step 6) (Chamberlain et al 2017).

In a review of high-intensity versus low-intensity exercise for osteoarthritis, measures of pain were selected based on relevance of the content and properties of the measurement tool (i.e. evidence of validity and reliability) (Regnaux et al 2015).

5. Specify how multiplicity of outcomes will be handled.

For a particular domain, multiple outcomes within a study may be available for inclusion. This may arise from:

Effects of the intervention calculated from these different sources of multiplicity are statistically dependent, since they have been calculated using the same participants. To deal with this dependency, select only one outcome per study for a particular comparison, or use a meta-analysis method that accounts for the dependency (see Step 6).

Pre-specify the method of selection from multiple outcomes or measures in the protocol, using an approach that is independent of the result (see ) (López-López et al 2018). Document all eligible outcomes or measures in the ‘Characteristics of included studies’ table, noting which was selected and why.

Multiplicity can arise from the reporting of multiple analyses of the same outcome (e.g. analyses that do and do not adjust for prognostic factors; intention-to-treat and per-protocol analyses) and multiple reports of the same study (e.g. journal articles, conference abstracts). Approaches for dealing with this type of multiplicity should also be specified in the protocol (López-López et al 2018).

It may be difficult to anticipate all forms of multiplicity when developing a protocol. Any post-hoc approaches used to select outcomes or results should be noted at the beginning of the Methods section, or if extensive, within an additional supplementary material.

The following hierarchy was specified to select one outcome per domain in a review examining the effects of portion, package or tableware size (Hollands et al 2015):

Selection of the outcome was made blinded to the results. All available outcome measures were documented in the ‘Characteristics of included studies’ table.

In a review of audit and feedback for healthcare providers, the outcome domains were ‘provider performance’ (e.g. compliance with recommended use of a laboratory test) and ‘patient health outcomes’ (e.g. smoking status, blood pressure) (Ivers et al 2012). For each domain, outcomes were selected using the following hierarchy:

6. Plan how the specified outcome domains will be used in the synthesis.

When different measurement methods or tools have been used across studies, consideration must be given to how these will be synthesized. Options include the following.

In a review of interventions to support women to stop smoking, separate outcome domains were specified for biochemically validated measures of smoking and self-report measures. The two domains were meta-analysed together, but sensitivity analyses were undertaken restricting the meta-analyses to studies with only biochemically validated outcomes, to examine if the results were robust to the method of measurement (Chamberlain et al 2017).

In a review of psychological therapies for youth internalizing and externalizing disorders, most studies contributed multiple effects (e.g. in one meta-analysis of 443 studies, there were 5139 included measures). The authors used multilevel modelling to address the dependency among multiple effects contributed from each study (Weisz et al 2017).

7. Where possible, build in contingencies by specifying both specific and broader outcome domains.

Consider building in flexibility to group outcomes at different levels or time intervals. Inflexible approaches can undermine the potential to synthesize, especially when few studies are anticipated, or there is likely to be diversity in the way outcomes are defined and measured and the timing of measurement. If insufficient studies report data for meaningful synthesis using the narrower domains, the broader domains can be used (see also ).

Consider a hypothetical review aiming to examine the effects of behavioural psychological interventions for the treatment of overweight and obese adults. A specific outcome is body mass index (BMI). However, also specifying a broader outcome domain ‘indicator of body mass’ will facilitate synthesis in the circumstance where few studies report BMI, but most report an indicator of body mass (such as weight or waist circumference). This is particularly important when few studies may be anticipated or there is expected diversity in the measurement methods or tools.

Predefining study designs ( )

Predefined, unambiguous eligibility criteria are a fundamental prerequisite for a systematic review. This is particularly important when non-randomized studies are considered. Some labels commonly used to define study designs can be ambiguous. For example a ‘double blind’ study may not make it clear who was blinded; a ‘case-control’ study may be nested within a cohort, or be undertaken in a cross-sectional manner; or a ‘prospective’ study may have only some features defined or undertaken prospectively.

Justifying choice of study designs ( )

It might be difficult to address some interventions or some outcomes in randomized trials. Authors should be able to justify why they have chosen either to restrict the review to randomized trials or to include non-randomized studies. The particular study designs included should be justified with regard to appropriateness to the review question and with regard to potential for bias.

Including randomized trials ( )

if it is feasible to conduct them to evaluate the interventions and outcomes of interest.

Randomized trials are the best study design for evaluating the efficacy of interventions. If it is feasible to conduct them to evaluate questions that are being addressed by the review, they must be considered eligible for the review. However, appropriate exclusion criteria may be put in place, for example regarding length of follow-up.

Excluding studies based on publication status ( )

Obtaining and including data from unpublished studies (including grey literature) can reduce the effects of publication bias. However, the unpublished studies that can be located may be an unrepresentative sample of all unpublished studies.

Chronic disease

Patient-led care models

Peters MDJ, Godfrey C, McInerney P, Munn Z, Tricco AC, Khalil, H. Chapter 11: Scoping Reviews (2020 version). In: Aromataris E, Munn Z (Editors). JBI Manual for Evidence Synthesis, JBI, 2020. Available from   https://synthesismanual.jbi.global  .    https://doi.org/10.46658/JBIMES-20-12

A model useful for qualitative and mixed method type research questions.

Example question: What are young parents’ experiences of attending antenatal education? (Cooke et al., 2012)

Cooke, A., Smith, D. and Booth, A. (2012) 'Beyond PICO: the SPIDER tool for qualitative evidence synthesis.' Qualitative Health Research , 22(10) pp. 1435-1443

A model useful for qualitative and mixed method type research questions. 

Example question: How effective is mindfulness used as a cognitive therapy in a counseling service in improving the attitudes of patients diagnosed with cancer?

Example question taken from: Tate, KJ., Newbury-Birch, D., and McGeechan, GJ. (2018) ‘A systematic review of qualitative evidence of  cancer patients’ attitudes to mindfulness.’ European Journal of Cancer Care , 27(2) pp. 1 – 10.

A model useful for qualitative and mixed method type research questions, especially for question examining particular services or professions.

Example question: Cross service communication in supporting adults with learning difficulties

You might find that your topic does not always fall into one of the models listed on this page. You can always modify a model to make it work for your topic, and either remove or incorporate additional elements.

The important thing is to ensure that you have a high quality question that can be separated into its component parts.

• << Previous: Timescales and processes
• Next: Inclusion and exclusion criteria >>
• Last Updated: Sep 12, 2024 11:12 AM
• URL: https://plymouth.libguides.com/systematicreviews

• The Hong Kong Polytechnic University
• Guides & Tutorials

Systematic Search for Systematic Review

• Formulate Research Question Using PICO
• Introduction
• Find Systematic Reviews (SR)
• Databases Selection for Conducting SR
• Step 1. Set Preferences in EndNote
• Step 2. Create Groups in EndNote
• Step 3. Export Search Results from Databases to EndNote
• Step 4. Add Name of Database to References
• Step 5. Remove Duplicate Records
• Step 6. Share References with Teammates
• Step 7. Find Full Text Articles
• [Optional] Export References to Excel
• Related Library Workshops & Courses
• FAQs on Literature Search for Conducting SR

Worksheets for Documenting & Reporting Search Process

Here are some resources for you to document and report your search process in a systematic review. 

• Workbook for documenting systematic search
• PRISMA Flow Diagram A flow diagram to depict the flow of information through the different phases of a systematic review. It maps out the number of records identified, included and excluded, and the reasons for exclusions.

Understanding SR

• What are systematic reviews? (Cochrane)
• Intro to Systematic Reviews & Meta-Analyses
• Using PICO to formulate a search question   (CEBM)
• Turning search terms into a search   (CEBM)
• Turning your search strategy into results: PubMed demonstration   (CEBM)

Understanding study design

• What is a randomised trial?
• Epidemiology Study Types: Randomized Control Trial
• Epidemiology Study Types: Cohort and Case-Control
• Cohort, Case-Control, Meta-Analysis, Cross-sectional Study Designs & Definition 

Copyright Disclaimer

Except where otherwise noted, the content of this guide is licensed under a  CC BY-NC 4.0 License .

A systematic review aims to answer a specific research (clinical) question. A well-formulated question will guide many aspects of the review process, including determining eligibility criteria, searching for studies, collecting data from included studies, and presenting findings ( Cochrane Handbook , Sec. 5.1.1).

To define a  researchable  question, the most commonly used structure is  PICO , which specifies the type of P atient or P opulation, type of I nterventions (and C omparisons if there is any), and the type of O utcomes that are of interest. 

The table below gives an example on how a research question is framed using the PICO structure. You may also use the PICO components to write the objective and title of your review, and later to structure your inclusion and exclusion criteria for study selection. This ensures that the whole review process is guided by your research question. 

Type of Question and Study Design

While formulating your research question, it's also important to consider the  type of question  you are asking because this will affect the type of studies (or study design ) to be included in your review.

Each type of question defines its type of studies in order to provide the best evidence. For example, to answer a therapeutic question, you need to include as many Randomized Controlled Trials (RCTs) as possible, because RCTs are considered to have the highest  level of evidence  (least bias) for solving a therapeutic problem. 

The table below suggests the best designs for specific type of question. The Level of Evidence pyramid, which is widely adopted in the medical research area, shows a hierarchy of the quality of medical research evidence in different type of studies ( Level of Evidence (2011), Oxford Centre for Evidence-based Medicine, CEBM ).

Usually, the study design of a research work will be clearly indicated either in its title or abstract, especially for RCT. Some databases also allow to search or refine results to one or a few study designs, which helps you locate as many as possible the relevant studies. If you are not sure the study design of a research work, refer to this brief guide for spotting study designs  (by CEBM).

Learn to Build a Good Clinical Question

Learn to build a good clinical question  from this  EBP Tutorial: Module 1:  "Introduction to Evidence-Based Practice"

It is provided by Duke University and University of North Carolina at Chapel Hill, USA.

PICO Framework and the Question Statement The above named section  in the Library guide:  Evidence-Based Practice in Health , provided by the University of Canberra Library, explains the PICO framework with examples and in various question types.

Documenting Your Search Process

Systematic review requires a detailed and structured reporting of the search strategy and selection criteria used in the review. Therefore we strongly advise you to document your search process from the very beginning. You may use this workbook  to help you with the documentation.

The documentation should include:

• Research concepts in PICO structure and research question ,
• Type of studies you intend to include, and
• Inclusion and exclusion criteria in PICO structure

and the whole search process, including:

• Databases searched (hosting platforms) , including journals and other sources covered in handsearching
• Date of search
• Search strategy , including keywords and subject headings used, the combination of searches (usually copy-paste from database search page)
• Filters used in initial search or refine results, including year coverage, type of studies, age, etc.
• Number of results retrieved after each search and refinement in each database
• Total number of results from all databases searched
• Duplicates identified from all results
• Number of results with full text

Eventually, you will need to include the information above when you start writing your review. A highly recommended structure for reporting the search process is the PRISMA Flow Diagram . You may also use PRISMA Flow Diagram Generator to generate a diagram in a different format (based on your input). 

• << Previous: Find Systematic Reviews (SR)
• Next: Databases Selection for Conducting SR >>
• Last Updated: Sep 6, 2024 5:03 PM
• URL: https://libguides.lb.polyu.edu.hk/syst_review

• Help and Support
• Research Guides

Systematic Reviews - Research Guide

• Defining your review question
• Starting a Systematic Review
• Developing your search strategy
• Where to search
• Appraising Your Results
• Documenting Your Review
• Find Systematic Reviews
• Software and Tools for Systematic Reviews
• Guidance for conducting systematic reviews by discipline
• Library Support

Review question

A systematic review aims to answer a clear and focused clinical question. The question guides the rest of the systematic review process. This includes determining inclusion and exclusion criteria, developing the search strategy, collecting data and presenting findings. Therefore, developing a clear, focused and well-formulated question is critical to successfully undertaking a systematic review.

 A good review question:

• allows you to find information quickly
• allows you to find relevant information (applicable to the patient) and valid (accurately measures stated objectives)
• provides a checklist for the main concepts to be included in your search strategy.

How to define your systematic review question and create your protocol

• Starting the process
• Defining the question
• Creating a protocol

Types of clinical questions

• PICO/PICo framework
• Other frameworks

Research topic vs review question

A  research topic   is the area of study you are researching, and the  review question   is the straightforward, focused question that your systematic review will attempt to answer. 

Developing a suitable review question from a research topic can take some time. You should:

• perform some scoping searches
• use a framework such as PICO  
• consider the FINER criteria; review questions should be  F easible, I nteresting, N ovel, E thical and R elevant
• check for existing or prospective systematic reviews.

When considering the feasibility of a potential review question, there should be enough evidence to answer the question whilst ensuring that the quantity of information retrieved remains manageable. A scoping search will aid in defining the boundaries of the question and determining feasibility.

For more information on FINER criteria in systematic review questions, read Section 2.1 of the Cochrane Handbook .

Check for existing or prospective systematic reviews

Before finalising your review question, you should determine if any other systematic review is in progress or has been completed on your intended question (i.e. consider if the review is N ovel).

To find systematic reviews you might search specialist resources such as the Cochrane Library , Joanna Briggs Institute EBP Database  or the Campbell Collaboration . "Systematic review" can also be used as a search term or limit when searching the recommended databases .

You should appraise any systematic reviews you find to assess their quality. An article may include ‘systematic review’ in its title without correctly following the systematic review methodology. Checklists, including those developed by AMSTAR and JBI , are useful tools for appraisal.

You may undertake a review on a similar question if that posed by a previously published review had issues with its methodology such as not having a comprehensive search strategy, for example. You may choose to narrow the parameters of a previously conducted search or to update the review if it was published some years ago. 

Searching a register of prospective systematic reviews such as PROSPERO  will allow you to check that you are not duplicating research already underway.

Once you have performed scoping searches and checked for other systematic reviews on your topic, you can focus and refine your review question. Any PICO elements identified during the initial development of the review question from the research topic should now be further refined.

The review question should always be:

• unambiguous
• structured.

Review questions may be broad or narrow in focus; however, you should consider the FINER criteria when determining the breadth of the PICO elements of your review question.

A question that is too broad may present difficulty with searching, data collection, analysis, and writing, as the number of studies retrieved would be unwieldy. A broad review question could be more suited to another type of review .

A question that is too narrow may not have enough evidence to allow you to answer your review question. Table 2.3.a in the Cochrane Handbook summarises the advantages and disadvantages of broad versus narrow reviews and provides examples of how you could broaden or narrow different PICO elements.

It is essential to formulate your research question with care to avoid missing relevant studies or collecting a potentially biased result set.

A systematic review protocol is a document that describes the rationale, question, and planned methods of a systematic review. Creating a protocol is an essential part of the systematic review process, ensuring careful planning and detailed documentation of what is planned before undertaking the review.

The Preferred Reporting Items for Systematic review and Meta-Analysis Protocols (PRISMA-P) checklist outlines recommended items to address in a systematic review protocol, including:

• review question, with PICO elements defined
• eligibility criteria 
• information sources (e.g. planned databases, trial registers, grey literature sources, etc.)
• draft search strategy. 

Systematic reviews must have pre-specified criteria for including and excluding studies in the review. The Cochrane Handbook states that "predefined, unambiguous eligibility criteria are a fundamental prerequisite for a systematic review." 

The first step in developing a protocol is determining the PICO elements   of the review question and how the intervention produces the expected outcomes in the specified population. You should then specify the types of studies   that will provide the evidence to answer your review question. Then outline the inclusion and exclusion criteria based on these PICO elements.

For more information on defining eligibility criteria, see Chapter 3 of the Cochrane Handbook .

A key purpose of a protocol is to make plans to minimise bias in the findings of the review; where possible, changes should not be made to the eligibility criteria of a published protocol. Where such changes are made, they must be justified and documented in the review. Appropriate time and consideration should be given to creating the protocol.

You may wish to register your protocol in a publicly accessible way. This will help prevent other people from completing a review on your topic.

If you intend to publish a systematic review in the health sciences, it should conform to the IOM Standards for Reporting Systematic Reviews .

If you intend to publish a systematic review in the Cochrane Database of Systematic Reviews , it should conform to the Methodological Expectations in Cochrane Intervention Review s (MECIR).

A clinical question needs to be directly relevant to the patient or problem and phrased to facilitate the search for an answer. A clear and focused question is more likely to lead to a credible and useful answer, whereas a poorly formulated question can lead to an uncertain answer and create confusion.

The population and intervention should be specific, but if any or both are described too narrowly, it may not be easy to find relevant studies or sufficient data to demonstrate a reliable answer.

PICO is a framework for developing a focused clinical question. 

Slightly different versions of this concept are used to search for   quantitative   and   qualitative reviews, examples are given below:

PICO for quantitative studies

Here is an example of a clinical question that outlines the PICO components:

PICo for qualitative studies

Here is an example of a clinical question that outlines the PICo components:

Two other mnemonics may be used to frame questions for qualitative and quantitative studies -  SPIDER   and  SPICE .

SPIDER for qualitative or quantitative studies

SPIDER   can be used for both qualitative and quantitative studies:

Within social sciences research,  SPICE  may be more appropriate for formulating research questions:

More question frameworks

For more question frameworks, see the following:

• Table 1 Types of reviews , from ' What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences. '
• Framing your research question , CQ University 
• Asking focused questions - Centre for Evidence Based Medicine Tips and examples for formulating focused questions
• Cochrane Handbook, Chapter 2: Determining the scope of the review and the questions it will address Discusses the formulation of review questions in detail
• PICO for Evidence-Based Veterinary Medicine EBVM Toolkit from the RCVS
• PICO worksheet
• PICo worksheet
• << Previous: Starting a Systematic Review
• Next: Developing your search strategy >>
• Last Updated: Aug 7, 2024 10:22 AM
• URL: https://libguides.murdoch.edu.au/systematic

Systematic & scoping reviews

Why use pico.

Systematic reviews require focused clinical questions. PICO is a useful tool for formulating such questions. For information on PICO and other frameworks please see our tutorial below.

Systematic Reviews: Formulating the Research Question [PDF, 191kB]

This PowerPoint covers:

• Formulating the research question
• The PICO framework
• Types of questions
• Types of studies
• Qualitative questions

PICO example for quantitative studies

The PICO (Patient, Intervention, Comparison, Outcome) framework is commonly used to develop focused clinical questions for quantitative systematic reviews.

Sample topic:

In middle aged women suffering migraines, is Botulinium toxin type A compared to placebo effective at decreasing migraine frequency?

P - Middle aged women suffering migraines

I - Botulinium toxin type A

C - Placebo

O - Decreased migraine frequency

Use the following worksheet to complete a search strategy:

PICO SR worksheet [DOCX, 17kB]

Completed PICO SR worksheet [PDF, 114kB]

PICo, SPICE or SPIDER example for qualitative studies

The PICO (Patient, Intervention, Comparison, Outcome) framework is commonly used to develop focused clinical questions for quantitative systematic reviews. A modified version, PICo , can be used for qualitative questions.

What are caregivers’ experiences with providing home-based care to patients with HIV/AIDS in Africa?

P - Caregivers providing home-based care to persons with HIV/AIDS

I - Experiences

Co - Africa

Use the following worksheet to create a search strategy:

PICo qualitative worksheet [DOCX, 18kB]

Completed PICo Qualitative worksheet [PDF, 115kB]

The SPIDER framework is an alternative search strategy tool (based on PICo) for qualitative/mixed methods research.

What are the experiences of women undergoing IVF treatment?

PI - IVF treatment

D - Questionnaire or survey or interview

E - Experiences or views or attitudes or feelings

R - Qualitative or mixed method

Cooke, A., Smith, D., & Booth, A. (2012). Beyond PICO: The SPIDER tool for qualitative evidence synthesis

Methley, A. M., Campbell, S., Chew-Graham, C., McNally, R., & Cheraghi-Sohi, S. (2014). PICO, PICOS and SPIDER: A comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews

SPICE can be used for both qualitative and quantitative studies. SPICE stands for S etting (where?), P erspective (for whom?), I ntervention (what?), C omparison (compared with what?) and E valuation (with what result?).

What are the coping skills of parents of children with autism undergoing behavioural therapy in schools?

S - Schools

P - Parents of children with autism

I - Behavioural therapy

E - Coping skills

Booth, A. (2006). Clear and present questions: Formulating questions for evidence based practice

Systematic Reviews

• Introduction & Review Types

The PICO Framework

Pico for quantitative studies, pico for qualitative studies, other frameworks (for qualitative studies), further reading.

• Searching in Bibliographic Databases
• Grey Literature
• Developing a Protocol
• Reference Management
• PRESS - Checklist for Search Strategies

The first key stage of a systematic review would be to formulate a focused, answerable research question . A well-defined and clear research question is an essential starting point for a systematic search. To create a logical search strategy, always start by identifying the key elements of the research question - i.e., establish what the main concepts of the topic are. With these concepts, you can then create the search blocks that form the basis for the search strategies used in the different databases.

Use the PICO framework to translate the research question into search concepts that can be applied in a structured search strategy. In general, you should not use all parts of the PICO question in the search. The key focus of the search would be generally on the P (Population / Patient / Problem) and the I (Intervention), and sometimes C (Comparison intervention) if the volume of search results is too great, or the concept is exceptionally clear and regularly reported.     

Karolinska Institutet University Library (2022). Systematic reviews [ Systematic Search Technique s] :  https://kib.ki.se/en/search-evaluate/systematic-reviews

An example of the use of PICO (Quantitative)

Formulate a clear and focused PICO question. An example of an initial unfocused question would be: Is caffeine effective in preventing daytime drowsiness (DTD)?

A focused clinical question would be: Among adults with a history of DTD, does a cup of caffeinated coffee in the morning improve alertness? A question is made focused by clearly specifying the PICO elements...

Dr Lorraine Tudor Car.  1.2 Focused Question & the Parallel Group RCT Design.

Murdoch University Library. [2022]. Using PICO or PICo - Systematic Reviews:  https://libguides.murdoch.edu.au/systematic/PICO .

An example of a qualitative research question: Do mindfulness programs improve the academic, behavioral, and socio-emotional functioning of primary and secondary students?

Let's apply this to the SPIDER framework.

Another example of a qualitative research question: What is the effect of climate change on the seed quality of legume crops?

Let's apply this to the SPICE framework.

Steven Chang (La Trobe University).  Systematic Searching for Systematic Reviews.

St Joseph's Health Centre Toronto. Study Design [Matching Question Types with Study Design].  https://library.stjoestoronto.ca/home/ebm/studydesign

Elsevier Author Services. FINR : a Research Framework.  https://scientific-publishing.webshop.elsevier.com/research-process/finer-research-framework/

• << Previous: Systematic Search Techniques
• Next: Searching in Bibliographic Databases >>
• Last Updated: Jun 28, 2024 12:51 PM
• URL: https://libguides.ntu.edu.sg/systematic-reviews

You are expected to comply with University policies and guidelines namely, Appropriate Use of Information Resources Policy , IT Usage Policy and Social Media Policy . Users will be personally liable for any infringement of Copyright and Licensing laws. Unless otherwise stated, all guide content is licensed by CC BY-NC 4.0 .

Systematic Reviews: Formulate your question and protocol

• Formulate your question and protocol
• Developing the review protocol
• Searching for evidence
• Search strategy
• Managing search results
• Evaluating results (critical appraisal)
• Synthesising and reporting
• Further resources

This video illustrates how to use the PICO framework to formulate an effective research question, and it also shows how to search a database using the search terms identified. The database used in this video is CINAHL but the process is very similar in databases from other companies as well.

Recommended Reading

• BMJ Best Practice Advice on using the PICO framework.

A longer on the important pre-planning and protocol development stages of systematic reviews, including tips for success and pitfalls to avoid. 

* You can start watching this video from around the 9 minute mark.*

Formulate Your Question

Having a focused and specific research question is especially important when undertaking a systematic review. If your search question is too broad you will retrieve too many search results and you will be unable to work with them all. If your question is too narrow, you may miss relevant papers. Taking the time to break down your question into separate, focused concepts will also help you search the databases effectively.

Deciding on your inclusion and exclusion criteria early on in the research process can also help you when it comes to focusing your research question and your search strategy.

A literature searching planning template can help to break your search question down into concepts and to record alternative search terms. Frameworks such as PICO and PEO can also help guide your search. A planning template is available to download below, and there is also information on PICO and other frameworks ( Adapted from: https://libguides.kcl.ac.uk/systematicreview/define).

Looking at published systematic reviews can give you ideas of how to construct a focused research question and an effective search strategy.

Example of an unfocused research question: How can deep vein thrombosis be prevented?

Example of a focused research question: What are the effects of wearing compression stockings versus not wearing them for preventing DVT in people travelling on flights lasting at least four hours.

In this Cochrane systematic review by Clarke et al. (2021), publications on randomised trials of compression stockings versus no stockings in passengers on flights lasting at least four hours were gathered. The appendix of the published review contains the comprehensive search strategy used.  This research question has focused on a particular method (wearing compression stockings) in a particular setting (flights of at least 4 hrs) and included only specific studies (randomised trails). An additional way of focusing a question could be to look at a particular section of the population.

Clarke  M. J., Broderick  C., Hopewell  S., Juszczak  E., and Eisinga  A., 20121. Compression stockings for preventing deep vein thrombosis in airline passengers. Cochrane Database of Systematic Reviews 2021, Issue 4. Art. No.: CD004002  [Accessed 30th April 2021].  Available from: 10.1002/14651858.CD004002.pub4

There are many different frameworks that you can use to structure your research question with clear parameters. The most commonly used framework is PICO:

• Population This could be the general population, or a specific group defined by: age, socioeconomic status, location and so on.
• Intervention This is the therapy/test/strategy to be investigated and can include medication, exercise, environmental factors, and counselling for example. It may help to think of this as 'the thing that will make a difference'.
• Comparator This is a measure that you will use to compare results against. This can be patients who received no treatment or a placebo, or people who received alternative treatment/exposure, for instance.
• Outcome What outcome is significant to your population or issue? This may be different from the outcome measures used in the studies.

Adapted from:  https://libguides.reading.ac.uk/systematic-review/protocol

• Developing an efficient search strategy using PICO A tool created by Health Evidence to help construct a search strategy using PICO

Other Frameworks: alternatives to PICO

As well as PICO, there are other frameworks available, for instance:

• PICOT : Population, Intervention, Comparison, Outcome, Time.
• PEO: Population and/or Problem, Exposures, Outcome
• SPICE: Setting, Population or Perspective, Intervention, Comparison, Evaluation
• ECLIPS: Expectations, Client Group, Location, Impact, Professionals Involved, Service
• SPIDER: Sample, Phenomenon of interest, Design, Evaluation, Research type

This page from City, University of London, contains useful information on several frameworks, including the ones listed above.

Develop Your Protocol

Atfer you have created your research question, the next step is to develop a protocol which outlines the study methodology. You need to include the following:

• Research question and aims
• Criteria for inclusion and exclusion
• search strategy
• selecting studies for inclusion
• quality assessment
• data extraction & analysis
• synthesis of results
• dissemination

To find out how much has been published on a particular topic, you can perform scoping searches in relevant databases. This can help you decide on the time limits of your study.

• Systematic review protocol template This template from the University of Reading can help you plan your protocol.
• Protocol Guidance This document from the University of York describes what each element of your protocol should cover.

Register Your Protocol

It is good practice to register your protocol and often this is a requirement for future publication of the review.

You can register your protocol here:

• PROSPERO: international prospective register of systematic review
• Cochrane Collaboration, Getting Involved
• Campbell Collaboration, Co-ordinating Groups

Adapted from:   https://libguides.bodleian.ox.ac.uk/systematic-reviews/methodology

• << Previous: Home
• Next: Developing the review protocol >>
• Last Updated: Aug 8, 2024 4:21 PM
• URL: https://libguides.qmu.ac.uk/systematic-reviews

• Library Hours
• (314) 362-7080
• [email protected]
• Library Website
• Electronic Books & Journals
• Database Directory
• Catalog Home
• Library Home

Systematic Review Guide

• Preliminary Search
• PICO format
• Full Text Retrieval
• Covidence Screening Application
• Evidence Grading & Appraisal
• Risk of Bias Tools
• Manuscript Preparation
• Other Review Types

On the Systematic Review Request form you will be asked to outline your research question in PICO format. This allows us to easily understand the main concepts of your research question. Here is what PICO stands for:

P = Problem/Population

I = Intervention (or the experimental variable)

C = Comparison (or the control variable) [Optional]

O = Outcome

If your research question does not fit neatly into PICO that is okay. Just try to include the elements of your question as closely as possible into the format. Your collaborating librarian will discuss any questions or concerns about your research topic before putting together your systematic review search strategy.

• << Previous: Preliminary Search
• Next: Full Text Retrieval >>
• Last Updated: May 14, 2024 1:58 PM
• URL: https://beckerguides.wustl.edu/SystematicReviews

PICO Framework for Systematic Reviews: A Comprehensive Overview

What is a systematic review.

Before diving into the meaning of a PICO framework systematic review, it is important to have a quick overview of the meaning of a systematic review. You have probably come across the hierarchy of evidenc e, which is used to rank the strength of evidence obtained from research studies to power clinical decision-making. Systematic reviews (and meta-analyses) are usually placed at the top of the pyramid as the highest level of evidence. For this reason, many universities are accepting systematic reviews and meta-analyses for undergraduate, masters, and PhD dissertations. Similarly, many reputable academic journals are accepting systematic reviews and meta-analyses as a source of credible evidence for publications.

Table of Contents

A systematic review is the highest level of evidence, especially in clinical sciences. It aims to answer focused clinical questions to power evidence-based clinical decision-making by collating findings from various empirical studies. These empirical studies must meet a pre-specified eligibility criteria. Therefore, when doing a systematic review, the first step is to formulate a research question . That is where the idea of a PICO framework emerges.

What is PICO Framework Systematic Review?

In summary, when formulating a research question to be answered using a systematic review methodology, you must follow a specified structure. One of the frameworks commonly employed in formulating systematic review research questions is the PICO framework. Similarly, the empirical studies whose findings are collated in a systematic review must meet pre-specified eligibility criteria. Also, the PICO framework is used in guiding the formulation of these eligibility criteria. Therefore, before describing the specific meaning of the PICO framework, it is important to note its two functions in systematic reviews, namely (a) framing research questions and (b) formulating eligibility criteria. The PICO framework is recommended by the Cochrane Collaboration in framing research questions and formulating eligibility criteria.

A PICO Framework systematic review refers to a systematic review or meta-analysis that relies on the PICO process.

The Elements of the PICO Framework

The “P” element of PICO refers to the target population of your research. For example, you could be focusing on adults diagnosed with diabetes. In your research question, you must indicate that you systematic review is focusing on adults diagnosed with diabetes. Similarly, in your eligibility criteria, you must indicate that you’re selecting only empirical studies that used adults diagnosed with diabetes in their sample. Thus, studies that use a sample of children or adolescents diagnosed with diabetes may not meet your eligibility criteria.

Intervention

The “I” element of PICO defines the intervention whose effects you’re investigating in your systematic review. For example, as a clinician or researcher, you might be interested in understanding the effectiveness of interventions derived from self-determination theory for adults diagnosed with diabetes . In this regard, the specific interventions are not yet identified. In that case, you can add an objective to your systematic review about identifying or summarizing interventions derived from the self-determination theory. Therefore, in your eligibility criteria, you must also mention that you’re only focusing on interventions derived from the self-determination theory. In other words, studies to be selected in your systematic review must focus on investigating such interventions.

The “C” element of a PICO framework outlines the comparison. In this case, the comparison refers to the alternative that you’re comparing your intervention against. Take the example of a systematic review investigating the effectiveness of interventions derived from the self-determination theory. For you to determine whether such interventions are effective, you can compare them with usual or routine care. If the intervention shows better effectiveness than usual or routine care, you can recommend it in your systematic review.

Finally, the “O” element of the PICO framework refers to the outcome, which defines the specific outcome or result you want to measure. In the case of a systematic review focusing on adult patients diagnosed with diabetes, interventions derived from self-determination theory, and usual treatment as the comparison, the outcome can include self-management. In other words, you want to determine the extent to which the intervention improves self-management among adult patients diagnosed with diabetes.

Framing a Research Question in a PICO Framework Systematic Review

In systematic reviews and meta-analyses, the PICO framework is commonly used to frame a research question. To do so, you must start by defining the PICO elements. In most cases, the population of interest is known by a clinician or researcher. The intervention may be known or unknown. The comparison is also usually known, in most cases determined as “usual care” in various healthcare settings. Finally, in most cases, the outcome(s) is unknown, but it can also be pre-defined.

What I mean in this case is that, sometimes the researcher or clinician may be interested in identifying interventions (unknown) that bring about a certain outcome or results in a given target population. Since the intervention is unknown, the researcher can formulate the research question to imply that they are investigating interventions that bring about a certain effect. For example, “In adults diagnosed with diabetes (P), which interventions (I), compared to usual care (C), can improve self-management? (O)” In such a research question, the researcher or clinician knows the population, the comparison, and the outcome, but does not know the interventions, which will be the focus on their systematic review.

However, in most cases, the researcher knows the population, intervention, and comparison, but does not know the outcomes. For instance, “In adults diagnosed with diabetes (P), compared to usual care (C), what is the effectiveness of self-determination-based interventions (I) in improving self-management?” In this case, the researcher does not know whether the identified interventions improves self-management or not. It is important noting that any element of the PICO framework can be unknown.

Formulating Eligibility Criteria in a PICO Systematic Review

Apart from framing a research question, the PICO framework can also be used in formulating eligibility criteria for systematic reviews. Take the example of a systematic review focusing on the effectiveness of self-determination theory-based interventions for improving self-management in adults diagnosed with diabetes. The eligibility criteria derived from the PICO framework can be as follows:

• P opulation – Adult patients diagnosed with diabetes. Therefore, studies using a sample of children or adolescents were excluded in the systematic review.
• I ntervention – Studies investigating the effectiveness of interventions derived from self-determination theories. Studies investigating interventions derived from other theoretical or conceptual models other than the self-determination theory will be excluded.
• C omparison – Studies comparing the intervention with usual care or routine management of adult patients with diabetes. In most cases, researchers are lenient defining the specific comparison used in studies.
• O utcome – Studies investigating the effectiveness of the intervention in improving self-management in adult patients diagnosed with diabetes. Studies not reporting on this outcome will be excluded from the meta-analysis.

The four elements above are usually combined with study design, year of publication, and article language to formulate a complete eligibility criteria in PICO framework systematic reviews. The PICO framework guides the formulation of an eligibility criteria that can be used to select empirical studies that can answer a specific research question for clinical practice decision-making.

The PICO Framework for a Literature Search Strategy

Finally, the PICO framework can be used in developing a literature search strategy. However, its use for this purpose is highly contentious. Some argue that when developing a search strategy, the outcomes should not be captured because they can lower the retrieval potential of that search strategy. Instead, the keywords used should only focus on the population and intervention elements. The outcomes will be determined when screening the articles for eligibility. Because of such disagreements among scholars, we did not include the development of a literature search strategy as one of the functions of a PICO framework in systematic reviews. Even so, we heavily recommend experimenting first. For example, you can harvest keywords relevant to each of the PICO elements. In the first search round, include keywords for population and intervention elements only. In the second round, introduce keywords from other PICO elements.

Research Question Types Suitable for PICO Systematic Reviews

There are different types of systematic reviews depending on the type of research question being answered. Some research question types are suitable for other frameworks like SPIDER and CIMO. Other research questions are strictly suitable for a PICO framework. Therefore, before choosing PICO as your preferred framework, it is important to begin by determining whether the type of your research question is suitable. The following types of research questions are suitable for a PICO framework systematic review:

Once you have identified the suitability of PICO for your research question, the next step is to formulate the research question accordingly. The table below provides a detailed guidance on how to frame each question type based on the PICO framework:

PICO framework systematic reviews are those whose research questions and eligibility criteria are derived from the PICO framework. The PICO framework is one of the most common frameworks used in systematic reviews because it ensures research questions and eligibility criteria are not too specific or too broad.

• Frandsen, T. F., Bruun Nielsen, M. F., Lindhardt, C. L., & Eriksen, M. B. (2020). Using the full PICO model as a search tool for systematic reviews resulted in lower recall for some PICO elements. Journal of Clinical Epidemiology , 127 , 69–75. https://doi.org/10.1016/j.jclinepi.2020.07.005
• Higgins, J. P. T., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M. J., & Welch, V. A. (Eds.). (2019). Cochrane handbook for systematic reviews of interventions (1st ed.). Wiley. https://doi.org/10.1002/9781119536604
• Mathiesen, A. S., Zoffmann, V., Lindschou, J., Jakobsen, J. C., Gluud, C., Due-Christensen, M., Rasmussen, B., Marqvorsen, E. H. S., Lund-Jacobsen, T., Skytte, T. B., Thomsen, T., & Rothmann, M. J. (2023). Self-determination theory interventions versus usual care in people with diabetes: A systematic review with meta-analysis and trial sequential analysis. Systematic Reviews , 12 (1), 158. https://doi.org/10.1186/s13643-023-02308-z
• Scells, H., Zuccon, G., Koopman, B., Deacon, A., Azzopardi, L., & Geva, S. (2017). Integrating the framing of clinical questions via PICO into the retrieval of medical literature for systematic reviews. Proceedings of the 2017 ACM on Conference on Information and Knowledge Management , 2291–2294. https://doi.org/10.1145/3132847.3133080

Approximate price: $ 22

Calculate the price of your order

• Free title page and bibliography
• Unlimited revisions
• Plagiarism-free guarantee

Money-back guarantee

• 24/7 support
• Systematic Review Service
• Meta Analysis Services
• Literature Search Service
• Literature Review Assistance
• Scientific Article Writing Service
• Manuscript Publication Assistance
• 275 words per page
• 12 pt Arial/Times New Roman
• Double line spacing
• Any citation style (APA, MLA, Chicago/Turabian, Harvard, etc)

Our guarantees

Delivering a high-quality product at a reasonable price is not enough anymore. That’s why we have developed 5 beneficial guarantees that will make your experience with our service enjoyable, easy, and safe.

You have to be 100% sure of the quality of your product to give a money-back guarantee. This describes us perfectly. Make sure that this guarantee is totally transparent.

Zero-plagiarism guarantee

Each paper is composed from scratch, according to your instructions. It is then checked by our plagiarism-detection software. There is no gap where plagiarism could squeeze in.

Free-revision policy

Thanks to our free revisions, there is no way for you to be unsatisfied. We will work on your paper until you are completely happy with the result.

Privacy policy

Your email is safe, as we store it according to international data protection rules. Your bank details are secure, as we use only reliable payment systems.

Fair-cooperation guarantee

By sending us your money, you buy the service we provide. Check out our terms and conditions if you prefer business talks to be laid out in official language.

Systematic and systematic-like review toolkit: Step 1: Formulating the research question

Systematic and systematic-like review toolkit.

• Systematic and systematic-like reviews overview

Step 1: Formulating the research question

• Step 2: Developing the search
• Step 3: Screening and selection of articles
• Step 4: Appraisal of articles
• Step 5: Writing and publishing
• Filters and complex search examples
• Evidence synthesis support services

Tip: Look for these icons for guidance on which technique is required

Email your Librarians

The first stage in a review is formulating the research question. The research question accurately and succinctly sums up the review's line of inquiry. This page outlines approaches to developing a research question that can be used as the basis for a review.

Research question frameworks

It can be useful to use a framework to aid in the development of a research question. Frameworks can help you identify searchable parts of a question and focus your search on relevant results

A technique often used in research for formulating a clinical research question is the PICO model. Slightly different versions of this concept are used to search for quantitative and qualitative reviews.

The PICO/ PECO   framework is an adaptable approach to help you focus your research question and guide you in developing search terms. The framework prompts you to consider your question in terms of these four elements:

P : P atient/ P opulation/ P roblem

I/E : I ntervention/ I ndicator/ E xposure/ E vent

C : C omparison/ C ontrol

O : O utcome

For more detail, there are also the PICOT and PICOS additions:

PICO T - adds T ime  

PICO S - adds S tudy design

PICO example

Consider this scenario:

Current guidelines indicate that nicotine replacement therapies (NRTs) should not be used as an intervention in young smokers.  Counselling is generally the recommended best practice for young smokers, however youth who are at high risk for smoking often live in regional or remote communities with limited access to counselling services.  You have been funded to review the evidence for the effectiveness of NRTs for smoking cessation in Australian youths to update the guidelines.

The research question stemming from this scenario could be phrased in this way:

In (P) adolescent smokers , how does (I) nicotine replacement therapy compared with (C) counselling affect (O) smoking cessation rates ?

Alternative frameworks

PICO is one of the most frequently used frameworks, but there are several other frameworks available to use, depending on your question.

Question type

• Qualitative; Aetiology or risk
• Services, policy, social care
• Prevalence & prognosis; Economics

Structuring qualitative questions?

Try PIC or SPIDER :

• P opulation, Phenomena of I nterest, C ontext
• S ample, P henomenon of I nterest, D esign, E valuation, R esearch type   

Cooke, A., Smith, D., & Booth, A. (2012). Beyond PICO: the SPIDER tool for qualitative evidence synthesis . Qualitative health research, 22(10), 1435-1443.

Question about aetiology or risk? 

• P opulation, E xposure, O utcomes

Moola, Sandeep; Munn, Zachary; Sears, Kim; Sfetcu, Ralucac; Currie, Marian; Lisy, Karolina; Tufanaru, Catalin; Qureshi, Rubab; Mattis, Patrick; Mu, Peifanf. Conducting systematic reviews of association (etiology) , International Journal of Evidence-Based Healthcare: September 2015 - Volume 13 - Issue 3 - p 163-169.

Evaluating an intervention, policy or service? 

Try SPICE :

• S etting, P opulation or P erspective, I ntervention, C omparison, E valuation

Booth, A. (2006), " Clear and present questions: formulating questions for evidence based practice ", Library Hi Tech, Vol. 24 No. 3, pp. 355-368. https://doi-org.ezproxy-b.deakin.edu.au/10.1108/07378830610692127

Investigating the outcome of a service or policy? 

Try ECLIPSE :

• E xpectation, C lient group, L ocation, I mpact, P rofessionals, SE rvice  

Wildridge, V., & Bell, L. (2002). How CLIP became ECLIPSE: a mnemonic to assist in searching for health policy/management information . Health Information & Libraries Journal, 19(2), 113-115.

Working out prevalence or incidence? 

Try CoCoPop :

• Co ndition, Co ntext, Pop ulation

Munn, Z., Moola, S., Lisy, K., Riitano, D., & Tufanaru, C. (2015). Methodological guidance for systematic reviews of observational epidemiological studies reporting prevalence and cumulative incidence data . International journal of evidence-based healthcare, 13(3), 147-153.

Determining prognosis?

• P opulation, Prognostic F actors, O utcome

Conducting an economic evaluation? 

Try PICOC :

• P opulation, I ntervention, C omparator/s, O utomes, Context

Petticrew, M., & Roberts, H. (2006). Systematic reviews in the social sciences: a practical guide . Blackwell Pub.

JBI recommends the PCC (Population (or Participants), Concept, and Context) search framework to develop the research question of a scoping review. In some instances, just the concept and context are used in the search.

The University of Notre Dame Australia provides information on some different frameworks available to help structure the research question.

Further Readings

Booth A, Noyes J, Flemming K, et al, Formulating questions to explore complex interventions within qualitative evidence synthesis . BMJ Global Health 2019;4:e001107. This paper explores the importance of focused, relevant questions in qualitative evidence syntheses to address complexity and context in interventions.

Kim, K. W., Lee, J., Choi, S. H., Huh, J., & Park, S. H. (2015). Systematic review and meta-analysis of studies evaluating diagnostic test accuracy: a practical review for clinical researchers-part I. General guidance and tips . Korean journal of radiology, 16(6), 1175-1187. As the use of systematic reviews and meta-analyses is increasing in the field of diagnostic test accuracy (DTA), this first of a two-part article provides a practical guide on how to conduct, report, and critically appraise studies of DTA. 

Methley, A. M., Campbell, S., Chew-Graham, C., McNally, R., & Cheraghi-Sohi, S. (2014). PICO, PICOS and SPIDER: A comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews . BMC Health Services Research, 14(1), 579. In this article the ‘SPIDER’ search framework, developed for more effective searching of qualitative research, was evaluated against PICO and PICOD. 

Munn, Z., Stern, C., Aromataris, E., Lockwood, C., & Jordan, Z. (2018). What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences . BMC medical research methodology, 18(1), 5. https://doi.org/10.1186/s12874-017-0468-4 This article aligns review types to question development frameworks.

Search for existing reviews

Before you start searching, find out whether any systematic reviews have been conducted recently on your topic. This is because similar systematic reviews could help with identifying your search terms, and information on your topic. It is also helpful to know if there is already a systematic review on your topic as it may mean you need to change your question.  

Cochrane Library and Joanna Briggs Institute publish systematic reviews. You can also search for the term "systematic review" in any of the subject databases. You can also search PROSPERO , an international register of systematic reviews, to see if there are any related reviews underway but not yet published; there are additional review registers detailed below.  

Watch this video to find out how to search for published systematic reviews

Protocols and Guidelines for reviews

It is recommended that authors consult relevant guidelines and create a protocol for their review.  

Protocols provide a clear plan for how the review will be conducted, including what will and will not be included in the final review. Protocols are widely recommended for any systematic review and are increasingly a requirement for publication of a completed systematic review.

Guidelines provide specific information on how to perform a review in your field of study. A completed review may be evaluated against the relevant guidelines by peer reviewers or readers, so it makes sense to follow the guidelines as best you can.

Click the headings below to learn more about the importance of protocols and guidelines.

Your protocol (or plan for conducting your review) should include the rationale, objectives, hypothesis, and planned methods used in searching, screening and analysing identified studies used in the review. The rationale should clearly state what will be included and excluded from the review. The aim is to minimise any bias by having pre-defined eligibility criteria.

Base the protocol on the relevant guidelines for the review that you are conducting.  PRISMA-P was developed for reporting and development of protocols for systematic reviews. Their Explanation and Elaboration paper includes examples of what to write in your protocol. York's CRD has also created a document on how to submit a protocol to PROSPERO .

There are several registers of protocols, often associated with the organisation publishing the review. Cochrane and Joanna Briggs Institute both have their own protocol registries, and PROSPERO is a wide-reaching registry covering protocols for Cochrane, non-Cochrane and non-JBI reviews on a range of health, social care, education, justice, and international development topics.

Before beginning your protocol, search within protocol registries such as those listed above, or Open Science Framework or Research Registry , or journals such as Systematic Reviews and BMJ Open . This is a useful step to see if a protocol has already been submitted on your review topic and to find examples of protocols in similar areas of research.    

While a protocol will contain details of the intended search strategy, a protocol should be registered before the search strategy is finalised and run, so that you can show that your intention for the review has remained true and to limit duplication of in progress reviews.  

A protocol should typically address points that define the kind of studies to be included and the kind of data required to ensure the systematic review is focused on the appropriate studies for the topic. Some points to think about are:

• What study types are you looking for? For example, randomised controlled trials, cohort studies, qualitative studies
• What sample size is acceptable in each study (power of the study)? 
• What population are you focusing on? Consider age ranges, gender, disease severity, geography of patients.
• What type of intervention are you focusing on?
• What outcomes are of importance to the review, including how those outcomes are measured?
• What context should you be looking for in a study? A lab, acute care, school, community...
• How will you appraise the studies? What methodology will you use?
• Does the study differentiate between the target population and other groups in the data? How will you handle it if it does not?
• Is the data available to access if the article does not specify the details you need? If not, what will you do?
• What languages are you able to review? Do you have funding to translate articles from languages other than English?  

Further reading

PLoS Medicine Editors. (2011). Best practice in systematic reviews: the importance of protocols and registration . PLoS medicine, 8(2), e1001009.

Systematic Review guidelines

The Cochrane handbook of systematic reviews of interventions is a world-renowned resource for information on designing systematic reviews of intervention.  

Many other guidelines have been developed from these extensive guidelines.

General systematic reviews

• The  PRISMA Statement  includes the well-used Checklist and Flow Diagram.
• Systematic Reviews: CRD's guidance on undertaking reviews in health care . One of the founding institutions that developed systematic review procedure. CRD's guide gives detailed clearly written explanations for different fields in Health.
• National Academies Press (US); 2011. 3, Standards for Finding and Assessing Individual Studies. Provides guidance on searching, screening, data collection, and appraisal of individual studies for a systematic review.

Meta-analyses

• An alternative to PRISMA is the Meta‐analysis Of Observational Studies in Epidemiology (MOOSE) for observational studies. It is a 35‐item checklist. It pays more attention to certain aspects of the search strategy, in particular the inclusion of unpublished and non‐English‐language studies.

Surgical systematic reviews

• Systematic reviews in surgery-recommendations from the Study Center of the German Society of Surgery . Provides recommendations for systematic reviews in surgery with or without meta-analysis, for each step of the process with specific recommendations important to surgical reviews.

Nursing/Allied Health systematic reviews

Joanna Briggs Institute Manual for Evidence Synthesis  a comprehensive guide to conducting JBI systematic and similar reviews

Nutrition systematic reviews

• Academy of Nutrition and Dietetics Evidence Analysis Manual  is designed to guide expert workgroup members and evidence analysts to understand and carry out the process of conducting a systematic review.

Occupational therapy

• American Occupational Therapy Association: Guidelines for Systematic reviews . The American Journal of Occupational Therapy (AJOT) provides guidance for authors conducting systematic reviews.

Education/Law/ Sociology systematic reviews

• Campbell Collaboration, Cochrane's sister organisation provides guidelines for systematic reviews in the social sciences:  MECIR
• Systematic Reviews in Educational Research: Methodology, Perspectives and Application

Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy

COSMIN Guideline for Systematic Reviews of Outcome Measurement Instruments – This was developed for patient reported outcomes (PROMs) but has since been adapted for use with other types of outcome measurements in systematic reviews.

Prinsen, C.A.C., Mokkink, L.B., Bouter, L.M. et al. COSMIN guideline for systematic reviews of patient-reported outcome measures . Qual Life Res 27, 1147–1157 (2018). https://doi.org/10.1007/s11136-018-1798-3

HuGENet™ Handbook of systematic reviews – particularly useful for describing population-based data and human genetic variants.

AHRQ: Methods Guide for Effectiveness and Comparative Effectiveness Reviews - from the US Department of Health and Human Services, guidelines on conducting systematic reviews of existing research on the effectiveness, comparative effectiveness, and harms of different health care interventions.

Mariano, D. C., Leite, C., Santos, L. H., Rocha, R. E., & de Melo-Minardi, R. C. (2017). A guide to performing systematic literature reviews in bioinformatics . arXiv preprint arXiv:1707.05813.

Integrative Review guidelines

Integrative reviews may incorporate experimental and non-experimental data, as well as theoretical information.  They differ from systematic reviews in the diversity of the study methodologies included.

Guidelines:

• Whittemore, R. and Knafl, K. (2005), The integrative review: updated methodology. Journal of Advanced Nursing, 52: 546–553. doi:10.1111/j.1365-2648.2005.03621.x
• A step-by-step guide to conducting an Integrative Review (2020), edited by C.E. Toronto & Ruth Remington, Springer Books

Rapid Review guidelines

Rapid reviews differ from systematic reviews in the shorter timeframe taken and reduced comprehensiveness of the search.

Cochrane has a methods group to inform the conduct of rapid reviews with a bibliography of relevant publications .

A modified approach to systematic review guidelines can be used for rapid reviews, but guidelines are beginning to appear:

Crawford C, Boyd C, Jain S, Khorsan R and Jonas W (2015), Rapid Evidence Assessment of the Literature (REAL©): streamlining the systematic review process and creating utility for evidence-based health care . BMC Res Notes 8:631 DOI 10.1186/s13104-015-1604-z

Philip Moons, Eva Goossens, David R. Thompson, Rapid reviews: the pros and cons of an accelerated review process , European Journal of Cardiovascular Nursing, Volume 20, Issue 5, June 2021, Pages 515–519, https://doi.org/10.1093/eurjcn/zvab041

Rapid Review Guidebook: Steps for conducting a rapid review National Collaborating Centre for Methods and Tools (McMaster University and Public Health Agency Canada) 2017

Tricco AC, Langlois EV, Straus SE, editors (2017) Rapid reviews to strengthen health policy and systems: a practical guide (World Health Organization). This guide is particularly aimed towards developing rapid reviews to inform health policy. 

Scoping Review guidelines

Scoping reviews can be used to map an area, or to determine the need for a subsequent systematic review. Scoping reviews tend to have a broader focus than many other types of reviews, however, still require a focused question.

• Peters MDJ, Godfrey C, McInerney P, Munn Z, Tricco AC, Khalil, H. Chapter 11: Scoping Reviews (2020 version). In: Aromataris E, Munn Z (Editors). Joanna Briggs Institute Reviewer's Manual, JBI, 2020. 
• Statement / Explanatory paper

Scoping reviews: what they are and how you can do them - Series of Cochrane Training videos presented by Dr. Andrea C. Tricco and Kafayat Oboirien

Martin, G. P., Jenkins, D. A., Bull, L., Sisk, R., Lin, L., Hulme, W., ... & Group, P. H. A. (2020). Toward a framework for the design, implementation, and reporting of methodology scoping reviews . Journal of Clinical Epidemiology, 127, 191-197.

Khalil, H., McInerney, P., Pollock, D., Alexander, L., Munn, Z., Tricco, A. C., ... & Peters, M. D. (2021). Practical guide to undertaking scoping reviews for pharmacy clinicians, researchers and policymakers . Journal of clinical pharmacy and therapeutics.

Colquhoun, H (2016) Current best practices for the conduct of scoping reviews (presentation)

Arksey H & O'Malley L (2005) Scoping studies: towards a methodological framework , International Journal of Social Research Methodology, 8:1, 19-32, DOI: 10.1080/1364557032000119616

Umbrella reviews

• Pollock M, Fernandes RM, Becker LA, Pieper D, Hartling L. Chapter V: Overviews of Reviews . In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). Cochrane, 2021. Available from www.training.cochrane.org/handbook .  
• Aromataris E, Fernandez R, Godfrey C, Holly C, Khalil H, Tungpunkom P. Chapter 10: Umbrella Reviews . In: Aromataris E, Munn Z (Editors). JBI Manual for Evidence Synthesis. JBI, 2020. Available from https://jbi-global-wiki.refined.site/space/MANUAL/4687363 .
• Aromataris, Edoardo; Fernandez, Ritin; Godfrey, Christina M.; Holly, Cheryl; Khalil, Hanan; Tungpunkom, Patraporn. Summarizing systematic reviews: methodological development, conduct and reporting of an umbrella review approach , International Journal of Evidence-Based Healthcare: September 2015 - Volume 13 - Issue 3 - p 132-140.

Meta-syntheses

Noyes, J., Booth, A., Cargo, M., Flemming, K., Garside, R., Hannes, K., ... & Thomas, J. (2018). Cochrane Qualitative and Implementation Methods Group guidance series—paper 1: introduction . Journal of clinical epidemiology, 97, 35-38.

Harris, J. L., Booth, A., Cargo, M., Hannes, K., Harden, A., Flemming, K., ... & Noyes, J. (2018). Cochrane Qualitative and Implementation Methods Group guidance series—paper 2: methods for question formulation, searching, and protocol development for qualitative evidence synthesis . Journal of clinical epidemiology, 97, 39-48.

Noyes, J., Booth, A., Flemming, K., Garside, R., Harden, A., Lewin, S., ... & Thomas, J. (2018). Cochrane Qualitative and Implementation Methods Group guidance series—paper 3: methods for assessing methodological limitations, data extraction and synthesis, and confidence in synthesized qualitative findings . Journal of clinical epidemiology, 97, 49-58.

Cargo, M., Harris, J., Pantoja, T., Booth, A., Harden, A., Hannes, K., ... & Noyes, J. (2018). Cochrane Qualitative and Implementation Methods Group guidance series—paper 4: methods for assessing evidence on intervention implementation . Journal of clinical epidemiology, 97, 59-69.

Harden, A., Thomas, J., Cargo, M., Harris, J., Pantoja, T., Flemming, K., ... & Noyes, J. (2018). Cochrane Qualitative and Implementation Methods Group guidance series—paper 5: methods for integrating qualitative and implementation evidence within intervention effectiveness reviews . Journal of clinical epidemiology, 97, 70-78.

Flemming, K., Booth, A., Hannes, K., Cargo, M., & Noyes, J. (2018). Cochrane Qualitative and Implementation Methods Group guidance series—Paper 6: Reporting guidelines for qualitative, implementation, and process evaluation evidence syntheses . Journal of Clinical Epidemiology, 97, 79-85.

Walsh, D. and Downe, S. (2005), Meta-synthesis method for qualitative research: a literature review . Journal of Advanced Nursing, 50: 204–211. doi:10.1111/j.1365-2648.2005.03380.x

Living reviews

• Akl, E.A., Meerpohl, J.J., Elliott, J., Kahale, L.A., Schünemann, H.J., Agoritsas, T., Hilton, J., Perron, C., Akl, E., Hodder, R. and Pestridge, C., 2017. Living systematic reviews: 4. Living guideline recommendations . Journal of clinical epidemiology, 91, pp.47-53.

Qualitative systematic reviews

• Dixon-Woods, M., Bonas, S., Booth, A., Jones, D. R., Miller, T., Sutton, A. J., . . . Young, B. (2006). How can systematic reviews incorporate qualitative research? A critical perspective . Qualitative Research,6(1), 27–44.
• Thomas, J., & Harden, A. (2008). Methods for the thematic synthesis of qualitative research in systematic reviews . BMC Medical Research Methodology,8, 45–45.

Mixed methods systematic review

• Lizarondo L, Stern C, Carrier J, Godfrey C, Rieger K, Salmond S, Apostolo J, Kirkpatrick P, Loveday H. Chapter 8: Mixed methods systematic reviews . In: Aromataris E, Munn Z (Editors). JBI Manual for Evidence Synthesis. JBI, 2020. Available from https://synthesismanual.jbi.global. https://doi.org/10.46658/JBIMES-20-09
• Pearson, A, White, H, Bath-Hextall, F, Salmond, S, Apostolo, J, & Kirkpatrick, P 2015, ' A mixed-methods approach to systematic reviews ', International Journal of Evidence-Based Healthcare, vol. 13, no. 3, p. 121-131. Available from: 10.1097/XEB.0000000000000052
• Dixon-Woods, M., Agarwal, S., Jones, D., Young, B., & Sutton, A. (2005). Synthesising qualitative and quantitative evidence: A review of possible methods . Journal of Health Services Research &Policy,10(1), 45–53.

Realist reviews

The RAMESES Projects - Includes information on publication, quality, and reporting standards, as well as training materials for realist reviews, meta-narrative reviews, and realist evaluation.

Rycroft-Malone, J., McCormack, B., Hutchinson, A. M., DeCorby, K., Bucknall, T. K., Kent, B., ... & Wilson, V. (2012). Realist synthesis: illustrating the method for implementation research . Implementation Science, 7(1), 1-10.

Wong, G., Westhorp, G., Manzano, A. et al. RAMESES II reporting standards for realist evaluations. BMC Med 14, 96 (2016). https://doi.org/10.1186/s12916-016-0643-1

Wong, G., Greenhalgh, T., Westhorp, G., Buckingham, J., & Pawson, R. (2013). RAMESES publication standards: realist syntheses. BMC medicine, 11, 21. https://doi.org/10.1186/1741-7015-11-21

Wong, G., Greenhalgh, T., Westhorp, G., Buckingham, J., & Pawson, R. (2013). RAMESES publication standards: realist syntheses. BMC medicine, 11(1), 1-14.  https://doi.org/10.1186/1741-7015-11-21

Social sciences

• Chapman, K. (2021). Characteristics of systematic reviews in the social sciences . The Journal of Academic Librarianship, 47(5), 102396.
• Crisp, B. R. (2015). Systematic reviews: A social work perspective . Australian Social Work, 68(3), 284-295.  

Further Reading

Uttley, L., Montgomery, P. The influence of the team in conducting a systematic review . Syst Rev 6, 149 (2017). https://doi.org/10.1186/s13643-017-0548-x

• << Previous: Review Process Steps
• Next: Step 2: Developing the search >>
• Last Updated: Jul 22, 2024 11:44 AM
• URL: https://deakin.libguides.com/systematicreview

Cleveland Clinic Florida - How to Conduct Systematic Reviews: What is PICO

• What is PICO
• Step 1: Choose your topic
• Step 2: Identify your keywords
• Step 3: Connect your keywords
• Step 4: Choose your databases
• Step 5: Find your subjects
• Step 6: Run your search
• Step 7: Apply your criteria
• Step 8: Manage your citations
• Outline of the Process
• Goldblatt Library Assistance
• Goldblatt Library Homepage This link opens in a new window

Using PICO to formulate a search question

The Cochrane Library Searching using PICOT

What is PICO?

According to the Centre for Evidence Based Medicine (CEBM), well-formed clinical questions are essential in practicing EBM. "To benefit patients and clinicians, such questions need to be both directly relevant to patients' problems and phrased in ways that direct your search to relevant and precise answers." - CEBM, University of Toronto,  Asking Focused Questions

The PICO model is a tool that can help you formulate a good clinical question. Sometimes it's referred to as PICO-T, containing an optional 5th factor. 

• Asking an Answerable Question (Cochrane Library)
• PICO Cochrane Library Tutorial (University of Oxford)
• Question Templates for PICOT (Sonoma State University)

This page was adapted from   (PA/MPH) PICO  by George Washington University, Himmelfarb Health Sciences Library.

• << Previous: Home
• Next: Step 1: Choose your topic >>
• Last Updated: Jul 26, 2023 12:07 PM
• URL: https://my.clevelandclinic.libguides.com/Florida_SystematicReview

Covidence website will be inaccessible as we upgrading our platform on Monday 23rd August at 10am AEST, / 2am CEST/1am BST (Sunday, 15th August 8pm EDT/5pm PDT) 

How to formulate the review question using PICO. 5 steps to get you started.

Home | Blog | How To | How to formulate the review question using PICO. 5 steps to get you started.

Covidence covers five key steps to formulate your review question using PICO

You’ve decided to go ahead. You have identified a gap in the evidence and you know that conducting a systematic review, with its explicit methods and replicable search, is the best way to fill it – great choice 🙌. 

The review will produce useful information to enable informed decision-making and to improve patient care. Your review team’s first job is to capture exactly what you need to know in a well-formulated review question.

At this stage there is a lot to plan. You might be recruiting people to your review team, thinking about the time-frame for completion and considering what software to use. It’s tempting to get straight on to the search for studies 🏃. 

Take it slowly: it’s vital to get the review question right. A clear and precise question will ensure that you gather the appropriate data to answer your question. Time invested up-front to consider every aspect of the question will pay off once the review is underway. The review question will shape all the subsequent stages in the review, particularly setting the criteria for including and excluding studies, the search strategy, and the way you choose to present the results. So it’s worth taking the time to get this right!

Let’s take a look at five key steps in formulating the question for a standard systematic review of interventions. It’s a process that requires careful thought from a range of stakeholders and meticulous planning. But what if, once you have started the review, you find that you need to tweak the question anyway? Don’t worry, we’ll cover that too ✅.

📌 Consider the audience of the review

Who will use this review? What do they want to know? How do they measure effectiveness? Good review teams partner with the people who will use the evidence and make sure that their research plan (or protocol) asks a question that is relevant and important for patients.

📌 Think about what you already know

How much do you need to know about the topic area at this stage? Ideally, enough to come up with a relevant, useful question but not so much that your knowledge influences the way in which you phrase it. Why? Because setting a review question when you are already familiar with the data can introduce bias by allowing you to direct the question in favour of achieving a particular result. In practice, the review team is very likely to have some knowledge of relevant studies and some preconceived ideas about how the treatments work. That’s fine – and it’s useful – but it’s also good practice to recognise the influence this knowledge and these ideas might have on the choice of question. Issues of bias will come up again as we work through the rest of these steps.

If not enough is known about the subject area to ask a useful question, you might undertake a scoping review . This is a separate exercise from a systematic review and is sometimes used by researchers to map the literature and highlight gaps in the evidence before they start work on a systematic review. 

📌 Use a framework

Faced with a heady mixture of concepts, ideas, aims and outcomes, researchers in every field have come up with question frameworks (and some great backronyms ) to help them. Question frameworks impose order on a complex thought process by breaking down a question into its component parts. A commonly used framework in clinical medicine is PICO:

👦 P opulation (or patients) refers to the characteristics of the people that you want to study. For example, the review might look at children with nocturnal enuresis.

💊 I ntervention is the treatment you are investigating. For example, the review might look at the effectiveness of enuresis alarms.

💊 C omparison, if you decide to use one, is the treatment you want to compare the intervention with. For example, the review might look at the effectiveness of enuresis alarms versus the effectiveness of drug therapy. 

📏 O utcomes are the measures used to assess the effectiveness of the treatment. It’s particularly important to select outcomes that matter to the end users of the review. In this example, a useful outcome might be bedwetting. (Helpfully, some clinical areas use standardised sets of outcomes in their clinical trials to facilitate the comparison of data between studies 👏.) 

But back to bedwetting. In our example, a PICO review question would look something like this:

“In children with nocturnal enuresis (population), how effective are alarms (intervention) versus drug treatments (comparison) for the prevention of bedwetting (outcome)?”

PICO is suitable for reviews of interventions. If you plan to review prognostic or qualitative data, or diagnostic test accuracy, PICO is unlikely to be a suitable framework for your question. In Covidence you can save your PICO for easy reference throughout the screening, extraction and quality assessment phases of your review.

📌 Set the scope

The scope of a review question requires careful thought. To answer the example PICO question above, the review would compare one treatment (alarms) with another (drug therapy). A broader question might consider all the available treatments for nocturnal enuresis in children. The broad scope of this question would still allow the review team to drill down and separate the data into groups of specific treatments later in the review process. And to minimise bias, the intended grouping of data would be pre-specified and justified in the protocol or research plan.

Broader systematic reviews are great because they summarise all the evidence on a given topic in one place. A potential disadvantage is that they can produce a large volume of data that is difficult to manage. 

If the size of the review has started to escalate beyond your comfort zone, you might consider narrowing the scope. This can make the size of the review more manageable, both for the review team and for the reader. But it’s worth examining the motivations for narrowing the scope more closely. Suppose we wanted to define a smaller population in the example PICO question. Is there a good reason (other than to reduce the review team’s workload) to restrict the population to boys with nocturnal enuresis? Or to children under 10 years old? On the basis of what is already known, could the treatment effect be expected to differ by sex or age of the study participants? 🤔 Be prepared to explain your choices and to demonstrate that they are legitimate. 

Some reviews with a narrow scope retrieve only a small number of studies. If this happens, there is a risk that the data collected from these studies might not be enough to produce a useful synthesis or to guide decision-making. It can be frustrating for review teams who have spent time defining the question, planning the methods, and conducting an extensive search to find that their question is unanswerable. This is another reason why it is useful for the review team to have prior knowledge of the subject area and some familiarity with the existing evidence. The Cochrane Handbook contains some useful contingencies for dealing with sparse data .

Covidence can help review teams to save time whatever the scope and size of the review. In Covidence, data can be grouped to the review team’s exact specification for seamless export into data analysis software. The intuitive workflow makes collaboration simple so if one reviewer spots a problem, they can alert the rest of the team quickly and easily.

📌 Adjust if necessary

Systematic reviews follow explicit, pre-specified methods. So it’s no surprise to learn that the review question needs to be considered carefully and explained in detail before the review gets underway. But what about the unknown unknowns – those issues that the review teams will have to deal with later in the process but that they cannot foresee at the outset, no matter how much time they spend on due diligence? 

Clearly, reviews need the agility to control for issues that the project plan did not anticipate – strict adherence to the pre-specified process when a good reason to deviate has come to light would carry its own risks for the quality of the review. So if an initial scan of, for example, the search results indicates that it would be sensible to modify the question, this can be done. The research plan might make explicit the process for dealing with these types of changes. It might also contain plans for sensitivity analysis , to examine whether these choices have any effect on the findings of the review. As mentioned above with regard to scope, it might be difficult to defend a data-driven change to the question. And as before, the issue is the risk of bias and the danger of producing a spurious result.

(Figure 4. Image from Eshun‐Wilson  I, Siegfried  N, Akena  DH, Stein  DJ, Obuku  EA, Joska  JA. Antidepressants for depression in adults with HIV infection. Cochrane Database of Systematic Reviews 2018, Issue 1. Art. No.: CD008525. DOI: 10.1002/14651858.CD008525.pub3. Accessed 27 May 2021.)

This blog post is part of the Covidence series on how to write a systematic review. 

Sign up for a free trial of Covidence today!

Laura Mellor. Portsmouth, UK

Perhaps you'd also like....

Top 5 Tips for High-Quality Systematic Review Data Extraction

Data extraction can be a complex step in the systematic review process. Here are 5 top tips from our experts to help prepare and achieve high quality data extraction.

How to get through study quality assessment Systematic Review

Find out 5 tops tips to conducting quality assessment and why it’s an important step in the systematic review process.

How to extract study data for your systematic review

Learn the basic process and some tips to build data extraction forms for your systematic review with Covidence.

Better systematic review management

Head office, working for an institution or organisation.

Find out why over 350 of the world’s leading institutions are seeing a surge in publications since using Covidence!

Request a consultation with one of our team members and start empowering your researchers: 

By using our site you consent to our use of cookies to measure and improve our site’s performance. Please see our Privacy Policy for more information. 

Library Services

UCL LIBRARY SERVICES

• Guides and databases
• Library skills
• Systematic reviews

Formulating a research question

• What are systematic reviews?
• Types of systematic reviews
• Identifying studies
• Searching databases
• Describing and appraising studies
• Synthesis and systematic maps
• Software for systematic reviews
• Online training and support
• Live and face to face training
• Individual support
• Further help

Clarifying the review question leads to specifying what type of studies can best address that question and setting out criteria for including such studies in the review. This is often called inclusion criteria or eligibility criteria. The criteria could relate to the review topic, the research methods of the studies, specific populations, settings, date limits, geographical areas, types of interventions, or something else.

Systematic reviews address clear and answerable research questions, rather than a general topic or problem of interest. They also have clear criteria about the studies that are being used to address the research questions. This is often called inclusion criteria or eligibility criteria.

Six examples of types of question are listed below, and the examples show different questions that a review might address based on the topic of influenza vaccination. Structuring questions in this way aids thinking about the different types of research that could address each type of question. Mneumonics can help in thinking about criteria that research must fulfil to address the question. The criteria could relate to the context, research methods of the studies, specific populations, settings, date limits, geographical areas, types of interventions, or something else.

Examples of review questions

• Needs - What do people want? Example: What are the information needs of healthcare workers regarding vaccination for seasonal influenza?
• Impact or effectiveness - What is the balance of benefit and harm of a given intervention? Example: What is the effectiveness of strategies to increase vaccination coverage among healthcare workers. What is the cost effectiveness of interventions that increase immunisation coverage?
• Process or explanation - Why does it work (or not work)? How does it work (or not work)?  Example: What factors are associated with uptake of vaccinations by healthcare workers?  What factors are associated with inequities in vaccination among healthcare workers?
• Correlation - What relationships are seen between phenomena? Example: How does influenza vaccination of healthcare workers vary with morbidity and mortality among patients? (Note: correlation does not in itself indicate causation).
• Views / perspectives - What are people's experiences? Example: What are the views and experiences of healthcare workers regarding vaccination for seasonal influenza?
• Service implementation - What is happening? Example: What is known about the implementation and context of interventions to promote vaccination for seasonal influenza among healthcare workers?

Examples in practice :  Seasonal influenza vaccination of health care workers: evidence synthesis / Loreno et al. 2017

Example of eligibility criteria

Research question: What are the views and experiences of UK healthcare workers regarding vaccination for seasonal influenza?

• Population: healthcare workers, any type, including those without direct contact with patients.
• Context: seasonal influenza vaccination for healthcare workers.
• Study design: qualitative data including interviews, focus groups, ethnographic data.
• Date of publication: all.
• Country: all UK regions.
• Studies focused on influenza vaccination for general population and pandemic influenza vaccination.
• Studies using survey data with only closed questions, studies that only report quantitative data.

Consider the research boundaries

It is important to consider the reasons that the research question is being asked. Any research question has ideological and theoretical assumptions around the meanings and processes it is focused on. A systematic review should either specify definitions and boundaries around these elements at the outset, or be clear about which elements are undefined. 

For example if we are interested in the topic of homework, there are likely to be pre-conceived ideas about what is meant by 'homework'. If we want to know the impact of homework on educational attainment, we need to set boundaries on the age range of children, or how educational attainment is measured. There may also be a particular setting or contexts: type of school, country, gender, the timeframe of the literature, or the study designs of the research.

Research question: What is the impact of homework on children's educational attainment?

• Scope : Homework - Tasks set by school teachers for students to complete out of school time, in any format or setting.
• Population: children aged 5-11 years.
• Outcomes: measures of literacy or numeracy from tests administered by researchers, school or other authorities.
• Study design: Studies with a comparison control group.
• Context: OECD countries, all settings within mainstream education.
• Date Limit: 2007 onwards.
• Any context not in mainstream primary schools.
• Non-English language studies.

Mnemonics for structuring questions

Some mnemonics that sometimes help to formulate research questions, set the boundaries of question and inform a search strategy.

Intervention effects

PICO  Population – Intervention– Outcome– Comparison

Variations: add T on for time, or ‘C’ for context, or S’ for study type,

Policy and management issues

ECLIPSE : Expectation – Client group – Location – Impact ‐ Professionals involved – Service

Expectation encourages  reflection on what the information is needed for i.e. improvement, innovation or information.  Impact looks at what  you would like to achieve e.g. improve team communication .

• How CLIP became ECLIPSE: a mnemonic to assist in searching for health policy/management information / Wildridge & Bell, 2002

Analysis tool for management and organisational strategy

PESTLE:  Political – Economic – Social – Technological – Environmental ‐ Legal

An analysis tool that can be used by organizations for identifying external factors which may influence their strategic development, marketing strategies, new technologies or organisational change.

• PESTLE analysis / CIPD, 2010

Service evaluations with qualitative study designs

SPICE:  Setting (context) – Perspective– Intervention – Comparison – Evaluation

Perspective relates to users or potential users. Evaluation is how you plan to measure the success of the intervention.

• Clear and present questions: formulating questions for evidence based practice / Booth, 2006

Read more about some of the frameworks for constructing review questions:

• Formulating the Evidence Based Practice Question: A Review of the Frameworks / Davis, 2011
• << Previous: Stages in a systematic review
• Next: Identifying studies >>
• Last Updated: Aug 2, 2024 9:22 AM
• URL: https://library-guides.ucl.ac.uk/systematic-reviews

• McGill Library

Systematic Reviews, Scoping Reviews, and other Knowledge Syntheses

• Identifying the research question
• Types of knowledge syntheses
• Process of conducting a knowledge synthesis

Constructing a good research question

Inclusion/exclusion criteria, has your review already been done, where to find other reviews or syntheses, references on question formulation frameworks.

• Developing the protocol
• Database-specific operators and fields
• Search filters and tools
• Exporting and documenting search results
• Deduplicating
• Grey literature and other supplementary search methods
• Documenting the search methods
• Updating the database searches
• Resources for screening, appraisal, and synthesis
• Writing the review
• Additional training resources

Formulating a well-constructed research question is essential for a successful review. You should have a draft research question before you choose the type of knowledge synthesis that you will conduct, as the type of answers you are looking for will help guide your choice of knowledge synthesis.

Examples of systematic review and scoping review questions

• Process of formulating a question

Developing a good research question is not a straightforward process and requires engaging with the literature as you refine and rework your idea.

Some questions that might be useful to ask yourself as you are drafting your question:

• Does the question fit into the PICO question format?
• What age group?
• What type or types of conditions?
• What intervention? How else might it be described?
• What outcomes? How else might they be described?
• What is the relationship between the different elements of your question?
• Do you have several questions lumped into one? If so, should you split them into more than one review? Alternatively, do you have many questions that could be lumped into one review?

A good knowledge synthesis question will have the following qualities:

• Be focused on a specific question with a meaningful answer
• Retrieve a number of results that is manageable for the research team (is the number of results on your topic feasible for you to finish the review? Your initial literature searches should give you an idea, and a librarian can help you with understanding the size of your question).

Considering the inclusion and exclusion criteria

It is important to think about which studies will be included in your review when you are writing your research question. The Cochrane Handbook chapter (linked below) offers guidance on this aspect.

McKenzie, J. E., Brennan, S. E., Ryan, R. E., Thomson, H. J., Johnston, R. V, & Thomas, J. (2021). Chapter 3: Defining the criteria for including studies and how they will be grouped for the synthesis. Retrieved from https://training.cochrane.org/handbook/current/chapter-03

Once you have a reasonably well defined research question, it is important to make sure your project has not already been recently and successfully undertaken. This means it is important to find out if there are other knowledge syntheses that have been published or that are in the process of being published on your topic.

If you are submitting your review or study for funding, for example, you may want to make a good case that your review or study is needed and not duplicating work that has already been successfully and recently completed—or that is in the process of being completed. It is also important to note that what is considered “recent” will depend on your discipline and the topic.

In the context of conducting a review, even if you do find one on your topic, it may be sufficiently out of date or you may find other defendable reasons to undertake a new or updated one. In addition, looking at other knowledge syntheses published around your topic may help you refocus your question or redirect your research toward other gaps in the literature.

• PROSPERO Search PROSPERO is an international, searchable database that allows free registration of systematic reviews, rapid reviews, and umbrella reviews with a health-related outcome in health & social care, welfare, public health, education, crime, justice, and international development. Note: PROSPERO does not accept scoping review protocols.
• Open Science Framework (OSF) At present, OSF does not allow for Boolean searching on their site. However, you can search via https://share.osf.io/, an aggregator, that allows you to search for major keywords using Boolean and truncation. Add "review*" to your search to narrow results down to scoping, systematic, umbrella or other types of reviews. Be sure to click on the drop-down menu for "Source" and select OSF and OSF Registries (search separately as you can't combine them). This will search for ongoing and/or registered reviews in OSF.

The Cochrane Library (including systematic reviews of interventions, diagnostic studies, prognostic studies, and more) is an excellent place to start, even if Cochrane reviews are also indexed in MEDLINE/PubMed.

By default, the Cochrane Library will display “ Cochrane Reviews ” (Cochrane Database of Systematic Reviews, aka CDSR). You can ignore the results which show up in the Trials tab when looking for systematic reviews: They are records of controlled trials. 

The example shows the number of Cochrane Reviews with hiv AND circumcision in the title, abstract, or keywords.

• Google Scholar

Subject-specific databases you can search to find existing or in-process reviews

Alternatively, you can use a search hedge/filter; for example, the filter used by  BMJ Best Practice  to find systematic reviews in Embase (can be copied and pasted into the Embase search box then combined with the concepts of your research question):

(exp review/ or (literature adj3 review$).ti,ab. or exp meta analysis/ or exp "Systematic Review"/) and ((medline or medlars or embase or pubmed or cinahl or amed or psychlit or psyclit or psychinfo or psycinfo or scisearch or cochrane).ti,ab. or RETRACTED ARTICLE/) or (systematic$ adj2 (review$ or overview)).ti,ab. or (meta?anal$ or meta anal$ or meta-anal$ or metaanal$ or metanal$).ti,ab.

Alternative interface to PubMed: You can also search MEDLINE on the Ovid platform, which we recommend for systematic searching. Perform a sufficiently developed search strategy (be as broad in your search as is reasonably possible) and then, from Additional Limits , select the publication type  Systematic Reviews, or select the subject subset  Systematic Reviews Pre 2019 for more sensitive/less precise results. 

The subject subset for Systematic Reviews is based on the filter version used in PubMed .

Perform a sufficiently developed search strategy (be as broad in your search as is reasonably possible) and then, from  Additional Limits , select, under  Methodology,  0830 Systematic Review

See Systematic Reviews Search Strategy Applied in PubMed for details.

• healthevidence.org Database of thousands of "quality-rated reviews on the effectiveness of public health interventions"
• See also: Evidence-informed resources for Public Health

Munn Z, Stern C, Aromataris E, Lockwood C, Jordan Z. What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences. BMC Med Res Methodol. 2018;18(1):5. doi: 10.1186/s12874-017-0468-4

Scoping reviews: Developing the title and question . In: Aromataris E, Munn Z (Editors) . JBI Manual for Evidence Synthesis.   JBI; 2020. https://doi.org/10.46658/JBIMES-20-01

Due to a large influx of requests, there may be an extended wait time for librarian support on knowledge syntheses.

Find a librarian in your subject area to help you with your knowledge synthesis project.

Or contact the librarians at the Schulich Library of Physical Sciences, Life Sciences, and Engineering s [email protected]

Need help? Ask us!

Online training resources.

• Advanced Research Skills: Conducting Literature and Systematic Reviews A short course for graduate students to increase their proficiency in conducting research for literature and systematic reviews developed by the Toronto Metropolitan University (formerly Ryerson).
• The Art and Science of Searching in Systematic Reviews Self-paced course on search strategies, information sources, project management, and reporting (National University of Singapore)
• CERTaIN: Knowledge Synthesis: Systematic Reviews and Clinical Decision Making "Learn how to interpret and report systematic review and meta-analysis results, and define strategies for searching and critically appraising scientific literature" (MDAndersonX)
• Cochrane Interactive Learning Online modules that walk you through the process of working on a Cochrane intervention review. Module 1 is free (login to access) but otherwise payment is required to complete the online training
• Compétences avancées en matière de recherche : Effectuer des revues de la littérature et des revues systématiques (2e édition) Ce cours destiné aux étudiant.e.s universitaires vise à peaufiner leurs compétences dans la réalisation de revues systématiques et de recherches dans la littérature en vue de mener avec succès leurs propres recherches durant leur parcours universitaire et leur éventuelle carrière.
• The Essentials of Conducting Systematic Reviews - Module 4: Searching for Eligible Studies Module 4 explores where and how to search for studies, how to manage search results, and how to report the search process.
• Evidence Synthesis for Librarians and Information Specialists Introduction to core components of evidence synthesis. Developed by the Evidence Synthesis Institute. Free for a limited time as of July 10, 2024.
• Evidence Synthesis Training (Environmental Evidence) Open educational training courses covering different evidence synthesis methods suitable for environmental and ecological research, such as systematic review and systematic mapping, stakeholder engagement, and evidence synthesis technologies. Free
• Introduction to Systematic Review and Meta-Analysis Free coursera MOOC offered by Johns Hopkins University; covers the whole process of conducting a systematic review; week 3 focuses on searching and assessing bias
• Mieux réussir un examen de la portée en sciences de la santé : une boîte à outils Cette ressource éducative libre (REL) est conçue pour soutenir les étudiant·e·s universitaires en sciences de la santé dans la préparation d’un examen de la portée de qualité.
• Scoping Review Methods for Producing Research Syntheses Two-part, online workshop sponsored by the Center on Knowledge Translation for Disability and Rehabilitation Research (KTDRR)
• Systematic Reviews and Meta-Analysis Online overview of the steps involved in systematic reviews of quantitative studies, with options to practice. Courtesy of the Campbell Collaboration and the Open Learning Initiative (Carnegie Mellon University). Free pilot
• Systematic Reviews of Animal Studies (SYRCLE) Introduction to systematic reviews of animal studies
• Systematic Searches Developed by the Harvey Cushing/John Hay Whitney Medical Library (Yale University)
• << Previous: Types of knowledge syntheses
• Next: Developing the protocol >>
• Last Updated: Sep 17, 2024 4:10 PM
• URL: https://libraryguides.mcgill.ca/knowledge-syntheses

McGill Libraries • Questions? Ask us! Privacy notice

Systematic Reviews in Health

• Introduction

Define your question with the PICO Framework

Writing your question statement.

• Research Protocol
• Database Searching
• Article Screening
• Data Extraction
• Quality Appraisal
• Evidence Synthesis
• Interpret Results
• Reporting with PRISMA 2020

Asking a Clinical Question with PICO (video)

This YouTube presentation by Jeffery Hill covers structuring a focussed clinical question using the PICO framework:

PICO Framework

The first step in performing a Systematic Review is to formulate the research question.    Without a well-focused question, it can be very difficult and time consuming to identify appropriate resources and search for relevant evidence. Practitioners of Evidence-Based Practice (EBP) often use a specialised framework, called PICO , to form the question and facilitate the literature search. 1 PICO stands for:

• P atient Problem, (or Population)
• I ntervention,
• C omparison or Control, and

For Systematic Reviews an additional item,  T  is sometimes added to the framework.  The T can stand for :

• T ype of study  (e.g. Randomised Controlled Trials), 
• T ype of question , (i.e. Therapy, Prevention, Diagnosis, Prognosis, Etiology), 
• T ype of study participants , 
• T ype of intervention , 
• T ype of outcome measure , or 
• T imeframe .

When forming your question using PICO , keep the following points in mind:

• Your P atient is a member of a population as well as a person with (or at risk of) a health problem. So, in addition to age and gender, you may also need to consider ethnicity, socioeconomic status or other demographic variables.
• A C omparison is not always present in a PICO analysis.
• O utcomes should be measurable as the best evidence comes from rigorous studies with statistically significant findings.
• An O utcome ideally measures clinical wellbeing or quality of life, and not alternates such as laboratory test results.

PICO Elements Change According to Question Type (Domain)

When forming your question using the PICO framework it is useful to think about what type of question it is you are asking, (therapy, prevention, diagnosis, prognosis, etiology). The table below illustrates ways in which P roblems, I nterventions, C omparisons and O utcomes vary according to the t ype (domain) of your question. 2

Once you have clearly identified the main elements of your question using the PICO framework, it is easy to write your question statement.  The following table provides some examples.

1. Schardt, C., Adams, M. B., Owens, T., Keitz, S., & Fontelo, P. (2007). Utilization of the PICO framework to improve searching PubMed for clinical questions. BMC Medical Informatics and Decision Making , 7, 16. doi: http://dx.doi.org/10.1186/1472-6947-7-1

2. Fineout-Overholt, E., & Johnston, L. (2005). Teaching EBP: asking searchable, answerable clinical questions. Worldviews On Evidence-Based Nursing , 2, 157-160. 

• << Previous: Introduction
• Next: Research Protocol >>
• Last Updated: Aug 14, 2024 4:07 PM
• URL: https://canberra.libguides.com/systematic

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

The PMC website is updating on October 15, 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• BMC Health Serv Res

PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews

Abigail m methley.

University of Manchester, Centre for Primary Care, Williamson Building, Oxford Road, Manchester, M13 9PL UK

Stephen Campbell

NIHR Greater Manchester Primary Care Patient Safety Translational Research Centre, Institute of Population Health, The University of Manchester, Manchester, M13 9WL UK

Carolyn Chew-Graham

Institute of Primary Care and Health Sciences, Keele University, Keele, UK

Rosalind McNally

Central Manchester Hospitals Site, Manchester Mental Health and Social Care Trust, Research and Innovation 3rd Floor, Rawnsley Building, Hathersage Road, Manchester, M13 9WL UK

Sudeh Cheraghi-Sohi

Qualitative systematic reviews are increasing in popularity in evidence based health care. Difficulties have been reported in conducting literature searches of qualitative research using the PICO search tool. An alternative search tool, entitled SPIDER, was recently developed for more effective searching of qualitative research, but remained untested beyond its development team.

In this article we tested the ‘SPIDER’ search tool in a systematic narrative review of qualitative literature investigating the health care experiences of people with Multiple Sclerosis. Identical search terms were combined into the PICO or SPIDER search tool and compared across Ovid MEDLINE, Ovid EMBASE and EBSCO CINAHL Plus databases. In addition, we added to this method by comparing initial SPIDER and PICO tools to a modified version of PICO with added qualitative search terms (PICOS).

Results showed a greater number of hits from the PICO searches, in comparison to the SPIDER searches, with greater sensitivity. SPIDER searches showed greatest specificity for every database. The modified PICO demonstrated equal or higher sensitivity than SPIDER searches, and equal or lower specificity than SPIDER searches. The modified PICO demonstrated lower sensitivity and greater specificity than PICO searches.

Conclusions

The recommendations for practice are therefore to use the PICO tool for a fully comprehensive search but the PICOS tool where time and resources are limited. Based on these limited findings the SPIDER tool would not be recommended due to the risk of not identifying relevant papers, but has potential due to its greater specificity.

Systematic reviews are a crucial method, underpinning evidence based practice and informing health care decisions [ 1 , 2 ]. Traditionally systematic reviews are completed using an objective and primarily quantitative approach [ 3 ] whereby a comprehensive search is conducted, attempting to identify all relevant articles which are then integrated and assimilated through statistical analysis. The comprehensiveness of the search process has been viewed as a key factor in preventing bias and providing a true representation of available research [ 4 ]. Current research investigating the process of quantitative systematic reviews therefore focuses on methods for ensuring the most comprehensive and bias free searches possible [ 5 ]. Because of the time and resources required to complete a systematic and comprehensive search, efforts have been made to investigate the sensitivity of searches, and thus lessen the amount of time spent reviewing irrelevant articles with no benefit [ 6 ].

However, conducting comprehensive searches also forms the bedrock of qualitative or narrative reviews, now commonly referred to as qualitative evidence syntheses [ 7 ]. Qualitative evidence syntheses are now acknowledged as a necessary and valuable type of information to answer health services research questions [ 8 ]. However, difficulties in completing a sensitive yet comprehensive search of qualitative literature have been previously noted [ 9 - 11 ] including: poor indexing and use of key words of qualitative studies, the common use of titles that lack the keywords describing the article, and unstructured abstracts.

When devising a search strategy, a search tool is used as an organising framework to list terms by the main concepts in the search question, especially in teams where it is not possible to have an experienced information specialist as a member of the review team. The PICO tool focuses on the Population, Intervention, Comparison and Outcomes of a (usually quantitative) article. It is commonly used to identify components of clinical evidence for systematic reviews in evidence based medicine and is endorsed by the Cochrane Collaboration [ 2 ]. Due to its target literature base several of these search terms such as “control group” and “intervention” are not relevant to qualitative research which traditionally does not utilise control groups or interventions, and therefore may not appropriately locate qualitative research. However, these terms may become more relevant in the future as more trials and interventions incorporate qualitative research [ 12 ].

As the PICO tool does not currently accommodate terms relating to qualitative research or specific qualitative designs, it has often been modified in practice to “PICOS” where the “S” refers to the Study design [ 4 ], thus limiting the number of irrelevant articles.

Cooke et al. also addressed this issue of relevance by developing a new search tool entitled “SPIDER” (sample, phenomenon of interest, design, evaluation, research type), designed specifically to identify relevant qualitative and mixed-method studies [ 9 ]. The key features and differences of the SPIDER and PICO search tools are shown in Table  1 . The addition of the “design” and “research type” categories to the SPIDER tool was intended to further increase the ability of this tool to identify qualitative articles, whilst removing irrelevant PICO categories such as the “comparison” group [ 9 ].

Search categories and SPIDER and PICO headings

Cooke et al. recommended that the SPIDER tool was tested further in qualitative literature searches [ 9 ]. Although it has been used previously in a scoping review to investigate gaps in an evidence base on community participation in rural health care [ 13 ], SPIDER has not yet been tested and evaluated in a qualitative systematic narrative review context. The authors of this article recently completed a systematic review of the qualitative research investigating experiences of health care services for people with Multiple Sclerosis [ 14 ]. On embarking on this review topic we faced many of the difficulties commonly discussed in identifying qualitative literature on a given topic, and identified SPIDER as a potential way of overcoming some of these difficulties. Therefore, the aim of this article was to test SPIDER by broadly replicating the work of Cooke et al. [ 9 ], specifically by comparing the two approaches: 1) the traditional PICO method of searching electronic databases with 2) the newly devised SPIDER tool, developed for qualitative and mixed-method research. In addition we wished to build and expand on the work of Cooke et al. [ 9 ] and so our third aim was to compare PICO and SPIDER to a modified PICO with qualitative study designs (PICOS, see Table  1 by investigating specificity and sensitivity across 3 major databases.

Inclusion and exclusion criteria

Studies eligible for inclusion were those that qualitatively investigated patients’ experiences, views, attitudes to and perceptions of health care services for Multiple Sclerosis. No date restriction was imposed on searches as this was an original review. Qualitative research, for this purpose, was defined by the Cochrane qualitative methods group [ 7 ] as using both a qualitative data collection method and qualitative analysis. Quantitative and mixed method studies were therefore excluded.

We define experience as “ Patients’ reports of how care was organised and delivered to meet their needs p.301” [ 15 ]. Patients’ reports could refer to either experience of health care services delivery and organisation overall or their experiences of care by specific health care personnel. We included studies that investigated adults (aged 18 years old and older) with a diagnosis of Multiple Sclerosis, who had experience of utilising health care services at any time point. There were no restrictions on subtype of Multiple Sclerosis, gender, ethnicity or frequency of use of health care. Health care in this sense referred to routine clinical care (either state funded or privately funded) not trial protocols or interventions. Excluded studies included studies that focussed on self-management and studies that investigated quality of life.

Because of the focus on Multiple Sclerosis, studies were excluded if they used a mixed sample of various conditions (e.g. studies reported a mixed sample of people with neurological conditions) or if they used a sample of mixed respondents (i.e. people with Multiple Sclerosis and their carers) where results of patients with Multiple Sclerosis could not be clearly separated. If an article had a section or subtheme on health care services but this was not the main research area of the article, then that article was included; however only data from the relevant subtheme were extracted and included in the findings. Additional exclusion criteria were articles that only described carer or health care professional experiences not patient experiences. Conference abstracts, editorials and commentaries were not included.

Search strategy

For this systematic search we developed a detailed search strategy in collaboration with a specialist librarian and information specialist. This search strategy was tailored to the three largest medical and nursing databases (Ovid MEDLINE, Ovid EMBASE, and EBSCO CINAHL Plus) as in Cooke et al.’s study [ 9 ] and search terms used a mixture of medical subject headings and keywords. To investigate the benefit of the SPIDER,PICO and PICOS tools we used identical search terms but combined them in different ways as shown in Tables  2 , ​ ,3 3 and ​ and4 4 below.

The search terms used in the SPIDER search

a [S AND P of I] AND [(D or E) AND R].

Footnote: * is a truncation symbol to retrieve terms with a common root within CINHAL Plus and MEDLINE. $ is a truncation symbol to retrieve terms with a common root within EMBASE.

The search terms used in the PICO search

a (P and I and O).

The terms used in the PICOS search

a (P AND I AND C AND O AND S).

One reviewer judged titles and abstracts against the inclusion criteria. If a title and abstract met the inclusion criteria then full text copies of all articles were retrieved for further investigation. Two authors reviewed these full text articles independently for relevance to the search aim (i.e. patients/service users with multiple sclerosis, experiences of health care services and qualitative research). Any disagreements were resolved via discussion. Data from included studies were extracted by both reviewers independently to ensure accuracy and then stored on a Microsoft Excel spread sheet. No ethical approval was required for this study.

All searches spanned from database inception until 12th October 2013. As in Cooke et al. [ 9 ], we reviewed our findings based on two metrics; the number of hits generated and of these, the number relevant to the search aim (see Table  5 ).

Hits generated and articles searched

Number of articles generated

As found in Cooke et al. [ 9 ], PICO created a much greater number of hits compared to SPIDER. A total of 23758 hits were generated using PICO, 448 hits were generated using PICOS and 239 hits were generated using SPIDER. Overall, the average reduction of hits (% across all three databases) was 98.58% for SPIDER vs. PICO, 97.94% for PICO vs. PICOS and 68.64% for PICOS vs. SPIDER. The time spent screening hits for relevant articles equated to weeks for the PICO hits and hours for the PICOS and SPIDER hits.

Proportion of relevant articles

Articles which met the inclusion criteria after full text review are displayed in Table  6 [ 16 - 33 ]. Examination of the titles and abstracts of the identified articles resulted in the obtainment of 18 full text articles relevant at full text, across all databases and search tools.

Articles identified by database and search tool

For the PICO tool in CINAHL Plus, 5.78% of hits were deemed relevant after the title and abstract stage (78 articles/1350 articles), and 14/78 articles (17.95%) were confirmed to meet the inclusion criteria after full text review. For the PICO tool in MEDLINE, 0.42% of hits were deemed relevant after the title and abstract stage (34 articles/8158 articles) and 12/34 (35.29%) articles were confirmed to meet the inclusion criteria after full text review. For the PICO tool in EMBASE, 0.25% hits were deemed relevant after the title and abstract stage (35 articles/ 14250 articles) and 14/35(40%) articles were confirmed to meet the inclusion criteria after full text review.

For the PICOS tool in CINAHL Plus, 38.36% of articles were relevant after the title and abstract stage (56 articles/146 articles) and 12/56 (21.43%) were confirmed to meet the inclusion criteria after full text review. For the PICOS tool in MEDLINE 14.16% of articles were relevant after the title and abstract stage (16 articles/ 113 articles) and 6/16 (37.5%) were confirmed to meet the inclusion criteria after full text review. For the PICOS tool in EMBASE 7.94% of articles were deemed relevant after the title and abstract stage (15 articles/189 articles) and 7/15 (46.67%) were confirmed to meet the inclusion criteria after full text review.

SPIDER tool

For the SPIDER tool in CINAHL Plus 38.36% of articles were relevant after the title and abstract stage (56 articles/146 articles) and 12/56 (21.43%) were confirmed to meet the inclusion criteria after full text review. For the SPIDER tool in MEDLINE, 36.81% hits were deemed relevant at the title stage (14 articles/38 articles) and 5/14 articles (35.71%) were confirmed to meet the inclusion criteria after full text review. For the SPIDER tool in EMBASE, 16.36% were relevant at the title stage (9 articles/55 articles) and 3/9 (33.33%) were confirmed to meet the inclusion criteria after full text review.

Sensitivity and specificity

The SPIDER tool identified 13 relevant articles out of 239 articles across all three databases (5.43%) compared to PICOS which identified 13 articles out of 448 articles (2.90%) and PICO which identified 18 articles out of 23758 articles (0.076%). Of the 18 relevant articles identified by the PICO tool, 66.66% came from both MEDLINE and CINAHL Plus (12 articles each), and 72.22% came from EMBASE (13 articles). Of the 13 relevant articles identified by the PICOS tool 46.15% came from MEDLINE (6 articles), 53.84% came from EMBASE (7 articles) and 92.31% came from CINAHL Plus (12 articles). Of the 13 relevant articles identified by SPIDER, 38.46% came from MEDLINE (5 articles) and 23.07% came from EMBASE (3 articles) and 92.30% came from CINAHL Plus (12 articles) Table  7 .

Sensitivity and specificity for each search tool by database

Different articles were found across different tools and databases (as shown in Table  6 ). All three databases were checked for all articles. One article was available in CINAHL Plus but not identified by any of the tools [ 17 ]. Two papers were identified in all databases through all search tools. Five papers were identified in MEDLINE through all search tools, three identified in EMBASE through all search tools and 12 identified in CINAHL through all search tools. Five papers were identified solely in CINAHL Plus, with one of these papers only identified using the PICO search method. One paper was identified by all search tools in EMBASE but not identified by any in MEDLINE. No new studies were identified using the SPIDER or PICOS tools alone in any database.

In this article we addressed the aim of replicating a comparison between the SPIDER, PICOS and PICO search tools. As previously described in Cooke et al. [ 9 ], the SPIDER tool produced a greatly reduced number of initial hits to sift through, however in this study it missed five studies that were identified through the PICO method. This may be partly be explained by the nature of the research question prompting the search. As this study included subthemes of studies whose focus differed from the initial research question (i.e. only a smaller section of the paper related to health care) then it’s possible that these studies were picked up by a broader search but not the highly specific SPIDER search. Other authors researching the process of qualitative literature reviews have previously commented that there appears to be a decision to be made about the benefits of comprehensiveness of findings versus the accuracy of the studies identified [ 11 ]. Given the common nature of using sub-sections of papers for systematic reviews then our findings suggest that comprehensiveness needs to be the key for this type of search.

The PICOS tool was more specific than the PICO tool, but did not identify any additional relevant hits to the SPIDER tool, suggesting it is of approximately equal sensitivity. PICOS identified the same number of papers as the SPIDER tool and both demonstrated a substantially lower number of hits generated than a regular PICO search. The SPIDER tool showed the greatest specificity due the small number of hits generated. This may mean that review teams with very limited resources or time, and who are not aiming for a totally comprehensive search (i.e. in the case of scoping studies), would benefit from using the SPIDER tool. This might be applicable particularly to studies such as qualitative syntheses, where the research aim is theoretical saturation, not a comprehensive search [ 34 ]. In addition, articles written to influence policy often require swift publication, providing another area in which either SPIDER or PICOS might improve current practice.

The issue of time was also related to the number of relevant articles identified per database. Whilst EMBASE generated nearly twice as many hits as MEDLINE, only one additional paper was found. The PICO tool identified all articles, suggesting that where time is not a factor, it might be of more benefit to use this tool, as SPIDER demonstrated lower sensitivity, did not identify any new articles and identified fewer relevant articles than PICO.

Our findings indicate that it is worthwhile testing a chosen search tool across various databases as they produce different results; i.e. CINHAL Plus identified papers not identified in MEDLINE or EMBASE databases. It is therefore important for future research to investigate the potential of the SPIDER vs. PICOS and PICO tools as a base for the recommended comprehensive searching process, by investigating the contribution of the SPIDER and PICOS tools at every stage from the initial search hits, to the final included relevant articles.

As CINAHL is a database dedicated to nursing and allied health research, it was expected that it would produce a greater number of relevant articles than more medically focussed databases [ 10 ], as nursing and allied areas have traditionally been at the forefront of qualitative investigations into Multiple Sclerosis.

SPIDER proved to be a tool designed to formulate search terms easily, as it naturally fits the crucial elements of the search question. However, even though some qualitative keywords are necessary to identify qualitative studies, including the words “ qualitative research ” AND the name of the type of research e.g. “ grounded theory ” might be too restrictive, particularly given the poor use of the qualitative index term, and might partially explain the fewer studies identified by SPIDER in comparison to PICO. Studies not identified by the SPIDER model in MEDLINE and EMBASE databases did not use keywords such as “ qualitative ”, but some described qualitative methods, such as “ phenomenological-hermeneutic ” [ 16 ] or “ interview(s) ” [ 20 , 23 ].

In all PICO searches for MEDLINE and EMBASE the word “ qualitative ” combined with the phrase “ multiple sclerosis ” identified many quantitative studies reporting brain scan assessments that were wholly unrelated to the search aim. This was because the word “ qualitative ” in this context referred to using a qualitative method to provide information about the quality of the scan and any potential flaws [ 35 ]. This caused a problem with specificity, resulting in thousands of inappropriate hits as there was no way to exclude studies with the word “ qualitative ” unless all articles clearly utilised and indexed qualitative research methods in the title, abstract and keywords.

Many studies were excluded at the full text stage on the basis that the samples were mixed: being comprised of either various neurological conditions or mixed groups of people i.e. patients and carers/patients and health care professionals and so forth. Without clearer titles and abstracts, and potentially an indexing phrase that indicates mixed samples, there is no way of avoiding this issue. Excluding the phrases “ caregivers ” or “ health care professionals ” would have excluded any studies that used these phrases (for example in the introduction or implication for future research sections) and therefore it is difficult to see how this could be prevented. A strength and limitation of our study is that whilst it details a real world example of evidence searching, it only addresses one topic. Further research should test these search tools against a wider variety of narrative review and meta-synthesis topics.

SPIDER greatly reduced the initial number of articles identified on a given search due to increased specificity, however because of lower sensitivity omitted many relevant papers. The PICOS tool resulted in an overall more sensitive search, but still demonstrated poor specificity on this topic. Further investigations of the specificity and sensitivity of SPIDER and PICOS on varied topics will be of benefit to research teams with limited time and resources or articles necessary to impact on policy or change current practice. However, where comprehensiveness is a key factor we suggest that the PICO tool should be used preferentially. Part of the lower identification rate for SPIDER (in comparison to PICO) was poor labelling and use of qualitative keywords in indexing studies. As both individual research submissions and journal/database indexers improve, or standardise, the indexing of qualitative studies, it is likely that the relevance of the SPIDER tool will increase. The recommendation for current practice therefore is to use the PICO tool across a variety of databases. In this article we have shown that SPIDER is relevant for those researchers completing systematic narrative reviews of qualitative literature but not as effective as PICO. Future research should investigate the use of SPIDER and PICOS across varied databases.

Acknowledgements

This study was funded by a School for Primary Care Research PhD studentship from the National Institute of Health Research. Support in selecting search terms is acknowledged from Olivia Walsby, Academic Engagement Librarian at the University of Manchester. We are grateful to Professor Peter Bower for his comments on the protocol.

Competing interests

The authors declare that they have no competing interests.

Authors’ contributions

AM designed the study, conducted all searches, appraised all potential studies and wrote and revised the draft manuscript and subsequent manuscripts. SC made significant contributions to the conception and design of the study, assisted with the presentation of findings and assisted with drafting and revising the manuscript. CCG and RM made significant contributions to the conception and design of the study, assisted with the presentation of findings and assisted with drafting and revising the manuscript. SCS conceived and designed the study, assisted with searches, appraised relevant studies and assisted with drafting and revising the manuscript. All authors read and approved the final manuscript.

Authors’ information

Caroly Chew-Graham is part-funded by the National Institute for Health Research (NIHR) Collaborations for Leadership in Applied Health Research and Care West Midlands.

Contributor Information

Abigail M Methley, Email: [email protected] .

Stephen Campbell, Email: [email protected] .

Carolyn Chew-Graham, Email: [email protected] .

Rosalind McNally, Email: [email protected] .

Sudeh Cheraghi-Sohi, Email: [email protected] .

A .gov website belongs to an official government organization in the United States.

A lock ( ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

• General Committee-Related Information
• ACIP Work Groups
• ACIP Meeting Information
• ACIP Recommendations
• Apply for ACIP Membership
• ACIP Committee Members
• Evidence-Based Recommendations—GRADE
• GRADE Evidence Tables – Recommendations in MMWR
• Evidence to Recommendations Frameworks

Related Topics:

• View All Home
• ACIP GRADE Handbook
• Vaccine-Specific Recommendations
• Vaccines & Immunizations

Grading of Recommendations, Assessment, Development, and Evaluation (GRADE): Novavax COVID-19 Vaccine

CDC vaccine recommendations are developed using an explicit evidence-based method based on the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) approach.

A Grading of Recommendations, Assessment, Development and Evaluation (GRADE) review of the evidence for benefits and harms for Novavax coronavirus disease 2019 (COVID-19) vaccine was presented to the Advisory Committee on Immunization Practices (ACIP) on July 19, 2022. GRADE evidence type indicates the certainty in estimates from the available body of evidence. Evidence certainty ranges from type 1 (high certainty) to type 4 (very low certainty) 1 .

The policy question was, "Should vaccination with Novavax COVID-19 vaccine be recommended for persons 18 years of age and older during an Emergency Use Authorization?" The potential benefits pre-specified by the ACIP COVID-19 Vaccines Work Group were prevention of symptomatic laboratory-confirmed COVID-19 (critical), hospitalization due to COVID-19 (critical), death due to COVID-19 (important), and asymptomatic SARS-CoV-2 infection (important). The two pre-specified harms were serious adverse events (critical) and reactogenicity grade ≥3 (important).

A systematic review of evidence on the efficacy and safety of a two-dose regimen of Novavax COVID-19 vaccine among persons aged 18 years and older was conducted. The quality of evidence from one Phase III randomized controlled trial was assessed using a modified GRADE approach. 2 3

A lower risk of symptomatic COVID-19 was observed with vaccination compared to placebo (relative risk [RR] 0.10, 95% confidence interval [CI]: 0.06, 0.18, evidence type 1), corresponding to a vaccine efficacy of 89.6% (95% CI: 82.4%, 93.8%). This was observed with a median follow-up of 2.5 months, during a period of Alpha variant predominance. The vaccine was also associated with a lower risk of severe illness due to COVID-19 (RR 0.00; 95% CI: 0.00, 1.00; evidence type 3), corresponding to a vaccine efficacy of 100% (95% CI: 0%, 100%). The measure of severe COVID-19 was used as surrogate for the GRADE outcome of hospitalization due to COVID-19. No hospitalizations or deaths due to COVID-19 were identified among vaccine recipients or placebo recipients in the per-protocol population.*

In terms of harms, the available data indicated that serious adverse events were balanced between the vaccine and placebo arms (RR 0.92; 95% CI: 0.73, 1.16; evidence type 1). Reactogenicity grade ≥3 was associated with vaccination (RR 4.11; 95% CI: 3.70, 4.57; evidence type 1), 16.3% of vaccine recipients and 4% of placebo recipients reported any grade ≥3 local or systemic reactions following either dose 1 or dose 2.

Introduction

On July 13, 2022, the U.S. Food and Drug Administration (FDA) issued an Emergency Use Authorization (EUA) for Novavax COVID-19 (NVX-CoV2373) vaccine for prevention of symptomatic COVID-19 for persons aged ≥18 years. 4 As part of the process employed by the ACIP, a systematic review and GRADE evaluation of the evidence for Novavax COVID-19 vaccine was conducted and presented to ACIP. The ACIP adopted a modified GRADE approach in 2010 as the framework for evaluating the scientific evidence that informs recommendations for vaccine use. Evidence of benefits and harms were reviewed based on the GRADE approach. 1

The policy question was, "Should vaccination with Novavax COVID-19 vaccine be recommended for persons 18 years of age and older under an Emergency Use Authorization?" (Table 1).

We conducted a systematic review of evidence on the efficacy and safety of a two-dose regimen of Novavax (5 μg antigen plus 50 μg Matrix-M adjuvant) COVID-19 vaccine. We assessed outcomes and evaluated the quality of evidence using the GRADE approach.

During Work Group calls, members were asked to pre-specify and rate the importance of relevant patient-important outcomes (including benefits and harms) before the GRADE assessment. No conflicts of interest were reported by CDC and ACIP COVID-19 Vaccines Work Group members involved in the GRADE analysis. Outcomes of interest included individual benefits and harms (Table 2). The critical benefits of interest are prevention of symptomatic laboratory-confirmed COVID-19 and prevention of hospitalization due to COVID-19. Other important outcomes include prevention of death due to COVID-19 and prevention of asymptomatic SARS-CoV-2 infection. The critical harm of interest was serious adverse events; reactogenicity grade ≥3 was deemed an important harm.

We identified clinical trials through clinicaltrials.gov. Records of relevant Phase I, II, or III RCTs of COVID-19 vaccine were included if they 1) provided data on persons aged ≥18 years vaccinated with NVX-CoV2373; 2) involved human subjects; 3) reported primary data; and 4) included data relevant to the efficacy and safety outcomes being measured. We identified relevant observational studies through an ongoing systematic review conducted by the International Vaccine Access Center (IVAC) and the World Health Organization (WHO). 4 Relevant observational studies were restricted to the defined population, intervention, comparison, and outcome outlined in the policy question, or related outcomes if direct data were not available. In addition, unpublished and other relevant data were obtained by hand-searching reference lists, and consulting with vaccine manufacturers and subject matter experts. The systematic review was limited to studies published from January 1, 2020 to July 7, 2022. Characteristics of all included studies are shown in Appendix 1 and evidence retrieval methods are found in Appendix 2.

The evidence certainty assessment addressed risk of bias, inconsistency, indirectness, imprecision, and other characteristics. The GRADE assessment across the body of evidence for each outcome was presented in an evidence profile; the evidence certainty of Type 1, 2, 3, or 4 corresponds to high, moderate, low, or very low certainty, respectively.

Relative risks (RR) were calculated from numerators and denominators available in the body of evidence. Vaccine efficacy estimates were defined as 100% x (1-RR).

GRADE Summary

The initial GRADE evidence level was type 1 (high) for each outcome because the body of evidence was from a randomized controlled trial (Table 4). In terms of critical benefits, the available data indicated that the vaccine was effective for preventing symptomatic COVID-19 during a period of Alpha variant predominance, and no serious concerns impacting certainty in the estimate were identified (type 1, high). The certainty in the effect estimate for severe illness due to COVID-19 was downgraded one point for serious concern of indirectness because severe illness due to COVID-19 was evaluated as a surrogate for hospitalization due to COVID-19 and one point for serious concern of imprecision due to the small number of events during the observation period (type 3, low). No serious concerns impacted the certainty in the estimate for serious adverse events or reactogenicity (both type 1, high) (Table 4).

* Cases were included in the analysis if they were confirmed in a designated central laboratory, per manufacturer protocol

† Additional studies which evaluated the Novavax vaccine were excluded as the vaccines were manufactured at a different facility and by a different process

Table 1: Policy Question and PICO

Abbreviations : IM = intramuscular.

a Assessed through serial PCR testing.

Table 2: Outcomes and Rankings

a Severe illness due to COVID-19 evaluated as a surrogate measure for this critical outcome.

b No events occurred in the study included in the review of evidence.

c Data were not available to inform an evaluation of asymptomatic SARS-CoV-2 infection in studies identified in the review of evidence.

Table 3a: Summary of Studies Reporting Symptomatic Laboratory-confirmed COVID-19

References in this table: 2 3

Abbreviations: RT-PCR = real-time polymerase chain reaction; CI = confidence interval; RR = relative risk.

a 19,965 and 9,984 persons were randomized to vaccine and placebo

b Primary outcome, defined as SARS-CoV-2 RT-PCR-positive symptomatic illness, in seronegative adults, ≥7 days post vaccination.

c Symptomatic illness defined as any mild, moderate or severe COVID-19. Mild COVID-19 was defined as fever, new onset cough OR ≥2 additional COVID-19 symptoms: pyretic, new onset or worsening of shortness of breath or difficulty breathing compared to baseline, new onset fatigue, new onset generalized muscle or body aches, new onset headache, new loss of taste or smell, acute onset of sore throat, congestion, and runny nose, or new onset nausea, vomiting, or diarrhea. Moderate COVID-19 was defined as high fever (≥38.4°C) for ≥3 days or any evidence of significant lower respiratory tract infection. Severe COVID-19 was defined as any of the following symptoms: tachypnea: ≥ 30 breaths per minute at rest, resting heart rate ≥125 beats per minute, oxygen saturation ≤93% on room air or ratio of the partial pressure of arterial oxygen to the fraction of inspired oxygen <300 mm Hg, high flow oxygen therapy or non-invasive ventilation/non-invasive positive pressure ventilation, mechanical ventilation or extracorporeal membrane oxygenation, one or more major organ system dysfunction or failure.

d Based on data cutoff September 27, 2021; participants had a median of two months follow-up.

Table 3b: Summary of Studies Reporting Severe Illness due to COVID-19 a,b

References in this table: 3

Abbreviations: CI = confidence interval; RR = relative risk; NE=Not estimable

a Severe COVID-19 was defined as any of the following symptoms: tachypnea: ≥ 30 breaths per minute at rest, resting heart rate ≥125 beats per minute, oxygen saturation ≤93% on room air or ratio of the partial pressure of arterial oxygen to the fraction of inspired oxygen <300 mm Hg, high flow oxygen therapy or non-invasive ventilation/non-invasive positive pressure ventilation, mechanical ventilation or extracorporeal membrane oxygenation, one or more major organ system dysfunction or failure.

b Based on data cutoff September 27, 2021.

Table 3c: Summary of Studies Reporting Serious Adverse Events a,b

Abbreviations: RR = relative risk; CI = confidence interval; RCT = randomized controlled trial.

a Death, life-threatening event, hospitalization, incapacity to perform normal life functions, medically important event, or congenital anomaly/birth defect

b Five participants in the vaccine arm experienced SAEs that were considered related to vaccination by the investigator (n=1 each of headache, angioedema, Basedow's disease, thrombocytopenia, nervous system disorder). Among these, FDA considered the event of angioedema as potentially related to vaccination. There was one event of myocarditis in a 67-year-old male, with concomitant COVID-19 infection, 28 days after dose 1, which was not considered related to vaccination. Additional cases of myocarditis were observed after placebo crossover

Table 3d: Summary of Studies Reporting Reactogenicity a,b

Abbreviations: RR = relative risk; CI = confidence interval.

a Reactogenicity outcome includes local and systemic events, grade ≥3. For local reactions, grade 3 pain or tenderness defined as any narcotic pain reliever or prevents daily activity, grade 4 defined as emergency department visit or hospitalization. Grade 3 redness or swelling is defined as >10 cm or prevents daily activity, grade 4 is necrosis or exfoliative dermatitis. For systemic reactions, grade 3 fever defined as 39.0 to 40.0 °C, grade 4 defined as >40°C. Grade 3 headache, fatigue/malaise, muscle pain, and joint pain defined as any use of narcotic pain reliever or prevents daily activity, grade 4 defined as emergency department visit or hospitalization. Grade 3 nausea/vomiting defined as prevents daily activity or requires outpatient IV hydration, grade 4 defined as emergency department visit or hospitalization for hypotensive shock.

b Based on interim analysis, data cutoff September 27, 2021.

Table 4. Grade Summary of Findings Table

Abbreviations: CI = confidence interval; RR = relative risk; COVID-19 = coronavirus disease 2019; RCT = randomized controlled trial.

a. Risk of bias related to blinding of participants and personnel was present. Although participants and study staff were blinded to intervention assignments, they may have inferred receipt of vaccine or placebo based on reactogenicity. This was deemed unlikely to overestimate efficacy or underestimate risk of serious adverse events, therefore the risk of bias was rated as not serious.

b. Concern for indirectness was noted due to the short duration of observation in the available body of evidence. The vaccine efficacy observed at a median 2-month follow-up may differ from the efficacy observed with ongoing follow-up. However, in consideration that this recommendation is under an Emergency Use Authorization, the length of follow up time was deemed sufficient to support efficacy in that context. Additionally, it should be noted that the efficacy assessment took place during Alpha variant predominance and efficacy may differ for other variants.

c. The effects noted are from a per protocol analysis with outcomes assessed at least 7 days post dose 2 among persons who received two doses and had no evidence of prior SARS-CoV-2 infection.

d. The RCT excluded persons with prior COVID-19 diagnosis, pregnant women or women planning to become pregnant within 3 months of vaccination, and persons who were immunocompromised due to conditions and/or treatments (participants with stable/well-controlled HIV infection were not excluded). The population included in the RCT may not represent all persons aged ≥18 years.

e. Absolute risk was calculated using the observed outcomes in the placebo arm during the available clinical trial follow-up. Absolute risk estimates should be interpreted in this context.

f. Serious concern for indirectness was noted because severe illness due to COVID-19 is being evaluated as a surrogate for hospitalization due to COVID-19.

g. Serious concern for imprecision was noted due to the small number of events during the observation period.

h. There were no events in the vaccine group, therefore the relative risk was calculated using the standard offset of 0.5.

i. Absolute risk based on relative risk calculated using an offset due to zero events in the vaccine group

Appendix 1: Studies Included in the Review of Evidence

a Additional data provided by sponsor.References in this table:

Abbreviations: SD = standard deviation; RCT = randomized controlled trial; COVID-19 = coronavirus disease 2019.

Appendix 2. Databases and strategies used for systematic review

a. Most recent search conducted July 11, 2022.

View the complete list of GRADE evidence tables‎

• Ahmed F. U.S. Advisory Committee on Immunization Practices (ACIP) Handbook for Developing Evidence-based Recommendations.
• Dunkle LM, Kotloff KL, Gay CL, et al. Efficacy and safety of NVX-CoV2373 in adults in the United States and Mexico. NEJM . 2022. DOI: 10.1056/NEJMoa2116185.
• Novavax, 2022 personal communication, May 11 – June 3, 2022.
• Food and Drug Administration. Novavax COVID-19 Vaccine Emergency Use Authorization. Novavax Letter of Authorization 07132022 (fda.gov) .
• Food and Drug Administration (FDA). FDA Briefing Document – Sponsor: Novavax COVID-19 Vaccine. https://www.fda.gov/media/158914/download .
• Food and Drug Administration (FDA). FDA Briefing Document Novavax COVID-19 vaccine. https://www.fda.gov/media/158912/download

ACIP comprises medical and public health experts who develop recommendations on the use of vaccines in the civilian population of the United States.

• Open access
• Published: 18 September 2024

Mapping the evaluation of the electronic health system PEC e-SUS APS in Brazil: a scoping review protocol

• Mariano Felisberto 1 , 2 ,
• Júlia Meller Dias de Oliveira 1 , 3 ,
• Eduarda Talita Bramorski Mohr 1 , 2 ,
• Daniel Henrique Scandolara 1 , 4 ,
• Ianka Cristina Celuppi 1 , 5 ,
• Miliane dos Santos Fantonelli 1 ,
• Raul Sidnei Wazlawick 1 , 6 &
• Eduardo Monguilhott Dalmarco   ORCID: orcid.org/0000-0002-5220-5396 1 , 7  

Systematic Reviews volume  13 , Article number:  237 ( 2024 ) Cite this article

Metrics details

The Brazilian Ministry of Health has developed and provided the Citizen’s Electronic Health Record (PEC e-SUS APS), a health information system freely available for utilization by all municipalities. Given the substantial financial investment being made to enhance the quality of health services in the country, it is crucial to understand how users evaluate this product. Consequently, this scoping review aims to map studies that have evaluated the PEC e-SUS APS.

This scoping review is guided by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocols (PRISMA-P) framework, as well as by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Checklist extension for scoping reviews (PRISMA-ScR). The research question was framed based on the “CoCoPop” mnemonic (Condition, Context, Population). The final question posed is, “How has the Citizen’s Electronic Health Record (PEC e-SUS APS) been evaluated?” The search strategy will be executed across various databases (LILACS, PubMed/MEDLINE, Scopus, Web of Science, ACM Digital Library, and IEEE Digital Library), along with gray literature from ProQuest Dissertation and Theses Global and Google Scholar, with assistance from a professional healthcare librarian skilled in supporting systematic reviews. The database search will encompass the period from 2013 to 2024. Articles included will be selected by three independent reviewers in two stages, and the findings will undergo a descriptive analysis and synthesis following a “narrative review” approach. Independent reviewers will chart the data as outlined in the literature.

The implementation process for the PEC e-SUS APS can be influenced by the varying characteristics of the over 5500 Brazilian municipalities. These factors and other challenges encountered by health professionals and managers may prove pivotal for a municipality’s adoption of the PEC e-SUS APS system. With the literature mapping to be obtained from this review, vital insights into how users have evaluated the PEC will be obtained.

Systematic review registration

The protocol has been registered prospectively at the Open Science Framework platform under the number 10.17605/OSF.IO/NPKRU.

Peer Review reports

The Brazilian Unified Health System (SUS) was launched in Brazil in 1998 [ 1 , 2 , 3 ]. Its structure adheres to a triad of principles: integrality, universality, and equity of health services offered to the nation’s population [ 4 ]. In 1990, Primary Health Care (PHC) was established as a national policy under Basic Operational Standard 96, which provided support for the implementation of Family Health and Community Health Agents programs throughout Brazil [ 5 , 6 ]. Currently, PHC has become a central component within the organization of the health care network and is considered the main entry point to the Brazilian health system, extending healthcare provision throughout the entire territory [ 3 , 7 , 8 ].

Examining Brazil’s demographic and epidemiological aspects is crucial to ensure these services reach all citizens. Hence, health policy planning depends on this information, which is typically sourced from healthcare system data [ 9 ]. This data may represent the reality and needs of a specific community, municipality, state, or country and, thus, directly influences health surveillance activities, forming the basis of health service management [ 10 ]. Health information systems aim to generate, organize, and analyze health indicators, thereby producing knowledge about the health status of the population [ 11 ].

To digitize SUS and facilitate health professionals’ efforts in care coordination, the Brazilian Ministry of Health instituted the e-SUS Primary Care Strategy in 2013. Its key objectives were to individualize records, integrate data between official systems, reduce redundancy in data collection, and computerize health units [ 12 ]. It is worth noting that this strategy extends beyond a federal management and national information system context; it touches on the daily routines of professionals, the challenges faced, and the information essential for individual care in territories [ 13 ]. To further facilitate this process, the Ministry introduced the Citizen’s Electronic Health Record (PEC), which is a freely available health information system for municipalities, aiding the computerization of Basic Health Units throughout Brazil [ 1 , 14 ].

The role of software products and intensive computer systems has grown to become essential for a broad array of business and personal operations. Consequently, achieving personal satisfaction, business success, and human security increasingly rely on the quality of these software and systems [ 15 ]. The development and implementation of these technologies are fundamental; however, they require substantial financial resources, and their success hinges on user acceptance [ 16 ]. Therefore, it is critical for those investing in technology to understand what factors affect acceptance and usage, aiding organizations in implementing user-level interventions [ 17 ].

Understanding how users evaluate a software product is critical in a nation of continental proportions like Brazil, especially given the significant financial investment to enhance health services’ quality. Given this context, this scoping review aims to map out the studies that have evaluated the PEC e-SUS APS using various quality models. This will be done using ISO/IEC 25010 as a theoretical foundation to define these models, which present in-depth quality models for computer systems, software products, data quality, and usage.

The protocol and its registration have been adapted based on elements taken from the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocols (PRISMA-P) and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Checklist extension for scoping reviews (PRISMA-ScR) [ 18 , 19 ]. The adapted protocol was subsequently registered on Open Science Framework under the number https://doi.org/10.17605/OSF.IO/NPKRU . The research question was formulated and structured around the CoCoPop approach (Condition, Context, and Population), as shown in Table  1 .

Inclusion and exclusion criteria

All studies evaluating the PEC e-SUS APS will be considered for the inclusion criteria. Given the myriad aspects of electronic health record systems open to analysis (e.g., user experience, usability, efficiency, accessibility, security, and economic aspects), this review will include studies evaluating the general function and effectiveness of the PEC e-SUS APS, regardless of the language. The exclusion criteria will include studies that will not clearly outline the evaluation method used for the health information system; will not employ an evaluative tool or method; will focus solely on medical records differing from the PEC e-SUS APS; will be published before 2013 (i.e., PEC e-SUS APS was first distributed to municipalities in 2013); will be conducted by authors from the Bridge Laboratory (i.e., the group responsible for the PEC implementation); will be review articles, letters, book chapters, conference abstracts, opinion articles, brief communications, editorials, and clinical guidelines; and if the full text will not found for full reading or correspondence authors will not reply to contact attempts.

Sources of information and search strategy

A comprehensive search strategy will be deployed across various databases: LILACS, PubMed/MEDLINE, Scopus, Web of Science, ACM Digital Library, and IEEE Digital Library. Moreover, the gray literature will also be explored using the ProQuest Dissertation and Theses Global and Google Scholar databases with support from a healthcare librarian experienced in systematic reviews. The search strategy developed for the PubMed/MEDLINE databases is presented in Table  2 .

Furthermore, experts will be contacted for the potential inclusion of more studies, with manual searches of bibliographies from included studies and key journals also conducted. The database search will cover the period from 2013 until 2024. The search will be implemented in March 2024, and the results will be imported into the EndNote Online reference software (Thomson Reuters, USA).

Methods to select the sources of evidence

Three independent reviewers will decide on what will be included in the final studies. In the first stage, the three reviewers will assess the titles and abstracts for eligibility. In the second stage, they will examine the full texts of the articles, applying the same criteria as in the first stage. The reviewers will then cross-validate all the information gathered during both stages. If disagreements occur, an arbitrator, not involved in the initial article selection stage, will be brought in before a final decision is reached. If review-critical data are missing or ambiguous, the study’s corresponding author will be contacted for resolution or clarification. The data mapping process and related entities will involve these same three independent reviewers.

Data extraction and synthesis

A descriptive analysis will synthesize the results, following the narrative review approach of Pawson and Bellamy [ 20 ]. Independent reviewers will chart the data based on the method of Hilary Arksey and Lisa O’Malley (2005), as depicted in Table  3 .

In the event of discrepancies, a consensus discussion will ensue and, if necessary, independent reviewers will be brought in to reach a final decision. Any disagreements will be addressed among the reviewers. The corresponding author will be contacted if any crucial information is unclear or missing. The studies included will be grouped according to the various characteristics and sub-characteristics pertinent to all software products and computer systems, as defined by the ISO/IEC 25010–2011 standard .

Tabular summaries will be employed to present the findings and cover study characteristics, methodologies, and aspects evaluated. Subsequently, a narrative synthesis will be carried out to elucidate the evidence found relating to the review objective.

The success of PEC implementation can be influenced by various characteristics of municipalities, including their location, population density, level of urbanization, municipal management assistance, computerization levels, and technological infrastructure, among others [ 21 ]. These factors, coupled with the challenges confronted by healthcare professionals and managers, may determine a municipality’s adoption of the PEC. Literature emphasizes several barriers or difficulties encountered during implementation and usage, such as inadequate material resources in municipalities, lack of professional technology training, and poor internet connectivity [ 22 , 23 , 24 ].

Considering the myriad software product quality assessment models available, this review will utilize ISO/IEC 25010–2011 as its theoretical foundation. This model provides precise definitions of the attributes that must be evaluated. It is crucial to note that this international standard underwent rigorous evaluation by numerous international organizations before publication, reinforcing its suitability for assessing software product quality.

The literature map derived from this review will provide crucial insights into user evaluations of the PEC. Through these insights, it will be possible to identify the strengths and weaknesses of this software product. This knowledge will empower those responsible for developing and implementing this system to make significant improvements, thereby ensuring a substantial return on investment.

Availability of data and materials

Not applicable.

Avila GS, Cavalcante RB, Almeida NG, Gontijo TL, DE Souza Barbosa S, Brito MJ. Diffusion of the Electronic Citizen's Record in Family Health Teams. REME-Revista Mineira de Enfermagem. 2021. 25(1). https://periodicos.ufmg.br/index.php/reme/article/view/44494 .

Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32.

Article   Google Scholar  

Barros RD, Aquino R, Souza LE. Evolution of the structure and results of Primary Health Care in Brazil between 2008 and 2019. Cien Saude Colet. 2022;27:4289–301. https://doi.org/10.1590/1413-812320222711.02272022EN .

Article   PubMed   Google Scholar  

Brasil - Ministério da Saúde. Subchefia para Assuntos Jurídicos. Lei nº 8.080, de 19 de setembro de 1990: Lei Orgânica da Saúde. Dispõe sobre as condições para a promoção, proteção e recuperação da saúde, a organização e o funcionamento dos serviços correspondentes e dá outras providências. Brasília, 1990. http://www.planalto.gov.br/ccivil_03/leis/L8080.htm .

Sousa AN, Shimizu HE. Integrality and comprehensiveness of service provision in Primary Health Care in Brazil (2012-2018). Revista Brasileira de Enfermagem. 2021 https://doi.org/10.1590/0034-7167-2020-0500

Tasca R, Massuda A, Carvalho WM, Buchweitz C, Harzheim E. Recommendations to strengthen primary health care in Brazil. Revista Panamericana de Salud Pública. 2020 https://doi.org/10.37774/9789275726426 .

Brasil - Ministério da Saúde. Portaria nº 2.436, de 21 de setembro de 2017. Aprova a Política Nacional de Atenção Básica, estabelecendo a revisão de diretrizes para a organização da Atenção Básica, no âmbito do Sistema Único de Saúde (SUS). Diário Oficial da União; 2017. https://bvsms.saude.gov.br/bvs/saudelegis/gm/2017/prt2436_22_09_2017.html .

Mendonça MH, Matta GC, Gondim R, Giovanella L. Atenção primária à saúde no Brasil: conceitos, práticas e pesquisa. SciELO-Editora Fiocruz; 2018.

Mendes, E. V. As redes de atenção à saúde. Brasília: Organização Pan-Americana da Saúde, 2011. 549p. http://bvsms.saude.gov.br/bvs/publicacoes/redes_de_atencao_saude.pdf .

Brasil - Ministério da Saúde. Secretaria de Vigilância em Saúde. Guia de vigilância epidemiológica. 6 ed. Brasília, 2005. http://bvsms.saude.gov.br/bvs/publicacoes/Guia_Vig_Epid_novo2.pdf .

Mota E, Carvalho D. Sistemas de informação em saúde. Epidemiología & saúde. Rio de Janeiro: Editora Médica e Científica (MEDSI). 2003.

Brasil - Ministério da Saúde. Secretaria de Atenção à Saúde. Departamento de Atenção Básica. Diretrizes Nacionais de Implantação da Estratégia e-SUS AB. Brasília, 2014. http://bvsms.saude.gov.br/bvs/publicacoes/diretrizes_nacionais_implantacao_estrategia_esus.pdf .

Gaete RAC, Leite TA. Estratégia e-SUS Atenção Básica: o processo de reestruturação do sistema de informação da atenção básica. In: Congresso Brasileiro em Informática em Saúde – CBIS, 14, 2014, Santos. [s.n.], 2014.

Brasil. Nota Técnica 07/2013: Estratégia e-SUS Atenção Básica e Sistema de Informação em Saúde da Atenção Básica-SISAB. 2013. Disponível em: https://www.conass.org.br/biblioteca/wp-content/uploads/2013/01/NT-07-2013-e-SUS-e-SISAB.pdf .

ISO/IEC 25010:2011: Systems and software engineering — Systems and software Quality Requirements and Evaluation (SQuaRE) — System and software quality models. Geneva, Switzerland.: ISO Copyright Office, 2011.

Venkatesh V, Morris MG, Davis GB, Davis FD. User acceptance of information technology: Toward a unified view. MIS Quarterly, 2003. 27(3), 425–478. https://doi.org/10.2307/30036540

Pinho C, Franco M, Mendes L. Web portals as tools to support information management in higher education institutions: A systematic literature review. International Journal of Information Management, 2018. 41, 80–92. https://doi.org/10.1016/j.ijinfomgt.2018.04.002

Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, Shekelle P, Stewart LA, Group, P.-P. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4:1–9.

Article   PubMed   PubMed Central   Google Scholar  

Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, Moher D, Peters MD, Horsley T, Weeks L. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467–73.

Pawson R, & Bellamy JL. Realist synthesis: an explanatory focus for systematic review. In: POPAY, J. (Ed.). Moving beyond effectiveness in evidence synthesis: Methodological issues in the synthesis of diverse sources of evidence, 2006. 83–94.

Cielo AC, Raiol T, Silva EN, Barreto JO. Implementation of the e-SUS Primary Care Strategy: an analysis based on official data. Revista de Saúde Pública. 2022 https://doi.org/10.11606/s1518-8787.2022056003405 .

Gontijo TL, Lima PKM, Guimarães EAA, Oliveira VC, Quites HFO, Belo VS, et al. Computerization of primary health care: the manager as a change agent. Rev Bras Enferm. 2021;74(2):e20180855. https://doi.org/10.1590/0034-7167-2018-0855 .

Santos LPR, Pereira AG, Graever L, Guimarães RM. e-SUS AB na cidade do Rio de Janeiro: projeto e implantação do sistema de informação em saúde. Cad saúde colet. 2021. 29(spe):199–204. https://doi.org/10.1590/1414-462X202199010232 .

Zacharias FC, Schönholzer TE, Oliveira VC, Gaete RA, Perez G, Fabriz LA, Amaral GG, Pinto IC. Primary Healthcare e-SUS: determinant attributes for the adoption and use of a technological innovation. Cad Saude Publica. 2021;37:e00219520. https://doi.org/10.1590/0102-311X00219520 .

Download references

Acknowledgements

The authors would like to thank Mrs. Karyn Munik Lehmkuhl for her support with the search strategies.

This study will be financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior [Coordination for the Improvement of Higher Education Personnel] – Brazil (CAPES) – Finance Code 001, and by Brazilian Ministry of Health (e-SUS PHC Project Stage 6). The RSW and EMD are productivity fellows in technology development and innovative extension of CNPq.

Author information

Authors and affiliations.

Bridge Laboratory, Federal University of Santa Catarina, Florianópolis, Brazil

Mariano Felisberto, Júlia Meller Dias de Oliveira, Eduarda Talita Bramorski Mohr, Daniel Henrique Scandolara, Ianka Cristina Celuppi, Miliane dos Santos Fantonelli, Raul Sidnei Wazlawick & Eduardo Monguilhott Dalmarco

Graduate Program in Pharmacy, Federal University of Santa Catarina, Florianópolis, Brazil

Mariano Felisberto & Eduarda Talita Bramorski Mohr

Graduate Program in Dentistry, Federal University of Santa Catarina, Florianópolis, Brazil

Júlia Meller Dias de Oliveira

Graduate Program in Engineering, Management, and Knowledge Media, Federal University of Santa Catarina, Florianópolis, Brazil

Daniel Henrique Scandolara

Department of Nursing, Federal University of Santa Catarina, Florianópolis, Brazil

Ianka Cristina Celuppi

Department of Informatics and Statistics, Federal University of Santa Catarina, Florianópolis, Brazil

Raul Sidnei Wazlawick

Department of Clinical Analysis, Federal University of Santa Catarina, Florianópolis, Brazil

Eduardo Monguilhott Dalmarco

You can also search for this author in PubMed   Google Scholar

Contributions

All co-authors constructed, read, and approved of the final manuscript.

Corresponding author

Correspondence to Eduardo Monguilhott Dalmarco .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare the presence of financial and political conflicts of interest related to the content of this study protocol. The Laboratório Bridge is involved in the development and maintenance of the PEC e-SUS APS, a health information system, in collaboration with the Brazilian Ministry of Health. This collaboration entails financial and political agreements, as the Ministry of Health is a governmental institution responsible for public health in Brazil. We acknowledge that these conflicts may influence our research and analysis. However, we are committed to reporting the results impartially and transparently in the future, following the ethical and editorial guidelines of the international scientific journal in which this article will be published. Our financial and political interests will not compromise the integrity of the research or the objectivity in presenting the results.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Felisberto, M., de Oliveira, J.M.D., Mohr, E.T.B. et al. Mapping the evaluation of the electronic health system PEC e-SUS APS in Brazil: a scoping review protocol. Syst Rev 13 , 237 (2024). https://doi.org/10.1186/s13643-024-02648-4

Download citation

Received : 09 April 2024

Accepted : 23 August 2024

Published : 18 September 2024

DOI : https://doi.org/10.1186/s13643-024-02648-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Electronic health records
• Management in health
• Primary health care
• Health information systems
• Diffusion of innovation
• Scoping review

Systematic Reviews

ISSN: 2046-4053

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

• Open access
• Published: 16 September 2024

Response is increased using postal rather than electronic questionnaires – new results from an updated Cochrane Systematic Review

• Phil Edwards 1 &
• Chloe Perkins 1  

BMC Medical Research Methodology volume  24 , Article number:  209 ( 2024 ) Cite this article

3 Altmetric

Metrics details

A decade ago paper questionnaires were more common in epidemiology than those administered online, but increasing Internet access may have changed this. Researchers planning to use a self-administered questionnaire should know whether response rates to questionnaires administered electronically differ to those of questionnaires administered by post. We analysed trials included in a recently updated Cochrane Review to answer this question.

We exported data of randomised controlled trials included in three comparisons in the Cochrane Review that had evaluated hypotheses relevant to our research objective and imported them into Stata for a series of meta-analyses not conducted in the Cochrane review. We pooled odds ratios for response using random effects meta-analyses. We explored causes of heterogeneity among study results using subgroups. We assessed evidence for reporting bias using Harbord’s modified test for small-study effects.

Twenty-seven trials (66,118 participants) evaluated the effect on response of an electronic questionnaire compared with postal. Results were heterogeneous (I-squared = 98%). There was evidence for biased (greater) effect estimates in studies at high risk of bias; A synthesis of studies at low risk of bias indicates that response was increased (OR = 1.43; 95% CI 1.08–1.89) using postal questionnaires. Ten trials (39,523 participants) evaluated the effect of providing a choice of mode (postal or electronic) compared to an electronic questionnaire only. Response was increased with a choice of mode (OR = 1.63; 95% CI 1.18–2.26). Eight trials (20,909 participants) evaluated the effect of a choice of mode (electronic or postal) compared to a postal questionnaire only. There was no evidence for an effect on response of a choice of mode compared with postal only (OR = 0.94; 95% CI 0.86–1.02).

Conclusions

Postal questionnaires should be used in preference to, or offered in addition to, electronic modes.

Peer Review reports

Introduction

When collecting information from large, geographically dispersed populations, a self-administered questionnaire is usually the only financially viable option [ 1 ]. Non-responses to questionnaires reduce effective sample sizes, reducing study power, and may introduce bias in study results [ 2 ]. The Cochrane Methodology Review of methods to increase response to self-administered questionnaires has provided a much-used scientific evidence base for effective data collection by questionnaire since the publication of the first version of the review in 2003 which focused on postal questionnaires [ 3 ].

A decade ago paper-and-pencil administration of questionnaires in epidemiological studies was twenty times more common than electronic administration [ 4 ], but increased Internet access and decreasing volumes of mailed letters suggests that electronic administration has gained favour [ 5 , 6 , 7 ]. Researchers planning to collect data from participants using a self-administered questionnaire need to know how will the proportion of participants responding to a questionnaire administered electronically compare with one administered by post? We conducted further analyses of the trials included in the recently updated Cochrane Review [ 8 ] to answer this question.

To assess whether response rates to questionnaires administered electronically differ to those of questionnaires administered by post.

Data sources/measurement

We exported data of randomised controlled trials included in the updated Cochrane Review [ 8 ] from RevMan and imported them into Stata for a series of meta-analyses not conducted in the Cochrane review.

Comparisons

We focused on data from trials included in three comparisons in the Cochrane Review that had evaluated hypotheses relevant to our research objective:

Postal vs. electronic questionnaire (Cochrane Comparison 81).

Electronic questionnaire only vs. choice (postal or electronic) (Cochrane Comparison 84).

Choice (electronic or postal) vs. postal questionnaire only (Cochrane Comparison 82).

These comparisons assess: response to questionnaires administered by post compared with questionnaires administered electronically, response to a questionnaire administered electronically compared with response when including a postal response option, and response when including an electronic response option compared with response to a questionnaire administered by post only, respectively.

Outcome measures

The data obtained from each trial included the numbers of participants randomised to each arm of the trial with the numbers of completed, or partially completed questionnaires returned after all mailings (for trials including a postal questionnaire), and the numbers of participants randomised to each arm of the trial with the numbers of participants submitting the completed, or partially completed online questionnaires after all contacts (electronic questionnaire).

Other variables

Additional data were also extracted on the:

Year of publication of the study.

Risk of bias in each included study (a judgment - high, low, or unclear); we assessed the overall risk of bias in each study using the Cochrane Collaboration’s tool [ 9 ].

Effect measures and synthesis

For each of the three comparisons (2.1.1 above), we pooled the odds ratios for response in each included study in a random effects meta-analysis (to allow for heterogeneity of effect estimates between studies) using the metan command in Stata [ 10 ]. This command also produced a forest plot (a visual display of the results of the individual studies and syntheses) for each comparison. We quantified any heterogeneity using the I 2 statistic that describes the percentage of the variability in effect estimates that is due to heterogeneity [ 11 ].

Subgroup analyses

We explored possible causes of heterogeneity among study results by conducting subgroup analyses according to two study-level factors: Year of study publication, and risk of bias in studies. We used a statistical test of homogeneity of the pooled effects in subgroups to assess evidence for subgroup differences. The statistical test of homogeneity used is Cochran’s Q test, where the Q statistic is distributed as a chi-square statistic with k-1 degrees of freedom, where k is the number of subgroups. If there was evidence for subgroup differences provided by the test of homogeneity, we chose the ‘best estimate of effect’ as the estimate from the subgroup of studies with low risk of bias, or the subgroup of studies published after 2012. If there was no evidence for subgroup differences, we chose our best estimate of effect based on the synthesis of all studies.

Year of study publication

From 2012, household access to a computer exceeded 40%: [ 5 ] As the odds ratios for response to questionnaires administered electronically may be associated with household access to a computer, we analysed trial results in two subgroups – before 2012 and after 2012, where we used the year of publication as an approximation of the year of study conduct.

Risk of bias

The odds ratios for response estimated in the included studies may be associated with trial quality. [ 12 , 13 ] For this reason we analysed trial results in two subgroups – trials judged to be at low and at high risk of bias.

Reporting bias assessment

We assessed evidence for reporting bias using Harbord’s modified test for small-study effects implemented in Stata using the metabias command [ 14 ]. This test maintains better control of the false-positive rate than the test proposed by Egger at al [ 14 ].

Study characteristics

Thirty-five studies [ 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 ] reported 45 independent trials included in one or more of the three comparisons (Table  1 ). The studies were conducted in the US ( n  = 20), Europe ( n  = 13), and Australasia ( n  = 2). The studies included between 133 and 12,734 participants and were published between 2001 and 2020. Eight studies were judged to be at high risk of bias [ 16 , 19 , 33 , 34 , 42 , 43 , 45 , 46 ].

Results of syntheses

Comparison 1 - postal vs. electronic questionnaire.

Twenty-seven trials (66,118 participants) evaluated the effect on questionnaire response of postal administration compared with electronic. [ 15 , 16 , 17 , 18 , 19 , 20 , 23 , 24 , 25 , 26 , 27 , 28 , 31 , 32 , 33 , 34 , 35 , 36 , 38 , 39 , 40 , 41 , 43 , 44 , 46 , 47 , 48 ] The odds of response were increased by over half (OR 1.76; 95% CI 1.34 to 2.32) using a postal questionnaire when compared with an electronic one (Fig.  1 ). There was considerable heterogeneity between the trial results (I-squared = 98%), but most of the studies showed response was greater with postal questionnaires than with electronic questionnaires, and the high I-squared is due to differences in the size of the benefit for postal questionnaires, rather than being due to an even spread of results between those favouring postal and those favouring electronic questionnaires.

Effect on response of mode of administration

Comparison 2 - electronic questionnaire only vs. choice (postal or electronic)

Ten trials (39,523 participants) evaluated the effect on questionnaire response of providing a choice of response mode (postal or electronic) compared to an electronic questionnaire only [ 20 , 21 , 27 , 29 , 30 , 35 , 37 , 40 , 42 , 45 ]. The odds of response were increased by over half when providing a choice of response mode (OR 1.63; 95% CI 1.18 to 2.26; Fig.  2 ). There was considerable heterogeneity between the trial results (I-squared = 97.1%), but again, most of the studies favoured giving people the choice of response mode rather than electronic questionnaire only, and the high I-squared is due to differences in the size of the benefit for choice, rather than being due to an even spread of results between those favouring choice and those favouring electronic only.

Effect on response of choice of response mode compared with electronic only

Comparison 3 - choice (electronic or postal) vs. postal only

Eight trials (20,909 participants) evaluated the effect of providing a choice of response mode (electronic or postal) compared to postal response only [ 20 , 22 , 27 , 29 , 34 , 35 , 40 , 49 ]. There was no evidence for an effect on response of providing a choice (OR 0.94; 95% CI 0.86 to 1.02; Fig.  3 ). There was moderate heterogeneity among the trial results (I-squared = 50.9%).

Effect on response of choice of response mode compared with postal only

Results of subgroup analyses

Table  2 presents the results of subgroup analyses according to the two study-level factors (forest plots of these subgroup analyses are included in supplementary figures).

Comparison 1 - postal vs. electronic questionnaire

Year of publication.

A third of studies were published before 2012 [ 15 , 16 , 17 , 23 , 24 , 33 , 35 , 40 , 47 , 48 ]. In this subgroup of studies the odds of response were 85% greater (OR 1.85; 95% CI 1.12 to 3.06) with a postal questionnaire compared with an electronic one. In the subgroup of studies published after 2012 the effect was lower (OR 1.70; 1.19 to 2.43), consistent with our concern (Sect.  2.4.1 ) that higher household access to a computer from 2012 may have improved preference for electronic questionnaires, however the statistical test of homogeneity of the pooled effects in these two subgroups was not significant ( p  = 0.788), indicating no evidence from these studies for different effects by year of study (Supplementary Fig.  1 a).

Seven of the trials [ 16 , 17 , 26 , 33 , 34 , 43 , 46 ] were judged to be at high risk of bias and for these trials the odds of response were more than tripled (OR 3.24; 95% CI 1.68 to 6.25) using a postal questionnaire when compared with an electronic one. There was considerable heterogeneity between the trial results (I-squared = 99%).

When only the 20 trials deemed to be at low risk of bias were synthesised, the odds of response were increased by two-fifths (OR 1.43; 95% CI 1.08 to 1.89). There was also considerable heterogeneity between these trial results (I-squared = 96.8%).

The statistical test of homogeneity of the pooled effects in these two subgroups ( p  = 0.025) provides some evidence for greater effect estimates in studies at high risk of bias (Supplementary Fig.  1 b). Our best estimate of the effect on response of mode of administration is hence from a synthesis of the studies at low risk of bias (OR 1.43; 95% CI 1.08 to 1.89). Results overall were thus confounded by risk of bias, but this did not explain the between study heterogeneity.

Year of study

Half of studies were published before 2012 [ 35 , 40 , 42 , 45 ]. In this subgroup of studies there was no evidence for an effect on response of providing a postal response option (OR 1.22; 95% CI 0.93 to 1.61). In the subgroup of studies published after 2012 there was evidence for an effect on response of providing a postal response option (OR 2.02; 95% CI 1.30 to 3.13). The statistical test of homogeneity of the pooled effects in these two subgroups was significant ( p  = 0.057), indicating some evidence from these studies for different effects by year of study (Supplementary Fig.  2 a). This apparent preference for a postal response option in studies published after 2012 was counter to our concern (Sect.  2.4.1 ) that higher household access to a computer from 2012 would improve preference for electronic questionnaires. There was considerable heterogeneity between the trial results (I-squared = 98.2%), but most of the studies favoured giving people the choice of response mode rather than electronic questionnaire only, and the high I-squared is due to differences in the size of the benefit for choice, rather than being due to an even spread of results between those favouring choice and those favouring electronic only.

Two of the trials were judged to be at high risk of bias [ 42 , 45 ]. There was no evidence for an effect on response of a postal option in these studies (OR 1.08; 95% CI 0.43 to 2.71). When only the 8 trials deemed to be at low risk of bias were synthesised, there was evidence that the odds of response were increased when providing a postal response option (OR 1.77; 95% CI 1.23 to 2.55). There was considerable heterogeneity between these trial results (I-squared = 97.7%). The statistical test of homogeneity of the pooled effects in these two subgroups ( p  = 0.326) provides no evidence for different effects by risk of bias (Supplementary Fig.  2 b). Our best estimate of the effect on response of providing a postal response option is hence from a synthesis of all of these studies (OR 1.63; 95% CI 1.18 to 2.26).

Comparison 3 - choice (electronic or postal) vs. postal questionnaire only

In the subgroup of studies published before 2012 there was very weak evidence that the odds of response were lower with an electronic option (OR 0.85; 0.73 to 0.98), whereas in studies published after 2012 there was no evidence for a difference between an electronic option and postal only – perhaps due to electronic methods being more acceptable with increased computer access. The results in both subgroups were more homogeneous (I-squared = 48.5% and 7.0% respectively). The statistical test of homogeneity of the pooled effects in these two subgroups ( p  = 0.04) provides some evidence for different effects by year of study (Supplementary Fig.  3 a). If we consider the most recent trials to better represent the situation today (i.e., greater access to computers than prior to 2012), then our best estimate of the effect on response of providing an electronic response option is from a synthesis of the studies published after 2012 (OR 1.01; 95% CI 0.93 to 1.08), i.e., no evidence for an effect.

There was one study at high risk of bias [ 34 ]. Its results were entirely consistent with the results of the seven studies at low risk of bias (the statistical test of homogeneity of the pooled effects in these two subgroups was not significant ( p  = 0.454), Supplementary Fig.  3 b).

Results of assessments of evidence for reporting bias

There was no evidence for small study effects (Harbord’s modified test p  = 0.148).

There was no evidence for small study effects (Harbord’s modified test p  = 0.841).

There was no evidence for small study effects (Harbord’s modified test p  = 0.139).

General interpretation of the results in the context of other evidence

This study has shown that response to a postal questionnaire is more likely than response to an electronic questionnaire. It has also shown that response is more likely when providing the option for postal response with an electronic questionnaire. It has further shown that providing an electronic response option with a postal questionnaire has no effect on response. Response is thus increased using postal rather than electronic questionnaires.

A previous meta-analysis of 43 mixed-mode surveys from 1996 to 2006 also found paper and postal administration produced greater response than electronic administration [ 50 ]. Our result that providing an electronic response option to postal administration does not increase response is consistent with a previous meta-analysis of randomised trials that found that mailed surveys that incorporate a concurrent Web option have significantly lower response rates than those that do not [ 51 ].

We suggest two possible reasons for these results:

Paper questionnaires are more  accessible  than electronic questionnaires .

Although access to the Internet increased over the period during which the studies included in this study were conducted [ 5 , 52 ], a ‘digital divide’ [ 53 ] persists in many populations where completion of a paper questionnaire may be possible, but completion of an electronic one may not.

Paper questionnaires are more  personal  than electronic questionnaires .

Personalised materials have been shown to increase response [ 54 ]. If participants perceive a paper questionnaire with a return envelope to be more ‘personal’ than a request to go to a website to answer some questions, we should expect a higher response with paper.

Strengths and limitations

The main strengths of this study are that our results are based on syntheses of the results of 45 randomised controlled trials that span two decades, and most of which were judged to be at low risk of bias.

There was, however, considerable heterogeneity between the results of the included studies. Our subgroup analyses did not identify any causes of heterogeneity among study results, but they did reveal confounding of the pooled result for postal versus electronic questionnaires. The unexplained heterogeneity means that we cannot be confident about the magnitude of the effects on response using postal rather than electronic questionnaires. However, from inspection of the forest plots we can be confident about the direction of these effects.

The evidence included in this review addresses ‘unit’ non-response only (i.e., return of questionnaires). ‘Item’ response (i.e., completion of individual questions) may be greater with electronic methods, but this was not addressed in this review and requires investigation in the future.

We assessed evidence for reporting bias using Harbord’s modified test for small-study effects and found no evidence for bias. This test may not be reliable given the substantial heterogeneity between the results of the included trials [ 55 ].

Due to the nature of this study (secondary analysis of a published review), there is no pre-registered protocol for the subgroup analyses provided in this study.

Implications for practice, policy, and future research

These results will help researchers and healthcare providers to improve data collection from study participants and patients, helping to maintain study power and reduce bias due to missing data in research studies. In addition to the methods already known to be effective in increasing questionnaire response [ 8 , 56 ], postal questionnaires should be used in preference to, or offered in addition to, electronic modes as this will help to increase the proportion of participants that responds. It should be noted, however, that the evidence upon which this recommendation is based is from studies published between 2001 and 2020, and this may change in the future as access to the Internet increases and more people become ‘tech-savvy’. Furthermore, we consider that the certainty of the evidence provided in this study is “Moderate”, due to the unexplained heterogeneity between the results of the included studies.

Future research

Evidence on effective data collection in low- and middle-income settings is needed. Research centres such as LSHTM can embed studies within trials (SWATs) in their research in these settings to help to increase the evidence base [ 57 ].

Participation rates for epidemiologic studies have been declining [ 58 ]. Our study has presented evidence that postal questionnaires are preferable to electronic questionnaires to improve participation, but it does not tell us why . Research is still needed to advance sociological and psychological theories of participation in data collection procedures [ 59 ].

Electronic administration provides benefits for researchers over paper administration which have not been addressed by this study: A well-designed Web questionnaire can control skip patterns, check for allowable values and ranges and response consistencies, and it can include instructions and explanations about why a question is being asked [ 60 ]. These options could help to improve the completeness and quality of self-administered data collection, maintaining study power, reducing the risk of bias in study results, and saving study resources. Further research into the cost-effectiveness of electronic administration compared with postal administration in different settings will be needed to inform practice [ 61 ].

Data availability

Data extracted from included studies will be available in the forthcoming update on the Cochrane Library.

Armstrong BK, White E, Saracci R. Principles of exposure measurement in Epidemiology. New York: Oxford University Press; 1992.

Book   Google Scholar  

Greenland S. Response and follow-up bias in cohort studies. Am J Epidemiol. 1977;106:184–7.

Article   CAS   PubMed   Google Scholar  

Edwards PJ, Roberts IG, Clarke MJ, DiGuiseppi C, Wentz R, Kwan I, Cooper R, Felix L, Pratap S. Methods to increase response rates to postal questionnaires. Cochrane Database of systematic reviews 2007, issue 2. Art No : MR000008 https://doi.org/10.1002/14651858.MR000008.pub3

van Gelder MMHJ, Reini W, Bretveld RW, Nel Roeleveld N. Web-based questionnaires: the Future in Epidemiology? Am J Epidemiol. 2010;172:1292–8.

Article   PubMed   Google Scholar  

Alsop T. Share of households with a computer at home worldwide from 2005 to 2019 https://www.statista.com/statistics/748551/worldwide-households-with-computer/ [Accessed 4 May 2024].

Mazareanu E. United States Postal Service - mail volume. https://www.statista.com/statistics/320234/mail-volume-of-the-usps/#:text=United%20States%20Postal%20Service%20%2D%20mail%20volume%202004%2D2020&text=After%20reaching%20a%20peak%20of,to%20just%20129.2%20billion%20units [Accessed 4 May 2024].

BBC. What is happening to the Royal Mail? http://news.bbc.co.uk/2/hi/business/8304722.stm [Accessed 4 May 2024].

Edwards PJ, Roberts I, Clarke MJ, DiGuiseppi C, Woolf B, Perkins C. Methods to increase response to postal and electronic questionnaires. Cochrane Database Syst Rev. 2023;11(11):MR000008. https://doi.org/10.1002/14651858.MR000008.pub5 .

Higgins JP, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne JA, Cochrane Bias Methods Group; Cochrane Statistical Methods Group. The Cochrane collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928. https://doi.org/10.1136/bmj.d5928 .

Article   PubMed   PubMed Central   Google Scholar  

Harris RJ, Deeks JJ, Altman DG, Bradburn MJ, Harbord RM, Sterne JAC. Metan: fixed- and Random-effects Meta-Analysis. Stata J. 2008).;8(1):3–28. https://doi.org/10.1177/1536867X0800800102 .

Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ. 2003;327:557–60.

Schultz, et al. Empirical evidence of Bias - dimensions of Methodological Quality Associated with estimates of Treatment effects in controlled trials. JAMA. 1995;273(5):408–12.

Article   Google Scholar  

Sterne JA, Jüni P, Schulz KF, Altman DG, Bartlett C, Egger M. Statistical methods for assessing the influence of study characteristics on treatment effects in ‘meta-epidemiological’ research. Stat Med . 2002;21(11):1513-24. https://doi.org/10.1002/sim.1184 . PMID: 12111917.

Harbord RM, Harris RJ, Sterne JA. Updated tests for small-study effects in meta-analyses. Stata J. 2009;9(2):197–210.

Akl EA, Maroun N, Klocke RA, Montori V, Schünemann HJ. Electronic mail was not better than postal mail for surveying residents and faculty. J Clin Epidemiol. 2005;58(4):425–9.

Basnov M, Kongsved SM, Bech P, Hjollund NH. Reliability of short form-36 in an Internet- and a pen-and-paper version. Inform Health Soc Care. 2009;34(1):53 – 8. doi: 10.1080/17538150902779527. PMID: 19306199.

Bech M, Kristensen MB. Differential response rates in postal and web-based surveys among older respondents. Surv Res Methods. 2009;3(1):1–6.

Google Scholar  

Beebe TJ, Jacobson RM, Jenkins SM, Lackore KA, Rutten LJF. Testing the impact of mixed-Mode designs (mail and web) and multiple contact attempts within Mode (Mail or web) on Clinician Survey Response. Health Serv Res. 2018;53(Suppl 1):3070–83. Epub 2018 Jan 22. PMID: 29355920; PMCID: PMC6056581.

Bergeson SC, Gray J, Ehrmantraut LA, Laibson T, Hays RD. Comparing Web-based with Mail Survey Administration of the Consumer Assessment of Healthcare Providers and Systems (CAHPS R) Clinician and Group Survey. Primary health care: open access. 2013;3.

Bjertnaes O, Iversen HH, Skrivarhaug T. A randomized comparison of three data collection models for the measurement of parent experiences with diabetes outpatient care. BMC Med Res Methodol. 2018;18(1):95. https://doi.org/10.1186/s12874-018-0557-z . PMID: 30236067; PMCID: PMC6149010.

Bray I, Noble S, Robinson R, Molloy L, Tilling K. Mode of delivery affected questionnaire response rates in a birth cohort study. J Clin Epidemiol. 2017;81:64–71.

Brøgger J, Nystad W, Cappelen I, Bakke P. No increase in response rate by adding a web response option to a postal population survey: a randomized trial. J Med Internet Res. 2007;9(5):e40.

Clark M, Rogers M, Foster A, Dvorchak F, Saadeh F, Weaver J, et al. A randomized trial of the impact of survey design characteristics on response rates among nursing home providers. Eval Health Prof. 2011;34(4):464–86.

Cobanoglu C, Moreo PJ, Warde B. A comparison of mail, fax and web-based Survey methods. Int J Market Res. 2001;43(4):1–15.

Fluss E, Bond CM, Jones GT, Macfarlane GJ. The effect of an internet option and single-sided printing format to increase the response rate to a population-based study: a randomized controlled trial. BMC Med Res Methodol 2014;14.

Fowler FJ Jr, Cosenza C, Cripps LA, Edgman-Levitan S, Cleary PD. The effect of administration mode on CAHPS survey response rates and results: a comparison of mail and web-based approaches. Health Serv Res. 2019;54(3):714–21.

Hardigan PC, Succar CT, Fleisher JM. An analysis of response rate and economic costs between mail and web-based surveys among practicing dentists: a randomized trial. J Community Health. 2012;37(2):383–94.

Hardigan PC, Popovici I, Carvajal MJ. Response rate, response time, and economic costs of survey research: a randomized trial of practicing pharmacists. Res Social Administrative Pharm. 2016;12(1):141–8.

Hohwu L, Lyshol H, Gissler M, Jonsson SH, Petzold M, Obel C. Web-based Versus Traditional Paper questionnaires: a mixed-Mode Survey with a nordic perspective. J Med Internet Res 2013;15(8).

Iversen HH, Holmboe O, Bjertnaes O. Patient-reported experiences with general practitioners: a randomised study of mail and web-based approaches following a national survey. BMJ open. 2020;10(10):e036533.

Jacob RT, Jacob B. Prenotification, incentives, and Survey Modality: an experimental test of methods to increase Survey Response Rates of School principals. J Res Educational Eff. 2012;5(4):401–18.

Lagerros YT, Sandin S, Bexelius C, Litton JE, Löf M. Estimating physical activity using a cell phone questionnaire sent by means of short message service (SMS): a randomized population-based study. Eur J Epidemiol. 2012;27(7):561–6.

Leece P, Bhandari M, Sprague S, Swiontkowski MF, Schemitsch EH, Tornetta P, et al. Internet versus mailed questionnaires: a randomized comparison (2). J Med Internet Res. 2004;6(3):26–33.

Mauz E, Hoffmann R, Houben R, Krause L, Kamtsiuris P, Gößwald A. Population-Based Health Interview Surveys for Children and Adolescents: Methodological Study. J Med Internet Res. 2018;20(3):e64. https://doi.org/10.2196/jmir.7802 . PMID: 29506967; PMCID: PMC5859740. Mode Equivalence of Health Indicators Between Data Collection Modes and Mixed-Mode Survey Designs in.

Millar MM, Dillman DA. Improving response to web and mixed-Mode surveys. Pub Opin Q. 2011;75(2):249–69.

Millar MM, Elena JW, Gallicchio L, Edwards SL, Carter ME, Herget KA, Sweeney C. The feasibility of web surveys for obtaining patient-reported outcomes from cancer survivors: a randomized experiment comparing survey modes and brochure enclosures. BMC Med Res Methodol. 2019;19(1):208.

Murphy CC, Craddock Lee SJ, Geiger AM, Cox JV, Ahn C, Nair R, Gerber DE, Halm EA, McCallister K, Skinner CS. A randomized trial of mail and email recruitment strategies for a physician survey on clinical trial accrual. BMC Med Res Methodol. 2020;20(1):123.

Reinisch JF, Yu DC, Li WY. Getting a Valid Survey Response From 662 Plastic Surgeons in the 21st Century. Ann Plast Surg. 2016;76(1):3–5. https://doi.org/10.1097/SAP.0000000000000546 . PMID: 26418779.

Sakshaug JW, Vicari B, Couper MP. Paper, e-mail, or both? Effects of contact mode on participation in a web survey of establishments. Social Sci Comput Rev. 2019;37(6):750–65.

Schmuhl P, Van Duker H, Gurley KL, Webster A, Olson LM. Reaching emergency medical services providers: is one survey mode better than another? Prehospital Emerg Care. 2010;14(3):361–9.

Schwartzenberger J, Presson A, Lyle A, O’Farrell A, Tyser AR. Remote Collection of patient-reported outcomes following outpatient hand surgery: a Randomized Trial of Telephone, Mail, and E-Mail. J Hand Surg. 2017;42(9):693–9.

Scott A, Jeon SH, Joyce CM, Humphreys JS, Kalb G, Witt J et al. A randomised trial and economic evaluation of the effect of response mode on response rate, response bias, and item non-response in a survey of doctors. BMC Med Res Methodol 2011;11.

Sebo P, Maisonneuve H, Cerutti B, Fournier JP, Senn N, Haller DM, Rates. Delays, and Completeness of General Practitioners’ Responses to a Postal Versus Web-Based Survey: A Randomized Trial. J Med Internet Res 2017;19(3).

Taylor S, Ferguson C, Peng F, Schoeneich M, Picard RW. Use of In-Game rewards to Motivate Daily Self-Report Compliance: Randomized Controlled Trial. J Med Internet Res. 2019;21(1):e11683. https://doi.org/10.2196/11683 . PMID: 30609986; PMCID: PMC6682282.

van den Berg MH, Overbeek A, van der Pal HJ, Versluys AB, Bresters D, van Leeuwen FE et al. Using web-based and paper-based questionnaires for Collecting Data on Fertility Issues among Female Childhood Cancer survivors: differences in response characteristics. J Med Internet Res 2011;13(3).

Weaver L, Beebe TJ, Rockwood T. The impact of survey mode on the response rate in a survey of the factors that influence Minnesota physicians’ disclosure practices. BMC Med Res Methodol. 2019;19(1):73.

Whitehead L. Methodological issues in internet-mediated research: a randomized comparison of internet versus mailed questionnaires. J Med Internet Res. 2011;13(4):e109.

Yetter G, Capaccioli K. Differences in responses to web and paper surveys among school professionals. Behav Res Methods. 2010;42(1):266–72.

Ziegenfuss JY, Beebe TJ, Rey E, Schleck C, Locke GR 3rd, Talley NJ. Internet option in a mail survey: more harm than good? Epidemiology. 2010;21(4):585–6. https://doi.org/10.1097/EDE.0b013e3181e09657 .

Shih TH, Fan X. Response Rates and Mode preferences in Web-Mail mixed-Mode surveys: a Meta-analysis. Int J Internet Sci. 2007;2(1):59–82.

Medway RL, Fulton J. When more gets you less - a Meta-analysis of the effect of concurrent web options on Mail Survey Response Rates. Pub Opin Q. 2012;76(4):733–46.

Petrosyan A. Global internet access rate 2005–2022. https://www.statista.com/statistics/209096/share-of-internet-users-worldwide-by-market-maturity/#:text=As%20of%202022%2C%2066%20percent,access%20rate%20was%2066%20percent [Accessed 4 May 2024].

Vestberg H. How can we bring 2.6 billion people online to bridge the digital divide? World Economic Forum Annual Meeting; Jan 2024. https://www.weforum.org/agenda/2024/01/digital-divide-internet-access-online-fwa/ [Accessed 4 May 2024].

Scott P, Edwards P. Personally addressed hand-signed letters increase questionnaire response: a meta-analysis of randomised controlled trials. BMC Health Serv Res. 2006;6:111. https://doi.org/10.1186/1472-6963-6-111 .

Harbord RM, Egger M, Sterne JA. A modified test for small-study effects in meta-analyses of controlled trials with binary endpoints. Stat Med. 2006;25(20):3443-57. https://doi.org/10.1002/sim.2380 . PMID: 16345038.

Edwards P, Cooper R, Roberts I, Frost C. Meta-analysis of randomised trials of monetary incentives and response to mailed questionnaires. J Epidemiol Community Health. 2005;59(11):987–99. https://doi.org/10.1136/jech.2005.034397 .

Treweek S, Bevan S, Bower P, Campbell M, Christie J, Clarke M, et al. Trial Forge Guidance 1: what is a study within a trial (SWAT)? Trials. 2018;19(1):139. https://doi.org/10.1186/s13063-018-2535-5 .

Galea S, Tracy M. Participation rates in epidemiologic studies. Ann Epidemiol. 2007;17(9):643–53.

Dillman DA. Towards survey response rate theories that no longer pass each other like strangers in the night. In: Brenner PS, editor. Understanding survey methodology - sociological theory and applications. Switzerland: Springer Nature; 2020. pp. 15–44.

Chapter   Google Scholar  

Griffin DH, Fischer DP, Morgan MT. Testing an Internet Response Option for the American Community Survey. Washington: U.S. Bureau of the Census, 2001. https://www.census.gov/library/working-papers/2001/acs/2001_Griffin_01.html [Accessed 27 October 2023].

Sinclair M, O’Toole J, Malawaraarachchi M, et al. Comparison of response rates and cost-effectiveness for a community-based survey: postal, internet and telephone modes with generic or personalised recruitment approaches. BMC Med Res Methodol. 2012;12:132. https://doi.org/10.1186/1471-2288-12-132 .

Download references

Acknowledgements

Not applicable.

This work was supported by the NIHR Evidence Synthesis Programme [Grant NIHR133238]. Conduct of the work was entirely independent of the funder.

Author information

Authors and affiliations.

Data Collection Methodology Group, Department of Population Health, Faculty of Epidemiology and Population Health, London School of Hygiene & Tropical Medicine, Keppel Street, London, WC1E 7HT, UK

Phil Edwards & Chloe Perkins

You can also search for this author in PubMed   Google Scholar

Contributions

PE independently screened the search results and the full-text reports, and extracted data from eligible studies. He assisted with data entry into RevMan 5. He exported data into Stata and he conducted the meta-analyses and other statistical analyses. He drafted the manuscript. PE is guarantor for the paper.CP ran the electronic searches, independently screened the search results and the full-text reports, and extracted data from eligible studies. CP assisted with data entry into RevMan 5, and commented on the manuscript.

Corresponding author

Correspondence to Phil Edwards .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary material 2, supplementary material 3, supplementary material 4, supplementary material 5, supplementary material 6, supplementary material 7, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Edwards, P., Perkins, C. Response is increased using postal rather than electronic questionnaires – new results from an updated Cochrane Systematic Review. BMC Med Res Methodol 24 , 209 (2024). https://doi.org/10.1186/s12874-024-02332-0

Download citation

Received : 06 November 2023

Accepted : 06 September 2024

Published : 16 September 2024

DOI : https://doi.org/10.1186/s12874-024-02332-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Data collection
• Questionnaires and surveys
• Survey methodology
• Nonrespondents
• Self report

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Journal Proposal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Objectively measured physical activity and sedentary behaviour on cardiovascular risk and health-related quality of life in adults: a systematic review.

1. Introduction

2. materials and methods, 2.1. search strategies, 2.2. eligibility criteria and selection of studies, 2.3. data extraction, 2.4. quality and risk of bias assessment, 3.1. data search, 3.2. characteristics of studies.

3.3. Quality and Risk of Bias Assessment

4. discussion, 5. conclusions, author contributions, institutional review board statement, informed consent statement, data availability statement, acknowledgments, conflicts of interest.

• European Society of Cardiology. SCORE2 risk prediction algorithms: New models to estimate 10-year risk of cardiovascular disease in Europe. Eur. Heart J. 2021 , 42 , 2439–2454. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Visseren, F.L.J.; Mach, F.; Smulders, Y.M.; Carballo, D.; Koskinas, K.C.; Bäck, M.; Benetos, A.; Biffi, A.; Boavida, J.-M.; Capodanno, D.; et al. ESC Guidelines on cardiovascular disease prevention in clinical practice. Eur. Heart J. 2021 , 42 , 3227–3337. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Timmis, A.; Vardas, P.; Townsend, N.; Torbica, A.; Katus, H.; De Smedt, D.; Gale, C.P.; Maggioni, A.P.; E. Petersen, S.; Huculeci, R.; et al. European Society of Cardiology: Cardiovascular disease statistics 2021: Executive Summary. Eur. Heart J.-Qual. Care Clin. Outcomes 2022 , 8 , 377–382. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Alwan, H.; William, J.; Viswanathan, B.; Paccaud, F.; Bovet, P. Perception of cardiovascular risk and comparison with actual cardiovascular risk. Eur. J. Prev. Cardiol. 2009 , 16 , 556–561. [ Google Scholar ] [ CrossRef ]
• Edwardson, C.L.; Gorely, T.; Davies, M.J.; Gray, L.J.; Khunti, K.; Wilmot, E.G.; Yates, T.; Biddle, S.J.H. Association of Sedentary Behaviour with Metabolic Syndrome: A Meta-Analysis. PLoS ONE 2012 , 7 , e34916. [ Google Scholar ] [ CrossRef ]
• Grundy, S.M. Metabolic syndrome update. Trends Cardiovasc. Med. 2015 , 26 , 364–373. [ Google Scholar ] [ CrossRef ]
• D’Agostino, R.B.; Vasan, R.S.; Pencina, M.J.; Wolf, P.A.; Cobain, M.; Massaro, J.M.; Kannel, W.B. General Cardiovascular Risk Profile for Use in Primary Care: The Framingham Heart Study. Circulation 2008 , 117 , 743–753. [ Google Scholar ] [ CrossRef ]
• Assmann, G.; Cullen, P.; Schulte, H. Simple Scoring Scheme for Calculating the Risk of Acute Coronary Events Based on the 10-Year Follow-Up of the Prospective Cardiovascular Münster (PROCAM) Study. Circulation 2002 , 105 , 310–315. [ Google Scholar ] [ CrossRef ]
• Cella, D.F. Quality of life: Concepts and definition. J. Pain Symptom Manag. 1994 , 9 , 186–192. [ Google Scholar ] [ CrossRef ]
• Karimi, M.; Brazier, J. Health, Health-Related Quality of Life, and Quality of Life: What is the Difference? PharmacoEconomics 2016 , 34 , 645–649. [ Google Scholar ] [ CrossRef ]
• Patrick, D.L.; Erickson, P. Health Status and Health Policy: Allocating Resources to Health Care ; Oxford University Press: New York, NY, USA, 1993. [ Google Scholar ]
• Ware, J.E., Jr.; Sherbourne, C.D. The MOS 36-item short-form health survey (SF-36). I. Conceptual framework and item selection. Med. Care 1992 , 30 , 473–483. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Whoqol Group Development of the WHOQOL: Rationale and Current Status. Int. J. Ment. Health 1994 , 23 , 24–56. [ CrossRef ]
• Martinelli, L.M.B.; Mizutani, B.M.; Mutti, A.; Dèlia, M.P.B.; Coltro, R.S.; Matsubara, B.B. Quality of Life and its Association with Cardiovascular Risk Factors in a Community Health Care Program Population. Clinics 2008 , 63 , 783–788. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Mitu, O.; Roca, M.; Leon, M.M.; Gherasim, A.; Graur, M.; Mitu, F. Association of Health-Related Quality of Life with Cardio-vascular Risk Factors and Subclinical Atherosclerosis in Non-Diabetic Asymptomatic Adults. Biomed. Res. 2016 , 27 , 687–694. [ Google Scholar ]
• Salinas-Rodríguez, A.; Manrique-Espinoza, B.; Palazuelos-González, R.; Rivera-Almaraz, A.; Jáuregui, A. Physical activity and sedentary behavior trajectories and their associations with quality of life, disability, and all-cause mortality. Eur. Rev. Aging Phys. Act. 2022 , 19 , 13. [ Google Scholar ] [ CrossRef ]
• Pool, L.R.; Ning, H.; Huffman, M.D.; Reis, J.P.; Lloyd-Jones, D.M.; Allen, N.B. Association of cardiovascular health through early adulthood and health-related quality of life in middle age: The Coronary Artery Risk Development in Young Adults (CARDIA) Study. Prev. Med. 2019 , 126 , 105772. [ Google Scholar ] [ CrossRef ]
• Pinheiro, L.C.; Reshetnyak, E.; Sterling, M.R.; Richman, J.S.; Kern, L.M.; Safford, M.M. Using health-related quality of life to predict cardiovascular disease events. Qual. Life Res. 2019 , 28 , 1465–1475. [ Google Scholar ] [ CrossRef ]
• Bull, F.C.; Al-Ansari, S.S.; Biddle, S.; Borodulin, K.; Buman, M.P.; Cardon, G.; Willumsen, J.F. WHO Guidelines on Physical Activity and Sedentary Behaviour. Br. J. Sport. Med. 2020 , 54 , 1451–1462. [ Google Scholar ] [ CrossRef ]
• World Health Organization. Who Guidelines on Physical Activity and Sedentary Behaviour ; World Health Organization: Geneva, Switzerland, 2020.
• Bames, J.; Behrens, T.K.; Benden, M.E.; Biddle, S.; Bond, D.; Brassard, P.; Brown, H.; Carr, L.; Carson, V.; Chaput, J. Standardized Use of the Terms “Sedentary” and “Sedentary Behaviours”. Appl. Physiol. Nutr. Metab. Appl. Nutr. Metab. 2012 , 37 , 540–542. [ Google Scholar ]
• Tremblay, M.S.; Aubert, S.; Barnes, J.D.; Saunders, T.J.; Carson, V.; Latimer-Cheung, A.E.; Chastin, S.F.M.; Altenburg, T.M.; Chinapaw, M.J.M.; Aminian, S.; et al. Sedentary Behavior Research Network (SBRN)—Terminology Consensus Project process and outcome. Int. J. Behav. Nutr. Phys. Act. 2017 , 14 , 75. [ Google Scholar ] [ CrossRef ]
• Ballin, M.; Nordström, P.; Niklasson, J.; Nordström, A. Associations of Objectively Measured Physical Activity and Sedentary Time with the Risk of Stroke, Myocardial Infarction or All-Cause Mortality in 70-Year-Old Men and Women: A Prospective Cohort Study. Sports Med. 2020 , 51 , 339–349. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Richardson, C.R.; Kriska, A.M.; Lantz, P.M.; Hayward, R.A. Physical Activity and Mortality across Cardiovascular Disease Risk Groups. Med. Sci. Sports Exerc. 2004 , 36 , 1923–1929. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Rynders, C.A.; Blanc, S.; DeJong, N.; Bessesen, D.H.; Bergouignan, A. Sedentary behaviour is a key determinant of metabolic inflexibility. J. Physiol. 2017 , 596 , 1319–1330. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Pedersen, B.K. Anti-inflammatory effects of exercise: Role in diabetes and cardiovascular disease. Eur. J. Clin. Investig. 2017 , 47 , 600–611. [ Google Scholar ] [ CrossRef ]
• Booth, F.W.; Roberts, C.K.; Thyfault, J.P.; Ruegsegger, G.N.; Toedebusch, R.G. Role of Inactivity in Chronic Diseases: Evolutionary Insight and Pathophysiological Mechanisms. Physiol. Rev. 2017 , 97 , 1351–1402. [ Google Scholar ] [ CrossRef ]
• Wildman, R.P.; Muntner, P.; Reynolds, K.; McGinn, A.P.; Rajpathak, S.; Wylie-Rosett, J.; Sowers, M.R. The Obese Without Cardiometabolic Risk Factor Clustering and the Normal Weight With Cardiometabolic Risk Factor Clustering Prevalence and Correlates of 2 Phenotypes Among the US Population (NHANES 1999–2004). Obstet. Gynecol. Surv. 2008 , 63 , 783–784. [ Google Scholar ] [ CrossRef ]
• Vallance, J.K.; Eurich, D.T.; Lavallee, C.M.; Johnson, S.T. Physical activity and health-related quality of life among older men: An examination of current physical activity recommendations. Prev. Med. 2012 , 54 , 234–236. [ Google Scholar ] [ CrossRef ]
• Pucci, G.C.M.F.; Rech, C.R.; Fermino, R.C.; Reis, R.S. Association between Physical Activity and Quality of Life in Adults. Rev. Saude Publica 2012 , 46 , 166–179. [ Google Scholar ] [ CrossRef ]
• Sanchez-Aguadero, N.; MARK Group; Alonso-Dominguez, R.; Garcia-Ortiz, L.; Agudo-Conde, C.; Rodriguez-Martin, C.; de Cabo-Laso, A.; Sanchez-Salgado, B.; Ramos, R.; Maderuelo-Fernandez, J.A.; et al. Diet and physical activity in people with intermediate cardiovascular risk and their relationship with the health-related quality of life: Results from the MARK study. Health Qual. Life Outcomes 2016 , 14 , 1–10. [ Google Scholar ] [ CrossRef ]
• Hamer, M.; Ingle, L.; Carroll, S.; Stamatakis, E. Physical Activity and Cardiovascular Mortality Risk. Med. Sci. Sports Exerc. 2012 , 44 , 84–88. [ Google Scholar ] [ CrossRef ]
• Hamer, M.; Stamatakis, E. Physical Activity and Risk of Cardiovascular Disease Events: Inflammatory and Metabolic Mechanisms. Med. Sci. Sports Exerc. 2009 , 41 , 1206–1211. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Oguma, Y.; Shinoda-Tagawa, T. Physical Activity Decreases Cardiovascular Disease Risk in Women: Review and Meta-Analysis. Am. J. Prev. Med. 2004 , 26 , 407–418. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Page, M.J.; Moher, D.; Bossuyt, P.M.; Boutron, I.; Hoffmann, T.C.; Mulrow, C.D.; Shamseer, L.; Tetzlaff, J.M.; Akl, E.A.; Brennan, S.E.; et al. PRISMA 2020 explanation and elaboration: Updated guidance and exemplars for reporting systematic reviews. BMJ 2021 , 372 , n160. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Methley, A.M.; Campbell, S.; Chew-Graham, C.; McNally, R.; Cheraghi-Sohi, S. PICO, PICOS and SPIDER: A comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv. Res. 2014 , 14 , 579. [ Google Scholar ] [ CrossRef ]
• National Heart, Lung, and Blood Institute. NIH Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies ; National Heart, Lung, and Blood Institute: Bethesda, MD, USA, 2019; pp. 1–4. [ Google Scholar ]
• Musa, S.; Elyamani, R.; Dergaa, I. COVID-19 and screen-based sedentary behaviour: Systematic review of digital screen time and metabolic syndrome in adolescents. PLoS ONE 2022 , 17 , e0265560. [ Google Scholar ] [ CrossRef ]
• Marín-Jiménez, N.B.; Ruiz-Montero, P.J.; De la Flor-Alemany, M.; Aranda, P.; Aparicio, V.A. Association of objectively measured sedentary behavior and physical activity levels with health-related quality of life in middle-aged women: The FLAMENCO project. Menopause 2020 , 27 , 437–443. [ Google Scholar ] [ CrossRef ]
• Tigbe, W.W.; Granat, M.H.; Sattar, N.; Lean, M.E.J. Time spent in sedentary posture is associated with waist circumference and cardiovascular risk. Int. J. Obes. 2017 , 41 , 689–696. [ Google Scholar ] [ CrossRef ]
• Niemelä, M.; Kangas, M.; Farrahi, V.; Kiviniemi, A.; Leinonen, A.-M.; Ahola, R.; Puukka, K.; Auvinen, J.; Korpelainen, R.; Jämsä, T. Intensity and temporal patterns of physical activity and cardiovascular disease risk in midlife. Prev. Med. 2019 , 124 , 33–41. [ Google Scholar ] [ CrossRef ]
• Frisk, M.K.; Hedner, J.; Grote, L.; Ekblom, Ö.; Arvidsson, D.; Bergström, G.; Börjesson, M.; Zou, D. Eveningness is associated with sedentary behavior and increased 10-year risk of cardiovascular disease: The SCAPIS pilot cohort. Sci. Rep. 2022 , 12 , 8203. [ Google Scholar ] [ CrossRef ]
• Kolt, G.S.; George, E.S.; Rebar, A.L.; Duncan, M.J.; Vandelanotte, C.; Caperchione, C.M.; Maeder, A.J.; Tague, R.; Savage, T.N.; Van Itallie, A.; et al. Associations between quality of life and duration and frequency of physical activity and sedentary behaviour: Baseline findings from the WALK 2.0 randomised controlled trial. PLoS ONE 2017 , 12 , e0180072. [ Google Scholar ] [ CrossRef ]
• Ainsworth, B.E.; Haskell, W.L.; Whitt, M.C.; Irwin, M.L.; Swartz, A.M.; Strath, S.J.; O’Brien, W.L.; Bassett, D.R., Jr.; Schmitz, K.H.; Emplaincourt, P.O.; et al. Compendium of Physical Activities: An update of activity codes and MET intensities. Med. Sci. Sports Exerc. 2000 , 32 , S498–S504. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Ware, J.E. SF-36 Health Survey. In Manual and Interpretation Guide ; The Health Institute: Boston, MA, USA, 1993; pp. 1–6. [ Google Scholar ]
• Lins-Kusterer, L.; Aguiar, I.; Santos-Lins, L.S.; Lins-Rocha, M.; Aguiar, C.V.N.; Menezes, M.S.; Sampaio, A.S.; Quarantini, L.C.; Cotrim, H.P.; Brites, C. Validation of the RAND 36-Item Health Survey questionnaire in Brazil. Arq. Gastroenterol. 2022 , 59 , 193–197. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Vásquez, P.M.; Durazo-Arvizu, R.A.; Marquez, D.X.; Argos, M.; Lamar, M.; Odoms-Young, A.; Wu, D.; González, H.M.; Tarraf, W.; Sotres-Alvarez, D.; et al. Moderate-vigorous physical activity and health-related quality of life among Hispanic/Latino adults in the Hispanic Community Health Study/Study of Latinos (HCHS/SOL). J. Patient-Rep. Outcomes 2019 , 3 , 45. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Loprinzi, P.D.; Davis, R.E. Bouted and non-bouted moderate-to-vigorous physical activity with health-related quality of life. Prev. Med. Rep. 2015 , 3 , 46–48. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Guallar-Castillón, P.; Bayán-Bravo, A.; León-Muñoz, L.M.; Balboa-Castillo, T.; López-García, E.; Gutierrez-Fisac, J.L.; Rodríguez-Artalejo, F. The association of major patterns of physical activity, sedentary behavior and sleep with health-related quality of life: A cohort study. Prev. Med. 2014 , 67 , 248–254. [ Google Scholar ] [ CrossRef ]
• Ekelund, U.; Tarp, J.; Steene-Johannessen, J.; Hansen, B.H.; Jefferis, B.; Fagerland, M.W.; Whincup, P.; Diaz, K.M.; Hooker, S.P.; Chernofsky, A.; et al. Dose-response associations between accelerometry measured physical activity and sedentary time and all cause mortality: Systematic review and harmonised meta-analysis. BMJ 2019 , 366 , l4570. [ Google Scholar ] [ CrossRef ]
• Schuch, F.B.; Vancampfort, D.; Firth, J.; Rosenbaum, S.; Ward, P.B.; Silva, E.S.; Hallgren, M.; De Leon, A.P.; Dunn, A.L.; Deslandes, A.C.; et al. Physical Activity and Incident Depression: A Meta-Analysis of Prospective Cohort Studies. Am. J. Psychiatry 2018 , 175 , 631–648. [ Google Scholar ] [ CrossRef ]
• Yang, P.-Y.; Ho, K.-H.; Chen, H.-C.; Chien, M.-Y. Exercise training improves sleep quality in middle-aged and older adults with sleep problems: A systematic review. J. Physiother. 2012 , 58 , 157–163. [ Google Scholar ] [ CrossRef ]
• Hawkins, M.S.; Gabriel, K.P.; Conroy, M.B.; Cooper, J.; Sutton-Tyrrell, K. Physical activity intensity and cardiovascular risk by ankle–brachial index. Vasc. Med. 2013 , 18 , 79–84. [ Google Scholar ] [ CrossRef ]
• Biswas, A.; Chen, C.; Prince, S.A.; Smith, P.M.; Mustard, C.A. Workers’ Activity Profiles Associated with Predicted 10-Year Cardiovascular Disease Risk. J. Am. Heart Assoc. 2022 , 11 , e025148. [ Google Scholar ] [ CrossRef ]
• Kraus, W.E.; Janz, K.F.; Powell, K.E.; Campbell, W.W.; Jakicic, J.M.; Troiano, R.P.; Sprow, K.; Torres, A.; Piercy, K.L. Daily Step Counts for Measuring Physical Activity Exposure and Its Relation to Health. Med. Sci. Sports Exerc. 2019 , 51 , 1206–1212. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Hall, K.S.; Hyde, E.T.; Bassett, D.R.; Carlson, S.A.; Carnethon, M.R.; Ekelund, U.; Evenson, K.R.; Galuska, D.A.; Kraus, W.E.; Lee, I.-M.; et al. Systematic review of the prospective association of daily step counts with risk of mortality, cardiovascular disease, and dysglycemia. Int. J. Behav. Nutr. Phys. Act. 2020 , 17 , 78. [ Google Scholar ] [ CrossRef ]
• Thomas, D.; Elliott, E.J.; A. Naughton, G. Exercise for type 2 diabetes mellitus. Cochrane Database Syst. Rev. 2006 , 2017 , CD002968. [ Google Scholar ] [ CrossRef ]
• Shaw, K.A.; Gennat, H.C.; O’Rourke, P.; Del Mar, C. Exercise for overweight or obesity. Cochrane Database Syst. Rev. 2006 , 2010 , CD003817. [ Google Scholar ] [ CrossRef ] [ PubMed ]

Click here to enlarge figure

Share and Cite

Santos, B.; Monteiro, D.; Silva, F.M.; Flores, G.; Bento, T.; Duarte-Mendes, P. Objectively Measured Physical Activity and Sedentary Behaviour on Cardiovascular Risk and Health-Related Quality of Life in Adults: A Systematic Review. Healthcare 2024 , 12 , 1866. https://doi.org/10.3390/healthcare12181866

Santos B, Monteiro D, Silva FM, Flores G, Bento T, Duarte-Mendes P. Objectively Measured Physical Activity and Sedentary Behaviour on Cardiovascular Risk and Health-Related Quality of Life in Adults: A Systematic Review. Healthcare . 2024; 12(18):1866. https://doi.org/10.3390/healthcare12181866

Santos, Beatriz, Diogo Monteiro, Fernanda M. Silva, Gonçalo Flores, Teresa Bento, and Pedro Duarte-Mendes. 2024. "Objectively Measured Physical Activity and Sedentary Behaviour on Cardiovascular Risk and Health-Related Quality of Life in Adults: A Systematic Review" Healthcare 12, no. 18: 1866. https://doi.org/10.3390/healthcare12181866

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

Advertisement

A systematic review of vision transformers and convolutional neural networks for Alzheimer’s disease classification using 3D MRI images

• Published: 17 September 2024

Cite this article

• Mario Alejandro Bravo-Ortiz 1 , 2 , 7   na1 ,
• Sergio Alejandro Holguin-Garcia 1 , 2 , 7   na1 ,
• Sebastián Quiñones-Arredondo 1 ,
• Alejandro Mora-Rubio 6 ,
• Ernesto Guevara-Navarro 1 , 2 ,
• Harold Brayan Arteaga-Arteaga 1 ,
• Gonzalo A. Ruz 3 , 4 , 5 &
• Reinel Tabares-Soto 1 , 2 , 3  

Alzheimer’s disease (AD) is a progressive neurodegenerative disorder that mainly affects memory and other cognitive functions, such as thinking, reasoning, and the ability to carry out daily activities. It is considered the most common form of dementia in older adults, but it can appear as early as the age of 25. Although the disease has no cure, treatment can be more effective if diagnosed early. In diagnosing AD, changes in the brain’s morphology are identified macroscopically, which is why deep learning models, such as convolutional neural networks (CNN) or vision transformers (ViT), excel in this task. We followed the Systematic Literature Review process, applying stages of the review protocol from it, which aims to detect the need for a review. Then, search equations were formulated and executed in several literature databases. Relevant publications were scanned and used to extract evidence to answer research questions. Several CNN and ViT approaches have already been tested on problems related to brain image analysis for disease detection. A total of 722 articles were found in the selected databases. Still, a series of filters were performed to decrease the number to 44 articles, focusing specifically on brain image analysis with CNN and ViT methods. Deep learning methods are effective for disease diagnosis, and the surge in research activity underscores its importance. However, the lack of access to repositories may introduce bias into the information. Full access demonstrates transparency and facilitates collaborative work in research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Explore related subjects

• Artificial Intelligence
• Medical Imaging

Data availability

Not applicable.

Ballard C, Gauthier S, Corbett A, Brayne C, Aarsland D, Jones E (2011) Alzheimer’s disease. Lancet 377(9770):1019–1031. https://doi.org/10.1016/S0140-6736(10)61349-9

Google Scholar  

DeTure MA, Dickson DW (2019) The neuropathological diagnosis of alzheimer’s disease. Mol Neurodegenerat 14(1):32. https://doi.org/10.1186/s13024-019-0333-5

Moradi E, Pepe A, Gaser C, Huttunen H, Tohka J, Initiative ADN et al (2015) Machine learning framework for early mri-based alzheimer’s conversion prediction in mci subjects. Neuroimage 104:398–412

Article   Google Scholar  

Hassan SA, Khan T (2017) A machine learning model to predict the onset of alzheimer disease using potential cerebrospinal fluid (csf) biomarkers. Int J Adv Comput Sci Appl 8(12)

Salvatore C, Cerasa A, Battista P, Gilardi MC, Quattrone A, Castiglioni I, Initiative ADN (2015) Magnetic resonance imaging biomarkers for the early diagnosis of alzheimer’s disease: a machine learning approach. Front Neurosci 9:307

Long X, Chen L, Jiang C, Zhang L, Initiative ADN (2017) Prediction and classification of alzheimer disease based on quantification of mri deformation. PloS One 12(3):0173372

LaMontagne PJ, Benzinger TL, Morris JC, Keefe S, Hornbeck R, Xiong C, Grant E, Hassenstab J, Moulder K, Vlassenko AG, et al (2019) Oasis-3: longitudinal neuroimaging, clinical, and cognitive dataset for normal aging and alzheimer disease. MedRxiv

Petersen RC, Aisen P, Beckett LA, Donohue M, Gamst A, Harvey DJ, Jack C, Jagust W, Shaw L, Toga A et al (2010) Alzheimer’s disease neuroimaging initiative (adni): clinical characterization. Neurology 74(3):201–209

Mora-Rubio A, Bravo-Ortíz MA, Arredondo SQ, Torres JMS, Ruz GA, Tabares-Soto R (2023) Classification of alzheimer’s disease stages from magnetic resonance images using deep learning. PeerJ Comput Sci 9:1490

Sharma S, Sharma V, Sharma A (2016) Performance based evaluation of various machine learning classification techniques for chronic kidney disease diagnosis. arXiv preprint arXiv:1606.09581

Chan H-P, Samala RK, Hadjiiski LM, Zhou C (2020) Deep learning in medical image analysis. Deep Learn Med Image Anal, pp 3–21

Bravo-Ortíz MA, Arteaga-Arteaga HB, Tabares-Soto KR, Padilla-Buriticá JI, Orozco-Arias S (2021) Cervical cancer classification using convolutional neural networks, transfer learning and data augmentation. Revista EIA 18(35):100–111

Liu S, Liu S, Cai W, Pujol S, Kikinis R, Feng D (2014) Early diagnosis of alzheimer’s disease with deep learning. In: 2014 IEEE 11th international symposium on biomedical imaging (ISBI), pp 1015–1018. IEEE

Chitradevi D, Prabha S (2020) Analysis of brain sub regions using optimization techniques and deep learning method in alzheimer disease. Appl Soft Comput 86:105857

Currie G, Rohren E (2021) Intelligent imaging in nuclear medicine: the principles of artificial intelligence, machine learning and deep learning. In: Seminars in nuclear medicine, vol 51, pp 102–111. Elsevier

Porumb M, Stranges S, Pescapè A, Pecchia L (2020) Precision medicine and artificial intelligence: a pilot study on deep learning for hypoglycemic events detection based on ecg. Sci Rep 10(1):1–16

Karami V, Nittari G, Amenta F (2019) Neuroimaging computer-aided diagnosis systems for alzheimer’s disease. Int J Imag Syst Technol 29(1):83–94

Mohapatra S, Swarnkar T, Das J (2021) Deep convolutional neural network in medical image processing. In: Handbook of deep learning in biomedical engineering, pp 25–60. Elsevier

Shanmugavadivel K, Sathishkumar V, Cho J, Subramanian M (2023) Advancements in computer-assisted diagnosis of alzheimer’s disease: a comprehensive survey of neuroimaging methods and ai techniques for early detection. Ageing Res Rev, 102072

Weimer DL, Sager MA (2009) Early identification and treatment of alzheimer’s disease: social and fiscal outcomes. Alzheimer’s Dementia 5(3):215–226

Sujith A, Sajja GS, Mahalakshmi V, Nuhmani S, Prasanalakshmi B (2022) Systematic review of smart health monitoring using deep learning and artificial intelligence. Neurosci Informat 2(3):100028

Maqsood M, Nazir F, Khan U, Aadil F, Jamal H, Mehmood I, Song O-y (2019) Transfer learning assisted classification and detection of alzheimer’s disease stages using 3d mri scans. Sensors 19(11), 2645

Basaia S, Agosta F, Wagner L, Canu E, Magnani G, Santangelo R, Filippi M (2019) Automated classification of alzheimer’s disease and mild cognitive impairment using a single mri and deep neural networks. NeuroImage: Clin 21:101645. https://doi.org/10.1016/j.nicl.2018.101645

Hawkins DM (2004) The problem of overfitting. J Chem Inf Comput Sci 44(1):1–12. https://doi.org/10.1021/ci0342472 . ( PMID: 14741005 )

Liu M, Li F, Yan H, Wang K, Ma Y, Shen L, Xu M (2020) A multi-model deep convolutional neural network for automatic hippocampus segmentation and classification in alzheimer’s disease. NeuroImage 208:116459. https://doi.org/10.1016/j.neuroimage.2019.116459

Hochreiter S (1998) The vanishing gradient problem during learning recurrent neural nets and problem solutions. Int J Uncertainty Fuzziness Knowl Based Syst 6(02):107–116

Li W, Lin X, Chen X (2020) Detecting alzheimer’s disease based on 4d fmri: an exploration under deep learning framework. Neurocomputing 388:280–287. https://doi.org/10.1016/j.neucom.2020.01.053

Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

Zhang J, Zheng B, Gao A, Feng X, Liang D, Long X (2021) A 3d densely connected convolution neural network with connection-wise attention mechanism for alzheimer’s disease classification. Magnetic Resonance Imag 78:119–126. https://doi.org/10.1016/j.mri.2021.02.001

Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. In: Advances in neural information processing systems, pp 5998–6008

Janghel RR, Rathore YK (2021) Deep convolution neural network based system for early diagnosis of alzheimer’s disease. IRBM 42(4):258–267. https://doi.org/10.1016/j.irbm.2020.06.006

Bi X, Li S, Xiao B, Li Y, Wang G, Ma X (2020) Computer aided alzheimer’s disease diagnosis by an unsupervised deep learning technology. Neurocomputing 392:296–304. https://doi.org/10.1016/j.neucom.2018.11.111

Hosseini-Asl E, Keynton R, El-Baz A (2016) Alzheimer’s disease diagnostics by adaptation of 3d convolutional network. In: 2016 IEEE international conference on image processing (ICIP), pp 126–130. https://doi.org/10.1109/ICIP.2016.7532332

Raghavaiah P, Varadarajan S (2021) A cad system design to diagnosize alzheimers disease from mri brain images using optimal deep neural network. Multimedia Tools Appl 80(17):26411–26428

Kang W, Lin L, Zhang B, Shen X, Wu S (2021) Multi-model and multi-slice ensemble learning architecture based on 2d convolutional neural networks for alzheimer’s disease diagnosis. Comput Biol Med 136:104678. https://doi.org/10.1016/j.compbiomed.2021.104678

Radford A, Metz L, Chintala S (2015) Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434

Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, et al (2020) An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929

Khan S, Naseer M, Hayat M, Zamir SW, Khan FS, Shah M (2022) Transformers in vision: a survey. ACM Comput Surv (CSUR) 54(10s):1–41

Jang J, Hwang D (2022) M3t: three-dimensional medical image classifier using multi-plane and multi-slice transformer. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 20718–20729

Cho K, Kim J, Kim KD, Park S, Kim J, Yun J, Ahn Y, Oh SY, Lee SM, Seo JB et al (2023) Music-vit: A multi-task siamese convolutional vision transformer for differentiating change from no-change in follow-up chest radiographs. Med Image Anal 89:102894

Li J, Liu Z, Li L, Lin J, Yao J, Tu J (2023) Multi-view convolutional vision transformer for 3d object recognition. J Vis Commun Image Represent 95:103906

Bravo-Ortiz MA, Mercado-Ruiz E, Villa-Pulgarin JP, Hormaza-Cardona CA, Quiñones-Arredondo S, Arteaga-Arteaga HB, Orozco-Arias S, Cardona-Morales O, Tabares-Soto R (2024) Cvtstego-net: A convolutional vision transformer architecture for spatial image steganalysis. J Inf Security Appl 81:103695

Holguin-Garcia SA, Guevara-Navarro E, Daza-Chica AE, Patiño-Claro MA, Arteaga-Arteaga HB, Ruz GA, Tabares-Soto R, Bravo-Ortiz MA (2024) A comparative study of cnn-capsule-net, cnn-transformer encoder, and traditional machine learning algorithms to classify epileptic seizure. BMC Med Informat Decis Making 24(1):60

Zhu J, Tan Y, Lin R, Miao J, Fan X, Zhu Y, Liang P, Gong J, He H (2022) Efficient self-attention mechanism and structural distilling model for alzheimer’s disease diagnosis. Comput Biol Med 147:105737

Yin Y, Jin W, Bai J, Liu R, Zhen H (2022) Smil-deit: Multiple instance learning and self-supervised vision transformer network for early alzheimer’s disease classification. In: 2022 international joint conference on neural networks (IJCNN), pp 1–6. IEEE

Li C, Cui Y, Luo N, Liu Y, Bourgeat P, Fripp J, Jiang T (2022) Trans-resnet: Integrating transformers and cnns for alzheimer’s disease classification. In: 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), pp. 1–5. IEEE

Kitchenham B, Charters S, et al (2007) Guidelines for performing systematic literature reviews in software engineering. UK

Reinel T-S, Raul R-P, Gustavo I (2019) Deep learning applied to steganalysis of digital images: a systematic review. IEEE Access 7:68970–68990

Kitchenham B, Brereton OP, Budgen D, Turner M, Bailey J, Linkman S (2009) Systematic literature reviews in software engineering-a systematic literature review. Inf Softw Technol 51(1):7–15

Anwer F, Aftab S (2017) Latest customizations of xp: a systematic literature review. Int J Mod Educ Comput Sci 9(12):26

Ashraf S, Aftab S (2017) Scrum with the spices of agile family: a systematic mapping. Int J Mod Educ Comput Sci 9(11):58–72

Ahmad M, Aftab S, Bashir MS, Hameed N (2018) Sentiment analysis using svm: a systematic literature review. Int J Adv Comput Sci Appl 9(2)

Matloob F, Ghazal TM, Taleb N, Aftab S, Ahmad M, Khan MA, Abbas S, Soomro TR (2021) Software defect prediction using ensemble learning: a systematic literature review. IEEE Access 9:98754–98771

Wen J, Li S, Lin Z, Hu Y, Huang C (2012) Systematic literature review of machine learning based software development effort estimation models. Inf Softw Technol 54(1):41–59

Yu X, Peng B, Shi J, Zhu J, Dai Y (2019) 3d convolutional networks based automatic diagnosis of alzheimer’s disease using structural mri. In: 2019 12th international congress on image and signal processing, BioMedical Engineering and Informatics (CISP-BMEI), pp 1–6. IEEE

Jin D, Xu J, Zhao K, Hu F, Yang Z, Liu B, Jiang T, Liu Y (2019) Attention-based 3d convolutional network for alzheimer’s disease diagnosis and biomarkers exploration. In: 2019 IEEE 16th international symposium on biomedical imaging (ISBI 2019), pp 1047–1051. IEEE

Ge C, Qu Q, Gu IY-H, Jakola AS (2019) Multiscale deep convolutional networks for characterization and detection of alzheimer’s disease using mr images. In: 2019 IEEE international conference on image processing (ICIP), pp 789–793. IEEE

Goenka N, Tiwari S (2021) Volumetric convolutional neural network for alzheimer detection. In: 2021 5th International conference on trends in electronics and informatics (ICOEI), pp 1500–1505. IEEE

Dua M, Makhija D, Manasa P, Mishra P (2020) A cnn-rnn-lstm based amalgamation for alzheimer’s disease detection. J Med Biol Eng 40(5):688–706

Raju M, Gopi VP, Anitha V, Wahid KA (2020) Multi-class diagnosis of alzheimer’s disease using cascaded three dimensional-convolutional neural network. Phys Eng Sci Med 43(4):1219–1228

Parmar H, Nutter B, Long R, Antani S, Mitra S (2020) Spatiotemporal feature extraction and classification of alzheimer’s disease using deep learning 3d-cnn for fmri data. J Med Imag 7(5):056001

Niu J, Tang X (2020) 3d residual dense convolutional network for diagnosis of alzheimer’s disease and mild cognitive impairment. In: 2020 IEEE international conference on mechatronics and automation (ICMA), pp 1581–1586. IEEE

Li A, Li F, Elahifasaee F, Liu M, Zhang L (2021) Hippocampal shape and asymmetry analysis by cascaded convolutional neural networks for alzheimer’s disease diagnosis. Brain Imag Behav 15(5):2330–2339

Basher A, Kim BC, Lee KH, Jung HY (2021) Volumetric feature-based alzheimer’s disease diagnosis from smri data using a convolutional neural network and a deep neural network. IEEE Access 9:29870–29882

Solano-Rojas B, Villalón-Fonseca R (2021) A low-cost three-dimensional densenet neural network for alzheimer’s disease early discovery. Sensors 21(4):1302

Zhang J, Zheng B, Gao A, Feng X, Liang D, Long X (2021) A 3d densely connected convolution neural network with connection-wise attention mechanism for alzheimer’s disease classification. Magnetic Resonance Imag 78:119–126

Katabathula S, Wang Q, Xu R (2021) Predict alzheimer’s disease using hippocampus mri data: a lightweight 3d deep convolutional network model with visual and global shape representations. Alzheimer’s Res Therapy 13(1):1–9

Pei Z, Gou Y, Ma M, Guo M, Leng C, Chen Y, Li J (2021) Alzheimer’s disease diagnosis based on long-range dependency mechanism using convolutional neural network. Multimedia Tools Appl, pp 1–16

Ebrahimi A, Luo S, Chiong R, Initiative ADN et al (2021) Deep sequence modelling for alzheimer’s disease detection using mri. Comput Biol Med 134:104537

Lin C-J, Lin C-W (2021) Using three-dimensional convolutional neural networks for alzheimer’s disease diagnosis. Sens Mater 33(10):3399–3413

Zhang P, Lin S, Qiao J, Tu Y (2021) Diagnosis of alzheimer’s disease with ensemble learning classifier and 3d convolutional neural network. Sensors 21(22):7634

Dwivedi S, Goel T, Sharma R, Murugan R (2021) Structural mri based alzheimer’s disease prognosis using 3d convolutional neural network and support vector machine. In: 2021 Advanced communication technologies and signal processing (ACTS), pp 1–4. IEEE

Bakkouri I, Afdel K, Benois-Pineau J et al (2022) Bg-3dm2f: Bidirectional gated 3d multi-scale feature fusion for alzheimer’s disease diagnosis. Multimedia Tools Appl 81(8):10743–10776

Kadri R, Bouaziz B, Tmar M, Gargouri F (2022) Multimodal deep learning based on the combination of efficientnetv2 and vit for alzheimer’s disease early diagnosis enhanced by sagan data augmentation. Int J Comput Inf Syst Ind Manag Appl 14:313–325

Sarraf S, Sarraf A, DeSouza DD, Anderson JA, Kabia M, Initiative ADN (2023) Ovitad: Optimized vision transformer to predict various stages of alzheimer’s disease using resting-state fmri and structural mri data. Brain Sci 13(2):260

Carcagnì P, Leo M, Del Coco M, Distante C, De Salve A (2023) Convolution neural networks and self-attention learners for alzheimer dementia diagnosis from brain mri. Sensors 23(3):1694

Odusami M, Maskeliūnas R, Damaševičius R (2023) Pixel-level fusion approach with vision transformer for early detection of alzheimer’s disease. Electronics 12(5):1218

Hoang GM, Kim U-H, Kim JG (2023) Vision transformers for the prediction of mild cognitive impairment to alzheimer’s disease progression using mid-sagittal smri. Front Aging Neurosci 15:1102869

Tang C, Wei M, Sun J, Wang S, Zhang Y, Initiative ADN, et al (2023) Csagp: Detecting alzheimer’s disease from multimodal images via dual-transformer with cross-attention and graph pooling. J King Saud Univ Comput Inf Sci, 101618

Zhao Q, Huang G, Xu P, Chen Z, Li W, Yuan X, Zhong G, Pun C-M, Huang Z (2023) Ida-net: Inheritable deformable attention network of structural mri for alzheimer’s disease diagnosis. Biomed Signal Process Control 84:104787

Lyu Y, Yu X, Zhu D, Zhang L (2022) Classification of alzheimer’s disease via vision transformer: Classification of alzheimer’s disease via vision transformer. In: Proceedings of the 15th international conference on pervasive technologies related to assistive environments, pp 463–468

Duan Y, Wang R, Li Y (2023) Aux-vit: Classification of alzheimer’s disease from mri based on vision transformer with auxiliary branch. In: 2023 5th International conference on communications, information system and computer engineering (CISCE), pp 382–386. IEEE

Liu L, Liu S, Zhang L, To XV, Nasrallah F, Chandra SS (2023) Cascaded multi-modal mixing transformers for alzheimer’s disease classification with incomplete data. NeuroImage 277:120267

Zhang J, He X, Liu Y, Cai Q, Chen H, Qing L (2023) Multi-modal cross-attention network for alzheimer’s disease diagnosis with multi-modality data. Comput Biol Med 162:107050

Xin J, Wang A, Guo R, Liu W, Tang X (2023) Cnn and swin-transformer based efficient model for alzheimer’s disease diagnosis with smri. Biomed Signal Process Control 86:105189

Dai Y, Zou B, Zhu C, Li Y, Chen Z, Ji Z, Kui X, Zhang W (2023) De-janet: A unified network based on dual encoder and joint attention for alzheimer’s disease classification using multi-modal data. Comput Biol Med 165:107396

Hu Z, Li Y, Wang Z, Zhang S, Hou W, Initiative ADN et al (2023) Conv-swinformer: Integration of cnn and shift window attention for alzheimer’s disease classification. Comput Biol Med 164:107304

Miao S, Xu Q, Li W, Yang C, Sheng B, Liu F, Bezabih TT, Yu X (2024) Mmtfn: Multi-modal multi-scale transformer fusion network for alzheimer’s disease diagnosis. Int J Imag Syst Technol 34(1):22970

Tang Y, Xiong X, Tong G, Yang Y, Zhang H (2024) Multimodal diagnosis model of alzheimer’s disease based on improved transformer. BioMedical Eng OnLine 23(1):8

Wang C, Piao S, Huang Z, Gao Q, Zhang J, Li Y, Shan H, Initiative ADN et al (2024) Joint learning framework of cross-modal synthesis and diagnosis for alzheimer’s disease by mining underlying shared modality information. Med Image Anal 91:103032

Khan AA, Mahendran RK, Perumal K, Faheem M (2024) Dual-3dm 3-ad: Mixed transformer based semantic segmentation and triplet pre-processing for early multi-class alzheimer’s diagnosis. IEEE Trans Neural Syst Rehabilit Eng

Khatri U, Kwon G-R (2024) Diagnosis of alzheimer’s disease via optimized lightweight convolution-attention and structural mri. Comput Biol Med, 108116

Chen Q, Fu Q, Bai H, Hong Y (2024) Longformer: Longitudinal transformer for alzheimer’s disease classification with structural mris. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp 3575–3584

Wang Y, Chen K, Zhang Y, Wang H (2024) Medtransformer: accurate ad diagnosis for 3d mri images through 2d vision transformers. arXiv preprint arXiv:2401.06349

Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group* t (2009) Preferred reporting items for systematic reviews and meta-analyses: the prisma statement. Ann Internal Med 151(4):264–269

Reinel T-S, Brayan A-AH, Alejandro B-OM, Alejandro M-R, Daniel A-G, Alejandro A-GJ, Buenaventura B-JA, Simon O-A, Gustavo I, Raul R-P (2021) Gbras-net: a convolutional neural network architecture for spatial image steganalysis. IEEE Access 9:14340–14350

Orozco-Arias S, Isaza G, Guyot R, Tabares-Soto R (2019) A systematic review of the application of machine learning in the detection and classification of transposable elements. PeerJ 7:8311

Chen T, Guestrin C (2016) Xgboost: A scalable tree boosting system. In: Proceedings of the 22nd Acm Sigkdd international conference on knowledge discovery and data mining, pp 785–794

Basher A, Choi KY, Lee JJ, Lee B, Kim BC, Lee KH, Jung HY (2019) Hippocampus localization using a two-stage ensemble hough convolutional neural network. IEEE Access 7:73436–73447

Basher A, Kim BC, Lee KH, Jung HY (2020) Automatic localization and discrete volume measurements of hippocampi from mri data using a convolutional neural network. IEEE Access 8:91725–91739. https://doi.org/10.1109/ACCESS.2020.2994388

Zavaliangos-Petropul A, Tubi MA, Zhu A, Haddad E, Jahanshad N, Thompson PM, Liew S-L (2020) Automated hippocampal segmentation improved by convolutional neural network approach in participants with a history of cerebrovascular accident: Neuroimaging/new imaging methods. Alzheimer’s Dementia 16:041634

Wu H, Xiao B, Codella N, Liu M, Dai X, Yuan L, Zhang L (2021) Cvt: Introducing convolutions to vision transformers. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 22–31

Download references

Acknowledgements

Mario Alejandro Bravo-Ortiz is supported by a Ph.D. grant Convocatoria 22 OCAD de Ciencia, Tecnología e Innovación del Sistema General de Regalías de Colombia y Ministerio de Ciencia, Tecnología e Innovación de Colombia. We would like to thank Universidad Autónoma de Manizales for making this paper as part of the “Clasificación de los estadios del Alzheimer utilizando Imágenes de Resonancia Magnética Nuclear y datos clínicos a partir de técnicas de Deep Learning” with code 873-139 and "Aplicación de Vision Transformer para clasificar estadios del Alzheimer utilizando imágenes de resonancia magnética nuclear y datos clínicos" project with code 847-2023 TD. Additionally, we acknowledge the support from the projects ANID PIA/BASAL FB0002 and ANID/PIA/ANILLO ACT210096. We also extend our gratitude to Universidad de Caldas for their support, as this paper is part of the project “Plataforma tecnológica para la clasificación de los estadios de la enfermedad de alzheimer utilizando imágenes de resonancia magnética nuclear, datos clínicos y técnicas de deep learning.” with code PRY-89. We also thank the National Agency for Research and Development (ANID); Applied Research Subdirection (SIA); through the instrument IDeA I+D 2023, code ID23I10357, and ORIGEN 0011323, Sistema General de Regalías (SGR) - Asignación para la Ciencia, Tecnología e Innovación, project BPIN 2021000100368, and PRY-121 - Interactive Virtual Didactic Strategy for the Promotion of ICT Skills and their Relationship with Computational Thinking.

This work was funded by Universidad Autonoma de Manizales as part of the project “Clasificación de los estadios del Alzheimer utilizando Imágenes de Resonancia Magnética Nuclear y datos clínicos a partir de técnicas de Deep Learning” with code 873-139, and also by the projects “CH-T1246: Oportunidades de Mercado para las Empresas de Tecnología-Compras Públicas de Algoritmos Responsables, Éticos y Transparentes,” ANID PIA/BASAL FB0002, and ANID/PIA/ANILLOS ACT210096.

Author information

Mario Alejandro Bravo-Ortiz and Sergio Alejandro Holguin-Garcia have contributed equally to this work.

Authors and Affiliations

Departamento de Electrónica y Automatización, Universidad Autónoma de Manizales, Manizales, 170001, Caldas, Colombia

Mario Alejandro Bravo-Ortiz, Sergio Alejandro Holguin-Garcia, Sebastián Quiñones-Arredondo, Ernesto Guevara-Navarro, Harold Brayan Arteaga-Arteaga & Reinel Tabares-Soto

Departamento de Sistemas e Informática, Universidad de Caldas, Manizales, 170004, Caldas, Colombia

Mario Alejandro Bravo-Ortiz, Sergio Alejandro Holguin-Garcia, Ernesto Guevara-Navarro & Reinel Tabares-Soto

Facultad de Ingeniería y Ciencias, Universidad Adolfo Ibáñez, 7941169, Santiago, Chile

Gonzalo A. Ruz & Reinel Tabares-Soto

Center of Applied Ecology and Sustainability (CAPES), 8331150, Santiago, Chile

Gonzalo A. Ruz

Data Observatory Foundation, 7941169, Santiago, Chile

Unidad Mixta de Imagen Biomédica FISABIO-CIPF, Fundación para el Fomento de la Investigación Sanitario y Biomédica de la Comunidad Valenciana, 46020, Valencia, Spain

Alejandro Mora-Rubio

Centro de Bioinformática y Biología Computacional (BIOS), 170001, Manizales, Colombia

Mario Alejandro Bravo-Ortiz & Sergio Alejandro Holguin-Garcia

You can also search for this author in PubMed   Google Scholar

Contributions

MABO contributed to Conceptualization, Methodology, Investigation, Writing—original draft, Writing—review and editing. SAHG contributed to Conceptualization, Methodology, Investigation, Writing—original draft, Writing—review and editing. SQA contributed to Writing—review and editing. EGN contributed to Writing—review and editing. AMR: Writing—review and editing. HBAA contributed to Writing—review and editing. GAR: Writing—review and editing, acquired the funding and provided the resources. RTS contributed to Writing—review and editing, acquired the funding and provided the resources.

Corresponding author

Correspondence to Mario Alejandro Bravo-Ortiz .

Ethics declarations

Conflict of interest.

The authors declare that they have no conflict of interest.

Ethics approval

Consent to participate, additional information, publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Bravo-Ortiz, M.A., Holguin-Garcia, S.A., Quiñones-Arredondo, S. et al. A systematic review of vision transformers and convolutional neural networks for Alzheimer’s disease classification using 3D MRI images. Neural Comput & Applic (2024). https://doi.org/10.1007/s00521-024-10420-x

Download citation

Received : 26 April 2024

Accepted : 20 August 2024

Published : 17 September 2024

DOI : https://doi.org/10.1007/s00521-024-10420-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Alzheimer’s disease
• Deep learning
• 3D magnetic resonance image
• Convolutional neural networks
• Vision transformer
• Find a journal
• Publish with us
• Track your research