systematic review thesis pdf

Advertisement

The ABC of systematic literature review: the basic methodological guidance for beginners

• Published: 23 October 2020
• Volume 55 , pages 1319–1346, ( 2021 )

Cite this article

• Hayrol Azril Mohamed Shaffril 1 ,
• Samsul Farid Samsuddin 2 &
• Asnarulkhadi Abu Samah 1 , 3  

18k Accesses

131 Citations

2 Altmetric

Explore all metrics

There is a need for more methodological-based articles on systematic literature review (SLR) for non-health researchers to address issues related to the lack of methodological references in SLR and less suitability of existing methodological guidance. With that, this study presented a beginner's guide to basic methodological guides and key points to perform SLR, especially for those from non-health related background. For that, a total of 75 articles that passed the minimum quality were retrieved using systematic searching strategies. Seven main points of SLR were discussed, namely (1) the development and validation of the review protocol/publication standard/reporting standard/guidelines, (2) the formulation of research questions, (3) systematic searching strategies, (4) quality appraisal, (5) data extraction, (6) data synthesis, and (7) data demonstration.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Similar content being viewed by others

Reading and interpreting reviews for health professionals: a practical review

Definition of a systematic review used in overviews of systematic reviews, meta-epidemiological studies and textbooks

What do meta-analysts need in primary studies guidelines and the semi checklist for facilitating cumulative knowledge.

Athukorala, K., Głowacka, D., Jacucci, G., Oulasvirta, A., Vreeken, J.: Is exploratory search different?: A comparison of information search behavior for exploratory and lookup tasks. J. Assoc. Inf. Sci. Technol. 67 (11), 2635–2651 (2016). https://doi.org/10.1002/asi.23617

Article   Google Scholar  

Barnett-Page, E., Thomas, J.: Methods for the synthesis of qualitative research: a critical review. BMC Med. Res. Methodol. (2009). https://doi.org/10.1186/1471-2288-9-59

Bates, J., Best, P., McQuilkin, J., Taylor, B.: Will web search engines replace bibliographic databases in the systematic identification of research? J. Acad. Librariansh. 43 (1), 8–17 (2017)

Google Scholar  

Berrang-Ford, L., Pearce, T., Ford, J.D.: Systematic review approaches for climate change adaptation research. Reg. Environ. Change 15 (5), 755–769 (2015). https://doi.org/10.1007/s10113-014-0708-7

Braun, V., Clarke, V.: Using thematic analysis in psychology. Qual. Res. Psychol. 3 (2), 77–101 (2006). https://doi.org/10.1191/1478088706qp063oa

Burgers, C., Brugman, B.C., Boeynaems, A.: Systematic literature reviews: four applications for interdisciplinary research. J. Pragmat. 145 , 102–109 (2019)

Brunton, G., Oliver, S., Oliver, K., Lorenc, T.: A Synthesis of Research Addressing Children’s, Young People’s and Parents’ Views of Walking and Cycling for Transport. EPPI-Centre, Social. Science Research Unit, Institute of Education, University of London, London (2006)

Cañón, M., Buitrago-Gómez, Q.: The research question in clinical practice: a guideline for its formulation. Rev Colomb Psiquiatr. 47 (3), 193–200 (2018). https://doi.org/10.1016/j.rcp.2016.06.004

Centre for Reviews and Dissemination.: Systematic Reviews: CRD’s Guidance for Undertaking Reviews in Health Care. University of York, York (2006)

Charrois, T.L.: Systematic reviews: What do you get to know to get started? Can. J. Hosp. Pharm. 68 (2), 144–148 (2015)

Cooke, A., Smith, D., Booth, A.: Beyond PICO: the SPIDER tool for qualitative evidence synthesis. Qual. Health Res. 22 (10), 1435–1443 (2012)

Cooper, C., Booth, A., Campbell, J., Britten, N., Garside, R.: Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies. BMC Med. Res. Methodol. 18 , 85 (2018)

Creswell, J.: Qualitative Inquiry & Research Design: Choosing Among Five Approaches, 3rd edn. Sage Publications, Inc., Thousand Oaks, CA (2013)

del Amo, I.F., Erkoyuncu, J.A., Roy, R., Palmarini, R., Onoufriou, D.: A systematic review of augmented reality content-related techniques for knowledge transfer in maintenance applications. Comput. Ind. 103 , 47–71 (2018)

Delaney, A., Tamás, P.A.: Searching for evidence or approval? A commentary on database search in systematic reviews and alternative information retrieval methodologies. Res. Synth. Method 9 (1), 124–131 (2018)

Dixon-Woods, M., Agarwal, S., Jones, D., Young, B., Sutton, A.: Synthesising qualitative and quantitative evidence: a review of possible methods. J. Health Serv. Res. Policy 10 (1), 45–53 (2005)

Doody, O., Bailey, M.E.: Setting a research question, aim and objective. Nurse Res. 23 (4), 19–23 (2016)

Durach, C.F., Kembro, J., Wieland, A.: A new paradigm for systematic literature reviews in supply chain management. J. Supply Chain Manag. 53 (4), 67–85 (2017)

Fagan, J.C.: An evidence-based review of academic web search engines, 2014–2016: implications for Librarians’ Practice and Research Agenda. Inf. Technol. Libr. 36 (2), 7–47 (2017)

Flemming, K., Booth, A., Garside, R., Tunc¸alp, O., Noyes J.: Qualitative evidence synthesis for complex interventions and guideline development: clarification of the purpose, designs and relevant methods. BMJ Global Health (2018)

Gomersall, J.S., Jadotte, Y.T., Xue, Y., Lockwood, S., Riddle, D., Preda, A.: Conducting systematic reviews of economic evaluations. Int. J. Evid.-Based Healthc. 13 (3), 170–178 (2015). https://doi.org/10.1097/XEB.0000000000000063

Green, B.N., Johnson, C.D., Adams, A.: Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. J. Chiropr. Med. 5 (3), 101–117 (2006). https://doi.org/10.1016/S0899-3467(07)60142-6

Greyson, D., Rafferty, E., Slater, L., MacDonald, N., Bettinger, J.A., Dubé, È., MacDonald, S.E.: Systematic review searches must be systematic, comprehensive, and transparent: a critique of Perman et al. BMC Public Health 19 (1), 1–6 (2019). https://doi.org/10.1186/s12889-018-6275-y

Gusenbauer, M.: Google Scholar to overshadow them all? Comparing the sizes of 12 academic search engines and bibliographic databases. Sciencetometrics 118 (1), 177–214 (2019)

Gusenbauer M, Haddaway NR (2020) Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Res. Synth. Methods. 11(2):181–217. https://doi.org/10.1002/jrsm.1378

Haddaway, N.R., Collins, A.M., Coughlin, D., Kirk, S.: The role of google scholar in evidence reviews and its applicability to grey literature searching. PLoS ONE 10 (9), e0138237 (2015). https://doi.org/10.1371/journal.pone.0138237

Haddaway, N.R., Macura, B., Whaley, P., Pulin, A.S.: ROSES Reporting standards for Systematic Evidence Syntheses: pro forma, flow-diagram and descriptive summary of the plan and conduct of environmental systematic reviews and systematic maps. Environ. Evid 7 , 7 (2018). https://doi.org/10.1186/s13750-018-0121-7

Halevi, G., Moed, H., Bar-Illan, J.: Suitability of Google Scholar as a source of scientific information and as a source of data for scientific evaluation. Revi Lit 11 (3), 823–834 (2017)

Hannes, K.: Critical appraisal of qualitative research. In: Noyes, J., Hannes, K., Harden, A., Harris, J., Lewin, S., Lockwood, C. (eds.) Supplementary Guidance for Inclusion of Qualitative Research in Cochrane Systematic Reviews of Interventions. Cochrane Collaboration Qualitative Methods Group, London (2011)

Higgins, J.P.T., Altman, D.G., Gotzsche, P.C., Juni, P., Moher, D., Oxman, A.D., Savovic, J., Schulz, K.F., Weeks, L., Sterne, J.A.C.: The cochrane collaboration's tool for assessing risk of bias in randomised trials. BMJ 343 (7829), 1–9 (2011). https://doi.org/10.1136/bmj.d5928

Higgins, J.P.T., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M.J., Welch, V.A. (eds.): Cochrane Handbook for Systematic Reviews of Interventions, 2nd edn. Wiley, Chichester (UK) (2019)

Hong, Q.N., Pluye, P., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M-C., Vedel, I.: Mixed Methods Appraisal Tool (MMAT), version 2018. Registration of Copyright (#1148552), Canadian Intellectual Property Office, Industry Canada (2018)

Housyar, M., Sotudeh, H.: A reflection on the applicability of Google Scholar as a tool for comprehensive retrieval in bibliometric research and systematic reviews. Int. J. Inf. Sci. Manag. 16 (2), 1–17 (2018)

Hopia, H., Latvala, E., Liimatainen, L.: Reviewing the methodology of an integrative review. Scand. J. Caring Sci. 30 (4), 662–669 (2016)

Johnson, B.T., Hennessy, E.A.: Systematic reviews and meta-analyses in the health sciences: best practice methods for research syntheses. Soc. Sci. Med. 233 , 237–251 (2019)

Kastner, M., Straus, S., Goldsmith, C.H.: Estimating the horizon of articles to decide when to stop searching in systematic reviews: an example using a systematic review of RCTs evaluating osteoporosis clinical decision support tools. AMIA Annu. Symp. Proc. Arch. 2007 , 389–393 (2007)

Kitchenham, B.A., Charters, S.M.: Guidelines for performing systematic literature reviews in software engineering. EBSE Technical Report (2007)

Kraus, S., Breier, M., Dasí-Rodríguez, S.: The art of crafting a systematic literature review in entrepreneurship research. Int. Entrep. Manag. J. 16 (3), 1023–1042 (2020). https://doi.org/10.1007/s11365-020-00635-4

Kushwah, S., Dhir, A., Sagar, M., Gupta, B.: Determinants of organic food consumption. A systematic literature review on motives and barriers. Appetite 143 , 104402 (2019)

Levy, Y., Ellis, T.J.: A systems approach to conduct an effective literature review in supports of information system research. Inf. Sci. J. 9 , 181–212 (2006)

Liberati, A., Altman, D.G., Tetzlaff, J., Mulrow, C., Gøtzsche, P.C., Ioannidis, J.P., Clarke, M., Devereaux, P.J., Kleijnen, J., Moher, D.: The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med 6 (7), e1000100 (2009)

Linares-Espinós, E., Hernández, V., Domínguez-Escrig, J.L., Fernández-Pello, S., Hevia, V., Mayor, J., Padilla-Fernández, B., Ribal, M.J.: Methodology of systematic review. Actas urologicas españolas 42 (8), 499–506 (2018)

Lockwood, C., Munn, Z., Porritt, K.: Qualitative research synthesis: methodological guidance for systematic reviewers utilizing meta-aggregation. Int. J. Evid. Based Healthc. 13(3), 179–187 (2015). https://doi.org/10.1097/XEB.0000000000000062

Long, A.F., Godfrey, M.: An evaluation tool to assess the quality of qualitative research studies. Int. J. Soc. Res. Methodol. 7 (2), 181–196 (2004)

Mallet, R., Hagen-Zanker, J., Slater, R., Duvendack, M.: The benefits and challenges of using systematic reviews in international development research. J. Dev. Eff. 4 , 445–455 (2012)

Mantzoukas, S.: Facilitating research students in formulating qualitative research questions. Nurse Educ. Today 28 (3), 371–377 (2008)

Mays, N., Pope, C., Popay, J.: Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field. J. Health Serv. Res. Policy 10 (1), 6–20 (2005)

Mengist, W., Soromessa, T., Legese, G.: Method for conducting systematic literature review and meta-analysis for environmental science research. MethodsX 7 , 100777 (2020). https://doi.org/10.1016/j.mex.2019.100777

Methley, A.M., Campbell, S., Chew-Graham, C., McNally, R., Cheraghi-Sohi, S.: PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv. Res. (2014). https://doi.org/10.1186/s12913-014-0579-0

Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G., PRISMA Group: Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 21;6 (7), e1000097 (2009)

Morton SC, Murad MH, O’Connor E, Lee CS, Booth M, Vandermeer BW, Snowden JM, D’Anci KE, Fu R, Gartlehner G, Wang Z, Steele DW (2018) Quantitative synthesis—an update. methods guide for comparative effectiveness reviews. (Prepared by the Scientific Resource Center under Contract No. 290-2012-0004-C). AHRQ Publication No. 18-EHC007- EF. Rockville, MD: Agency for Healthcare Research and Quality. Posted final reports are located on the Effective Health Care Program search page. https://doi.org/10.23970/AHRQEPCMETHGUIDE3

Noble, H., Mitchell, M.: What is grounded theory? Evid. Based Nurs. 19 (2), 34–35 (2016). https://doi.org/10.1136/eb-2016-102306

Okoli, C.: A guide to conducting a standalone systematic literature review. Commun. Assoc. Inf. Syst. 37 , 879–910 (2015)

Okoli, C., Schabram, K.: A guide to conducting a systematic literature review of information systems research. Philos. Methodol. Econ. eJ. (2010). https://doi.org/10.2139/ssrn.1954824

Onwuegbuzie, A.J., Leech, N.L.: Sampling designs in qualitative research: making the sampling process more public. Qual. Rep. 12 (2), 238–254 (2007)

Pace, R., Pluye, P., Bartlett, G., Macaulay, A.C., Salsberg, J., Jagosh, J., Seller, R.: Testing the reliability and efficiency of the pilot Mixed Methods Appraisal Tool (MMAT) for systematic mixed studies review. Int. J. Nurs Stud. 49 (1), 47–53 (2012). https://doi.org/10.1016/j.ijnurstu.2011.07.002

Page, M.J., McKenzie, J.E., Bossuyt, P.M., Boutron, I., Hoffmann, T., Mulrow, C.D., Shamseer, L., Moher, D.: Mapping of reporting guidance for systematic reviews and meta-analyses generated a comprehensive item bank for future reporting guidelines. J. Clin. Epidemiol. 118 , 60–68 (2020)

Paterson, B.L., Thorne, S.E., Canam, C., Jillings, C.: Meta-Study of Qualitative Health Research. A Practical Guide to Meta-Analysis and Meta-Synthesis. Sage Publications, Thousand Oaks (2001)

Palaskar, J.N.: Framing the research question using PICO strategy. J. Dent. Allied Sci. 6 (2), 55 (2017)

Patino, C.M., Ferreira, J.C.: Inclusion and exclusion criteria in research studies: definitions and why they matter. J. Bras. Pneumol. 44 (2), 84 (2018)

Peters, M.D., Godfrey, C.M., Khalil, H., McInerney, P., Parker, D., Soares, C.B.: Guidance for conducting systematic scoping reviews. Int. J. Evid.-Based Healthc. 13 (3), 141–146 (2015). https://doi.org/10.1097/XEB.0000000000000050

Petticrew, M., Roberts, H.: Systematic Reviews in the Social Sciences: A Practical Guide. Blackwell Publishing Ltd, Oxford (2006)

Prieto and Rumbo-Prieto: The systematic review: plurality of approaches and methodologies. Enferm. Clín. (English Edition) 28 (6), 387–393 (2018)

Reim, W., Parida, V., Örtqvist, D.: Product-Service Systems (PSS) business models and tactics – a systematic literature review. J. Clean. Prod. 97 , 61–75 (2015). https://doi.org/10.1016/j.jclepro.2014.07.003

Robinson, P., Lowe, J.: Literature reviews vs systematic reviews. Aust. N. Z. J. Public Health 39 (2), 103 (2015)

Rousseau, D. M., Manning, J., Denyer, D.: Evidence in management and organizational science: assembling the field’s full weight of scientific knowledge through syntheses. In: AIM Research Working Paper Series: Advanced Institute of Management Research (2008)

Sandelowski, M., Barroso, J., Voils, C.I.: Using qualitative metasummary to synthesize qualitative and quantitative descriptive findings. Res. Nuurs. Health 30 (1), 99–111 (2007). https://doi.org/10.1002/nur.20176

Sandelowski, M., Voils, C.I., Barroso, J.: Defining and designing mixed research synthesis studies. Res. Schools: Nat. Ref. J. Spons. Mid-South Educ. Res. Assoc. Univ. Alabama 13 (1), 29 (2006)

Schardt, C., Adams, M.B., Owens, T., Keitz, S., Fontelo, P.: Utilization of the PICO framework to improve searching PubMed for clinical questions. BMC Med. Inf. Decis. Mak. 7 , 16 (2007). https://doi.org/10.1186/1472-6947-7-16

Seehra, J., Pandis, N., Koletsi, D.: Use of quality assessment tools in systematic reviews was varied and inconsistent. J. Clin. Epidemiol. 69 , 179–184 (2016)

Shaffril, H.A.M., Krauss, S.E., Samsuddin, S.F.: A systematic review on Asian’s farmers’ adaptation practices towards climate change. Sci. Total Environ. 644 , 683–695 (2018)

Shaffril, H.A.M., Abu Samah, A., Samsuddin, S.F., Ali, Z.: Mirror-mirror on the wall, what climate change adaptation strategies are practiced by the Asian’s fishermen of all? J. Clean. Prod. 232 , 104–117 (2019)

Shorten, A., Shorten, B.: What is meta-analysis. Evid. Based Nurs. 16 (1), 3–4 (2013)

Siering, U., Eikermann, M., Hausner, E., Hoffmann-Eßer, W., Neugebauer, E.A.: Appraisal tools for clinical practice guidelines: a systematic review. PLoS ONE 8 (12), e82915 (2013)

Soares, C.B., Hoga, L., Sangaleti, C., Yonekura, T., Peduzzi, M., Silva, D.: Integrative review in nursing research and EBP: a type of systematic review? Int. J.Evid.-Based Healthcare 11 (3), 246–247 (2013)

Thomas, J., Noel-Storrb, A., Marshall, I., Wallace, B., McDonald, S., Mavergames, C., Glasziou, P., Shemilta, I., Synnote, A., Turnere, T., Elliott, J.: Living systematic reviews: 2. Combining human and machine effort. J. Clin. Epidemiol. 91 , 31–37 (2017)

Thomas, J., Kneale, D., McKenzie, J.E., Brennan, S.E., Bhaumik, S.: Chapter 2: Determining the scope of the review and the questions it will address. In: Higgins, J.P.T., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M.J., Welch, V.A. (eds.) Cochrane Handbook for Systematic Reviews of Interventions version 6.1 (2020). Cochrane (2020). https://www.training.cochrane.org/handbook

Wanden-Berghe, C., Sanz-Valero, J.: Systematic reviews in nutrition: standardized methodology. Br. J. Nutr. 107 , S3–S7 (2012)

Whittemore, R., Knafl, K.: The integrative review: updated methodology. J. Adv. Nurs. 52 (5), 546–553 (2005)

Wong, G., Greenhalgh, T., Westhorp, G., Buckingham, J., Pawson, R.: RAMESES publication standards: realist syntheses. BMC Med. 11 , 21 (2013). https://doi.org/10.1186/1741-7015-11-21

Xiao, Y., Watson, M.: Guidance on conducting a systematic literature review. J. Plan. Educ. 39 (1), 93–112 (2019)

Younger, P.: Using Google Scholar to conduct a literature search. Nurs. Stand. 24 (45), 40–46 (2010)

Download references

Author information

Authors and affiliations.

Institute for Social Science Studies, Universiti Putra Malaysia, Putra Infoport, 43400, Serdang, Selangor, Malaysia

Hayrol Azril Mohamed Shaffril & Asnarulkhadi Abu Samah

Faculty of Computer Science and Information Technology, University of Malaya, 50603, Kuala Lumpur, Malaysia

Samsul Farid Samsuddin

Faculty of Human Ecology, Universiti Putra Malaysia, 43400, Serdang, Selangor, Malaysia

Asnarulkhadi Abu Samah

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Hayrol Azril Mohamed Shaffril .

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Mohamed Shaffril, H.A., Samsuddin, S.F. & Abu Samah, A. The ABC of systematic literature review: the basic methodological guidance for beginners. Qual Quant 55 , 1319–1346 (2021). https://doi.org/10.1007/s11135-020-01059-6

Download citation

Accepted : 12 October 2020

Published : 23 October 2020

Issue Date : August 2021

DOI : https://doi.org/10.1007/s11135-020-01059-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Systematic literature review
• Basic methodology
• Non-health related studies
• Find a journal
• Publish with us
• Track your research

• Open access
• Published: 12 December 2017

Acceptance of a systematic review as a thesis: survey of biomedical doctoral programs in Europe

• Livia Puljak   ORCID: orcid.org/0000-0002-8467-6061 1 , 2 , 3 &
• Damir Sapunar 3  

Systematic Reviews volume  6 , Article number:  253 ( 2017 ) Cite this article

15k Accesses

23 Citations

66 Altmetric

Metrics details

Systematic reviews (SRs) have been proposed as a type of research methodology that should be acceptable for a graduate research thesis. The aim of this study was to analyse whether PhD theses in European biomedical graduate programs can be partly or entirely based on SRs.

In 2016, we surveyed individuals in charge of European PhD programs from 105 institutions. The survey asked about acceptance of SRs as the partial or entire basis for a PhD thesis, their attitude towards such a model for PhD theses, and their knowledge about SR methodology.

We received responses from 86 individuals running PhD programs in 68 institutions (institutional response rate of 65%). In 47% of the programs, SRs were an acceptable study design for a PhD thesis. However, only 20% of participants expressed a personal opinion that SRs meet the criteria for a PhD thesis. The most common reasons for not accepting SRs as the basis for PhD theses were that SRs are ‘not a result of a PhD candidate’s independent work, but more of a team effort’ and that SRs ‘do not produce enough new knowledge for a dissertation’. The majority of participants were not familiar with basic concepts related to SRs; questions about meta-analyses and the type of plots frequently used in SRs were correctly answered by only one third of the participants.

Conclusions

Raising awareness about the importance of SRs and their methodology could contribute to higher acceptance of SRs as a type of research that forms the basis of a PhD thesis.

Peer Review reports

Systematic reviews (SRs) are a type of secondary research, which refers to the analysis of data that have already been collected through primary research [ 1 ]. Even though SRs are a secondary type of research, a SR needs to start with a clearly defined research question and must follow rigorous research methodology, including definition of the study design a priori, data collection, appraisal of study quality, numerical analyses in the form of meta-analyses and other analyses when relevant and formulation of results and conclusions. Aveyard and Sharp defined SRs as ‘original empirical research’ because they ‘review, evaluate and synthesise all the available primary data, which can be either quantitative or qualitative’ [ 2 ]. Therefore, a SR represents a new research contribution to society and is considered the highest level in the hierarchy of evidence in medicine [ 3 ].

SRs have been proposed as a type of research methodology that should be acceptable as the basis for a graduate research thesis [ 4 , 5 ]. To the best of our knowledge, there are no reports on the acceptance of SRs as the basis for PhD theses. A recent review addressed potential advantages and disadvantages of such a thesis type and presented opposing arguments about the issue [ 5 ]. However, there were no actual data that would indicate how prevalent one opinion is over another with regard to the acceptance of a SR as the primary research methodology for a PhD thesis. The aim of this cross-sectional study was to assess whether a PhD thesis in European biomedical graduate programs can be partly or entirely based on a SR, as well as to explore the attitudes and knowledge of individuals in charge of PhD programs with regard to a thesis of this type.

Participants

The Organization of PhD Education in Biomedicine and Health Sciences in the European System (ORPHEUS) includes 105 institutional members from 40 countries and six associate members from Canada, Georgia, Iran, Kyrgyzstan, Kazakhstan and the USA [ 6 ]. The ORPHEUS encompasses a network of higher education institutions committed to developing and disseminating best practice within PhD training programs in biomedicine, health sciences and public health. ORPHEUS approved the use of their mailing list for the purpose of this study. The mailing list had 1049 contacts. The study authors were not given the mailing list due to data protection and privacy. Instead, it was agreed that ORPHEUS officials would send the survey via email to the mailing list. The General Secretary of the ORPHEUS contacted individuals responsible for PhD programs (directors or deputy directors) among the institutional members, via e-mail, on 5th of July 2016. These individuals were sent an invitation to complete an online survey about SRs as the basis for PhD theses. We invited only individuals responsible for PhD programs (e.g., directors, deputy directors, head of graduate school, vice deans for graduate school or similar). We also asked them to communicate with other individuals in charge of their program to make sure that only one person per PhD program filled out the survey. If there were several PhD programs within one institution, we asked for participation of one senior person per program.

The survey was administered via Survey Monkey (Portland, OR, USA). The survey took 5–10 min to complete. One reminder was sent to the targeted participants 1 month after the first mail.

The ethics committee of the University of Split School of Medicine approved this study, which formed part of the Croatian Science Foundation grant no. IP-2014-09-7672 ‘Professionalism in Health Care’.

Questionnaire

The 20-item questionnaire, designed specifically for this study by both authors (LP and DS), was first tested for face validity and clarity among five individuals in charge of PhD programs. The questionnaire was then modified according to their feedback. The questionnaire included questions about their PhD program; whether PhD candidates are required to publish manuscript(s) before thesis defence; the minimum number of required manuscripts for defending a PhD thesis; the authorship requirements for a PhD candidate with regard to published manuscript(s); whether there is a requirement for a PhD candidate to publish manuscript(s) in journals indexed in certain databases or journals of certain quality, and how the quality is defined; the description about other requirements for defending a PhD thesis; whether a SR partly or fully meets requirements for approval of a PhD thesis in their graduate program; what are the rules related to the use of a SR as the basis for a PhD thesis; and the number of PhD theses based on SRs relative to other types of research methods.

Participants were also asked about their opinion with regard to the main reasons that SRs are not recognised in some institutions as the basis for a doctoral dissertation, and their opinion about literature reviews, using a four-item Likert scale, ranging from ‘agree’ to ‘disagree’, including an option for ‘don’t know’. In the last question, the participants’ knowledge about SR methodology was examined using nine statements; participants had to rate each statement as either ‘correct’, ‘incorrect’, ‘unsure’ or ‘I don’t know’. Finally, participants were invited to leave their email address if they wanted to receive survey results. The survey sent to the study participants can be found in an additional file (Additional file  1 ).

Data analysis

Survey responses were entered into a spreadsheet, checked by both authors and analysed using Microsoft Excel (Microsoft Inc., Redmond, WA, USA). Descriptive data are presented as frequencies and percentages. All raw data and analysed data sets used in the manuscript are available from authors on request. A point-biserial correlation (SPSS, IBM, Chicago, IL, USA) was used to measure the strength of the association between results on the knowledge test (continuous variable) and the attitude towards SRs as the basis for dissertations (dichotomous variable; we used the answer to the following question as this measure: ‘Do you agree that a systematic review, in whole or in part, meets the criteria for a publication on which a doctoral dissertation can be based?’).

Study participants

There are 105 institutions included in the ORPHEUS network. We received a response from 86 individuals representing 68 institutions from 37 countries (65% institutional response rate). There were more respondents than institutions because some institutions have several PhD programs and thus several program directors. Those responders were used as a unit of analysis in the analysis of attitudes and knowledge; institutions were the unit of analysis when analysing criteria for theses. Some of the questionnaires ( n  = 15) were only partly completed. In most cases, the missing data were related to knowledge about SR methodology.

Overview of requirements for a dissertation

Based on the information provided by the graduate program directors, in the majority of the included PhD programs, students were required to publish a research manuscript prepared within their PhD thesis prior to their thesis defence (83%; n  = 64). Among 13 programs (17%) that did not have this requirement, five respondents (38%) indicated that in their opinion their school’s rules related to a PhD thesis should be changed such as to specify that each thesis should be based on work that is already published in a journal.

The minimum number of published manuscripts necessary for the PhD thesis defence was prespecified in 94% ( n  = 60) of the programs that required publication of research manuscripts prior to the thesis defence. In most of the programs (37%; n  = 22), the number of required manuscripts was three or more. Two manuscripts were required in 30% ( n  = 18) and one was required in 33% ( n  = 20) of the programs. In four programs, there was no formal policy on this matter, but there was a strong expectation that the student will have contributed substantially to several manuscripts in peer-reviewed journals.

In most cases, the PhD candidates’ contribution to published manuscripts within the PhD thesis was determined through first authorship. A requirement that a PhD candidate should be the first author on a manuscript(s) that constitutes a PhD thesis was reported in 82% ( n  = 64) of the graduate programs.

In 60% ( n  = 52) of the graduate programs, the quality of the journals where a PhD candidate has to publish research manuscripts as a part of a PhD thesis was defined by the database in which these journals are indexed. The most commonly specified databases were Web of Science (41%; n  = 35) and MEDLINE/PubMed (13%; n  = 11), followed by Science Citation Index, Scopus, Current Contents, a combination of several databases or, in two cases, a combination of journals from a list defined by some governing body.

Systematic reviews as a PhD thesis

SRs, in whole or in part, met the criteria for acceptable research methodology for a PhD thesis in 47% ( n  = 40) of programs, whereas 53% ( n  = 46) of programs specifically stated that they did not accept SRs in this context (Fig.  1 a, b). Among the programs that accepted SRs, theses could be exclusively based on a SR in 42% ( n  = 17) of programs, while in the remaining programs, SRs were acceptable as one publication among others in a dissertation.

a European PhD programs that recognise a systematic review as a PhD thesis (green dot) and those that do not (red dot). Half red and half green dots indicate the five universities with institutions that have opposite rules regarding recognition of a systematic review as a PhD thesis. The pie chart presents b the percentage of the programs in which systematic reviews, in whole or in part, meet the criteria for a dissertation and c the opinion of participants about whether systematic reviews should form the basis of a publication within a PhD dissertation

The majority of participants (80%; n  = 69) indicated that SRs did not meet criteria for a publication on which a PhD dissertation should be based (Fig.  1 c). The main arguments for not recognising a SR as the basis for a PhD thesis are listed in Table  1 . The majority of respondents were neutral regarding the idea that scoping reviews or SRs should replace traditional narrative reviews preceding the results of clinical and basic studies in doctoral theses. Most of the respondents agreed that narrative or critical/discursive literature reviews preceding clinical studies planned as part of a dissertation should be replaced with systematic reviews (Table  2 ).

Most of the programs that accepted SRs as a research methodology acceptable for PhD theses had defined rules related to the use of an SR as part of a PhD thesis (Fig.  2 ). The most common rule was that a SR can be one publication among others within a PhD thesis. Some of the respondents indicated that empty (reviews that did not find a single study that should be included after literature search) or updated reviews could also be used for a PhD thesis (Fig.  2 ).

Frequency of different rules that define the use of systematic reviews as a part of a PhD thesis in European biomedical graduate programs

The results of the survey regarding knowledge about SR methodology indicated that the majority of respondents were not familiar with this methodology. Only three out of nine questions were correctly answered by more than 80% of the participants, and questions about meta-analyses and the type of plots frequently used in a SR were correctly answered by only one third of the participants (Table  3 ). The association between participants’ results on the knowledge test and attitudes towards SRs was tested using a point-biserial correlation; this revealed that lack of knowledge was not correlated with negative attitudes towards SRs ( r pb  = 0.011; P  = 0.94).

In this study conducted among individuals in charge of biomedical graduate programs in Europe, we found that 47% of programs accepted SRs as research methodology that can partly or fully fulfil the criteria for a PhD thesis. However, most of the participants had negative attitudes about such a model for a PhD thesis, and most had insufficient knowledge about the basic aspects of SR methodology. These negative attitudes and lack of knowledge likely contribute to low acceptance of SRs as an acceptable study design to include in a PhD thesis.

A limitation of this study was that we relied on participants’ responses and not on assessments of formal rules of PhD programs. Due to a lack of familiarity with SRs, it is possible that the respondents gave incorrect answers. We believe that this might be the case since we received answers from different programs in the same university, where one person claimed that SRs were accepted in their program, and the other person claimed that they were not accepted in the other program. We had five such cases, so it is possible that institutions within the same university have different rules related to accepted research methodology in graduate PhD programs. This study may not be generalisable to different PhD programs worldwide that were not surveyed. The study is also not generalisable to Europe, as there are no universal criteria or expectations for PhD theses in Europe. Even in the same country, there may be different models and expectations for a PhD in different higher education institutions.

A recent study indicated a number of opposing views and disadvantages related to SRs as research methodology for graduate theses, including lack of knowledge and understanding by potential supervisors, which may prevent them from being mentors and assisting students to complete such a study [ 5 ]. This same manuscript emphasised that there may be constraints if the study is conducted in a resource-limited environment without access to electronic databases, that there may be a very high or very low number of relevant studies that can impact the review process, that methods may not be well developed for certain types of research syntheses and that it may be difficult to publish SRs [ 5 ].

Some individuals believe that a SR is not original research. Indeed, it has been suggested that SRs as ‘secondary research’ are different than ‘primary or original research’, implying that they are inferior and lacking in novelty and methodological rigour as compared to studies that are considered primary research. In 1995, Feinstein suggested that such studies are ‘statistical alchemy for the 21st century’ and that a meta-analysis removes or destructs ‘scientific requirements that have been so carefully developed and established during the 19th and 20th centuries’ [ 7 ]. There is little research about this methodological issue. Meerpohl et al. surveyed journal editors and asked whether they consider SRs to be original studies. The majority of the editors indicated that they do think that SRs are original scientific contributions (71%) and almost all journals (93%) published SRs. That study also highlighted that the definition of original research may be a grey area [ 8 ]. They argued that, in an ideal situation, ‘the research community would accept systematic reviews as a research category of its own, which is defined by methodological criteria, as is the case for other types of research’ [ 8 ]. Biondi-Zoccai et al. pointed out that the main criteria to judge a SR should be its novelty and usefulness, and not whether it is original/primary or secondary research [ 9 ].

In our study, 80% of the participants reported negative attitudes, and more than half of the respondents agreed with a statement that SRs are ‘not a result of the candidate’s independent work since systematic reviews tend to be conducted by a team’. This opinion is surprising since other types of research are also conducted within a team, and single authorship is very rare in publications that are published within a PhD thesis. On the contrary, the mean number of authors of research manuscripts is continuously increasing [ 10 ]. At the very least, the authors of manuscripts within a PhD will include the PhD candidate and a mentor, which is a team in and of itself. Therefore, it is unclear why somebody would consider it a problem that a SR is conducted within a team.

The second most commonly chosen argument against such a thesis was that SRs ‘do not produce enough new knowledge for a dissertation’. The volume of a SR largely depends on the number of included studies and the available data for numerical analyses. Therefore, it is unfair to label a SR as a priori lacking in new knowledge. There are SRs with tens or hundreds of included studies, and some of them not only include meta-analyses, but also network meta-analyses, which are highly sophisticated statistical methods. However, limiting SRs within a thesis only to those with meta-analysis would be unfair because sometimes meta-analysis is not justified due to clinical or statistical heterogeneity [ 11 ] and the presence or absence of a meta-analysis is not an indicator of the quality of a SR. Instead, there are relevant checklists for appraising methodological and reporting quality of a SR [ 12 , 13 ].

The third most commonly chosen argument against SRs within PhD theses was ‘lack of adequate training of candidates in methodology of systematic reviews’. This could refer to either insufficient formal training or insufficient mentoring. The graduate program and the mentor need to ensure that a PhD candidate receives sufficient knowledge to complete the proposed thesis topic. Successful mentoring in academic medicine requires not only commitment and interpersonal skills from both the mentor and mentee, but also a facilitating institutional environment [ 14 ]. This finding could be a result of a lack of capacity and knowledge for conducting SRs in the particular institutions where the survey was conducted, and not general opinion related to learning a research method when conducting a PhD study. Formal training in skills related to SRs and research synthesis methods [ 15 , 16 ], as well as establishing research collaborations with researchers experienced in this methodology, could alleviate this concern.

One third of the participants indicated a ‘lack of appreciation of systematic review methodology among faculty members’ as a reason against such a thesis model. This argument, as well as the prevalent negative attitude towards SRs as PhD theses, perhaps can be traced to a lack of knowledge about SR methodology; however, although the level of knowledge was quite low in our study, there was no statistically significant correlation between knowledge and negative attitudes. Of the nine questions about SR research methodology, only three questions were correctly answered by more than half of the participants. This could be a cause for concern because it has been argued that any health research should begin with a SR of the literature [ 17 ]. It has also been argued that the absence of SRs in the context of research training might severely hamper research trainees and may negatively impact the research conducted [ 18 ]. Thus, it has been recommended that SRs should be included ‘whenever appropriate, as a mandatory part of any PhD program or candidature’ [ 18 ].

It has recently been suggested that the overwhelming majority of investment in research represents an ‘avoidable waste’ [ 19 ]. Research that is not necessary harms both the public and patients, because funds are not invested where they are really necessary, and necessary research may not be conducted [ 17 ]. This is valid not only for clinical trials, but also for other types of animal and human experiments [ 20 ]. SRs can help improve the design of new experiments by relying on current evidence in the field and by helping to clarify which questions still need to be addressed. SRs can be instrumental in improving methodological quality of new experiments, providing evidence-based recommendations for research models, reducing avoidable waste, and enabling evidence-based translational research [ 20 ].

Four respondents from three institutions indicated that empty SRs are accepted as a PhD thesis. While it makes sense to include such a SR as a part of the thesis to indicate lack of evidence in a certain field, it is highly unlikely that an entire thesis can be based on an empty SR, without a single included study.

There are many advantages of a SR as a graduate thesis [ 4 , 5 ], especially as a research methodology suitable for low-resource settings. A PhD candidate can prepare a Cochrane SR as a part of the PhD thesis, yielding a high-impact publication [ 4 ]. Non-Cochrane SRs can also be published in high-impact journals. A PhD candidate involved in producing a SR within a PhD thesis goes through the same research process as those conducting primary research, from setting up a hypothesis and a research question, to development of a protocol, data collection, data analysis and appraisal, and formulation of conclusions. Graduate programs can set limits, such as the prevention of empty reviews and the recognition of updated reviews as valid for a PhD thesis, and engage experienced researchers as advisors and within thesis evaluation committees, to ensure that a candidate will conduct a high-quality SR [ 4 ]. Conducting a SR should not be mandatory, but candidates and mentors willing to produce such research within a graduate program should be allowed to do so.

Further studies in this field could provide better insight into attitudes related to SRs as graduate theses and explore interventions that can be used to change negative attitudes and improve knowledge of SRs among decision-makers in graduate education.

Raising awareness about the importance of SRs in biomedicine, the basic aspects of SR methodology and the status of SRs as original secondary research could contribute to greater acceptance of SRs as potential PhD theses. Our results can be used to create strategies that will enhance acceptance of SRs among graduate education program directors.

Gopalakrishnan S, Ganeshkumar P. Systematic reviews and meta-analysis: understanding the best evidence in primary healthcare. J. Fam. Med Prim Care. 2013;2(1):9–14.

Article   CAS   Google Scholar  

Aveyard H, Sharp P. A beginner’s guide to evidence-based practice in health and social care. Glasgow: McGraw Open Press University; 2011.

Google Scholar  

Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med. 1997;126(5):376–80.

Article   CAS   PubMed   Google Scholar  

Puljak L, Sambunjak D. Cochrane systematic review as a PhD thesis: an alternative with numerous advantages. Biochemia Medica. 2010;20(3):319–2.

ten Ham-Baloyi W, Jordan P. Systematic review as a research method in post-graduate nursing education. Health SA Gesondheid. 2016;21:120–8.

Article   Google Scholar  

Organisation for PhD Education in Biomedicine and Health Sciences in the European System (ORPHEUS). Available at: http://www.orpheus-med.org/ .

Feinstein AR. Meta-analysis: statistical alchemy for the 21st century. J Clin Epidemiol. 1995;48(1):71–9.

Meerpohl JJ, Herrle F, Reinders S, Antes G, von Elm E. Scientific value of systematic reviews: survey of editors of core clinical journals. PLoS One. 2012;7(5):e35732.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Biondi-Zoccai G, Lotrionte M, Landoni G, Modena MG. The rough guide to systematic reviews and meta-analyses. HSR proc intensive care cardiovascular anesth. 2011;3(3):161–73.

CAS   Google Scholar  

Baethge C. Publish together or perish: the increasing number of authors per article in academic journals is the consequence of a changing scientific culture. Some researchers define authorship quite loosely. Dtsch. Arztebl. Int. 2008;105(20):380–3.

PubMed   PubMed Central   Google Scholar  

Higgins JPT, Thompson SG. Quantifying heterogeneity in a meta-analysis. Statist Med. 2002;21:1539–58.

Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, Clarke M, Devereaux PJ, Kleijnen J, Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009;6(7):e1000100.

Article   PubMed   PubMed Central   Google Scholar  

Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, Porter AC, Tugwell P, Moher D, Bouter LM. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7:10.

Sambunjak D, Straus SE, Marusic A. A systematic review of qualitative research on the meaning and characteristics of mentoring in academic medicine. J Gen Intern Med. 2010;25(1):72–8.

Article   PubMed   Google Scholar  

Balajic K, Barac-Latas V, Drenjancevic I, Ostojic M, Fabijanic D, Puljak L. Influence of a vertical subject on research in biomedicine and activities of the Cochrane collaboration branch on medical students’ knowledge and attitudes toward evidence-based medicine. Croat Med J. 2012;53(4):367–73.

Marusic A, Sambunjak D, Jeroncic A, Malicki M, Marusic M. No health research without education for research—experience from an integrated course in undergraduate medical curriculum. Med Teach. 2013;35(7):609.

Mahtani KR. All health researchers should begin their training by preparing at least one systematic review. J R Soc Med. 2016;109(7):264–8.

Olsson C, Ringner A, Borglin G. Including systematic reviews in PhD programmes and candidatures in nursing - ‘Hobson’s choice’? Nurse Educ Pract. 2014;14(2):102–5.

Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet. 2009;374(9683):86–9.

de Vries RB, Wever KE, Avey MT, Stephens ML, Sena ES, Leenaars M. The usefulness of systematic reviews of animal experiments for the design of preclinical and clinical studies. ILAR J. 2014;55(3):427–37.

Download references

Acknowledgements

The authors thank the ORPHEUS secretariat for administering the survey and the study participants for taking time to participate in the survey. We are grateful to Prof. Ana Marušić for the critical reading of the manuscript.

This research was funded by the Croatian Science Foundation, grant no. IP-2014-09-7672 ‘Professionalism in Health Care’. The funder had no role in the design of this study or its execution and data interpretation.

Availability of data and materials

The datasets used and/or analysed for the current study are available from the corresponding author on reasonable request.

Author information

Authors and affiliations.

Cochrane Croatia, University of Split School of Medicine, Šoltanska 2, 21000, Split, Croatia

Livia Puljak

Department for Development, Research and Health Technology Assessment, Agency for Quality and Accreditation in Health Care and Social Welfare, Planinska 13, 10000, Zagreb, Croatia

Laboratory for Pain Research, University of Split School of Medicine, Šoltanska 2, 21000, Split, Croatia

Livia Puljak & Damir Sapunar

You can also search for this author in PubMed   Google Scholar

Contributions

Both authors participated in the study design, data collection and analysis and writing of the manuscript, and both read and approved the final manuscript.

Corresponding author

Correspondence to Livia Puljak .

Ethics declarations

Ethics approval and consent to participate.

The Ethics Committee of the University of Split School of Medicine approved the study. All respondents consented to participate in the study.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional file

Additional file 1:.

Online survey used in the study. Full online survey that was sent to the study participants. (PDF 293 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Puljak, L., Sapunar, D. Acceptance of a systematic review as a thesis: survey of biomedical doctoral programs in Europe. Syst Rev 6 , 253 (2017). https://doi.org/10.1186/s13643-017-0653-x

Download citation

Received : 29 August 2017

Accepted : 30 November 2017

Published : 12 December 2017

DOI : https://doi.org/10.1186/s13643-017-0653-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Systematic review
• PhD program
• Biomedicine
• Study design

Systematic Reviews

ISSN: 2046-4053

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]