literature review with software

Something went wrong when searching for seed articles. Please try again soon.

No articles were found for that search term.

Author, year The title of the article goes here

June Trials for Institutions

LITERATURE REVIEW SOFTWARE FOR BETTER RESEARCH

“Litmaps is a game changer for finding novel literature... it has been invaluable for my productivity.... I also got my PhD student to use it and they also found it invaluable, finding several gaps they missed”

Varun Venkatesh

Austin Health, Australia

As a full-time researcher, Litmaps has become an indispensable tool in my arsenal. The Seed Maps and Discover features of Litmaps have transformed my literature review process, streamlining the identification of key citations while revealing previously overlooked relevant literature, ensuring no crucial connection goes unnoticed. A true game-changer indeed!

Ritwik Pandey

Doctoral Research Scholar – Sri Sathya Sai Institute of Higher Learning

Using Litmaps for my research papers has significantly improved my workflow. Typically, I start with a single paper related to my topic. Whenever I find an interesting work, I add it to my search. From there, I can quickly cover my entire Related Work section.

David Fischer

Research Associate – University of Applied Sciences Kempten

“It's nice to get a quick overview of related literature. Really easy to use, and it helps getting on top of the often complicated structures of referencing”

Christoph Ludwig

Technische Universität Dresden, Germany

“This has helped me so much in researching the literature. Currently, I am beginning to investigate new fields and this has helped me hugely”

Aran Warren

Canterbury University, NZ

“I can’t live without you anymore! I also recommend you to my students.”

Professor at The Chinese University of Hong Kong

“Seeing my literature list as a network enhances my thinking process!”

Katholieke Universiteit Leuven, Belgium

“Incredibly useful tool to get to know more literature, and to gain insight in existing research”

KU Leuven, Belgium

“As a student just venturing into the world of lit reviews, this is a tool that is outstanding and helping me find deeper results for my work.”

Franklin Jeffers

South Oregon University, USA

“Any researcher could use it! The paper recommendations are great for anyone and everyone”

Swansea University, Wales

“This tool really helped me to create good bibtex references for my research papers”

Ali Mohammed-Djafari

Director of Research at LSS-CNRS, France

“Litmaps is extremely helpful with my research. It helps me organize each one of my projects and see how they relate to each other, as well as to keep up to date on publications done in my field”

Daniel Fuller

Clarkson University, USA

As a person who is an early researcher and identifies as dyslexic, I can say that having research articles laid out in the date vs cite graph format is much more approachable than looking at a standard database interface. I feel that the maps Litmaps offers lower the barrier of entry for researchers by giving them the connections between articles spaced out visually. This helps me orientate where a paper is in the history of a field. Thus, new researchers can look at one of Litmap's "seed maps" and have the same information as hours of digging through a database.

Baylor Fain

Postdoctoral Associate – University of Florida

We use cookies to ensure that we give you the best experience on our website. If you continue to use this site we will assume that you are happy with it.

Start your free trial

Arrange a trial for your organisation and discover why FSTA is the leading database for reliable research on the sciences of food and health.

REQUEST A FREE TRIAL

• Research Skills Blog

5 software tools to support your systematic review processes

By Dr. Mina Kalantar on 19-Jan-2021 13:01:01

Systematic reviews are a reassessment of scholarly literature to facilitate decision making. This methodical approach of re-evaluating evidence was initially applied in healthcare, to set policies, create guidelines and answer medical questions.

Systematic reviews are large, complex projects and, depending on the purpose, they can be quite expensive to conduct. A team of researchers, data analysts and experts from various fields may collaborate to review and examine incredibly large numbers of research articles for evidence synthesis. Depending on the spectrum, systematic reviews often take at least 6 months, and sometimes upwards of 18 months to complete.

The main principles of transparency and reproducibility require a pragmatic approach in the organisation of the required research activities and detailed documentation of the outcomes. As a result, many software tools have been developed to help researchers with some of the tedious tasks required as part of the systematic review process.

hbspt.cta._relativeUrls=true;hbspt.cta.load(97439, 'ccc20645-09e2-4098-838f-091ed1bf1f4e', {"useNewLoader":"true","region":"na1"});

The first generation of these software tools were produced to accommodate and manage collaborations, but gradually developed to help with screening literature and reporting outcomes. Some of these software packages were initially designed for medical and healthcare studies and have specific protocols and customised steps integrated for various types of systematic reviews. However, some are designed for general processing, and by extending the application of the systematic review approach to other fields, they are being increasingly adopted and used in software engineering, health-related nutrition, agriculture, environmental science, social sciences and education.

Software tools

There are various free and subscription-based tools to help with conducting a systematic review. Many of these tools are designed to assist with the key stages of the process, including title and abstract screening, data synthesis, and critical appraisal. Some are designed to facilitate the entire process of review, including protocol development, reporting of the outcomes and help with fast project completion.

As time goes on, more functions are being integrated into such software tools. Technological advancement has allowed for more sophisticated and user-friendly features, including visual graphics for pattern recognition and linking multiple concepts. The idea is to digitalise the cumbersome parts of the process to increase efficiency, thus allowing researchers to focus their time and efforts on assessing the rigorousness and robustness of the research articles.

This article introduces commonly used systematic review tools that are relevant to food research and related disciplines, which can be used in a similar context to the process in healthcare disciplines.

These reviews are based on IFIS' internal research, thus are unbiased and not affiliated with the companies.

This online platform is a core component of the Cochrane toolkit, supporting parts of the systematic review process, including title/abstract and full-text screening, documentation, and reporting.

The Covidence platform enables collaboration of the entire systematic reviews team and is suitable for researchers and students at all levels of experience.

From a user perspective, the interface is intuitive, and the citation screening is directed step-by-step through a well-defined workflow. Imports and exports are straightforward, with easy export options to Excel and CVS.

Access is free for Cochrane authors (a single reviewer), and Cochrane provides a free trial to other researchers in healthcare. Universities can also subscribe on an institutional basis.

Rayyan is a free and open access web-based platform funded by the Qatar Foundation, a non-profit organisation supporting education and community development initiative . Rayyan is used to screen and code literature through a systematic review process.

Unlike Covidence, Rayyan does not follow a standard SR workflow and simply helps with citation screening. It is accessible through a mobile application with compatibility for offline screening. The web-based platform is known for its accessible user interface, with easy and clear export options.

Function comparison of 5 software tools to support the systematic review process

EPPI-Reviewer

EPPI-Reviewer is a web-based software programme developed by the Evidence for Policy and Practice Information and Co-ordinating Centre  (EPPI) at the UCL Institute for Education, London .

It provides comprehensive functionalities for coding and screening. Users can create different levels of coding in a code set tool for clustering, screening, and administration of documents. EPPI-Reviewer allows direct search and import from PubMed. The import of search results from other databases is feasible in different formats. It stores, references, identifies and removes duplicates automatically. EPPI-Reviewer allows full-text screening, text mining, meta-analysis and the export of data into different types of reports.

There is no limit for concurrent use of the software and the number of articles being reviewed. Cochrane reviewers can access EPPI reviews using their Cochrane subscription details.

EPPI-Centre has other tools for facilitating the systematic review process, including coding guidelines and data management tools.

CADIMA is a free, online, open access review management tool, developed to facilitate research synthesis and structure documentation of the outcomes.

The Julius Institute and the Collaboration for Environmental Evidence established the software programme to support and guide users through the entire systematic review process, including protocol development, literature searching, study selection, critical appraisal, and documentation of the outcomes. The flexibility in choosing the steps also makes CADIMA suitable for conducting systematic mapping and rapid reviews.

CADIMA was initially developed for research questions in agriculture and environment but it is not limited to these, and as such, can be used for managing review processes in other disciplines. It enables users to export files and work offline.

The software allows for statistical analysis of the collated data using the R statistical software. Unlike EPPI-Reviewer, CADIMA does not have a built-in search engine to allow for searching in literature databases like PubMed.

DistillerSR

DistillerSR is an online software maintained by the Canadian company, Evidence Partners which specialises in literature review automation. DistillerSR provides a collaborative platform for every stage of literature review management. The framework is flexible and can accommodate literature reviews of different sizes. It is configurable to different data curation procedures, workflows and reporting standards. The platform integrates necessary features for screening, quality assessment, data extraction and reporting. The software uses Artificial Learning (AL)-enabled technologies in priority screening. It is to cut the screening process short by reranking the most relevant references nearer to the top. It can also use AL, as a second reviewer, in quality control checks of screened studies by human reviewers. DistillerSR is used to manage systematic reviews in various medical disciplines, surveillance, pharmacovigilance and public health reviews including food and nutrition topics. The software does not support statistical analyses. It provides configurable forms in standard formats for data extraction.

DistillerSR allows direct search and import of references from PubMed. It provides an add on feature called LitConnect which can be set to automatically import newly published references from data providers to keep reviews up to date during their progress.

The systematic review Toolbox is a web-based catalogue of various tools, including software packages which can assist with single or multiple tasks within the evidence synthesis process. Researchers can run a quick search or tailor a more sophisticated search by choosing their approach, budget, discipline, and preferred support features, to find the right tools for their research.

If you enjoyed this blog post, you may also be interested in our recently published blog post addressing the difference between a systematic review and a systematic literature review.

• FSTA - Food Science & Technology Abstracts
• IFIS Collections
• Resources Hub
• Diversity Statement
• Sustainability Commitment
• Company news
• Frequently Asked Questions
• Privacy Policy
• Terms of Use for IFIS Collections

Ground Floor, 115 Wharfedale Road,  Winnersh Triangle, Wokingham, Berkshire RG41 5RB

Get in touch with IFIS

© International Food Information Service (IFIS Publishing) operating as IFIS – All Rights Reserved     |     Charity Reg. No. 1068176     |     Limited Company No. 3507902     |     Designed by Blend

10 Best Literature Review Tools for Researchers

Boost your research game with these Best Literature Review Tools for Researchers! Uncover hidden gems, organize your findings, and ace your next research paper!

Researchers struggle to identify key sources, extract relevant information, and maintain accuracy while manually conducting literature reviews. This leads to inefficiency, errors, and difficulty in identifying gaps or trends in existing literature.

Table of Contents

Top 10 Literature Review Tools for Researchers: In A Nutshell (2023)

#1. Semantic Scholar – A free, AI-powered research tool for scientific literature

Semantic Scholar is a cutting-edge literature review tool that researchers rely on for its comprehensive access to academic publications. With its advanced AI algorithms and extensive database, it simplifies the discovery of relevant research papers. 

Not all scholarly content may be indexed, and occasional false positives or inaccurate associations can occur. Furthermore, the tool primarily focuses on computer science and related fields, potentially limiting coverage in other disciplines. 

#2. Elicit – Research assistant using language models like GPT-3

However, users should be cautious when using Elicit. It is important to verify the credibility and accuracy of the sources found through the tool, as the database encompasses a wide range of publications. 

#3. Scite.Ai – Your personal research assistant

However, while Scite.Ai offers numerous advantages, there are a few aspects to be cautious about. As with any data-driven tool, occasional errors or inaccuracies may arise, necessitating researchers to cross-reference and verify results with other reputable sources. 

Rayyan offers the following paid plans:

#4. DistillerSR – Literature Review Software

Despite occasional technical glitches reported by some users, the developers actively address these issues through updates and improvements, ensuring a better user experience. 

#5. Rayyan – AI Powered Tool for Systematic Literature Reviews

However, it’s important to be aware of a few aspects. The free version of Rayyan has limitations, and upgrading to a premium subscription may be necessary for additional functionalities. 

#6. Consensus – Use AI to find you answers in scientific research

With Consensus, researchers can save significant time by efficiently organizing and accessing relevant research material.People consider Consensus for several reasons. 

Consensus offers both free and paid plans:

#7. RAx – AI-powered reading assistant

#8. lateral – advance your research with ai.

Additionally, researchers must be mindful of potential biases introduced by the tool’s algorithms and should critically evaluate and interpret the results. 

#9. Iris AI – Introducing the researcher workspace

Researchers are drawn to this tool because it saves valuable time by automating the tedious task of literature review and provides comprehensive coverage across multiple disciplines. 

#10. Scholarcy – Summarize your literature through AI

Scholarcy’s automated summarization may not capture the nuanced interpretations or contextual information presented in the full text. 

Final Thoughts

In conclusion, conducting a comprehensive literature review is a crucial aspect of any research project, and the availability of reliable and efficient tools can greatly facilitate this process for researchers. This article has explored the top 10 literature review tools that have gained popularity among researchers.

Q1. What are literature review tools for researchers?

Q2. what criteria should researchers consider when choosing literature review tools.

When choosing literature review tools, researchers should consider factors such as the tool’s search capabilities, database coverage, user interface, collaboration features, citation management, annotation and highlighting options, integration with reference management software, and data extraction capabilities. 

Q3. Are there any literature review tools specifically designed for systematic reviews or meta-analyses?

Meta-analysis support: Some literature review tools include statistical analysis features that assist in conducting meta-analyses. These features can help calculate effect sizes, perform statistical tests, and generate forest plots or other visual representations of the meta-analytic results.

Q4. Can literature review tools help with organizing and annotating collected references?

Integration with citation managers: Some literature review tools integrate with popular citation managers like Zotero, Mendeley, or EndNote, allowing seamless transfer of references and annotations between platforms.

Leave a Comment Cancel reply

Accelerate your research with the best systematic literature review tools

The ideal literature review tool helps you make sense of the most important insights in your research field. ATLAS.ti empowers researchers to perform powerful and collaborative analysis using the leading software for literature review.

Finalize your literature review faster with comfort

ATLAS.ti makes it easy to manage, organize, and analyze articles, PDFs, excerpts, and more for your projects. Conduct a deep systematic literature review and get the insights you need with a comprehensive toolset built specifically for your research projects.

Figure out the "why" behind your participant's motivations

Understand the behaviors and emotions that are driving your focus group participants. With ATLAS.ti, you can transform your raw data and turn it into qualitative insights you can learn from. Easily determine user intent in the same spot you're deciphering your overall focus group data.

Visualize your research findings like never before

We make it simple to present your analysis results with meaningful charts, networks, and diagrams. Instead of figuring out how to communicate the insights you just unlocked, we enable you to leverage easy-to-use visualizations that support your goals.

Everything you need to elevate your literature review

Import and organize literature data.

Import and analyze any type of text content – ATLAS.ti supports all standard text and transcription files such as Word and PDF.

Analyze with ease and speed

Utilize easy-to-learn workflows that save valuable time, such as auto coding, sentiment analysis, team collaboration, and more.

Leverage AI-driven tools

Make efficiency a priority and let ATLAS.ti do your work with AI-powered research tools and features for faster results.

Visualize and present findings

With just a few clicks, you can create meaningful visualizations like charts, word clouds, tables, networks, among others for your literature data.

The faster way to make sense of your literature review. Try it for free, today.

A literature review analyzes the most current research within a research area. A literature review consists of published studies from many sources:

• Peer-reviewed academic publications
• Full-length books
• University bulletins
• Conference proceedings
• Dissertations and theses

Literature reviews allow researchers to:

• Summarize the state of the research
• Identify unexplored research inquiries
• Recommend practical applications
• Critique currently published research

Literature reviews are either standalone publications or part of a paper as background for an original research project. A literature review, as a section of a more extensive research article, summarizes the current state of the research to justify the primary research described in the paper.

For example, a researcher may have reviewed the literature on a new supplement's health benefits and concluded that more research needs to be conducted on those with a particular condition. This research gap warrants a study examining how this understudied population reacted to the supplement. Researchers need to establish this research gap through a literature review to persuade journal editors and reviewers of the value of their research.

Consider a literature review as a typical research publication presenting a study, its results, and the salient points scholars can infer from the study. The only significant difference with a literature review treats existing literature as the research data to collect and analyze. From that analysis, a literature review can suggest new inquiries to pursue.

Identify a focus

Similar to a typical study, a literature review should have a research question or questions that analysis can answer. This sort of inquiry typically targets a particular phenomenon, population, or even research method to examine how different studies have looked at the same thing differently. A literature review, then, should center the literature collection around that focus.

Collect and analyze the literature

With a focus in mind, a researcher can collect studies that provide relevant information for that focus. They can then analyze the collected studies by finding and identifying patterns or themes that occur frequently. This analysis allows the researcher to point out what the field has frequently explored or, on the other hand, overlooked.

Suggest implications

The literature review allows the researcher to argue a particular point through the evidence provided by the analysis. For example, suppose the analysis makes it apparent that the published research on people's sleep patterns has not adequately explored the connection between sleep and a particular factor (e.g., television-watching habits, indoor air quality). In that case, the researcher can argue that further study can address this research gap.

External requirements aside (e.g., many academic journals have a word limit of 6,000-8,000 words), a literature review as a standalone publication is as long as necessary to allow readers to understand the current state of the field. Even if it is just a section in a larger paper, a literature review is long enough to allow the researcher to justify the study that is the paper's focus.

Note that a literature review needs only to incorporate a representative number of studies relevant to the research inquiry. For term papers in university courses, 10 to 20 references might be appropriate for demonstrating analytical skills. Published literature reviews in peer-reviewed journals might have 40 to 50 references. One of the essential goals of a literature review is to persuade readers that you have analyzed a representative segment of the research you are reviewing.

Researchers can find published research from various online sources:

• Journal websites
• Research databases
• Search engines (Google Scholar, Semantic Scholar)
• Research repositories
• Social networking sites (Academia, ResearchGate)

Many journals make articles freely available under the term "open access," meaning that there are no restrictions to viewing and downloading such articles. Otherwise, collecting research articles from restricted journals usually requires access from an institution such as a university or a library.

Evidence of a rigorous literature review is more important than the word count or the number of articles that undergo data analysis. Especially when writing for a peer-reviewed journal, it is essential to consider how to demonstrate research rigor in your literature review to persuade reviewers of its scholarly value.

Select field-specific journals

The most significant research relevant to your field focuses on a narrow set of journals similar in aims and scope. Consider who the most prominent scholars in your field are and determine which journals publish their research or have them as editors or reviewers. Journals tend to look favorably on systematic reviews that include articles they have published.

Incorporate recent research

Recently published studies have greater value in determining the gaps in the current state of research. Older research is likely to have encountered challenges and critiques that may render their findings outdated or refuted. What counts as recent differs by field; start by looking for research published within the last three years and gradually expand to older research when you need to collect more articles for your review.

Consider the quality of the research

Literature reviews are only as strong as the quality of the studies that the researcher collects. You can judge any particular study by many factors, including:

• the quality of the article's journal
• the article's research rigor
• the timeliness of the research

The critical point here is that you should consider more than just a study's findings or research outputs when including research in your literature review.

Narrow your research focus

Ideally, the articles you collect for your literature review have something in common, such as a research method or research context. For example, if you are conducting a literature review about teaching practices in high school contexts, it is best to narrow your literature search to studies focusing on high school. You should consider expanding your search to junior high school and university contexts only when there are not enough studies that match your focus.

You can create a project in ATLAS.ti for keeping track of your collected literature. ATLAS.ti allows you to view and analyze full text articles and PDF files in a single project. Within projects, you can use document groups to separate studies into different categories for easier and faster analysis.

For example, a researcher with a literature review that examines studies across different countries can create document groups labeled "United Kingdom," "Germany," and "United States," among others. A researcher can also use ATLAS.ti's global filters to narrow analysis to a particular set of studies and gain insights about a smaller set of literature.

ATLAS.ti allows you to search, code, and analyze text documents and PDF files. You can treat a set of research articles like other forms of qualitative data. The codes you apply to your literature collection allow for analysis through many powerful tools in ATLAS.ti:

• Code Co-Occurrence Explorer
• Code Co-Occurrence Table
• Code-Document Table

Other tools in ATLAS.ti employ machine learning to facilitate parts of the coding process for you. Some of our software tools that are effective for analyzing literature include:

• Named Entity Recognition
• Opinion Mining
• Sentiment Analysis

As long as your documents are text documents or text-enable PDF files, ATLAS.ti's automated tools can provide essential assistance in the data analysis process.

• Resources Home 🏠
• Try SciSpace Copilot
• Search research papers
• Add Copilot Extension
• Try AI Detector
• Try Paraphraser
• Try Citation Generator
• April Papers
• June Papers
• July Papers

5 literature review tools to ace your research (+2 bonus tools)

Table of Contents

Your literature review is the lore behind your research paper . It comes in two forms, systematic and scoping , both serving the purpose of rounding up previously published works in your research area that led you to write and finish your own.

A literature review is vital as it provides the reader with a critical overview of the existing body of knowledge, your methodology, and an opportunity for research applications.

Some steps to follow while writing your review:

• Pick an accessible topic for your paper
• Do thorough research and gather evidence surrounding your topic
• Read and take notes diligently
• Create a rough structure for your review
• Synthesis your notes and write the first draft
• Edit and proofread your literature review

To make your workload a little lighter, there are many literature review AI tools. These tools can help you find academic articles through AI and answer questions about a research paper.  

Best literature review tools to improve research workflow

A literature review is one of the most critical yet tedious stages in composing a research paper. Many students find it an uphill task since it requires extensive reading and careful organization .

Using some of the best literature review tools listed here, you can make your life easier by overcoming some of the existing challenges in literature reviews. From collecting and classifying to analyzing and publishing research outputs, these tools help you with your literature review and improve your productivity without additional effort or expenses.

1. SciSpace

SciSpace is an AI for academic research that will help find research papers and answer questions about a research paper. You can discover, read, and understand research papers with SciSpace making it an excellent platform for literature review. Featuring a repository with over 270 million research papers, it comes with your AI research assistant called Copilot that offers explanations, summaries , and answers as you read.

Get started now:

Find academic articles through AI

SciSpace has a dedicated literature review tool that finds scientific articles when you search for a question. Based on semantic search, it shows all the research papers relevant for your subject. You can then gather quick insights for all the papers displayed in your search results like methodology, dataset, etc., and figure out all the papers relevant for your research.

Identify relevant articles faster

Abstracts are not always enough to determine whether a paper is relevant to your research question. For starters, you can ask questions to your AI research assistant, SciSpace Copilot to explore the content and better understand the article. Additionally, use the summarize feature to quickly review the methodology and results of a paper and decide if it is worth reading in detail.

Learn in your preferred language

A big barrier non-native English speakers face while conducting a literature review is that a significant portion of scientific literature is published in English. But with SciSpace Copilot, you can review, interact, and learn from research papers in any language you prefer — presently, it supports 75+ languages. The AI will answer questions about a research paper in your mother tongue.

Integrates with Zotero

Many researchers use Zotero to create a library and manage research papers. SciSpace lets you import your scientific articles directly from Zotero into your SciSpace library and use Copilot to comprehend your research papers. You can also highlight key sections, add notes to the PDF as you read, and even turn helpful explanations and answers from Copilot into notes for future review.

Understand math and complex concepts quickly

Come across complex mathematical equations or difficult concepts? Simply highlight the text or select the formula or table, and Copilot will provide an explanation or breakdown of the same in an easy-to-understand manner. You can ask follow-up questions if you need further clarification.

Discover new papers to read without leaving

Highlight phrases or sentences in your research paper to get suggestions for related papers in the field and save time on literature reviews. You can also use the 'Trace' feature to move across and discover connected papers, authors, topics, and more.

SciSpace Copilot is now available as a Chrome extension , allowing you to access its features directly while you browse scientific literature anywhere across the web.

Get citation-backed answers

When you're conducting a literature review, you want credible information with proper references.  Copilot ensures that every piece of information provided by SciSpace Copilot is backed by a direct reference, boosting transparency, accuracy, and trustworthiness.

Ask a question related to the paper you're delving into. Every response from Copilot comes with a clickable citation. This citation leads you straight to the section of the PDF from which the answer was extracted.

By seamlessly integrating answers with citations, SciSpace Copilot assures you of the authenticity and relevance of the information you receive.

2. Mendeley

Mendeley Citation Manager is a free web and desktop application. It helps simplify your citation management workflow significantly. Here are some ways you can speed up your referencing game with Mendeley.

Generate citations and bibliographies

Easily add references from your Mendeley library to your Word document, change your citation style, and create a bibliography, all without leaving your document.

Retrieve references

It allows you to access your references quickly. Search for a term, and it will return results by referencing the year, author, or source.

Add sources to your Mendeley library by dragging PDF to Mendeley Reference Manager. Mendeley will automatically remove the PDF(s) metadata and create a library entry.‌

Read and annotate documents

It helps you highlight and comment across multiple PDFs while keep them all in one place using Mendeley Notebook . Notebook pages are not tied to a reference and let you quote from many PDFs.

A big part of many literature review workflows, Zotero is a free, open-source tool for managing citations that works as a plug-in on your browser. It helps you gather the information you need, cite your sources, lets you attach PDFs, notes, and images to your citations, and create bibliographies.

Import research articles to your database

Search for research articles on a keyword, and add relevant results to your database. Then, select the articles you are most interested in, and import them into Zotero.

Add bibliography in a variety of formats

With Zotero, you don’t have to scramble for different bibliography formats. Simply use the Zotero-Word plug-in to insert in-text citations and generate a bibliography.

Share your research

You can save a paper and sync it with an online library to easily share your research for group projects. Zotero can be used to create your database and decrease the time you spend formatting citations.

Sysrev is an AI too for article review that facilitates screening, collaboration, and data extraction from academic publications, abstracts, and PDF documents using machine learning. The platform is free and supports public and Open Access projects only.

Some of the features of Sysrev include:

Group labels

Group labels can be a powerful concept for creating database tables from documents. When exported and re-imported, each group label creates a new table. To make labels for a project, go into the manage -> labels section of the project.

Group labels enable project managers to pull table information from documents. It makes it easier to communicate review results for specific articles.

Track reviewer performance

Sysrev's label counting tool provides filtering and visualization options for keeping track of the distribution of labels throughout the project's progress. Project managers can check their projects at any point to track progress and the reviewer's performance.

Tool for concordance

The Sysrev tool for concordance allows project administrators and reviewers to perform analysis on their labels. Concordance is measured by calculating the number of times users agree on the labels they have extracted.

Colandr is a free, open-source, internet-based analysis and screening software used as an AI for academic research. It was designed to ease collaboration across various stages of the systematic review procedure. The tool can be a little complex to use. So, here are the steps involved in working with Colandr.

Create a review

The first step to using Colandr is setting up an organized review project. This is helpful to librarians who are assisting researchers with systematic reviews.

The planning stage is setting the review's objectives along with research queries. Any reviewer can review the details of the planning stage. However, they can only be modified by the author for the review.

Citation screening/import

In this phase, users can upload their results from database searches. Colandr also offers an automated deduplication system.

Full-text screening

The system in Colandr will discover the combination of terms and expressions that are most useful for the reader. If an article is selected, it will be moved to the final step.

Data extraction/export

Colandr data extraction is more efficient than the manual method. It creates the form fields for data extraction during the planning stage of the review procedure. Users can decide to revisit or modify the form for data extraction after completing the initial screening.

Bonus literature review tools

SRDR+ is a web-based tool for extracting and managing systematic review or meta-analysis data. It is open and has a searchable archive of systematic reviews and their data.

7. Plot Digitizer

Plot Digitizer is an efficient tool for extracting information from graphs and images, equipped with many features that facilitate data extraction. The program comes with a free online application, which is adequate to extract data quickly.

Final thoughts

Writing a literature review is not easy. It’s a time-consuming process, which can become tiring at times. The literature review tools mentioned in this blog do an excellent job of maximizing your efforts and helping you write literature reviews much more efficiently. With them, you can breathe a sigh of relief and give more time to your research.

As you dive into your literature review, don’t forget to use SciSpace ResearchGPT to streamline the process. It facilitates your research and helps you explore key findings, summary, and other components of the paper easily.

Frequently Asked Questions (FAQs)

1. what is rrl in research.

RRL stands for Review of Related Literature and sometimes interchanged with ‘Literature Review.’ RRL is a body of studies relevant to the topic being researched. These studies may be in the form of journal articles, books, reports, and other similar documents. Review of related literature is used to support an argument or theory being made by the researcher, as well as to provide information on how others have approached the same topic.

2. What are few softwares and tools available for literature review?

• SciSpace Discover

• Mendeley

• Zotero

• Sysrev

• Colandr

• SRDR+

3. How to generate an online literature review?

The Scispace Discover tool, which offers an excellent repository of millions of peer-reviewed articles and resources, will help you generate or create a literature review easily. You may find relevant information by utilizing the filter option, checking its credibility, tracing related topics and articles, and citing in widely accepted formats with a single click.

4. What does it mean to synthesize literature?

To synthesize literature is to take the main points and ideas from a number of sources and present them in a new way. The goal is to create a new piece of writing that pulls together the most important elements of all the sources you read. Make recommendations based on them, and connect them to the research.

5. Should we write abstract for literature review?

Abstracts, particularly for the literature review section, are not required. However, an abstract for the research paper, on the whole, is useful for summarizing the paper and letting readers know what to expect from it. It can also be used to summarize the main points of the paper so that readers have a better understanding of the paper's content before they read it.

6. How do you evaluate the quality of a literature review?

• Whether it is clear and well-written.

• Whether Information is current and up to date.

• Does it cover all of the relevant sources on the topic.

• Does it provide enough evidence to support its conclusions.

7. Is literature review mandatory?

Yes. Literature review is a mandatory part of any research project. It is a critical step in the process that allows you to establish the scope of your research and provide a background for the rest of your work.

8. What are the sources for a literature review?

• Reports

• Theses

• Conference proceedings

• Company reports

• Some government publications

• Journals

• Books

• Newspapers

• Articles by professional associations

• Indexes

• Databases

• Catalogues

• Encyclopaedias

• Dictionaries

• Bibliographies

• Citation indexes

• Statistical data from government websites

9. What is the difference between a systematic review and a literature review?

A systematic review is a form of research that uses a rigorous method to generate knowledge from both published and unpublished data. A literature review, on the other hand, is a critical summary of an area of research within the context of what has already been published.

Suggested reads!

Types of essays in academic writing Citation Machine Alternatives — A comparison of top citation tools 2023

QuillBot vs SciSpace: Choose the best AI-paraphrasing tool

ChatPDF vs. SciSpace Copilot: Unveiling the best tool for your research

You might also like

Consensus GPT vs. SciSpace GPT: Choose the Best GPT for Research

Literature Review and Theoretical Framework: Understanding the Differences

Types of Essays in Academic Writing - Quick Guide (2024)

• Help Center

GET STARTED

COLLABORATE ON YOUR REVIEWS WITH ANYONE, ANYWHERE, ANYTIME

Save precious time and maximize your productivity with a Rayyan membership. Receive training, priority support, and access features to complete your systematic reviews efficiently.

Rayyan Teams+ makes your job easier. It includes VIP Support, AI-powered in-app help, and powerful tools to create, share and organize systematic reviews, review teams, searches, and full-texts.

RESEARCHERS

Rayyan makes collaborative systematic reviews faster, easier, and more convenient. Training, VIP support, and access to new features maximize your productivity. Get started now!

Over 1 billion reference articles reviewed by research teams, and counting...

Intelligent, scalable and intuitive.

Rayyan understands language, learns from your decisions and helps you work quickly through even your largest systematic literature reviews.

WATCH A TUTORIAL NOW

Solutions for Organizations and Businesses

Rayyan Enterprise and Rayyan Teams+ make it faster, easier and more convenient for you to manage your research process across your organization.

• Accelerate your research across your team or organization and save valuable researcher time.
• Build and preserve institutional assets, including literature searches, systematic reviews, and full-text articles.
• Onboard team members quickly with access to group trainings for beginners and experts.
• Receive priority support to stay productive when questions arise.
• SCHEDULE A DEMO
• LEARN MORE ABOUT RAYYAN TEAMS+

RAYYAN SYSTEMATIC LITERATURE REVIEW OVERVIEW

LEARN ABOUT RAYYAN’S PICO HIGHLIGHTS AND FILTERS

Join now to learn why Rayyan is trusted by already more than 500,000 researchers

Individual plans, teams plans.

For early career researchers just getting started with research.

Free forever

• 3 Active Reviews
• Invite Unlimited Reviewers
• Import Directly from Mendeley
• Industry Leading De-Duplication
• 5-Star Relevance Ranking
• Advanced Filtration Facets
• Mobile App Access
• 100 Decisions on Mobile App
• Standard Support
• Revoke Reviewer
• Online Training
• PICO Highlights & Filters
• PRISMA (Beta)
• Auto-Resolver 
• Multiple Teams & Management Roles
• Monitor & Manage Users, Searches, Reviews, Full Texts
• Onboarding and Regular Training

Professional

For researchers who want more tools for research acceleration.

per month, billed annually

• Unlimited Active Reviews
• Unlimited Decisions on Mobile App
• Priority Support
• Auto-Resolver

For currently enrolled students with valid student ID.

per month, billed quarterly

For a team that wants professional licenses for all members.

per month, per user, billed annually

• Single Team
• High Priority Support

For teams that want support and advanced tools for members.

• Multiple Teams
• Management Roles

For organizations who want access to all of their members.

Annual Subscription

Contact Sales

• Organizational Ownership
• For an organization or a company
• Access to all the premium features such as PICO Filters, Auto-Resolver, PRISMA and Mobile App
• Store and Reuse Searches and Full Texts
• A management console to view, organize and manage users, teams, review projects, searches and full texts
• Highest tier of support – Support via email, chat and AI-powered in-app help
• GDPR Compliant
• Single Sign-On
• API Integration
• Training for Experts
• Training Sessions Students Each Semester
• More options for secure access control

———————–

ANNUAL ONLY

Rayyan Subscription

membership starts with 2 users. You can select the number of additional members that you’d like to add to your membership.

Total amount:

Click Proceed to get started.

Great usability and functionality. Rayyan has saved me countless hours. I even received timely feedback from staff when I did not understand the capabilities of the system, and was pleasantly surprised with the time they dedicated to my problem. Thanks again!

This is a great piece of software. It has made the independent viewing process so much quicker. The whole thing is very intuitive.

Rayyan makes ordering articles and extracting data very easy. A great tool for undertaking literature and systematic reviews!

Excellent interface to do title and abstract screening. Also helps to keep a track on the the reasons for exclusion from the review. That too in a blinded manner.

Rayyan is a fantastic tool to save time and improve systematic reviews!!! It has changed my life as a researcher!!! thanks

Easy to use, friendly, has everything you need for cooperative work on the systematic review.

Rayyan makes life easy in every way when conducting a systematic review and it is easy to use.

Literature Review with MAXQDA

Interview transcription examples, make the most out of your literature review.

Literature reviews are an important step in the data analysis journey of many research projects, but often it is a time-consuming and arduous affair. Whether you are reviewing literature for writing a meta-analysis or for the background section of your thesis, work with MAXQDA. Our product comes with many exciting features which make your literature review faster and easier than ever before. Whether you are a first-time researcher or an old pro, MAXQDA is your professional software solution with advanced tools for you and your team.

How to conduct a literature review with MAXQDA

Conducting a literature review with MAXQDA is easy because you can easily import bibliographic information and full texts. In addition, MAXQDA provides excellent tools to facilitate each phase of your literature review, such as notes, paraphrases, auto-coding, summaries, and tools to integrate your findings.

Step one: Plan your literature review

Similar to other research projects, one should carefully plan a literature review. Before getting started with searching and analyzing literature, carefully think about the purpose of your literature review and the questions you want to answer. This will help you to develop a search strategy which is needed to stay on top of things. A search strategy involves deciding on literature databases, search terms, and practical and methodological criteria for the selection of high-quality scientific literature.

MAXQDA supports you during this stage with memos and the newly developed Questions-Themes-Theories tool (QTT). Both are the ideal place to store your research questions and search parameters. Moreover, the Question-Themes-Theories tool is perfectly suited to support your literature review project because it provides a bridge between your MAXQDA project and your research report. It offers the perfect enviornment to bring together findings, record conclusions and develop theories.

Step two: Search, Select, Save your material

Follow your search strategy. Use the databases and search terms you have identified to find the literature you need. Then, scan the search results for relevance by reading the title, abstract, or keywords. Try to determine whether the paper falls within the narrower area of the research question and whether it fulfills the objectives of the review. In addition, check whether the search results fulfill your pre-specified eligibility criteria. As this step typically requires precise reading rather than a quick scan, you might want to perform it in MAXQDA. If the piece of literature fulfills your criteria and context, you can save the bibliographic information using a reference management system which is a common approach among researchers as these programs automatically extract a paper’s meta-data from the publishing website. You can easily import this bibliographic data into MAXQDA via a specialized import tool. MAXQDA is compatible with all reference management programs that are able to export their literature databases in RIS format which is a standard format for bibliographic information. This is the case with all mainstream literature management programs such as Citavi, DocEar, Endnote, JabRef, Mendeley, and Zotero.

Step three: Import & Organize your material in MAXQDA

Importing bibliographic data into MAXQDA is easy and works seamlessly for all reference management programs that use the standard RIS files. MAXQDA offers an import option dedicated to bibliographic data which you can find in the MAXQDA Import tab. To import the selected literature, just click on the corresponding button, select the data you want to import, and click okay. Upon import, each literature entry becomes its own text document. If full texts are imported, MAXQDA automatically connects the full text to the literature entry with an internal link. The individual information in the literature entries is automatically coded for later analysis so that, for example, all titles or abstracts can be compiled and searched. To help you keeping your literature (review) organized, MAXQDA automatically creates a document group called “References” which contains the individual literature entries. Like full texts or interview documents, the bibliographic entries can be searched, coded, linked, edited, and you can add memos for further qualitative and quantitative content analysis (Kuckartz & Rädiker, 2019). Especially, when running multiple searches using different databases or search terms, you should carefully document your approach. Besides being a great place to store the respective search parameters, memos are perfectly suited to capture your ideas while reviewing our literature and can be attached to text segments, documents, document groups, and much more.

Analyze your literature with MAXQDA

Once imported into MAXQDA, you can explore your material using a variety of tools and functions. With MAXQDA as your literature review & analysis software, you have numerous possibilities for analyzing your literature and writing your literature review – impossible to mention all. Thus, we can present only a subset of tools here. Check out our literature about performing literature reviews with MAXQDA to discover more possibilities.

AI Assist: Introducing AI to literature reviews

AI Assist – MAXQDA’s AI-based add-on module – can simplify your literature reviews in many ways. Chat with your data and ask the AI questions about your documents. Let AI Assist automatically summarize entire papers and text segments. Automatically create summaries of all coded segments of a code or generate suggestions for subcodes, and if you don’t know a word’s or concept’s meaning, use AI Assist to get a definition without leaving MAXQDA. Visit our research guide for even more ideas on how AI can support your literature review:

AI for Literature Review

Code & Retrieve important segments

Coding qualitative data lies at the heart of many qualitative data analysis approaches and can be useful for literature reviews as well. Coding refers to the process of labeling segments of your material. For example, you may want to code definitions of certain terms, pro and con arguments, how a specific method is used, and so on. In a later step, MAXQDA allows you to compile all text segments coded with one (or more) codes of interest from one or more papers, so that you can for example compare definitions across papers.

But there is more. MAXQDA offers multiple ways of coding, such as in-vivo coding, highlighters, emoticodes, Creative Coding, or the Smart Coding Tool. The compiled segments can be enriched with variables and the segment’s context accessed with just one click. MAXQDA’s Text Search & Autocode tool is especially well-suited for a literature review, as it allows one to explore large amounts of text without reading or coding them first. Automatically search for keywords (or dictionaries of keywords), such as important concepts for your literature review, and automatically code them with just a few clicks.

Paraphrase literature into your own words

Another approach is to paraphrase the existing literature. A paraphrase is a restatement of a text or passage in your own words, while retaining the meaning and the main ideas of the original. Paraphrasing can be especially helpful in the context of literature reviews, because paraphrases force you to systematically summarize the most important statements (and only the most important statements) which can help to stay on top of things.

With MAXQDA as your literature review software, you not only have a tool for paraphrasing literature but also tools to analyze the paraphrases you have written. For example, the Categorize Paraphrases tool (allows you to code your parpahrases) or the Paraphrases Matrix (allows you to compare paraphrases side-by-side between individual documents or groups of documents.)

Summaries & Overview tables: A look at the Bigger Picture

When conducting a literature review you can easily get lost. But with MAXQDA as your literature review software, you will never lose track of the bigger picture. Among other tools, MAXQDA’s overview and summary tables are especially useful for aggregating your literature review results. MAXQDA offers overview tables for almost everything, codes, memos, coded segments, links, and so on. With MAXQDA literature review tools you can create compressed summaries of sources that can be effectively compared and represented, and with just one click you can easily export your overview and summary tables and integrate them into your literature review report.

Visualize your qualitative data

The proverb “a picture is worth a thousand words” also applies to literature reviews. That is why MAXQDA offers a variety of Visual Tools that allow you to get a quick overview of the data, and help you to identify patterns. Of course, you can export your visualizations in various formats to enrich your final report. One particularly useful visual tool for literature reviews is the Word Cloud. It visualizes the most frequent words and allows you to explore key terms and the central themes of one or more papers. Thanks to the interactive connection between your visualizations with your MAXQDA data, you will never lose sight of the big picture. Another particularly useful tool is MAXQDA’s word/code frequency tool with which you can analyze and visualize the frequencies of words or codes in one or more documents. As with Word Clouds, nonsensical words can be added to the stop list and excluded from the analysis.

QTT: Synthesize your results and write up the review

MAXQDA has an innovative workspace to gather important visualization, notes, segments, and other analytics results. The perfect tool to organize your thoughts and data. Create a separate worksheet for your topics and research questions, fill it with associated analysis elements from MAXQDA, and add your conclusions, theories, and insights as you go. For example, you can add Word Clouds, important coded segments, and your literature summaries and write down your insights. Subsequently, you can view all analysis elements and insights to write your final conclusion. The Questions-Themes-Theories tool is perfectly suited to help you finalize your literature review reports. With just one click you can export your worksheet and use it as a starting point for your literature review report.

Literature about Literature Reviews and Analysis

We offer a variety of free learning materials to help you get started with your literature review. Check out our Getting Started Guide to get a quick overview of MAXQDA and step-by-step instructions on setting up your software and creating your first project with your brand new QDA software. In addition, the free Literature Reviews Guide explains how to conduct a literature review with MAXQDA in more detail.

Getting Started with MAXQDA

Literature Reviews with MAXQDA

A literature review is a critical analysis and summary of existing research and literature on a particular topic or research question. It involves systematically searching and evaluating a range of sources, such as books, academic journals, conference proceedings, and other published or unpublished works, to identify and analyze the relevant findings, methodologies, theories, and arguments related to the research question or topic.

A literature review’s purpose is to provide a comprehensive and critical overview of the current state of knowledge and understanding of a topic, to identify gaps and inconsistencies in existing research, and to highlight areas where further research is needed. Literature reviews are commonly used in academic research, as they provide a framework for developing new research and help to situate the research within the broader context of existing knowledge.

A literature review is a critical evaluation of existing research on a particular topic and is part of almost every research project. The literature review’s purpose is to identify gaps in current knowledge, synthesize existing research findings, and provide a foundation for further research. Over the years, numerous types of literature reviews have emerged. To empower you in coming to an informed decision, we briefly present the most common literature review methods.

• Narrative Review : A narrative review summarizes and synthesizes the existing literature on a particular topic in a narrative or story-like format. This type of review is often used to provide an overview of the current state of knowledge on a topic, for example in scientific papers or final theses.
• Systematic Review : A systematic review is a comprehensive and structured approach to reviewing the literature on a particular topic with the aim of answering a defined research question. It involves a systematic search of the literature using pre-specified eligibility criteria and a structured evaluation of the quality of the research.
• Meta-Analysis : A meta-analysis is a type of systematic review that uses statistical techniques to combine and analyze the results from multiple studies on the same topic. The goal of a meta-analysis is to provide a more robust and reliable estimate of the effect size than can be obtained from any single study.
• Scoping Review : A scoping review is a type of systematic review that aims to map the existing literature on a particular topic in order to identify the scope and nature of the research that has been done. It is often used to identify gaps in the literature and inform future research.

There is no “best” way to do a literature review, as the process can vary depending on the research question, field of study, and personal preferences. However, here are some general guidelines that can help to ensure that your literature review is comprehensive and effective:

• Carefully plan your literature review : Before you start searching and analyzing literature you should define a research question and develop a search strategy (for example identify relevant databases, and search terms). A clearly defined research question and search strategy will help you to focus your search and ensure that you are gathering relevant information. MAXQDA’s Questions-Themes-Theories tool is the perfect place to store your analysis plan.
• Evaluate your sources : Screen your search results for relevance to your research question, for example by reading abstracts. Once you have identified relevant sources, read them critically and evaluate their quality and relevance to your research question. Consider factors such as the methodology used, the reliability of the data, and the overall strength of the argument presented.
• Synthesize your findings : After evaluating your sources, synthesize your findings by identifying common themes, arguments, and gaps in the existing research. This will help you to develop a comprehensive understanding of the current state of knowledge on your topic.
• Write up your review : Finally, write up your literature review, ensuring that it is well-structured and clearly communicates your findings. Include a critical analysis of the sources you have reviewed, and use evidence from the literature to support your arguments and conclusions.

Overall, the key to a successful literature review is to be systematic, critical, and comprehensive in your search and evaluation of sources.

As in all aspects of scientific work, preparation is the key to success. Carefully think about the purpose of your literature review, the questions you want to answer, and your search strategy. The writing process itself will differ depending on the your literature review method. For example, when writing a narrative review use the identified literature to support your arguments, approach, and conclusions. By contrast, a systematic review typically contains the same parts as other scientific papers: Abstract, Introduction (purpose and scope), Methods (Search strategy, inclusion/exclusion characteristics, …), Results (identified sources, their main arguments, findings, …), Discussion (critical analysis of the sources you have reviewed), Conclusion (gaps or inconsistencies in the existing research, future research, implications, etc.).

Start your free trial

Your trial will end automatically after 14 days and will not renew. There is no need for cancelation.

Advanced Literature Review Software

Synthesis provides advanced literature review software with analytical and automation functionality for delivering timely evidence-based information in hours, not months, for better decisions.

Strategic Analysis

Perform Scoping and Systematic Reviews quickly and accurately using the latest automation and information management algorithms.

Reference Management

Synthesis organizes and manages all your references and PDFs. You can then quickly search the Abstract and Full-Text PDFs for keywords and phrases.

Advanced Analytics

Quickly summarize the reference by searching and tagging for keywords, preform topic clustering or word clouds on the literature, and then graph all your data.

Multiple Databases

PubMed, PubMed Central, IEEE, US Patents, Ovid (Medline, Embase, Global Health), Web of Science, Scopus, ProQuest, and many others..

Distribution

Export capabilities for sharing the Knowledge that you have just created as either CSV files or for importing into Cite and Write managers.

Internationally Recognized

Synthesis is used in academic research universities, hospitals, government agencies, private corporations and non-governmental organziations throughout the world.

Synthesis applies the latest in automation and enhanced analytic functionality for improving the efficiency and effectiveness of conducting literature reviews...

How to get started

Explore the features of Synthesis to see what truly sets it apart from other approaches for managing and analyzing the academic and business literature.

Synthesis provides online embedded searching on major bibliographical databases, validated automated de-duplication of references, automated importing of PDFs, methods to analyze the literature, and many more features.

Synthesis is available for Windows, Macintosh, Linux and as a Java application that can be run on any platform.

Find out more

I want to have Access to the latest Literature in the Fastest Possible way and Quickly Assess it. Physician

We need Systems with Automation and Artificial Intelligence that Allows Literature Reviews to be conducted quickly and efficiently. Academic Researcher

We need a Computer System for Healthcare that Puts the Information at My Finger Tips and Tells Me Everything I Need to Know. Hospital Administrator

We need Information Systems that Aren't Based in 1970s Technology Medical Student

Keep Up to Date about Synthesis

Synthesis research inc..

Synthesis Research Inc is a software development company focused on improving the way that literature is managed and analyzed. This desire is based around the goal of providing the best synthesized knowledge for supporting evidence-based decision making.

Synthesis Research Inc applies the latest computer science algorithms based around automation and information retrieval and management for improving the efficiency and effectiveness of conducting literature reviews through automating manual processes and enhancing the workflow.

Covidence website will be inaccessible as we upgrading our platform on Monday 23rd August at 10am AEST, / 2am CEST/1am BST (Sunday, 15th August 8pm EDT/5pm PDT) 

The World's #1 Systematic Review Tool

See your systematic reviews like never before

Faster reviews.

An average 35% reduction in time spent per review, saving an average of 71 hours per review.

Expert, online support

Easy to learn and use, with 24/7 support from product experts who are also seasoned reviewers!

Seamless collaboration

Enable the whole review team to collaborate from anywhere.

Suits all levels of experience and sectors

Suitable for reviewers in a variety of sectors including health, education, social science and many others.

Supporting the world's largest systematic review community

See how it works.

Step inside Covidence to see a more intuitive, streamlined way to manage systematic reviews.

Unlimited use for every organization

With no restrictions on reviews and users, Covidence gets out of the way so you can bring the best evidence to the world, more quickly.

Covidence is used by world-leading evidence organizations

Whether you’re an academic institution, a hospital or society, Covidence is working for organizations like yours right now.

See a list of organizations already using Covidence →

How Covidence has enabled living guidelines for Australians impacted by stroke

Clinical guidelines took 7 years to update prior to moving to a living evidence approach. Learn how Covidence streamlined workflows and created real time savings for the guidelines team.

University of Ottawa Drives Systematic Review Excellence Across Many Academic Disciplines

University of Ottawa

Top Ranked U.S. Teaching Hospital Delivers Effective Systematic Review Management

Top Ranked U.S. Teaching Hospital

See more Case Studies

Better systematic review management

Head office, working for an institution or organisation.

Find out why over 350 of the world’s leading institutions are seeing a surge in publications since using Covidence!

Request a consultation with one of our team members and start empowering your researchers:

By using our site you consent to our use of cookies to measure and improve our site’s performance. Please see our Privacy Policy for more information. 

7 open source tools to make literature reviews easy

Opensource.com

A good literature review is critical for academic research in any field, whether it is for a research article, a critical review for coursework, or a dissertation. In a recent article, I presented detailed steps for doing  a literature review using open source software .

The following is a brief summary of seven free and open source software tools described in that article that will make your next literature review much easier.

1. GNU Linux

Most literature reviews are accomplished by graduate students working in research labs in universities. For absurd reasons, graduate students often have the worst computers on campus. They are often old, slow, and clunky Windows machines that have been discarded and recycled from the undergraduate computer labs. Installing a flavor of GNU Linux will breathe new life into these outdated PCs. There are more than 100 distributions , all of which can be downloaded and installed for free on computers. Most popular Linux distributions come with a "try-before-you-buy" feature. For example, with Ubuntu you can make a bootable USB stick that allows you to test-run the Ubuntu desktop experience without interfering in any way with your PC configuration. If you like the experience, you can use the stick to install Ubuntu on your machine permanently.

Linux distributions generally come with a free web browser, and the most popular is Firefox . Two Firefox plugins that are particularly useful for literature reviews are Unpaywall and Zotero. Keep reading to learn why.

3. Unpaywall

Often one of the hardest parts of a literature review is gaining access to the papers you want to read for your review. The unintended consequence of copyright restrictions and paywalls is it has narrowed access to the peer-reviewed literature to the point that even Harvard University is challenged to pay for it. Fortunately, there are a lot of open access articles—about a third of the literature is free (and the percentage is growing). Unpaywall is a Firefox plugin that enables researchers to click a green tab on the side of the browser and skip the paywall on millions of peer-reviewed journal articles. This makes finding accessible copies of articles much faster that searching each database individually. Unpaywall is fast, free, and legal, as it accesses many of the open access sites that I covered in my paper on using open source in lit reviews .

Formatting references is the most tedious of academic tasks. Zotero can save you from ever doing it again. It operates as an Android app, desktop program, and a Firefox plugin (which I recommend). It is a free, easy-to-use tool to help you collect, organize, cite, and share research. It replaces the functionality of proprietary packages such as RefWorks, Endnote, and Papers for zero cost. Zotero can auto-add bibliographic information directly from websites. In addition, it can scrape bibliographic data from PDF files. Notes can be easily added on each reference. Finally, and most importantly, it can import and export the bibliography databases in all publishers' various formats. With this feature, you can export bibliographic information to paste into a document editor for a paper or thesis—or even to a wiki for dynamic collaborative literature reviews (see tool #7 for more on the value of wikis in lit reviews).

5. LibreOffice

Your thesis or academic article can be written conventionally with the free office suite LibreOffice , which operates similarly to Microsoft's Office products but respects your freedom. Zotero has a word processor plugin to integrate directly with LibreOffice. LibreOffice is more than adequate for the vast majority of academic paper writing.

If LibreOffice is not enough for your layout needs, you can take your paper writing one step further with LaTeX , a high-quality typesetting system specifically designed for producing technical and scientific documentation. LaTeX is particularly useful if your writing has a lot of equations in it. Also, Zotero libraries can be directly exported to BibTeX files for use with LaTeX.

7. MediaWiki

If you want to leverage the open source way to get help with your literature review, you can facilitate a dynamic collaborative literature review . A wiki is a website that allows anyone to add, delete, or revise content directly using a web browser. MediaWiki is free software that enables you to set up your own wikis.

Researchers can (in decreasing order of complexity): 1) set up their own research group wiki with MediaWiki, 2) utilize wikis already established at their universities (e.g., Aalto University ), or 3) use wikis dedicated to areas that they research. For example, several university research groups that focus on sustainability (including mine ) use Appropedia , which is set up for collaborative solutions on sustainability, appropriate technology, poverty reduction, and permaculture.

Using a wiki makes it easy for anyone in the group to keep track of the status of and update literature reviews (both current and older or from other researchers). It also enables multiple members of the group to easily collaborate on a literature review asynchronously. Most importantly, it enables people outside the research group to help make a literature review more complete, accurate, and up-to-date.

Wrapping up

Free and open source software can cover the entire lit review toolchain, meaning there's no need for anyone to use proprietary solutions. Do you use other libre tools for making literature reviews or other academic work easier? Please let us know your favorites in the comments.

Related Content

Buyer’s Guide To Literature Review Software

About this guide.

Our team has been developing literature review software for the world’s leading research organizations for over 15 years. Though the software has evolved dramatically over that period, the questions we are asked about the features and benefits of review software haven’t changed much.

In this guide, we present a comprehensive list of things to consider when evaluating a literature review software solution.

This guide will:

• Explain what literature review software does and how it is used
• Discuss where literature review software fits within the overall review process
• Provide a checklist of features to help you with the evaluation process

Who should read this guide?

If you are doing literature reviews today, you already know that they are increasingly required for regulatory compliance and safety monitoring. You also probably know that, while reviews sound simple on the surface, they are big projects that can consume significant amounts of time and resources. Doing reviews well can be a challenge.

This guide can benefit you if:

You are struggling with the amount of time it takes to conduct a review

If you are involved in the preparation of literature reviews for Clinical Evaluation Reports (CERs), Performance Evaluation Reports (PER), or if you track literature for safety monitoring, you need to be able to enforce standardized review processes and methods across your organization. Since your work could be subject to an audit, you need to be prepared.

You need to reduce the time it takes to conduct a review

You’re concerned about manual errors compromising the quality of your review

Did I make a transcription error? Did we forget to review that paper by Nosyk? Has any of my data changed? Worries like these can keep a researcher up at night and can seriously impact the quality and integrity of your review.

You’re not sure which literature review software is the best fit for you

What does literature review software do.

Today’s literature review software automates the many manual tasks involved in conducting a review. Literature reviews are process intensive and data heavy, and not so long ago they typically involved circulating paper copies of articles and screening forms to the review team who captured their work on spreadsheets.

Most reviewers currently use some form of technology to help manage the information and data in their review projects. In fact, a recent survey showed that the vast majority of reviewers still use spreadsheets at some stage of their review process.

Of course, it is possible to produce results using spreadsheets, or even paper forms. That said, each of these methods has a number of drawbacks that can have significant impact on both the quality and the volume of research produced.

Just Say No To Spreadsheets

When using spreadsheets for review tasks such as screening, data extraction, or storing references, you may find yourself dealing with some or all of the following:

• A reviewing “bottleneck” because each stage of the review must be completed before the next one is started
• Manual data entry errors that can be difficult or even impossible to catch
• Excessive manual work in checking for disagreements and creating reports
• Questions about the validity of your results due to lost files or undocumented processes

Where does literature review software fit in the process?

Literature review software is designed to reduce the manual work involved in conducting reviews and maintain a complete record of the work that’s been done on your review projects.

But how does it do this?

Once you’ve defined your research question and completed your search of relevant databases, you can typically import your search results into your literature review software and start your screening and data extraction processes.

Similar to the paper forms used in the past, literature review software uses electronic forms to record the answers to inclusion/exclusion questions. Some forms can be configured for data extraction. One of the main advantages that electronic forms provide is that they collect all your review data in one place, eliminating the need to manually cut and paste collate individual responses for processing and analysis. 

“Why input data twice when it only needs to be done once?”

Digital forms can be reused an unlimited number of times. Depending on the form and the reviewer, they can usually be completed faster than writing or typing since they can incorporate easy-to-use answer formats like checkboxes or radio buttons. They can also validate your data and even perform calculations before you submit it, giving you cleaner results and fewer errors.

Screening and data extraction are the most common review tasks facilitated by literature review software, but there are often other valuable features such as direct connection to popular databases such as PubMed, automated report generation, and reviewer roles and permissions management.

With regulatory bodies calling for continuous monitoring and assessment of safety data, having your entire review project and all its references, full text articles and audit trail stored within your literature review software can be a huge time saver when it comes time for updates.

As literature reviews have become a fundamental component of the risk management system for many organizations, they are increasingly scrutinized for thoroughness, standardized processes, and data integrity. By maintaining complete, accurate records of every reviewer action and decision, and allowing you to establish and enforce repeatable processes, literature review software makes it easier to deliver regulatory compliant, audit-ready literature reviews on time and on budget.

Top 5 Ways Systematic Review Software Can Help You

#1 compliance.

If there’s one thing that almost every reviewer wishes for, it’s more time. In our Survey of Literature Reviews, approximately one quarter of the respondents mentioned their greatest review challenge is the time involved in completing a review – to conduct searches, remove duplicates and irrelevant articles, complete screening, extract data, and prepare reports. In a recent survey of our user community, reviewers reported that literature review software reduced the time required to produce reviews by 40%-60%.

#3 Automation

No one wants to discover a mistake in their review right before – or worse, during – an audit.

Duplicate references, transcription errors, and data entry errors can skew, or even invalidate, your results. Literature review software can provide built-in automation and validation tools that dramatically reduce the potential for errors in your reviews.

#4 Compatibility

Although literature review software can help with many tasks throughout the review lifecycle, your process likely includes other tools for searching and storing references and data. You also likely need to use the information from your completed review in reports and submissions. Your literature review software should allow you to import and export your data in all the most common file formats, such as CSV, Excel, Word, PDF, RIS, and ENLX.

#5 Collaboration

Literature review software packages today are typically cloud-based and can be used from any browser on any device. With a centralized, shared data set, your team can collaborate in real time, regardless of location.

Your Literature Review Software Checklist

Deciding to adopt literature review software is more than just a monetary investment – it’s a commitment to a new way of doing things. And just like any significant purchase, it’s always a good idea to do your research first.

Make sure you conduct a thorough assessment of each of the available options to choose the software that is the best fit for your needs. Below is a list of features that may be offered by systematic review software packages.

This requirement applies to my assessement

Automatic reference updates to prevent the review from becoming out-of-date

Compatible with standard reference file types (RIS, CSV, and ENLX)

Direct integration with reference databases

Keyword highlighting for faster screening

Full-Text Retrieval

Data extraction, project management.

Real-time updates on project progress to inform stakeholders and facilitate planning

Live customer support, professional services offerings and training

Enterprise-Grade Software (High availability and redundancy,  scalable to handle hundreds of thousands of references per project, secure and regulatory compliant )

Download this Ebook

Learn more about distillersr.

RAxter is now Enago Read! Enjoy the same licensing and pricing with enhanced capabilities. No action required for existing customers.

Your all in one AI-powered Reading Assistant

A Reading Space to Ideate, Create Knowledge, and Collaborate on Your Research

• Smartly organize your research
• Receive recommendations that cannot be ignored
• Collaborate with your team to read, discuss, and share knowledge

From Surface-Level Exploration to Critical Reading - All in one Place!

Fine-tune your literature search.

Our AI-powered reading assistant saves time spent on the exploration of relevant resources and allows you to focus more on reading.

Select phrases or specific sections and explore more research papers related to the core aspects of your selections. Pin the useful ones for future references.

Our platform brings you the latest research related to your and project work.

Speed up your literature review

Quickly generate a summary of key sections of any paper with our summarizer.

Make informed decisions about which papers are relevant, and where to invest your time in further reading.

Get key insights from the paper, quickly comprehend the paper’s unique approach, and recall the key points.

Bring order to your research projects

Organize your reading lists into different projects and maintain the context of your research.

Quickly sort items into collections and tag or filter them according to keywords and color codes.

Experience the power of sharing by finding all the shared literature at one place.

Decode papers effortlessly for faster comprehension

Highlight what is important so that you can retrieve it faster next time.

Select any text in the paper and ask Copilot to explain it to help you get a deeper understanding.

Ask questions and follow-ups from AI-powered Copilot.

Collaborate to read with your team, professors, or students

Share and discuss literature and drafts with your study group, colleagues, experts, and advisors. Recommend valuable resources and help each other for better understanding.

Work in shared projects efficiently and improve visibility within your study group or lab members.

Keep track of your team's progress by being constantly connected and engaging in active knowledge transfer by requesting full access to relevant papers and drafts.

Find papers from across the world's largest repositories

Testimonials

Privacy and security of your research data are integral to our mission..

Everything you add or create on Enago Read is private by default. It is visible if and when you share it with other users.

You can put Creative Commons license on original drafts to protect your IP. For shared files, Enago Read always maintains a copy in case of deletion by collaborators or revoked access.

We use state-of-the-art security protocols and algorithms including MD5 Encryption, SSL, and HTTPS to secure your data.

Living systematic review software, optimized for clinical literature. Scroll

Autolit + synthesis:.

Nested Knowledge ® offers a comprehensive software platform for systematic literature review and meta-analysis. The software is composed of two parts which work in tandem. Search, screen, extract data, and complete critical appraisal with AutoLit ® . Visualize, analyze, publish and share insights with Synthesis.

MA Extraction

Qualitative

Quantitative

Search, Import, or Bibliomine.

Literature search.

Create updatable searches of PubMed, or import studies from a variety of common databases.

Add studies by mining from existing reviews, or add individual studies of interest. No matter how you get studies, we’ll set them up to be included in your living review.

If you add studies to AutoLit, we’ll trace the path of studies from Search to Synthesis.

Use AI to find relevant concepts.

Automatic PICO highlighting, or your own keywords, directs your eye to the key phrases from any abstract.

Inclusion Prediction AI can anticipate which studies are most relevant to your research question.

Dual Screening can help you quality-control your decisions, so only the studies that actually contain data relevant to you make it through.

If you screen out irrelevant references in AutoLit, we’ll automatically generate your PRISMA diagram in Synthesis.

Connect concepts across the literature.

You understand how key concepts in your field relate to each other – but those ideas are stuck in your head unless you lay them out for your readers.

By building a tagging hierarchy, you structure your ideas. By applying those tags to the studies in your review, you capture the evidence to support each concept.

We help out by enabling you to borrow from past hierarchies, create tags ‘on the fly’ as you read studies, and by connecting your tags to the quantitative data you’ll extract.

If you build and apply your hierarchy in AutoLit, we’ll also create interactive, qualitative visuals in Synthesis.

Continuous. Dichotomous. Categorical.

Meta-analytical data extraction.

Turn your tags into data elements to connect your qualitative and quantitative concepts. 

Identify which part of your hierarchy contains your interventions of interest.

Then, you’re all set to gather continuous, dichotomous and categorical metrics across multiple arms and time points from the text and tables in your studies of interest.

If you gather data in AutoLit, we’ll summarize and analyze your quantitative findings in Synthesis.

Publish, Share, Visualize with Synthesis:

Catch a ray from the qualitative sunburst..

Qualitative Synthesis

Tagging in AutoLit continuously and automatically updates Qualitative Synthesis.

Each segment of the sunburst diagram represents a concept you tagged, and immediately directs your readers to the underlying studies.

Try it!  Select one or more of the segments in the sunburst to filter the studies from our sample review on strokes that impact the brain stem to those that report on your concept of interest. Then, select a study from the list to view the abstract and gathered data.

Was this published? Yes, as a part of our  Stanford collaboration .

What are the odds? Drill into the Data.

Quantitative synthesis.

Gathering data in AutoLit continuously and automatically updates Quantitative Synthesis.

We slice the data three ways. First, we summarize your findings at the intervention and study level in Summary. Then, we let you create scatter plots of findings in Distribution. Finally, we compute odds ratios and build forest plots in our Network Meta-Analysis.

Was this published? Yes, as a part of our  Stanford collaboration .

Excel beyond words.

Draft in Manuscript Editor, and you’ll never need to update your data manually. Whenever you add data to your review, we add it automatically to your tables!

Rich text, point-and-click citation tools, auto-updating tables, and embeddable Synthesis visuals.

Was this published? Yes, as a part of our  Stanford collaboration .

Living Systematic Reviews

Dive into nested knowledge:.

First review is free • No credit card required

Buying for a team? Contact Sales:

Have a question?

Send us an email and we’ll get back to you as quickly as we can!

Save up to 71 hours on your literature review.  

Join over 200,000 researchers and students finishing their review faster., ✓ no credit card required ✓ no time limit on free trial ✓ invite unlimited team members on paid plans, deliver your systematic literature reviews like never before, faster reviews, expert, online support.

Save an average of 35% time spent per review, or up to 71 hours per review. 

Easy to learn and use, with 24/7 support from product experts who are also seasoned reviewers!

Seamless collaboration

No expertise required.

Enable the whole review team to collaborate from anywhere.

Suitable for all levels of experienced reviewers in a variety of sectors including health, education, social science and many others.

Supporting the world's largest systematic literature review community

Researchers and students, universities, societies and hospitals, reviews started, take a look inside, watch a quick demo.

See how Covidence makes doing your review more intuitive, streamlined and fun. 

HOW IT WORKS

Start streamlining your review, import citations.

Covidence works seamlessly with your favourite reference managers like EndNote, Zotero, Refworks, Mendeley or any tool that support RIS, CSV or PubMedXML formats.

Screen titles & abstracts

Breeze through screening with keyword highlighting & a lightning quick interface. Covidence keeps full records of who voted and also supports single or dual screeners.

Upload references

Transfer PDFs stored in your reference manager to Covidence in a few clicks.

Screen full text

Decide quickly on studies in full text. Capture reasons for exclusion and any notes so you can resolve any disagreements quickly, with a click of a button.

Risk of bias

Automatically populate your risk of bias tables by highlighting and commenting on text directly in your PDF. No more cut and paste.

Data extraction & export

Create customised extraction templates to suit you, then export a single, machine-readable file that easily integrates into all the common statistics packages.

TESTIMONIALS

What other reviewers say.

We're a non-profit on a mission to streamline the review process while improving research quality.

We love our community, and they say some very nice things about us.

Syreeta Nolan @nolan_syreeta

"I love you @Covidence. I love the YouTube series Covidence put out and my librarian is my hero. Let others help you and be careful when you set Inclusion and exclusion as to when you want to adhere to it at what stage as some information may be more visible in full-text review."

Rebecca Venchiarutti

@RebeccaVenchers

"Big fan of using @Covidence for conducting systematic reviews - this is my first systematic review I'm leading from start to finish, and hard to see myself using another platform. Step by step, organised, and very user-friendly."

Stephanie Craig, PhD

@DrStephCraig

"Second using covidence. It's great for the abstract and full text review. Also helps you track rationale for removal which is needed in the methods"

Unlimited use for every organization

With no restrictions on reviews and users, Covidence gets out of the way so you can bring the best evidence to the world, more quickly.

Start delivering your systematic literature review faster and better today.

Copyright - Covidence 2020

View website

Terms & Conditions

Privacy Policy

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• How to Write a Literature Review | Guide, Examples, & Templates

How to Write a Literature Review | Guide, Examples, & Templates

Published on January 2, 2023 by Shona McCombes . Revised on September 11, 2023.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research that you can later apply to your paper, thesis, or dissertation topic .

There are five key steps to writing a literature review:

• Search for relevant literature
• Evaluate sources
• Identify themes, debates, and gaps
• Outline the structure
• Write your literature review

A good literature review doesn’t just summarize sources—it analyzes, synthesizes , and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

Table of contents

What is the purpose of a literature review, examples of literature reviews, step 1 – search for relevant literature, step 2 – evaluate and select sources, step 3 – identify themes, debates, and gaps, step 4 – outline your literature review’s structure, step 5 – write your literature review, free lecture slides, other interesting articles, frequently asked questions, introduction.

• Quick Run-through
• Step 1 & 2

When you write a thesis , dissertation , or research paper , you will likely have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

• Demonstrate your familiarity with the topic and its scholarly context
• Develop a theoretical framework and methodology for your research
• Position your work in relation to other researchers and theorists
• Show how your research addresses a gap or contributes to a debate
• Evaluate the current state of research and demonstrate your knowledge of the scholarly debates around your topic.

Writing literature reviews is a particularly important skill if you want to apply for graduate school or pursue a career in research. We’ve written a step-by-step guide that you can follow below.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

• Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
• Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
• Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
• Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research problem and questions .

Make a list of keywords

Start by creating a list of keywords related to your research question. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list as you discover new keywords in the process of your literature search.

• Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
• Body image, self-perception, self-esteem, mental health
• Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some useful databases to search for journals and articles include:

• Your university’s library catalogue
• Google Scholar
• Project Muse (humanities and social sciences)
• Medline (life sciences and biomedicine)
• EconLit (economics)
• Inspec (physics, engineering and computer science)

You can also use boolean operators to help narrow down your search.

Make sure to read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

You likely won’t be able to read absolutely everything that has been written on your topic, so it will be necessary to evaluate which sources are most relevant to your research question.

For each publication, ask yourself:

• What question or problem is the author addressing?
• What are the key concepts and how are they defined?
• What are the key theories, models, and methods?
• Does the research use established frameworks or take an innovative approach?
• What are the results and conclusions of the study?
• How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
• What are the strengths and weaknesses of the research?

Make sure the sources you use are credible , and make sure you read any landmark studies and major theories in your field of research.

You can use our template to summarize and evaluate sources you’re thinking about using. Click on either button below to download.

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It is important to keep track of your sources with citations to avoid plagiarism . It can be helpful to make an annotated bibliography , where you compile full citation information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

Prevent plagiarism. Run a free check.

To begin organizing your literature review’s argument and structure, be sure you understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

• Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
• Themes: what questions or concepts recur across the literature?
• Debates, conflicts and contradictions: where do sources disagree?
• Pivotal publications: are there any influential theories or studies that changed the direction of the field?
• Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

• Most research has focused on young women.
• There is an increasing interest in the visual aspects of social media.
• But there is still a lack of robust research on highly visual platforms like Instagram and Snapchat—this is a gap that you could address in your own research.

There are various approaches to organizing the body of a literature review. Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarizing sources in order.

Try to analyze patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organize your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

• Look at what results have emerged in qualitative versus quantitative research
• Discuss how the topic has been approached by empirical versus theoretical scholarship
• Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text , your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, you can follow these tips:

• Summarize and synthesize: give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: don’t just paraphrase other researchers — add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically evaluate: mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: use transition words and topic sentences to draw connections, comparisons and contrasts

In the conclusion, you should summarize the key findings you have taken from the literature and emphasize their significance.

When you’ve finished writing and revising your literature review, don’t forget to proofread thoroughly before submitting. Not a language expert? Check out Scribbr’s professional proofreading services !

This article has been adapted into lecture slides that you can use to teach your students about writing a literature review.

Scribbr slides are free to use, customize, and distribute for educational purposes.

Open Google Slides Download PowerPoint

If you want to know more about the research process , methodology , research bias , or statistics , make sure to check out some of our other articles with explanations and examples.

• Sampling methods
• Simple random sampling
• Stratified sampling
• Cluster sampling
• Likert scales
• Reproducibility

 Statistics

• Null hypothesis
• Statistical power
• Probability distribution
• Effect size
• Poisson distribution

Research bias

• Optimism bias
• Cognitive bias
• Implicit bias
• Hawthorne effect
• Anchoring bias
• Explicit bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarize yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your thesis or dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. (2023, September 11). How to Write a Literature Review | Guide, Examples, & Templates. Scribbr. Retrieved June 18, 2024, from https://www.scribbr.com/dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, what is a theoretical framework | guide to organizing, what is a research methodology | steps & tips, how to write a research proposal | examples & templates, "i thought ai proofreading was useless but..".

I've been using Scribbr for years now and I know it's a service that won't disappoint. It does a good job spotting mistakes”

Doing a literature review using digital tools (with Notion template)

I’ve recently revamped my literature review workflow since discovering Notion . Notion is an organization application that allows you to make various pages and databases. It’s kind of like your own personal wiki- you can link your pages and embed databases into another page, adding filters and sorting them using user-set properties. The databases are what I use the most. I’ve essentially transferred all of my excel sheets into Notion databases and find it much easier to filter and sort things now. In this post, I’ll go through how I do my literature review and share a Notion template that you can use.

I like to organize my literature review using various literature review tools along with two relational Notion databases: a ‘literature tracker’ and a ‘literature notes’ matrix. You can see a flow chart of my literature review process below (it’s inspired by this post by Jenn’s Studious Life and the three pass method for reading papers which I wrote about last week in this post ):

As you can see, this process involves a couple of decision points which helps me focus on the most important papers. This is an iterative process that keeps me up to date on relevant research in my field as I am getting new paper alerts in my inbox most days. I used this method quite successfully to write the literature review for my confirmation report and regularly add to it for the expanded version that will become part of my PhD thesis. In this post, I’ll break down how this works for me and how I implement my Notion databases to synthesise the literature I read into a coherent argument.

You can click on the links below to navigate to a particular section of this article:

The literature search

The literature tracker, the literature synthesis matrix, writing your literature review, iterating your literature review, my literature review notion template, some useful resources.

This is always the first step in building your literature review. There are plenty of resources online all about how to start with your search- I find a mixture of database search tools works for me.

The first thing to do when starting your literature review is to identify some keywords to use in your initial searches. It might be worth chatting to your supervisor to make a list of these and then add or remove terms to it as you go down different research routes. You can use keyword searches relevant to your research questions as well tools that find ‘similar’ papers and look at citation links. I also find that just looking through the bibliographies of literature in your field and seeing which papers are regularly cited gives you a good idea of the core papers in your area (you’ll start recognising the key ones after a while). Another method for finding literature is the snowballing method which is particularly useful for conducting a systematic review.

Here are some digital tools I use to help me find literature relevant to my research questions:

Library building and suggestions

Mendeley was my research management tool of choice prior to when I started using Notion to organize all of my literature and create my synthesis matrix. I still use Mendeley as a library just in case anything happens to my Notion. It’s easy to add new papers to your library using the browser extension with just one click. I like that Mendeley allows you to share your folders with colleagues and that I can export bib.tex files straight from my library into overleaf documents where I’m writing up papers and my thesis. You do need to make sure that all of the details are correct before you export the bib.tex files though as this is taken straight from the information plane. I also like to use the tag function in Mendeley to add more specific identifiers than my folders.

Mendeley is also useful for finding literature related to those in your library- I’ve found quite a few interesting papers through the email updates they send out each week with ‘suggested papers’. You can also browse these suggestions from within Mendeley and use its interface to do initial keyword searches. The key is to just scan the titles and then decide whether it’s worth your time reading the abstract and then the rest of it. It’s easy to get overwhelmed by the sheer amount of papers being published every day so being picky in what you read is important (and something I need to work on more!).

Some similar tools that allow you to build a library and get literature recommendations include Zotero , Researcher , Academia , and ResearchGate . It’s up to you which one you use for your own purposes. One big factor for me when choosing Mendeley was that my supervisor and colleagues use it so it makes it much easier to share libraries with them, so maybe ask your colleagues what they use before settling on one.

Literature databases and keyword alerts

There are a variety of databases out there for finding literature. My go-to is Web of Science as it shows you citation data and has a nice interface. I used this to begin my initial literature search using my keywords.

The other thing you can do with these kinds of tools is set up email alerts to get a list of recent work that has just been published with any keywords you set. These alerts are usually where I find papers to read during journal club with my supervisor. You can customize these emails to what suits you- mine are set to the top 10 most relevant new papers for each keyword weekly and I track around 5 words/phrases. This allows me to stay on top of the most recent literature in my field- I have alerts set up on a variety of services to ensure that I don’t miss anything crucial (and alerts from the ArXiv mean I see preprints too). Again, you need to be picky about what you read from these to ensure that they are very relevant to your research. At this stage, it’s important to spend as little time as possible scanning titles as this can easily become a time suck.

Some of the other tools I have keyword (and author) email alerts set up on are: Scopus , Google Scholar , Dimensions , and ArXiv alerts . I set 10 minutes maximum aside per day to scan through any new email alerts and save anything relevant to me into my literature tracker (which I’ll come to more later).

Literature mapping tools

There are loads of these kinds of tools out there. Literature mapping can be helpful for finding what the seminal papers are in your field and seeing how literature connects. It’s like a huge web and I find these visual interfaces make it much easier to get my head around the relationships between papers. I use two of these tools during the literature search phase of the flowchart: Citation Gecko and Connected Papers .

Citation Gecko builds you a citation tree using ‘seed papers’. You can import these from various reference management software (like Mendeley), bib.tex files or manually search for papers. This is particularly useful if your supervisor has provided you with some core papers to start off with, or you can use the key papers you identified through scanning the bibliographies of literature you read. My project is split into fairly clear ‘subprojects’ so these tools help me see connections between the various things I’m working on (or a lack of them which is good in some ways as it shows I’ve found a clear research gap!).

You can switch between different views and add connecting papers as new seed papers to expand your network. I use this tool from time to time with various different papers associated with my subprojects. It’s helped me make sure I haven’t missed any key papers when doing my literature review and I’ve found it to be fairly accurate, although sometimes more recent papers don’t have any citation data on it so that’s something to bear in mind.

Connected Papers uses a ‘similarity’ algorithm to show paper relationships. This isn’t a citation tree like Citation Gecko but it does also give you prior and derivative works if you want to look at them. All you do is put one of your key papers into the search box and ‘build a graph’. It will then show you related papers, including those which don’t have direct citation links to the key paper. I think this is great for ensuring that you’re not staying inside an insular bubble of the people who all cite each other. It also allows me to see some of the research which is perhaps a bit more tangential to my project and get an overview of where my work sits within the field more broadly.

I like Connected Paper’s key for the generated tree and that it shows where related papers connect between themselves. Again, it’s helpful for ensuring that you haven’t missed a really important work when compiling your literature review and doesn’t just rely on citation links between papers.

This is where I record the details of any paper I come across that I think might be relevant to my PhD. In some ways, it’s very similar to Mendeley but it’s a version that sits within Notion so I have some more customised filtering categories set up, like my ‘status’ field where I track which pass I am on.

Here’s what my literature tracker looks like:

The beauty of Notion is that you can decide which properties you want to record in your database and customize it to your needs. You can sort and filter using these properties including making nested filters and using multiple filters at once. This makes it really easy to find what you’re looking for. For example, say I’m doing my literature review for my ‘FIB etching’ subproject and want to see all of the papers that I marked as relevant to my PhD but haven’t started reading yet. All I need to do is add a couple of filters:

And it filters everything so that I’m just looking at the papers I want to check out. It’s this flexibility that I think really gives Notion the edge when it comes to my literature review process.

The other thing I really like about using Notion rather than excel is that I can add different database views. I especially like using the kanban board view to see where I’m at with my reading workflow:

When I add something to the literature tracker database, I scan the abstract for keywords to add and categorize it in terms of relevant topics. It’s essentially the first pass of the paper, so that involves reading the title, abstract, introduction, section headings, conclusions, and checking the references for anything you recognise. After this is done, I decide whether it’s relevant enough to my PhD to proceed to do a second pass of the paper, at which point I will progress to populating my literature notes database.

Once I’ve decided that I want to do a second pass on a paper, I then add it to the ‘literature notes’ database. This is part of the beauty of Notion: relational databases. I have ‘rollup’ properties set in the literature notes database which shows all of the things I added during my first pass and allows me to filter the matrix using them. You can watch the video below to see exactly how to add a new paper to the ‘notes’ database from the ‘tracker’ database:

During the second pass, I populate the new fields in the ‘notes’ database. These are:

Summary | Objective of study | Key Results | Theory | Materials | Methods | Conclusions | Future work suggested | Critiques | Key connected papers.

I also have various themes/questions/ideas as properties which I add a few notes on for each relevant paper. I then complete my ‘questions for critical engagement’ which are on the entry’s ‘Notes’ page and are stored in the ‘Article Template’. If you want to read more about this process, check out my ‘how to read a scientific paper’ post .

By, doing this I create a synthesis matrix where I can see a breakdown of the key aspects of each paper and can scan down a column to get an overview of all of the papers I have read. For example, if I wanted to see all of the papers about Quantum Point Contacts to get an idea of what previous work has been done so that I can identify my research gap, I can filter using the tag property and can then see the notes I wrote for each entry, broken down by section. I also have tags for my research questions or themes, materials used, experimental techniques, fabrication techniques, and anything else that comes to mind really! The more tags I have for a paper, the easier it is to filter when I want to find a specific thing.

The other property I have included in the literature notes database is ‘Key connected papers’. This is a relation but is within the database itself. So it means that I can link to the page of other papers in the literature matrix. I’ve found this to be useful for connecting to what I call ‘core’ papers. I can also filter using this property, allowing me to see my notes on all of the papers I’ve read that are related to a certain ‘core’ paper. This helps with synthesising all of the information and forming my argument.

For those papers most relevant to my research (the ‘core’ papers) I’ll also do a third pass which involves reimplementing the paper in my own words. This is quite a time-consuming task so not many papers reach this stage, but those which I have done a third pass on are the ones I know really well. My hope is that this will stand me in good stead for my viva. This process also helps me refine my research questions further as I gain a deeper understanding of the field.

I find that writing up a review is extremely intimidating, but having the literature matrix makes this process that bit easier. I won’t go into too many details as there are already loads of resources out there going into the details of writing up a review, but here’s a brief overview of my own process:

Identify your research themes

Using your literature matrix, review each research theme or question and decide which ones you are going to focus on. These will form the different sections of your literature review and help you write your thesis statement(s). You can also think about how your questions link to ensure that you’re telling a coherent story with your review.

Choose and summarize literature related to each theme

For each section, gather up the most important related literature and summarize the key points of each source. A good literature review doesn’t need to cover all the literature out there, just the most significant sources. I try to stick to around 10 or fewer key sources per section.

Critical evaluation of sources

This is where you utilize the ‘questions for critical engagement’. Make sure you evaluate the strengths and weaknesses of the studies you’re writing about. By doing this, you can establish where our knowledge is lacking which will come in helpful later when establishing a research gap.

Analyse each source in relation to other literature

Try to make sure that you are telling a coherent story by linking between your sources. You can go back to the literature matrix here and use it to group similar studies to compare and contrast them. You should also discuss the relevance of the source’s findings in relation to the broader field and core papers.

Situate your research in a research gap

This is where you justify your own research. Using what you have laid out in the rest of the review, show that there is a research gap that you plan to fill and explain how you are going to do that. This should mean that your thesis flows nicely into the next section where you’ll cover the materials and methods you used in your research project.

In some ways, a literature review never really ends. As you can see in the flowchart at the beginning of this post, I regularly update and revise my literature review as well as refining my research questions. At this point in my PhD, I think that most of my research questions are quite well defined, so I’m mostly just adding any newly published work into my review. I don’t spend much time reading literature at the moment but I’m sure I’ll return to it more regularly when I’m in the write-up phase of my PhD. There is a balance to be had between reading and writing for your literature review and actually getting on with your own research!

Here’s the link to my Notion Literature Review Template . You can duplicate it and adapt it however you want, but this should save you some time setting up the initial databases if you’d like to use my method for organizing your own literature review.

Here are some resources on how to do a literature review that I’ve found useful during my PhD:

• The Literature Review: Step-by-Step Guide for Students
• 3 Steps to Save You From Drowning in Your Literature Review
• How to write a literature review
• How to become a literature searching ninja
• Mind the gap
• 7 Secrets to Write a PhD Literature Review The Right Way

If you like my work, I’d love your support!

Share this:

• Click to share on Twitter (Opens in new window)
• Click to share on Facebook (Opens in new window)
• Click to share on LinkedIn (Opens in new window)

11 thoughts on “Doing a literature review using digital tools (with Notion template)”

Thank you so much for your insight and structured process. This will help me a lot kicking off my Master Thesis.

The perfect method to organize the literature that I have read and will read in the future. I am so glad to have found your website, this will save me from thrashing around in the swamp of literature. I was already feeling the limits of my memory when I was doing my master thesis and this will be so helpful during my PhD.

Thank you so much for this detailed post! Lily 🙂

Thank you very much for this. I’m doing my undergrad atm and reading a lot of papers. This seems like an excellent way of tracking everything.

Thank you, you made my beginning less stressful. I like your system and i helped me a lot. I have one question (more might come later), What do you mean by " journal club with my supervisor."

This piece is really really helpful! I started from this one and went through the rest blog writings. I agree on many points with Daisy. I had an unhappy experience of PhD two years ago and now just started a new one in another country. I will take it as an adventure and enjoy it.

This is an AMAZING template. I've found this so helpful for my own workflow. Thank you so much!

I found this post really helpful. Thank you.

thank you very much!

Hi! Thank you very much for posting this guide and sharing your notion template! I do have a question—do you manually enter the references into Notion, or is there any way to speed up the process? Ta x

Leave a Reply Cancel reply

Discover more from notes from the physics lab.

Subscribe now to keep reading and get access to the full archive.

Type your email…

Continue reading

Screen, analyse and summarise articles faster with Scholarcy

Try it for free, subscribe today.

Scholarcy is used by students around the world to read and analyse research papers in less time. Upload your articles to Scholarcy to:

• Cut your reading time in half and feel more in control
• Identify the papers that matter in less time
• Jump straight to the most important information
• Compare a collection of articles more easily

With Scholarcy Library, you can import all your papers and search results, and quickly screen them with the automatically generated ‘key takeaway’ headline.

Take the stress out of your literature review

While there are lots of tools that help you discover articles for your research, how do you analyse and synthesise the information from all of those papers?

3 easy ways to import articles

Scholarcy lets you quickly import your articles for screening and analysing.

Import papers in PDF, Word, HTML and LaTeX format

Import search results from PubMed or any service that provides results in RIS or BibTeX format

Import publisher RSS feeds

Build your literature matrix in minutes

Our Excel export feature generates a literature synthesis matrix for you, so you can

Compare papers side by side for their study sizes, key contributions, limitations, and more.

Export literature-review ready data in Excel, Word, RIS or Markdown format

Integrates with your reference manager and ‘second brain’ tools such as Roam, Notion and Obsidian

Carrying out a systematic review?

Scholarcy breaks papers down into our unique summary flashcard format.

The Study subjects and analysis tab shows you study population, intervention, outcome, and statistical analyses from the paper.

And the Excel synthesis matrix generated shows the key methods and quantitative findings of each paper, side by side.

Build a knowledge graph from your papers

If you’re a fan of the latest generation of knowledge management tools such as  Roam  or  Obsidian , you’ll love our  Markdown  export.

This creates a knowledge graph of all the papers in your library by connecting them via key terms, methods, and shared citations.

What People Are Saying

“Quick processing time, successfully summarized important points.”

“It’s really good for case study analysis, thank you for this too.”

“I love this website so much it has made my research a lot easier thanks!”

“The instant feedback I get from this tool is amazing.”

“Thank you for making my life easier.”

A living systematic review of prevention, diagnosis and treatments for every disease.

Pitts offers web application software for record screening and data-extraction specially developed for living systematic review and (network) meta-analysis.

Automated search

When conducting a living systematic review, checking if new evidence is published continuously is essential. Automated Search automatically executes your search strategy in bibliographic databases using application programming interface integration. We have Automated Search integrated for PubMed and are working on integrating more databases. When new studies that fit your search strategy become available, they will automatically appear in the screening workflow.

Real-time collaboration with reviewers is critical when conducting a systematic review. When starting the review, you can configure customized exclusion criteria. You can choose to do the independent blinded screening. It is easy to resolve conflicts when they arrive. A machine-learning algorithm can be used to classify randomized controlled trials based on the abstracts.

Data extraction

Setting up a data extraction form is often difficult, especially for network meta-analysis. You can easily pre-configure the data extraction form in the web application to fit your needs. While conducting a living systematic review, adding data is sometimes necessary due to unforeseen circumstances. You can edit the data extraction form during the living systematic review. You can also do data extraction alone or with two reviewers independently with blinding.

Using integration with Chat GPT (using OpenAI API), we make it possible to receive GPT-generated recommendations of values and sentences during data extraction for all sort of data items like population characteristics, percentage of men and women in a study, mean age of the participants in the trial etc. Try it out in our demo: app.pitts.ai

Every human life has extraordinary value and is worth saving. In the past few years, the workload to synthesize reliable, unbiased, up-to-date, and complete health information has rapidly increased due to increased publications in biomedical literature. The most optimal treatment for nearly all medical conditions is unknown due to an incomplete overview of the evidence. Even less is known about disease prevention and healthy living. The incomplete overview of evidence undoubtedly resulted in non-optimal outcomes for many patients. The answer is living systematic reviews and (network) meta-analyses of diagnosis, treatments, and prevention for every disease. Our mission is to facilitate trust in medicine by building the technical infrastructure to make rigorous, complete, and up-to-date evidence synthesis possible.

Already in use by:

Software Developer

AI developer

We are a small but dedicated team of software engineers working on building the future of evidence-based medicine.

Chief Executive Officer

Chief Technology Officer

Teaching Literature Reviewing for Software Engineering Research

The goal of this chapter is to support teachers in holistically introducing graduate students to literature reviews, with a particular focus on secondary research. It provides an overview of the overall literature review process and the different types of literature review before diving into guidelines for selecting and conducting different types of literature review. The chapter also provides recommendations for evaluating the quality of existing literature reviews and concludes with a summary of our learning goals and how the chapter supports teachers in addressing them.

1 Introduction

Literature reviews are a crucial part of academic research. Although this is apparent to experienced researchers, it might not be immediately clear to early-stage graduate students. To motivate the topic, a teacher might start with a rhetorical question:

Have you ever written a paper for a seminar, a thesis, or a research project? If so, you probably already have done a literature review.

While students often perform ad hoc reviews as part of seminars or theses, those reviews are usually informal. It is important to motivate that such rather unstructured ad hoc literature reviews might be acceptable for these contexts, but rigorous scientific research requires more systematic approaches.

After introducing the topic of literature reviews, one can take a step back and talk about different ways of doing research. In general, we can stratify scholarship and scholarly writing into primary , secondary , and tertiary research. This distinction is crucial to understanding the different types of literature review that can be performed.

Primary Research Primary research involves making observations in the broadest sense of collecting data about objects that are not studies themselves. Computing code metrics, administering questionnaires, interviewing participants, counting bugs, collecting documents, downloading source code, and taking field notes while observing a retrospective meeting are all primary research.

Ideally, different research groups study the same phenomenon independently, allowing other researchers to collect, compare, and aggregate individual results. Since these researchers did not conduct the study themselves but reused their results, this process is called secondary research .

Secondary Research Secondary research involves analyzing, synthesizing, and critiquing primary studies. Ad hoc reviews, case surveys, critical reviews, meta-analysis, meta-synthesis, and scoping reviews are all types of secondary research.

Secondary research is central to evidence-based practice  ( Kitchenham et al , 2004 ) . On the one hand, important decisions should typically be made based on the balance of evidence rather than a single study. On the other hand, from the perspective of a practitioner, it is infeasible to read every study on a certain topic. Secondary research can help solve this discrepancy. Another aspect is that in many fields, individual studies are too small to produce accurate estimates of population parameters, for example, to assess the strength of the relationship between two variables. However, not every paper that somehow combines the results of multiple other papers is automatically of value. Some secondary research degrades into predominately descriptive “papers about papers” with limited scope and usefulness.

At a certain point in time, the body of knowledge on a certain phenomenon or a group of phenomena might be large enough to enable tertiary research .

Tertiary Research Tertiary research has two related meanings: (1) analyses of groups of secondary studies (meta-reviews); (2) summaries or indices of broad areas of scientific knowledge as found in textbooks, encyclopedia entries, etc.

Although tertiary studies can be valuable in specific circumstances, we should expect them to be rare.

In summary, literature reviews are crucial for understanding, structuring, and synthesizing collective knowledge on a certain topic, making empirical results more accessible to researchers and practitioners. This is particularly true for topics discussed in the software engineering research community.

The purpose of this chapter is to outline how to holistically introduce students to literature reviews, with a particular focus on secondary research. We provide an overview of the different types of literature review before diving into guidelines for selecting and conducting the appropriate literature review type depending on the research context. We also want to support teaching students to evaluate the quality of existing literature reviews and conclude the chapter with a summary of our learning goals and how the chapter supports teachers in addressing them.

2 Types of Literature Review

This section provides a broad overview of the literature review landscape, including references for further reading. As mentioned in the introduction, the first literature review performed by students is usually ad hoc . Although such reviews might be appropriate for seminar papers or theses, it is nonetheless important to understand their limitations. This section is partially based on a previously published short paper that motivates the need for more mature secondary research in software engineering  ( Ralph and Baltes , 2022 ) . We are convinced that educating graduate students is an essential step toward achieving that goal.

In-class SuggestionStarting the lecture with all types of literature review might overwhelm students. An alternative to this bottom-up approach would be to start top-down, i.e., first outline the process (see the in-class suggestion in Section  3.1 ) and then provide detailed information on the types. The types themselves can be introduced starting with examples that are aligned with the groups’ interests and backgrounds.

In the following, we introduce the seven types of literature review that are outlined in the above-mentioned paper. Tables  2.6 and 2.6 summarize the content of this chapter.

Ad Hoc ReviewAn ad hoc literature review is simply a discussion of unsystematically selected literature, as contained in most research papers as part of a background or related work section.

Ad hoc reviews may develop theory  ( Ralph , 2018 ) , or integrate a new theory into existing literature  ( Stol et al , 2016 ) . They can further support a position paper or tertiary scholarship. Ad hoc reviews use purposive sampling  ( Baltes and Ralph , 2022 ) ; that is, researchers purposefully select papers or studies that are useful, relevant, or support their arguments. Ad hoc reviews are often appropriate, for example, to support theory development, identify promising research topics, or prepare for a comprehensive exam. However, they also come with major limitations.

AttentionAd hoc reviews suffer from important limitations. Their unsystematic nature introduces sampling bias and defies replication.

An unsystematic ad hoc approach may lead to cherry-picking of evidence supporting authors’ arguments  ( Gough et al , 2017 ) . Therefore, ad hoc reviews are inappropriate for supporting empirical statements such as “x causes y”, “most people/objects have property P”, or “process P has structure S or follows rules R”. Those limitations are best understood by contrasting ad hoc reviews with the various systematic approaches that exist, which we will introduce in the following.

Before we dive into the different types of systematic reviews , we want to clarify the term. Historically, its meaning differs between research communities. The term can be used to describe a systematic review that uses meta-analysis of quantitative studies (especially experiments) to assess the strength of the evidence for specific, usually causal, propositions. In the context of this chapter, to avoid confusion, we refer to this type of review as meta-analysis and define systematic reviews as follows.

Systematic Review (Definition)A systematic review is a literature review that employs a systematic (hence the name), replicable process of selecting primary studies for inclusion, including case surveys, critical reviews, meta-analyses, meta-syntheses, and scoping reviews.

This double meaning is due to the history of systematic reviews. The concept of a meta-analytic systematic review emerged from health and medicine in the late 20 th century  ( Purssell and McCrae , 2020 ) , when practitioners could not keep up with the accelerating production of research. When Chalmers  ( Chalmers , 1993 ) founded the Cochrane Library in 1993, the medical community coalesced around using meta-analysis to inform evidence-based practice. Now, systematic review is often conflated with meta-analysis . However, other types of systematic reviews have also been around for decades. Scoping reviews were proposed no later than 2005 ( Arksey and O’Malley , 2005 ) . Meta-synthesis goes back at least to 1996 ( Jensen and Allen , 1996 ) . Case surveys were proposed as far back as 1974 ( Lucas , 1974 ) .

Systematic Review (Process) Systematic reviews begin by applying a search strategy to identify primary studies that meet pre-established criteria. Most types of systematic reviews seek to identify all the primary studies that meet the selection criteria by combining various techniques to mitigate sampling bias and publication bias as part of the search strategy. Guidelines for developing search strategies are available.

The ACM SIGSOFT Empirical Standards for Software Engineering 1 1 1 https://www2.sigsoft.org/EmpiricalStandards/docs/standards list various techniques to mitigate sampling bias and publication bias  ( Ralph et al , 2020 ) , which researchers can adopt when developing their search strategy:

backward and forward snowballing searches

checking profiles of prolific authors in the area

searching both formal databases (e.g., DBLP) and indexes (e.g., Google Scholar)

searching for relevant dissertations

searching pre-print servers (e.g., arXiv)

soliciting unpublished manuscripts through mailing lists or social media

contacting known authors in the area

This systematic search yields a list of primary studies. The way these primary studies are then analyzed determines the type of systematic review. If the literature review includes gray literature such as blog posts and whitepapers, it is sometimes referred to as a multivocal literature review   ( Garousi et al , 2019 ) .

2.1 Meta-analysis

Although systematic reviews can be conducted for various different types of primary studies, an archetypal systematic review analyzes a set of randomized controlled experiments with the same independent and dependent variables. Meta-analyses are then used to aggregate the results of these primary studies.

Meta-analysisA meta-analysis   ( Glass , 1976 ) analyzes a set of quantitative studies, usually randomized controlled experiments, with the same independent and dependent variables to statistically aggregate the results of the primary studies into a global effect size estimate.

Consider the following scenario: Ten different experiments randomized software engineering undergraduate students into a control group (who complete tasks individually) and a treatment group (who complete tasks in pairs) to compare individual vs. pair programming. The dependent variable was the number of tasks completed successfully. Each primary study reports the results of an independent samples t-test including the mean and standard deviation for each group, the t-statistic, the p-value, Cohen’s D 𝐷 D italic_D (effect size), and the 95% confidence interval for D 𝐷 D italic_D . Our aim is to combine the results of these ten experiments to estimate the effect of pair programming on effectiveness. Suppose that four studies found a negative effect, three found no significant effect, and three found a positive effect. Can we conclude, based on these study results, that the effect is negative?

Attention Do not use vote counting to aggregate the results of primary studies, for example, by concluding that an effect is negative because the studies reporting negative results outweigh the studies reporting positive results.

In our scenario, counting votes would mean that we concluded that the effect is negative because four negative results outweigh three positive results. Vote counting is invalid because primary studies can have wildly different sample sizes and quality levels. What if the studies that found positive effects were much larger and more rigorous while the studies that found negative effects were small and confounded? What if, when the three studies without significant results are aggregated, together, their results are significant?

Instead of vote-counting, we apply meta-analysis  ( Glass , 1976 ) , that is, we statistically aggregate primary study results into a global effect size estimate. This is often possible with the summary data reported in papers, without the original datasets. Meta-analysis can aggregate results from other kinds of (quantitative) methods as long as the studies have the same independent and dependent variables or overlapping sets of variables. The more complicated the overlaps, the more complicated the meta-analytic model. A comprehensive tutorial on statistical procedures for meta-analysis is beyond the scope of this paper but is readily available  ( Borenstein et al , 2021 ) .

Meta-analytic reviews essentially have the same research question as the studies being reviewed. The purpose of the meta-analysis is to reach a more reliable and robust conclusion by aggregating all available data, implying two important criteria for meta-analysis:

researchers should go to great lengths to find all relevant studies

researchers must evaluate the quality of each primary study and either exclude low quality studies or include study quality as a covariate in the meta-analytic model

In summary, when scientists equate systematic reviews with evidence-based practice, they usually mean meta-analytic reviews. Meta-analysis aggregates quantitative studies that investigate the same or overlapping hypotheses. They do not simply describe existing research. Meta-analysis is rare in software engineering. While there are several good examples ( Shepperd et al , 2014 ; Hannay et al , 2009 ; Rafique and Mišić , 2012 ) , quality meta-analysis is dwarfed by superficial scoping reviews  ( Cruzes and Dybå , 2011 ) .

2.2 Meta-synthesis

Since quantitative and qualitative research are both common in the software engineering research community, we want to introduce the analogue of meta-analysis for qualitative research: meta-synthesis .

Meta-synthesis Meta-synthesis refers to a family of methods of aggregating qualitative studies  ( Jensen and Allen , 1996 ) . After identifying the primary studies, the researcher applies hermeneutical and dialectical analyses to understand each primary study, translate them into each other, and construct an account of the body of research; for example, a theory of the central phenomenon that unites the primary studies.

Other names for meta-synthesis are thematic synthesis, narrative synthesis, meta-ethnography, and interpretive synthesis. Meta-synthesis requires expertise in qualitative methods and familiarity with the underlying philosophical assumptions. Without a deep understanding of hermeneutical  (see Ricoeur , 1981 ) and dialectical  (see Maybee , 2020 ) analyses, one should not attempt to perform meta-synthesis. In principle, meta-synthesis can be applied to both qualitative and quantitative work. In practice, such combinations are philosophically strained.

AttentionMeta-synthesis is not organizing papers into categories (as in scoping reviews). Meta-synthesis is the process of synthesizing a credible, nuanced account of a phenomenon from prior qualitative findings.

2.3 Case Survey

A case survey’s primary studies are (typically qualitative) case studies in the broadest sense (i.e., a scholarly account of some events). Experience reports and gray literature may or may not be included, depending on the study’s purposes.

Unlike meta-synthesis, however, a case survey transforms qualitative accounts into a quantitative dataset that supports null-hypothesis testing. Case surveys share the philosophy of meta-analysis (positivism), not meta-synthesis (constructivism)  ( Bullock and Tubbs , 1987 ) .

Case SurveyA case survey transforms the results of (typically qualitative) case studies into a quantitative dataset that supports null-hypothesis testing.

Case surveys typically begin with a priori hypotheses and an a priori coding scheme. The researcher reads each case and extracts data into the coding scheme, often using simple dichotomous variables like ‘did the team have retrospective meetings? [yes/no]’ or ‘does the case mention coordination problems? [yes/no]’ The resulting dataset is often too sparse for regression modeling, so researchers use simple bivariate correlations to test hypotheses  ( Bullock and Tubbs , 1987 ) .

The Rand Corporation proposed case studies as a “way to aggregate existing research”  ( Lucas , 1974 ) , quickly picked up by Yin  ( Yin and Heald , 1975 ) , and later elaborated in management  ( Bullock and Tubbs , 1987 ; Larsson , 1993 ) . Today, case surveys, or case meta-analyses, are widely used in management and information systems research  ( Jurisch et al , 2013 ) . Although software-engineering-specific case surveys are available  ( Melegati and Wang , 2020 ; Petersen , 2020 ) , they remain rare. However, case surveys have been used to investigate strategic pivots in software start-ups  ( Bajwa et al , 2017 ) and how organizations select component sourcing options  ( Petersen et al , 2017 ) . Case surveys have great potential in software engineering research because case studies are so common.

2.4 Critical Review

The term critical review has different meanings in different research communities. We focus on its meaning in software engineering research.

Critical ReviewA critical review analyzes a sample of qualitative or quantitative primary studies to support an argument or critique, often of a meta-scientific nature.

For example, Stol et al. ( Stol et al , 2016 ) ’s critical review of the use of grounded theory in software engineering criticizes method slurring; that is, claiming to have used a research methodology that was not actually used to create illusory legitimacy. Similarly, Baltes and Ralph’s critical review  ( Baltes and Ralph , 2022 ) criticizes how software engineering researchers often overstate sample representativeness and conflate random sampling with representative sampling. In fact, critical reviews in software engineering often investigate methodological topics such as how ethnography is reported ( Zhang et al , 2019 ) or how qualitative research is synthesized ( Huang et al , 2018 ) . Critical reviews differ from case surveys and meta-analyses in two important ways.

First, meta-analytic reviews aggregate evidence on causal relationships to generate evidence-based recommendations, while critical reviews critically evaluate (methodological) issues. Critical reviews are not done to support evidence-based practice or to summarize evidence for a theory. Critical reviews are part of the meta-scientific discourse , that is, the conversations that a scientific community has internally about how it conducts research.

Second, for many critical reviews, including all relevant primary studies is impossible and unnecessary. For example, a critical review of adherence to the Introduction, Method, Results and Discussion framework (IMRaD) framework  ( Sollaci and Pereira , 2004 ) could include all software engineering papers ever written. Instead, a random sample of papers from a selection of leading journals and conferences is sufficient because critical reviews do not assess causal claims, therefore publication bias—“what if significant results were published but non-significant results were not?”—is irrelevant.

Analysis performed within a critical review can be quantitative, qualitative, or both. However, critical reviews typically adopt a critical stance; that is, they go beyond mere description and offer specific critiques of the work being reviewed.

2.5 Scoping Review

What is often called a systematic mapping study in software engineering  ( Petersen et al , 2008 ) is generally called a scoping review elsewhere, for example, in health, medicine and psychology.

Scoping ReviewThe purpose of a scoping review is to understand the state of research on a particular topic. This is typically done by mapping primary studies into categories. Therefore, they are often called systematic mapping studies in software engineering research.

Scoping reviews are primarily descriptive; they count the number of studies on a topic. They often organize studies by research method, subtopic, authors, geographical location, publication venue, etc. They often conclude that more research is needed on particular subtopics. For example, Mohanani et al. ( Mohanani et al , 2018 ) mapped primary studies according to which cognitive bias (e.g., confirmation bias) they investigated and in which area of software development (e.g., design, management) they investigated it, then called for more research on debiasing , that is, preventing or mitigating cognitive biases.

AttentionThe problem with scoping reviews is that they typically do not meet any of the core purposes of secondary research.

When comparing scoping reviews to meta-analyses and case surveys , one notices that the latter synthesize the results of many studies to answer specific empirical (often causal) questions about the world. Scoping reviews include a similar search, but typically do not provide sufficient quantitative synthesis to answer important empirical questions. Therefore, scoping reviews do not inform evidence-based practice as meta-analyzes and case surveys do. Meta-synthesis involves deep, theory-oriented reinterpretation of related qualitative studies. Although scoping reviews can include qualitative analysis (e.g., mapping or categorization), that analysis is often too superficial to generate novel and useful theories.

Critical reviews use a sample of papers to demonstrate an important pattern for the internal discourse of a scientific community. While scoping reviews often give recommendations regarding future research, they focus on an empirical topic (e.g., cognitive biases in SE); not a meta-scientific topic (e.g., construct validity); therefore, they are not configured, from the outset, to deliver useful meta-scientific critique.

In summary, scoping reviews begin like other kinds of systematic reviews but stop short of synthesizing the data into aggregate empirical results, theory, or meta-scientific critique. This is by definition: If a scoping review applies a meta-analytic model to aggregate primary study results, or applies hermeneutics and dialectics to synthesize qualitative accounts, or develops an evidence-based critique of a scientific practice, it is no longer a scoping review; it is a meta-analysis, a case survey, a meta-synthesis, or a critical review. Therefore, some authors recommend a scoping review “as a precursor to a systematic review”  ( Munn et al , 2018 ) .

2.6 Rapid Review

Before conducting a systematic literature review, one should be aware that they require a considerable time investment. In some situations, decision-making cannot wait until this time-consuming process is finished.

Rapid ReviewA rapid review is a meta-analysis that makes methodological compromises to reduce the completion time  ( Ganann et al , 2010 ) .

Ganann et al. found many such compromises including restricting the literature search, truncating results, omitting techniques for overcoming publication bias (e.g., reference snowballing), streamlining screening and data extraction, and skipping quality assessment  ( Ganann et al , 2010 ) . Some authors argue that rapid reviews are a suitable means for practitioners to provide evidence in a timely manner  ( Cartaxo et al , 2020 ) . However, from a scientific point of view, rapid reviews are justified if and only if evidence is needed to support imminent decisions, and waiting for a comprehensive meta-analytic review would be harmful. These conditions occur in health and medicine, for example, when an unprecedented viral pandemic strikes. These conditions do not occur frequently in software engineering.

AttentionThe term rapid review should not be used to legitimize bad systematic reviews where there is no urgent need for immediate results.

In general, software-engineering-related topics rarely have so many primary studies that it would take more than a year to complete a comprehensive review. Therefore, in scholarly research, rapid reviews should be the exception.

3 Guidelines

After the previous section introduced different types of literature review, this section provides advice for selecting, performing, and evaluating them. We start by describing the overall literature review process (Section  3.1 ) and then continue to discuss how students can use ad hoc reviews and scoping reviews to screen literature (Section  3.2 ) and how they can subsequently reflect on which secondary research method might be appropriate (Section  3.3 ). We continue with recommendations on performing secondary research (Section  3.4 ) and conclude with advise on evaluating literature reviews (Section  3.5 ). Rather than comprehensive guidelines, we outline the main pitfalls as well as anti-patterns, and highlight crucial aspects.

Learning GoalsThe learning goals of a lecture on literature reviews are that students (1) understand the overall process, (2) can independently conduct ad hoc and scoping reviews according to our suggestions, (3) are aware of the existing secondary research methods, and (4) can evaluate existing literature reviews. Actually performing secondary research is out of scope for such a lecture. If a teacher wants to expand the scope of the lecture, we suggest focusing on case surveys rather than meta-analysis and meta-synthesis (reasons for that are mentioned throughout this section).

3.1 The Literature Review Process

Figure  1 outlines the process of selecting appropriate literature review types at different stages of a student’s research project. In some cases, one might start with a rough research idea and then perform an ad hoc review to assess the idea and its novelty, informally gathering a first overview of existing related work. This information can then be used to formulate more specific research questions, which could also be defined without first conducting an ad hoc review. Having defined specific research questions, the next step would be to perform a scoping review to thoroughly understand the research on the topic of interest.

In-class SuggestionIn class, the teacher can demonstrate how tools such as Google Scholar and DBLP can be used for ad hoc reviews . The teacher can then share a research idea that students are asked to explore in a buzz group setting. The resulting papers are jointly collected, e.g., via Etherpad . Based on the resulting list, the teacher can discuss additional aspects of ad hoc reviews, such as selecting only peer-reviewed articles.

After the initial screening phase is completed, a researcher might decide that no further literature review is required. The next steps would then be to plan and conduct a novel study against the background of the identified related work. Reasons for not conducting a scoping review after the ad hoc review include:

The ad hoc review yielded existing studies that are so close to the research idea or research questions that one can either continue planning a replication study or decide that the idea is not worth pursuing.

The ad hoc review yielded recent scoping reviews on the topic that can be used to further plan the intended study.

The ad hoc review yielded recent secondary research on the topic that needs to be further screened.

The research question is of meta-scientific nature and warrants a critical review (see Sections  2.4 and 3.4 ).

It is important to explain to students that, generally, a larger research project should always involve a scoping review . Finding very similar related work late in the research process, limiting the novelty of an ongoing study, is very demotivating, not only for students.

Critical reviews are a special case, because they are often performed on a random sample of papers selected from specific venues in a defined period of time. As motivated in Section  2.4 , including all relevant papers is often infeasible for critical reviews, and therefore a scoping review is not a logical predecessor of a critical review.

If the ad hoc review warrants performing a scoping review , the resulting list of articles needs to be screened and the researcher needs to decide which form secondary research is suitable (Section  3.3 ) or if the scoping review itself is sufficient. Reasons for not conducting secondary research include:

The research methods of the identified papers are too diverse.

The research questions of the identified papers are not aligned.

Papers lack statistics or other information required to perform secondary research.

The quantity of the identified papers is too low.

In-class SuggestionTo underline the diversity of research methods and ways of presenting empirical studies in software engineering , the teacher might contrast papers on human aspects of software engineering, mining software repositories, and software testing (to name a few examples). Papers in psychology or medical research can serve as examples for disciplines with more uniform research methods and a more standardized structure for presenting empirical studies.

3.2 Screening Literature

In this section, we focus on the screening phase of the process outlined in Figure  1 . What both ad hoc reviews and scoping reviews have in common is that they should start with a clearly formulated research idea, purpose, goal, or question (see also the chapter Research Design in Software Engineering by Molléri and Petersen).

In-class SuggestionIn class, the teacher might provide a counterexample, that is, a vague and ambiguous research goal , and then refine it together with the students.

Based on the research idea or question, the next step is to define a search strategy that typically includes an initial set of keywords, inclusion/exclusion criteria, and the search engines to use. Popular choices for computer science research are Google Scholar and DBLP . Semantic Scholar is another freely available search engine. Commercial options include Web of Science and Scopus .

Some portals support backward and/or forward snowballing . Backward snowballing refers to analyzing the papers that an included paper cites, while forward snowballing refers to analyzing papers that cite an included paper. Guidelines for snowballing in literature reviews  ( Wohlin , 2014 ) and for combining keyword-based portal searches with snowballing  ( Wohlin et al , 2022 ) are available.

The inclusion and exclusion criteria define which paper the researchers are interested in. Typical filters restrict the time span (e.g., only papers published in the last five years) or exclude articles not meeting certain standards (e.g., non-peer-reviewed articles or book chapters). If keywords have different meaning in different contexts/disciplines, the inclusion/exclusion criteria can clarify the instances that the researchers want to consider.

In-class SuggestionBased on the previously defined research goal, the teacher can guide students through the definition of an initial set of keywords and inclusion/exclusion criteria . It is important to stress that both the keywords and the criteria can evolve in the course of a research project. However, all changes need to be documented.

Conducting ad hoc or scoping reviews involves reading many paper titles, abstracts, and sections. It should be stressed how important deep reading is for research. Knowing when to skim over (parts of) a paper is a skill that students have to develop over time. In the beginning, students should err on the side of reading more and skimming less, even if it slows them down. We are well aware of the pressure to publish early and much, but this is hard to do without a deep understanding of the field. Part of that understanding is, besides published research articles, being able to distinguish scientific evidence from opinionated articles, “laws”, and recommendations. Interesting examples in this regard are the Gartner Hype Cycle , which claimed that software engineering was at its “peak of inflated expectations” in 2023, over 55 years after the first NATO Conference on Software Engineering  ( Naur and Randell , 1969 ) , or the consulting company McKinsey’s claim that their consultants can measure software developer productivity, despite scientific research suggesting a much more nuanced view  ( Sadowski and Zimmermann , 2019 ) . Finally, students must understand that ad hoc and scoping reviews are usually not a study on their own, but a means of preparing themselves to perform novel studies or replications.

Exercise SuggestionAn exercise could be to let students conduct a limited ad hoc review at home, using the portals, keywords, and criteria discussed during the lecture. Identifying a fixed number of papers (e.g., up to 20) and then summarizing a subset (e.g., three of them) in their own words, without repeating sentences from the abstracts, could be reasonable guardrails. Bonus exercise: include snowballing.

3.3 Reflecting on Screened Literature

In this section, we focus on the decision points in Figure  1 . In Section  3.1 , we already described in which cases one might not perform a scoping review after an ad hoc review or not perform secondary research after a scoping review . Here, we will focus mainly on the selection of appropriate secondary research methods.

As mentioned above, critical reviews are a special case. They are meta-scientific in nature and analyze a sample of qualitative or quantitative primary studies to support an argument or critique (see Sections  2.4 ). They would usually not be done after a scoping review, but instead. We do not expect students to conduct a critical review early in their careers, and therefore we only refer to our recommendations in Section  3.4 .

Assuming that the students have conducted a scoping review , the question arises how they can decide whether the corpus of identified articles warrants secondary research. If there is indeed a considerable body of knowledge, most studies are quantitative, and the goal is to test one or more hypotheses, the next step would be to perform meta-analysis (see Sections  2.1 and 3.4 ). If most studies are qualitative and the goal is theory building, performing meta-synthesis is possible (see Sections  2.2 ), given that the researcher has the appropriate expertise and experience (see 3.4 ). If most studies are case studies and the objective is to test hypotheses, students might consider conducting a case survey (see Sections  2.3 and 3.4 ).

In a lecture on literature reviews, the concepts of meta-analysis and meta-synthesis can be presented, but we consider case surveys to be the most useful type of secondary research for students in software engineering. In the following section, we elaborate on this further.

In-class SuggestionTeachers can stress that, unfortunately, secondary research is rare in software engineering . In class, best-practice examples from software engineering can be shown along with examples from other fields.

3.4 Literature Review Recommendations

As mentioned above, performing meta-analysis or meta-synthesis will probably be too challenging for most graduate students. Performing critical reviews as a student is also rare. We provide recommendations for all of these types of secondary research, but would like to stress that, besides conducting ad hoc reviews and scoping reviews , case surveys are most appropriate for software engineering graduate students and hence teachers should focus on those three methods while outlining the others.

In-class SuggestionFocus on ad hoc reviews , scoping reviews , and case surveys but also summarize meta-analysis and meta-synthesis , making clear that these are advanced secondary research methods usually out of scope for graduate students in software engineering.

Meta-analysis

In our experience, the overwhelming problem with meta-analysis in software engineering is that, for a given topic, we almost never have enough similar quantitative studies to aggregate statistically. The research questions and statistics methods are usually too diverse, or essential properties such as effect sizes are missing. Moreover, software engineering research can be performed in very diverse contexts, further limiting the comparability of studies. Therefore, an attempt to conduct a true meta-analysis almost always devolves into a scoping review.

In addition, meta-analysis is hard to do well by one person alone. Multiple researchers need to perform the abstract-and-title screening and then the full-text screening. They need to double-enter all the data, which subsequently needs to be compared and aligned. Hence, this is not something a graduate student can and should do alone. One needs a team.

Moreover, meta-analysis is very time-consuming and often takes more than a year. This results in pragmatic issues such as new papers being published in the process of performing the meta-analysis, but also during the paper review process after submission. The dataset must be constantly kept up-to-date, and the statistical methods used in the meta-analysis should be automated so that statistics can be regularly updated.

Our specific recommendations for students (and supervisors) who nevertheless want to conduct a meta-analysis are:

If you want to conduct a meta-analysis, you need to read dedicated text books on the topic (e.g., ( Borenstein et al , 2021 ) ). A solid background in statistics is helpful.

Do not vote-count (see Section  2.1 ).

It is possible to statistically model study quality instead of excluding low-quality studies. Refer to the empirical standards  ( Ralph et al , 2020 ) to assess study quality.

In summary, the situation around meta-analysis in software engineering is a chicken-and-egg problem. We are convinced that software engineering needs more meta-analysis, but, as we just motivated, this kind of secondary research is very hard to do and the success of students in that endeavor is improbable, because research and reporting quality in software engineering do not meet the standards of other disciplines such as medical research or psychology.

Meta-synthesis

Meta-synthesis requires a strong appreciation of qualitative research. Crucial to this is understanding constructivism and interpretivism. Without this understanding, meta-synthesis is virtually impossible. Moreover, meta-synthesis should only be attempted by people with considerable experience in conducting qualitative studies. Again, without such experience, meta-synthesis is virtually impossible.

Ideally, to do meta-synthesis well, researchers need to understand the differences between different qualitative research traditions (e.g., case study, ethnography, grounded theory, phenomenology, critical theory). Very few researchers in software engineering have such a deep methodological understanding, because the methods are rooted in other disciplines; hence, “method slurring” is common  ( Stol et al , 2016 ) . Although evidence standards  ( Ralph et al , 2020 ) can be used to assess study quality, assessing the quality of qualitative research can be more difficult because there are not as many obvious red flags, such as missing effect sizes. All this further complicates meta-synthesis.

Finally, when performing a meta-synthesis, one must organize the findings of the primary studies into reasonable narratives without forcing one’s own expectations or assumptions onto the data. Auto-reflection  ( Walsh , 2003 ) is crucial. In summary, our recommendation is to only attempt meta-synthesis after:

reading many books on qualitative methods,

conducting several qualitative studies,

reading at least one book on meta-synthesis (e.g., ( Jensen and Allen , 1996 ) ),

reading many studies from a range of qualitative research traditions.

Unfortunately, having a teacher who has done all of the above is not enough. The primary analyst needs the expertise that comes from a combination of first-hand experience and second-hand study. Therefore, meta-synthesis is not a great choice for most graduate students unless an experienced supervisor is very engaged in the analysis.

Case Survey

In our opinion, case surveys are a great option for graduate students because case surveys are more structured than meta-synthesis but less difficult to perform than meta-analysis . One does not need to know as much about research methods as for a critical review . In our experience, case study is one of the most common research methods in software engineering, so there are a lot of good—and not so good—examples out there.

One problem is that, as mentioned in the previous section, method slurring is a problem in software engineering research, and this also applies to case studies . As a researcher, one needs to have clear rules to decide what is a case study (to include it) and what is not (to exclude it). One cannot make this decision based on the presence (or absence) of the words “case study” alone. There are different schools of thought on what constitutes a case study .

Yin  ( Yin , 2009 ) argued that a case must have clear evidence of triangulation across different kinds of data (e.g., interviews, online discussion threads, change logs); whereas Bullock and Tubbs  ( Bullock and Tubbs , 1987 ) included just about any account of the phenomenon of interest. Another way is to include only articles that clearly identify a site that the researchers visited. This third way is good for excluding a lot of low-quality pseudo-cases but is problematic for studies of remote-first teams and historical cases.

A problem is that, while guidelines exist  ( Runeson and Höst , 2009 ; Runeson et al , 2012 ) , software engineering research does not have a history of rich description in case study research that, for example, sociology has. Therefore, students might run into the problem that the analyzed articles do not report the required details.

In-class SuggestionGiven the large number of case studies in software engineering research a and the existence of guidelines and examples, teachers should focus on this secondary research method and present the existing guidelines and best-practice examples.

For case surveys , the statistics are simpler than for meta-analysis , because the resulting data set is much more sparse and statistics is limited to fundamental methods such as correlations. Structural equation modeling (SEM) or Partial least squares regression (PLS) are usually not possible to use for the reasons outlined above.

Our recommendation is that all data extraction should be performed by two researchers independently and then compared and aligned to ensure quality and assess reliability. Bullock and Tubbs  ( Bullock and Tubbs , 1987 ) recommend asking all the authors of primary studies to fill out the data collection for their papers in a survey, so that they act as a second coder and can enter information they know that was not reported in the article. However, it is not easy to motivate the authors to do this (or to contact them in the first place), especially for older papers. It might help if not the student, but a senior supervisor that is well known in the field, sends out the invites but even then the response rate will most likely be far away from 100%.

Critical Review

In theory, critical reviews can be a great choice for a graduate student because doing a critical review usually means analyzing the research methods used in an area, finding all the common problems, and figuring out how to do a method better. This can be a great prelude to conducting one’s own research because it helps the student avoid all these pitfalls they identified in the critical review and hence design better studies.

However, critical reviews suffer from some of the same problems as meta-analysis —you need a team and the analysis can be time consuming. Moreover, performing a critical review on a method that the student has not or only rarely used themselves, is challenging. An advantage of critical reviews is that they do not require a comprehensive sample that includes—in the best case—all studies on a topic. One only needs a defensible sample of studies. Therefore, the sampling and analysis of paper does not take as long as for meta-analysis and meta-synthesis , and a slightly outdated sample does not affect the critical review as much.

The biggest challenges we have encountered with critical reviews are that:

Most students (and also many supervisors) have insufficient research methods training to assess the primary studies. One reason for that is also the diversity of research methods in software engineering, as motivated above.

Reviewers may be unfamiliar with critical reviews and therefore treat them as scoping reviews, expecting the authors to include all relevant studies. As described in Section  2.4 , this is not required for critical reviews.

Critical reviews have ethical implications (see below).

Attention Ethics: It is hard to critique primary studies without publicly blaming the authors. We recommend to describe problems without citing the primary studies that exemplify those problems. One can state explicitly that this is done because the additional credibility of pointing to examples does not justify the hurt feelings of the authors and the potential impact on the reputation of the researchers conducting the review. A risk for us as authors is that reviewers might not follow this argument.

3.5 Evaluating Literature Reviews

Most of the time, students will read or review literature reviews at some point during their studies. Therefore, we want to teach them which aspects to look out for when assessing the quality of literature reviews.

Learning to assess the quality of existing research is a crucial skill for any scientist. We cannot infer that a study is good just because it was peer reviewed and published in a reputable venue (or bad because it is a preprint or published in an outlet of ill repute). The criteria described in this section are adapted from the ACM SIGSOFT Empirical Standards for Software Engineering   ( Ralph et al , 2020 ) . Rather than repeating all the criteria, we will focus on the most common red flags.

Evaluating Ad Hoc and Scoping Reviews

An ad hoc review is what students would report in the related work section of a research paper. They are not a standalone publication. Similarly, scoping reviews are basically pilot studies on the way to conducting a real systematic review. However, given the diversity of research methods and the issues with reporting empirical studies in software engineering, which we described above, they are often published as standalone articles in journals or full-paper conference tracks. This is uncommon in other disciplines and will hopefully change as secondary research in software engineering matures  ( Ralph and Baltes , 2022 ) .

The most common red flag in an ad hoc review is the vague feeling that the authors have not actually read, in detail, the papers being discussed. When the reader is familiar with some of the reviewed works, and the authors’ comments on them suggest serious misunderstandings, the reader worries that the review is superficial.

In contrast, the authors of scoping reviews do not necessarily read the primary studies in depth; they rigorously extract specific data from the studies. Some things that suggest low quality in a scoping review are:

The process of selecting the primary studies is not described in sufficient detail that other researchers could replicate it.

There is no public data set.

The authors did not apply techniques for mitigating sampling bias  ( Baltes and Ralph , 2022 ) (e.g., reference snowballing).

Evaluating Meta-analysis

Meta-analysis usually involves estimating the effect size of one or more causal relationships by creating a statistical meta-model to aggregate the results of numerous similar studies. Two of the biggest validity threats to meta-analysis are publication bias and the quality of the primary studies. This leads to several common red flags:

The study does not have a statistical meta-model, or the model does not take into account differences in primary study size and quality. Sometimes, there might be no meta-model because the paper is a scoping review masquerading as meta-analysis.

Insufficient attempts to find all relevant studies (e.g., overly restrictive search terms, no reference snowballing).

Insufficient attempts to mitigate and asses publication bias (e.g., including pre-prints and dissertations)

No discussion of inter-rater reliability for abstract/title and full-text screening.

Missing or incomplete replication package, including data set.

For a really quick and dirty assessment, if the meta-analysis paper does not have a PRISMA diagram 2 2 2 https://www.prisma-statement.org/prisma-2020-flow-diagram (showing study selection) or a funnel plot  ( Sterne and Harbord , 2004 ) (visualizing publication bias), it is probably of low quality.

Evaluating Meta-synthesis

The synthesis of qualitative research does not involve complicated statistical models, but it does involve a complicated process of comparing, contrasting, and translating concepts and findings between studies. Qualitative studies often use (and invent) different concepts to describe and explain related phenomena, so synthesizing the findings involves retelling one paper’s narratives in the languages of other papers. This is much more difficult than it sounds, so a good meta-synthesis explains— in vivid detail —how the primary studies were coded, compared, contrasted, translated, and (eventually) integrated into themes. Red flags include:

The themes are not clearly defined, vividly explained, and grounded in many quotations from the primary studies,

The data analysis process is only briefly or superficially explained.

Evaluating Case Surveys

Case surveys are quite similar to meta-analysis except that a case survey’s statistical meta-model is much simpler. Each case study has the same size ( n = 1 𝑛 1 n=1 italic_n = 1 ), so we do not model study size. Since we are only concerned with the facts of the case (not the interpretations), we do not exclude low-quality cases or model study quality unless we have reason to believe a case was fabricated. However, there are red flags to watch out for, including the following:

No discussion of inter-rater reliability for abstract and full-text screening; no discussion of quality control during data extraction.

Insufficient a priori justification of hypotheses.

About that last red flag: case surveys lend themselves to HARKing (Hypothesizing After Results are Known). In an ideal world, case surveys, including all of their hypotheses, would be publicly pre-registered such that HARKing is easily identifiable and thus discouraged. Without pre-registration, however, we can only look for how well each hypothesis is justified conceptually, or based on previous research.

Evaluating Critical Reviews

At risk of repeating ourselves, critical reviews do not need to include every study that meets the selection criteria. Critical reviews often just take a (stratified) random sample of papers from some venues the authors consider reputable, and—for the scope of a critical review —that is fine, because the point of a critical review is to make some argument about the way research is done in some area, not to estimate the effect size of a causal relationship. Publication bias is usually irrelevant; therefore, critical reviews do not need PRISMA diagrams or funnel plots. Critical reviews do not exclude low-quality studies because the whole point is to critique the quality of the primary studies.

The quality of a critical review is all about the critique itself. A good critical review identifies several problems with existing research and suggests specific ways to address these problems. Common red flags include:

The critique seems superficial.

The suggested solutions are intractable.

The authors of the critical review do not seem to be experts in the method they are critiquing.

An example of the last point: a critical review of experimental designs mixing up quasi-experiments with true experiments would be a serious red flag.

4 Conclusion

As described in Section  3 , the learning goals of a graduate lecture on literature reviews should be that students:

understand the overall literature review process and can reproduce Figure  1 capturing that process,

can independently conduct ad hoc and scoping reviews according to our suggestions in Sections  2 and 3 ,

are aware of the secondary research methods, that is, they can reproduce the description of meta-analysis , meta-synthesis , and case survey ,

can evaluate existing literature reviews according to the recommendations in Section  3.5 .

As mentioned earlier, if a teacher wants to expand the scope of the lecture, we suggest focusing on case surveys rather than meta-analysis and meta-synthesis because case studies are common in software engineering research and the barriers to performing that kind of secondary research are lower for students. We hope that this chapter, together with the in-class and exercise suggestions, will support teachers in preparing a lecture on literature reviews.

• Arksey and O’Malley (2005) Arksey H, O’Malley L (2005) Scoping studies: towards a methodological framework. International Journal of Social Research Methodology 8(1):19–32
• Bajwa et al (2017) Bajwa SS, Wang X, Duc AN, Abrahamsson P (2017) “failures” to be celebrated: an analysis of major pivots of software startups. Empirical Software Engineering 22(5):2373–2408
• Baltes and Ralph (2022) Baltes S, Ralph P (2022) Sampling in software engineering research: A critical review and guidelines. Empirical Software Engineering 27:article number 94
• Borenstein et al (2021) Borenstein M, Hedges LV, Higgins JP, Rothstein HR (2021) Introduction to Meta-Analysis. John Wiley & Sons, Chichester, GBR
• Bullock and Tubbs (1987) Bullock R, Tubbs ME (1987) The case meta-analysis method for od. In: Research in organizational change and development, vol 1, JAI Press, Inc., pp 171–228
• Cartaxo et al (2020) Cartaxo B, Pinto G, Soares S (2020) Rapid reviews in software engineering. In: Felderer M, Travassos GH (eds) Contemporary Empirical Methods in Software Engineering, Springer International Publishing, Cham, pp 357––384, DOI  10.1007/978-3-030-32489-6“˙3 , URL https://doi.org/10.1007/978-3-030-32489-6_3
• Chalmers (1993) Chalmers I (1993) The cochrane collaboration: preparing, maintaining, and disseminating systematic reviews of the effects of health care. Annals of the New York Academy of Sciences 703(1):156–165
• Cruzes and Dybå (2011) Cruzes DS, Dybå T (2011) Research synthesis in software engineering: A tertiary study. Information and Software Technology 53(5):440–455
• Ganann et al (2010) Ganann R, Ciliska D, Thomas H (2010) Expediting systematic reviews: methods and implications of rapid reviews. Implementation Science 5(1):1–10
• Garousi et al (2019) Garousi V, Felderer M, Mäntylä MV (2019) Guidelines for including grey literature and conducting multivocal literature reviews in software engineering. Inf Softw Technol 106:101–121, DOI  10.1016/J.INFSOF.2018.09.006 , URL https://doi.org/10.1016/j.infsof.2018.09.006
• Glass (1976) Glass GV (1976) Primary, secondary, and meta-analysis of research. Educational Researcher 5(10):3–8
• Gough et al (2017) Gough D, Oliver S, Thomas J (2017) An Introduction to Systematic Reviews. Sage, London, GBR
• Hannay et al (2009) Hannay JE, Dybå T, Arisholm E, Sjøberg DI (2009) The effectiveness of pair programming: A meta-analysis. Information and Software Technology 51(7):1110–1122
• Huang et al (2018) Huang X, Zhang H, Zhou X, Babar MA, Yang S (2018) Synthesizing qualitative research in software engineering: A critical review. In: Proceedings of the 40th International Conference on Software Engineering, Association for Computing Machinery, New York, NY, USA, ICSE ’18, p 1207–1218, DOI  10.1145/3180155.3180235 , URL https://doi.org/10.1145/3180155.3180235
• Jensen and Allen (1996) Jensen LA, Allen MN (1996) Meta-synthesis of qualitative findings. Qualitative Health Research 6(4):553–560
• Jurisch et al (2013) Jurisch MC, Wolf P, Krcmar H (2013) Using the case survey method for synthesizing case study evidence in information systems research. In: Proceedings of the 19th Americas Conference on Information Systems (AMCIS), Association for Information Systems, Chicago, USA, article 5
• Kitchenham et al (2004) Kitchenham BA, Dyba T, Jorgensen M (2004) Evidence-based software engineering. In: Proceedings of the 26th International Conference on Software Engineering, IEEE Computer Society, USA, ICSE ’04, p 273–281
• Larsson (1993) Larsson R (1993) Case survey methodology: Quantitative analysis of patterns across case studies. Academy of Management Journal 36(6):1515–1546
• Lucas (1974) Lucas WA (1974) The case survey method: Aggregating case experience. Tech. Rep. 1515, Rand, URL https://www.rand.org/pubs/reports/R1515.html
• Maybee (2020) Maybee JE (2020) Hegel’s Dialectics. In: Zalta EN (ed) The Stanford Encyclopedia of Philosophy, Winter 2020 edn, Metaphysics Research Lab, Stanford University
• Melegati and Wang (2020) Melegati J, Wang X (2020) Case survey studies in software engineering research. In: Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), Association for Computing Machinery, New York, NY, USA, ESEM ’20, DOI  10.1145/3382494.3410683 , URL https://doi.org/10.1145/3382494.3410683
• Mohanani et al (2018) Mohanani R, Salman I, Turhan B, Rodríguez P, Ralph P (2018) Cognitive biases in software engineering: a systematic mapping study. IEEE Transactions on Software Engineering 46(12):1318–1339
• Munn et al (2018) Munn Z, Peters MD, Stern C, Tufanaru C, McArthur A, Aromataris E (2018) Systematic review or scoping review? guidance for authors when choosing between a systematic or scoping review approach. BMC Medical Research Methodology 18(1):1–7
• Naur and Randell (1969) Naur P, Randell B (1969) Software Engineering: Report of a conference sponsored by the NATO Science Committee, Garmisch, Germany, 7-11 Oct. 1968, Brussels, Scientific Affairs Division, NATO
• Petersen (2020) Petersen K (2020) Guidelines for case survey research in software engineering. In: Felderer M, Travassos GH (eds) Contemporary Empirical Methods in Software Engineering, Springer International Publishing, Cham, pp 63–92, DOI  10.1007/978-3-030-32489-6“˙3 , URL https://doi.org/10.1007/978-3-030-32489-6_3
• Petersen et al (2008) Petersen K, Feldt R, Mujtaba S, Mattsson M (2008) Systematic mapping studies in software engineering. In: Proceedings of the 12th International Conference on Evaluation and Assessment in Software Engineering, BCS Learning & Development Ltd., Swindon, GBR, EASE’08, p 68–77
• Petersen et al (2017) Petersen K, Badampudi D, Shah SMA, Wnuk K, Gorschek T, Papatheocharous E, Axelsson J, Sentilles S, Crnkovic I, Cicchetti A (2017) Choosing component origins for software intensive systems: In-house, cots, oss or outsourcing?—a case survey. IEEE Transactions on Software Engineering 44(3):237–261
• Purssell and McCrae (2020) Purssell E, McCrae N (2020) How to Perform a Systematic Literature Review: A Guide for Healthcare Researchers, Practitioners and Students. Springer International Publishing, Cham, DOI  10.1007/978-3-030-49672-2“˙2 , URL https://doi.org/10.1007/978-3-030-49672-2_2
• Rafique and Mišić (2012) Rafique Y, Mišić VB (2012) The effects of test-driven development on external quality and productivity: A meta-analysis. IEEE Transactions on Software Engineering 39(6):835–856
• Ralph (2018) Ralph P (2018) Toward methodological guidelines for process theories and taxonomies in software engineering. IEEE Transactions on Software Engineering 45(7):712–735
• Ralph and Baltes (2022) Ralph P, Baltes S (2022) Paving the way for mature secondary research: the seven types of literature review. In: Roychoudhury A, Cadar C, Kim M (eds) Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, Singapore, Singapore, November 14-18, 2022, ACM, pp 1632–1636, DOI  10.1145/3540250.3560877 , URL https://doi.org/10.1145/3540250.3560877
• Ralph et al (2020) Ralph P, Baltes S, Bianculli D, Dittrich Y, Felderer M, Feldt R, Filieri A, Furia CA, Graziotin D, He P, et al (2020) Acm sigsoft empirical standards. arXiv: 201003525 [csSE]
• Ricoeur (1981) Ricoeur P (1981) Hermeneutics and the Human Sciences: Essays on Language, Action and Interpretation. Cambridge University Press, New York
• Runeson and Höst (2009) Runeson P, Höst M (2009) Guidelines for conducting and reporting case study research in software engineering. Empir Softw Eng 14(2):131–164, DOI  10.1007/S10664-008-9102-8 , URL https://doi.org/10.1007/s10664-008-9102-8
• Runeson et al (2012) Runeson P, Höst M, Rainer A, Regnell B (2012) Case Study Research in Software Engineering - Guidelines and Examples. Wiley, URL http://eu.wiley.com/WileyCDA/WileyTitle/productCd-1118104358.html
• Sadowski and Zimmermann (2019) Sadowski C, Zimmermann T (eds) (2019) Rethinking Productivity in Software Engineering. Apress open / Springer, DOI  10.1007/978-1-4842-4221-6 , URL https://doi.org/10.1007/978-1-4842-4221-6
• Shepperd et al (2014) Shepperd M, Bowes D, Hall T (2014) Researcher bias: The use of machine learning in software defect prediction. IEEE Transactions on Software Engineering 40(6):603–616
• Sollaci and Pereira (2004) Sollaci LB, Pereira MG (2004) The introduction, methods, results, and discussion (imrad) structure: a fifty-year survey. Journal of the Medical Library Association 92(3):364
• Sterne and Harbord (2004) Sterne JAC, Harbord RM (2004) Funnel plots in meta-analysis. The Stata Journal 4(2):127–141, DOI  10.1177/1536867X0400400204
• Stol et al (2016) Stol KJ, Ralph P, Fitzgerald B (2016) Grounded theory in software engineering research: A critical review and guidelines. In: Proceedings of the 38th International Conference on Software Engineering, Association for Computing Machinery, New York, NY, USA, ICSE ’16, p 120–131, DOI  10.1145/2884781.2884833 , URL https://doi.org/10.1145/2884781.2884833
• Walsh (2003) Walsh R (2003) The methods of reflexivity. The Humanistic Psychologist 31(4):51–66, DOI  10.1080/08873267.2003.9986934
• Wohlin (2014) Wohlin C (2014) Guidelines for snowballing in systematic literature studies and a replication in software engineering. In: Shepperd MJ, Hall T, Myrtveit I (eds) 18th International Conference on Evaluation and Assessment in Software Engineering, EASE ’14, London, England, United Kingdom, May 13-14, 2014, ACM, pp 38:1–38:10, DOI  10.1145/2601248.2601268 , URL https://doi.org/10.1145/2601248.2601268
• Wohlin et al (2022) Wohlin C, Kalinowski M, Felizardo KR, Mendes E (2022) Successful combination of database search and snowballing for identification of primary studies in systematic literature studies. Inf Softw Technol 147:106908, DOI  10.1016/J.INFSOF.2022.106908 , URL https://doi.org/10.1016/j.infsof.2022.106908
• Yin (2009) Yin RK (2009) Case study research: Design and methods, vol 5. sage
• Yin and Heald (1975) Yin RK, Heald KA (1975) Using the case survey method to analyze policy studies. Administrative Science Quarterly 20(3):371–381, URL http://www.jstor.org/stable/2391997
• Zhang et al (2019) Zhang H, Huang X, Zhou X, Huang H, Babar MA (2019) Ethnographic research in software engineering: A critical review and checklist. In: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Association for Computing Machinery, New York, NY, USA, ESEC/FSE 2019, p 659–670, DOI  10.1145/3338906.3338976 , URL https://doi.org/10.1145/3338906.3338976

Teaching Literature Reviewing for Software Engineering Research

• Baltes, Sebastian
• Ralph, Paul

The goal of this chapter is to support teachers in holistically introducing graduate students to literature reviews, with a particular focus on secondary research. It provides an overview of the overall literature review process and the different types of literature review before diving into guidelines for selecting and conducting different types of literature review. The chapter also provides recommendations for evaluating the quality of existing literature reviews and concludes with a summary of our learning goals and how the chapter supports teachers in addressing them.

• Computer Science - Software Engineering

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

Research Techniques Made Simple: Assessing Risk of Bias in Systematic Reviews

Affiliations.

• 1 Department of Dermatology, Warren Alpert Medical School, Brown University, Providence, Rhode Island, USA. Electronic address: [email protected].
• 2 Division of Dermatology, University of Toronto, Toronto, Ontario, Canada.
• 3 Division of Dermatology, University of Toronto and Women's College Research Institute, Toronto, Ontario, Canada.
• PMID: 27772550
• DOI: 10.1016/j.jid.2016.08.021

Systematic reviews are increasingly utilized in the medical literature to summarize available evidence on a research question. Like other studies, systematic reviews are at risk for bias from a number of sources. A systematic review should be based on a formal protocol developed and made publicly available before the conduct of the review; deviations from a protocol with selective presentation of data can result in reporting bias. Evidence selection bias occurs when a systematic review does not identify all available data on a topic. This can arise from publication bias, where data from statistically significant studies are more likely to be published than those that are not statistically significant. Systematic reviews are also susceptible to bias that arises in any of the included primary studies, each of which needs to be critically appraised. Finally, competing interests can lead to bias in favor of a particular intervention. Awareness of these sources of bias is important for authors and consumers of the scientific literature as they conduct and read systematic reviews and incorporate their findings into clinical practice and policy making.

Copyright © 2016 The Authors. Published by Elsevier Inc. All rights reserved.

PubMed Disclaimer

Similar articles

• Publication bias in dermatology systematic reviews and meta-analyses. Atakpo P, Vassar M. Atakpo P, et al. J Dermatol Sci. 2016 May;82(2):69-74. doi: 10.1016/j.jdermsci.2016.02.005. Epub 2016 Feb 24. J Dermatol Sci. 2016. PMID: 26925817
• Usefulness of Cochrane Skin Group reviews for clinical practice. Davila-Seijo P, Batalla A, Garcia-Doval I. Davila-Seijo P, et al. Actas Dermosifiliogr. 2013 Oct;104(8):679-84. doi: 10.1016/j.adengl.2012.12.009. Epub 2013 Aug 13. Actas Dermosifiliogr. 2013. PMID: 23954044
• [Volume and health outcomes: evidence from systematic reviews and from evaluation of Italian hospital data]. Amato L, Colais P, Davoli M, Ferroni E, Fusco D, Minozzi S, Moirano F, Sciattella P, Vecchi S, Ventura M, Perucci CA. Amato L, et al. Epidemiol Prev. 2013 Mar-Jun;37(2-3 Suppl 2):1-100. Epidemiol Prev. 2013. PMID: 23851286 Review. Italian.
• The contribution of systematic reviews to the practice of pediatric nephrology. Hodson E, Craig JC. Hodson E, et al. Pediatr Nephrol. 2013 Feb;28(2):197-206. doi: 10.1007/s00467-012-2155-3. Epub 2012 Apr 4. Pediatr Nephrol. 2013. PMID: 22476252 Review.
• Dissemination and publication of research findings: an updated review of related biases. Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ, Hing C, Kwok CS, Pang C, Harvey I. Song F, et al. Health Technol Assess. 2010 Feb;14(8):iii, ix-xi, 1-193. doi: 10.3310/hta14080. Health Technol Assess. 2010. PMID: 20181324 Review.
• Systematic review of feedback literacy instruments for health professions students. Mohd Noor MN, Fatima S, Grace Cockburn J, Romli MH, Pallath V, Hong WH, Vadivelu J, Foong CC. Mohd Noor MN, et al. Heliyon. 2024 May 10;10(10):e31070. doi: 10.1016/j.heliyon.2024.e31070. eCollection 2024 May 30. Heliyon. 2024. PMID: 38813152 Free PMC article.
• XR and mental wellbeing: state of the art and future research directions for the Metaverse. Taylor A, Tom Dieck MC, Jung T, Cho J, Kwon O. Taylor A, et al. Front Psychol. 2024 Mar 8;15:1360260. doi: 10.3389/fpsyg.2024.1360260. eCollection 2024. Front Psychol. 2024. PMID: 38524293 Free PMC article. Review.
• A diverse portfolio of marine protected areas can better advance global conservation and equity. Gill DA, Lester SE, Free CM, Pfaff A, Iversen E, Reich BJ, Yang S, Ahmadia G, Andradi-Brown DA, Darling ES, Edgar GJ, Fox HE, Geldmann J, Trung Le D, Mascia MB, Mesa-Gutiérrez R, Mumby PJ, Veverka L, Warmuth LM. Gill DA, et al. Proc Natl Acad Sci U S A. 2024 Mar 5;121(10):e2313205121. doi: 10.1073/pnas.2313205121. Epub 2024 Feb 26. Proc Natl Acad Sci U S A. 2024. PMID: 38408235
• Online peer support for people with Amyotrophic Lateral Sclerosis (ALS): a narrative synthesis systematic review. Gerritzen EV, Lee AR, McDermott O, Coulson N, Orrell M. Gerritzen EV, et al. Front Digit Health. 2024 Jan 31;6:1138530. doi: 10.3389/fdgth.2024.1138530. eCollection 2024. Front Digit Health. 2024. PMID: 38357638 Free PMC article. Review.
• Effects of different triticale inclusion levels on broilers' growth parameters: A meta-analysis. Poaty Ditengou JIC, Cho S, Ahn SI, Chae B, Jeon E, Choi NJ. Poaty Ditengou JIC, et al. Vet Anim Sci. 2023 Dec 15;23:100328. doi: 10.1016/j.vas.2023.100328. eCollection 2024 Mar. Vet Anim Sci. 2023. PMID: 38192373 Free PMC article. Review.

Publication types

• Search in MeSH

LinkOut - more resources

Full text sources.

• Elsevier Science

Other Literature Sources

• scite Smart Citations
• MedlinePlus Health Information

• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

New Software Product Development: Bibliometric Analysis

• Open access
• Published: 15 June 2024

Cite this article

You have full access to this open access article

• Pedro Neves Mata   ORCID: orcid.org/0000-0001-8465-9539 1 ,
• José Moleiro Martins 2 , 3 &
• João Carlos Ferreira 4 , 5 , 6  

77 Accesses

Explore all metrics

The aim of this article is to conduct a comprehensive bibliometric analysis of the existing literature in the domain of new software product development. The research methodology is based on bibliometrics, cluster analysis and meta-analysis of key indicators of knowledge dissemination and publications about the new software product in the Scopus scientometric database for 2003–2022.Through bibliometric analysis, this study provides valuable information on the trends, patterns and impact of research and publications in the field of new software product development. Based on the literature reviewed, known research areas, influential authors and new directions are identified, thus providing a comprehensive overview of the current state of knowledge in the field of new software product development. The number of publications dedicated to new software products has steadily increased over the past two decades, indicating a growing interest in this field among researchers and scholars. The United States, China, and Germany emerged as the leading countries in terms of the number of publications, highlighting their strong presence in software development and technological advancements. Meta-analysis demonstrates the significant impact of Engineering, Mathematics, and Social Sciences on the publication count of the new software product. Overall, the findings highlight the widespread interest and interdisciplinary nature of software product development, with particular emphasis on the role of Computer Science and Engineering in driving advancements in this domain. The conducted bibliometric analysis can serve as a basis for further research and help researchers and practitioners to identify knowledge gaps and directions for future research.

Similar content being viewed by others

Quantity versus impact of software engineering papers: a quantitative study.

A Trend Analysis of Software Business Research

An analysis of top author citations in software engineering and a comparison with other fields

Avoid common mistakes on your manuscript.

Introduction

In today’s fast-paced and interconnected world, software has become an integral part of our daily lives, enabling us to accomplish tasks efficiently, connect with others effortlessly, and access information instantaneously. As technology continues to advance at an unprecedented rate, the demand for innovative software products has surged, leading to a highly competitive landscape in the software development industry. Developing new software products is a complex endeavour. Apart from the necessity of technical expertise, creative thinking and understanding of the market (Appio et al., 2021 ) it also requires a systematic approach. The process is comprised of ideation, planning, design, development, testing, and deployment. The goal is to deliver a solution that meets users’ evolving needs (Cooper & Sommer, 2018 ).

In this study, we delve into the fascinating world of new software product development and explore the key aspects that drive innovation in the digital age. We will examine the challenges faced by software development teams, highlight best practices and methodologies employed throughout the process, and shed light on the transformative impact of emerging technologies in shaping the future of software product development. One of the fundamental challenges in new software product development lies in identifying and understanding the needs and desires of the target audience (Melegati et al., 2019 ). Effective market research and user-centric design methodologies play a crucial role in ensuring that the software product meets the expectations of its intended users. It’s helpful for the developers to gain insight into the preferences, pain points, and behaviours of the users. That way they can create solutions that offer meaningful and delightful experiences (Jansen et al., 2020 ).

The rapid advancement of technology presents both opportunities and hurdles for software development teams. The advent of artificial intelligence, machine learning, cloud computing, and the Internet of Things has opened up new avenues for innovation, enabling the creation of intelligent, scalable, and interconnected software products (Gill et al., 2019 ). However, with technological progress comes the need for continuous learning, adaptation, and keeping pace with the ever-changing landscape.

New software product development is a dynamic and exhilarating journey that requires a harmonious blend of technical expertise, innovation, and market understanding (Parthasarathy, 2022 ). As the digital landscape continues to evolve, software development teams must embrace emerging technologies, adapt to changing user expectations, and foster a culture of continuous learning and improvement (Marion & Fixson, 2021 ). By doing so, they can unlock the full potential of their creativity and bring to life software products that redefine the way we live, work, and interact in the digital age. The aim of this article is to conduct a comprehensive bibliometric analysis of the existing literature in the domain of new software product development. The study conducted a comprehensive bibliometric analysis to understand the trends, patterns, and impacts of research and publications in this field. The main tasks solved during the study are:

Through the analysis of publication patterns, citation networks, and co-authorship collaborations, the study identified the key research trends that have emerged in the field of new software product development. This involved examining the frequency and distribution of research publications over time to uncover the evolution of topics, methodologies, and technologies employed in this area.

By mapping the citation network and analyzing co-citation patterns, the study aimed to identify seminal works and influential factors that have shaped the landscape of new software product development. This analysis provided insights into the most referenced studies, influential authors, institutions, and countries, thereby highlighting the key contributors to the knowledge base in this field.

The study examined the distribution of research across different subtopics to identify potential gaps in the existing literature. This analysis will help researchers and practitioners identify areas that have received less attention or require further exploration, providing valuable directions for future research and development efforts in new software product development.

Through this comprehensive bibliometric analysis, we strive to offer a holistic view of the research landscape in new software product development. By uncovering research trends, influential factors, and potential gaps, this analysis will provide researchers, practitioners, and decision-makers with valuable insights to shape future strategies, foster collaboration, and advance the field of software product development in an informed and impactful manner.

State-of-the-art

Mapping the landscape of software development publication data.

In recent years, the field of software product development has witnessed rapid advancements and transformations, driven by technological innovations, evolving user needs, and changing market dynamics. This literature review explores the key themes that have emerged in the context of new software product development. In the study, a bibliometric analysis methodology was applied, using cluster analysis and meta-analysis of key indicators of knowledge dissemination and publications on the development of new software products. The analysis was based on data from the Scopus scientometric database for the period from 2003 to 2022, covering 22,116 publications (Table  1 ).

The dataset comprises an impressive collection of 22,116 documents sourced from 8,475 journals, books, and other scholarly outlets. This diversity of sources indicates a broad interest and cross-disciplinary engagement in software product development. The annual growth rate of publications stands at 4.09%, pointing to a steady increase in research output. This growth rate, though modest, is significant given the dataset’s starting volume. The average age of the documents is 9.75 years, which suggests that while the field continues to evolve, it also has a substantial body of established knowledge. With an average of 14.37 citations per document, the research on new software product development demonstrates impactful contributions to the field, indicating that the works are widely recognized and referenced within the academic community.

The analysis reveals a rich lexicon of 79,973 “Keywords plus (ID)” and 39,947 “Author’s keywords (DE)”. These figures not only underscore the extensive thematic diversity and specialized focus areas within the field but also highlight the evolving nature of software product development discourse.

The human capital involved in this academic endeavor is significant, with 51,843 authors contributing to the corpus. However, only 2812 of these have published single-authored documents, suggesting a strong tendency towards collaborative research in the field. This tendency is further supported by the fact that there are 3958 single-authored documents compared to a higher average of 3.52 co-authors per document. The percentage of international co-authorships stands at 17.11%, indicating a healthy level of global collaboration, although there is still room for improvement in terms of international cooperation.

The breakdown of document types reveals a dominant preference for articles, which constitute 12,633 of the total documents, underscoring the academic community’s inclination towards traditional scholarly communication. Conference papers also make up a substantial portion with 8759 entries, reflecting the field’s dynamic nature and the importance of conferences as venues for sharing cutting-edge research.

For the cluster analysis, we used VosViewer and Biblioshiny tools in the R environment. These tools helped identify connections between research areas and distinct concepts. Furthermore, it aided the determination of thematic patterns and interconnections in new software product research. These tools contributed to the identification of important domains and directions in the research literature, highlighting the interdisciplinary nature of software product development and the particular focus on the role of computer science and engineering in advancing this field.

Thematic Insights into new Software Product Development

The cluster analysis of publications was carried out using the concept “new software product” in keywords for 2003–2022. Using the VOSviewer software environment, we identified the relationships between studies in this area and distinct concepts (Fig.  1 ). The cluster analysis aimed to uncover the thematic patterns and interconnections within the research landscape concerning new software products. By analyzing the co-occurrence of keywords in the publications, VOSviewer identified clusters of studies that shared similar concepts or themes. These clusters provided insights into the various aspects and subtopics related to new software products. Researchers and practitioners could examine the identified clusters to gain a better understanding of the research trends, emerging topics, and potential collaborations in this area. Moreover, the analysis facilitated the identification of key concepts that were frequently associated with new software products, thereby highlighting the important domains and areas of focus within the research literature.

Cluster analysis of “new software product” publications in the Scopus database from 2003 to 2022.  Source: formed by the author

Throughout the analyzed period, the largest (red) cluster can be identified, which demonstrates the interconnection of research on a new software product in the context of its development, application creation, artificial intelligence, and ensuring the necessary characteristics for its advancement. The remaining clusters mainly reveal the development direction of medical and biotechnological research for the new software product, which integrates engineering and human science. The cluster analysis conducted using VOSviewer helped reveal the relationships and connections between studies focused on new software products and other relevant concepts, thereby enhancing our understanding of the research landscape in this domain.

Agile and lean development methodologies have gained significant attention in the software industry, emphasizing iterative and incremental development, customer collaboration, and rapid delivery. The literature highlights the benefits of these approaches in enabling flexibility, responsiveness, and adaptive product development, thereby reducing time-to-market and enhancing customer satisfaction. The key themes within agile and lean development include continuous integration, continuous delivery, cross-functional teams, and the adoption of agile practices such as Scrum and Kanban (Zayat & Senvar, 2020 ). User-centered design (UCD) focuses on understanding and addressing user needs, preferences, and behaviors throughout the product development lifecycle (Zorzetti et al., 2022 ). The literature reveals the importance of involving end-users in the design process through techniques like personas, user interviews, usability testing, and prototyping (Herumurti et al., 2023 ). The key themes within UCD encompass user research, usability engineering, interaction design, and the integration of user feedback to create intuitive and user-friendly software products (Dopp et al., 2020 ; Park et al., 2022 ).

DevOps practices emphasize collaboration, automation, and continuous integration and deployment (CI/CD) to bridge the gap between development and operations teams. The literature highlights the role of DevOps in streamlining software development, enhancing product quality, and improving deployment processes. Key themes within DevOps include infrastructure automation, version control, automated testing, monitoring, and the use of cloud computing platforms for scalable and reliable deployments. Effective software product management is crucial for successful new product development (Shameem, 2022 ). Literature sheds light on product managers’ roles. They define the product’s vision, strategy, and roadmaps. They also prioritize features according to market demands and customer feedback. Key themes within software product management include requirements engineering, product planning, product lifecycle management, and product analytics for data-driven decision-making (Mishra & Otaiwi, 2020 ; Wiedemann et al., 2019a ; Lwakatare et al., 2019 ).

The literature highlights the impact of emerging technologies, such as artificial intelligence (AI), machine learning, Internet of Things (IoT), and blockchain, on software product development. These technologies offer new opportunities and challenges, including intelligent automation, predictive analytics, enhanced user experiences, and secure and decentralized applications (Gerke et al., 2020 ; Merenda et al., 2020 ). Key themes within emerging technologies include the integration of AI algorithms, IoT connectivity, blockchain-based solutions, and the ethical implications of deploying these technologies in software products (Mugarza et al., 2020 ; Nica & Stehel, 2021 ).

Software product development is increasingly influenced by the emergence of software ecosystems and open-source communities. The literature explores the benefits and challenges of leveraging open-source software components, libraries, and frameworks in new product development. Key themes within software ecosystems and open source include collaborative development models, licensing considerations, community engagement, and the role of open-source platforms in fostering innovation (Coetzee et al., 2020 ). One of the major challenges in new software product development is the rapidly changing technology landscape. The software industry is characterized by constant advancements and evolving trends. Developers must keep up with the latest technologies, programming languages, frameworks, and tools to build cutting-edge software products (Colón-Ramos et al., 2019 ; Salvato & Laplume, 2020 ). Failure to adapt to these changes may result in outdated and less competitive products. Additionally, the integration of new technologies into existing software ecosystems can present compatibility and interoperability challenges. Developing a successful software product requires a deep understanding of market dynamics and customer needs (Marion & Fixson, 2021 ). The market for software products is highly volatile and unpredictable. Identifying the right target audience, determining their needs, and creating a product that resonates with their expectations is a challenging task. Additionally, customer needs and preferences change quickly. Developers need to constantly collect feedback from customers and adjust their products to remain relevant. Failure to do so may result in a product that fails to meet market demand (Bianchi et al., 2020 ).

New software product development often faces resource constraints, including budgetary limitations, time constraints, and limited availability of skilled personnel. Adequate funding is crucial for conducting research, acquiring necessary tools and technologies, and marketing the product effectively (Chiang & Lin, 2020 ; Beecham et al., 2021 ). Software development projects often face tight deadlines, putting pressure on developers to deliver high-quality products within limited timeframes. The shortage of skilled developers and technical expertise can also hamper the development process, leading to delays or compromised quality (Berg et al., 2020 ).

Modern software products are increasingly complex and require integration with various hardware and software components. Developing software products that seamlessly integrate with existing systems while ensuring performance, security, and scalability is a significant challenge. The scale and complexity of software systems can lead to issues such as system crashes, performance bottlenecks, and security vulnerabilities. Ensuring robustness, maintainability, and scalability while managing dependencies and interconnections is crucial but demanding (Thota et al., 2020 ). Ensuring the quality of software products is a critical challenge in new software product development. Inadequate testing and quality assurance processes can result in software products with functional defects, poor user experience, and security vulnerabilities (Saputri & Lee, 2021 ). It’s essential to develop comprehensive testing strategies and automate the testing process. Conducting thorough quality checks throughout the development lifecycle is also crucial. However, these activities require additional time, effort, and resources, which can be challenging to allocate in fast-paced development environments (Issa Mattos et al., 2023 ). New software product development plays a vital role in driving innovation and competitiveness in the rapidly evolving digital landscape. As organizations strive to create cutting-edge software products, it is essential to adopt best practices that can enhance the efficiency and effectiveness of the development process (Cooper, 2019 ).

Agile methodologies have become more popular in recent years as they help increase flexibility and adaptability in software fevelopment. This approach focuses on iterative development, teamwork, and customer feedback, allowing teams to adapt to changes and deliver quality software products (Aldave et al., 2019 ; Najihi et al., 2022 ). By involving users throughout the development process, software teams can ensure that the product meets user expectations and delivers a superior user experience. The concept of a minimum viable product (MVP) involves releasing a product with the minimum set of features necessary to meet user needs and gather feedback. Adopting an MVP approach allows teams to validate assumptions and gather early feedback. This allows the creation of informed decisions about the product’s direction (Tripathi et al., 2019 ; Dennehy et al., 2019 ). This iterative process allows for faster time-to-market and reduces the risk of developing features that users may not find valuable. DevOps practices emphasize collaboration, automation, and continuous improvement between development and operations teams. Adopting a DevOps mindset and using CI/CD pipelines helps automate building, testing, and deployment. This speeds up and improves the reliability of software releases (Almeida et al., 2022 ; Wiedemann et al., 2019a ). This approach ensures that software products are regularly updated, and any issues are addressed promptly.

Efficient project management practices are crucial for successful software product development. Techniques such as scrum, Kanban, and lean project management can help teams streamline development processes, manage resources effectively, and deliver software products on time and within budget. Clear communication, proper task allocation, and regular progress tracking are essential aspects of effective project management. Quality assurance and testing are integral to software product development to ensure that the final product meets the desired quality standards (Zonnenshain & Kenett, 2020 ). Best practices in this area include test-driven development (TDD), continuous testing, and the use of automated testing frameworks. Rigorous testing throughout the development lifecycle helps identify and address bugs and performance issues early, resulting in a more reliable and stable software product (Iqbal & Suzianti, 2021 ).

With the rapid advancement of technology, the creation of new software products is increasingly influenced by the rise of emerging technologies. However, the literature has not sufficiently tackled the specific challenges and strategies inherent in the development of software products within these new domains. Traditional studies have often centered on conventional software development methodologies, failing to grasp the complexity and specific nuances associated with cutting-edge technological advancements. There’s a noticeable gap in literature regarding practical insights, frameworks, and approaches designed expressly for software product development in these burgeoning fields. Bridging this gap requires a detailed examination and synthesis of research that delves into the challenges faced by software development teams as they navigate emerging technologies. Furthermore, it necessitates the identification of effective strategies, methodologies, and frameworks that can serve as a guide for the development of software products within these areas. By addressing this gap, researchers and practitioners can gain a deeper understanding of the unique challenges and opportunities in developing new software products for emerging technology sectors. This understanding will promote informed decision-making. Consequently, it may lead to the creation of successful software products within these innovative areas.

The Common Research Trends

Over the past two decades, the number of publications dedicated to the development of new software products has been steadily increasing. In 2003, only 762 publications were found, but by 2022, the number had reached 1558, which is twice as much. This indicates a growing interest in the field of creating new software products and their development. Figure  2 illustrates the growth of publications with the keyword “new software product” in the Scopus database from 2003 to 2022. The growth rate of publications featuring “new software product” keywords is not constant. While there is an overall upward trend, the rate of growth varies from year to year. Some years show significant increases, while others show smaller or even negative growth. The number of publications experienced some fluctuations between 2007 and 2015. Despite the dynamic growth in the volume of publications in the subject areas of “Computer Science” and “Mathematics,” most other subject areas did not exhibit a consistent growth trend. However, starting from 2016, the volume of publications related to new software products has been steadily increasing for over 5 years. This indicates a growing scientific interest in the field of new software products and the effectiveness of project management in this sphere. Figure  2 illustrates the growth of publications with the keywords “new software product” in the Scopus database from 2003 to 2022.

Increase in the quantity of publications featuring “new software product” keywords in the Scopus database from 2003 to 2022. Source: formed by the author

There is a general upward trend in the number of publications over the years, indicating an increasing interest in new software products among researchers and scholars. From 2003 to 2009, there was a relatively stable period with the number of publications ranging between 762 and 1011. However, from 2010 onwards, there was an increase in the number of publications, with occasional dips in certain years. The steepest increase in the quantity of publications occurred with a significant rise from 1097 in 2014 to 1324 in 2019.

The quantity of publications has shown a gradual increase with some fluctuations. There are fluctuations in the number of publications from year to year, with some years showing higher numbers than others. Notable peaks can be observed in 2018, 2019, 2021, and 2022, where the number of publications reached its highest points. In recent years, from 2019 to 2022, there has been a continuous growth in the number of publications featuring “new software product” keywords, indicating a sustained interest in this area of research. Several factors may explain the rise in publications. Namely, advancements in software development methods, the growing role of software across domains, the rise of open-source software, the spread of software-driven technologies, and more recognition of software as a research area.

Figure  3 presents a map illustrating the geographical representation of publications on the new software product in the Scopus database from 2003 to 2022. The map showcases the distribution of these publications according to their country of origin.

Mapping publication sources in the Scopus database from 2003–2022 with the keywords “new software product”.  Source: compiled by the author

The analysis indicates that the United States has the highest number of publications (4615) related to new software products, followed by China (2412) and Germany (1935). These countries have a significant presence in the software development field and are known for their technological advancements (Fig.  4 ). In addition to the United States, China, and Germany, other notable countries with a considerable number of publications include the United Kingdom (1173), India (979), and Italy (916). This suggests that software development and new software products are of interest and importance in these regions.

Europe shows a strong presence in terms of the number of publications. This highlights Europe’s active involvement in researching and publishing on new software products. In addition to China and India, other countries from the Asia-Pacific region with significant publication numbers include Japan, South Korea, Malaysia, Taiwan, Iran, Singapore, Indonesia, New Zealand, Pakistan, Thailand, and Australia. This indicates a widespread interest and contribution to research on new software products from this region. Brazil and Mexico represent Latin America, while South Africa and Nigeria are the only African countries mentioned. These numbers suggest that more research and publication on new software products are needed from these regions to match the levels seen in other parts of the world. Countries from the Middle East and North Africa include Saudi Arabia, Israel, Egypt, Tunisia, Algeria, and Morocco. While the numbers are relatively lower compared to other regions, it still indicates some level of research activity related to new software products.

Leading countries in research publications within the specified field during 2003–2022.  Source: formed by the author

United States: The United States stands out as the leading country with a significant number of publications, with 4615 publications during the studied period. The US has traditionally been a powerhouse in scientific research and development, with numerous world-renowned universities, research institutions, and funding opportunities. China follows closely behind the United States. Over the years, China has made significant investments in research and development, leading to substantial growth in scientific output. The country has been focusing on bolstering its research infrastructure, attracting talented researchers, and promoting innovation. Germany ranks third. Known for its strong research culture, Germany has a robust academic system with a focus on scientific excellence. The country boasts renowned universities, research institutes, and industrial collaborations, contributing to its high publication output. The United Kingdom holds the fourth position. The UK has a rich scientific heritage and is home to prestigious universities and research institutions. Despite its smaller size compared to some other countries on the list, the UK maintains a strong research presence and fosters a collaborative research environment. India demonstrates its research potential by securing the fifth spot with 979 publications. India has been steadily increasing its investment in research and development, particularly in fields such as technology, medicine, and space science. The country’s large population and diverse academic institutions contribute to its growing scientific output. Italy appears sixth on the list. Italy has a long history of scientific contributions, particularly in fields such as engineering, physics, and art. The country is renowned for its universities and research centers, attracting scientists from around the world. France occupies the seventh position. France has a strong tradition of scientific research, and its universities and research organizations have made significant contributions to various disciplines. The country promotes scientific collaboration and innovation through funding programs and research grants. Spain ranks eighth. Spain has been making notable progress in scientific research, focusing on areas such as renewable energy, biomedicine, and materials science. The country’s universities and research institutions have been actively engaged in research collaborations with international partners. Canada secures the ninth spot. Despite its smaller population, Canada has a strong research ecosystem and invests heavily in scientific research and development. The country’s universities and research institutions excel in fields such as natural sciences, engineering, and health sciences. Brazil rounds up the top 10 list. Brazil has been making significant efforts to boost its scientific research output, particularly in areas such as environmental sciences, agriculture, and bioengineering. The country’s universities and research institutions have been actively collaborating with international partners to advance knowledge and innovation.

Figure  5 depicts the breakdown of publications containing the keywords “new software product” in the Scopus database from 2003 to 2022, categorized by field of study.

Distribution of knowledge domains in Scopus publications with the keywords “new software product” from 2003 to 2022.  Source: formed by the author

The top two fields with the highest number of publications are Computer Science (9339) and Engineering (9222). This indicates the significant interest and research in the development of new software products within these fields. Along with Computer Science and Engineering, several other STEM (Science, Technology, Engineering, and Mathematics) fields also show considerable publication numbers. These include Mathematics, Materials Science, Physics and Astronomy, Chemical Engineering, Energy, and Chemistry. This suggests that the development of new software products is relevant and impactful across various STEM disciplines. Business and Management: The field of Business, Management, and Accounting demonstrates a significant interest in new software products, likely indicating the growing importance of software applications in business operations and management. Some fields, such as Decision Sciences, Social Sciences, Medicine, Environmental Science, Economics, Econometrics, and Finance, and Earth and Planetary Sciences, show notable publication numbers. This suggests the interdisciplinary nature of software product development, with implications in various domains, including healthcare, environmental science, and decision-making processes. The analysis demonstrates that the development of new software products is a widespread research interest across multiple fields of study. The prominence of Computer Science and Engineering indicates their central role in this domain, while other fields also contribute significantly, reflecting the interdisciplinary nature of software product development.

The bibliometric analysis of new software product development highlights a complex network of themes and connections across various keywords, representing the state and evolution of research in this area. The thematic map is organized into clusters, with “quality control” and “computer software” as the dominant clusters (Fig.  6 ).

The “quality control” cluster focuses on aspects such as computer simulation, automation, algorithms, and artificial intelligence, pointing towards a strong emphasis on ensuring the quality and reliability of software through sophisticated methods. Risk assessment, technology, data handling, and simulation are notable for their high betweenness centrality, indicating their role as pivotal topics that bridge various research areas within this cluster. The high pagerank centrality of keywords like “computer simulation” and “algorithms” underscores their influence and the frequent engagement by the academic community.

Thematic map.  Source: formed by the author

The “computer software” cluster dives into the more technical and applied aspects of software development, including software design, engineering, testing, and the lifecycle of software products. It covers a wide array of subjects from “computer aided design” to “software architecture” and “embedded systems”, reflecting the breadth of research and development activities in the software industry. “Product design” and “software engineering” emerge as central themes, highlighted by their pagerank centrality, suggesting a strong focus on these areas within the community. Keywords like “software”, “human”, and “humans” suggest a focus on the human aspect in software development, emphasizing user-centric design and human-computer interaction. The presence of “priority journal”, “computer program”, and “controlled study” indicates a strong emphasis on publishing high-quality research and the use of controlled methodologies to validate findings. Across all clusters, the analysis reveals a landscape marked by a focus on quality control, the application of computer science principles in software development, and the importance of considering the human element. The centrality measures highlight the key topics and terms that serve as connectors or significant points of focus within the network, suggesting areas that are currently of high interest or emerging as important fields of study within new software product development.

Meta-analysis on new Software Product Research

A meta-analysis was performed using the publication count in various scientific research fields from 2003 to 2022, and this allowed for the calculation of the penetration index for a new software product focused on microalgae-based processes and products. Figure  7 illustrates the findings from this analysis.

Penetration index of publications on the new software product across subject areas of scientific research. Source: formed by the author

The penetration index has been generally increasing over the years, indicating a growing presence and influence of the new software product across subject areas of scientific research. From 2003 to 2010, the penetration index shows a relatively stable growth, ranging from 0.710 to 0.714. This suggests a consistent level of adoption and interest in the software product during this period. Starting from 2011, there is a noticeable increase in the penetration index. From 2011 to 2016, the index shows a significant jump from 0.704 to 0.766, indicating a period of accelerated adoption and usage across subject areas. From 2016 to 2021, the penetration index continues to rise steadily, with some minor fluctuations. This period reflects a sustained and increasing presence of the software product in scientific research, with the index reaching 0.831 in 2021. In 2022, there is a slight decrease in the penetration index to 0.818, breaking the trend of continuous growth. This dip may indicate a temporary decrease in publication activity or a shift in research focus, but it’s important to consider other factors that may have influenced this change. The analysis suggests a positive trend in the adoption and penetration of the new software product across subject areas of scientific research. The steady growth followed by a period of accelerated adoption indicates the software’s increasing importance and utilization in the scientific community. The fluctuations in the index should be further examined to understand any underlying factors contributing to the observed changes.

In order to ascertain the connections between the five most densely populated regions, a correlation analysis was performed, considering the quantity of publications. The findings are displayed as a correlation matrix (Fig.  8 ).

Correlation matrix of subject areas in relation to publication volumes within the scope of a newl software product.  Note: CS - “Computer Science”, EN - “Engineering”, MT - “Mathematics”, BMA - “Business, Management and Accounting”, MS - “Materials Science”.  Source: formed by the author

The highest correlation is observed between Computer Science and Mathematics (0.907), indicating a strong relationship between these two subject areas in terms of publication volumes within the scope of the new software product. The strongest correlation within a subject area is found between Engineering and Materials Science (0.869), suggesting a significant relationship between these two disciplines in terms of publication volumes. For most of the studied subject areas, there is a sufficient correlation level above 0.5, indicating a positive relationship between them in terms of publication volumes. These subject areas include Computer Science, Engineering, Mathematics, and Business, Management, and Accounting. The weakest correlation is observed between Computer Science and Engineering (0.035), indicating a relatively weaker relationship in terms of publication volumes within the scope of the new software product. In order to assess the influence of the growing number of publications in specific subject areas on the adoption rate of a novel software product in scientific research, an analysis was performed. The key metrics of this analysis are outlined in Table  2 .

Based on the available data, it is not possible to construct an adequate regression model, as the p-values for the “Engineering” and “Business, Management and Accounting” subject areas exceed the permissible value of 0.05, with values of 0.6102 and 0.4678 respectively. This indicates the inadequacy of the obtained model and suggests the need to exclude the identified subject areas from the model. The results obtained after excluding the data for these subject areas are presented in Table  3 .

The p-values for all the coefficients are below 0.05, indicating that the coefficients are statistically significant. The coefficient of determination (R-squared) is 0.93. Based on the results of the regression analysis, the following model can be formulated to understand the relationship between subject areas and the publication count in the context of a new software product:

The t-tests indicate that the developed model is applicable, as the t-statistics for all coefficients are greater than the critical t-value ( \({ t}_{obs}\) > \({ t}_{crit}\) ). The Fisher criterion confirms the adequacy of the model, as the F-statistic is greater than the critical F-value ( \({ F}_{tabl}\) < \(F\) ). It is worth noting that the coefficient values indicate the strength and direction of the relationship between each subject area and the publication count. The subject areas “Engineering” (EN), “Mathematics” (MT), and “Social Sciences” (SS) all have positive coefficients, suggesting a positive relationship with the publication count. Among them, “Mathematics” has the highest coefficient, indicating the strongest positive influence on the publication count. The intercept term represents the expected publication count when all subject areas have zero influence. The regression analysis suggests that the subject areas of Engineering, Mathematics, and Social Sciences have a significant impact on the publication count in the context of the new software product.

Potential gaps in the Studied Literature

The literature offers a strong foundation. However, it appears there’s a need for more in-depth exploration of the changing user expectations and market demands. These are quickly influenced by technological progress and societal shifts. New technologies are developed and adopted quickly in the software industry, challenging research to stay current. This suggests there might be a gap in studies focused on how these technologies affect product development, team dynamics, and product-market fit (Table  4 ).

The main potential gap identified is the integration of interdisciplinary approaches to software product development. While some publications mention the interdisciplinary nature of the field, incorporating insights from areas such as psychology, sociology, and business could enhance understanding of how software products are developed, adopted, and utilized within various societal and cultural contexts. This could also include exploring the ethical, legal, and social implications of software products, particularly those utilizing artificial intelligence, machine learning, and big data analytics.

The analysis also points to a geographical concentration of research output, with the United States, China, and Germany being the most prolific countries in software product development research. This indicates potential gaps in literature from regions that are underrepresented in research outputs, such as Latin America, Africa, and parts of Asia. Studies from these regions could provide unique perspectives on local market needs, cultural influences on product development, and innovative practices that emerge from different constraints and opportunities. While agile methodologies, user-centered design, and DevOps practices are well-covered in the literature, there is a lack of comprehensive studies that examine the long-term impacts of these practices on product success, team well-being, and organizational culture. Exploring these areas could provide valuable insights into the sustainability of current software development practices and their adaptability to future challenges. While the existing literature on new software product development is extensive and covers a wide range of topics, there are potential gaps that future research could address. These include the integration of emerging technologies, interdisciplinary approaches, geographical diversity in research outputs, long-term impacts of prevalent development practices, and the synthesis of existing research to distill key insights and directions for future studies.

Conclusions

Through bibliometric analysis, this study provides valuable information on the trends, patterns, and impact of research and publications in the field of new software product development. Using the reviewed literature as a basis, it’s possible to identify known research areas, influential authors, and new directions. This provides a comprehensive overview of the current state of knowledge in the field of new software product development. The conducted bibliometric analysis can serve as a basis for further research and help researchers and practitioners to identify knowledge gaps and directions for future research.

The paper emphasises the importance of theoretical foundations in the development of new software products. By building on established theories and models such as user-centred design, agile methodologies and DevOps practices, developers and project teams can benefit from a solid foundation of knowledge and best practices. Theoretical input provides a conceptual framework that guides decision making, problem solving and the realisation of innovative ideas. It enables software teams to make informed choices and develop software products that meet user needs and market requirements.

The study emphasises the importance of hands-on implementation, iterative processes, and collaboration within cross-functional teams. It also highlights the value of agile methodologies, rapid prototyping, and continuous feedback. These approaches ensure the timely delivery of high-quality software products. By incorporating practical input, software teams can translate theoretical knowledge into tangible results, test assumptions, and improve their solutions based on real-world feedback.

The number of publications dedicated to new software products has steadily increased over the past two decades. This indicates a growing interest in this field among researchers and scholars. The growth rate of these publications has varied. Some years have experienced significant increases, while other year have shown smaller or negative growth. The United States, China, and Germany emerged as the leading countries in terms of the number of publications, highlighting their strong presence in software development and technological advancements. Europe, Asia-Pacific, and select countries from other regions also contributed significantly to the research and publication on new software products. However, there is room for increased research and publication from Latin America, Africa, and the Middle East. The rising number of publications reflects the increasing importance of software development and the recognition of software as a significant research topic.

The meta-analysis conducted across various scientific research fields reveals significant interest in developing new software products, being computer Science and Engineering the most prominent fields. However, other STEM disciplines, such as Mathematics, Materials Science, Physics and Astronomy, Chemical Engineering, Energy, and Chemistry, also contribute significantly to software product development. The interdisciplinary nature of software product development is evident, with implications in domains such as business, decision sciences, social sciences, medicine, environmental science, economics, and earth and planetary sciences. The penetration index of the new software product shows a growing presence and influence across subject areas, with a sustained increase in adoption and usage. The correlation analysis reveals strong relationships between subject areas such as Computer Science and Mathematics, and Engineering and Materials Science. Regression analysis demonstrates the significant impact of Engineering, Mathematics, and Social Sciences on the publication count of the new software product. Overall, the findings highlight the widespread interest and interdisciplinary nature of software product development, with particular emphasis on the role of Computer Science and Engineering in driving advancements in this domain.

The research conducted on the development of new software products provides a comprehensive overview of the subject through bibliometric analysis, theoretical contributions and practical ideas. It emphasises the importance of research and analysis in understanding the current situation in software product development. It emphasises the importance of theoretical frameworks in managing the development process and outlines practical strategies and methodologies to stimulate innovation and create successful software products. By combining theoretical and practical aspects, software development teams can optimise their processes, create user-centric solutions and remain at the forefront of the rapidly evolving digital age.

Aldave, A., Vara, J. M., Granada, D., & Marcos, E. (2019). Leveraging creativity in requirements elicitation within agile software development: A systematic literature review. Journal of Systems and Software , 157 , 110396.

Article   Google Scholar  

Almeida, F., Simões, J., & Lopes, S. (2022). Exploring the benefits of combining devops and agile. Future Internet , 14 (2), 63.

Appio, F. P., Frattini, F., Petruzzelli, A. M., & Neirotti, P. (2021). Digital transformation and innovation management: A synthesis of existing research and an agenda for future studies. Journal of Product Innovation Management , 38 (1), 4–20.

Beecham, S., Clear, T., Lal, R., & Noll, J. (2021). Do scaling agile frameworks address global software development risks? An empirical study. Journal of Systems and Software , 171 , 110823.

Berg, V., Birkeland, J., Nguyen-Duc, A., Pappas, I. O., & Jaccheri, L. (2020). Achieving agility and quality in product development-an empirical study of hardware startups. Journal of Systems and Software , 167 , 110599.

Bianchi, M., Marzi, G., & Guerini, M. (2020). Agile, stage-gate and their combination: Exploring how they relate to performance in software development. Journal of Business Research , 110 , 538–553.

Chiang, H. Y., & Lin, B. M. (2020). A decision model for human resource allocation in project management of software development. Ieee Access : Practical Innovations, Open Solutions , 8 , 38073–38081.

Coetzee, S., Ivánová, I., Mitasova, H., & Brovelli, M. A. (2020). Open geospatial software and data: A review of the current state and a perspective into the future. ISPRS International Journal of Geo-Information , 9 (2), 90.

Colón-Ramos, D. A., La Riviere, P., Shroff, H., & Oldenbourg, R. (2019). Transforming the development and dissemination of cutting-edge microscopy and computation. Nature Methods , 16 (8), 667–669.

Cooper, R. G. (2019). The drivers of success in new-product development. Industrial Marketing Management , 76 , 36–47.

Cooper, R. G., & Sommer, A. F. (2018). Agile–stage-gate for manufacturers: Changing the Way New products are developed integrating agile project management methods into a Stage-gate system offers both opportunities and challenges. Research-Technology Management , 61 (2), 17–26.

Dennehy, D., Kasraian, L., O’Raghallaigh, P., Conboy, K., Sammon, D., & Lynch, P. (2019). A lean start-up approach for developing minimum viable products in an established company. Journal of Decision Systems , 28 (3), 224–232.

Dopp, A. R., Parisi, K. E., Munson, S. A., & Lyon, A. R. (2020). Aligning implementation and user-centered design strategies to enhance the impact of health services: Results from a concept mapping study. Implementation Science Communications , 1 (1), 1–13.

Gerke, S., Babic, B., Evgeniou, T., & Cohen, I. G. (2020). The need for a system view to regulate artificial intelligence/machine learning-based software as medical device. NPJ Digital Medicine , 3 (1), 53.

Gill, S. S., Tuli, S., Xu, M., Singh, I., Singh, K. V., Lindsay, D., & Garraghan, P. (2019). Transformative effects of IoT, Blockchain and Artificial Intelligence on cloud computing: Evolution, vision, trends and open challenges. Internet of Things , 8 , 100118.

Herumurti, D., Bimantara, I. M. S., & Supriana, I. W. (2023). User-centered design-based Approach in Scheduling Management Application Design and Development. IPTEK The Journal for Technology and Science , 34 (1), 26–43.

Iqbal, M., & Suzianti, A. (2021). New product development process design for small and medium enterprises: A systematic literature review from the perspective of open innovation. Journal of Open Innovation: Technology Market and Complexity , 7 (2), 153.

Issa Mattos, D., Dakkak, A., Bosch, J., & Olsson, H. H. (2023). The HURRIER process for experimentation in business-to‐business mission‐critical systems. Journal of Software: Evolution and Process , 35(5), e2390.

Jansen, B. J., Salminen, J. O., & Jung, S. G. (2020). Data-driven personas for enhanced user understanding: Combining empathy with rationality for better insights to analytics. Data and Information Management , 4 (1), 1–17.

Lwakatare, L. E., Kilamo, T., Karvonen, T., Sauvola, T., Heikkilä, V., Itkonen, J., & Lassenius, C. (2019). DevOps in practice: A multiple case study of five companies. Information and Software Technology , 114 , 217–230.

Marion, T. J., & Fixson, S. K. (2021). The transformation of the innovation process: How digital tools are changing work, collaboration, and organizations in new product development. Journal of Product Innovation Management , 38 (1), 192–215.

Melegati, J., Goldman, A., Kon, F., & Wang, X. (2019). A model of requirements engineering in software startups. Information and Software Technology , 109 , 92–107.

Merenda, M., Porcaro, C., & Iero, D. (2020). Edge machine learning for Ai-enabled iot devices: A review. Sensors (Basel, Switzerland) , 20 (9), 2533.

Mishra, A., & Otaiwi, Z. (2020). DevOps and software quality: A systematic mapping. Computer Science Review , 38 , 100308.

Mugarza, I., Flores, J. L., & Montero, J. L. (2020). Security issues and software updates management in the industrial internet of things (iiot) era. Sensors (Basel, Switzerland) , 20 (24), 7160.

Najihi, S., Elhadi, S., Abdelouahid, R. A., & Marzak, A. (2022). Software Testing from an agile and traditional view. Procedia Computer Science , 203 , 775–782.

Nica, E., & Stehel, V. (2021). Internet of things sensing networks, artificial intelligence-based decision-making algorithms, and real-time process monitoring in sustainable industry 4.0. Journal of Self-Governance and Management Economics , 9 (3), 35–47.

Google Scholar  

Park, D. Y., Choi, J., Ryu, S., & Kim, M. J. (2022). A user-centered Approach to the application of BIM in Smart Working environments. Sensors (Basel, Switzerland) , 22 (8), 2871.

Parthasarathy, S. (2022). A decision framework for software startups to succeed in COVID-19 environment. Decision Analytics Journal , 3 , 100037.

Salvato, J. J., & Laplume, A. O. (2020). Agile stage-gate management (ASGM) for physical products. R&d Management , 50 (5), 631–647.

Saputri, T. R. D., & Lee, S. W. (2021). Integrated framework for incorporating sustainability design in software engineering life-cycle: An empirical study. Information and Software Technology , 129 , 106407.

Shameem, M. (2022). A systematic literature review of challenges factors for implementing DevOps practices in software development organizations: A development and operation teams perspective. Evolving Software Processes: Trends and Future Directions , 187–199.

Thota, M. K., Shajin, F. H., & Rajesh, P. (2020). Survey on software defect prediction techniques. International Journal of Applied Science and Engineering , 17 (4), 331–344.

Tripathi, N., Oivo, M., Liukkunen, K., & Markkula, J. (2019). Startup ecosystem effect on minimum viable product development in software startups. Information and Software Technology , 114 , 77–91.

Wiedemann, A., Forsgren, N., Wiesche, M., Gewald, H., & Krcmar, H. (2019). The DevOps Phenomenon. ACM Queue , 17 (2), 40.

Wiedemann, A., Forsgren, N., Wiesche, M., Gewald, H., & Krcmar, H. (2019a). Research for practice: The DevOps phenomenon. Communications of the ACM , 62 (8), 44–49.

Zayat, W., & Senvar, O. (2020). Framework study for agile software development via scrum and Kanban. International Journal of Innovation and Technology Management , 17 (04), 2030002.

Zonnenshain, A., & Kenett, R. S. (2020). Quality 4.0—the challenging future of quality engineering. Quality Engineering , 32 (4), 614–626.

Zorzetti, M., Signoretti, I., Salerno, L., Marczak, S., & Bastos, R. (2022). Improving agile software development using user-centered design and lean startup. Information and Software Technology , 141 , 106718.

Download references

Open access funding provided by FCT|FCCN (b-on).

Author information

Authors and affiliations.

Instituto Universitário de Lisboa (ISCTE-IUL), ISTAR, 1600-189, Lisbon, Portugal

Pedro Neves Mata

Instituto Superior de Contabilidade E Administração de Lisboa, Avenida Miguel Bombarda, 20, 1069-035, Lisbon, Portugal

José Moleiro Martins

Instituto Universitário de Lisboa (ISCTE-IUL), Business Research Unit (BRU-IUL), Lisbon, Portugal

Instituto Universitário de Lisboa (ISCTE-IUL), 1649-026, Lisbon, Portugal

João Carlos Ferreira

Information Sciences and Technologies and Architecture Research Centre (ISTAR), Lisbon, Portugal

Molde University College - Specialized University in Logistics, Molde, Norway

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Pedro Neves Mata .

Ethics declarations

Conflict of interest.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Mata, P.N., Martins, J.M. & Ferreira, J.C. New Software Product Development: Bibliometric Analysis. J Knowl Econ (2024). https://doi.org/10.1007/s13132-024-02095-5

Download citation

Received : 28 November 2023

Accepted : 14 May 2024

Published : 15 June 2024

DOI : https://doi.org/10.1007/s13132-024-02095-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Software Product
• Innovation Management
• Development
• Engineering
• Effectiveness
• Find a journal
• Publish with us
• Track your research

IEEE Account

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.