literature review research quiz

• Request new password
• Create a new account

Research Methods in Early Childhood: An Introductory Guide

Student resources, multiple choice quiz.

Test your understanding of each chapter by taking the quiz below. Click anywhere on the question to reveal the answer. Good luck!

1. What is an abstract?

• A short footnote in the text of your work
• A chance to acknowledge those who have helped you in your research project, placed at the beginning of the written-up project
• A brief resume of the study to entice the reader to read more
• An introduction to the research

2. What should always be included in an introduction?

• Detail of your research findings
• Your interview schedule (or detail of any other research instruments)
• A rationale for conducting the study and key research questions
• Recommendations for early childhood practice in detail

3. What is a literature review?

• A place to provide autobiographic detail
• A place to outline all the reading you have done in your studies
• A place to link your reading to your research findings
• A place to discuss literature which helps to frame your study

4. Where is a literature review typically found in written-up research?

• Before the methodology chapter
• After the findings chapter
• In the appendices
• Integrated into the data analysis

5. What would you expect to find in a methodology chapter in a piece of empirical research?

• Detail of the methodological approach taken
• Detail of the methods employed
• Detail of ethical considerations linked to the research project
• All of the above

6. Should data and discussion of data be presented as two separate chapters?

• Never. They are inappropriate for early childhood research
• Always. Students tend to undertake qualitative research projects
• Possibly, depending on the kind of research undertaken and tutor advice
• Possibly, depending if there is time to complete two chapters as opposed to one larger chapter

7. What should a conclusion chapter contain?

• A sense of the research story
• A summary of the key findings
• Reflection on what these findings mean
• Discussion of possible implications for practice or future research

8. Having read the chapter, what do Mukherji and Albon advise in relation to using ‘I’ when writing up research?

• You should always use the personal pronoun when writing up research as it reflects that it is a personal piece of work
• Writing seems far too ‘chatty’ if you use ‘I’ when writing. In order to write academically use of ‘I’ should always be avoided
• If talking about the decisions you made in research and your own viewpoint, it is hard to avoid use of ‘I’ and it does not necessarily mean lack of academic tone if used judiciously (but do ask your tutor!)
• You could be in danger of failing a module if you use the personal pronoun in your work. You will never see academic journal articles using ‘I’

Indiana University Indianapolis Indiana University Indianapolis IU Indianapolis

• Herron School of Art
• Ruth Lilly Law
• Ruth Lilly Medical
• School of Dentistry

Literature Review - A Self-Guided Tutorial

• Literature Reviews: A Recap
• Reading Journal Articles
• Does it describe a Literature Review?
• 1. Identify the question
• 2. Review discipline styles
• Searching article databases - video
• Finding the article full-text
• Citation chaining
• When to stop searching
• 4. Manage your references
• 5. Critically analyze and evaluate
• 6. Synthesize
• 7. Write literature review

Quiz: What Have You Learned?

If quiz does not load, click here http://library.indianapolis.iu.edu/files/html5/LitReviewFinalQuizSWF.html .

• << Previous: 7. Write literature review
• Last Updated: Jun 20, 2024 1:47 PM
• URL: https://iu.libguides.com/literaturereview

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• How to Write a Literature Review | Guide, Examples, & Templates

How to Write a Literature Review | Guide, Examples, & Templates

Published on January 2, 2023 by Shona McCombes . Revised on September 11, 2023.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research that you can later apply to your paper, thesis, or dissertation topic .

There are five key steps to writing a literature review:

• Search for relevant literature
• Evaluate sources
• Identify themes, debates, and gaps
• Outline the structure
• Write your literature review

A good literature review doesn’t just summarize sources—it analyzes, synthesizes , and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

Table of contents

What is the purpose of a literature review, examples of literature reviews, step 1 – search for relevant literature, step 2 – evaluate and select sources, step 3 – identify themes, debates, and gaps, step 4 – outline your literature review’s structure, step 5 – write your literature review, free lecture slides, other interesting articles, frequently asked questions, introduction.

• Quick Run-through
• Step 1 & 2

When you write a thesis , dissertation , or research paper , you will likely have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

• Demonstrate your familiarity with the topic and its scholarly context
• Develop a theoretical framework and methodology for your research
• Position your work in relation to other researchers and theorists
• Show how your research addresses a gap or contributes to a debate
• Evaluate the current state of research and demonstrate your knowledge of the scholarly debates around your topic.

Writing literature reviews is a particularly important skill if you want to apply for graduate school or pursue a career in research. We’ve written a step-by-step guide that you can follow below.

Don't submit your assignments before you do this

The academic proofreading tool has been trained on 1000s of academic texts. Making it the most accurate and reliable proofreading tool for students. Free citation check included.

Try for free

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

• Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
• Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
• Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
• Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research problem and questions .

Make a list of keywords

Start by creating a list of keywords related to your research question. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list as you discover new keywords in the process of your literature search.

• Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
• Body image, self-perception, self-esteem, mental health
• Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some useful databases to search for journals and articles include:

• Your university’s library catalogue
• Google Scholar
• Project Muse (humanities and social sciences)
• Medline (life sciences and biomedicine)
• EconLit (economics)
• Inspec (physics, engineering and computer science)

You can also use boolean operators to help narrow down your search.

Make sure to read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

You likely won’t be able to read absolutely everything that has been written on your topic, so it will be necessary to evaluate which sources are most relevant to your research question.

For each publication, ask yourself:

• What question or problem is the author addressing?
• What are the key concepts and how are they defined?
• What are the key theories, models, and methods?
• Does the research use established frameworks or take an innovative approach?
• What are the results and conclusions of the study?
• How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
• What are the strengths and weaknesses of the research?

Make sure the sources you use are credible , and make sure you read any landmark studies and major theories in your field of research.

You can use our template to summarize and evaluate sources you’re thinking about using. Click on either button below to download.

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It is important to keep track of your sources with citations to avoid plagiarism . It can be helpful to make an annotated bibliography , where you compile full citation information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

Prevent plagiarism. Run a free check.

To begin organizing your literature review’s argument and structure, be sure you understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

• Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
• Themes: what questions or concepts recur across the literature?
• Debates, conflicts and contradictions: where do sources disagree?
• Pivotal publications: are there any influential theories or studies that changed the direction of the field?
• Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

• Most research has focused on young women.
• There is an increasing interest in the visual aspects of social media.
• But there is still a lack of robust research on highly visual platforms like Instagram and Snapchat—this is a gap that you could address in your own research.

There are various approaches to organizing the body of a literature review. Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarizing sources in order.

Try to analyze patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organize your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

• Look at what results have emerged in qualitative versus quantitative research
• Discuss how the topic has been approached by empirical versus theoretical scholarship
• Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text , your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, you can follow these tips:

• Summarize and synthesize: give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: don’t just paraphrase other researchers — add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically evaluate: mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: use transition words and topic sentences to draw connections, comparisons and contrasts

In the conclusion, you should summarize the key findings you have taken from the literature and emphasize their significance.

When you’ve finished writing and revising your literature review, don’t forget to proofread thoroughly before submitting. Not a language expert? Check out Scribbr’s professional proofreading services !

This article has been adapted into lecture slides that you can use to teach your students about writing a literature review.

Scribbr slides are free to use, customize, and distribute for educational purposes.

Open Google Slides Download PowerPoint

If you want to know more about the research process , methodology , research bias , or statistics , make sure to check out some of our other articles with explanations and examples.

• Sampling methods
• Simple random sampling
• Stratified sampling
• Cluster sampling
• Likert scales
• Reproducibility

 Statistics

• Null hypothesis
• Statistical power
• Probability distribution
• Effect size
• Poisson distribution

Research bias

• Optimism bias
• Cognitive bias
• Implicit bias
• Hawthorne effect
• Anchoring bias
• Explicit bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarize yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your thesis or dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. (2023, September 11). How to Write a Literature Review | Guide, Examples, & Templates. Scribbr. Retrieved June 24, 2024, from https://www.scribbr.com/dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, what is a theoretical framework | guide to organizing, what is a research methodology | steps & tips, how to write a research proposal | examples & templates, get unlimited documents corrected.

✔ Free APA citation check included ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

• University of Oregon Libraries
• Research Guides

How to Write a Literature Review

• 1. Identify the Question
• Literature Reviews: A Recap
• Reading Journal Articles
• Does it Describe a Literature Review?

Identify the question

Developing a research question.

• 2. Review Discipline Styles
• Searching Article Databases
• Finding Full-Text of an Article
• Citation Chaining
• When to Stop Searching
• 4. Manage Your References
• 5. Critically Analyze and Evaluate
• 6. Synthesize
• 7. Write a Literature Review

From Topic to Question (Infographic)

This graphic emphasizes how reading various sources can play a role in defining your research topic.

( Click to Enlarge Image )

Text description of "From Topic to Question" for web accessibility

In some cases, such as for a course assignment or a research project you're working on with a faculty mentor, your research question will be determined by your professor. If that's the case, you can move on to the next step .  Otherwise, you may need to explore questions on your own. 

A few suggestions

Photo Credit: UO Libraries

According to The Craft of Research (2003) , a research question is more than a practical problem or something with a yes/no answer. A research question helps you learn more about something you don't already know and it needs to be significant enough to interest your readers.

Your Curiosity + Significance to Others = Research Question

How to get started.

In a research paper, you develop a unique question and then synthesize scholarly and primary sources into a paper that supports your argument about the topic.

• Identify your Topic (This is the starting place from where you develop a research question.)
• Refine by Searching (find background information) (Before you can start to develop a research question, you may need to do some preliminary background research to see (1) what has already been done on the topic and (2) what are the issues surrounding the topic.) HINT: Find background information in Google and Books.
• Refine by Narrowing (Once you begin to understand the topic and the issues surrounding it, you can start to narrow your topic and develop a research question. Do this by asking the 6 journalistic question words.

Ask yourself these 6 questions 

These 6 journalistic question words can help you narrow your focus from a broad topic to a specific question.

Who : Are you interested in a specific group of people? Can your topic be narrowed by gender, sex, age, ethnicity, socio-economic status or something else? Are there any key figures related to your topic?

What : What are the issues surrounding your topic? Are there subtopics? In looking at background information, did you notice any gaps or questions that seemed unanswered?

Where : Can your topic be narrowed down to a geographic location? Warning: Don't get too narrow here. You might not be able to find enough information on a town or state.

When : Is your topic current or historical? Is it confined to a specific time period? Was there a causative event that led your topic to become an area of study?

Why : Why are you interested in this topic? Why should others be interested?

How : What kinds of information do you need? Primary sources, statistics? What is your methodology?

Detailed description of, "Developing a Research Question" for web accessibility

• << Previous: Does it Describe a Literature Review?
• Next: 2. Review Discipline Styles >>
• Last Updated: May 3, 2024 5:17 PM
• URL: https://researchguides.uoregon.edu/litreview

Contact Us Library Accessibility UO Libraries Privacy Notices and Procedures

1501 Kincaid Street Eugene, OR 97403 P: 541-346-3053 F: 541-346-3485

• Visit us on Facebook
• Visit us on Twitter
• Visit us on Youtube
• Visit us on Instagram
• Report a Concern
• Nondiscrimination and Title IX
• Accessibility
• Privacy Policy
• Find People

Harvey Cushing/John Hay Whitney Medical Library

• Collections
• Research Help

YSN Doctoral Programs: Steps in Conducting a Literature Review

• Biomedical Databases
• Global (Public Health) Databases
• Soc. Sci., History, and Law Databases
• Grey Literature
• Trials Registers
• Data and Statistics
• Public Policy
• Google Tips
• Recommended Books
• Steps in Conducting a Literature Review

What is a literature review?

A literature review is an integrated analysis -- not just a summary-- of scholarly writings and other relevant evidence related directly to your research question.  That is, it represents a synthesis of the evidence that provides background information on your topic and shows a association between the evidence and your research question.

A literature review may be a stand alone work or the introduction to a larger research paper, depending on the assignment.  Rely heavily on the guidelines your instructor has given you.

Why is it important?

A literature review is important because it:

• Explains the background of research on a topic.
• Demonstrates why a topic is significant to a subject area.
• Discovers relationships between research studies/ideas.
• Identifies major themes, concepts, and researchers on a topic.
• Identifies critical gaps and points of disagreement.
• Discusses further research questions that logically come out of the previous studies.

APA7 Style resources

APA Style Blog - for those harder to find answers

1. Choose a topic. Define your research question.

Your literature review should be guided by your central research question.  The literature represents background and research developments related to a specific research question, interpreted and analyzed by you in a synthesized way.

• Make sure your research question is not too broad or too narrow.  Is it manageable?
• Begin writing down terms that are related to your question. These will be useful for searches later.
• If you have the opportunity, discuss your topic with your professor and your class mates.

2. Decide on the scope of your review

How many studies do you need to look at? How comprehensive should it be? How many years should it cover? 

• This may depend on your assignment.  How many sources does the assignment require?

3. Select the databases you will use to conduct your searches.

Make a list of the databases you will search. 

Where to find databases:

• use the tabs on this guide
• Find other databases in the Nursing Information Resources web page
• More on the Medical Library web page
• ... and more on the Yale University Library web page

4. Conduct your searches to find the evidence. Keep track of your searches.

• Use the key words in your question, as well as synonyms for those words, as terms in your search. Use the database tutorials for help.
• Save the searches in the databases. This saves time when you want to redo, or modify, the searches. It is also helpful to use as a guide is the searches are not finding any useful results.
• Review the abstracts of research studies carefully. This will save you time.
• Use the bibliographies and references of research studies you find to locate others.
• Check with your professor, or a subject expert in the field, if you are missing any key works in the field.
• Ask your librarian for help at any time.
• Use a citation manager, such as EndNote as the repository for your citations. See the EndNote tutorials for help.

Review the literature

Some questions to help you analyze the research:

• What was the research question of the study you are reviewing? What were the authors trying to discover?
• Was the research funded by a source that could influence the findings?
• What were the research methodologies? Analyze its literature review, the samples and variables used, the results, and the conclusions.
• Does the research seem to be complete? Could it have been conducted more soundly? What further questions does it raise?
• If there are conflicting studies, why do you think that is?
• How are the authors viewed in the field? Has this study been cited? If so, how has it been analyzed?

Tips: 

• Review the abstracts carefully.  
• Keep careful notes so that you may track your thought processes during the research process.
• Create a matrix of the studies for easy analysis, and synthesis, across all of the studies.
• << Previous: Recommended Books
• Last Updated: Jun 20, 2024 9:08 AM
• URL: https://guides.library.yale.edu/YSNDoctoral

Literature Review & the research process

Quiz   by sarinder.

Feel free to use or edit a copy

includes Teacher and Student dashboards

Measure skills from any curriculum

Tag the questions with any skills you have. Your dashboard will track each student's mastery of each skill.

• edit the questions
• save a copy for later
• start a class game
• automatically assign follow-up activities based on students’ scores
• assign as homework
• share a link with colleagues
• print as a bubble sheet
• Q 1 / 8 Score 0 What is a Literature Review? 29 A review of information sources on English Literature A critical review of published sources of information on your research topic Just a list of the published sources of information Just references such as Johnson (2016) on your research topic

Our brand new solo games combine with your quiz, on the same screen

Correct quiz answers unlock more play!

• Q 1 What is a Literature Review? A review of information sources on English Literature A critical review of published sources of information on your research topic Just a list of the published sources of information Just references such as Johnson (2016) on your research topic 30 s
• Q 2 Why conduct a Literature Review? To show how widely you have read Identify themes in the Literature and gaps in Knowledge To acknowledge your references 30 s
• Q 3 How would you cite sources in a Literature Review? Johnson (2016,p.7) F.H. Johnson( 2016) Fred Johnson (2016) 30 s
• Q 4 How would you assess claims made by companies that they are market leaders in the field of organising Group Educational Visits? Check existing Market Research on Group Educational Visits Check their websites Check Customer Reviews Check financial status at Company House 30 s
• Q 5 How would you assess whether information sources provide you with accurate information? Check Factual claims in Websites Check factual claims in books Check several sources to see whether they consistently report the same factual claims 30 s
• Q 6 What do we mean by reliability in Market Research? Different researchers get different results It measures whether the feedback given by customers represents the population as a whole If different Market researchers were to devise Questionnaires they would get the same results 30 s
• Q 7 What do we mean by Validity in Market Research? Different researchers get different results Do the results arrived at in consumer research reflect the views of the population as whole All researchers get the same results 30 s
• Q 8 How do you assess the credibility of information provided by websites? The ease of navigating the website Look at how well the website is designed Check the Web address and the type of organisation it is 30 s

Teachers give this quiz to your class

Literature Reviews

• Getting Started
• Choosing a Type of Review

Developing a Research Question

Finding example literature reviews.

• Searching the Literature
• Searching Tips
• ChatGPT [beta]
• Documenting your Search
• Using Citation Managers
• Concept Mapping
• Writing the Review
• Further Resources

Goldilocker Tool

UM Librarians have developed a quick tool called Goldilocker  to help beginners who are struggling to refine their Research Question. 

DEVELOPING A RESEARCH QUESTION

Before searching for sources, you need to formulate a Research Question — this is what you are trying to answer using the existing academic literature. The Research Question pinpoints the focus of the review .

Your first step involves choosing, exploring, and focusing a topic. At this stage you might discover that you need to tweak your topic or the scope of your research as you learn more about the topic in the literature.

THINGS TO KEEP IN MIND: 

• The question must be "researchable" — it can be answered with accessible facts and data
• Questions often start with How, Why, What, Which
• The question opens the door for other areas of inquiry — it identifies a gap in existing research
• Questions should be open-ended and focus on cause and effect

TRY TO AVOID: 

• Simple yes/no questions, or questions with an easy answer (what is the radius of the moon?)
• Questions that can only be answered by an opinion (does it smell nice when it rains?)
• Questions that involve secret information (what is the recipe for Coca-Cola?)
• Questions that are too broad or too narrow

REFINING YOUR RESEARCH QUESTION

Two examples of refining research questions that could be considered either too broad or too narrow.

USING DATABASE FILTER TOOLS

It can be helpful to read existing literature reviews on your topic to get an idea of major themes, how authors structure their arguments, or what reviews look like in your discipline.

DOCUMENT TYPE FILTERS

Many library databases have the option to highlight just Review Articles after you perform a search. Filters above show what the Document Type filter looks like, with a "Review" option. These examples are from Scopus and ProQuest. The "Review" filter here refers to free-standing, comprehensive Review Articles on a topic, as opposed to a shorter literature review inside a scholarly article.

LIT REVIEWS INSIDE ARTICLES

It is also worth taking a look at the shorter literature reviews inside scholarly articles. These can sometimes be called "Background" or "Background Literature." Look for a section typically following the Introduction that covers the history or gives context on the paper's topic.

EXAMPLE REVIEW ARTICLES

Here are a few examples of Review Articles in different disciplines. Note sometimes an article can be a Review Article without the word "review" in the title.

HUMANITIES — Art — " Art and Crime: Conceptualising Graffiti in the City " from the journal Geography Compass

SCIENCES — Climate Change — " Mercury Isotopes in Earth and Environmental Sciences " from the journal  Annual Review of Earth and Planetary Sciences

SOCIAL SCIENCES — Psychology — " Structural Competency and the Future of Firearm Research " from the journal Social Science & Medicine

• << Previous: Choosing a Type of Review
• Next: Searching the Literature >>
• Last Updated: May 9, 2024 11:44 AM
• URL: https://guides.lib.umich.edu/litreview

Literature Review

A literature review is a written summary of the findings of a literature ________ search review proposal

Rate this question:

In the powerpoint we indicate there are three key aspects to a literature review : summarising, synthesising and ______________ referencing citing paraphrasing evaluating

Your literature review is a list of all the work published on your topic and who argues what. true or false , there are different ways to organise your literature review. tick all which apply.

Chronologically

Thematically

Methodologically

Anyway that makes sense to me; it's my work!

A visual ______ map might help you plan the structure of the literature review

When you are referencing correctly you will probably have an in-_____ citation and a bibliography/reference list. (although this might depend on which referencing style you are using).

Quiz Review Timeline +

Our quizzes are rigorously reviewed, monitored and continuously updated by our expert board to maintain accuracy, relevance, and timeliness.

• Current Version
• Mar 22, 2023 Quiz Edited by ProProfs Editorial Team
• Feb 24, 2015 Quiz Created by Ncllibsoc

Related Topics

• Short Story

Recent Quizzes

Featured Quizzes

Popular Topics

• Animation Quizzes
• Art History Quizzes
• Art Terms Quizzes
• Artist Quizzes
• Craft Quizzes
• Culinary Art Quizzes
• Dance Quizzes
• Drawing Quizzes
• Fine Art Quizzes
• Magic Quizzes
• Painting Quizzes
• Photography Quizzes

Related Quizzes

Wait! Here's an interesting quiz for you.

• Subject guides
• Researching for your literature review
• Before you start

Researching for your literature review: Before you start

• Literature reviews
• Literature sources
• Develop a search strategy
• Keyword search activity
• Subject search activity
• Combined keyword and subject searching
• Online tutorials
• Apply search limits
• Run a search in different databases
• Supplementary searching
• Save your searches
• Manage results

The research question

If you are having difficulty finding a suitable topic for your review, try thinking about the following:

• Areas of uncertainty
• Variations of practice 
• Assumptions in practice
• Existing review topics that may be outdated and could be revised to address a new element

Once you have your topic, put it into the format of a question or questions to be answered by the literature . 

Essentially a research question puts forward an hypothesis about a relationship, such as the relationship between an intervention and an outcome. For example: In P (population group) does I (intervention) result in O (outcome)? or Will I perform better than C at achieving O in population P?

For an overview of five main steps to creating a good research question see the online library resources . See also this online video on YouTube.

If you would like to try generative AI (ChatGPT) to assist with developing your research question, see this video by an academic language advisor in the faculty of education for some advice.

T he research question will guide the development of your search strategy so it's important that you take time to do some testing of your proposed question. Having done the preliminary scoping searches as noted above will be helpful in understanding the volume of the literature. This guide will provide a sample search for a health/medical topic, as well as for an education/social science topic. Choose which section you would like to work through.

Start with scoping searches

Doing some scoping searches are an essential step to help you understand the quantity of existing literature in your area of interest, and the terminology used in its discussion. This preliminary searching is non-systematic in its nature and is not documented, but helps establish a basis for the subsequent development of a comprehensive search strategy.

Scoping searches usually involve a series of very targeted searches , perhaps looking for your main keywords in the article title. You might also look for examples of review articles on similar topics, as these might be useful for gaining an overview of a facet of your own topic. We can use a search engine such as Google Scholar , large inter-disciplinary databases such as Scopus or Proquest central , or a discipline-specific database that you are familiar with.

Our scoping searches will help us to understand whether our research question has the following elements:

• the question addresses a gap in the literature ( has not recently been answered in another review)
• the question is specific and focused (and therefore feasible)
• the question is answerable in the literature.

Develop a 'gold set' of target papers

Your scoping searches should enable you to locate a 'gold set' of relevant articles that you would expect to use in your review.

This gold set is a curated collection of highly relevant papers for your research question. These articles are important as a reliable foundation for compiling your search terms.

The gold set is also needed to test the strength of the search strategy that you will later develop. This testing is a hallmark of a rigorous and comprehensive search. Unless you test your search on a target set of papers that are definitely relevant, you have no tangible means of assessing whether you have any errors or omissions in your search strategy!

• << Previous: Literature sources
• Next: Search strategies - Health/Medical topic example >>

Research guidance, Research Journals, Top Universities

Top 20 MCQs on literature review with answers

MCQs on literature review : The primary purpose of literature review is to facilitate detailed background of the previous studies to the readers on the topic of research.

In this blog post, we have published 20 MCQs on Literature Review (Literature Review in Research) with answers.

20 Multiple Choice Questions on Literature Review

1. Literature is a 

Written Record

Published Record

Unpublished Record

All of these

2. Which method of literature review involves a non-statistical method to present data having the feature of systematic Method too?

Narrative Method

Systematic Method

Meta-Analysis Method of Literature Review

Meta-Synthesis Method of Literature Review

3. Comparisons of non-statistical variables are performed under which method of literature review?

4. Literature review is not similar to

Annotated Bibliography 

5. APA Style, MLA Style, Chicago Manual, Blue Book, OSCOLA are famously known as

Citation Manuals

Directories

Abbreviation Manuals

6. Literature collected is reviewed and preferably arranged 

Alphabetically

Chronologically

None of these

7. Literature collected for review includes

Primary and Secondary Sources

Secondary and Tertiary Sources

Primary and Tertiary Sources

8. Literature includes

Previous Studies

Scholarly publications

Research Findings

9. No time frame is set to collect literature in which of the following method of compiling reviews?

Traditional Method

10. Which method of the literature review is more reliable for drawing conclusions of each individual researcher for new conceptualizations and interpretations?

11. The main purpose of finalization of research topics and sub-topics is

Collection of Literature

Collection of Questions

Collection of Statistics

Collection of Responses

12. Literature review is basically to bridge the gap between

Newly established facts

Previously established facts

Facts established time to time

Previous to current established facts

13. The last step in writing the literature review is 

Developing a Final Essay

Developing a Coherent Essay

Developing a Collaborated Essay

Developing a Coordinated Essay

14. The primary purpose of literature review is to facilitate detailed background of 

Present Studies

Previous studies

Future Studies

15. Narrative Literature Review method is also known as 

Advanced Method

Scientific Method

16. Which method of literature review starts with formulating research questions?

17. Which method of literature review involves application of clinical approach based on a specific subject.

18. Which literature review involves timeline based collection of literature for review

19. Which method of literature review involves application of statistical approach?

20. Which literature review method involves conclusions in numeric/statistical form?

More MCQs Related to MCQs on Literature Review

• MCQs on Qualitative Research with answers
• Research Proposal MCQs with answers PDF
• Solved MCQ on legal Reasoning in Research
• MCQ on data analysis in research methodology
• Research Report writing MCQs with answers
• All Solved MCQs on Research Methodology
• MCQs on Legal Research with answers
• MCQs on sampling in research methodology with answers
• MCQs with answers on plagiarism
• MCQ on Citation and Referencing in Research
• Research Ethics MCQs with answers
• Solved MCQs on Sampling in research methodology
• Solved MCQs on Basic Research

MCQs  on literature review  with answers PDF | Research methods multiple choice questions | Literature review  questions and answers

Share this:

1 thought on “top 20 mcqs on literature review with answers”.

Very nice questions for revision

Leave a Comment Cancel reply

Save my name, email, and website in this browser for the next time I comment.

Notify me of follow-up comments by email.

Notify me of new posts by email.

How To Structure Your Literature Review

3 options to help structure your chapter.

By: Amy Rommelspacher (PhD) | Reviewer: Dr Eunice Rautenbach | November 2020 (Updated May 2023)

Writing the literature review chapter can seem pretty daunting when you’re piecing together your dissertation or thesis. As  we’ve discussed before , a good literature review needs to achieve a few very important objectives – it should:

• Demonstrate your knowledge of the research topic
• Identify the gaps in the literature and show how your research links to these
• Provide the foundation for your conceptual framework (if you have one)
• Inform your own  methodology and research design

To achieve this, your literature review needs a well-thought-out structure . Get the structure of your literature review chapter wrong and you’ll struggle to achieve these objectives. Don’t worry though – in this post, we’ll look at how to structure your literature review for maximum impact (and marks!).

But wait – is this the right time?

Deciding on the structure of your literature review should come towards the end of the literature review process – after you have collected and digested the literature, but before you start writing the chapter. 

In other words, you need to first develop a rich understanding of the literature before you even attempt to map out a structure. There’s no use trying to develop a structure before you’ve fully wrapped your head around the existing research.

Equally importantly, you need to have a structure in place before you start writing , or your literature review will most likely end up a rambling, disjointed mess. 

Importantly, don’t feel that once you’ve defined a structure you can’t iterate on it. It’s perfectly natural to adjust as you engage in the writing process. As we’ve discussed before , writing is a way of developing your thinking, so it’s quite common for your thinking to change – and therefore, for your chapter structure to change – as you write. 

Need a helping hand?

Like any other chapter in your thesis or dissertation, your literature review needs to have a clear, logical structure. At a minimum, it should have three essential components – an  introduction , a  body   and a  conclusion . 

Let’s take a closer look at each of these.

1: The Introduction Section

Just like any good introduction, the introduction section of your literature review should introduce the purpose and layout (organisation) of the chapter. In other words, your introduction needs to give the reader a taste of what’s to come, and how you’re going to lay that out. Essentially, you should provide the reader with a high-level roadmap of your chapter to give them a taste of the journey that lies ahead.

Here’s an example of the layout visualised in a literature review introduction:

Your introduction should also outline your topic (including any tricky terminology or jargon) and provide an explanation of the scope of your literature review – in other words, what you  will   and  won’t   be covering (the delimitations ). This helps ringfence your review and achieve a clear focus . The clearer and narrower your focus, the deeper you can dive into the topic (which is typically where the magic lies). 

Depending on the nature of your project, you could also present your stance or point of view at this stage. In other words, after grappling with the literature you’ll have an opinion about what the trends and concerns are in the field as well as what’s lacking. The introduction section can then present these ideas so that it is clear to examiners that you’re aware of how your research connects with existing knowledge .

2: The Body Section

The body of your literature review is the centre of your work. This is where you’ll present, analyse, evaluate and synthesise the existing research. In other words, this is where you’re going to earn (or lose) the most marks. Therefore, it’s important to carefully think about how you will organise your discussion to present it in a clear way. 

The body of your literature review should do just as the description of this chapter suggests. It should “review” the literature – in other words, identify, analyse, and synthesise it. So, when thinking about structuring your literature review, you need to think about which structural approach will provide the best “review” for your specific type of research and objectives (we’ll get to this shortly).

There are (broadly speaking)  three options  for organising your literature review.

Option 1: Chronological (according to date)

Organising the literature chronologically is one of the simplest ways to structure your literature review. You start with what was published first and work your way through the literature until you reach the work published most recently. Pretty straightforward.

The benefit of this option is that it makes it easy to discuss the developments and debates in the field as they emerged over time. Organising your literature chronologically also allows you to highlight how specific articles or pieces of work might have changed the course of the field – in other words, which research has had the most impact . Therefore, this approach is very useful when your research is aimed at understanding how the topic has unfolded over time and is often used by scholars in the field of history. That said, this approach can be utilised by anyone that wants to explore change over time .

For example , if a student of politics is investigating how the understanding of democracy has evolved over time, they could use the chronological approach to provide a narrative that demonstrates how this understanding has changed through the ages.

Here are some questions you can ask yourself to help you structure your literature review chronologically.

• What is the earliest literature published relating to this topic?
• How has the field changed over time? Why?
• What are the most recent discoveries/theories?

In some ways, chronology plays a part whichever way you decide to structure your literature review, because you will always, to a certain extent, be analysing how the literature has developed. However, with the chronological approach, the emphasis is very firmly on how the discussion has evolved over time , as opposed to how all the literature links together (which we’ll discuss next ).

Option 2: Thematic (grouped by theme)

The thematic approach to structuring a literature review means organising your literature by theme or category – for example, by independent variables (i.e. factors that have an impact on a specific outcome).

As you’ve been collecting and synthesising literature , you’ll likely have started seeing some themes or patterns emerging. You can then use these themes or patterns as a structure for your body discussion. The thematic approach is the most common approach and is useful for structuring literature reviews in most fields.

For example, if you were researching which factors contributed towards people trusting an organisation, you might find themes such as consumers’ perceptions of an organisation’s competence, benevolence and integrity. Structuring your literature review thematically would mean structuring your literature review’s body section to discuss each of these themes, one section at a time.

Here are some questions to ask yourself when structuring your literature review by themes:

• Are there any patterns that have come to light in the literature?
• What are the central themes and categories used by the researchers?
• Do I have enough evidence of these themes?

PS – you can see an example of a thematically structured literature review in our literature review sample walkthrough video here.

Option 3: Methodological

The methodological option is a way of structuring your literature review by the research methodologies used . In other words, organising your discussion based on the angle from which each piece of research was approached – for example, qualitative , quantitative or mixed  methodologies.

Structuring your literature review by methodology can be useful if you are drawing research from a variety of disciplines and are critiquing different methodologies. The point of this approach is to question  how  existing research has been conducted, as opposed to  what  the conclusions and/or findings the research were.

For example, a sociologist might centre their research around critiquing specific fieldwork practices. Their literature review will then be a summary of the fieldwork methodologies used by different studies.

Here are some questions you can ask yourself when structuring your literature review according to methodology:

• Which methodologies have been utilised in this field?
• Which methodology is the most popular (and why)?
• What are the strengths and weaknesses of the various methodologies?
• How can the existing methodologies inform my own methodology?

3: The Conclusion Section

Once you’ve completed the body section of your literature review using one of the structural approaches we discussed above, you’ll need to “wrap up” your literature review and pull all the pieces together to set the direction for the rest of your dissertation or thesis.

The conclusion is where you’ll present the key findings of your literature review. In this section, you should emphasise the research that is especially important to your research questions and highlight the gaps that exist in the literature. Based on this, you need to make it clear what you will add to the literature – in other words, justify your own research by showing how it will help fill one or more of the gaps you just identified.

Last but not least, if it’s your intention to develop a conceptual framework for your dissertation or thesis, the conclusion section is a good place to present this.

Example: Thematically Structured Review

In the video below, we unpack a literature review chapter so that you can see an example of a thematically structure review in practice.

Let’s Recap

In this article, we’ve  discussed how to structure your literature review for maximum impact. Here’s a quick recap of what  you need to keep in mind when deciding on your literature review structure:

• Just like other chapters, your literature review needs a clear introduction , body and conclusion .
• The introduction section should provide an overview of what you will discuss in your literature review.
• The body section of your literature review can be organised by chronology , theme or methodology . The right structural approach depends on what you’re trying to achieve with your research.
• The conclusion section should draw together the key findings of your literature review and link them to your research questions.

If you’re ready to get started, be sure to download our free literature review template to fast-track your chapter outline.

Psst… there’s more!

This post is an extract from our bestselling short course, Literature Review Bootcamp . If you want to work smart, you don't want to miss this .

You Might Also Like:

27 Comments

Great work. This is exactly what I was looking for and helps a lot together with your previous post on literature review. One last thing is missing: a link to a great literature chapter of an journal article (maybe with comments of the different sections in this review chapter). Do you know any great literature review chapters?

I agree with you Marin… A great piece

I agree with Marin. This would be quite helpful if you annotate a nicely structured literature from previously published research articles.

Awesome article for my research.

I thank you immensely for this wonderful guide

It is indeed thought and supportive work for the futurist researcher and students

Very educative and good time to get guide. Thank you

Great work, very insightful. Thank you.

Thanks for this wonderful presentation. My question is that do I put all the variables into a single conceptual framework or each hypothesis will have it own conceptual framework?

Thank you very much, very helpful

This is very educative and precise . Thank you very much for dropping this kind of write up .

Pheeww, so damn helpful, thank you for this informative piece.

I’m doing a research project topic ; stool analysis for parasitic worm (enteric) worm, how do I structure it, thanks.

comprehensive explanation. Help us by pasting the URL of some good “literature review” for better understanding.

great piece. thanks for the awesome explanation. it is really worth sharing. I have a little question, if anyone can help me out, which of the options in the body of literature can be best fit if you are writing an architectural thesis that deals with design?

I am doing a research on nanofluids how can l structure it?

Beautifully clear.nThank you!

Lucid! Thankyou!

Brilliant work, well understood, many thanks

I like how this was so clear with simple language 😊😊 thank you so much 😊 for these information 😊

Insightful. I was struggling to come up with a sensible literature review but this has been really helpful. Thank you!

You have given thought-provoking information about the review of the literature.

Thank you. It has made my own research better and to impart your work to students I teach

I learnt a lot from this teaching. It’s a great piece.

I am doing research on EFL teacher motivation for his/her job. How Can I structure it? Is there any detailed template, additional to this?

You are so cool! I do not think I’ve read through something like this before. So nice to find somebody with some genuine thoughts on this issue. Seriously.. thank you for starting this up. This site is one thing that is required on the internet, someone with a little originality!

I’m asked to do conceptual, theoretical and empirical literature, and i just don’t know how to structure it

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

• Print Friendly

• Reserve a study room
• Library Account
• Undergraduate Students
• Graduate Students
• Faculty & Staff

How to Conduct a Literature Review (Health Sciences and Beyond)

• What is a Literature Review?

The Research Questions

• Selection Criteria
• Database Search
• Documenting Your Search
• Organize Key Findings
• Reference Management

Background vs. Foreground Questions

You may need to find answers to background questions (i.e. about general knowledge) before seeking answers to foreground questions (i.e. about specific knowledge, such as information that might inform a clinical decision).

The research questions on this page are for foreground questions.

A well-formulated research question:

• starts your entire search process
• provides focus for your searches
• guides the selection of literature sources

Question formats are helpful tools researchers can use to structure a question that will facilitate a focused search. Such formats include: PICO , PEO , SPIDER , and  COSMIN . Other formats can be found here .  

The  PICO  format is commonly used in evidence-based clinical practice.  This format creates a "well-built" question that identifies four concepts: (1) the P atient problem or P opulation, (2) the I ntervention, (3) the C omparison (if there is one), and (4) the O utcome(s) .

Example : In adults with recurrent furunculosis (skin boils), do prophylactic antibiotics, compared to no treatment, reduce the recurrence rate?  ( Cochrane Library Tutorial, 2005 )

The  PEO  question format is useful for qualitative research questions. Questions based on this format identify three concepts: (1) P opulation, (2) E xposure, and (3) O utcome(s) .

Example:  In infants, is there an association between exposure to soy milk and the subsequent development of peanut allergy ( Levine, Ioannidis, Haines, & Guyatt, 2014 )?

The  SPIDER  question format was adapted from the PICO tool to search for qualitative and mixed-methods research.  Questions based on this format identify the following concepts: (1) S ample, (2) P henomenon of I nterest, (3) D esign, (4) E valuation, and (5) R esearch type .

Example:  What are young parents’ experiences of attending antenatal education? 

Search for ( S  AND  P of I   AND ( D  OR  E ) AND  R ) ( Cooke, Smith, & Booth, 2012 ).

The COSMIN  ( CO nsensus-based  S tandards for the selection of health status M easurement IN struments ) format is used for systematic review of measurement properties.  Questions based on this format identify (1) the construct or the name(s) of the outcome measurement instrument(s) of interest,  (2) the target population, (3) the type of measurement instrument of interest, and (4) the measurement properties on which the review focuses.

Visit the COSMIN website to view the COSMIN manual and checklist.

• << Previous: What is a Literature Review?
• Next: Selection Criteria >>
• Last Updated: Mar 15, 2024 12:22 PM
• URL: https://guides.library.vcu.edu/health-sciences-lit-review

Conducting a Literature Review

• Literature Review
• Developing a Topic
• Planning Your Literature Review
• Developing a Search Strategy
• Managing Citations
• Critical Appraisal Tools
• Writing a Literature Review

Developing a Research Question

One of the hardest parts of a literature review is to develop a good research question.  You don't want a research question that is so broad it encompasses too many research areas, and can't be reasonably answered. 

Defining your topic may require an initial review of literature on your topic to get a sense of the scope about your topic.   Select a topic of interest, and do a preliminary search to see what kinds of research is being done and what is trending in that topic area.  This will give you a better sense of the topic, and help you focus your research question

In specifying your topic or research question, you should think about setting appropriate limitations on the research you are seeking. Limiting, for example, by time, personnel, gender, age, location, nationality etc. results in a more focused and meaningful topic. 

You may start with a general question:  

Why did the chicken cross the road? 

This question is so general that you could be gathering relevant research for days. 

A more precise research question might be:  

What are some of the environmental factors that occurred in New Jersey between January and June 2021 that would cause a chicken to cross Route 3?  

This research question is specific about a number of variables like time, geography, etc.  

Some Questions to Ask

Some questions to think about as you develop your literature review:

• What is known about the subject?
• Are there any gaps in the knowledge of the subject?
• Have areas of further study been identified by other researchers that you may want to consider?
• Who are the significant research personalities in this area?
• Is there consensus about the topic?
• What aspects have generated significant debate on the topic?
• What methods or problems were identified by others studying in the field and how might they impact your research?
• What is the most productive methodology for your research based on the literature you have reviewed?
• What is the current status of research in this area?
• What sources of information or data were identified that might be useful to you?
• How detailed? Will it be a review of ALL relevant material or will the scope be limited to more recent material, e.g., the last five years.
• Are you focusing on methodological approaches; on theoretical issues; on qualitative or quantitative research?
• The Interprofessional Health Sciences Library
• 123 Metro Boulevard
• Nutley, NJ 07110
• [email protected]
• Student Services
• Parents and Families
• Career Center
• Web Accessibility
• Visiting Campus
• Public Safety
• Disability Support Services
• Campus Security Report
• Report a Problem
• Login to LibApps

Have an account?

Research Methodology(literature review)

Professional development, social studies.

3 questions

Introducing new   Paper mode

No student devices needed.   Know more

Which statements are TRUE about literature review?

Literature review should focus on a broader topic.

Literature review is just the summary of current research.

Literature review is a summary of your ow research.

Literature review is an examination of the literature and it enables the researcher to know what has been learned.

What should you do with the literature review?

Keep bibliographic for every source

Try to read everything

Reading but not writing

Failing to keep complete bibliographic

What is the critical analysis of the relationship among

different works?

introduction

literature review

Explore all questions with a free account

Continue with email

Continue with phone

• Case report
• Open access
• Published: 28 June 2024

Diagnosis and management of COVID toes in outpatients: a case report

• Marie Pouxe 1 ,
• Aziz Abdulkarim 1 ,
• Serge de Vallière   ORCID: orcid.org/0000-0001-9394-629X 1 , 2 ,
• Teofila Seremet   ORCID: orcid.org/0000-0002-3789-6256 3 ,
• Bernard Favrat   ORCID: orcid.org/0000-0001-6920-099X 1 &
• Ioannis Kokkinakis   ORCID: orcid.org/0000-0002-0033-0988 1  

Journal of Medical Case Reports volume  18 , Article number:  307 ( 2024 ) Cite this article

168 Accesses

Metrics details

Since the beginning of the coronavirus disease 2019 pandemic, the most common skin lesions observed due to infection with the severe acute respiratory syndrome coronavirus 2 are pseudochilblains (or coronavirus disease toes). However, this pathology remains infrequent and difficult to diagnose, as no specific test exists.

Case presentation

Two Caucasian women, 30 and 22 years old, presented to our General Medicine Unit with perniosis lesions on the feet during the first two waves of the coronavirus disease 2019 pandemic. They did not have respiratory or general symptoms of severe acute respiratory syndrome coronavirus 2 infection, the reverse transcription polymerase chain reaction on nasopharyngeal swabs was negative, and the serology was positive only in the first case. The clinical presentation differed for the two cases, as the second patient suffered from swelling and burning after cold application. The diagnosis was based on clinical presentation, temporality, exclusion of other differential diagnoses, and blood test results (positive serology in the first case and high level of CXCL13 and VEGF in the second), supported by current literature. Lesions resolved spontaneously in the first patient. The second case was hospitalized for pain management and received corticosteroid therapy with resolution of the symptoms.

These two cases with different clinical presentations illustrate the diagnostic approach to coronavirus disease 2019, a challenging disease with diverse manifestations, including, in some cases, coronavirus disease toes. We present a literature review that illustrates the progression of scientific research. Skin lesions associated with coronavirus disease 2019 infection could be the expression of an important interferon type 1 response and should be considered in the differential diagnosis in a primary care setting.

Peer Review reports

In December 2019, a new virus was identified in Wuhan, China, called severe acute respiratory syndrome coronavirus (SARS-CoV2-). The virus spread rapidly worldwide, and the World Health Organization (WHO) declared this outbreak a pandemic on 12 March 2020 [ 1 ]. At the onset of the pandemic, COVID-19 infection was described as pneumonia that could induce respiratory failure. Epidemiological studies showed that the most frequent symptoms were fever, malaise, fatigue, cough, sputum, and dyspnea [ 2 ]. Other common symptoms included neurological symptoms, myalgia, diarrhea, headache, rhinitis, and chest pain [ 3 ]. In addition, an abnormally high number of skin lesions associated with COVID-19 were described [ 4 ].

Pseudochilblains, also called pernio or COVID toes, are the most common skin manifestations of COVID-19 disease [ 5 ]. They are mainly described in Europe and the USA but are rarely reported in Asia [ 6 ]. Depending on the studies, the prevalence of pseudochilblains in suspected or confirmed patients with COVID-19 ranges from 3.7% to 29% [ 4 , 7 ]. The prevalence of any skin lesion in suspected or confirmed COVID-19 patients was 60%, of which 48% were pseudochilblains [ 4 ]. In a study with patients with confirmed COVID-19 suffering from skin lesions, the prevalence of pseudochilblains was 30% [ 8 ].

Lesions appear as acral inflammatory erythematous–violaceous macules or papules that can sometimes become blisters or swelling [ 9 ]. Clinical presentation and pathologic features are similar to idiopathic and autoimmune-related chilblains, without the same epidemiologic characteristics (no cold exposure or autoimmune disorder) and are thus called “pseudochilblains” [ 10 ]. Patients frequently show no or few symptoms of COVID-19 before or during pseudochilblains lesions, and biological tests are often negative, making it difficult to prove a link between the skin lesions and the infection. As scientific progress is being made, literature shows evidence of this link [ 11 ].

In this article, we present two cases of COVID toes, which we observed in our emergency department in Lausanne, Switzerland, during two different local waves of COVID-19.

A 30-year-old Caucasian woman presented at the beginning of the SARS-CoV-2 pandemic with foot lesions. She complained of erythematous swelling and a burning sensation on the left foot that had started at T  = 0, initially limited to the third and fourth toes and then progressed to other toes and increased in intensity. At T  = 10 days, she developed blisters, which evolved into desquamation with a diminution of the erythema. The patient complained of tiredness but had no fever or other systemic symptoms during this period. She had no relevant past medical history or family disease.

During the clinical evaluation at T  = 1 month, we noticed a desquamation of several toes without pain, swelling, or redness. Complete blood count and C-reactive protein were normal. A reverse transcription polymerase chain reaction (RT–PCR) test for SARS-CoV-2 on a nasopharyngeal swab was not performed due to the delay since the onset of symptoms and the absence of respiratory manifestations. The SARS-CoV-2 serology at T  = 1 month showed a weak positive IgM level at 2.58 g/L (normal value < 2.41 g/L). A biopsy was not performed, as no skin lesions were found on examination. We retained the pseudochilblain diagnosis based on current literature supporting clinical diagnosis during the COVID-19 outbreak and after excluding other differential diagnoses [ 12 ]. Symptoms resolved spontaneously without treatment within T  = 2 months.

Second case

A 22-year-old Caucasian woman with no other comorbidities ans not vaccinated against COVID-19 presented in the second year of the SARS-CoV-2 pandemic with acute foot lesions without fever or other systemic manifestations. She reported redness, heat, swelling, and itching of all toes associated with pain at T  = 0 (Fig.  1 ). The symptoms were constant, relieved by cold and worsened by heat. She described the appearance of a nodule on the right second toe that lasted for a few days (Fig.  2 ). Dark red punctiform lesions appeared on some toes. During the examination at T  = 5 days, she declared that she was not relieved by topical and oral anti-inflammatory drugs, paracetamol, or antihistamine medication.

Initial clinical presentation of chilblain-like lesions ( T  = 0). Diffuse redness macules and slight swelling of the toes as initial clinical presentation

Evolution two days after the onset of symptoms ( T  = 2 days). A purpuric lesion appeared on the second right toe. Improvement of diffuse redness and edema

The SARS-CoV-2 RT–PCR on nasopharyngeal swabs at T  = 5 and T  = 45 days were negative. At T  = 3 weeks, the toe edema temporarily subsided. A few days later, inflammatory swelling of the two feet reappeared with a burning sensation and diffuse dysesthesia. Other dark red lesions, punctiform or linear, appeared on the back of the two feet (Fig.  3 ), which became crusted.

Evolution 1 month after the onset of symptoms ( T  = 1 month). Inflammatory swelling of the two feet with dark red lesions, punctiform or linear, on the back of the two feet

The patient was finally hospitalized for better pain management at T  = 6 weeks. Clinical examination revealed warm, erythematous edema of both feet and ankles, spreading upwards over both legs’ lower third. There were dark red, punctiform, or linear lesions on the toes and back of the feet. A cyanotic border surrounded some lesions; others were cracked and had a crusty or fibrinous background (Fig.  4 ). The rest of the examination showed a slight decrease in pallesthesia (7.5/8), probably related to edema. A screening for autoimmune diseases and a SARS-CoV-2 serology were negative. The CXCL13 chemokine was elevated at 1021 pg/ml (normal value < 114 pg/ml), and VEGF-A was slightly increased at 663 pg/ml (normal value < 569 pg/ml). Magnetic resonance imaging (MRI) of the feet and ankles showed tenosynovitis of the long extensor digitorum tendons of the right foot, possibly secondary to adjacent tissue compression and fatty infiltration of the dorsum of both feet. Following a dermatological assessment, a skin biopsy was not performed due to extensive swelling.

Evolution after 6 weeks after the onset of symptoms ( T  = 6 weeks). Erythematous edema with dark red, punctiform, or linear crusted lesions on the back of the feet

Although the SARS-CoV-2 RT–PCR and serologies were negative, we retained the pseudochilblain diagnosis after excluding other differential diagnoses and considering COVID toes as a clinical diagnosis during a COVID-19 outbreak following the current literature [ 12 ]. The patient's skin lesions resembled those described in literature in association with waves of COVID-19, and we found an increase of CXCL13 chemokine in the blood, which was related to severe COVID-19 infection [ 13 ]. The diagnosis of erythromelalgia was considered unlikely due to the nonparoxysmal nature of the symptoms.

Symptoms improved significantly after a few days of oral prednisone therapy, 40 mg per day, followed by 20 mg per day for 7 days and 10 mg per day for 7 days. The patient was also treated with pregabalin and tramadol for pain relief and 300 mg of aspirin per day. The pain resolved in a few weeks, she stopped taking painkillers, and the lesions did not recur at T  = 1 year (Fig.  5 ).

Evolution 1 year after the onset of symptoms ( T  = 1 year). No redness or swelling. There are still residual marks from the crusts on the back of the left foot

These two cases of COVID toes had different clinical presentations and evolution, but they both developed symptoms during pandemic waves within a context of high COVID-19 infection incidence, indicating a temporal link. We retained the diagnosis of pseudochilblains or COVID toes based mainly on clinical elements after excluding other differential diagnoses and following current literature for indirect evidence of this association during the COVID-19 outbreak.

The first patient presented typical symptoms and recovered within 1 month. Serology revealed positive IgM for SARS-CoV-2 without positive IgG at T  = 1 month. Studies have shown that antibody titers appear related to disease severity, and a low or undetectable IgG rate might be related to a strong viral clearance [ 14 ].

The second patient had different interesting features of COVID toes, with swelling and a burning sensation after cold application. Although the patient had a negative serology, current literature has shown that an excessive interferon-alpha (IFN-α) response might clear the virus before humoral immunity occurs and explain the negative serology [ 15 ]. Furthermore, the second patient had a high level of CXCL13, which has been shown to be a marker of COVID-19 severity [ 16 ], and a high level of VEGF-A, an endothelial marker, that has been found in late pseudochilblains 20 days after symptom onset [ 17 ]. The evolution was slow over about 2 months and required corticosteroid therapy.

COVID toes occur mainly in children and young adults, last on average 14 days (up to 1 month) and resolve spontaneously. Despite the frequent absence or low-grade typical COVID-19 symptoms, recent contact with COVID-19 patients is often found. The skin lesions occur 2 to 4 weeks after the initial manifestation [ 18 ]. These lesions are primarily associated with mild COVID-19 with a good prognosis [ 19 ]. Laboratory tests are often negative. A study found 7% positive tests without distinguishing RT–PCR or serologies [ 19 ]. In another study, 30% of patients had positive serologies without positive RT–PCR [ 15 ]. Finally, some authors showed negative IgG with positive IgA in children in 31.6% of patients [ 20 ]. The prevalence of these lesions is not known with certainty, but it has been estimated to be between 3.7% and 29% of patients with suspected or confirmed COVID-19 [ 4 , 7 ].

The formal causal link between pseudochilblains and COVID-19 remains challenging to prove. The characterization of this syndrome relies primarily on retrospective studies with many limitations, and the RT–PCR or serologies are often negative [ 20 ]. The RT–PCR is negative in most patients, probably because pseudochilblains appear late during the infection. It has also been shown that patients with COVID-19 with mild or no symptoms have lower viral loads that are more challenging to detect [ 15 ]. Also, we currently know that specific early humoral response to SARS-CoV-2 is dominated by IgA, which has a crucial role in virus neutralization. Serologies might be negative because IgA is not analyzed in routine serology [ 21 , 22 ]. At last, as mentioned above, this could be related to a strong IFN-α response that clears the virus before the humoral response [ 15 ].

Type 1 IFN is known as essential for host defense against viruses and a high concentration of type 1 interferon-inducible myxovirus resistance protein A has been found in pseudochilblain biopsies, similar to the transcriptional signature of lupus [ 23 ]. Therefore, some people would have a strong type 1 IFN response that would favor rapid elimination of the virus without suffering from the disease, thus blocking seroconversion. Pseudochilblains would, therefore, be a viral-induced type I interferonopathy [ 24 ]. A study has identified a stimulator of interferon genes (STING)-dependent type I IFN signature that is primarily mediated by macrophages adjacent to areas of endothelial cell damage [ 25 ].

A recent review proposes that the mechanism of COVID toes is related to an interaction between SARS-CoV-2 cell infection via ACE2, the RAAS (renin–angiotensin–aldosterone system), sex hormones, and the IFN type I immune response [ 26 ]. In a small case series, viral particles were found in endothelial cells, suggesting that the vessel damage could be directly related to SARS-CoV2 [ 27 ]. Finally, one study compared idiopathic chilblains to pseudochilblains, demonstrating histological similarities. However, pseudochilblains showed a systemic response with a high prevalence of ANCA, IgA, and IFN type 1 and an endothelial alteration [ 17 ].

Pseudochilblain or COVID toes is a recent disease that emerged with the COVID-19 pandemic. Current literature suggests pathophysiological mechanisms related to a cutaneous expression of an important IFN type 1 response. The typical clinical presentation, history, and examination, the consideration and exclusion of other differential diagnoses, the absence of a history of perniosis, and the timing of COVID waves allow the diagnosis of pseudochilblains following a COVID-19 infection. Clinicians in primary care should be aware of chilblain-like lesions and suspect COVID-19 infection, even in the presence of a negative RT–PCR test and negative serology.

Availability of data and materials

All data generated or analyzed during this study are included in this published article.

Muralidar S, Ambi SV, Sekaran S, Krishnan UM. The emergence of COVID-19 as a global pandemic: understanding the epidemiology, immune response and potential therapeutic targets of SARS-CoV-2. Biochimie. 2020;179:85–100.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Wu D, Wu T, Liu Q, Yang Z. The SARS-CoV-2 outbreak: What we know. Int J Infect Dis. 2020;94:44–8.

da Rosa Mesquita R, Francelino Silva Junior LC, Santos Santana FM, Farias de Oliveira T, Campos Alcântara R, Monteiro Arnozo G, et al . Clinical manifestations of COVID-19 in the general population: systematic review. Wien Klin Wochenschr. 2021;133:377–82.

Perna A, Passiatore M, Massaro A, Terrinoni A, Bianchi L, Cilli V, et al . Skin manifestations in COVID-19 patients, state of the art A systematic review. Int J Dermatol. 2021;60:547–53.

Jimenez-Cebrian AM, Castro-Mendez A, García-Podadera B, Romero-Galisteo R, Medina-Alcántara M, Garcia-Paya I, et al . Clinical manifestations of COVID-19 in the feet: a review of reviews. J Clin Med. 2021;10:2201.

Tan SW, Tam YC, Oh CC. Skin manifestations of COVID-19: a worldwide review. JAAD Int. 2021;2:119–33.

Article   PubMed   Google Scholar  

Mascitti H, Jourdain P, Bleibtreu A, Jaulmes L, Dechartres A, Lescure X, et al . Prognosis of rash and chilblain-like lesions among outpatients with COVID-19: a large cohort study. Eur J Clin Microbiol Infect Dis. 2021;40:2243–8.

Giavedoni P, Podlipnik S, Pericàs JM, Fuertes de Vega I, García-Herrera A, Alós L, et al . Skin Manifestations in COVID-19: prevalence and relationship with disease severity. J Clin Med. 2020;9:3261.

Kashetsky N, Mukovozov IM, Bergman J. Chilblain-Like Lesions (CLL) Associated With COVID-19 (“COVID Toes”): A Systematic Review. J Cutan Med Surg. 2021;120347542110045.

Kanitakis J, Lesort C, Danset M, Jullien D. Chilblain-like acral lesions during the COVID-19 pandemic (“COVID toes”): Histologic, immunofluorescence, and immunohistochemical study of 17 cases. J Am Acad Dermatol. 2020;83:870–5.

Feito-Rodríguez M, Mayor-Ibarguren A, Cámara-Hijón C, Montero-Vega D, Servera-Negre G, Ruiz-Bravo E, et al . Chilblain-like lesions and COVID-19 infection: a prospective observational study at Spain’s ground zero. J Am Acad Dermatol. 2021;84:507–9.

Kolivras A, Thompson C, Pastushenko I, Mathieu M, Bruderer P, Vicq M, et al . A clinicopathological description of COVID ‐19‐induced chilblains ( COVID ‐toes) correlated with a published literature review. J Cutan Pathol. 2021;cup.14099.

Alturaiki W, Alkadi H, Alamri S, Awadalla ME, Alfaez A, Mubarak A, et al . Association between the expression of toll-like receptors, cytokines, and homeostatic chemokines in SARS-CoV-2 infection and COVID-19 severity. Heliyon. 2023;9: e12653.

Article   CAS   PubMed   Google Scholar  

Tan W, Lu Y, Zhang J, Wang J, Dan Y, Tan Z, et al . Viral Kinetics and Antibody Responses in Patients with COVID-19. Infect Dis (except HIV/AIDS); 2020. https://doi.org/10.1101/2020.03.24.20042382

Hubiche T, Cardot-Leccia N, Le Duff F, Seitz-Polski B, Giordana P, Chiaverini C, et al . Clinical, laboratory, and interferon-alpha response characteristics of patients with chilblain-like lesions during the COVID-19 pandemic. JAMA Dermatol. 2021;157:202.

Perreau M, Suffiotti M, Marques-Vidal P, Wiedemann A, Levy Y, Laouénan C, et al . The cytokines HGF and CXCL13 predict the severity and the mortality in COVID-19 patients. Nat Commun. 2021;12:4888.

Frumholtz L, Bouaziz J ‐D., Battistella M, Hadjadj J, Chocron R, Bengoufa D, et al . Type I interferon response and vascular alteration in chilblain‐like lesions during the COVID‐19 outbreak. Br J Dermatol. 2021;bjd.20707.

Gisondi P, Di Leo S, Bellinato F, Cazzaniga S, Piaserico S, Naldi L. Time of onset of selected skin lesions associated with COVID-19: a systematic review. Dermatol Ther. 2021;11:695–705.

Article   Google Scholar  

Freeman EE, McMahon DE, Lipoff JB, Rosenbach M, Kovarik C, Takeshita J, et al . Pernio-like skin lesions associated with COVID-19: A case series of 318 patients from 8 countries. J Am Acad Dermatol. 2020;83:486–92.

El Hachem M, Diociaiuti A, Concato C, Carsetti R, Carnevale C, CiofiDegliAtti M, et al . A clinical, histopathological and laboratory study of 19 consecutive Italian paediatric patients with chilblain-like lesions: lights and shadows on the relationship with COVID-19 infection. J Eur Acad Dermatol Venereol. 2020;34:2620–9.

Article   PubMed   PubMed Central   Google Scholar  

Lippi G, Mattiuzzi C. Clinical value of anti-SARS-COV-2 serum IgA titration in patients with COVID-19. J Med Virol. 2021;93:1210–1.

Mathian A, Sterlin D, Miyara M, Mohr A, Anna F, Quentric P, et al . Les Immunoglobulines A dominent la réponse anticorps neutralisante précoce anti-SARS-CoV-2. Rev Médecine Interne. 2020;41:A63–4.

Magro CM, Mulvey JJ, Laurence J, Sanders S, Crowson AN, Grossman M, et al . The differing pathophysiologies that underlie COVID-19-associated perniosis and thrombotic retiform purpura: a case series. Br J Dermatol. 2021;184:141–50.

Yatim A. Gilliet M [COVID-toes, a cutaneous sign of innate resistance to SARS-CoV-2]. Rev Med Suisse. 2021;17:646–52.

PubMed   Google Scholar  

Domizio JD, Gulen MF, Saidoune F, Thacker VV, Yatim A, Sharma K, et al . The cGAS–STING pathway drives type I IFN immunopathology in COVID-19. Nature. 2022;603:145–51.

Cappel MA, Cappel JA, Wetter DA. Pernio (Chilblains), SARS-CoV-2, and COVID toes unified through cutaneous and systemic mechanisms. Mayo Clin Proc. 2021;96:989–1005.

Colmenero I, Santonja C, Alonso-Riaño M, Noguera-Morel L, Hernández-Martín A, Andina D, et al . SARS-CoV-2 endothelial infection causes COVID-19 chilblains: histopathological, immunohistochemical and ultrastructural study of seven paediatric cases. Br J Dermatol. 2020;183:729–37.

Download references

Acknowledgements

Not applicable.

Open access funding provided by University of Lausanne

Author information

Authors and affiliations.

University Center for Primary Care and Public Health, Unisanté, Lausanne, Switzerland

Marie Pouxe, Aziz Abdulkarim, Serge de Vallière, Bernard Favrat & Ioannis Kokkinakis

Service of Infectious Diseases, Lausanne University Hospital (CHUV), Lausanne, Switzerland

Serge de Vallière

Service of Dermatology, Lausanne University Hospital (CHUV), Lausanne, Switzerland

Teofila Seremet

You can also search for this author in PubMed   Google Scholar

Contributions

MP drafted the case reports, reviewed literature, and wrote the manuscript. AA cared for one patient and revised the manuscript. BF cared for one patient and revised the manuscript. TS and SDV revised the manuscript as dermatology and infectious diseases consultants. IK reviewed the literature, revised and supervised the manuscript, and was the correspondace author. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Ioannis Kokkinakis .

Ethics declarations

Ethics approval and consent to participate.

This case report did not include any treatment differing from the management of the case suggested by current guidelines, so approval from the ethics committee was not required.

Consent for publication

Written informed consent was obtained from the patient for publication of this case report and any accompanying images. A copy of the written consent is available for review by the Editor-in-Chief of this journal.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Pouxe, M., Abdulkarim, A., de Vallière, S. et al. Diagnosis and management of COVID toes in outpatients: a case report. J Med Case Reports 18 , 307 (2024). https://doi.org/10.1186/s13256-024-04626-9

Download citation

Received : 17 August 2022

Accepted : 07 June 2024

Published : 28 June 2024

DOI : https://doi.org/10.1186/s13256-024-04626-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Journal of Medical Case Reports

ISSN: 1752-1947

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Autonomous self-healing agents in cementitious materials: parameters and impacts on mortar properties.

1. Introduction

2. systematic literature review, bibliometric analysis, 3. results and discussion, 3.1. parameters for the preparation and composition of self-healing material mixtures, 3.2. parameters adopted in the encapsulation of healing agents, 3.3. parameters adopted for crack control and curing environment, 3.4. the influence of pellets/granules/capsules on the properties of mortar, 3.5. the influence of healing agents on the self-healing properties of mortars, 4. conclusions and research gaps.

• The predominant encapsulation method used for healing agents was the simultaneous use of granules and pellets in equipment designated as granulators and pelletizers, respectively. Additionally, the predominant surface coating mainly consists of a PVA film. However, cementitious materials incorporating pellets with a PVA film coating significantly reduced spreading and compressive strength properties.
• Adverse behavior was observed in these properties for capsules coated with Portland cement and sprayed with a water and sodium silicate solution. This indicates a trend towards their use, as it reduces the negative effect of pellet/capsule incorporation on mortar properties. Additionally, this type of coating, regarding the use of expansive agents through pellets/capsules as the core, may provide benefits in crack sealing and increase the incorporation content in cementitious materials to promote self-healing.
• It was observed in this systematic review that the preferred pre-crack test method is the three-point flexural test. Steel wires are used to prevent the complete rupture of the prismatic sample, and a small notch (1.5 mm) at the bottom center of the sample is made to induce a crack at a specific point. The CMOD device is the central apparatus for crack control to observe crack opening.
• Regarding the parameters for evaluating self-healing, it was observed that for pellets, granules, or capsules composed of expansive agents, the pre-crack test is generally conducted at 7 days, and the curing time and duration of the crack sealing monitoring process are both approximately 28 days. Additionally, the environment for conducting the autonomous healing of cement-based materials was water.

Supplementary Materials

Author contributions, data availability statement, acknowledgments, conflicts of interest.

• Gwon, S.; Ahn, E.; Shin, M. Water Permeability and Rapid Self-Healing of Sustainable Sulfur Composites Using Superabsorbent Polymer and Binary Cement. Constr. Build. Mater. 2020 , 265 , 120306. [ Google Scholar ] [ CrossRef ]
• Gwon, S.; Ahn, E.; Shin, M. Self-Healing of Modified Sulfur Composites with Calcium Sulfoaluminate Cement and Superabsorbent Polymer. Compos. Part B 2019 , 162 , 469–483. [ Google Scholar ] [ CrossRef ]
• Park, B.; Cheol, Y. Effect of Healing Products on the Self-Healing Performance of Cementitious Materials with Crystalline Admixtures. Constr. Build. Mater. 2021 , 270 , 121389. [ Google Scholar ] [ CrossRef ]
• Sinha, A.; Wang, Q.; Wei, J. Feasibility and Compatibility of a Biomass Capsule System in Self-Healing Concrete. Materials 2021 , 14 , 958. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Cheol, Y.; Park, B. Enhanced Autogenous Healing of Ground Granulated Blast Furnace Slag Blended Cements and Mortars. Integr. Med. Res. 2019 , 8 , 3443–3452. [ Google Scholar ] [ CrossRef ]
• Qureshi, T.; Kanellopoulos, A.; Al-Tabbaa, A. Autogenous Self-Healing of Cement with Expansive Minerals-I: Impact in Early Age Crack Healing. Constr. Build. Mater. 2018 , 192 , 768–784. [ Google Scholar ] [ CrossRef ]
• Litina, C.; Bumanis, G.; Anglani, G.; Dudek, M.; Maddalena, R.; Amenta, M.; Papaioannou, S.; Pérez, G.; García Calvo, J.L.; Asensio, E.; et al. Evaluation of Methodologies for Assessing Self-Healing Performance of Concrete with Mineral Expansive Agents: An Interlaboratory Study. Materials 2021 , 14 , 2024. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Youssef Namnoum, C.; Hilloulin, B.; Grondin, F.; Loukili, A. Determination of the Origin of the Strength Regain after Self-Healing of Binary and Ternary Cementitious Materials Including Slag and Metakaolin. J. Build. Eng. 2021 , 41 , 102739. [ Google Scholar ] [ CrossRef ]
• Feng, J.; Qian, S. Accelerating Autonomic Healing of Cementitious Composites by Using Nano Calcium Carbonate Coated Polypropylene Fibers. Mater. Des. 2023 , 225 , 111549. [ Google Scholar ] [ CrossRef ]
• Huang, H.; Yuan, Y.; Zhang, W.; Zhu, L. Property Assessment of High-Performance Concrete Containing Three Types of Fibers. Int. J. Concr. Struct. Mater. 2021 , 15 , 39. [ Google Scholar ] [ CrossRef ]
• Van Tittelboom, K.; Gruyaert, E.; Rahier, H.; De Belie, N. Influence of Mix Composition on the Extent of Autogenous Crack Healing by Continued Hydration or Calcium Carbonate Formation. Constr. Build. Mater. 2012 , 37 , 349–359. [ Google Scholar ] [ CrossRef ]
• Sisomphon, K.; Copuroglu, O.; Koenders, E.A.B. Self-Healing of Surface Cracks in Mortars with Expansive Additive and Crystalline Additive. Cem. Concr. Compos. 2012 , 34 , 566–574. [ Google Scholar ] [ CrossRef ]
• Belie, N.D.; Gruyaert, E.; Al-tabbaa, A.; Antonaci, P.; Baera, C.; Bajare, D.; Darquennes, A.; Davies, R.; Ferrara, L.; Jefferson, T.; et al. A Review of Self-Healing Concrete for Damage Management of Structures. Adv. Mater. Interfaces 2018 , 5 , 1800074. [ Google Scholar ] [ CrossRef ]
• Reinhardt, H.W.; Jonkers, H.; Tittelboom, K.V.; Snoeck, D.; De Belie, N.; De Muynck, W.; Verstraete, W.; Wang, J.; Mechtcherine, V. Recovery against Environmental Action. In Self-Healing Phenomena in Cement-Based Materials: State-of-the-Art Report of RILEM Technical Committee ; Springer: Dordrecht, The Netherlands, 2013; p. 240. [ Google Scholar ]
• Zhang, W.; Zheng, Q.; Ashour, A.; Han, B. Self-Healing Cement Concrete Composites for Resilient Infrastructures: A Review. Compos. Part B 2020 , 189 , 107892. [ Google Scholar ] [ CrossRef ]
• Reinhardt, H.-W.; Jooss, M. Permeability and Self-Healing of Cracked Concrete as a Function of Temperature and Crack Width. Cem. Concr. Res. 2003 , 33 , 981–985. [ Google Scholar ] [ CrossRef ]
• De Brabandere, L.; Van Mullem, T.; Lee, J.H.; Suh, J.-I.; Lee, K.-M.; De Belie, N. Comparative Analysis of Three Different Types of Self-Healing Concrete via Permeability Testing and a Quasi-Steady-State Chloride Migration Test. Constr. Build. Mater. 2024 , 411 , 134288. [ Google Scholar ] [ CrossRef ]
• Anglani, G.; Van Mullem, T.; Tulliani, J.M.; Van Tittelboom, K.; De Belie, N.; Antonaci, P. Durability of Self-Healing Cementitious Systems with Encapsulated Polyurethane Evaluated with a New Pre-Standard Test Method. Mater. Struct. Constr. 2022 , 55 , 143. [ Google Scholar ] [ CrossRef ]
• Zhang, C.; Khorshidi, H.; Najafi, E.; Ghasemi, M. Fresh, Mechanical and Microstructural Properties of Alkali-Activated Composites Incorporating Nanomaterials: A Comprehensive Review. J. Clean. Prod. 2023 , 384 , 135390. [ Google Scholar ] [ CrossRef ]
• Wei, C.; Li, Y.; Liu, X.; Zhang, Z.; Wu, P.; Gu, J. Large-Scale Application of Coal Gasification Slag in Nonburnt Bricks: Hydration Characteristics and Mechanism Analysis. Constr. Build. Mater. 2024 , 421 , 135674. [ Google Scholar ] [ CrossRef ]
• Van Tittelboom, K.V.; Belie, N. Self-Healing in Cementitious Materials—A Review. Materials 2013 , 6 , 2182–2217. [ Google Scholar ] [ CrossRef ]
• Oh, S.; Choi, Y.W.; Yong, J.K. Effect of Cement Powder Based Self-Healing Solid Capsule on the Quality of Mortar. Constr. Build. Mater. 2019 , 214 , 574–580. [ Google Scholar ] [ CrossRef ]
• Ge, Y.; Hu, H.; Zhan, Q.; Zhang, X.; Su, Y.; Zhou, J. Study on Self-Healing Effect of Cement-Based Materials Cracks Based on Various Inorganic Minerals. J. Build. Eng. 2024 , 82 , 108202. [ Google Scholar ] [ CrossRef ]
• Wani, S.; Jan Gęca, M.; Selvaraj, T.; Shanmuga Priya, T. Assessing the Influence of Bacillus Megaterium and Bacillus Sphaericus in Cementitious Materials: Promoting Sustainability towards Strength, Durability and Crack Repair. Ain Shams Eng. J. 2024 , 15 , 102748. [ Google Scholar ] [ CrossRef ]
• Xiao, X.; Tan, A.C.Y.; Unluer, C.; Yang, E.-H. Development of a Functionally Graded Bacteria Capsule for Self-Healing Concrete. Cem. Concr. Compos. 2023 , 136 , 104863. [ Google Scholar ] [ CrossRef ]
• Lo Monte, F.; Repesa, L.; Snoeck, D.; Doostkami, H.; Roig-Flores, M.; Jackson, S.J.P.; Alvarez, A.B.; Nasner, M.; Borg, R.P.; Schröfl, C.; et al. Multi-Performance Experimental Assessment of Autogenous and Crystalline Admixture-Stimulated Self-Healing in UHPFRCCs: Validation and Reliability Analysis through an Inter-Laboratory Study. Cem. Concr. Compos. 2024 , 145 , 105315. [ Google Scholar ] [ CrossRef ]
• Alghamri, R.; Kanellopoulos, A.; Litina, C.; Al-Tabbaa, A. Preparation and Polymeric Encapsulation of Powder Mineral Pellets for Self-Healing Cement Based Materials. Constr. Build. Mater. 2018 , 186 , 247–262. [ Google Scholar ] [ CrossRef ]
• Wang, X.; Li, W.; Jiang, Z. Preparation and Characterization of Self-Healing Mortar Based on “Build-In” Carbonation. Materials 2020 , 13 , 21. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Papaioannou, S.; Amenta, M.; Kilikoglou, V.; Gournis, D.; Karatasios, I. Synthesis and Integration of Cement-Based Capsules Modified with Sodium Silicate for Developing Self-Healing Cements. Constr. Build. Mater. 2022 , 316 , 125803. [ Google Scholar ] [ CrossRef ]
• Lima, G.T.d.S.; Oliveira, M.V.C.; Pinto, R.C.d.A.; Rocha, J.C. Is the Diffuse Ultrasound Method Reliable for Evaluating Autonomous Self-Healing in Cementitious Materials with Expansive Agent Pellets? Mater. Lett. 2023 , 351 , 135058. [ Google Scholar ] [ CrossRef ]
• Lee, Y.; Ryou, J. Self Healing Behavior for Crack Closing of Expansive Agent via Granulation/Film Coating Method. Constr. Build. Mater. 2014 , 71 , 188–193. [ Google Scholar ] [ CrossRef ]
• Lee, Y.S.; Ryou, J.S. Crack Healing Performance of PVA-Coated Granules Made of Cement, CSA, and Na 2 CO 3 in the Cement Matrix. Materials 2016 , 9 , 555. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Kim, H.G.; Atta-ur-Rehman; Qudoos, A.; Ryou, J.S. Self-Healing Performance of GGBFS Based Cementitious Mortar with Granulated Activators Exposed to a Seawater Environment. Constr. Build. Mater. 2018 , 188 , 569–582. [ Google Scholar ] [ CrossRef ]
• Litina, C.; Al-Tabbaa, A. First Generation Microcapsule-Based Self-Healing Cementitious Construction Repair Materials. Constr. Build. Mater. 2020 , 255 , 119389. [ Google Scholar ] [ CrossRef ]
• Kanellopoulos, A.; Giannaros, P.; Al-Tabbaa, A. The Effect of Varying Volume Fraction of Microcapsules on Fresh, Mechanical and Self-Healing Properties of Mortars. Constr. Build. Mater. 2016 , 122 , 577–593. [ Google Scholar ] [ CrossRef ]
• Qureshi, T.S.; Kanellopoulos, A. Encapsulation of Expansive Powder Minerals within a Concentric Glass Capsule System for Self-Healing Concrete. Constr. Build. Mater. 2016 , 121 , 629–643. [ Google Scholar ] [ CrossRef ]
• Restuccia, L.; Reggio, A.; Ferro, G.A.; Tulliani, J.M. New Self-Healing Techniques for Cement-Based Materials. Procedia Struct. Integr. 2017 , 3 , 253–260. [ Google Scholar ] [ CrossRef ]
• Tittelboom, K.V.; De Belie, N.; Van Loo, D.; Jacobs, P. Self-Healing Efficiency of Cementitious Materials Containing Tubular Capsules Filled with Healing Agent. Cem. Concr. Compos. 2011 , 33 , 497–505. [ Google Scholar ] [ CrossRef ]
• Perez, G.; Erkizia, E.; Gaitero, J.J.; Kaltzakorta, I.; Jiménez, I.; Guerrero, A. Synthesis and Characterization of Epoxy Encapsulating Silica Microcapsules and Amine Functionalized Silica Nanoparticles for Development of an Innovative Self-Healing Concrete. Mater. Chem. Phys. 2015 , 165 , 39–48. [ Google Scholar ] [ CrossRef ]
• Zhu, D.Y.; Rong, M.Z.; Zhang, M.Q. Self-Healing Polymeric Materials Based on Microencapsulated Healing Agents: From Design to Preparation. Prog. Polym. Sci. 2015 , 49–50 , 175–220. [ Google Scholar ] [ CrossRef ]
• Zhang, C.; Wang, M.; Liu, R.; Li, X.; Yan, J.; Du, H. Enhancing Self-Healing Efficiency of Concrete Using Multifunctional Granules and PVA Fibers. J. Build. Eng. 2023 , 76 , 107314. [ Google Scholar ] [ CrossRef ]
• Shao, L.; Feng, P.; Liu, Q.; Chen, C.; Cai, Y.; Xu, G. A Review on Performance Improvement and Multi-Functionalization of Cement Composites Using Capsules. Constr. Build. Mater. 2023 , 409 , 133977. [ Google Scholar ] [ CrossRef ]
• Gupta, S.; Pang, S.D.; Kua, H.W. Autonomous Healing in Concrete by Bio-Based Healing Agents—A Review. Constr. Build. Mater. 2017 , 146 , 419–428. [ Google Scholar ] [ CrossRef ]
• Rajadesingu, S.; Palani, N.; Mendonce, K.C.; Vijayakumar, P.; Monisha, P.; Ayyadurai, S. State-of-the-Art Review on Advancements of Eco-Friendly Bacterial-Infused Self-Healing Concrete for Sustainable Constructions. J. Build. Eng. 2024 , 91 , 109669. [ Google Scholar ] [ CrossRef ]
• Silvestro, L.; Jean, P.; Gleize, P. Effect of Carbon Nanotubes on Compressive, Flexural and Tensile Strengths of Portland Cement-Based Materials: A Systematic Literature Review. Constr. Build. Mater. 2020 , 264 , 120237. [ Google Scholar ] [ CrossRef ]
• Maes, M.; Van Tittelboom, K.; De Belie, N. The Efficiency of Self-Healing Cementitious Materials by Means of Encapsulated Polyurethane in Chloride Containing Environments. Constr. Build. Mater. 2014 , 71 , 528–537. [ Google Scholar ] [ CrossRef ]
• Van Tittelboom, K.; Tsangouri, E.; Van Hemelrijck, D.; De Belie, N. The Efficiency of Self-Healing Concrete Using Alternative Manufacturing Procedures and More Realistic Crack Patterns. Cem. Concr. Compos. 2015 , 57 , 142–152. [ Google Scholar ] [ CrossRef ]
• Perez, G.; Gaitero, J.J.; Erkizia, E.; Jimenez, I.; Guerrero, A. Characterisation of Cement Pastes with Innovative Self-Healing System Based in Epoxy-Amine Adhesive. Cem. Concr. Compos. 2015 , 60 , 55–64. [ Google Scholar ] [ CrossRef ]
• Mostavi, E.; Asadi, S.; Hassan, M.M.; Alansari, M. Evaluation of Self-Healing Mechanisms in Concrete with Double-Walled Sodium Silicate Microcapsules. J. Mater. Civ. Eng. 2015 , 27 , 04015035. [ Google Scholar ] [ CrossRef ]
• Lee, Y.S.; Kim, H.G.; Song, T.H.; Ryou, J.S. The Control of Response Time in Self-Healing of Granulated Cementitious Material by Water-Soluble Film Coating. Procedia Eng. 2015 , 125 , 663–668. [ Google Scholar ] [ CrossRef ]
• Kanellopoulos, A.; Qureshi, T.S.; Al-Tabbaa, A. Glass Encapsulated Minerals for Self-Healing in Cement Based Composites. Constr. Build. Mater. 2015 , 98 , 780–791. [ Google Scholar ] [ CrossRef ]
• Formia, A.; Terranova, S.; Antonaci, P.; Pugno, N.M.; Tulliani, J.M. Setup of Extruded Cementitious Hollow Tubes as Containing/Releasing Devices in Self-Healing Systems. Materials 2015 , 8 , 1897–1923. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Gruyaert, E.; Van Tittelboom, K.; Sucaet, J.; Anrijs, J.; Van Vlierberghe, S.; Dubruel, P.; De Geest, B.G.; Remon, J.P.; De Belie, N. Capsules with Evolving Brittleness to Resist the Preparation of Self-Healing Concrete. Mater. Constr. 2016 , 66 , e092. [ Google Scholar ]
• Li, W.; Zhu, X.; Zhao, N.; Jiang, Z. Preparation and Properties of Melamine Urea-Formaldehyde Microcapsules for Self-Healing of Cementitious Materials. Materials 2016 , 9 , 152. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Feiteira, J.; Gruyaert, E.; De Belie, N. Self-Healing of Moving Cracks in Concrete by Means of Encapsulated Polymer Precursors. Constr. Build. Mater. 2016 , 102 , 671–678. [ Google Scholar ] [ CrossRef ]
• Araújo, M.; Van Vlierberghe, S.; Feiteira, J.; Graulus, G.J.; Van Tittelboom, K.; Martins, J.C.; Dubruel, P.; De Belie, N. Cross-Linkable Polyethers as Healing/Sealing Agents for Self-Healing of Cementitious Materials. Mater. Des. 2016 , 98 , 215–222. [ Google Scholar ] [ CrossRef ]
• Alghamri, R.; Kanellopoulos, A.; Al-Tabbaa, A. Impregnation and Encapsulation of Lightweight Aggregates for Self-Healing Concrete. Constr. Build. Mater. 2016 , 124 , 910–921. [ Google Scholar ] [ CrossRef ]
• De Nardi, C.; Bullo, S.; Ferrara, L.; Ronchin, L.; Vavasori, A. Effectiveness of Crystalline Admixtures and Lime/Cement Coated Granules in Engineered Self-Healing Capacity of Lime Mortars. Mater. Struct. Constr. 2017 , 50 , 191. [ Google Scholar ] [ CrossRef ]
• Dong, B.; Fang, G.; Wang, Y.; Liu, Y.; Hong, S.; Zhang, J.; Lin, S.; Xing, F. Performance Recovery Concerning the Permeability of Concrete by Means of a Microcapsule Based Self-Healing System. Cem. Concr. Compos. 2017 , 78 , 84–96. [ Google Scholar ] [ CrossRef ]
• Al-Ansari, M.; Abu-Taqa, A.G.; Hassan, M.M.; Senouci, A.; Milla, J. Performance of Modified Self-Healing Concrete with Calcium Nitrate Microencapsulation. Constr. Build. Mater. 2017 , 149 , 525–534. [ Google Scholar ] [ CrossRef ]
• Milla, J.; Hassan, M.M.; Rupnow, T. Evaluation of Self-Healing Concrete with Microencapsulated Calcium Nitrate. J. Mater. Civ. Eng. 2017 , 29 , 04017235. [ Google Scholar ] [ CrossRef ]
• Wang, X.; Sun, P.; Han, N.; Xing, F. Experimental Study on Mechanical Properties and Porosity of Organic Microcapsules Based Self-Healing Cementitious Composite. Materials 2017 , 10 , 20. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Arce, G.A.; Hassan, M.M.; Mohammad, L.N.; Rupnow, T. Characterization of Self-Healing Processes Induced by Calcium Nitrate Microcapsules in Cement Mortar. J. Mater. Civ. Eng. 2017 , 29 , 04016189. [ Google Scholar ] [ CrossRef ]
• Araújo, M.; Van Tittelboom, K.; Feiteira, J.; Gruyaert, E.; Chatrabhuti, S.; Raquez, J.M.; Šavija, B.; Alderete, N.; Schlangen, E.; De Belie, N. Design and Testing of Tubular Polymeric Capsules for Self-Healing of Concrete. IOP Conf. Ser. Mater. Sci. Eng. 2017 , 251 , 012003. [ Google Scholar ] [ CrossRef ]
• Kim, D.M.; Song, I.H.; Choi, J.Y.; Jin, S.W.; Nam, K.N.; Chung, C.M. Self-Healing Coatings Based on Linseed-Oil-Loaded Microcapsules for Protection of Cementitious Materials. Coatings 2018 , 8 , 404. [ Google Scholar ] [ CrossRef ]
• Hu, Z.X.; Hu, X.M.; Cheng, W.M.; Zhao, Y.Y.; Wu, M.Y. Performance Optimization of One-Component Polyurethane Healing Agent for Self-Healing Concrete. Constr. Build. Mater. 2018 , 179 , 151–159. [ Google Scholar ] [ CrossRef ]
• Al-Ansari, M.; Abu Taqa, A.G.; Senouci, A.; Hassan, M.M.; Shaat, A. Effect of Calcium Nitrate Healing Microcapsules on Concrete Strength and Air Permeability. Mag. Concr. Res. 2019 , 71 , 195–206. [ Google Scholar ] [ CrossRef ]
• Milla, J.; Hassan, M.M.; Rupnow, T.; Daly, W.H. Measuring the Crack-Repair Efficiency of Steel Fiber Reinforced Concrete Beams with Microencapsulated Calcium Nitrate. Constr. Build. Mater. 2019 , 201 , 526–538. [ Google Scholar ] [ CrossRef ]
• Arce, G.A.; Hassan, M.M.; Mohammad, L.N.; Rupnow, T. Self-Healing of SMA and Steel-Reinforced Mortar with Microcapsules. J. Mater. Civ. Eng. 2019 , 31 , 04018366. [ Google Scholar ] [ CrossRef ]
• Du, W.; Yu, J.; Gu, Y.; Li, Y.; Han, X.; Liu, Q. Preparation and Application of Microcapsules Containing Toluene-Di-Isocyanate for Self-Healing of Concrete. Constr. Build. Mater. 2019 , 202 , 762–769. [ Google Scholar ] [ CrossRef ]
• Lv, L.; Guo, P.; Liu, G.; Han, N.; Xing, F. Light Induced Self-Healing in Concrete Using Novel Cementitious Capsules Containing UV Curable Adhesive. Cem. Concr. Compos. 2020 , 105 , 103445. [ Google Scholar ] [ CrossRef ]
• Feng, J.; Dong, H.; Wang, R.; Su, Y. A Novel Capsule by Poly (Ethylene Glycol) Granulation for Self-Healing Concrete. Cem. Concr. Res. 2020 , 133 , 106053. [ Google Scholar ] [ CrossRef ]
• Li, Y.; Yu, J.; Cao, Z.; Du, W.; Zhang, Y.; Zou, Y. Preparation and Characterization of Nano-Fe 3 O 4 /Paraffin Encapsulated Isocyanate Microcapsule by Electromagnetic Controlled Rupture for Self-Healing Cementitious Materials. Constr. Build. Mater. 2020 , 265 , 120703. [ Google Scholar ] [ CrossRef ]
• Alghamri, R.; Al-Tabbaa, A. Self-Healing of Cracks in Mortars Using Novel PVA-Coated Pellets of Different Expansive Agents. Constr. Build. Mater. 2020 , 254 , 119254. [ Google Scholar ] [ CrossRef ]
• Wang, X.; Chen, S.; Yang, Z.; Ren, J.; Zhang, X.; Xing, F. Self-Healing Concrete Incorporating Mineral Additives and Encapsulated Lightweight Aggregates: Preparation and Application. Constr. Build. Mater. 2021 , 301 , 124119. [ Google Scholar ] [ CrossRef ]
• Du, W.; Liu, Q.; Lin, R.; Su, X. Preparation and Characterization of Microcrystalline Wax/Epoxy Resin Microcapsules for Self-Healing of Cementitious Materials. Materials 2021 , 14 , 1725. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Jiang, S.; Lin, Z.; Tang, C.; Hao, W. Preparation and Mechanical Properties of Microcapsule-Based Self-Healing Cementitious Composites. Materials 2021 , 14 , 4866. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• He, J.; Shi, X. Laboratory Assessment of Early-Age Durability Benefits of a Self-Healing Sytem to Cementitious Composites. J. Build. Eng. 2021 , 44 , 102602. [ Google Scholar ] [ CrossRef ]
• Beglarigale, A.; Eyice, D.; Seki, Y.; Yalçınkaya, Ç.; Çopuroğlu, O.; Yazıcı, H. Sodium Silicate/Polyurethane Microcapsules Synthesized for Enhancing Self-Healing Ability of Cementitious Materials: Optimization of Stirring Speeds and Evaluation of Self-Healing Efficiency. J. Build. Eng. 2021 , 39 , 102279. [ Google Scholar ] [ CrossRef ]
• Ren, J.; Wang, X.; Li, D.; Xu, S.; Dong, B.; Xing, F. Performance of Temperature Adaptive Microcapsules in Self-Healing Cementitious Materials under Different Mixing Temperatures. Constr. Build. Mater. 2021 , 299 , 124254. [ Google Scholar ] [ CrossRef ]
• Li, Y.; Yu, J.; Cao, Z.; He, P.; Liu, Q.; Han, X.; Wan, Y. Preparation and Application of Novel Microcapsules Ruptured by Microwave for Self-Healing Concrete. Constr. Build. Mater. 2021 , 304 , 124616. [ Google Scholar ] [ CrossRef ]
• Cao, B.; Al-Tabbaa, A. Reactive MgO-Based Self-Healing Slag-Cement-Bentonite Slurry Walls. Cem. Concr. Compos. 2022 , 131 , 104565. [ Google Scholar ] [ CrossRef ]
• Zhang, C.; Liu, R.; Chen, M.; Li, X.; Zhu, Z. Coupled Effect of Self-Healing Granules and Permeable Crystalline Additive on Early-Age Cracks Repair in Cement Material. Mater. Lett. 2022 , 323 , 132560. [ Google Scholar ] [ CrossRef ]
• Du, W.; Li, E.; Lin, R. Preparation and Characterization of Nano-CaCO 3 /Ceresine Wax Composite Shell Microcapsules Containing E-44 Epoxy Resin for Self-Healing of Cement-Based Materials. Nanomaterials 2022 , 12 , 197. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Ying, Y.; Hu, M.; Han, J.; Liu, W.; Qi, B.; Guo, J. Self-Healing in Cementitious System Using Interface Enhanced Capsules Prepared at Room Temperature. J. Clean. Prod. 2023 , 395 , 136465. [ Google Scholar ] [ CrossRef ]
• Taqa, A.A.; Suleiman, G.; Senouci, A.; Mohsen, M.O. Using Aerosol OT in Hexane Solution to Synthesize Calcium Nitrate Self-Healing Refined Microcapsules for Construction Applications. Buildings 2022 , 12 , 751. [ Google Scholar ] [ CrossRef ]
• Mao, Q.; Chen, J.; Qi, W.; Liu, H.; Wang, Z.; Cui, S. Improving Self-Healing and Shrinkage Reduction of Cementitious Materials Using Water-Absorbing Polymer Microcapsules. Materials 2022 , 15 , 847. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Choi, S.J.; Bae, S.H.; Ji, D.M.; Kim, S.H. Effects of Capsule Type on the Characteristics of Cement Mortars Containing Powder Compacted Capsules. Materials 2022 , 15 , 6773. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Mao, Q.; Chen, J.; Wu, W.; Li, R.; Shi, S.; Wang, Z.; Cui, S. Multiple Self-Healing Effects of Water-Absorbing Microcapsules in Cementitious Materials. Polymers 2023 , 15 , 428. [ Google Scholar ] [ CrossRef ]
• Arroyave, S.; Asensio, E.; Perilla, J.E.; Narváez-Rincón, P.C.; Cadavid, A.; Guerrero, A. Evaluation and Characterization of Autonomous Self-Healing Cementitious Materials with Low Carbon Footprint Using Hybrid Organic/Inorganic Microcapsules. Mater. Today Proc. 2023 . In Press . [ Google Scholar ] [ CrossRef ]
• Du, W.; Liu, B.; Feng, Z.; Liu, Q.; Wu, M.; Zuo, D. Influence of Electromagnetic Inductive Microcapsules on Self-Healing Ability of Limestone Calcined Clay Cement (LC3) Mortar. Polymers 2023 , 15 , 3081. [ Google Scholar ] [ CrossRef ]
• Zamir, M.; Dvorkin, D.; Peled, A. Fabric Cement-Based Composites with Nanoparticles Filler, Interfacial Characteristics. In Proceedings of the 3rd International RILEM Conference on Strain Hardening Cementitious Composites (SHCC3), Dordrecht, The Netherlands, 3–5 November 2014; pp. 171–178. [ Google Scholar ]
• Sherir, M.A.A.; Hossain, K.M.A.; Lachemi, M. The Influence of MgO-Type Expansive Agent Incorporated in Self-Healing System of Engineered Cementitious Composites. Constr. Build. Mater. 2017 , 149 , 164–185. [ Google Scholar ] [ CrossRef ]
• Wu, M.; Johannesson, B.; Geiker, M. A Review: Self-Healing in Cementitious Materials and Engineered Cementitious Composite as a Self-Healing Material. Constr. Build. Mater. 2012 , 28 , 571–583. [ Google Scholar ] [ CrossRef ]
• Kanellopoulos, A.; Giannaros, P.; Palmer, D.; Kerr, A.; Al-Tabbaa, A. Polymeric Microcapsules with Switchable Mechanical Properties for Self-Healing Concrete: Synthesis, Characterisation and Proof of Concept. Smart Mater. Struct. 2017 , 26 , 045025. [ Google Scholar ] [ CrossRef ]
• Iii, J.G.; Hassan, M.M.; Asce, M.; Rupnow, T.; Asce, M.; Barbato, M.; Asce, M.; Okeil, A.; Asce, M.; Asadi, S. Dicyclopentadiene and Sodium Silicate Microencapsulation for Self-Healing of Concrete. J. Mater. Civ. Eng. 2014 , 26 , 886–896. [ Google Scholar ] [ CrossRef ]
• Ach, D.; Briançon, S.; Broze, G.; Puel, F.; Rivoire, A.; Galvan, J.; Chevalier, Y. Formation of Microcapsules by Complex Coacervation. Can. J. Chem. Eng. 2015 , 93 , 183–191. [ Google Scholar ] [ CrossRef ]
• Jamekhorshid, A.; Sadrameli, S.M.; Farid, M. A Review of Microencapsulation Methods of Phase Change Materials (PCMs) as a Thermal Energy Storage (TES) Medium. Renew. Sustain. Energy Rev. 2014 , 31 , 531–542. [ Google Scholar ] [ CrossRef ]
• Toda, K.; Furuse, H. Extension of Einstein’s Viscosity Equation to That for Concentrated Dispersions of Solutes and Particles. J. Biosci. Bioeng. 2006 , 102 , 524–528. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Faroughi, S.A.; Huber, C. Crowding-Based Rheological Model for Suspensions of Rigid Bimodal-Sized Particles with Interfering Size Ratios. Phys. Rev. E 2014 , 90 , 052303. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Papaioannou, S.; Amenta, M.; Kilikoglou, V.; Gournis, D.; Karatasios, I. Critical Aspects in the Development and Integration of Encapsulated Healing Agents in Cement and Concrete. J. Adv. Concr. Technol. 2021 , 19 , 301–320. [ Google Scholar ] [ CrossRef ]
• Oh, S.-R.; Lee, K.-M.; Choi, S.; Choi, Y.-W. Fundamental Properties and Self-Healing Performance of Repair Mortar with Solid Capsules Made Using Inorganic Reactive Powder. Materials 2022 , 15 , 1710. [ Google Scholar ] [ CrossRef ]

Click here to enlarge figure

Share and Cite

Lima, G.T.d.S.; Silvestro, L.; Tambara Júnior, L.U.D.; Cheriaf, M.; Rocha, J.C. Autonomous Self-Healing Agents in Cementitious Materials: Parameters and Impacts on Mortar Properties. Buildings 2024 , 14 , 2000. https://doi.org/10.3390/buildings14072000

Lima GTdS, Silvestro L, Tambara Júnior LUD, Cheriaf M, Rocha JC. Autonomous Self-Healing Agents in Cementitious Materials: Parameters and Impacts on Mortar Properties. Buildings . 2024; 14(7):2000. https://doi.org/10.3390/buildings14072000

Lima, Geannina Terezinha dos Santos, Laura Silvestro, Luís Urbano Durlo Tambara Júnior, Malik Cheriaf, and Janaíde Cavalcante Rocha. 2024. "Autonomous Self-Healing Agents in Cementitious Materials: Parameters and Impacts on Mortar Properties" Buildings 14, no. 7: 2000. https://doi.org/10.3390/buildings14072000

Article Metrics

Supplementary material.

ZIP-Document (ZIP, 983 KiB)

Further Information

Mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

• Open access
• Published: 25 June 2024

GPT-4 performance on querying scientific publications: reproducibility, accuracy, and impact of an instruction sheet

• Kaiming Tao 1 ,
• Zachary A. Osman 1 ,
• Philip L. Tzou 1 ,
• Soo-Yon Rhee 1 ,
• Vineet Ahluwalia 2 &
• Robert W. Shafer 1  

BMC Medical Research Methodology volume  24 , Article number:  139 ( 2024 ) Cite this article

210 Accesses

1 Altmetric

Metrics details

Large language models (LLMs) that can efficiently screen and identify studies meeting specific criteria would streamline literature reviews. Additionally, those capable of extracting data from publications would enhance knowledge discovery by reducing the burden on human reviewers.

We created an automated pipeline utilizing OpenAI GPT-4 32 K API version “2023–05-15” to evaluate the accuracy of the LLM GPT-4 responses to queries about published papers on HIV drug resistance (HIVDR) with and without an instruction sheet. The instruction sheet contained specialized knowledge designed to assist a person trying to answer questions about an HIVDR paper. We designed 60 questions pertaining to HIVDR and created markdown versions of 60 published HIVDR papers in PubMed. We presented the 60 papers to GPT-4 in four configurations: (1) all 60 questions simultaneously; (2) all 60 questions simultaneously with the instruction sheet; (3) each of the 60 questions individually; and (4) each of the 60 questions individually with the instruction sheet.

GPT-4 achieved a mean accuracy of 86.9% – 24.0% higher than when the answers to papers were permuted. The overall recall and precision were 72.5% and 87.4%, respectively. The standard deviation of three replicates for the 60 questions ranged from 0 to 5.3% with a median of 1.2%. The instruction sheet did not significantly increase GPT-4’s accuracy, recall, or precision. GPT-4 was more likely to provide false positive answers when the 60 questions were submitted individually compared to when they were submitted together.

Conclusions

GPT-4 reproducibly answered 3600 questions about 60 papers on HIVDR with moderately high accuracy, recall, and precision. The instruction sheet's failure to improve these metrics suggests that more sophisticated approaches are necessary. Either enhanced prompt engineering or finetuning an open-source model could further improve an LLM's ability to answer questions about highly specialized HIVDR papers.

Peer Review reports

The systematic review of data from multiple research studies is often required to answer many of the most significant biomedical questions. However, the literature searches required for a systematic review often suffer from low sensitivity (recall) and specificity (precision) in part as a result of the limitations of current search tools which rely on the Medical Subject Headings (MeSH) key words, the National Library of Medicine’s controlled vocabulary used for indexing articles [ 1 ]. Extracting data from relevant studies also requires painstaking review by highly trained human reviewers.

The use of automated software tools to assist in reviewing research papers has become a topic of increasing interest. Most such tools have used natural language processing (NLP) and machine learning (ML) algorithms primarily to screen the titles and abstracts of publications to determine whether they meet the search criteria for a systematic review [ 2 , 3 , 4 , 5 , 6 , 7 , 8 ]. Several studies have also described the potential for using the representational language model Bidirectional Encoder Representations from Transformers (BERT) and the Generative Pre-trained Transformer (GPT) large language models (LLMs) for reviewing the full text of published studies [ 9 , 10 , 11 , 12 ]. LLMs have also been evaluated for their ability to summarize research studies [ 13 , 14 ].

We have extensive experience reviewing published studies on the topic of human immunodeficiency virus (HIV) drug resistance having maintained the Stanford HIV Drug Resistance Database (HIVDB; https://hivdb.stanford.edu ) and performed multiple systematic literature reviews [ 15 , 16 ]. In this study, we evaluated the ability of GPT-4 to correctly answer questions about publications on HIV drug resistance with and without an instruction sheet designed to provide GPT-4 with specialized HIV drug resistance knowledge. We evaluated publications considered for inclusion in a curated database. This database primarily links mutations in the genetic targets of HIV therapy to the antiviral treatments of the persons from whom the sequences were obtained and to the impact of these mutations on the i n vitro susceptibility to individual HIV drugs.

HIV drug resistance questions

We designed 60 questions pertaining to HIV drug resistance reflecting the type of information typically extracted from published papers evaluated for possible addition to HIVDB. Most of the questions dealt with linking HIV genetic sequence data to two other forms of data: (1) the antiviral treatments received by the patients from whom the sequenced viruses were obtained and (2) the effect of mutations in these viruses on their susceptibility to antiviral drugs. The questions were of three types: Boolean, requiring yes or no answers; numerical, where the correct response was an integer; and list-based, where a series of items constituted the correct answer. The complete list of questions can be found in Supplementary File 1.

Published papers

We selected 60 published papers on HIV drug resistance identified in recent PubMed searches and in recent GenBank database submissions including 19 published after September 2021, the cut-off date for the dataset used to train the GPT-4 model that was used. Nearly two-thirds of the papers reported HIV genotypic resistance data (e.g., genetic sequence data or lists of HIV drug-resistance mutations). Nearly one-half reported that their sequences had been submitted to GenBank, the standard public repository for sequence data, and provided GenBank accession numbers. The selected papers often reported the antiviral treatment histories of patients undergoing virus sequencing, the samples submitted for sequencing, the technology used for sequencing, and the results of in vitro susceptibility testing. Two authors reviewed each paper to determine the answers to the 60 questions. A third author designated the correct answer when there was a disagreement between the first two authors. The complete list of papers can be found in Supplementary File 2.

Instruction sheet

The instruction sheet contained 2002 words that provided background knowledge about HIV drug resistance and the type of information that a human curator would need to know to identify the relevant data for inclusion in HIVDB (Supplementary File 3). This document encapsulated fundamental antiviral therapy and HIV drug resistance concepts, alongside a description of frequently used terms and abbreviations within the field. The instruction sheet was not designed to be a comprehensive treatise on antiviral therapy and HIV drug resistance but rather to offer practical guidance to human curators with some background HIV knowledge. The instruction sheet contained information considered useful to answering many of the 60 questions. However, it was not designed specifically to answer each of the questions developed for this study.

Automated query pipeline

We designed an automated pipeline utilizing OpenAI GPT-4 32K API version “2023–05-15” (Microsoft Azure, accessed Sep 15th, 2023) (Fig. 1 ). A Python script was used to transform a published paper to a markdown format containing the text of the study methods, study results, tabular data, and figure legends. The abstract, introduction, discussion, and references were excluded from the markdown version of the paper. The median number of tokens in each markdown paper was 5338 (range:1282 to 13,861). On average, one token is about 0.75 words or about four English language characters. We chose to not submit the introduction, discussion, and references to GPT-4 because these parts of a paper often refer to the work of other studies.

Automated query pipeline work flow. The first step involved developing 60 questions relevant to HIV drug resistance, identifying 60 published papers, and developing an approximately 2000 word instruction sheet with HIV drug resistance information. Each paper was reviewed by two human reviewers and a markdown version of each paper’s full text was created. The second step involved querying GPT-4: building a prompt that included (1) the marked down version of each paper, (2) all 60 questions, and (3) the instruction sheet. The third step evaluated the GPT-4 answers to assess whether they were the same as the answers determined by the human curators. Three sample questions are shown including one for which the correct answer was Yes or No, another for which the correct answer was a list of items, and a third for which the correct answer was a number

Each GPT-4 query consisted of one markdown paper plus one of the following: (1) all 60 questions presented simultaneously; (2) all 60 questions presented simultaneously with the instruction sheet; (3) each of the 60 questions presented individually; and (4) each of the 60 questions presented individually with the instruction sheet.

We refer to the process of submitting the 60 questions simultaneously as the multiple-question mode and the process of submitting each question individually as the single-question mode. We refer to the process of presenting all questions without the instruction sheet as the base model. The single-question mode necessitated repeatedly submitting the same markdown paper with each question. It was therefore much more time consuming and expensive than the multiple-question mode.

If GPT-4 failed to answer all 60 questions for a paper or if a time-out error occurred when questions were presented in the multiple-question mode, the unanswered questions were resubmitted along with the paper.

Supplementary File 4 provides an example GPT-4 prompt. Supplementary File 5 provides the Python code used to generate the GPT-4 prompts.

Automated response evaluation pipeline

We evaluated the accuracy of GPT-4 responses using the following approach: (1) for Boolean questions, a script was used to determine if the response began with “yes” or “no”; (2) for numerical questions, a script was used to determine if the response contained a single number; (3) all other responses were evaluated manually. Accuracy was defined as concordance between the correct answer and the GPT-4 response for Boolean and numerical questions. For list questions, we considered the GPT-4 response to be accurate if it identified at least one element of the correct list. The response was considered inaccurate if it did not identify any element of the correct list or if it identified elements that were not part of the correct list. A manual review of half of the responses to the Boolean and numerical questions confirmed that the script used to determine whether the response began with “yes” or “no” or contained a single number accurately gauged GPT-4’s answers to these questions.

Experimental design and analyses

To evaluate the performance of GPT-4 in answering questions about a paper, we designed a series of experiments: (1) We assessed the reproducibility of the base model in the multiple-question mode by performing each query in triplicate. (2) We calculated the recall, precision, and F1 score – the harmonic mean of precision and recall, calculated as 2 x (recall * precision) / (recall + precision) – for the base model in multiple-question mode. This analysis was performed on the median of the triplicate results and it was performed separately for results obtained with and without the instruction sheet. (3) We compared the accuracy – measured as the proportion of correct answers – of the base model in the multiple-question mode to its performance when the responses were from randomly permuted papers. In essence, we assessed the accuracy of GPT-4's responses to the submitted paper compared with its accuracy when the answers were drawn from ten randomly selected papers, distinct from the actual paper. (4) We compared the accuracy of the base model in the multiple-question mode to the accuracy with the instruction sheet in the multiple-question mode. (5) Finally, we compared the accuracy of the base model in the multiple-question mode with the accuracy of the base model in the single-question mode, also in triplicate.

Figure 2 displays triplicate determinations of the accuracy of GPT-4 on each of the 60 questions applied to each of the 60 papers in the multiple-question mode without the instruction sheet (i.e., base model). The median accuracy for the 60 questions over the three replicates was 91.8% (range: 50.7%-100%). The mean accuracy for the 60 questions over the three replicates was 86.9%. The mean accuracies were similar for Boolean (86.6%), numerical (84.7%), and list (90.2%) questions. The standard deviation (SD) of three replicates for the 60 questions ranged from 0 to 5.3% with a median SD of 1.2% across all questions. The coefficient of variation (CV) of three replicates for the 60 questions ranged from 0 to 0.068 with a median CV of 0.012. The maximum difference between any two of the three replicates was 6 for one question, 4 for two questions, and 3 for three questions.

Triplicate determinations of the accuracy of each of the 60 questions applied to each of the 60 papers in the multiple question mode (i.e., all 60 questions presented simultaneously) without the instruction sheet (i.e., base model). The Y-axis indicates the percentage of times in which the GPT4 response was accurate across the 60 papers. The X-axis shows the question ID in descending order of median accuracy. The three bars shown for each question ID indicate separate replicates. The median accuracy for the 60 questions over the three replicates was 91.8% (range: 50.7%-100%). The mean accuracy across all questions and all papers were 86.8%, 86.9%, and 87.1%. Different colors mean different replicates

Figure 3 compares the results of one of the three replicates for the base model in multiple-question mode with the results obtained when the answers to the 60 papers were permuted. The mean accuracy for 10 permutations of the papers was 62.9%. Therefore, the increased accuracy of GPT-4 on the actual papers was 24.0% higher than expected by chance on the permuted set of papers (95% CI: 18.6%-29.4%; p  < 0.000001; paired Student’s t-test). The surprisingly high level of accuracy for permuted answers is explained by the uniformity of responses across many papers. Specifically, for Boolean questions, the answers were not infrequently always ‘yes’ or ‘no.’ Similarly, for numerical questions, the answer was often 0, and for list questions, the answer was often an empty list. Figure 3 demonstrates this in 10 questions where ≥ 90% of the Boolean answers were either ‘yes’ or ‘no’, in two numerical questions where the answer was usually 0, and in two list questions where the answer was usually an empty list.

Figure 4 shows the precision, recall, and F1 score with and without the instruction sheet separately for the Boolean, numerical, and list questions. Without the instruction sheet, GPT-4 demonstrated a recall of 68.1% and a precision of 84.6% on the 2280 Boolean questions (i.e., 48 questions × 60 papers); a recall of 61.6% and a precision of 88.1% on the 660 numerical questions (i.e., 11 questions × 60 papers); and a recall of 88.6% and a precision of 91.9% on the 660 list questions (i.e., 11 questions × 60 papers). Of the 296 true positive answers for list questions, 273 (92.2%) were identical to the manual answers whereas 23 (7.8%) contained a subset of the manual answers.

Recall and precision of GPT-4 at answering questions about HIV drug resistance papers: comparison with manual curators. The manual result (obtained by two human curators and a third to break ties) was considered to be the correct answer. Each entry in the six sections containing raw data represents the median of 3 repeats. *Includes questions for which GPT reported a number > 0 when the correct answer was 0 and questions for which GPT reported an incorrect number (i.e., one that differed from the manual review). **The results were considered to be false positives when GPT-4 identified items not identified by manual review. Additionally, 12 of 16 answers obtained without the instruction sheet and 14 of 18 answers obtained with the instruction sheet were considered to be false negatives because GPT-4 also failed to identify any of the items that were identified by manual review. Abbreviations: GPT (GPT-4), TP (true positive), TN (true negative), FP (false positive), FN (false negative), F1 score = 2 * (Recall * Precision) / (Recall + Precision)

Figure 5 displays the triplicate determinations of the accuracy of GPT-4 in the multiple-question mode with and without the instruction sheet. Across the 60 questions, the mean net change in accuracy was + 1.2% resulting in an overall accuracy of 88.1% across all questions and papers with the instruction sheet. On average, across the three replicates, the instruction sheet improved the accuracy of 3.2% ( n  = 114) of questions that were initially answered incorrectly. Conversely, 2.0% ( n  = 72) of questions initially answered correctly were incorrect with the instruction sheet. For all 60 questions combined, recall (76.2% vs. 72.5%; p  = 0.08; Fisher Exact Test) and precision 87.4% vs. 87.1%; NS) were not significantly higher with the instruction sheet compared to without the instruction sheet (Fig. 4 ).

Triplicate determinations of the accuracy of GPT-4 in multiple-question mode with and without the instruction sheet. For each question, two histograms are shown. The left histogram shows the median of triplicate accuracy determinations without the instruction sheet. The right histogram shows the median of triplicate accuracy determinations with the instruction sheet. Increased accuracy associated with the instruction sheet is shown by coloring part of the right-sided histogram in blue while reductions in accuracy are shown by coloring part of the left-sided histogram in red. The sizes of the colored regions indicate the sizes of the increases or decreases in accuracy associated with the instruction sheet. The questions are shown in descending order of the increased accuracy associated with instruction sheet (i.e., the size of the blue histograms)

The instruction sheet significantly impacted three questions: one showed a net accuracy increase of 26.1%, resulting in 16 additional correct responses; another recorded an 8.3% improvement (5 papers); and the third saw a 6.7% increase (4 papers). The remaining questions displayed net changes in accuracy that were no greater or lower than three. The question “Does the paper report GenBank accession numbers for sequenced HIV isolates other than those for laboratory HIV isolates?" was the one associated with a net increased accuracy of 26.1%. The question “How many samples in the paper were reported to have undergone plasma virus sequencing?” was the one associated with a net increased accuracy of 8.3%.

In an attempt to edit the instruction sheet to increase GPT-4 accuracy for questions that were often answered incorrectly, we modified the query pipeline as follows. Rather than submitting each paper in multiple-question mode, we submitted each paper with just the one question that we were targeting for improvement (i.e., in single-question mode). After running several questions in both modes, we noticed marked differences in GPT-4 accuracy between the multiple-question and single-question modes. Figure 6 compares the accuracy of the multiple-question and single-question mode for all 60 questions without the instruction sheet. Each histogram represents the median of three replicates. Overall, the median and mean accuracy for the single-question mode were significantly lower than the multiple-question mode across all 60 questions: median (83.0% vs 91.8%, p  = 0.0006; Wilcoxon signed-rank test) and mean (77.6% vs 86.9%, p  = 0.0005; Wilcoxon signed-rank test).

Figure 6 groups the questions according to whether the accuracy was ≥ 10% lower in the single-question mode ( n  = 21 questions), ≥ 10% higher in the single-question mode ( n  = 3 questions), or less than 10% different between the multiple-question and single-question modes. The largest differences in accuracy between the two modes was for questions for which the answer was usually no for Boolean questions, 0 for numerical questions, and an empty list for list questions. We refer to these usually negative questions as No_0_Empty. Indeed, for the 21 questions that belonged to this category, the median accuracy was 91.0% in the multiple-question mode but only 60.8% in the single-question mode. There was also a strong correlation between the frequency of answers that were No_0_Empty and the reduced accuracy when questions were presented in the single-question mode ( r  = 0.45; p  = 0.0003).

The cost of using the GPT-4 API to obtain responses for 60 papers × 60 questions in the multiple-question mode without and with the instruction sheet was $240 and $300, respectively. The cost of obtaining responses for 60 papers × 60 questions in the single-question mode without and with the instruction sheet was $1500 and $2000, respectively. The overall cost of this study, considering that most experiments were performed in triplicate was $8120: $240 × 3 plus $300 × 3 plus $1500 × 3 plus $2000 × 1. After completing this study, OpenAI released a new model, GPT-4-turbo (gpt-4–1106-preview), on November 6, 2023. This model significantly reduced costs, decreasing from $0.06 to $0.01 per prompt and from $0.12 to $0.03 per completion. Consequently, the overall cost of this study would have been approximately five times lower.

We submitted the text of the methods, results, tables, and figure legends of 60 published papers on HIV drug resistance together with 60 questions related to HIV drug resistance with and without an instruction sheet to the GPT-4 API. We found that the accuracy of GPT-4 responses was approximately 87%, which was 24% greater than that obtained when the answers to the papers were permuted. With the exception of one question, the accuracy of GPT-4 was not improved with an approximately 2000 word instruction sheet. Notably, GPT-4 was also less likely to answer certain types of questions accurately when they were submitted individually (single-question mode) compared to when they were submitted together (multiple-question mode).

This study differs from most previous studies of automated software tools designed to assist with systematic reviews. First, we prompted the LLM GPT-4 to answer specific questions about entire papers whereas previous studies were often optimized for screening paper title and abstracts [ 5 , 6 , 7 , 8 ]. Second, we used GPT without providing training examples, whereas previous studies, were often interactive in that they combined NLP and ML algorithms with user feedback [ 5 , 6 , 9 , 10 , 11 ]. Finally, the results presented in this study were quantitative and transparent, whereas several previous studies, particularly those using LLMs, presented their results in a qualitative manner.

GPT-4 performs well at summarizing research papers because LLMs are adept at distilling and condensing information into simpler shorter formats [ 13 , 14 ]. However, answering specific questions can be challenging for LLMs because they only process a limited amount of text at once. This limitation hampers their ability to cross-reference details within longer documents effectively. Indeed, the questions that GPT-4 was likely to answer correctly were those for which the answer could be found in a single paragraph or sentence in a paper. In contrast, those questions that required reasoning about information found in different parts of a paper were less likely to be answered correctly. For example, the question “Does the paper provide complete ART history for all of the individuals in the study?” was answered correctly only about 50% of the time.

The instruction sheet contained information that would have been expected to be helpful for several questions such as “Does the paper report the results of HIV pol sequences?” and “Were the individuals in the study INSTI-naïve?”. Despite this, when GPT-4 was equipped with the information that 'pol' refers to the gene encoding the viral enzymes protease, reverse transcriptase, and integrase, 'INSTI' denotes integrase strand transfer inhibitors, and 'naïve' implies untreated, it still only correctly answered these questions 57% and 62% of the time, respectively.

After completing the experiments outlined in this study, we performed ten queries in an attempt to determine the extent of GPT-4’s HIV drug resistance knowledge. Supplementary File 6 lists each of the ten queries and the entire GPT-4 response (version last updated April 2023). The responses to these additional queries, demonstrated that GPT-4 possesses extensive information about HIV drug resistance. Although the experiments outlined in the results were performed using an earlier version of GPT-4 (last updated September 2021), all of the information included in the GPT-4 response had been publicly available prior to this earlier date. Given that GPT-4 already contained most of the information provided in the instruction sheet, enhancing its performance would likely hinge on providing prompts that demonstrate how to apply its knowledge to a published paper.

In an attempt to edit the instruction sheet to increase GPT-4 accuracy we modified the query pipeline by submitting each paper with just the one question that we were targeting for improvement. This modification led to the study’s second major new finding: when questions were presented individually, GPT-4 tended to provide incorrect affirmative answers to questions that generally warranted a negative response such as ‘no’, ‘0’, or an empty list. For instance, when the query "Which drugs were tested on phenotypic susceptibility in the paper?" was posed separately, there were 40 instances where GPT-4 erroneously referenced drugs that were administered to patients instead of those used in a susceptibility assay. This mistake was infrequent when all 60 questions were asked at once, indicating that presenting the full batch of questions improves GPT-4's understanding of each question's context. The enhanced accuracy observed when presenting multiple questions simultaneously may resemble automatic chain-of-thought prompting [ 17 ]. This technique, used in AI interactions, involves supplying step-by-step questions that guide the system through a logical thought sequence, thereby improving its comprehension of complex inquiries.

While enhancing the questions and instruction sheet was of interest, undertaking such revisions methodically would have required an open-ended approach beyond the scope of this study. Nonetheless, we observed that rephrasing two of the questions led to a significantly increased GPT-4 accuracy. For example, the question “Were sequences obtained from individuals with active HIV replication?” was true for 26 of the 60 papers. The median accuracy of GPT-4 on three replicates, with and without the instruction sheet, was 61%. However, the median accuracy of GPT-4 was 97% when we rephrased the question as follows: “Were sequences in the paper obtained from individuals with virological failure while receiving antiretroviral therapy?”. In contrast, the few changes we made to the instruction sheet did not yield substantial increases in GPT-4 accuracy for any of the questions.

GPT-4 possesses extensive knowledge about HIV drug resistance and it reproducibly answers Boolean, numerical, and list questions about HIV drug resistance papers. Its accuracy, recall, and precision of approximately 87%, 73%, and 87% without human feedback demonstrate its potential at performing this task. GPT-4 faced several challenges beginning with the specialized nature of the questions that were on topics that likely represented a small part of its training corpus [ 18 ]. In addition, addressing queries that necessitate making inferences, particularly when dealing with unsaid elements within the text, can be difficult. A more robust familiarity with the subject of HIV drug resistance would potentially have empowered GPT-4 to make better inferences. Finally, the instruction sheet was designed for human comprehension without the multiple examples usually necessary for optimizing a language model’s performance. The inability of GPT-4 to utilize the instruction sheet suggests that more sophisticated prompt engineering approaches or the finetuning of an open source model are likely required to improve accuracy when answering questions on highly specialized research papers.

Availability of data and materials

All data generated or analyzed during this study are included in this published article and its supplementary information files.

2R24AI13661806. The funder played no role in this review.

Jin Q, Kim W, Chen Q, Comeau DC, Yeganova L, Wilbur WJ, et al. MedCPT: contrastive pre-trained transformers with large-scale PubMed search logs for zero-shot biomedical information retrieval. Bioinformatics. 2023;39(11):btad651.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Cierco Jimenez R, Lee T, Rosillo N, Cordova R, Cree IA, Gonzalez A, et al. Machine learning computational tools to assist the performance of systematic reviews: A mapping review. BMC Med Res Methodol. 2022D 16;22(1):322.

Article   PubMed   PubMed Central   Google Scholar  

Blaizot A, Veettil SK, Saidoung P, Moreno-Garcia CF, Wiratunga N, Aceves-Martins M, et al. Using artificial intelligence methods for systematic review in health sciences: A systematic review. Res Synth Methods. 2022;13(3):353–62.

Article   PubMed   Google Scholar  

dos Santos ÁO, da Silva ES, Couto LM, Reis GVL, Belo VS. The use of artificial intelligence for automating or semi-automating biomedical literature analyses: A scoping review. J Biomed Inform. 2023J;1(142).

Article   Google Scholar  

van Dijk SHB, Brusse-Keizer MGJ, Bucsán CC, van der Palen J, Doggen CJM, Lenferink A. Artificial intelligence in systematic reviews: promising when appropriately used. BMJ Open. 2023J 1;13(7).

van de Schoot R, de Bruin J, Schram R, Zahedi P, de Boer J, Weijdema F, et al. An open source machine learning framework for efficient and transparent systematic reviews. Nat Mach Intell. 2021F;3(2):125–33.

Schopow N, Osterhoff G, Baur D. Applications of the Natural Language Processing Tool ChatGPT in Clinical Practice: Comparative Study and Augmented Systematic Review. JMIR Med Inform. 2023N 28;11(1).

Guo E, Gupta M, Deng J, Park YJ, Paget M, Naugler C. Automated paper screening for clinical reviews using large language models. J Med Internet Res. 2023 [cited 2023 Nov 26]; Available from: http://arxiv.org/abs/2305.00844 .

Weissenbacher D, O’Connor K, Klein A, Golder S, Flores I, Elyaderani A, et al. Text mining biomedical literature to identify extremely unbalanced data for digital epidemiology and systematic reviews: dataset and methods for a SARS-CoV-2 genomic epidemiology study. medRxiv. 2023 . 2023.07.29.23293370. [cited 2024 Jan 3]. Available from: https://www.medrxiv.org/content/https://doi.org/10.1101/2023.07.29.23293370v1 .

Syriani E, David I, Kumar G. Assessing the ability of ChatGPT to screen articles for systematic reviews. arXiv; 2023. [cited 2023 Nov 14]. Available from: https://arxiv.org/abs/2307.06464 .

Google Scholar  

Alshami A, Elsayed M, Ali E, Eltoukhy AEE, Zayed T. Harnessing the power of ChatGPT for automating systematic review process: methodology, case study, limitations, and future directions. Systems. 2023;11(7):351.

Khraisha Q, Put S, Kappenberg J, Warraitch A, Hadfield K. Can large language models replace humans in the systematic review process? Evaluating GPT-4’s efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages. 2023. [cited 2023 Nov 13]. Available from: http://arxiv.org/abs/2310.17526 .

Liang W, Zhang Y, Cao H, Wang B, Ding D, Yang X, et al. Can large language models provide useful feedback on research papers? A large-scale empirical analysis. arXiv; 2023. [cited 2023 Nov 14]. Available from: http://arxiv.org/abs/2310.01783 .

Liu R, Shah NB. ReviewerGPT? An exploratory study on using large language models for paper reviewing. arXiv; 2023. [cited 2024 Jan 5]. Available from: http://arxiv.org/abs/2306.00622 .

Rhee S-Y, Kassaye SG, Jordan MR, Kouamou V, Katzenstein D, Shafer RW. Public availability of HIV-1 drug resistance sequence and treatment data: a systematic review. Lancet Microbe. 2022;3:e392–8.

Tao K, Rhee SY, Chu C, Avalos A, Ahluwalia AK, Gupta RK, et al. Treatment Emergent Dolutegravir Resistance Mutations in Individuals Naïve to HIV-1 Integrase Inhibitors: A Rapid Scoping Review. Viruses. 2023S;15(9):1932.

Zhang Z, Zhang A, Li M, Smola A. Automatic chain of thought prompting in large language models. arXiv; 2022. [cited 2024 Jan 4]. Available from: http://arxiv.org/abs/2210.03493 .

Kandpal N, Deng H, Roberts A, Wallace E, Raffel C. Large language models struggle to learn long-tail knowledge. arXiv; 2023. [cited 2024 Jan 2]. Available from: http://arxiv.org/abs/2211.08411 .

Download references

Acknowledgements

Not applicable.

This work was funded by a grant from the National Institutes of Health:

Author information

Authors and affiliations.

Division of Infectious Diseases, Department of Medicine, Stanford University, Stanford, CA, USA

Kaiming Tao, Zachary A. Osman, Philip L. Tzou, Soo-Yon Rhee & Robert W. Shafer

Aphorism Labs, Palo Alto, CA, USA

Vineet Ahluwalia

You can also search for this author in PubMed   Google Scholar

Contributions

R.W.S contributed to the design of the work, interpretation of the data, and drafting the manuscript; K.T contributed to the design of the work, acquisition, analysis, and interpretation of the data, and creation of the software used in the work; Z.A.O and P.L.T contributed to the acquisition of the data; S.Y.R contributed to the design of the work and acquisition of the data; V.A contributed to conception of the work and revision of the manuscript.

Corresponding author

Correspondence to Robert W. Shafer .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

Robert W. Shafer has received honoraria for participation in advisory boards from Gilead Sciences and GlaxoSmithKline and speaking honoraria from Gilead Sciences and ViiV Healthcare. Other authors declare that they have no competing interest.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., supplementary material 3., supplementary material 4., supplementary material 5., supplementary material 6., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Tao, K., Osman, Z.A., Tzou, P.L. et al. GPT-4 performance on querying scientific publications: reproducibility, accuracy, and impact of an instruction sheet. BMC Med Res Methodol 24 , 139 (2024). https://doi.org/10.1186/s12874-024-02253-y

Download citation

Received : 09 January 2024

Accepted : 21 May 2024

Published : 25 June 2024

DOI : https://doi.org/10.1186/s12874-024-02253-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Large language model
• HIV drug resistance
• Systematic review
• Data extraction

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]