qualitative research delphi method

The Delphi Method

• Living reference work entry
• Later version available View entry history
• First Online: 26 February 2023
• Cite this living reference work entry

• Ghouwa Ismail 2 &
• Naiema Taliep 3  

139 Accesses

Researchers in Social Sciences and Global Public Health have found the Delphi method to be a successful tool for identifying and prioritizing issues in decision-making. The Delphi method is a systematic technique for gathering information about a topic by asking a group of subject matter experts a series of questions. The Delphi method involves experts who are knowledgeable about a certain subject so that they can forecast outcomes for future research, predict the likelihood of an event, or reach a consensus on certain topics. While the Delphi method is not an ideal tool for all research within the social sciences and global public health, it can be adapted to address a range of issues in a variety of disciplines and research areas. The ensuing chapter discusses the characteristics of the Delphi method, including criticisms, steps for conducting a Delphi study, its uses as well as highlighting certain advantages and disadvantages.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Institutional subscriptions

Similar content being viewed by others

The Delphi Technique

Adler M, Ziglio E (1996) Gazing into the oracle: the Delphi method and its application to social policy and public health. Jessica Kingsley Publishers, London

Google Scholar  

Aigbavboa C, Thwala W (2012) An exploration of the use of Delphi methodology in housing satisfaction studies. In: Laryea S, Agyepong SA, Leiringer R, Hughes W (eds) Proceedings of 4th West Africa built environment research (WABER) conference, 24–26 July 2012. Abuja, Nigeria, pp 147–161

Avella JR (2016) Delphi panels: Research design, procedures, advantages, and challenges. Int J Dr Stud, 11:305–321. Retrieved from http://www.informingscience.org/Publications/3561

Baker J, Lovell K, Harris N (2006) How expert are the experts? An exploration of the concept of ‘expert’ within Delphi panel techniques. Nurse Res 14(1):59–70

Article   PubMed   Google Scholar  

Beretta R (1996) A critical review of the Delphi technique. Nurse Res 3(4):79–89

Article   CAS   PubMed   Google Scholar  

Bleijenbergh I, Korzilius H, Verschuren P (2011) Methodological criteria for the internal validity and utility of practice oriented research. Qual Quant 45(1):145–156

Article   Google Scholar  

Brand JC, Rossi MJ, Lubowitz JH (2017) Reliable references, accountable authors, and steadfast reviewers mitigate against retracted publications and scientific misconduct. Arthroscopy 33:1435–1436

Crisp J, Pelletier D, Dufield C, Adams A, Nagy S (1997) The Delphi method? Nurs Res 46:116–118

Dalkey NC (1972) The Delphi method: an experimental study of group opinion. In: Dalkey NC, Rourke DL, Lewis R, Snyder D (eds) Studies in the quality of life: Delphi and decision-making. Lexington Books, Lexington, MA, pp 13–54

Dalkey N, Heimer O (1963) An experimental application of the Delphi method to the use of experts. Manag Sci 9(3):458–467

Davidson PH (2013) The Delphi technique in doctoral research: considerations and rationale. Rev High Educ Self Learn 6(2):53–65

Delbecq AL, Van de Hen AH, Gustafson DH (1975) Group techniques for program planning: a guide to nominal group and Delphi processes. Scott, Foresman and Company

Devaney L, Henchion M (2018) Who is a Delphi ‘expert’? Reflections in a bioeconomy expert selection procedure from Ireland. Futures 99:45–55. https://doi.org/10.1016/j.futures.2018.03.017

Donohoe HM, Needham RD (2009) Moving best practice forward: Delphi characteristics, advantages, potential problems, and solutions. Int J Tour Res 11(5):415–437

Giannarou L, Zervas E (2014) Using Delphi technique to build consensus in practice. Int J Bus Sci Appl Manag 9:65–82

Gorman GE (2011) Method (or methodology) in their madness? How researchers confuse “method” and “methodology”. Online Inf Rev 35(1)

Gupta UG, Clarke RE (1996) Theory and applications of the Delphi technique: a bibliography (1975–1994). Technol Forecast Soc Chang 53(2):185–211

Guzys D, Dickson-Swift V, Kenny A, Threlkeld (2015) Gadamerian philosophical hermeneutics as a useful methodological framework for the Delphi technique. Int J Qual Stud Health Well Being 10:1–15

Hallowell MR, Gambatese JA (2010) Qualitative research: application of the Delphi method to CEM research. J Constr Eng Manag 136:99–107

Hanafin S (2004) Review of literature on the Delphi technique . https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.521.4998&rep=rep1&type=pdf

Hardy JD, O’Brien AP, Gaskin CJ et al (2004) Practical application of the Delphi technique in a bicultural mental health nursing study in New Zealand. J Adv Nurs 46(1):95–109

Hasson F, Keeney S (2011) Enhancing rigour in the Delphi technique research. Technol Forecast Soc Chang 78(9):1695–1704

Hasson F, Mckenna H, Keeney S (2000) Research guidelines for the Delphi survey technique. J Adv Nurs 32(4):1008–1015

CAS   PubMed   Google Scholar  

Helmer O, Rescher N (1959) On the epistemology of the inexact science. Manag Sci 6:25–53

Hsu C, Sandford BA (2007) The Delphi technique: making sense of consensus. Pract Assess Res Eval 12(10). https://doi.org/10.7275/pdz9-th90

Iqbal S, Pipon-Young L (2009) The Delphi method. Methods 22(7):598–601

Ismail G (2018) A mixed methods approach to the development and validation of an assessment tool to measure psycho-social factors associated with willingness to participate in child- centred initiatives . (Unpublished doctoral dissertation). University of South Africa, Pretoria, South Africa

Ismail G, Taliep N (2020) Chapter 7: Community engagement in instrument design: the Delphi consensus technique. In: Kramer S, Laher A, Fynn HH, van Vuuren J (eds) Online readings in research methods (ORIM). Psychological Society of South Africa, pp 1–37

Ismail G, Van Niekerk A (2020) Enablers and inhibitors associated with community willingness to participate in child-centred safety initiatives. Community Psychol Glob Perspect, 1(2/1), 1–19

Ismail G, Van Niekerk A (2021) Validation of an assessment tool to measure psycho-social factors associated with willingness to participate in child-centered initiatives. SAGE Open 11:1–16

Keeney S, Hasson F, Mckenna H (2006) Consulting the oracle: ten lessons from using the Delphi technique in nursing research. J Adv Nurs 53(2):205–212. https://doi.org/10.1111/j.1365-2648.2006.03716.x

Kobus J, Westner M (2016) Ranking-type delphi studies in IS research: step-by-step guide and analytical extension. In: Nunes M, Isaías P, Powell P (eds) 9th IADIS international conference information systems, 2016. Portugal. Conference Proceedings, pp 28–38

Lang T (1994) An overview of four futures methodologies. Retrieved from http://www.futures.hawaii.edu/publications/half-fried-ideas/J7/LANG.pdf

Linstone H, Turoff M (1975) The Delphi method: techniques and applications. Addison-Wesley, Reading, MA

Linstone HA, Turoff M (2002) Introduction. In: Linstone HA, Turoff M (eds) The Delphi method: techniques and applications. http://is.njit.edu/pubs/delphibook/

McGregor SLT, Murnane JA (2010) Paradigm, methodology and method: intellectual integrity in consumer scholarship. Int J Consum Stud 34(4):419427

McKenna HP (1994) The Delphi technique: a worthwhile research approach for nursing? J Adv Nurs 19:1221–1225

Mullen PM (2003) Delphi: myths and reality. J Health Organ Manag 17(1):37–52. https://doi.org/10.1108/14777260310469319

Munnik E, Smith M (2019) Methodological rigour and coherence in the construction of instruments: the emotional social screening tool for school readiness. Afr J Psychol Assess 1. https://doi.org/10.4102/ajopa.v1i0.2

Ogbeifun E, Agwa-Ejon J, Mbohwa C, Pretorius JH (2016) The Delphi technique: a credible research methodology . Proceedings of the 2016 International Conference on Industrial Engineering and Operations Management Kuala Lumpur, Malaysia, March 8–10, 2016

Ökmen B, Şahin Ş, Kiliç A, Adigüzel A (2019) A needs analysis on the competences that students should have for curriculum and instruction graduate programs: a Delphi study. J High Educ (Turkey) 9(2):149–158

Petry K, Maes B, Vlaskamp C (2007) Operationalizing quality of life for people with profound multiple disabilities: a Delphi study. J Intellect Disabil Res 51(1):334–349

Polit DF, Beck CT (2008) Nursing research: generating and assessment evidence for nursing practice. Wolters Kluwer Health/Lippincott, Williams & Wilkins, Philadelphia, PA

Rauch W (1979) The decision Delphi. Technol Forecast Soc Chang 15(3):159–169

Rowe G, Wright G (1999) The Delphi technique as a forecasting tool: issues and analysis. Int J Forecast 15:353–375

Salihu HM, Salinas-Miranda AA, Wang W, Turner D, Lo Berry E, Zoorob R (2015) Community priority index: utility, applicability and validation for priority setting in community-based participatory research. J Public Health Res 4(2):443. https://doi.org/10.4081/jphr.2015.443

Article   PubMed   PubMed Central   Google Scholar  

Schmidt RC (1997) Managing Delphi surveys using nonparametric statistical techniques. Decis Sci 28(3):763–774

Shariff NJ (2015) Utilizing the Delphi approach: a review. Nurs Care 4(3). https://doi.org/10.4172/2167-1168.1000246

Skinner R, Nelson R, Chin W, Land L (2015) The Delphi method research strategy in studies of information systems. Commun Assoc Inf Syst 37(2). https://doi.org/10.17705/1CAIS.03702

Skulmoski G, Hartman F, Krahn J (2007) The Delphi method for graduate research. J Inform Technol Educ Res 6(1):1–21

Stewart J (2001) Is the Delphi technique a qualitative method? Med Educ 35:922–923

Strasser A (2017) Delphi method variants in information systems research: taxonomy development and application. Electron J Bus Res Methods 15(2):120–132

Szpilko D (2014) The use of Delphi method in the process of building a tourism development strategy in the region. Econ Manag 4:329–346. https://doi.org/10.12846/j.em.2014.04.24

Taliep N (2016) Process evaluation of the development of a community-based participatory intervention promoting positive masculinity and peace and safety: Addressing interpersonal violence in a Western Cape community [Unpublished doctoral dissertation]. University of South Africa

Taliep N, Ismail G, Seedat M, Suffla S (2014) Development of a family functioning scale for the south African context: the substantive validity phase. Child Abuse Res: South Afr J 15(1):73–82

Turoff M (1970) The design of a policy Delphi. Technol Forecast Soc Chang 2:149–171

Van Niekerk A, Ismail G (2013) Barriers to caregiver involvement in a child safety intervention in South Africa. S Afr J Psychol, 43(4):470–481

Veziari Y, Kumar S, Leach M (2018) The development of a survey instrument to measure the barriers to the conduct and application of research in complementary and alternative medicine: a Delphi study. BMC Complement Altern Med 18:335. https://doi.org/10.1186/s12906-018-2352-0

Vogel C, Zwolinsky S, Griffiths C, Hobbs M, Henderson E, Wilkins E (2019) A Delphi study to build consensus on the definition and use of big data in obesity research. Int J Obes 43:2573–2586. https://doi.org/10.1038/s41366-018-0313-9

Download references

Author information

Authors and affiliations.

Institute for Social and Health Sciences, University of South Africa, Johannesburg, South Africa

Ghouwa Ismail

South African Medical Research Council-University of South Africa Masculinity and Health Research Unit, Cape Town, South Africa

Naiema Taliep

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Ghouwa Ismail .

Editor information

Editors and affiliations.

School of Health Sciences, Western Sydney University, Camberwell, VIC, Australia

Pranee Liamputtong

Rights and permissions

Reprints and permissions

Copyright information

© 2023 Springer Nature Switzerland AG

About this entry

Cite this entry.

Ismail, G., Taliep, N. (2023). The Delphi Method. In: Liamputtong, P. (eds) Handbook of Social Sciences and Global Public Health. Springer, Cham. https://doi.org/10.1007/978-3-030-96778-9_66-1

Download citation

DOI : https://doi.org/10.1007/978-3-030-96778-9_66-1

Received : 24 July 2022

Accepted : 20 December 2022

Published : 26 February 2023

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-96778-9

Online ISBN : 978-3-030-96778-9

eBook Packages : Springer Reference Social Sciences Reference Module Humanities and Social Sciences Reference Module Business, Economics and Social Sciences

• Publish with us

Policies and ethics

Chapter history

DOI: https://doi.org/10.1007/978-3-030-96778-9_66-2

DOI: https://doi.org/10.1007/978-3-030-96778-9_66-1

• Find a journal
• Track your research

Delphi Method

A qualitative method of forecasting where experts answer several rounds of questionnaires

What is the Delphi Method?

The Delphi method, also known as the estimate-talk-estimate technique (ETE), is a systematic and qualitative method of forecasting by collecting opinions from a group of experts through several rounds of questions. The Delphi method relies on experts who are knowledgeable about a certain topic so they can forecast the outcome of future scenarios, predict the likelihood of an event, or reach consensus about a particular topic.

Understanding the Delphi Method

The Delphi method consists of several rounds of written questionnaires that allow experts to give their opinions. After the experts answer each round of questionnaires, the facilitator collects all the answers and hands out a summary report of the answers to each expert. Then, the experts review the summary report and either agree or disagree with the other experts’ answers.

The experts then fill out another questionnaire that gives them the opportunity to provide updated opinions based on what they understand from the summary report. The Delphi method becomes complete when a consensus of forecasts is achieved.

Delphi Method Process

1. identify the issue and objective.

It is important to define the issue you are trying to solve and what you want to achieve by using the Delphi method. Make sure that you know what you are trying to forecast.

2. Choose a group of experts and a facilitator

The experts can be individuals who are internal or external to the organization . The facilitator should take a neutral position and be someone with experience with research and data collection.

3. Round one questionnaire

The facilitator provides the first questionnaire for the experts. The type of questions in round one are usually open-ended, as it allows experts to brainstorm their ideas. The facilitator collects all the answers from the questionnaire and hands out a summary report of answers to the experts. In the summary report, the experts’ identities remain anonymous to encourage them to state their opinions freely.

4. Round two questionnaire

The second questionnaire should be created by analyzing the answers that you observe from the first round. Identify any similarities between answers and eliminate irrelevant content, so the second questionnaire can go in the direction where there is consensus within the experts.

When the experts answer the second questionnaire, their opinions may remain the same, or they may change their opinions after reading the summary report from the first round and understanding the other experts’ opinions. After the completion of the second questionnaire, the facilitator hands out a second summary report of answers to the experts.

5. Round three questionnaire

Proceed to the third round with the same idea as the second round. The third questionnaire should be created by analyzing answers from the second questionnaire. The experts will answer the third questionnaire based on their opinions from the summary report of answers in the second round.

You may continue to go through additional rounds of questionnaires or choose to stop here if you feel like you’ve reached enough consensus between the experts, and all their forecasts agree with each other.

Practical Applications

The Delphi method was initially used to forecast trends and outcomes in the fields of science and technology. For example, it’s been used to predict trends in aerospace, automation, broadband connections, and the use of technology in schools.

Additionally, it’s been used to forecast outcomes related to the economy , education, healthcare, and public policy . It is also useful in business situations, helping forecast sales with 96%-97% accuracy compared to actual sales numbers. As a result, the Delphi method can be used as a technique to predict the success of business events.

Additional Sources

CFI offers the Commercial Banking & Credit Analyst (CBCA)™ certification program for those looking to take their careers to the next level. To keep learning and advancing your career, the following resources will be helpful:

• Cluster Sampling
• Ensemble Methods
• Sample Selection Bias
• Hypothesis Testing
• See all economics resources

Financial Analyst Certification

Become a certified Financial Modeling and Valuation Analyst (FMVA)® by completing CFI’s online financial modeling classes!

• Share this article

Create a free account to unlock this Template

Access and download collection of free Templates to help power your productivity and performance.

Already have an account? Log in

Supercharge your skills with Premium Templates

Take your learning and productivity to the next level with our Premium Templates.

Upgrading to a paid membership gives you access to our extensive collection of plug-and-play Templates designed to power your performance—as well as CFI's full course catalog and accredited Certification Programs.

Already have a Self-Study or Full-Immersion membership? Log in

Access Exclusive Templates

Gain unlimited access to more than 250 productivity Templates, CFI's full course catalog and accredited Certification Programs, hundreds of resources, expert reviews and support, the chance to work with real-world finance and research tools, and more.

Already have a Full-Immersion membership? Log in

• Search Search Please fill out this field.

What Is the Delphi Method?

Understanding the delphi method.

• The Process
• Disadvantages
• Applications
• Alternatives

The Bottom Line

• Fundamental Analysis

What Is the Delphi Method, and How Is It Useful in Forecasting?

Pete Rathburn is a copy editor and fact-checker with expertise in economics and personal finance and over twenty years of experience in the classroom.

Xiaojie Liu / Investopedia

The Delphi method is a forecasting process and structured communication framework based on the results of multiple rounds of questionnaires sent to a panel of experts. After each round of questionnaires, the experts are presented with an aggregated summary of the last round, allowing each expert to adjust their answers according to the group response. This process combines the benefits of expert analysis with elements of the wisdom of crowds.

Key Takeaways

• The Delphi method is a process used to arrive at a group opinion or decision by surveying a panel of experts.
• Experts respond to several rounds of questionnaires, and the responses are aggregated and shared with the group after each round.
• The experts can adjust their answers each round, based on how they interpret the “group response” provided to them.
• The ultimate result is meant to be a true consensus of what the group thinks.

Several rounds of questionnaires are sent out to the group of experts, and the anonymous responses are aggregated and shared with the group after each round. The experts are allowed to adjust their answers in subsequent rounds, based on how they interpret the “group response” that has been provided to them. Since multiple rounds of questions are asked and the panel is told what the group thinks as a whole, the Delphi method seeks to reach the correct response through consensus.

The Delphi method was originally conceived in the 1950s by Olaf Helmer and Norman Dalkey of Rand Corp. The name refers to the Oracle of Delphi, a priestess at the temple of Apollo in ancient Greece known for her prophecies. The Delphi method allows experts to work toward a mutual agreement by conducting a circulating series of questionnaires and releasing related feedback to further the discussion with each subsequent round. The experts’ responses shift as rounds are completed based on the information brought forth by other experts participating in the analysis.

The Delphi method is a process of arriving at group consensus by providing experts with rounds of questionnaires, as well as the group response before each subsequent round.

Delphi Method Process

First, the group facilitator selects a group of experts based on the topic being examined. Once all participants are confirmed, each member of the group is sent a questionnaire with instructions to comment on each topic based on their personal opinion, experience, or previous research.

The questionnaires are returned to the facilitator, who groups the comments and prepares copies of the information. A copy of the compiled comments is sent to each participant, along with the opportunity to comment further. At the end of each comment session, all questionnaires are returned to the facilitator, who decides if another round is necessary or if the results are ready for publishing.

The questionnaire rounds can be repeated as many times as necessary to achieve a general sense of consensus.

Advantages of the Delphi Method

The Delphi method seeks to aggregate opinions from a diverse set of experts, and it can be done without having to bring everyone together for a physical meeting. Since the responses of the participants are anonymous, individual panelists don’t have to worry about repercussions for their opinions. The anonymity of the participants also helps prevent the “halo effect,” which sees higher priority given to the views of more powerful or higher-ranking members of the group.

By conducting Delphi studies, consensus can be reached over time as opinions are swayed, making the method very effective. In contrast with many other types of interviews and focus groups, Delphi studies allow participants to rethink and refine their opinions based on the input of others, contributing to a more reflective and thoughtful process.

Disadvantages of the Delphi Method

Although it provides the benefits of anonymity and the possibility for reevaluation and reflection, the Delphi method does not result in the same sort of interactions as a live discussion. A live discussion can sometimes produce a better example of consensus, as ideas and perceptions are introduced, broken down, and reassessed. Response times with the Delphi method can be long, which slows the rate of discussion. It is also possible that the information received back from the experts will provide no innate value.

The deliberate and drawn-out nature of the Delphi method also presents some challenges. Since the method often requires multiple rounds of questionnaires, there is a chance that some participants may drop out from the study before it has been completed. In addition, while there are benefits to giving participants the opportunity to reassess their views, there is a chance that they will adjust their responses so that they are more closely aligned with the views of the majority, reducing the diversity of opinions represented and diminishing the validity of the results.

Applications of the Delphi Method

Let's take a look at some general examples of when and how the Delphi method can be applied. This list is not meant to be exhaustive, but consider these options:

• Healthcare and Medicine : The Delphi method is frequently used in healthcare to forecast future medical advancements. Experts collaborate to predict technological innovations such as new diagnostic tools and treatment methods, ensuring that healthcare providers are prepared for future challenges. We'll take a look at more specific examples in the next section.
• Education : In the field of education, the Delphi method helps in curriculum development, future needs assessment, and policy formulation. Educators and industry experts work together to identify the skills and knowledge necessary for future graduates, shaping curricula that a broader group has agreed upon (or at least collaborated on).
• Business and Management : Businesses utilize the Delphi method for strategic planning, market forecasting, and identifying critical success factors. By engaging with industry experts, companies can anticipate market trends, threats, and areas of growth. There are more niche examples here, too. For instance, marketing professionals use the method to predict consumer behavior and product demand.
• Environmental Studies : Environmental researchers use the Delphi method to assess risks, predict climate change impacts, and develop sustainability strategies. Experts forecast potential environmental hazards and their consequences as part of the Delphi method here.
• Public Policy : The Delphi method also plays a part in setting public policy. Policymakers collaborate with experts to create comprehensive policy frameworks that tackle complex issues like healthcare reform or economic inequality.
• Transportation : Transportation planning leans in on the Delphi method through forecasting traffic patterns, planning infrastructure projects, and developing policies. Experts predict future traffic trends that legislative officials can then use to shape reform.
• Military and Defense : Much like in business, the Delphi method supports military and defense planning by aiding in strategic planning, threat assessment, and resource allocation. However, military strategists also use it to anticipate future threats, conflicts, and security challenges to national security; this could expand to cyberattacks, terrorism, or much smaller-scale risks.
• Tourism and Hospitality : The tourism and hospitality industry can use the Delphi method to also forecast trends. Like other industries, experts predict travel trends and tourist preferences, helping destinations and businesses plan for future demand. As the travel and entertainment industry is very diverse, the Delphi method helps aggregate some of these differing perspectives.

Real-World Example of the Delphi Method

The objective of one medical study was to develop guidelines for monitoring high-risk medications. The study aimed to assess the prevalence of laboratory testing. As part of the study guidelines, an advisory committee of national experts and local leaders employed a two-round Internet-based Delphi process to identify key medications that require monitoring.

The Delphi method achieved consensus on the medications to be included in the guidelines within those two rounds. The guidelines covered 35 drugs or drug classes and 61 lab tests. The findings bring some attention to the fact that despite general agreement on the importance of laboratory monitoring for high-risk medications, actual monitoring practices are inconsistent. Therefore, the study found that even though there was a positive general consensus towards lab monitoring, there was some variability to this.

Alternatives to the Delphi Method

If the Delphi method doesn't quite sound like the methodology for you, there are many other similar yet technically different methods. Below are some alternative examples.

• Nominal Group Technique (NGT) : Under NGT, experts independently generate ideas on a topic and then share these ideas in a round-robin format. Unlike the Delphi method, NGT involves face-to-face interactions and immediate ranking and discussion.
• Brainstorming: Brainstorming (or ideation ) occurs when a group of experts gathers to freely generate ideas and solutions. The goal is to produce as many ideas as possible which are later reviewed and refined. Brainstorming focuses on generating a large quantity of ideas through spontaneous and uninhibited thinking, whereas the Delphi method is more structured and iterative.
• Focus Groups : Focus groups are small, diverse groups of experts that engage in guided discussions to explore a specific topic in-depth. Focus groups involve real-time, interactive discussions with immediate feedback, while the Delphi method uses iterative rounds of anonymous questionnaires.
• Surveys/Questionnaires : When using surveys, experts provide their opinions and insights through structured questionnaires. Surveys gather data in a single round or a few rounds without the iterative feedback loop characteristic of the Delphi method.
• Interviews : One-on-one interviews with experts provide insights (and maybe some opinions) on a specific topic. While interviews involve direct, personal interaction and in-depth exploration of individual viewpoints, the Delphi method is more anonymous and usually with bigger groups.
• Workshops : Finally, let's touch on workshops. Workshops are interactive sessions where experts collaborate to address specific problems. Like some of the other bullets above, workshops involve real-time collaboration and hands-on activities, whereas the Delphi method uses a structured, iterative process.

What Is the Delphi Method Used for?

The Delphi method is used to establish a consensus opinion about an issue or set of issues by seeking mutual agreement from a group of experts in the relevant field. The Delphi method has been used to conduct research in numerous areas, from the defense industry to healthcare.

How Is the Delphi Method Conducted?

The group facilitator selects a group of experts based on the topic being examined and sends them a questionnaire with instructions to comment on each topic based on their personal opinion, experience, or previous research. The facilitator groups the comments from the returned questionnaires and sends copies to each participant, along with the opportunity to comment further. At the end of this session, the questionnaires are returned to the facilitator, who decides if another round is necessary or if the results are ready for publishing. This process can be repeated multiple times until a general sense of consensus is reached.

How Is "Consensus" Defined When Using the Delphi Method?

Although the Delphi method seeks to pinpoint an area of mutual agreement among the pool of experts, it is unlikely that the participants will be in complete agreement on all issues—even after several rounds of questionnaires and opportunities for reassessment. Researchers applying the Delphi method may have different thresholds for exactly what constitutes a consensus, and some critics of the method point to the subjective nature of this determination as a shortcoming.

How Many Rounds Are Generally Conducted in a Delphi Study?

Generally, a Delphi study is conducted in two to four rounds. The exact number of rounds will vary depending on the study's objectives and the complexity of the issue being addressed with a higher number of rounds needed for more advanced topics.

The Delphi method uses multiple rounds of questionnaires sent to a panel of experts to work toward a mutual agreement or consensus opinion. The participants modify their responses based on the information brought forth by other experts participating in the analysis. The Delphi method benefits from the anonymity of the participants and the opportunities it provides for reassessment, but it can also be time-consuming and in some cases may be less effective than a live discussion or focus group.

Rand Corp. “ An Experimental Application of the Delphi Method to the Use of Experts .”

BMJ Journals, Evidence-Based Nursing. “ What Are Delphi Studies? ”

National Library of Medicine. " PubMed ."

• Terms of Service
• Editorial Policy
• Privacy Policy

• View  PDF
• Download full issue

Method Article Preparing, conducting, and analyzing Delphi surveys: Cross-disciplinary practices, new directions, and advancements

• • The paper comprehensively demonstrates how to prepare, conduct, and analyze a Delphi study. In this regard, it combines several methodological advancements of the recent past (e.g., dissent analyses, scenario analyses) with state-of-the-art impulses from other disciplines like strategic management (e.g., fuzzy clustering), psychology (e.g., sentiment analyses), or clinical trials (e.g., consensus measurement).
• • By offering insights on the variety of possibilities to exploit Delphi-based data, we aim to support researchers across all disciplines in conducting Delphi studies and potentially expand and improve the method's field of application.

Graphical Abstract

• Download: Download high-res image (278KB)
• Download: Download full-size image
• Previous article in issue
• Next article in issue

Method name

Cited by (0).

• Social Science
• Qualitative Social Research
• Delphi Method

Qualitative Research: Application of the Delphi Method to CEM Research

• January 2010
• Journal of Construction Engineering and Management 136(1)

• University of Colorado Boulder

• Oregon State University

Abstract and Figures

Discover the world's research

• 25+ million members
• 160+ million publication pages
• 2.3+ billion citations

• Clinton Aigbavboa

• Anam Fatima

• Rehan Masood

• Mir Shahid Satar

• Mohammad Shoaib Alhawsawi
• Fanny E. Beriguete

• Torsten Masseck
• Ebenezer Olukanni

• Saratu Terreno

• Hyosoo Moon

• Yonghan Ahn
• Emad Ameen Alharbi
• Abdullah Alsehaimi

• D. O. Issakhova
• L. A. Bimendiyeva

• Keith J. Kowalkowski
• Daniela Schröter

• Emile Durkheim

• Irving M. Lane
• Technol Forecast
• N. C. Dalkey

• J SAFETY RES
• Anthony Veltri

• Roger J. Best
• John B. L. Robinson
• Recruit researchers
• Join for free
• Login Email Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google Welcome back! Please log in. Email · Hint Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google No account? Sign up

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

The PMC website is updating on October 15, 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Rev Lat Am Enfermagem

Language: English | Portuguese | Spanish

Guideline for incorporating the Delphi method in the evaluation of nursing theories

Diretriz para incorporao do mtodo delphi na avaliao de teorias de enfermagem, directriz para la incorporacin del mtodo delphi en la evaluacin de las teoras de enfermera, manuela campos gomes borel.

1 Universidade Federal de Juiz de Fora, Faculdade de Enfermagem, Juiz de Fora, MG, Brazil.

Rafael Oliveira Pitta Lopes

2 Universidade Federal do Rio de Janeiro, Campus Maca, Maca, RJ, Brazil.

Maira Buss Thofehrn

Maria miriam lima da nbrega.

3 Universidade Federal da Paraba, Centro de Cincias da Sade, Joo Pessoa, PB, Brazil.

Cristina Arreguy-Sena

Marcos antnio gomes brando.

4 Universidade Federal do Rio de Janeiro, Escola de Enfermagem Anna Nery, Rio de Janeiro, RJ, Brazil.

Authors Contribution:

to describe a guideline for the use of the Delphi method to evaluate nursing theories, from the perspective of internal validation.

a methodological study, targeted at the development of a guideline for the use of the Delphi method in the evaluation of nursing theories.

the Delphi method, principles of collective wisdom and levels of proficiency are used in the production of a guideline for organizing, searching, selecting and coordinating the activities of theoretical evaluators in teams. It distinguishes three phases for the theoretical evaluation process: Preparatory Phase (PP); Intermediate Phase (IP) and Theory Evaluation (TE) phase, incorporating Delphi-type selection procedures; search, selection and classification of judges/evaluators for the theory; definition of criteria for carrying out rounds and maintenance or removal of units of the theory evaluated.

Conclusion:

the developed guideline was able to adapt the elements of the Delphi method as a favorable strategy for the internal validation of nursing theories.

descrever uma diretriz de uso do mtodo Delphi para avaliao de teorias de enfermagem, na perspectiva de validao interna.

estudo metodolgico, dirigido ao desenvolvimento de uma diretriz para o uso do mtodo Delphi na avaliao de teorias de enfermagem.

Resultados:

o mtodo Delphi, os princpios da sabedoria coletiva e os nveis de proficincia so utilizados na produo de uma diretriz para organizao, busca, seleo e coordenao de atividades de avaliadores tericos em equipes. Distingue trs fases para o processo de avaliao terica: fase preparatria (FP); fase intermediria (FI) e fase de avaliao da teoria (AT), incorporando procedimentos de seleo do tipo de Delphi; busca, seleo e classificao de juzes/avaliadores da teoria; definio de critrios para a realizao de rodadas e de manuteno ou descarte de unidades da teoria avaliada.

a diretriz elaborada foi capaz de adequar os elementos do mtodo Delphi como uma estratgia favorvel validao interna de teorias de enfermagem.

describir una directriz para el uso del mtodo Delphi en la evaluacin de teoras de enfermera, desde la perspectiva de la validacin interna.

estudio metodolgico, dirigido a la elaboracin de una directriz para el uso del mtodo Delphi en la evaluacin de teoras de enfermera.

el mtodo Delphi, los principios de sabidura colectiva y los niveles de competencia se utilizan en la elaboracin de una directriz para organizacin, bsqueda, seleccin y coordinacin de las actividades de los evaluadores tericos en equipos. Distingue tres fases para el proceso de evaluacin terica: fase preparatoria (FP); fase intermedia (FI) y fase de evaluacin terica (ET), con la incorporacin de procedimientos de seleccin tipo Delphi; bsqueda, seleccin y clasificacin de jueces/evaluadores de la teora; definicin de criterios para la realizacin de rondas y mantenimiento o disposicin de unidades de la teora evaluada.

la directriz desarrollada logr adaptar los elementos del mtodo Delphi como estrategia favorable para la validacin interna de las teoras de enfermera.

Introduction

The theoretical construction through the theory-research strategy is a process initiated in the elaboration of an explicit theory in the phases of conceptual development and operationalization, later on, needing to proceed to the confirmation or non-confirmation phases, until reaching practical application conditions ( 1 ) . In the applied disciplines, empirical validation is emphasized to produce a judgment about usefulness and application.

The total undertaking of theoretical construction requires judgments to be carried out to estimate the value of a good theory, which is that rich in theoretical virtues, as science philosophers point out. Among these virtues, we can highlight the observance of singularity, falsification, parsimony, prediction, explanation, conservationism, capacity for generalization, fecundity, internal consistency, empirical wealth and abstraction ( 2 ) .

In identifying the good theory, an internal perspective can guide validation, the judgment of the intrinsic elements or by the external perspective, the judgment using the empirical test ( 3 ) . In nursing, internal validation is commonly referred to as theory evaluation which aims to determine the appropriateness of its use and the epistemological approach ( 4 ) . Depending on the formal criteria to be used, the evaluation can incorporate analysis or theoretical breakdown.

Despite the relevance of theory evaluation and the existence of dozens of structured and systematic criteria for its realization, it is still unusual to verify the application of these criteria in international literature. External validations are more common in empirical studies with statistical analyses or literature reviews ( 4 ) .

There is a continuous interest in the production of middle-range theories to describe better, explain, predict or prescribe the phenomena, facts, events or interventions with which nursing deals in daily life. However, these theories, together with those of a specific situation, are the most rarely evaluated ( 4 ) .

If, on the one hand, this continuous movement contributes to the progress of the discipline, on the other hand, it requires the availability of instruments and guidelines that promote good theoretical development practices. Therefore, access to resources that can assist in the program of elaboration, validation, refinement and theoretical application is indispensable.

The internal validation (evaluation) of a theory requires, at the same time knowledge of the theory and a high level of meta-theoretical knowledge. For such reason, it is difficult to find experienced and available meta-theorists to perform this task. Whenever possible, the coordination of this task is difficult or, even, is a complicated procedure the identification of an analyst considered proficient by the application of epistemologically consistent criteria.

Given the difficulty in locating meta-theoretical experts, the principles and criteria of collective wisdom or crowd wisdom can be useful for the construction of guidelines, methods or techniques that guide the formation of a team, capable of developing the task of theoretical evaluation with the same or superior result, compared to that of a single meta-theoretical expert. In the crowd wisdom theory, criteria such as independence, decentralization, diversity and aggregation would guide the constitution of groups, in which the aggregate decision would surpass that of the specialist, separately ( 5 ) .

In this way, analysts, not necessarily experts in meta-theory, act as judges for the content, the structure and other criteria to be judged. From the aggregate judgment, consistent results are achieved that allow for the theory evaluation to be carried out successfully. However, guidelines, methods or techniques with this conformation are not available for use with nursing theories.

Presumably, the Delphi method is adequate to evaluate a nursing theory supported by the crowd wisdom criteria, demonstrating which groups can judge adequately under conditions of uncertainty, defining the fundamental concepts, judging and adding the collective value of the ideas ( 5 - 7 ) . It has been used to deal with issues not clarified by experimental approaches in which the opinion of a group has value to clarify them, therefore, being compatible with internal validation ( 6 ) .

However, its application for this purpose is scarce. Its use was identified in the literature only in a theory of the education-informatics interface in the evaluation of the criteria of importance, precision and clarity, parsimony or simplicity, understanding, operationalization, empirical validity, fruiting and application ( 8 ) . The methodological description in the study mentioned above does not provide enough elements for its use in the evaluation of nursing theories with formal criteria, usually applied in the discipline ( 4 ) .

In Brazil, the Delphi method has helped in addressing practical problems such as trend indication, obtaining consensus on a program or intervention, expert opinion for comparing treatments and, more widely, in the construction of tools for evaluation and in the creation and validation of instruments ( 9 ) . The adaptation of the method for nursing theories evaluation remains a potential that has not yet been explored, even given its innovative character. This article was prepared given the scarcity of research studies and the potential from the development of a guideline.

The article aims to describe a guideline for the use of the Delphi method to evaluate nursing theories, from the perspective of internal validation.

This research is a methodological study for developing a guideline for the use of the Delphi method in nursing theories evaluation, indicating procedures for organizing, searching, selecting and coordinating the activities of theoretical evaluators in teams. The criteria of collective wisdom and levels of proficiency ( 5 ) were the reference basis and its elaboration took place in Rio de Janeiro, RJ, Brazil, between the months of November and December 2019.

The elements used in the methodological frameworks for the design, construction and testing of guidelines were incorporated, highlighting the following: selecting the topic and scope; adapting a prototype of a theoretical evaluation strategy guideline, using the Delphi method; group formation for development; systematic search for evidence; analysis and synthesis of available evidence and elaboration of the recommendation ( 10 ) .

The specific procedures for developing the guideline were the following: a simple review of manuscripts on the use of the Delphi method in theories evaluation and other applications; interpreting nursing theory evaluation methods ( 4 , 11 - 12 ) ; selection of the complementary material on the topic of collective wisdom; compiling and interpreting the results of using a prototype of a theoretical evaluation guideline developed in a masters thesis by one of the authors, incorporating features of the Delphi method; elaborating the guideline, taking into account the principles of construction of guidelines in health and the necessary adaptations to the theoretical-philosophical object; discussion and review by the authors; final elaboration of the guideline with diagramming interpretation of nursing theory evaluation methods.

The prototype developed in the masters thesis had the following stages: (a) selection of the experts ; (b) contact with experts and invitation for participation by those selected; (c) electronically sending the instrument to those who agreed to participate; (d) appreciation of theory evaluation items based on an agreement Likert scale; (e) receiving the answers; (f) qualitative and quantitative analysis of the results; (g) adaptation of the content for a new round of theoretical evaluation; (h) forwarding with feedback containing the data that led to the modification or maintenance of the items to perform a new evaluation; (i) receiving the answers to the adapted instrument; (j) analysis of the data from the second version; (l) final construction by consensus; (m) grammatical and orthographic review and (n) closing the theoretical evaluation.

The masters dissertation that incorporated the use of the prototype evaluated the Theory of Professional Links ( 13 ) by Meleis theoretical evaluation strategy ( 14 ) . The study that applied the prototype of the guideline respected the ethical principles of research contained in Resolution 466/2012 of the National Health Council, obtaining an approval opinion from the Research Ethics Committee, under number 3.237.583.

Encompassing the Delphi method in the nursing theories evaluation, the guideline has three phases: Preparatory Phase (PP), Intermediate Phase (IP) and Theory Evaluation (TE) phase. This study details the intermediate phase, as shown in Figure 1 .

In the preparatory phase (PP) the theory to be evaluated is chosen and the strategy to be employed is selected from the alternatives available in the literature.

In the intermediate phase, nine procedures related to the use of the Delphi method are outlined. The first procedure is related to the type of Delphi to be used, influenced by the level of the theory to be evaluated and by its application maturity.

In the second procedure, the coordination role of the theoretical evaluation is defined, which can be accumulated with the condition of a primary evaluator. This can be done by a member of the theory development team or by another researcher with consistent knowledge about it, being responsible for preparing and conducting the evaluation process. The role of the primary evaluator is to provide help to the secondary evaluators in accessing the analysis materials, with the ability to produce a preliminary report with results of the task performed that serves as a primary document intended for consultation by the other evaluators. Otherwise, the secondary operate as judges from the primary document or by performing a primary evaluative function.

The third procedure is the location and selection of candidates for evaluators/judges. The search can be performed in platforms that contain electronic resumes, the following being used as filters: nationality, academic background, language and professional performance, among others necessary. Appraisal of publications, projects or research studies is recommended, as well as verification of the performance time in the area for the careful selection of the candidate for evaluator. The selection of candidates for secondary evaluators is usually difficult, in the absence of classification criteria for a presumed meta-theoretical expertise.

So, based on collective wisdom ( 5 ) , the teams of evaluators (judges) must have diversity in their expertise levels. The expertise of the judges is analyzed by criteria in five domains, namely: Educational training in nursing theories; Professional experience in the theoretical area; Meta-theoretical experience and knowledge; Dissemination of knowledge produced on the topic of nursing meta-theories or theories and Peer-recognition of expertise on the topic of nursing meta-theories or theories ( Figure 2 ).

The candidate for evaluator/judge has their level of expertise ranked by the score obtained by the sum of the items of the five domains. The maximum score for each domain is 4 points and the minimum is 2. When the evaluators do not meet any of the criteria for a domain, they receive a zero score. The total score ranges from 0 to 20, 4 points being the minimum arbitrated value to consider the candidate suitable to be a member of the theoretical evaluation team.

The level of presumed expertise for a candidate for evaluator is established by the score obtained from the analysis of the five domains. Five levels are proposed, namely: beginner; advanced beginner; competent; proficient and expert.

The classification of a candidates presumed expertise level is established from their total score obtained (see Figure 1 ).

The definition of the minimum number of evaluators in the team will depend on the sum of the individual points of each evaluator and on the mixed composition that guarantees members with at least two different levels of expertise ( Figure 3 ).

For example, by applying the proposed equation, composing a team of seven evaluators by calculating the total required points of the sum of the evaluators will require approximately 74 points. Exemplifying, an appropriate configuration would include six advanced beginner evaluators with 10 points each and a competent evaluator with 14 points. Obviously, other configurations that respect the minimum score for the team can be applied. In addition to the total score, the requirement for mixed teams in terms of expertise levels aims to guarantee the criterion on diversity of judges.

Knowing that the losses in the face of invitations and during initial Delphi rounds are common, it is recommended to select a higher number of judges required for the minimum composition of the teams, seeking to maintain proportionality between the levels.

The fourth procedure is the invitation to the selected candidates, based on written or electronic communication and respecting ethical research principles.

The fifth procedure involves the reapplication of the criteria for categorizing the evaluators/judges by their expertise levels and adjustments to the composition of the teams considering refusals to participate.

The sixth procedure is the planning of rounds and interruption criteria. This planning considers the level of abstraction, the number of concepts and the complexity of the theory to be evaluated. The number of judges developing the evaluations is also noteworthy, as is the consideration regarding the number of criteria to be evaluated in the theory. The interruption of the rounds must be supported by the explicit judgment on the part of the evaluators after reaching an evaluative consensus or constitution of a multiplicity of ideas in the dissent. Another decision is to establish or not, a priori , a maximum number of rounds. This decision refers more to time available for the task than to the evaluative judgment.

Subsequently, to guide the evaluators, explicit criteria must be established for discarding items in each round. The items of a theory subjected to evaluation are its components like concepts, assumptions, suppositions, statements, and model schemes. Therefore, the evaluators must be certain that the decision to exclude is driven by the selected strategy and not, only, by their freely-expressed personal opinions. Their function is to judge a given theory item against the evaluation criteria established in the strategy.

The eighth procedure encompasses the definition of the consensus scope and of the stability of the answers. The consensus can be verified by formal measures of agreement, measures of central tendency, percentage of agreement, and measure of central tendency within a specific interval, among others ( 15 ) . The use of a five-point Likert scale can be planned with two purposes: (1) to verify the agreement of secondary evaluators with the result of the primary evaluation or (2) to organize the secondary evaluations in assertions that will be submitted to agreement analysis in a later round.

Even when scales are applied, it is recommended to guarantee free editing fields so that the evaluators/judges can express their suggestions, recommendations and detailed appraisals.

The ninth procedure is to provide specific guidance on the theoretical evaluation strategy. When the coordinator or primary evaluator deems it necessary, complementary and specific training on the content of the strategy can be carried out.

Theoretical evaluation is able to provide elements about a good theory, with several formal and systematic criteria available in the literature ( 2 , 11 ) . However, human resources with the competence and knowledge required to properly develop the process of judging theoretical virtues are not always available. And, in this respect, by using the principle of collective wisdom by consensus or dissent, the Delphi method can multiply the groups expertise, further expanding the universe of alternatives for the evaluation ( 15 - 16 ) . Likewise, it assists in the coordination of the process.

Through the evaluation, relationships and links of concepts are perceived, allowing the reviewer to verify the theorys strengths and limitations; identifying the need for new elements of the theory or improving the existing ones and, as a final goal, determining the potential contribution of the evaluated theory for the scientific knowledge ( 11 ) .

Unlike the theory analysis that decomposes a theory to examine its parts or components ( 4 ) , theoretical evaluation also judges them. However, even a theory judged to be good can prove to be inadequate in its descriptive, explanatory, predictive or prescriptive value from its confirmation or application. This places internal validation as a relevant stage, although not terminal of a theoretical development program.

Theories that violate the virtues of a good theory are more difficult to refute and tend not to, actually, contribute to knowledge ( 17 ) . The inadequacy of elements and constructions hinders theoretical evaluation and testing. Thus, it is fundamentally important to plan internal and/or external validation as part of a more comprehensive program. Using the guideline herein presented may avoid the expenditure of resources, when collaborating in the identification of theories that do not have sufficient virtues to support validation by field research.

The reasons for the reduced use of nursing theory evaluation strategies through formal systematic criteria are uncertain ( 4 ) . However, influence can be attributed to the difficulty in obtaining evaluators with sufficient epistemic authority to judge the meta-theoretical items of internal validation. It is supposed that the strategies linked to collective wisdom can overcome this problem of dependence on the expert with substantial advantages ( 18 ) .

The Delphi method is based on the John Deweys assumptions, emphasizing anonymous communication between individuals with expertise in a given topic, with the goal of seeking the opinion of experts in an iterative and structured way and usually seeking to achieve a consensual position ( 15 , 19 ) . The freedom and observance of the judges personal opinions guarantees the independence criterion of collective wisdom ( 5 ) .

Regarding the use in research studies, although it is used predominantly in mixed and quantitative, it has its qualitative application and even in the construction of practical theories, in the context of community organization ( 15 ) . Theory evaluation is a qualitative process permeated by subjectivity and by standards, conducts and codes of the evaluator ( 8 ) .

The Delphi method can coordinate these qualitative characteristics of the evaluation process, dealing with personal variables of the independence criterion, making the most of group work. It can be used for interpretation, for predictions and for obtaining recommendations of the evaluation developed ( 8 ) .

In choosing the Delphi method, the most common approach is the traditional one, also being referred to as normative or of consensus. It aims to reduce variance in the estimates and biases among experts. However the Delphi Policy or Policy of dissent, seeks to obtain a wide range of opinions, but without seeking consensus ( 16 ) .

For the theoretical evaluation, consensus Delphi is the most likely indication; however, the use of dissent can be recommended for theories of high originality, conceptual density, complexity and theoretical abstraction or when it is difficult to determine the consensus criteria. Additionally, one of the goals of the evaluation can be to explore the contradictions in the production of definitions or theoretical proposals.

Regarding the characteristics of the theory, consensus Delphi can be indicated for those of micro- or middle-range with conceptualization described in more than one empirical study or to evaluate partially disseminated, tested or used theories.

Supposedly, for consensus Delphi the composition of teams with a high number of evaluators is only justified when it is difficult to obtain evaluators with higher levels of expertise, because it is challenging to obtain consensus in groups of many components. On the other hand, it is assumed that the dissent approach benefits from the composition of larger teams and with a wide range of proficiency levels, tending to broaden the debate from different perspectives and to bring original elements that differ from the original theory and from the primary evaluation.

Panels with more participants tend to have lower answer rates, with an estimated reduction of 0.08 percentage points for each added participant ( 20 ) . A number of 5 to 20 experts are indicated if it is a recommendation based exclusively on the characteristics of the Delphi method ( 20 ) . Studies on the development and application of Core Outcome Set (COS) have used the Delphi method to determine which results to measure, with the predominance of Delphi panels of up to 50 people ( 20 ) .

In the theoretical evaluation it is challenging to establish a minimum and maximum number of evaluators/judges, due to its philosophical character and abstract epistemological nature inherent to theorization. For example, for new or poorly disseminated theories, it can be difficult to have many secondary evaluators with adequate expertise. On the other hand, large teams of beginner evaluators may not have knowledge of a meta-theoretical nature, causing a dispersion of perspectives that would hinder the aggregation of ideas. In this case, the guideline seeks to circumvent the limits by combining a balance between the criterion of diversity of the principle of collective wisdom and the expertise required for theoretical evaluation ( 4 - 5 ) .

The prototype of the guideline included four evaluators with three different expertise levels, and three secondary evaluators who together collectively summed 36 points (14, 13, and 9 individual points). According to the expertise points, the criteria for defining the team were useful for the composition of this small group, as the configuration of fewer participants guaranteed the maximum answer rate, as expected for this panel size ( 20 ) . The differences in training levels and stories of the evaluators ensured the decentralization criterion ( 5 ) . However, whenever possible, it is recommended to assemble teams with five or more judges.

Patricia Benners model ( 21 - 22 ) with its five levels of competence acquisition was the basis for creating the judges criteria of expertise in the guideline and sought to recognize the professional experience as an essential component for validation. The wide dissemination of studies by these authors and their criteria helped in the definition. There are more sophisticated models of aggregation rules to define the composition of the team, for example, the Contribution Weighted Model (CWM) that weighs the prognosis based on the relative performance of each judge and the accuracy of the group ( 18 , 23 ) .

The contributions of evaluators/judges have knowledge, experiences, and particular points of view in the evaluation of the theory. The iterative process of the Delphi method can allow that, in the rounds, the obscure criteria of the evaluation can be clarified or modified, through a careful interpretation of the answers of the secondary evaluators, by the coordinator. The composition of teams with different levels of competence guarantees the diversity criterion of collective wisdom ( 5 ) .

The studies commonly apply two to three rounds for the Delphi method ( 19 ) . However, the multiple criteria to be evaluated, the high number and diversity of profiles of the evaluators may require more rounds to reach consensus. It is desirable to plan a minimum according to the number of evaluators, to ensure that an excessive effort to manage the task results does not fall on the Delphi coordinator, compromising their quality.

The scope level of theories can influence the definition of criteria to be evaluated by judges; for example, when a given middle-range theory is evaluated as a model, even more specific and empirical criteria can be used ( 12 ) . However, this does not, directly, interfere with the nature of the Delphi method as a strategy.

The decision to reach consensus among judges is a type of mechanism to meet the criteria of aggregating collective wisdom, transforming individual judgments into a teams decision ( 5 ) . The consensual decision can start from the evaluators own opinion that a consensus was reached; however, it is recommended that this does not happen automatically after completion of the Delphi technique ( 19 ) .

It is necessary to specify which conditions are required for reaching consensus when the decision is qualitative. When quantitative measurement procedures are adopted, establishing the measures and cut-off points will be used to establish the degree of agreement or disagreement, compatible with the consensus or dissent ( 19 ) .

There are no mandatory rules for consensus building, but the five-point Likert scale is the most common among the scales used to estimate disagreements or agreements ( 24 ) . It makes it possible to check the degree of agreement for each item or set. For consensus reach estimates using the Likert scale, formal agreement measures such as the Kappa statistic can be applied, to verify the judges concordant judgment on the elements of the theory.

Usually, the percentages of agreement adopt the value of 0.8 or 80% as a minimum cut-off point ( 15 ) . However, the researcher can consider other cut-off points, supported by evidence or by a consistent recommendation. An explicit statement on the reach of consensus is indicated with an indication of the reasons that were considered in decision-making.

Another useful measure that can be used on the data obtained by the Likert scale is the content validity coefficient. The Aiken coefficient and its caudal probability table can be used to indicate the validity of a particular item evaluated by several judges, estimating a consensus. It can also be applied to judge the validity, by a single judge, of the content for all the items of the theory. The coefficient range varies from 0 to 1, with higher values indicating validation ( 25 ) .

Despite the literature generally recommending the use of quantitative scales to signal consensus, qualitative justifications must be added, especially when the recommendation is for the items invalidity ( 24 , 26 ) . The simple exclusion of an item can make the whole theory incoherent or illogical. This is because units of a theory play roles and have different relevance in the theoretical structure.

For example, the exclusion of an assumption can de-characterize the theory as a whole, since this typology of element functions as premises not given to the empirical test. Thus, its removal negates the ideas that guided the theorists themselves in constructing the theory. On the other hand, proposal type units are submitted, precisely, to generate testing hypotheses in empirical validation studies; therefore, they are naturally subjected to exclusion or maintenance after evidence obtained from experimentation or field research ( 1 ) .

It is highlighted that, from the evaluation of the Theory of Professional Links ( 27 - 28 ) , emerging factors demanded changes to criteria not detailed when the prototype was elaborated, which contributed to the deepening for the creation of the guideline presented in this article.

The study is not limitation-free. The focus of any research using the Delphi method will always be obtaining high-quality answers from a selection of expert individuals ( 29 ) . However, the internal validation of a theory deals with theoretical-philosophical criteria that can make it difficult, for a secondary evaluator, to produce or judge the quality of the answers by the nature of the object evaluated and by the judgment property to be performed. For example, the conceptual definition is one of the elements of a theory, evaluated in its semantics, logic, and context ( 14 ) . Notably, it can be difficult to make a good answer judgment for such a complex construct, given such properties.

The limitation for the subjectivity of the judges judgment in theories evaluation must be confronted with the philosophical root of the theorist and of the evaluator. Critical-social, hermeneutic or new pragmatism roots tend to deal with greater fluidity in the face of different perspectives, including exploring them in consensus or in dissent. On the other hand, as it requires greater objectivity of reality, post-positivism requires more stable, generalizable or measurement criteria ( 30 ) . In this last philosophical root, methods such as structural equation modeling, factorial analysis and multiple regressions may be the best choice for theory evaluation, obviously with criteria closer to external validation ( 4 ) .

Among the contributions for the advancement of scientific knowledge, the study adds an unexplored dimension of the incorporation of evaluators of different levels of meta-thematic expertise in the task of theoretical evaluation, including incorporating guidelines for the phases of this process. Given the growth in developing middle- and micro-range nursing theories and of a specific situation, with the consequent training of new theorists, the guideline can facilitate the validation process for the new theories, making up a solid base of disciplinary knowledge ( 4 , 31 ) .

On the other hand, higher levels of meta-theoretical expertise tend to require long years of training in this field, being more common to be verified in academia and among senior researchers. By exploring the principle of diversity of expertise levels for theoretical evaluation, the study encourages the creation of teams of different expertise levels, promoting cooperation and the circulation of knowledge to those involved in this process.

Finally, the application of the Delphi method in nursing theory evaluation must be clearly understood as different from the search for consensus on events, phenomena, facts, technologies, conducts or any other fundamentally empirical elements. In the empirical Delphi method, the removal of an item can have minimal implication; however, in the theory, the judgement of the inadequacy of central suppositions or concepts can place the whole theory in the condition of inadequate. Obviously, the main goal of the evaluation is to identify a good theory, which implies judging the adequacy of its components; however, this procedure must be performed with extreme caution by the evaluators, understanding that, in a theory, there is hierarchy and relationship between the elements.

The guideline developed was able to adapt the elements of the Delphi method as a favorable resource for the internal validation of nursing theories, enhancing it with the incorporation of judges with different views of the world, experiences, scientific knowledge, and creativity. The criteria displayed in the guideline adapt and articulate the proficiency levels of the evaluators with the principle of crowd wisdom, serving as a guide for the selection and composition of teams of judges, as well as facilitating the coordination of the theoretical evaluation work. Due to its innovative character, the guideline can instrumentalize nursing meta-theorists and, possibly, speed up the process of applying theories in practice.

The use of a guideline prototype in the evaluation of middle-range nursing theory, the Theory of Professional Links, brought satisfactory results that presume its feasibility and pointed out ways for refinement.

It is understood that it is essential that other researchers replicate its use in the evaluation of grand- and micro-range theories for future adjustments and updates of the guideline, also adopting evaluation strategies by formal criteria different from the one used in the prototype.