study designs research articles

• En español – ExME
• Em português – EME

An introduction to different types of study design

Posted on 6th April 2021 by Hadi Abbas

Study designs are the set of methods and procedures used to collect and analyze data in a study.

Broadly speaking, there are 2 types of study designs: descriptive studies and analytical studies.

Descriptive studies

• Describes specific characteristics in a population of interest
• The most common forms are case reports and case series
• In a case report, we discuss our experience with the patient’s symptoms, signs, diagnosis, and treatment
• In a case series, several patients with similar experiences are grouped.

Analytical Studies

Analytical studies are of 2 types: observational and experimental.

Observational studies are studies that we conduct without any intervention or experiment. In those studies, we purely observe the outcomes.  On the other hand, in experimental studies, we conduct experiments and interventions.

Observational studies

Observational studies include many subtypes. Below, I will discuss the most common designs.

Cross-sectional study:

• This design is transverse where we take a specific sample at a specific time without any follow-up
• It allows us to calculate the frequency of disease ( p revalence ) or the frequency of a risk factor
• This design is easy to conduct
• For example – if we want to know the prevalence of migraine in a population, we can conduct a cross-sectional study whereby we take a sample from the population and calculate the number of patients with migraine headaches.

Cohort study:

• We conduct this study by comparing two samples from the population: one sample with a risk factor while the other lacks this risk factor
• It shows us the risk of developing the disease in individuals with the risk factor compared to those without the risk factor ( RR = relative risk )
• Prospective : we follow the individuals in the future to know who will develop the disease
• Retrospective : we look to the past to know who developed the disease (e.g. using medical records)
• This design is the strongest among the observational studies
• For example – to find out the relative risk of developing chronic obstructive pulmonary disease (COPD) among smokers, we take a sample including smokers and non-smokers. Then, we calculate the number of individuals with COPD among both.

Case-Control Study:

• We conduct this study by comparing 2 groups: one group with the disease (cases) and another group without the disease (controls)
• This design is always retrospective
•  We aim to find out the odds of having a risk factor or an exposure if an individual has a specific disease (Odds ratio)
•  Relatively easy to conduct
• For example – we want to study the odds of being a smoker among hypertensive patients compared to normotensive ones. To do so, we choose a group of patients diagnosed with hypertension and another group that serves as the control (normal blood pressure). Then we study their smoking history to find out if there is a correlation.

Experimental Studies

• Also known as interventional studies
• Can involve animals and humans
• Pre-clinical trials involve animals
• Clinical trials are experimental studies involving humans
• In clinical trials, we study the effect of an intervention compared to another intervention or placebo. As an example, I have listed the four phases of a drug trial:

I:  We aim to assess the safety of the drug ( is it safe ? )

II: We aim to assess the efficacy of the drug ( does it work ? )

III: We want to know if this drug is better than the old treatment ( is it better ? )

IV: We follow-up to detect long-term side effects ( can it stay in the market ? )

• In randomized controlled trials, one group of participants receives the control, while the other receives the tested drug/intervention. Those studies are the best way to evaluate the efficacy of a treatment.

Finally, the figure below will help you with your understanding of different types of study designs.

References (pdf)

You may also be interested in the following blogs for further reading:

An introduction to randomized controlled trials

Case-control and cohort studies: a brief overview

Cohort studies: prospective and retrospective designs

Prevalence vs Incidence: what is the difference?

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

No Comments on An introduction to different types of study design

you are amazing one!! if I get you I’m working with you! I’m student from Ethiopian higher education. health sciences student

Very informative and easy understandable

You are my kind of doctor. Do not lose sight of your objective.

Wow very erll explained and easy to understand

I’m Khamisu Habibu community health officer student from Abubakar Tafawa Balewa university teaching hospital Bauchi, Nigeria, I really appreciate your write up and you have make it clear for the learner. thank you

well understood,thank you so much

Well understood…thanks

Simply explained. Thank You.

Thanks a lot for this nice informative article which help me to understand different study designs that I felt difficult before

That’s lovely to hear, Mona, thank you for letting the author know how useful this was. If there are any other particular topics you think would be useful to you, and are not already on the website, please do let us know.

it is very informative and useful.

thank you statistician

Fabulous to hear, thank you John.

Thanks for this information

Thanks so much for this information….I have clearly known the types of study design Thanks

That’s so good to hear, Mirembe, thank you for letting the author know.

Very helpful article!! U have simplified everything for easy understanding

I’m a health science major currently taking statistics for health care workers…this is a challenging class…thanks for the simified feedback.

That’s good to hear this has helped you. Hopefully you will find some of the other blogs useful too. If you see any topics that are missing from the website, please do let us know!

Hello. I liked your presentation, the fact that you ranked them clearly is very helpful to understand for people like me who is a novelist researcher. However, I was expecting to read much more about the Experimental studies. So please direct me if you already have or will one day. Thank you

Dear Ay. My sincere apologies for not responding to your comment sooner. You may find it useful to filter the blogs by the topic of ‘Study design and research methods’ – here is a link to that filter: https://s4be.cochrane.org/blog/topic/study-design/ This will cover more detail about experimental studies. Or have a look on our library page for further resources there – you’ll find that on the ‘Resources’ drop down from the home page.

However, if there are specific things you feel you would like to learn about experimental studies, that are missing from the website, it would be great if you could let me know too. Thank you, and best of luck. Emma

Great job Mr Hadi. I advise you to prepare and study for the Australian Medical Board Exams as soon as you finish your undergrad study in Lebanon. Good luck and hope we can meet sometime in the future. Regards ;)

You have give a good explaination of what am looking for. However, references am not sure of where to get them from.

Subscribe to our newsletter

You will receive our monthly newsletter and free access to Trip Premium.

Related Articles

Cluster Randomized Trials: Concepts

This blog summarizes the concepts of cluster randomization, and the logistical and statistical considerations while designing a cluster randomized controlled trial.

Expertise-based Randomized Controlled Trials

This blog summarizes the concepts of Expertise-based randomized controlled trials with a focus on the advantages and challenges associated with this type of study.

A well-designed cohort study can provide powerful results. This blog introduces prospective and retrospective cohort studies, discussing the advantages, disadvantages and use of these type of study designs.

• Get new issue alerts Get alerts
• Submit a Manuscript

Secondary Logo

Journal logo.

Colleague's E-mail is Invalid

Your message has been successfully sent to your colleague.

Save my selection

Study designs

Part 1 – an overview and classification.

Ranganathan, Priya; Aggarwal, Rakesh 1

Department of Anaesthesiology, Tata Memorial Centre, Mumbai, Maharashtra, India

1 Department of Gastroenterology, Sanjay Gandhi Postgraduate Institute of Medical Sciences, Lucknow, Uttar Pradesh, India

Address for correspondence: Dr. Priya Ranganathan, Department of Anaesthesiology, Tata Memorial Centre, Ernest Borges Road, Parel, Mumbai - 400 012, Maharashtra, India. E-mail: [email protected]

This is an open access journal, and articles are distributed under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 License, which allows others to remix, tweak, and build upon the work non-commercially, as long as appropriate credit is given and the new creations are licensed under the identical terms.

There are several types of research study designs, each with its inherent strengths and flaws. The study design used to answer a particular research question depends on the nature of the question and the availability of resources. In this article, which is the first part of a series on “study designs,” we provide an overview of research study designs and their classification. The subsequent articles will focus on individual designs.

INTRODUCTION

Research study design is a framework, or the set of methods and procedures used to collect and analyze data on variables specified in a particular research problem.

Research study designs are of many types, each with its advantages and limitations. The type of study design used to answer a particular research question is determined by the nature of question, the goal of research, and the availability of resources. Since the design of a study can affect the validity of its results, it is important to understand the different types of study designs and their strengths and limitations.

There are some terms that are used frequently while classifying study designs which are described in the following sections.

A variable represents a measurable attribute that varies across study units, for example, individual participants in a study, or at times even when measured in an individual person over time. Some examples of variables include age, sex, weight, height, health status, alive/dead, diseased/healthy, annual income, smoking yes/no, and treated/untreated.

Exposure (or intervention) and outcome variables

A large proportion of research studies assess the relationship between two variables. Here, the question is whether one variable is associated with or responsible for change in the value of the other variable. Exposure (or intervention) refers to the risk factor whose effect is being studied. It is also referred to as the independent or the predictor variable. The outcome (or predicted or dependent) variable develops as a consequence of the exposure (or intervention). Typically, the term “exposure” is used when the “causative” variable is naturally determined (as in observational studies – examples include age, sex, smoking, and educational status), and the term “intervention” is preferred where the researcher assigns some or all participants to receive a particular treatment for the purpose of the study (experimental studies – e.g., administration of a drug). If a drug had been started in some individuals but not in the others, before the study started, this counts as exposure, and not as intervention – since the drug was not started specifically for the study.

Observational versus interventional (or experimental) studies

Observational studies are those where the researcher is documenting a naturally occurring relationship between the exposure and the outcome that he/she is studying. The researcher does not do any active intervention in any individual, and the exposure has already been decided naturally or by some other factor. For example, looking at the incidence of lung cancer in smokers versus nonsmokers, or comparing the antenatal dietary habits of mothers with normal and low-birth babies. In these studies, the investigator did not play any role in determining the smoking or dietary habit in individuals.

For an exposure to determine the outcome, it must precede the latter. Any variable that occurs simultaneously with or following the outcome cannot be causative, and hence is not considered as an “exposure.”

Observational studies can be either descriptive (nonanalytical) or analytical (inferential) – this is discussed later in this article.

Interventional studies are experiments where the researcher actively performs an intervention in some or all members of a group of participants. This intervention could take many forms – for example, administration of a drug or vaccine, performance of a diagnostic or therapeutic procedure, and introduction of an educational tool. For example, a study could randomly assign persons to receive aspirin or placebo for a specific duration and assess the effect on the risk of developing cerebrovascular events.

Descriptive versus analytical studies

Descriptive (or nonanalytical) studies, as the name suggests, merely try to describe the data on one or more characteristics of a group of individuals. These do not try to answer questions or establish relationships between variables. Examples of descriptive studies include case reports, case series, and cross-sectional surveys (please note that cross-sectional surveys may be analytical studies as well – this will be discussed in the next article in this series). Examples of descriptive studies include a survey of dietary habits among pregnant women or a case series of patients with an unusual reaction to a drug.

Analytical studies attempt to test a hypothesis and establish causal relationships between variables. In these studies, the researcher assesses the effect of an exposure (or intervention) on an outcome. As described earlier, analytical studies can be observational (if the exposure is naturally determined) or interventional (if the researcher actively administers the intervention).

Directionality of study designs

Based on the direction of inquiry, study designs may be classified as forward-direction or backward-direction. In forward-direction studies, the researcher starts with determining the exposure to a risk factor and then assesses whether the outcome occurs at a future time point. This design is known as a cohort study. For example, a researcher can follow a group of smokers and a group of nonsmokers to determine the incidence of lung cancer in each. In backward-direction studies, the researcher begins by determining whether the outcome is present (cases vs. noncases [also called controls]) and then traces the presence of prior exposure to a risk factor. These are known as case–control studies. For example, a researcher identifies a group of normal-weight babies and a group of low-birth weight babies and then asks the mothers about their dietary habits during the index pregnancy.

Prospective versus retrospective study designs

The terms “prospective” and “retrospective” refer to the timing of the research in relation to the development of the outcome. In retrospective studies, the outcome of interest has already occurred (or not occurred – e.g., in controls) in each individual by the time s/he is enrolled, and the data are collected either from records or by asking participants to recall exposures. There is no follow-up of participants. By contrast, in prospective studies, the outcome (and sometimes even the exposure or intervention) has not occurred when the study starts and participants are followed up over a period of time to determine the occurrence of outcomes. Typically, most cohort studies are prospective studies (though there may be retrospective cohorts), whereas case–control studies are retrospective studies. An interventional study has to be, by definition, a prospective study since the investigator determines the exposure for each study participant and then follows them to observe outcomes.

The terms “prospective” versus “retrospective” studies can be confusing. Let us think of an investigator who starts a case–control study. To him/her, the process of enrolling cases and controls over a period of several months appears prospective. Hence, the use of these terms is best avoided. Or, at the very least, one must be clear that the terms relate to work flow for each individual study participant, and not to the study as a whole.

Classification of study designs

Figure 1 depicts a simple classification of research study designs. The Centre for Evidence-based Medicine has put forward a useful three-point algorithm which can help determine the design of a research study from its methods section:[ 1 ]

• Does the study describe the characteristics of a sample or does it attempt to analyze (or draw inferences about) the relationship between two variables? – If no, then it is a descriptive study, and if yes, it is an analytical (inferential) study
• If analytical, did the investigator determine the exposure? – If no, it is an observational study, and if yes, it is an experimental study
• If observational, when was the outcome determined? – at the start of the study (case–control study), at the end of a period of follow-up (cohort study), or simultaneously (cross sectional).

In the next few pieces in the series, we will discuss various study designs in greater detail.

Financial support and sponsorship

Conflicts of interest.

There are no conflicts of interest.

Epidemiologic methods; research design; research methodology

• + Favorites
• View in Gallery

Readers Of this Article Also Read

Study designs: part 2 – descriptive studies, study designs: part 3 - analytical observational studies, research studies on screening tests, introduction to qualitative research methods – part i, investigator-initiated studies: challenges and solutions.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Types of Research Designs Compared | Guide & Examples

Types of Research Designs Compared | Guide & Examples

Published on June 20, 2019 by Shona McCombes . Revised on June 22, 2023.

When you start planning a research project, developing research questions and creating a  research design , you will have to make various decisions about the type of research you want to do.

There are many ways to categorize different types of research. The words you use to describe your research depend on your discipline and field. In general, though, the form your research design takes will be shaped by:

• The type of knowledge you aim to produce
• The type of data you will collect and analyze
• The sampling methods , timescale and location of the research

This article takes a look at some common distinctions made between different types of research and outlines the key differences between them.

Table of contents

Types of research aims, types of research data, types of sampling, timescale, and location, other interesting articles.

The first thing to consider is what kind of knowledge your research aims to contribute.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

The next thing to consider is what type of data you will collect. Each kind of data is associated with a range of specific research methods and procedures.

Finally, you have to consider three closely related questions: how will you select the subjects or participants of the research? When and how often will you collect data from your subjects? And where will the research take place?

Keep in mind that the methods that you choose bring with them different risk factors and types of research bias . Biases aren’t completely avoidable, but can heavily impact the validity and reliability of your findings if left unchecked.

Choosing between all these different research types is part of the process of creating your research design , which determines exactly how your research will be conducted. But the type of research is only the first step: next, you have to make more concrete decisions about your research methods and the details of the study.

Read more about creating a research design

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Normal distribution
• Degrees of freedom
• Null hypothesis
• Discourse analysis
• Control groups
• Mixed methods research
• Non-probability sampling
• Quantitative research
• Ecological validity

Research bias

• Rosenthal effect
• Implicit bias
• Cognitive bias
• Selection bias
• Negativity bias
• Status quo bias

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. (2023, June 22). Types of Research Designs Compared | Guide & Examples. Scribbr. Retrieved August 29, 2024, from https://www.scribbr.com/methodology/types-of-research/

Is this article helpful?

Shona McCombes

Other students also liked, what is a research design | types, guide & examples, qualitative vs. quantitative research | differences, examples & methods, what is a research methodology | steps & tips, "i thought ai proofreading was useless but..".

I've been using Scribbr for years now and I know it's a service that won't disappoint. It does a good job spotting mistakes”

• Open access
• Published: 30 August 2024

Integrating causal pathway diagrams into practice facilitation to address colorectal cancer screening disparities in primary care

• Brooke Ike 1 ,
• Ashley Johnson 1 ,
• Rosemary Meza 2 &
• Allison Cole 1  

BMC Health Services Research volume  24 , Article number:  1007 ( 2024 ) Cite this article

Metrics details

Colorectal cancer (CRC) is the second leading cause of cancer death and the second most common cancer diagnosis among the Hispanic population in the United States. However, CRC screening prevalence remains lower among Hispanic adults than among non-Hispanic white adults. To reduce CRC screening disparities, efforts to implement CRC screening evidence-based interventions in primary care organizations (PCOs) must consider their potential effect on existing screening disparities. More research is needed to understand how to leverage existing implementation science methodologies to improve health disparities. The Coaching to Improve Colorectal Cancer Screening Equity (CoachIQ) pilot study explores whether integrating two implementation science tools, Causal Pathway Diagrams and practice facilitation, is a feasible and effective way to address CRC screening disparities among Hispanic patients.

We used a quasi-experimental, mixed methods design to evaluate feasibility and assess initial signals of effectiveness of the CoachIQ approach. Three PCOs received coaching from CoachIQ practice facilitators over a 12-month period. Three non-equivalent comparison group PCOs received coaching during the same period as participants in a state quality improvement program. We conducted descriptive analyses of screening rates and coaching activities.

The CoachIQ practice facilitators discussed equity, facilitated prioritization of QI activities, and reviewed CRC screening disparities during a higher proportion of coaching encounters than the comparison group practice facilitator. While the mean overall CRC screening rate in the comparison PCOs increased from 34 to 41%, the mean CRC screening rate for Hispanic patients did not increase from 30%. In contrast, the mean overall CRC screening rate at the CoachIQ PCOs increased from 41 to 44%, and the mean CRC screening rate for Hispanic patients increased from 35 to 39%.

Conclusions

The CoachIQ program merges two implementation science methodologies, practice facilitation and causal pathway diagrams, to help PCOs focus quality improvement efforts on improving CRC screening while also reducing screening disparities. Results from this pilot study demonstrate key differences between CoachIQ facilitation and standard facilitation, and point to the potential of the CoachIQ approach to decrease disparities in CRC screening.

Peer Review reports

Colorectal cancer (CRC) is the second leading cause of cancer death and the second most common cancer diagnosis among the Hispanic population in the United States (US) [ 1 ]. The US Preventive Services Task Force recommends that adults age 45–75 screen for CRC as screening reduces CRC incidence and mortality [ 2 , 3 , 4 ]. However, CRC screening prevalence remains lower among Hispanic adults 45 years of age and older than among non-Hispanic white adults (64% vs. 74% in 2020) [ 5 ]. Primary care organizations (PCOs) have a range of evidence-based interventions (EBIs) to utilize for increasing CRC screening, including small media, clinician assessment and feedback, and patient reminders [ 6 ]. To reduce CRC screening disparities, it is imperative that efforts to implement CRC screening EBIs also consider their potential effect on existing screening disparities. Yet, there is no established approach for ensuring equity is integrated into implementation efforts in PCOs.

There have been recent calls to bring more of an equity focus to implementation science [ 7 , 8 , 9 ]. Brownson et al. suggest further examination of how to leverage existing implementation science methodologies to address equity determinants and improve health disparities [ 7 ]. Practice facilitation (PF) is an established implementation method for guiding PCOs in implementing EBIs [ 10 , 11 , 12 , 13 ]. PF draws on the Model for Improvement [ 14 ], which guides practice facilitators to ask three key questions: (1) What are we trying to accomplish? (2) How will we know that a change is an improvement? (3) What change can we make that will result in improvement? In order to select what changes to make, PF involves assessing existing systems, barriers to improvement, and potential interventions for improvement [ 14 ]. PF may be an approach to improving health equity, however, minimal research has been done on how or the degree to which PF may decrease health disparities [ 15 ].

A complementary implementation science visualization tool, the causal pathway diagram (CPD), provides a structure for implementers to be explicit about the outcomes they are trying to influence, barriers that are inhibiting those outcomes, and change strategies that may be poised to bring out improved outcomes [ 16 ]. By carefully articulating how strategies work, CPDs aim to improve their effectiveness [ 17 , 18 ]. CPDs help implementers to consider whether a strategy will work under the local conditions by considering what is necessary for the strategy to work (i.e., preconditions) and what might enhance or diminish the effectiveness of the strategy (i.e., moderators). Within the context of PF, there could be potential in applying CPDs as a means to help facilitators ensure that the EBIs PCOs choose and the quality improvement (QI) strategies PCOs apply have genuine potential to address important local barriers to decreasing CRC screening disparities.

The Coaching to Improve Colorectal Cancer Screening Equity (CoachIQ) pilot study explores whether integrating CPDs into PF is a feasible and effective way to address disparities in CRC screening among Hispanic patients. The goal of this paper is to describe the CoachIQ practice facilitation approach and report changes in overall CRC screening rates and changes in CRC screening disparities before and after CoachIQ PF.

Study design

For our pilot study, we used a quasi-experimental, mixed methods design to evaluate feasibility and assess initial signals of effectiveness of the CoachIQ approach. Study procedures were reviewed by the University of Washington Human Subjects Division (STUDY00016086) and deemed to be human subjects research that qualifies for exempt status. Participants provided informed consent prior to participation.

Study setting and recruitment

We partnered with the Washington, Wyoming, Alaska, Montana, and Idaho (WWAMI) region Practice and Research Network and the Washington Association for Community Health to recruit PCOs with Hispanic patient populations and CRC screening disparities. Of the eight PCOs approached directly about participating in CoachIQ, four had the capacity and interest to participate, and the study team selected the three PCOs with the larger Hispanic patient populations for inclusion. Coaching was provided at the organization level at three PCOs located in Wyoming ( n  = 1), Washington ( n  = 1), and Idaho ( n  = 1), and involved 4 practices. Two of the PCOs were federally qualified health centers and one was a hospital affiliated health center. We provided $1500 to each PCO to compensate them for time spent on research activities.

We worked with an organization that provides PF support to Washington state PCOs to improve CRC screening to identify and engage PCOs for the non-equivalent comparison group. Of the five PCOs approached to participate, three had the interest and capacity to share CRC screening data for the study. The three PCOs received coaching support during the same time period as the CoachIQ intervention. The three non-equivalent comparison group PCOs were federally qualified health centers and included 24 practices across three organizations providing care in Washington. Coaching support was provided at the organization level. We provided $500 to each comparison group PCO to compensate them for time spent on study-specific evaluation activities.

Data collection

Throughout the study period (January 2023 – December 2023), the two CoachIQ practice facilitators kept field notes on their work with practices. At the end of each month, the two CoachIQ practice facilitators and the one comparison group practice facilitator completed a survey about coaching and QI activities. The survey was developed collaboratively with all three practice facilitators to include standard practice facilitator activities along with coaching elements related to CPD, such as whether the facilitator worked with the PCOs to understand how QI activities were expected to affect a prioritized barrier. The three practice facilitators received standardized instructions on how to interpret and respond to the survey questions. A copy of the survey is available in supplementary materials.

For each study site, we requested data on CRC screening rates from the electronic health records at two time points, the beginning and ending of coaching as was feasible for the participating practices. Data included CRC screening rates overall, among Hispanic patients, and among non-Hispanic patients. Additionally, we collected descriptive data about the participating PCOs and demographics information about their patient populations. Patient demographics data came from electronic health records data prior to the start of coaching (January 2024 for intervention practices, 2019 for comparison group practices).

Data analysis

For our qualitative analysis, we used a basic qualitative descriptive approach [ 19 ] as the aim of the qualitative work was to identify and illustrate case examples of the CoachIQ approach as experienced by participating PCOs. A trained and experienced qualitative analyst (BI) independently reviewed and hand-coded practice facilitator field notes for examples of the facilitator and PCO applying CPDs to the QI work. The qualitative analyst created data displays of poignant examples of the application of the CoachIQ approach and reviewed these displays with the larger study team (AC, AJ, and RM) to reflect on their accurate representation of the data and experiences of practices.

For our quantitative analysis, we conducted descriptive analyses. We determined the baseline CRC screening disparity by calculating the difference between the non-Hispanic CRC screening rate and the Hispanic CRC screening rate. We compared baseline data with post-coaching data for the CoachIQ and comparison PF organizations. We also conducted descriptive analyses of participant PCO descriptive information and practice facilitator monthly coaching activity data.

CoachIQ program

Each CoachIQ organization received approximately 12 months of QI support from two practice facilitators (one lead and one support) and a clinical advisor, who were also members of the study team. The CoachIQ organizations had no prior coaching on improving CRC screening. CoachIQ practice facilitators had 8 years (BI) and 3 years (AJ) of prior coaching experience. The clinical advisor was a family medicine physician (AC). The CoachIQ study team also collaborated with an implementation scientist with expertise in CPDs (RM) who trained the CoachIQ practice facilitators and clinical advisor on the CPD methodology and contributed to the CoachIQ program development.

The CoachIQ program design was derived from creating a CPD model specific to decreasing CRC screening disparities in primary care (Fig.  1 ) and blending that model into standard PF approaches. The structure of the CoachIQ program was an adaptation of key elements of study team members’ (BI and AC) prior QI work around supporting PCOs in implementing system based changes to improve opioid prescribing, The Six Building Blocks, particularly the use of three facilitation stages: prepare, implement, and sustain [ 20 , 21 , 22 ]. The CoachIQ program incorporated an equity focus and used CPDs to inform the strategies used in three practice facilitation stages as outlined in Table  1 and detailed below.

CoachIQ Causal Pathway Diagram

Stage 1: prepare

The first stage, Prepare, occurred during the first four months of the intervention and involved building the QI team, assessing baseline, and prioritizing the work. When building their QI team, PCOs were encouraged to consider including members who represented the targeted underserved demographics, Hispanic patients, and representatives with knowledge and experience about CRC screening. To assess the baseline, the QI team completed a survey and individual members participated in interviews with the practice facilitator to assess current CRC screening practices, past improvement activities, existing barriers to screening, potential strategies to overcome those barriers, and factors that might support or impede the success of those strategies. To prioritize work, the QI team participated in a coaching meeting to discuss results of the baseline assessment and identify and prioritize barriers faced by their Hispanic patient populations, and QI improvement strategies to try that could potentially overcome those prioritized barriers. The practice facilitator used CPDs to lead the team in vetting the effectiveness of alternative strategies by assessing (1) whether strategies were clearly matched with the barriers by facilitating discussions on how the strategy would address the barrier (i.e., the mechanism), (2) whether strategies were feasible by considering the preconditions for a strategy to work and factors that could moderate how well it works, and (3) what early indicators would signal whether the strategy was working to reduce the prioritized barrier. The final product of the meeting was a CPD Action Plan outlining steps to achieve “SMART” (specific, measurable, actionable, realistic, and timebound) goals and outlining the relationships in the related CPD figures guiding the work.

Stage 2: implement

During the seven months of the second stage, Implement, the QI team implemented the work prioritized during Stage 1 using CPD Action Plans developed at each monthly QI meeting with the practice facilitator. During monthly meetings, the facilitator used the CPD to guide the PCO in assessing their progress, reviewing early outcomes and equitable screening data, and adjusting implementation plans, as needed. A key aspect of the CoachIQ practice facilitator’s role was to interrogate whether QI team implementation activities were targeting the prioritized barriers identified during CPD assessment work in Stage 1. CoachIQ practice facilitators also aided the QI team in investigating why QI strategy implementation was struggling by checking in on the necessary strategy preconditions (e.g., clinicians available to attend the health equity training) or moderators (e.g., training materials relevant to clinician CRC screening work) that might be reducing the effectiveness of the strategy. Finally, CoachIQ practice facilitators worked with the QI team to review early outcome measures that were expected as a precursor to eventual decreasing of CRC screening disparities.

Stage 3: sustain

During the last month of the program, the practice facilitator worked with the QI team to assess progress and what facilitated and held back the work. The practice facilitator met with the team to discuss work left to accomplish, and helped the PCO make a sustainability plan to continue the work.

Comparison group

Throughout the study period, each PCO in the comparison group received approximately 12 months of QI support from one practice facilitator. The comparison group practice facilitator had 7 years of prior coaching experience and was not affiliated with the study. The comparison group practices had been receiving coaching on improving CRC screening for several years prior to the study start and were focused on reestablishing effective CRC screening practices and sustaining those still in effect. The QI strategies for the study period were chosen by PCOs from a list of EBIs provided by the practice facilitator. PF support involved quarterly meetings where the practice facilitator checked in on QI activities, worked through challenges, and connected the QI team to resources. There was also financial support available to these practices for staffing, patient navigation, population tracking, and patient colonoscopies.

Characteristics

The characteristics of the CoachIQ and comparison group PF organizations are shown in Table  2 . CoachIQ PCOs ranged in size from 11 to 36 primary care clinicians. The comparison group PCOs ranged in size from 19 to 74 primary care clinicians. All CoachIQ PCOs and comparison group PCOs reported using clinician reminders as an evidence-based CRC screening intervention at the start of the study. Two organizations in the comparison group and one organization in CoachIQ reported efforts to reduce structural barriers to CRC screening.

Each participating PCO reported patient characteristics for the population of patients eligible for CRC screening. In the CoachIQ PCOs, the proportion of patients identified as Hispanic ranged from 8 to 13%, and in the comparison group organizations, the range was 4–6% Hispanic. At one CoachIQ PCO, the proportion of patients without health insurance was 20%. At the remaining CoachIQ and comparison group organizations, the proportions of patients without health insurance ranged from 1 to 9%.

The two CoachIQ practice facilitators and the single comparison group practice facilitator entered data each month to record the coaching activities. For each activity, we calculated the proportion of months during the 12-month coaching cycle that the practice facilitator reported doing the coaching activity. CoachIQ practice facilitators reported discussing equity in the majority (75%) of monthly coaching encounters, compared to the comparison group practice facilitator who reported discussing equity in only 25% of coaching encounters (Table  3 ). The CoachIQ practice facilitators also reported that they facilitated prioritization of QI activities, facilitated development of an action plan, reviewed process steps and measures, and reviewed CRC screening disparities during a higher proportion of coaching encounters, compared to the comparison group practice facilitator. The comparison group practice facilitator reported providing technical support or education, connecting to others doing similar work, and sharing relevant resources at a higher proportion of coaching encounters than the CoachIQ practice facilitators.

Examples of CPD application in CoachIQ

In addition to examining the differences reported by practice facilitators in monthly surveys about their coaching activities, we developed two case examples of how CoachIQ practice facilitators used CPDs to guide the selection and implementation of QI activities with an equity focus (see the two example CPDs in supplementary materials).

The first example involved a PCO QI team that needed to adjust their implementation approach to meet a precondition for the strategy to be effective. This PCO planned to use educational brochures to address the barriers of (1) Hispanic patients’ limited knowledge of the need for CRC screening and (2) clinicians forgetting to recommend screening. The PCO QI team theorized that the brochures would help Hispanic patients learn about the importance of CRC screening and motivate them to ask about screening during busy appointments with their primary care clinician. During a CoachIQ meeting, the QI team reported that they received the brochures and placed them in their waiting rooms. The CoachIQ practice facilitator used the CPD to prompt the team to think through whether this deployment of educational brochures would be effective. It emerged that an important precondition for the strategy to be effective might not be met. If the brochures were only in the waiting rooms, it was unclear whether the patients would notice and access them prior to their appointments. Therefore, the team adjusted their implementation approach to instead incorporate giving the brochures to patients during rooming, which would make it much more likely that the brochures would address their intended barriers and outcomes.

The CoachIQ practice facilitator also helped the QI teams use early outcome measures to confirm the strength of the strategy and barrier match. The second CPD example was for a practice using targeted patient reminders to address two prioritized barriers: (1) Patients not knowing about or forgetting about needing to be screened, (2) clinicians forgetting to recommend screening. During each CoachIQ meeting with the practice facilitator, the early outcome measure of number of Hispanic patients due for screening without a referral in the chart was monitored. This PCO made targeted outreach calls to Hispanic patients who were due for screening to encourage them to schedule an appointment and to enter information in their chart highlighting their CRC screening gap. After implementing targeted patient reminders, the number of Hispanic patients without referrals who were due for screening went from 188 in May 2023 to 16 in October 2023, serving as a strong initial indicator that the strategy was working as planned.

Figure  2 summarizes the pre/post change in the primary outcomes for the CoachIQ PCOs and the comparison group PCOs: mean CRC screening rate overall, Hispanic CRC screening rate, and non-Hispanic CRC screening rate. While the mean overall CRC screening rate in the comparison PCOs increased from 34 to 41%, the mean CRC screening rate for Hispanic patients did not increase from 30% after the period of coaching. In contrast, the mean overall CRC screening rate in the CoachIQ PCOs increased from 41 to 44%, and the mean CRC screening rate for Hispanic patients increased from 35 to 39%.

Pre-Post Colorectal Cancer Screening Rates

In Table  4 , we report the baseline CRC screening rate, change in overall CRC screening rate, baseline CRC screening disparity, and change in CRC screening disparity for each of the CoachIQ and comparison group PCOs. The change in disparity for the CoachIQ PCOs ranged from growing by 1% in PCO 3 to reducing by 6% in PCO 1. In the comparison group, the change in disparity ranged from growing by 2% in PCO 1 to growing by 22% in PCO 3.

This study designed and piloted the CoachIQ program which utilized a novel application of CPD within a PF model for decreasing CRC screening disparities. CoachIQ practice facilitators worked with PCO QI teams to prioritize barriers to equitable CRC screening and design and implement QI strategies to overcome those barriers. We demonstrate that CPD can be utilized to guide practice facilitators and PCOs in their efforts to decrease disparities in CRC screening. CPD provides an operational approach to principles for equitable QI outlined by Galifant et al., [ 23 ] including using tools for health disparity tracking and understanding contextual differences when planning implementation. The practice facilitators used CPD to help guide QI teams in selecting QI activities (i.e., strategies) that would be feasible within their context considering existing circumstances (i.e., preconditions and moderators) and those that had a clear relationship to prioritized local barriers to equitable CRC screening (i.e., outcome). The practice facilitators also took time to explore how the QI teams anticipated the activities would work to affect the barriers (i.e., mechanisms), and how to measure what the QI teams expected to see as an early result of implementation (i.e., early outcomes). Through the CoachIQ approach, the practice facilitators tracked the details of the CPDs for QI teams, prompting them through targeted questions during meetings to fine tune their QI implementation. One potential strength of the CoachIQ model is that the integration of CPD methods was accomplished with the practice facilitators, rather than direct training in the method to PCO QI teams. PCO QI teams may not have sufficient time or expertise to translate implementation science methods into actionable QI activities, [ 24 ] and the CoachIQ model provides a means to bring implementation science to PCOs without the burden of them having to identify and learn these methods.

In this study, CoachIQ practice facilitators recorded completing several activities at a greater proportion of coaching encounters compared to the comparison group practice facilitator: (1) incorporated identified barriers and prioritized activities into Action Plans for the PCOs, (2) kept equity at the forefront of coaching, and (3) consistently assessed progress to check that QI activities were having the intended effect. These activities align with the core components of CPD, suggesting practice facilitators maintained fidelity to the CoachIQ approach. Few published studies describing PF programs provide detailed data about the activities performed by practice facilitators [ 25 ]. Our approach for collecting this data was assessed as feasible by the practice facilitators and may contribute to future efforts to better characterize and compare PF approaches. Demonstrating feasible measurement and documentation of implementation strategies is a critical need in the field of implementation science [ 26 ].

QI efforts have historically failed to address, or even exacerbated health disparities [ 27 , 28 , 29 ]. In our study, among the three practices receiving support through CoachIQ, all three increased their CRC screening rates overall, and two practices successfully reduced CRC screening disparities for Hispanic patients. In the comparison practices, there was an increase in the Hispanic/Non-Hispanic CRC screening disparity in all three practices, despite improved overall CRC screening rates. Although we are uncertain as to why all comparison group practices increased CRC screening disparities, including a significant increase in disparities in comparison group PCO 3, we hypothesize that without intentionally focusing on equity through the QI process, there is risk for further exacerbating disparities [ 30 ]. A strength of the CoachIQ program is using both the dynamic role of the practice facilitator and a systematic approach (CPD) to potentially help the PCO engage with equity as an ongoing practice rather than a QI project finished after one cycle [ 23 ]. Improving health equity requires a systematic approach that aligns well with the CPD approach [ 31 ].

Though we did compare CRC screening outcomes for organizations receiving support from CoachIQ practice facilitators to the CRC screening outcomes for organizations receiving standard coaching through an ongoing practice facilitation program, these two groups were non-equivalent. Despite the lack of equivalency, the detailed description of the CoachIQ program and its incorporation of CPD into practice facilitation, and the demonstration that practices receiving CoachIQ support made progress in improving equitable CRC screening contributes important data on a promising implementation science approach to decreasing CRC screening disparities. The organizations in the two arms received different financial incentives, which is a potential cofounder of the effect observed. For the pilot study, CoachIQ teams were encouraged to include Hispanic patients. Future versions of the program could go farther and include patients more intentionally as part of the baseline assessment process, and throughout implementation.

The CoachIQ program merges two implementation science methodologies, PF and CPD, to help PCOs focus QI efforts on improving CRC screening while also reducing screening disparities. Results from this pilot study demonstrate key differences between CoachIQ facilitation and standard facilitation, and point to the potential of the CoachIQ approach in decreasing disparities in CRC screening.

Data availability

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Coaching to Improve Colorectal Cancer Screening Equity

Causal Pathway Diagram

Colorectal Cancer

Evidence Based Intervention

Primary Care Organization

Practice Facilitation

Quality Improvement

United States

Cancer Facts & Figures for Hispanic and Latino People. Accessed April 30. 2024. https://www.cancer.org/research/cancer-facts-statistics/hispanics-latinos-facts-figures.html

Wolf AMD, Fontham ETH, Church TR, et al. Colorectal cancer screening for average-risk adults: 2018 guideline update from the American Cancer Society. Cancer J Clin. 2018;68(4):250–81. https://doi.org/10.3322/caac.21457 .

Article   Google Scholar  

Siegel RL, Miller KD, Wagle NS, Jemal A. Cancer statistics, 2023. Cancer J Clin. 2023;73(1):17–48. https://doi.org/10.3322/caac.21763 .

US Preventive Services Task Force. Screening for Colorectal Cancer: US Preventive Services Task Force Recommendation Statement. JAMA. 2021;325(19):1965–77. https://doi.org/10.1001/jama.2021.6238 .

CDC. Use of Colorectal Cancer Screening Tests. Colorectal Cancer. February 28. 2024. Accessed May 23, 2024. https://www.cdc.gov/colorectal-cancer/use-screening-tests/index.html

Brouwers MC, De Vito C, Bahirathan L, et al. What implementation interventions increase cancer screening rates? A systematic review. Implement Sci. 2011;6:111. https://doi.org/10.1186/1748-5908-6-111 .

Article   PubMed   PubMed Central   Google Scholar  

Brownson RC, Kumanyika SK, Kreuter MW, Haire-Joshu D. Implementation science should give higher priority to health equity. Implement Sci. 2021;16(1):28. https://doi.org/10.1186/s13012-021-01097-0 .

Kerkhoff AD, Farrand E, Marquez C, Cattamanchi A, Handley MA. Addressing health disparities through implementation science-a need to integrate an equity lens from the outset. Implement Sci. 2022;17(1):13. https://doi.org/10.1186/s13012-022-01189-5 .

Adsul P, Chambers D, Brandt HM, et al. Grounding implementation science in health equity for cancer prevention and control. Implement Sci Commun. 2022;3(1):56. https://doi.org/10.1186/s43058-022-00311-4 .

Baskerville NB, Liddy C, Hogg W. Systematic review and Meta-analysis of Practice Facilitation within Primary Care settings. Ann Fam Med. 2012;10(1):63–74. https://doi.org/10.1370/afm.1312 .

Dogherty EJ, Harrison MB, Graham ID. Facilitation as a role and process in achieving evidence-based practice in nursing: a focused review of Concept and meaning. Worldviews Evidence-Based Nurs. 2010;7(2):76–89. https://doi.org/10.1111/j.1741-6787.2010.00186.x .

Weiner BJ, Rohweder CL, Scott JE, et al. Using practice facilitation to increase Rates of Colorectal Cancer Screening in Community Health Centers, North Carolina, 2012–2013: feasibility, facilitators, and barriers. Prev Chronic Dis. 2017;14:E66. https://doi.org/10.5888/pcd14.160454 .

Kilbourne AM, Geng E, Eshun-Wilson I, et al. How does facilitation in healthcare work? Using mechanism mapping to illuminate the black box of a meta-implementation strategy. Implement Sci Commun. 2023;4(1):53. https://doi.org/10.1186/s43058-023-00435-1 .

How to Improve: Model for Improvement | Institute for Healthcare Improvement. Accessed May 23. 2024. https://www.ihi.org/resources/how-to-improve

Glaser KM, Crabtree-Ide CR, McNulty AD, et al. Improving Guideline-recommended Colorectal Cancer Screening in a federally qualified Health Center (FQHC): implementing a patient Navigation and Practice Facilitation Intervention to Promote Health Equity. Int J Environ Res Public Health. 2024;21(2):126. https://doi.org/10.3390/ijerph21020126 .

Klasnja P, Meza RD, Pullmann MD, et al. Getting cozy with causality: advances to the causal pathway diagramming method to enhance implementation precision. Implement Res Pract. 2024;5:26334895241248851. https://doi.org/10.1177/26334895241248851 .

Lewis CC, Klasnja P, Tuzzio L, Jones S, Walsh-Bailey C, Weiner B. From classification to causality: advancing understanding of mechanisms of change in implementation science. Front Public Health. 2018;6:336504. https://doi.org/10.3389/fpubh.2018.00136 .

Lewis CC, Klasnja P, Lyon AR, et al. The mechanics of implementation strategies and measures: advancing the study of implementation mechanisms. Implement Sci Commun. 2022;3(1):114. https://doi.org/10.1186/s43058-022-00358-3 .

Sandelowski M. Whatever happened to qualitative description? Res Nurs Health. 2000;23(4):334–40. https://doi.org/10.1002/1098-240X(200008)23:4%3C334::AID-NUR9%3E3.0.CO;2-G .

Article   CAS   PubMed   Google Scholar  

Six Building Blocks: A Team-Based Approach to Improving Opioid Management in Primary Care Self-Service How-To Guide. Accessed May 23. 2024. https://www.ahrq.gov/patient-safety/settings/ambulatory/improve/six-building-blocks-guide.html

Shoemaker-Hunt SJ, Evans L, Swan H, et al. Study protocol for evaluating six building blocks for opioid management implementation in primary care practices. Implement Sci Commun. 2020;1:16. https://doi.org/10.1186/s43058-020-00008-6 .

Parchman ML, Korff MV, Baldwin LM, et al. Primary care clinic re-design for prescription opioid management. J Am Board Fam Med. 2017;30(1):44–51. https://doi.org/10.3122/jabfm.2017.01.160183 .

Gallifant J, Griffin M, Pierce RL, Celi LA. From quality improvement to equality improvement projects: a scoping review and framework. iScience. 2023;26(10):107924. https://doi.org/10.1016/j.isci.2023.107924 .

King O, West E, Alston L, et al. Models and approaches for building knowledge translation capacity and capability in health services: a scoping review. Implement Sci. 2024;19:7. https://doi.org/10.1186/s13012-024-01336-0 .

Cole AM, Keppel GA, Baldwin LM, Holden E, Parchman M. Implementation Strategies Used by Facilitators to Improve Control of Cardiovascular Risk Factors in Primary Care. J Am Board Fam Med . Published online June 28, 2024:jabfm.2023.230312R1. https://doi.org/10.3122/jabfm.2023.230312R1

Kim B, Cruden G, Crable EL, Quanbeck A, Mittman BS, Wagner AD. A structured approach to applying systems analysis methods for examining implementation mechanisms. Implement Sci Commun. 2023;4(1):127. https://doi.org/10.1186/s43058-023-00504-5 .

Lion KC, Raphael JL. Partnering Health disparities Research with Quality Improvement Science in Pediatrics. Pediatrics. 2015;135(2):354–61. https://doi.org/10.1542/peds.2014-2982 .

Burnett-Hartman AN, Mehta SJ, Zheng Y, et al. Racial/Ethnic disparities in Colorectal Cancer Screening Across Healthcare systems. Am J Prev Med. 2016;51(4):e107–15. https://doi.org/10.1016/j.amepre.2016.02.025 .

Guillaume E, Dejardin O, Bouvier V, et al. Patient navigation to reduce social inequalities in colorectal cancer screening participation: a cluster randomized controlled trial. Prev Med. 2017;103:76–83. https://doi.org/10.1016/j.ypmed.2017.08.012 .

Article   PubMed   Google Scholar  

Barrell AM, Johnson L, Dehn Lunn A, Ford JA. Do primary care quality improvement frameworks consider equity? BMJ Open Qual. 2024;13(3):e002839. https://doi.org/10.1136/bmjoq-2024-002839 .

Mutha S, Marks A, Bau I, Regenstein M. Bringing Equity into Quality Improvement: An Overview and Opportunities Ahead; 2012. Retrieved from https://hsrc.himmelfarb.gwu.edu/sphhs_policy_facpubs/761

Download references

Acknowledgements

The authors would like to sincerely thank our community partners for their enthusiastic participation. The project is also supported by the National Center for Advancing Translational Sciences of the National Institutes of Health under Award Number UL1 TR002319. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Funding for this study comes from the National Institutes of Health National Cancer Institute (5P50CA244432-03).

Author information

Authors and affiliations.

Department of Family Medicine, University of Washington, Box 354982, Seattle, WA, 98195-4982, USA

Brooke Ike, Ashley Johnson & Allison Cole

Kaiser Permanente Washington, Health Research Institute, 1730 Minor Ave, Suite 1360, Seattle, WA, 98101-1466, USA

Rosemary Meza

You can also search for this author in PubMed   Google Scholar

Contributions

AC conceived the original research idea. BI, AJ, RM, and AC contributed to the design and development of the study. BI, AJ, and AC took field notes. BI and AC conducted the analyses with additional input from AJ and RM. BI, AJ, and AC drafted the manuscript. All authors read, edited, and approved the final manuscript.

Corresponding author

Correspondence to Brooke Ike .

Ethics declarations

Ethics approval and consent to participate.

Study procedures were reviewed by the University of Washington Human Subjects Division (STUDY00016086) and deemed to be human subjects research that qualifies for exempt status. Participants provided informed consent prior to participation.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

12913_2024_11471_MOESM1_ESM.docx

Supplementary Material 1: Additional File 1: CoachIQ Causal Pathway Diagram Example 1. Description of data: A diagram outlining the first case study example of applying the Causal Pathway Diagram in the CoachIQ program.

12913_2024_11471_MOESM2_ESM.docx

Supplementary Material 2: Additional File 2: CoachIQ Causal Pathway Diagram Example 2. Description of data: A diagram outlining the second case study example of applying the Causal Pathway Diagram in the CoachIQ program.

12913_2024_11471_MOESM3_ESM.docx

Supplementary Material 3: Additional File 3: CoachIQ Practice Facilitator Monthly Survey. Description of data: A monthly electronic survey practice facilitators completed about coaching activities conducted with each primary care organization during the prior month.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Ike, B., Johnson, A., Meza, R. et al. Integrating causal pathway diagrams into practice facilitation to address colorectal cancer screening disparities in primary care. BMC Health Serv Res 24 , 1007 (2024). https://doi.org/10.1186/s12913-024-11471-5

Download citation

Received : 10 July 2024

Accepted : 20 August 2024

Published : 30 August 2024

DOI : https://doi.org/10.1186/s12913-024-11471-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Colorectal cancer screening
• Practice facilitation
• Causal pathway diagram
• Implementation
• Screening disparities
• Primary care
• Quality improvement

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

• Research Guides
• Vanderbilt University Libraries
• Central Library

HART 1740W: Introduction to Design Studies - Worsnick

• Finding Books & Articles
• Getting Started With Research
• Related Resources
• Writing About Art
• Citing What You Find
• Evaluating Sources
• Interlibrary Loan & Scan and Deliver at Vanderbilt

What's This Number?

• How to Read a Call Number

Need help making sense of the Call # you found in Library Catalog?  or where to find the book in the library? 

Librarian for Art, History of Art and Architecture, and French & Italian

Books & Articles

• Finding Books

Vanderbilt University Library Search

Selecting the best search scope for your research needs:

Everything vs. Library Catalog vs. Articles

For a more detailed explanation of your search scope options consult What am I searching?

Keyword vs. Subject Searching

• << Previous: Getting Started With Research
• Next: Related Resources >>
• Last Updated: Aug 28, 2024 5:05 PM
• URL: https://researchguides.library.vanderbilt.edu/IntroDesignStudies-Worsnick

Going beyond the comparison: toward experimental instructional design research with impact

• Methodology
• Published: 28 August 2024

Cite this article

• Adam G. Gavarkovs 1 ,
• Rashmi A. Kusurkar 2 , 3 , 4 ,
• Kulamakan Kulasegaram 5 , 6 &
• Ryan Brydges 6 , 7  

Explore all metrics

To design effective instruction, educators need to know what design strategies are generally effective and why these strategies work, based on the mechanisms through which they operate. Experimental comparison studies, which compare one instructional design against another, can generate much needed evidence in support of effective design strategies. However, experimental comparison studies are often not equipped to generate evidence regarding the mechanisms through which strategies operate. Therefore, simply conducting experimental comparison studies may not provide educators with all the information they need to design more effective instruction. To generate evidence for the what and the why of design strategies, we advocate for researchers to conduct experimental comparison studies that include mediation or moderation analyses, which can illuminate the mechanisms through which design strategies operate. The purpose of this article is to provide a conceptual overview of mediation and moderation analyses for researchers who conduct experimental comparison studies in instructional design. While these statistical techniques add complexity to study design and analysis, they hold great promise for providing educators with more powerful information upon which to base their instructional design decisions. Using two real-world examples from our own work, we describe the structure of mediation and moderation analyses, emphasizing the need to control for confounding even in the context of experimental studies. We also discuss the importance of using learning theories to help identify mediating or moderating variables to test.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Similar content being viewed by others

Research-Based Instructional Perspectives

To prove or improve, that is the question: the resurgence of comparative, confounded research between 2010 and 2019

Instructional Design Methods and Practice

Explore related subjects.

• Artificial Intelligence

Data availability

No datasets were generated or analysed during the current study.

As an alternative to the regression approach, structural equation modelling (SEM) has gained popularity in the health professions education literature (Stoffels et al., 2023 ). SEM requires that a researcher make additional assumptions regarding the functional relationships between the covariates, the mediator(s), and the outcome(s) (VanderWeele, 2012 ). Though specifying these relationships can increase power, it comes with an increased risk of model misspecification (VanderWeele, 2012 ). Accordingly, we recommend that researchers beginning with experimental comparison studies involving a single mediator opt for using the regression-based approach with controls for mediator-outcome confounding (VanderWeele, 2012 ).

We did not actually analyze our data in the manner described below, for reasons described in our published manuscript. Here, we describe an alternative data analysis strategy for clarity.

Baron, R. M., & Kenny, D. A. (1986). The moderator–mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51 (6), 1173–1182. https://doi.org/10.1037/0022-3514.51.6.1173

Article   Google Scholar  

Bürkner, P. C. (2017). brms: An R package for Bayesian multilevel models using Stan . Journal of Statistical Software . https://doi.org/10.18637/jss.v080.i01

Carver, C. S., & Scheier, M. F. (1998). On the Self-Regulation of Behavior (1st ed.). Cambridge University Press. https://doi.org/10.1017/CBO9781139174794

Book   Google Scholar  

Cheung, J. J. H., & Kulasegaram, K. M. (2022). Beyond the tensions within transfer theories: Implications for adaptive expertise in the health professions. Advances in Health Sciences Education, 27 (5), 1293–1315. https://doi.org/10.1007/s10459-022-10174-y

Cheung, J. J. H., Kulasegaram, K. M., Woods, N. N., & Brydges, R. (2019). Why Content and cognition matter: Integrating conceptual knowledge to support simulation-based procedural skills transfer. Journal of General Internal Medicine, 34 (6), 969–977. https://doi.org/10.1007/s11606-019-04959-y

Cheung, J. J. H., Kulasegaram, K. M., Woods, N. N., & Brydges, R. (2021). Making concepts material: A randomized trial exploring simulation as a medium to enhance cognitive integration and transfer of learning. Simulation in Healthcare: THe Journal of the Society for Simulation in Healthcare, 16 (6), 392–400. https://doi.org/10.1097/SIH.0000000000000543

Cheung, J. J. H., Kulasegaram, K. M., Woods, N. N., Moulton, C., Ringsted, C. V., & Brydges, R. (2018). Knowing How and Knowing Why: Testing the effect of instruction designed for cognitive integration on procedural skills transfer. Advances in Health Sciences Education, 23 (1), 61–74. https://doi.org/10.1007/s10459-017-9774-1

Cook, D. A. (2005). The research we still are not doing: An agenda for the study of computer-based learning. Academic Medicine, 80 (6), 541–548. https://doi.org/10.1097/00001888-200506000-00005

Cook, D. A. (2009). The failure of e-learning research to inform educational practice, and what we can do about it. Medical Teacher, 31 (2), 158–162. https://doi.org/10.1080/01421590802691393

Durik, A. M., Shechter, O. G., Noh, M., Rozek, C. S., & Harackiewicz, J. M. (2015). What if I can’t? Success expectancies moderate the effects of utility value information on situational interest and performance. Motivation and Emotion, 39 (1), 104–118. https://doi.org/10.1007/s11031-014-9419-0

Ertmer, P. A., & Stepich, D. A. (2005). Instructional design expertise: How will we know it when we see it? Educational Technology, 45 (6), 38–43.

Google Scholar  

Fiorella, L., & Mayer, R. E. (2016). Eight ways to promote generative learning. Educational Psychology Review, 28 (4), 717–741. https://doi.org/10.1007/s10648-015-9348-9

Friedman, C. P. (1994). The research we should be doing. Academic Medicine, 69 (6), 455–457. https://doi.org/10.1097/00001888-199406000-00005

Gavarkovs, A. G., Crukley, J., Miller, E., Kusurkar, R. A., Kulasegaram, K., & Brydges, R. (2023a). Effectiveness of life goal framing to motivate medical students during online learning: A randomized controlled trial. Perspectives on Medical Education, 12 (1), 444–454. https://doi.org/10.5334/pme.1017

Gavarkovs, A. G., Finan, E., Jensen, R. D., & Brydges, R. (2024). When I say … active learning. Medical Education . https://doi.org/10.1111/medu.15383

Gavarkovs, A. G., Kusurkar, R. A., & Brydges, R. (2023b). The purpose, adaptability, confidence, and engrossment model: A novel approach for supporting professional trainees’ motivation, engagement, and academic achievement. Frontiers in Education, 8 , 1036539. https://doi.org/10.3389/feduc.2023.1036539

Hardré, P. L., Ge, X., & Thomas, M. K. (2005). Toward a model of development for instructional design expertise. Educational Technology, 45 (1), 53–57.

Hatano, G. & Inagaki, I. (1986). Two courses of expertise. In Child Development and Education in Japan (pp. 262–272). W. H. Freeman.

Hayes, A. F. (2022). Introduction to mediation, moderation, and conditional process analysis: A regression-based approach (3rd ed.). The Guilford Press.

Kalyuga, S. (2007). Expertise reversal effect and its implications for learner-tailored instruction. Educational Psychology Review, 19 (4), 509–539. https://doi.org/10.1007/s10648-007-9054-3

Kusurkar, R. A. (2023). Self-determination theory in health professions education research and practice. In R. M. Ryan (Ed.), The oxford handbook of self-determination theory (pp. 665–683). Oxford University Press. https://doi.org/10.1093/oxfordhb/9780197600047.013.33

Chapter   Google Scholar  

Kusurkar, R. A., Croiset, G., & Ten Cate, OTh. J. (2011). Twelve tips to stimulate intrinsic motivation in students through autonomy-supportive classroom teaching derived from Self-Determination Theory. Medical Teacher, 33 (12), 978–982. https://doi.org/10.3109/0142159X.2011.599896

Laidley, T. L., & Braddock, C. H. (2000). Role of adult learning theory in evaluating and designing strategies for teaching residents in ambulatory settings. Advances in Health Sciences Education, 5 (1), 43–54. https://doi.org/10.1023/A:1009863211233

Lawson, A. P., & Mayer, R. E. (2021). Benefits of writing an explanation during pauses in multimedia lessons. Educational Psychology Review, 33 (4), 1859–1885. https://doi.org/10.1007/s10648-021-09594-w

Maheu-Cadotte, M.-A., Cossette, S., Dubé, V., Fontaine, G., Lavallée, A., Lavoie, P., Mailhot, T., & Deschênes, M.-F. (2021). Efficacy of serious games in healthcare professions education: A systematic review and meta-analysis. Simulation in Healthcare: THe Journal of the Society for Simulation in Healthcare, 16 (3), 199–212. https://doi.org/10.1097/SIH.0000000000000512

Mann, K. V. (2004). The role of educational theory in continuing medical education: Has it helped us? Journal of Continuing Education in the Health Professions, 24 (Supplement 1), S22–S30. https://doi.org/10.1002/chp.1340240505

Mayer, R. E. (2023). How to assess whether an instructional intervention has an effect on learning. Educational Psychology Review, 35 (2), 64. https://doi.org/10.1007/s10648-023-09783-9

Schoemann, A. M., Boulton, A. J., & Short, S. D. (2017). Determining power and sample size for simple and complex mediation models. Social Psychological and Personality Science, 8 (4), 379–386. https://doi.org/10.1177/1948550617715068

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2001). Experimental and quasi-experimental designs for generalized causal inference . Houghton Mifflin.

Spencer, S. J., Zanna, M. P., & Fong, G. T. (2005). Establishing a causal chain: Why experiments are often more effective than mediational analyses in examining psychological processes. Journal of Personality and Social Psychology, 89 (6), 845–851. https://doi.org/10.1037/0022-3514.89.6.845

Stoffels, M., Torre, D. M., Sturgis, P., Koster, A. S., Westein, M. P. D., & Kusurkar, R. A. (2023). Steps and decisions involved when conducting structural equation modeling (SEM) analysis. Medical Teacher . https://doi.org/10.1080/0142159X.2023.2263233

Tai, A.-S., Lin, S.-H., Chu, Y.-C., Yu, T., Puhan, M. A., & VanderWeele, T. (2023). Causal mediation analysis with multiple time-varying mediators. Epidemiology, 34 (1), 8–19. https://doi.org/10.1097/EDE.0000000000001555

VanderWeele, T. J. (2012). Invited commentary: Structural equation models and epidemiologic analysis. American Journal of Epidemiology, 176 (7), 608–612. https://doi.org/10.1093/aje/kws213

VanderWeele, T. J. (2015). Explanation in causal inference: Methods for mediation and interaction . Oxford University Press.

VanderWeele, T. J. (2016). Mediation analysis: A practitioner’s guide. Annual Review of Public Health, 37 (1), 17–32. https://doi.org/10.1146/annurev-publhealth-032315-021402

VanderWeele, T. J., & Knol, M. J. (2014). A tutorial on interaction. Epidemiologic Methods . https://doi.org/10.1515/em-2013-0005

Woods, N. N., Brooks, L. R., & Norman, G. R. (2007). It all make sense: Biomedical knowledge, causal connections and memory in the novice diagnostician. Advances in Health Sciences Education, 12 (4), 405–415. https://doi.org/10.1007/s10459-006-9055-x

Download references

Author information

Authors and affiliations.

Faculty of Medicine, University of British Columbia, City Square East Tower, 555 W 12th Ave, Suite 200, Vancouver, BC, V5Z 3X7, Canada

Adam G. Gavarkovs

Research in Education, Amsterdam UMC Location Vrije Universiteit Amsterdam, De Boelelaan 1118, Amsterdam, The Netherlands

Rashmi A. Kusurkar

LEARN! Research Institute for Learning and Education, Faculty of Psychology and Education, VU University Amsterdam, Amsterdam, The Netherlands

Amsterdam Public Health, Quality of Care, Amsterdam, The Netherlands

Department of Family and Community Medicine, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada

Kulamakan Kulasegaram

The Wilson Centre, University of Toronto/University Health Network, Toronto, ON, Canada

Kulamakan Kulasegaram & Ryan Brydges

Department of Medicine, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada

Ryan Brydges

You can also search for this author in PubMed   Google Scholar

Contributions

A.G. conceptualized the topic of the manuscript and wrote the first draft. R.K., K.K., and R.B. provided contributions to subsequent drafts of the manuscript. All authors reviewed the final version of the manuscript.

Corresponding author

Correspondence to Adam G. Gavarkovs .

Ethics declarations

Conflict of interest.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Gavarkovs, A.G., Kusurkar, R.A., Kulasegaram, K. et al. Going beyond the comparison: toward experimental instructional design research with impact. Adv in Health Sci Educ (2024). https://doi.org/10.1007/s10459-024-10365-9

Download citation

Received : 06 March 2024

Accepted : 05 August 2024

Published : 28 August 2024

DOI : https://doi.org/10.1007/s10459-024-10365-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Randomized controlled trial
• Quantitative data analysis
• Learning theory
• Find a journal
• Publish with us
• Track your research

• Open access
• Published: 28 August 2024

The design, implementation, and evaluation of a blended (in-person and virtual) Clinical Competency Examination for final-year nursing students

• Rita Mojtahedzadeh 1 ,
• Tahereh Toulabi 2 , 3 &
• Aeen Mohammadi 1  

BMC Medical Education volume  24 , Article number:  936 ( 2024 ) Cite this article

7 Altmetric

Metrics details

Introduction

Studies have reported different results of evaluation methods of clinical competency tests. Therefore, this study aimed to design, implement, and evaluate a blended (in-person and virtual) Competency Examination for final-year Nursing Students.

This interventional study was conducted in two semesters of 2020–2021 using an educational action research method in the nursing and midwifery faculty. Thirteen faculty members and 84 final-year nursing students were included in the study using a census method. Eight programs and related activities were designed and conducted during the examination process. Students completed the Spielberger Anxiety Inventory before the examination, and both faculty members and students completed the Acceptance and Satisfaction questionnaire.

The results of the analysis of focused group discussions and reflections indicated that the virtual CCE was not capable of adequately assessing clinical skills. Therefore, it was decided that the CCE for final-year nursing students would be conducted using a blended method. The activities required for performing the examination were designed and implemented based on action plans. Anxiety and satisfaction were also evaluated as outcomes of the study. There was no statistically significant difference in overt, covert, and overall anxiety scores between the in-person and virtual sections of the examination ( p  > 0.05). The mean (SD) acceptance and satisfaction scores for students in virtual, in-person, and blended sections were 25.49 (4.73), 27.60 (4.70), and 25.57 (4.97), respectively, out of 30 points, in which there was a significant increase in the in-person section compared to the other sections. ( p  = 0.008). The mean acceptance and satisfaction scores for faculty members were 30.31 (4.47) in the virtual, 29.86 (3.94) in the in-person, and 30.00 (4.16) out of 33 in the blended, and there was no significant difference between the three sections ( p  = 0.864).

Evaluating nursing students’ clinical competency using a blended method was implemented and solved the problem of students’ graduation. Therefore, it is suggested that the blended method be used instead of traditional in-person or entirely virtual exams in epidemics or based on conditions, facilities, and human resources. Also, the use of patient simulation, virtual reality, and the development of necessary virtual and in-person training infrastructure for students is recommended for future research. Furthermore, considering that the acceptance of traditional in-person exams among students is higher, it is necessary to develop virtual teaching strategies.

Peer Review reports

The primary mission of the nursing profession is to educate competent, capable, and qualified nurses with the necessary knowledge and skills to provide quality nursing care to preserve and improve the community’s health [ 1 ]. Clinical education is one of the most essential and fundamental components of nursing education, in which students gain clinical experience by interacting with actual patients and addressing real problems. Therefore, assessing clinical skills is very challenging. The main goal of educational evaluation is to improve, ensure, and enhance the quality of the academic program. In this regard, evaluating learners’ performance is one of the critical and sensitive aspects of the teaching and learning process. It is considered one of the fundamental elements of the educational program [ 2 ]. The study area is educational evaluation.

Various methods are used to evaluate nursing students. The Objective Structured Clinical Examination (OSCE) is a valid and reliable method for assessing clinical competence [ 1 , 2 ]. In the last twenty years, the use of OSCE has increased significantly in evaluating medical and paramedical students to overcome the limitations of traditional practical evaluation systems [ 3 , 4 ]. The advantages of this method include providing rapid feedback, uniformity for all examinees, and providing conditions close to reality. However, the time-consuming nature and the need for a lot of personnel and equipment are some disadvantages of OSCE [ 5 , 6 ]. Additionally, some studies have shown that this method is anxiety-provoking for some students and, due to time constraints, being observed by the evaluator and other factors can cause dissatisfaction among students [ 7 , 8 ].

However, some studies have also reported that this method is not only not associated with high levels of stress among students [ 9 ] but also has higher satisfaction than traditional evaluation methods [ 4 ]. In addition, during the COVID-19 pandemic, problems such as overcrowding and student quarantine during the exam have arisen. Therefore, reducing time and costs, eliminating or reducing the tiring quarantine time, optimizing the exam, utilizing all facilities for simulating the clinical environment, using innovative methods for conducting the exam, reducing stress, increasing satisfaction, and ultimately preventing the transmission of COVID-19 are significant problems that need to be further investigated.

Studies show that using virtual space as an alternative solution is strongly felt [ 10 , 11 , 12 ]. In the fall of 2009, following the outbreak of H1N1, educational classes in the United States were held virtually [ 13 ]. Also, in 2005, during Hurricane Katrina, 27 universities in the Gulf of Texas used emergency virtual education and evaluation [ 14 ].

One of the challenges faced by healthcare providers in Iran, like most countries in the world, especially during the COVID-19 outbreak, was the shortage of nursing staff [ 15 , 16 ]. Also, in evaluating and conducting CCE for final-year students and subsequent job seekers in the Clinical Skills Center, problems such as student overcrowding and the need for quarantine during the implementation of OSCE existed. This problem has been reported not only for us but also in other countries [ 17 ]. The intelligent use of technology can solve many of these problems. Therefore, almost all educational institutions have quickly started changing their policies’ paradigms to introduce online teaching and evaluation methods [ 18 , 19 ].

During the COVID-19 pandemic, for the first time, this exam was held virtually in our school. However, feedback from professors and students and the experiences of researchers have shown that the virtual exam can only partially evaluate clinical and practical skills in some stations, such as basic skills, resuscitation, and pediatrics [ 20 ].

Additionally, using OSCE in skills assessment facilitates the evaluation of psychological-motor knowledge and attitudes and helps identify strengths and weaknesses [ 21 ]. Clinical competency is a combination of theoretical knowledge and clinical skills. Therefore, using an effective blended method focusing on the quality and safety of healthcare that measures students’ clinical skills and theoretical expertise more accurately in both in-person and virtual environments is essential. The participation of students, professors, managers, education and training staff, and the Clinical Skills Center was necessary to achieve this important and inevitable goal. Therefore, the Clinical Competency Examination (CCE) for nursing students in our nursing and midwifery school was held in the form of an educational action research process to design, implement, and evaluate a blended method. Implementing this process during the COVID-19 pandemic, when it was impossible to hold an utterly in-person exam, helped improve the quality of the exam and address its limitations and weaknesses while providing the necessary evaluation for students.

The innovation of this research lies in evaluating the clinical competency of final-year nursing students using a blended method that focuses on clinical and practical aspects. In the searches conducted, only a few studies have been done on virtual exams and simulations, and a similar study using a blended method was not found.

The research investigates the scientific and clinical abilities of nursing students through the clinical competency exam. This exam, traditionally administered in person, is a crucial milestone for final-year nursing students, marking their readiness for graduation. However, the unforeseen circumstances of the COVID-19 pandemic and the resulting restrictions rendered in-person exams impractical in 2020. This necessitated a swift and significant transition to an online format, a decision that has profound implications for the future of nursing education. While the adoption of online assessment was a necessary step to ensure student graduation and address the nursing workforce shortage during the pandemic, it was not without its challenges. The accurate assessment of clinical skills, such as dressing and CPR, proved to be a significant hurdle. This underscored the urgent need for a change in the exam format, prompting a deeper exploration of innovative solutions.

To address these problems, the research was conducted collaboratively with stakeholders, considering the context and necessity for change in exam administration. Employing an Action Research (AR) approach, a blend of online and in-person exam modalities was adopted. Necessary changes were implemented through a cyclic process involving problem identification, program design, implementation, reflection, and continuous evaluation.

The research began by posing the following questions:

What are the problems of conducting the CCE for final-year nursing students during COVID-19?

How can these problems be addressed?

What are the solutions and suggestions from the involved stakeholders?

How can the CCE be designed, implemented, and evaluated?

What is the impact of exam type on student anxiety and satisfaction?

These questions guided the research in exploring the complexities of administering the CCE amidst the COVID-19 pandemic and in devising practical solutions to ensure the validity and reliability of the assessment while meeting stakeholders’ needs.

Materials and methods

Research setting, expert panel members, job analysis, and role delineation.

This action research was conducted at the Nursing and Midwifery School of Lorestan University of Medical Sciences, with a history of approximately 40 years. The school accommodates 500 undergraduate and graduate nursing students across six specialized fields, with 84 students enrolled in their final year of undergraduate studies. Additionally, the school employs 26 full-time faculty members in nursing education departments.

An expert panel was assembled, consisting of faculty members specializing in various areas, including medical-surgical nursing, psychiatric nursing, community health nursing, pediatric nursing, and intensive care nursing. The panel also included educational department managers and the examination department supervisor. Through focused group discussions, the panel identified and examined issues regarding the exam format, and members proposed various solutions. Subsequently, after analyzing the proposed solutions and drawing upon the panel members’ experiences, specific roles for each member were delineated.

Sampling and participant selection

Given the nature of the research, purposive sampling was employed, ensuring that all individuals involved in the design, implementation, and evaluation of the exam participated in this study.

The participants in this study included final-year nursing students, faculty members, clinical skills center experts, the dean of the school, the educational deputy, group managers, and the exam department head. However, in the outcome evaluation phase, 13 faculty members participated in-person and virtually (26 times), and 84 final-year nursing students enrolled in the study using a census method in two semesters of 2020–2021 completed the questionnaires, including 37 females and 47 males. In addition, three male and ten female faculty members participated in this study; of this number, 2 were instructors, and 11 were assistant professors.

Data collection tools

In order to enhance the validity and credibility of the study and thoroughly examine the results, this study utilized a triangulation method consisting of demographic information, focus group discussions, the Spielberger Anxiety Scale questionnaire, and an Acceptance and Satisfaction Questionnaire.

Demographic information

A questionnaire was used to gather demographic information from both students and faculty members. For students, this included age, gender, and place of residence, while for faculty members, it included age, gender, field of study, and employment status.

Focus group discussion

Multiple focused group discussions were conducted with the participation of professors, administrators, experts, and students. These discussions were held through various platforms such as WhatsApp Skype, and in-person meetings while adhering to health protocols. The researcher guided the talks toward the research objectives and raised fundamental questions, such as describing the strengths and weaknesses of the previous exam, determining how to conduct the CCE considering the COVID-19 situation, deciding on virtual and in-person stations, specifying the evaluation checklists for stations, and explaining the weighting and scoring of each station.

Spielberger anxiety scale questionnaire

This study used the Spielberger Anxiety Questionnaire to measure students’ overt and covert anxiety levels. This questionnaire is an internationally standardized tool known as the STAI questionnaire that measures both overt (state) and covert (trait) anxiety [ 22 ]. The state anxiety scale (Form Y-1 of STAI) comprises twenty statements that assess the individual’s feelings at the moment of responding. The trait anxiety scale (Form Y-2 of STAI) also includes twenty statements that measure individuals’ general and typical feelings. The scores of each of the two scales ranged from 20 to 80 in the current study. The reliability coefficient of the test for the apparent and hidden anxiety scales, based on Cronbach’s alpha, was confirmed to be 0.9084 and 0.9025, respectively [ 23 , 24 ]. Furthermore, in the present study, Cronbach’s alpha value for the total anxiety questionnaire, overt anxiety, and covert anxiety scales were 0.935, 0.921, and 0.760, respectively.

Acceptance and satisfaction questionnaire

The Acceptability and Satisfaction Questionnaire for Clinical Competency Test was developed by Farajpour et al. (2012). The student questionnaire consists of ten questions, and the professor questionnaire consists of eleven questions, using a four-point Likert scale. Experts have confirmed the validity of these questionnaires, and their Cronbach’s alpha coefficients have been determined to be 0.85 and 0.87 for the professor and student questionnaires, respectively [ 6 ]. In the current study, ten medical education experts also confirmed the validity of the questionnaires. Regarding internal reliability, Cronbach’s alpha coefficients for the student satisfaction questionnaire for both virtual and in-person sections were 0.76 and 0.87, respectively. The professor satisfaction questionnaires were 0.84 and 0.87, respectively. An online platform was used to collect data for the virtual exam.

Data analysis and rigor of study

Qualitative data analysis was conducted using the method proposed by Graneheim and Lundman. Additionally, the criteria established by Lincoln and Guba (1985) were employed to confirm the rigor and validity of the data, including credibility, transferability, dependability, and confirmability [ 26 ].

In this research, data synthesis was performed by combining the collected data with various tools and methods. The findings of this study were reviewed and confirmed by participants, supervisors, mentors, and experts in qualitative research, reflecting their opinions on the alignment of findings with their experiences and perspectives on clinical competence examinations. Therefore, the member check method was used to validate credibility.

Moreover, efforts were made in this study to provide a comprehensive description of the research steps, create a suitable context for implementation, assess the views of others, and ensure the transferability of the results.

Furthermore, researchers’ interest in identifying and describing problems, reflecting, designing, implementing, and evaluating clinical competence examinations, along with the engagement of stakeholders in these examinations, was ensured by the researchers’ long-term engagement of over 25 years with the environment and stakeholders, seeking their opinions and considering their ideas and views. These factors contributed to ensuring confirmability.

In this research, by reflecting the results to the participants and making revisions by the researchers, problem clarification and solution presentation, design, implementation, and evaluation of operational programs with stakeholder participation and continuous presence were attempted to prevent biases, assumptions, and research hypotheses, and to confirm dependability.

Data analysis was performed using SPSS version 21, and descriptive statistical tests (absolute and relative frequency, mean, and standard deviation) and inferential tests (paired t-test, independent t-test, and analysis of variance) were used. The significance level was set at 0.05. Parametric tests were used based on the normality of the data according to the Kolmogorov-Smirnov statistical test.

Given that conducting the CCE for final-year nursing students required the active participation of managers, faculty members, staff, and students, and to answer the research question “How can the CCE for final-year nursing students be conducted?” and achieve the research objective of “designing, implementing, and evaluating the clinical competency exam,” the action research method was employed.

The present study was conducted based on the Dickens & Watkins model. There are four primary stages (Fig.  1 ) in the cyclical action research process: reflect, plan, act, observe, and then reflect to continue through the cycle [ 27 ].

The cyclical process of action research [ 27 ]

Stage 1: Reflection

Identification of the problem.

According to the educational regulations, final semester nursing students must complete the clinical competency exam. However, due to the COVID-19 pandemic and the critical situation in most provinces, inter-city travel restrictions, and insufficient dormitory space, conducting the CCE in-person was not feasible.

This exam was conducted virtually at our institution. However, based on the reflections from experts, researchers have found that virtual exams can only partially assess clinical and practical skills in certain stations, such as basic skills, resuscitation, and pediatrics. Furthermore, utilizing Objective Structured Clinical Examination (OSCE) in skills assessment facilitates the evaluation of psychomotor skills, knowledge, and attitudes, aiding in identifying strengths and weaknesses.

P3, “Due to the COVID-19 pandemic and the critical situation in most provinces, inter-city travel restrictions, and insufficient dormitory space, conducting the CCE in-person is not feasible.”

Stage 2: Planning

Based on the reflections gathered from the participants, the exam was designed using a blended approach (combining in-person and virtual components) as per the schedule outlined in Fig.  2 . All planned activities for the blended CCE for final-year nursing students were executed over two semesters.

P5, “Taking the exam virtually might seem easier for us and the students, but in my opinion, it’s not realistic. For instance, performing wound dressing or airway management is very practical, and it’s not possible to assess students with a virtual scenario. We need to see them in person.”

P6"I believe it’s better to conduct those activities that are highly practical in person, but for those involving communication skills like report writing, professional ethics, etc., we can opt for virtual assessment.”

Design and implementation of the blended CCE

Stage 3: Act

Cce implementation steps.

The CCE was conducted based on the flowchart in Fig.  3 and the following steps:

Steps for conducting the CCE for final-year nursing students using a blended method

Step 1: Designing the framework for conducting the blended Clinical Competency Examination

The panelists were guided to design the blended exam in focused group sessions and virtual panels based on the ADDIE (Analysis, Design, Development, Implementation, Evaluation) model [ 28 ]. Initially, needs assessment and opinion polling were conducted, followed by the operational planning of the exam, including the design of the blueprint table (Table  1 ), determination of station types (in-person or virtual), designing question stems in the form of scenarios, creating checklists and station procedure guides by expert panel groups based on participant analysis, and the development of exam implementation guidelines with participant input [ 27 ]. The design, execution, and evaluation were as follows:

In-person and virtual meetings with professors were held to determine the exam schedule, deadlines for submitting checklists, decision-making regarding the virtual or in-person nature of stations based on the type of skill (practical, communication), and presenting problems and solutions. Based on the decisions, primary skill stations, as well as cardiac and pediatric resuscitation stations, were held in person. In contrast, virtual stations for health, nursing ethics, nursing reports, nursing diagnosis, physical examinations, and psychiatric nursing were held.

News about the exam was communicated to students through the college website and text messages. Then, an online orientation session was held on Skype with students regarding the need assessment of pre-exam educational workshops, virtual and in-person exam standards, how to use exam software, how to conduct virtual exams, explaining the necessary infrastructure for participating in the exam by students, completing anxiety and satisfaction questionnaires, rules and regulations, how to deal with rejected individuals, and exam testing and Q&A. Additionally, a pre-exam in-person orientation session was held.

To inform students about the entire educational process, the resources and educational content recommended by the professors, including PDF files, photos and videos, instructions, and links, were shared through a virtual group on the social media messenger, and scientific information was also, questions were asked and answered through this platform.

Correspondence and necessary coordination were made with the university clinical skills center to conduct in-person workshops and exams.

Following the Test-centered approach, the Angoff Modified method [ 29 , 30 ] was used to determine the scoring criteria for each station by panelists tasked with assigning scores.

Additionally, in establishing standards for this blended CCE for fourth-year nursing students, for whom graduation was a prerequisite, the panelists, as experienced clinical educators familiar with the performance and future roles of these students and the assessment method of the blended exam, were involved [ 29 , 30 ](Table 1 ).

Step 2: Preparing the necessary infrastructure for conducting the exam

Software infrastructure.

The pre- and post-virtual exam questions, scenarios, and questionnaires were uploaded using online software.

The exam was conducted on a trial basis in multiple sessions with the participation of several faculty members, and any issues were addressed. Students were authenticated to enter the exam environment via email and personal information verification. The questions for each station were designed and entered into the software by the respective station instructors and the examination coordinator, who facilitated the exam. The questions were formatted as clinical scenarios, images, descriptive questions, and multiple-choice questions, emphasizing the clinical and practical aspects. This software had various features for administering different types of exams and various question formats, including multiple-choice, descriptive, scenario-based, image-based, video-based, matching, Excel output, and graphical and descriptive statistical analyses. It also had automatic questionnaire completion, notification emails, score addition to questionnaires, prevention of multiple answer submissions, and the ability to upload files up to 4 gigabytes. Student authentication was based on national identification numbers and student IDs, serving as user IDs and passwords. Students could enter the exam environment using their email and multi-level personal information verification. If the information did not match, individuals could not access the exam environment.

Checklists and questionnaires

A student list was prepared, and checklists for the in-person exam and anxiety and satisfaction questionnaires were reproduced.

Empowerment workshops for professors and education staff

Educational needs of faculty members and academic staff include conducting clinical competency exams using the OSCE method; simulating and evaluating OSCE exams; designing standardized questions, checklists, and scenarios; innovative approaches in clinical evaluations; designing physical spaces and setting up stations; and assessing ethics and professional commitment in clinical competency exams.

Student empowerment programs

According to the students’ needs assessment results, in-person workshops on cardiopulmonary resuscitation and airway management and online workshops were held on health, pediatrics, cardiopulmonary resuscitation, ethics, nursing diagnosis, and report writing through Skype messenger. In addition, vaccination notes, psychiatric nursing, and educational files on clinical examinations and basic skills were recorded by instructors and made available to students via virtual groups.

Step 3: CCE implementation

The CCE was held in two parts, in-person and virtual.

In-person exam

The OSCE method was used for this section of the exam. The basic skills station exam included dressing and injections, and the CPR and pediatrics stations were conducted in person. The students were divided into two groups of 21 each semester, and the exam was held in two shifts. While adhering to quarantine protocols, the students performed the procedures for seven minutes at each station, and instructors evaluated them using a checklist. An additional minute was allotted for transitioning to the next station.

Virtual exam

The professional ethics, nursing diagnosis, nursing report, health, psychiatric nursing, and physical examination stations were conducted virtually after the in-person exam. This exam was made available to students via a primary and a secondary link in a virtual space at the scheduled time. Students were first verified, and after the specified time elapsed, the ability to respond to inactive questions and submitted answers was sent. During the exam, full support was provided by the examination center.

The examination coordinator conducted the entire virtual exam process. The exam results were announced 48 h after the exam. A passing grade was considered to be a score higher than 60% in all stations. Students who failed in various stations were given the opportunity for remediation based on faculty feedback, either through additional study or participation in educational workshops. Subsequent exams were held one week apart from the initial exam. It was stipulated that students who failed in more than half of the stations would be evaluated in the following semester. If they failed in more than three sessions at a station, a decision would be made by the faculty’s educational council. However, no students met these situations.

Step 4: Evaluation

The evaluation of the exam was conducted by examiners using a checklist, and the results were announced as pass or fail.

Stage 4: Observation / evaluation

In this study, both process and outcome evaluations were conducted:

Process evaluation

All programs and activities implemented during the test design and administration process were evaluated in the process evaluation. This evaluation was based on operational program control and reflections received from participants through group discussion sessions and virtual groups.

Sample reflections received from faculty members, managers, experts, and students through group discussions and social messaging platforms after the changes:

P7: “The implementation of the blended virtual exam, in the conditions of the COVID-19 crisis where the possibility of holding in-person exams was not fully available, in my opinion, was able to improve the quality of exam administration and address the limitations and weaknesses of the exam entirely virtually.”

P5: “In my opinion, this blended method was able to better evaluate students in terms of clinical readiness for entering clinical practice.”

Outcomes evaluation

The study outcomes were student anxiety, student acceptance and satisfaction, and faculty acceptance and satisfaction. Before the start of the in-person and virtual exams, the Spielberger Anxiety Questionnaire was provided to students. Additionally, immediately after the exam, students and instructors completed the acceptance and satisfaction questionnaire for the relevant section. After the exam, students and instructors completed the acceptance and satisfaction questionnaire again for the entire exam process, including feasibility, satisfaction with its implementation, and educational impact.

Design framework and implementation for the blended Clinical Competency Examination

The exam was planned using a blended method (part in-person, part virtual) according to the Fig.  2 schedule, and all planned programs for the blended CCE for final-year nursing students were implemented in two semesters.

Evaluation results

In this study, 84 final-year nursing students participated, including 37 females (44.05%) and 47 males (55.95%). Among them, 28 (33.3%) were dormitory residents, and 56 (66.7%) were non-dormitory residents.

In this study, both process and outcome evaluations were conducted.

All programs and activities implemented during the test design and administration process were evaluated in the process evaluation (Table  2 ). This evaluation was based on operational program control and reflections received from participants through group discussion sessions and virtual groups on social media.

Anxiety and satisfaction were examined and evaluated as study outcomes, and the results are presented below.

The paired t-test results in Table  3 showed no statistically significant difference in overt anxiety ( p  = 0.56), covert anxiety ( p  = 0.13), and total anxiety scores ( p  = 0.167) between the in-person and virtual sections before the blended Clinical Competency Examination.

However, the mean (SD) of overt anxiety in persons in males and females was 49.27 (11.16) and 43.63 (13.60), respectively, and this difference was statistically significant ( p  = 0.03). Also, the mean (SD) of overt virtual anxiety in males and females was 45.70 (11.88) and 51.00 (9.51), respectively, and this difference was statistically significant ( p  = 0.03). However, there was no significant difference between males and females regarding covert anxiety in the person ( p  = 0.94) and virtual ( p  = 0.60) sections. In addition, the highest percentage of overt anxiety was apparent in the virtual section among women (15.40%) and the in-person section among men (21.28%) and was prevalent at a moderate to high level.

According to Table  4 , One-way analysis of variance showed a significant difference between the virtual, in-person, and blended sections in terms of acceptance and satisfaction scores.

The results of the One-way analysis of variance showed that the mean (SD) acceptance and satisfaction scores of nursing students of the CCE in virtual, in-person, and blended sections were 25.49 (4.73), 27.60 (4.70), and 25.57 (4.97) out of 30, respectively. There was a significant difference between the three sections ( p  = 0.008).

In addition, 3 (7.23%) male and 10 (76.3%) female faculty members participated in this study; of this number, 2 (15.38%) were instructors, and 11 (84.62%) were assistant professors. Moreover, they were between 29 and 50 years old, with a mean (SD) of 41.37 (6.27). Furthermore, they had 4 to 20 years of work experience with a mean and standard deviation of 13.22(4.43).

The results of the analysis of variance showed that the mean (SD) acceptance and satisfaction scores of faculty members of the CCE in virtual, in-person, and blended sections were 30.31 (4.47), 29.86 (3.94), and 30.00 (4.16) out of 33, respectively. There was no significant difference between the three sections ( p  = 0.864).

This action research study showed that the blended CCE for nursing students is feasible and, depending on the conditions and objectives, evaluation stations can be designed and implemented virtually or in person.

The blended exam, combining in-person and virtual elements, managed to address some of the weaknesses of entirely virtual exams conducted in previous terms due to the COVID-19 pandemic. Given the pandemic conditions, the possibility of performing all in-person stations was not feasible due to the risk of students and evaluators contracting the virus, as well as the need for prolonged quarantine. Additionally, to meet the staffing needs of hospitals, nursing students needed to graduate. By implementing the blended exam idea and conducting in-person evaluations at clinical stations, the assessment of nursing students’ clinical competence was brought closer to reality compared to the entirely virtual method.

Furthermore, the need for human resources, station setup costs, and time spent was less than the entirely in-person method. Therefore, in pandemics or conditions where sufficient financial resources and human resources are not available, the blended approach can be utilized.

Additionally, the evaluation results showed that students’ total and overt anxiety in both virtual and in-person sections of the blended CCE did not differ significantly. However, the overt anxiety of female students in the virtual section and male students in the in-person section was considerably higher. Nevertheless, students’ covert anxiety related to personal characteristics did not differ in virtual and in-person exam sections. However, students’ acceptance and satisfaction in the in-person section were higher than in the virtual and blended sections, with a significant difference. The acceptance and satisfaction of faculty members from the CCE in in-person, virtual, and blended sections were the same and relatively high.

A blended CCE nursing competency exam was not found in the literature review. However, recent studies, especially during the COVID-19 pandemic, have designed and implemented this exam using virtual OSCE. Previously, the CCE was held in-person or through traditional OSCE methods.

During the COVID-19 pandemic, nursing schools worldwide faced difficulties administering clinical competency exams for students. The virtual simulation was used to evaluate clinical competency and develop nursing students’ clinical skills in the United States, including standard videos, home videos, and clinical scenarios. Additionally, an online virtual simulation program was designed to assess the clinical competency of senior nursing students in Hong Kong as a potential alternative to traditional clinical training [ 31 ].

A traditional in-person OSCE was also redesigned and developed through a virtual conferencing platform for nursing students at the University of Texas Medical Branch in Galveston. Survey findings showed that most professors and students considered virtual OSCE a highly effective tool for evaluating communication skills, obtaining a medical history, making differential diagnoses, and managing patients. However, professors noted that evaluating examination techniques in a virtual environment is challenging [ 32 ].

However, Biranvand reported that less than half of the nursing students believed the in-person OSCE was stressful [ 33 ]. At the same time, the results of another study showed that 96.2% of nursing students perceived the exam as anxiety-provoking [ 1 ]. Students believe that the stress of this exam is primarily related to exam time, complexity, and the execution of techniques, as well as confusion about exam methods [ 7 ]. In contrast to previous research results, in a study conducted in Egypt, 75% of students reported that the OSCE method has less stress than other examination methods [ 9 ]. However, there has yet to be a consensus across studies on the causes and extent of anxiety-provoking in the OSCE exam. In a study, the researchers found that in addition to the factors mentioned above, the evaluator’s presence could also be a cause of stress [ 34 ]. Another survey study showed that students perceived the OSCE method as more stressful than the traditional method, mainly due to the large number of stations, exam items, and time constraints [ 7 ]. Another study in Egypt, which designed two stages of the OSCE exam for 75 nursing students, found that 65.6% of students reported that the second stage exam was stressful due to the problem-solving station. In contrast, only 38.9% of participants considered the first-stage exam stressful [ 35 ]. Given that various studies have reported anxiety as one of the disadvantages of the OSCE exam, in this study, one of the outcomes evaluated was the anxiety of final-year nursing students. There was no significant difference in total anxiety and overt anxiety between students in the in-person and virtual sections of the blended Clinical Competency Examination. The overt anxiety was higher in male students in the in-person part and female students in the virtual section, which may be due to their personality traits, but further research is needed to confirm this. Moreover, since students’ total and overt anxiety in the in-person and virtual sections of the exam are the same in resource and workforce shortages or pandemics, the blended CCE is suggested as a suitable alternative to the traditional OSCE test. However, for generalization of the results, it is recommended that future studies consider three intervention groups, where all OSCE stations are conducted virtually in the first group, in-person in the second group, and a blend of in-person and virtual in the third group. Furthermore, the results of the study by Rafati et al. showed that the use of the OSCE clinical competency exam using the OSCE method is acceptable, valid, and reliable for assessing nursing skills, as 50% of the students were delighted, and 34.6% were relatively satisfied with the OSCE clinical competency exam. Additionally, 57.7% of the students believed the exam revealed learning weaknesses [ 1 ]. Another survey study showed that despite higher anxiety about the OSCE exam, students thought that this exam provides equal opportunities for everyone, is less complicated than the traditional method, and encourages the active participation of students [ 7 ]. In another study on maternal and infant care, 95% of the students believed the traditional exam only evaluates memory or practical skills. In contrast, the OSCE exam assesses knowledge, understanding, cognitive and analytical skills, communication, and emotional skills. They believed that explicit evaluation goals, appropriate implementation guidelines, appropriate scheduling, wearing uniforms, equipping the workroom, evaluating many skills, and providing fast feedback are among the advantages of this exam [ 36 ]. Moreover, in a survey study, most students were satisfied with the clinical environment offered by the OSCE CCE using the OSCE method, which is close to reality and involves a hypothetical patient in necessary situations that increase work safety. On the other hand, factors such as the scheduling of stations and time constraints have led to dissatisfaction among students [ 37 ].

Furthermore, another study showed that virtual simulations effectively improve students’ skills in tracheostomy suctioning, triage concepts, evaluation, life-saving interventions, clinical reasoning skills, clinical judgment skills, intravenous catheterization skills, role-based nursing care, individual readiness, critical thinking, reducing anxiety levels, and increasing confidence in the laboratory, clinical nursing education, interactive communication, and health evaluation skills. In addition to knowledge and skills, new findings indicate that virtual simulations can increase confidence, change attitudes and behaviors, and be an innovative, flexible, and hopeful approach for new nurses and nursing students [ 38 ].

Various studies have evaluated the satisfaction of students and faculty members with the OSCE Clinical Competency Examination. In this study, one of the evaluated outcomes was the acceptability and satisfaction of students and faculty members with implementing the CCE in blended, virtual, and in-person sections, which was relatively high and consistent with other studies. One crucial factor that influenced the satisfaction of this study was the provision of virtual justification sessions for students and coordination sessions with faculty members. Social messaging groups were formed through virtual and in-person communication, instructions were explained, expectations and tasks were clarified, and questions were answered. Students and faculty members could access the required information with minimal presence in medical education centers and time and cost constraints. Moreover, with the blended evaluation, the researcher’s communication with participants was more accessible. The written guidelines and uploaded educational content of the workshops enabled students to save the desired topics and review them later if needed. Students had easy access to scientific and up-to-date information, and the application of social messengers and Skype allowed for sending photos and videos, conducting workshops, and questions and answering questions. However, the clinical workshops and examinations were held in-person to ensure accuracy. The virtual part of the examination was conducted through online software, and questions focused on each station’s clinical and practical aspects. Students answered various questions, including multiple-choice, descriptive, scenario, picture, and puzzle questions, within a specified time. The blended examination evaluated clinical competency and did not delay these individuals’ entry into the job market. Moreover, during the severe human resource shortage faced by the healthcare system, the examination allowed several nurses to enter the country’s healthcare system. The blended examination can substitute in-person examination in pandemic and non-pandemic situations, saving facilities, equipment, and human resources. The results of this study can also serve as a model to guide other nursing departments that require appropriate planning and arrangements for Conducting Clinical Competency Examinations in blended formats. This examination can also be developed to evaluate students’ clinical performance.

One of the practical limitations of the study was the possibility that participants might need to complete the questionnaires accurately or be concerned about losing marks. Therefore, in a virtual session before the in-person exam, the objectives and importance of the study were explained. Participants were assured that it would not affect their evaluation and that they should not worry about losing marks. Additionally, active participation from all nursing students, faculty members, and staff was necessary for implementing this plan, achieved through prior coordination, virtual meetings, virtual group formation, and continuous reflection of results, creating the motivation for continued collaboration and participation.

Among other limitations of this study included the use of the Spielberger Anxiety Questionnaire to measure students’ anxiety. It is suggested that future studies use a dedicated anxiety questionnaire designed explicitly for pre-exam anxiety measurement. Another limitation of the current research was its implementation in nursing and midwifery faculty. Therefore, it is recommended that similar studies be conducted in nursing and midwifery faculties of other universities, as well as in related fields, and over multiple consecutive semesters. Additionally, for more precise effectiveness assessment, intervention studies in three separate virtual, in-person, and hybrid groups using electronic checklists are proposed. Furthermore, it is recommended that students be evaluated in terms of other dimensions and variables such as awareness, clinical skill acquisition, self-confidence, and self-efficacy.

Conducting in-person Clinical Competency Examination (CCE) during critical situations, such as the COVID-19 pandemic, is challenging. Instead of virtual exams, blended evaluation is a feasible approach to overcome the shortages of virtual ones and closely mimic in-person scenarios. Using a blended method in pandemics or resource shortages, it is possible to design, implement, and evaluate stations that evaluate basic and advanced clinical skills in in-person section, as well as stations that focus on communication, reporting, nursing diagnosis, professional ethics, mental health, and community health based on scenarios in a virtual section, and replace traditional OSCE exams. Furthermore, the use of patient simulators, virtual reality, virtual practice, and the development of virtual and in-person training infrastructure to improve the quality of clinical education and evaluation and obtain the necessary clinical competencies for students is recommended. Also, since few studies have been conducted using the blended method, it is suggested that future research be conducted in three intervention groups, over longer semesters, based on clinical evaluation models and influential on other outcomes such as awareness and clinical skill acquisition self-efficacy, confidence, obtained grades, and estimation of material and human resources costs. This approach reduced the need for physical space for in-person exams, ensuring participant quarantine and health safety with higher quality. Additionally, a more accurate assessment of nursing students’ practical abilities was achieved compared to a solely virtual exam.

Data availability

The datasets generated and analyzed during the current study are available on request from the corresponding author.

Rafati F, Pilevarzade M, Kiani A. Designing, implementing and evaluating once to assess nursing students’ clinical competence in Jiroft faculty of nursing and midwifery. Nurs Midwifery J. 2020;18(2):118–28.

Google Scholar  

Sadeghi T, Ravari A, Shahabinejad M, Hallakoei M, Shafiee M, Khodadadi H. Performing of OSCE method in nursing students of Rafsanjan University of Medical science before entering the clinical field in the year 2010: a process for quality improvement. Community Health J. 2012;6(1):1–8.

Ali GA, Mehdi AY, Ali HA. Objective structured clinical examination (OSCE) as an assessment tool for clinical skills in Sohag University: nursing students’ perspective. J Environ Stud. 2012;8(1):59–69.

Article   Google Scholar  

Bolourchifard F, Neishabouri M, Ashktorab T, Nasrollahzadeh S. Satisfaction of nursing students with two clinical evaluation methods: objective structured clinical examination (OSCE) and practical examination of clinical competence. Adv Nurs Midwifery. 2010;19(66):38–42.

Noohi E, Motesadi M, Haghdoost A. Clinical teachers’ viewpoints towards Objective Structured Clinical examination in Kerman University of Medical Science. Iran J Med Educ. 2008;8(1):113–20.

Reza Masouleh S, Zare A, Chehrzad M, Atrkarruoshan Z. Comparing two methods of evaluation, objective structured practical examination and traditional examination, on the satisfaction of students in Shahid Beheshti faculty of nursing and midwifery. J Holist Nurs Midwifery. 2008;18(1):22–30.

Bagheri M, Sadeghineajad Forotagheh M, Shaghayee Fallah M. The comparison of stressors in the assessment of basic clinical skills with traditional method and OSCE in nursing students. Life Sci J. 2012;9(4):1748–52.

Eldarir SH, El Sebaae HA, El Feky HA, Hussein HA, El Fadil NA, El Shaeer IH. An introduction of OSCE versus the traditional method in nursing education: Faculty capacity building and students’ perspectives. J Am Sci. 2010;6(12):1002–14.

Al-Zeftawy AM, Khaton SE. Student evaluation of an OSCE in Community Health nursing clinical course at Faculty of nursing, Tanta University. J Nurs Health Sci. 2016;5(4):68–76.

Hayter M, Jackson D. Pre-registration undergraduate nurses and the COVID-19 pandemic: students or workers? J Clin Nurs. 2020;29(17–18):3115–6.

Bayham J, Fenichel EP. Impact of school closures for COVID-19 on the US health-care workforce and net mortality: a modeling study. Lancet Public Health. 2020;5(5):e271–8.

Murphy MPA. COVID-19 and emergency eLearning: consequences of the securitization of higher education for post-pandemic pedagogy. Contemp Secur Policy. 2020;41(3):492–505.

Allen IE, Seaman J. Learning on demand: Online education in the United States, 2009.

Meyer KA, Wilson JL. The role of Online Learning in the emergency plans of Flagship Institutions. Online J Distance Learn Adm. 2011;14(1):110–8.

Kursumovic E, Lennane S, Cook TM. Deaths in healthcare workers due to COVID-19: the need for robust data and analysis. Anaesthesia. 2020;75(8):989–92.

Malekshahi Beiranvand F, Hatami Varzaneh A. Health care workers challenges during coronavirus outbreak: the qualitative study. J Res Behav Sci. 2020;18(2):180–90.

Boursicot K, Kemp S, Ong TH, Wijaya L, Goh SH, Freeman K, Curran I. Conducting a high-stakes OSCE in a COVID-19 environment. MedEdPublish. 2020;9:285–89.

Atwa H, Shehata MH, Al-Ansari A, Kumar A, Jaradat A, Ahmed J, Deifalla A, Online. Face-to-Face, or blended learning? Faculty and Medical Students’ perceptions during the COVID-19 pandemic: a mixed-method study. Front Med. 2022;9:791352.

Chan MMK, Yu DS, Lam VS, Wong JY. Online clinical training in the COVID-19 pandemic. Clin Teach. 2020;17(4):445–6.

Toulabi T, Yarahmadi S. Conducting a clinical competency test for nursing students in a virtual method during the Covid-19 pandemic: a case study. J Nurs Educ. 2021;9(5):33–42.

Meskell P, Burke E, Kropmans TJB, Byrne E, Setyonugroho W, Kennedy KM. Back to the future: an online OSCE Management Information System for nursing OSCEs. Nurse Educ Today. 2015;35(11):1091–6.

Lichtenberg PA. (2010). Handbook of Assessment in Clinical Gerontology, 2nd Ed. Academic Press, https://doi.org/10.1016/B978-0-12-374961-1.10030-2

Gholami Booreng F, Mahram B, Kareshki H. Construction and validation of a scale of research anxiety for students. IJPCP. 2017;23(1):78–93.

Esmaili M. A survey of the influence of Murita therapy on reducing the rate of anxiety in clients of counseling centers. Res Clin Psychol Couns. 2011;1(1):15–30.

Farajpour A, Amini M, Pishbin E, Arshadi H, Sanjarmusavi N, Yousefi J, Sarafrazyazdi M. Teachers’ and students’ satisfaction with DOPS Examination in Islamic Azad University of Mashhad, a study in Year 2012. Iran J Med Educ. 2014;14(2):165–73.

StraussAC, Corbin JM. Basics of qualitative research: grounded theory procedures and technique. 2nd ed. London: Sage, Newbury Park; 1998.

Dickens L, Watkins K. Action research: rethinking Lewin. Manage Learn. 1999;30(2):127–40.

Rezaeerad M, Nadri Kh, Mohammadi Etergoleh R. The effect of ADDIE (analysis, design, development, implementation, evaluation) designing method with emphasizing on mobile learning on students’ self-conception, development motivation and academic development in English course. Educational Adm Res Q. 2013;4(15):15–32.

Ben-David MF. AMEE Guide 18: standard setting in student assessment. Med Teach. 2000;22(2):120–30.

McKinley DW, Norcini JJ. How to set standards on performance-based examinations: AMEE Guide 85. Med Teach. 2014;36(2):97–110.

Fung JTC, Zhang W, Yeung MN, Pang MTH, Lam VSF, Chan BKY, Wong JYH. Evaluation of students perceived clinical competence and learning needs following an online virtual simulation education programmed with debriefing during the COVID-19 pandemic. Nurs Open. 2021;8(6):3045–54.

Luke S, Petitt E, Tombrella J, McGoff E. Virtual evaluation of clinical competence in nurse practitioner students. Med Sci Educ. 2021;31:1267–71.

Beiranvand SH, Hosseinabadi R, Ghasemi F, Anbari KH. An Assessment of nursing and Midwifery Student Veiwwpoin, Performance, and feedback with an objective structured clinical examination. J Nurs Educ. 2017;6(1):63–7.

Sheikh Abumasoudi R, Moghimian M, Hashemi M, Kashani F, Karimi T, Atashi V. Comparison of the Effect of Objective Structured Clinical evaluation (OSCE) with Direct and Indirect Supervision on nursing student’s test anxiety. J Nurs Educ. 2015;4(2):1–8.

Zahran EM, Taha EE. Students’ feedback on Objective Structured Clinical examinations (OSCEs) experience in emergency nursing. J High Inst Public Health. 2009;39(2):370–87.

Na A-G. Assessment of Students’ knowledge, clinical performance and satisfaction with objective structured clinical exam. Med J Cairo Univ. 2009;77(4):287–93.

Adib-Hajbaghery M, Yazdani M. Effects of OSCE on learning, satisfaction and test anxiety of nursing students: a review study. Iran J Med Educ. 2018;18:70–83.

Purwanti LE, Sukartini T, Kurniawati ND, Nursalam N, Susilowati T. Virtual Simulation in clinical nursing education to improve knowledge and clinical skills: Literature Review. Open Access Maced J Med Sci. 2022;10(F):396–404.

Download references

Acknowledgements

We want to thank the Research and Technology deputy of Smart University of Medical Sciences, Tehran, Iran, the faculty members, staff, and officials of the School of Nursing and Midwifery, Lorestan University of Medical Sciences, Khorramabad, Iran, and all individuals who participated in this study.

All steps of the study, including study design and data collection, analysis, interpretation, and manuscript drafting, were supported by the Deputy of Research of Smart University of Medical Sciences.

Author information

Authors and affiliations.

Department of E-Learning in Medical Education, Center of Excellence for E-learning in Medical Education, School of Medicine, Tehran University of Medical Sciences, Tehran, Iran

Rita Mojtahedzadeh & Aeen Mohammadi

Department of Medical Education, Smart University of Medical Sciences, Tehran, Iran

Tahereh Toulabi

Cardiovascular Research Center, School of Nursing and Midwifery, Lorestan University of Medical Sciences, Khorramabad, Iran

You can also search for this author in PubMed   Google Scholar

Contributions

RM. Participating in study design, accrual of study participants, review of the manuscript, and critical revisions for important intellectual content. TT : The investigator; participated in study design, data collection, accrual of study participants, and writing and reviewing the manuscript. AM: Participating in study design, data analysis, accrual of study participants, and reviewing the manuscript. All authors read and approved the final version of the manuscript.

Corresponding author

Correspondence to Tahereh Toulabi .

Ethics declarations

Ethics approval and consent to participate.

This action research was conducted following the participatory method. All methods were performed according to the relevant guidelines and regulations in the Declaration of Helsinki (ethics approval and consent to participate). The study’s aims and procedures were explained to all participants, and necessary assurance was given to them for the anonymity and confidentiality of their information. The results were continuously provided as feedback to the participants. Informed consent (explaining the goals and methods of the study) was obtained from participants. The Smart University of Medical Sciences Ethics Committee approved the study protocol (IR.VUMS.REC.1400.011).

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Mojtahedzadeh, R., Toulabi, T. & Mohammadi, A. The design, implementation, and evaluation of a blended (in-person and virtual) Clinical Competency Examination for final-year nursing students. BMC Med Educ 24 , 936 (2024). https://doi.org/10.1186/s12909-024-05935-9

Download citation

Received : 21 July 2023

Accepted : 20 August 2024

Published : 28 August 2024

DOI : https://doi.org/10.1186/s12909-024-05935-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Clinical Competency Examination (CCE)
• Objective Structural Clinical Examination (OSCE)
• Blended method
• Satisfaction

BMC Medical Education

ISSN: 1472-6920

• General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

How to choose your study design

Affiliation.

• 1 Department of Medicine, Sydney Medical School, Faculty of Medicine and Health, University of Sydney, Sydney, New South Wales, Australia.
• PMID: 32479703
• DOI: 10.1111/jpc.14929

Research designs are broadly divided into observational studies (i.e. cross-sectional; case-control and cohort studies) and experimental studies (randomised control trials, RCTs). Each design has a specific role, and each has both advantages and disadvantages. Moreover, while the typical RCT is a parallel group design, there are now many variants to consider. It is important that both researchers and paediatricians are aware of the role of each study design, their respective pros and cons, and the inherent risk of bias with each design. While there are numerous quantitative study designs available to researchers, the final choice is dictated by two key factors. First, by the specific research question. That is, if the question is one of 'prevalence' (disease burden) then the ideal is a cross-sectional study; if it is a question of 'harm' - a case-control study; prognosis - a cohort and therapy - a RCT. Second, by what resources are available to you. This includes budget, time, feasibility re-patient numbers and research expertise. All these factors will severely limit the choice. While paediatricians would like to see more RCTs, these require a huge amount of resources, and in many situations will be unethical (e.g. potentially harmful intervention) or impractical (e.g. rare diseases). This paper gives a brief overview of the common study types, and for those embarking on such studies you will need far more comprehensive, detailed sources of information.

Keywords: experimental studies; observational studies; research method.

© 2020 Paediatrics and Child Health Division (The Royal Australasian College of Physicians).

PubMed Disclaimer

Similar articles

• Observational Studies. Hess DR. Hess DR. Respir Care. 2023 Nov;68(11):1585-1597. doi: 10.4187/respcare.11170. Epub 2023 Jun 20. Respir Care. 2023. PMID: 37339891
• Observational designs in clinical multiple sclerosis research: Particulars, practices and potentialities. Jongen PJ. Jongen PJ. Mult Scler Relat Disord. 2019 Oct;35:142-149. doi: 10.1016/j.msard.2019.07.006. Epub 2019 Jul 20. Mult Scler Relat Disord. 2019. PMID: 31394404 Review.
• Study designs in clinical research. Noordzij M, Dekker FW, Zoccali C, Jager KJ. Noordzij M, et al. Nephron Clin Pract. 2009;113(3):c218-21. doi: 10.1159/000235610. Epub 2009 Aug 18. Nephron Clin Pract. 2009. PMID: 19690439 Review.
• Study Types in Orthopaedics Research: Is My Study Design Appropriate for the Research Question? Zaniletti I, Devick KL, Larson DR, Lewallen DG, Berry DJ, Maradit Kremers H. Zaniletti I, et al. J Arthroplasty. 2022 Oct;37(10):1939-1944. doi: 10.1016/j.arth.2022.05.028. Epub 2022 Sep 6. J Arthroplasty. 2022. PMID: 36162926 Free PMC article.
• Design choices for observational studies of the effect of exposure on disease incidence. Gail MH, Altman DG, Cadarette SM, Collins G, Evans SJ, Sekula P, Williamson E, Woodward M. Gail MH, et al. BMJ Open. 2019 Dec 9;9(12):e031031. doi: 10.1136/bmjopen-2019-031031. BMJ Open. 2019. PMID: 31822541 Free PMC article.
• Effects of Electronic Serious Games on Older Adults With Alzheimer's Disease and Mild Cognitive Impairment: Systematic Review With Meta-Analysis of Randomized Controlled Trials. Zuo X, Tang Y, Chen Y, Zhou Z. Zuo X, et al. JMIR Serious Games. 2024 Jul 31;12:e55785. doi: 10.2196/55785. JMIR Serious Games. 2024. PMID: 39083796 Free PMC article. Review.
• Nurses' Adherence to the Portuguese Standard to Prevent Catheter-Associated Urinary Tract Infections (CAUTIs): An Observational Study. Paiva-Santos F, Santos-Costa P, Bastos C, Graveto J. Paiva-Santos F, et al. Nurs Rep. 2023 Oct 10;13(4):1432-1441. doi: 10.3390/nursrep13040120. Nurs Rep. 2023. PMID: 37873827 Free PMC article.
• Effects of regional anaesthesia on mortality in patients undergoing lower extremity amputation: A retrospective pooled analysis. Quak SM, Pillay N, Wong SN, Karthekeyan RB, Chan DXH, Liu CWY. Quak SM, et al. Indian J Anaesth. 2022 Jun;66(6):419-430. doi: 10.4103/ija.ija_917_21. Epub 2022 Jun 21. Indian J Anaesth. 2022. PMID: 35903599 Free PMC article.
• Peat J, Mellis CM, Williams K, Xuan W. Health Science Research: A Handbook of Quantitative Methods Chapter 2, Planning the Study. Sydney: Allen & Unwin; 2001.
• Guyatt G, Rennie D, Meade MO, Cook DJ. Users Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice, 3rd edn; Chapter 14, Harm (observational studies). New York, NY: McGraw-Hill; 2015.
• Centre for Evidence Based Medicine. Oxford EBM ‘Critical Appraisal tools’. Oxford University, UK. Available from: cebm.net [Accessed March 2020].
• Kahlert J, Bjerge Gribsholt S, Gammelager H, Dekkers OMet al. Control of confounding in the analysis phase - An overview for clinicians. Clin. Epidemiol. 2017; 9: 195-204.
• Sedgwick P. Cross sectional studies: Advantages and disadvantages. BMJ 2014; 348: g2276.
• Search in MeSH

LinkOut - more resources

Full text sources.

• Ovid Technologies, Inc.

Miscellaneous

• NCI CPTAC Assay Portal

• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 27 August 2024

Charting sustainable urban development through a systematic review of SDG11 research

• Abdulaziz I. Almulhim   ORCID: orcid.org/0000-0002-5384-7219 1 ,
• Ayyoob Sharifi   ORCID: orcid.org/0000-0002-8983-8613 2 ,
• Yusuf A. Aina   ORCID: orcid.org/0000-0002-0763-9865 3 ,
• Shakil Ahmad 4 ,
• Luca Mora 5 , 6 ,
• Walter Leal Filho 7 , 8 &
• Ismaila Rimi Abubakar   ORCID: orcid.org/0000-0002-7994-2302 9  

Nature Cities ( 2024 ) Cite this article

936 Accesses

1 Altmetric

Metrics details

• Environmental studies
• Social policy

The Sustainable Development Goal (SDG) 11 underscores the imperative of creating inclusive, safe, resilient and sustainable cities and communities by 2030. Here we employ bibliometric techniques to assess the evolving landscape of SDG11 research. Using a comprehensive dataset of over 21,000 scholarly publications, we investigate publication trends, thematic focus areas, authorship patterns, keyword co-occurrences and citation networks related to SDG11 research. The results reveal a consistent increase in research output, reflecting the growing global interest in urban sustainability studies. We identify influential authors, organizations and countries shaping the research landscape, highlighting existing global collaborative networks and emerging research hubs. Core thematic areas emphasize critical topics and interdisciplinary connections. Citation networks underscore the impacts of disseminating research outputs, including seminal works. This study offers insights for policymakers, academics and practitioners to align their collective efforts toward sustainable, inclusive and climate-resilient urban development. Moreover, it advances SDG11 by noting opportunities for further research, knowledge dissemination and international collaboration.

Similar content being viewed by others

A big data approach to assess progress towards Sustainable Development Goals for cities of varying sizes

Examining the unsustainable relationship between SDG performance, ecological footprint and international spillovers

Sustainable Development Goals (SDGs): Are we successful in turning trade-offs into synergies?

The growing interest in sustainable urban development is driven by challenges posed by urbanization, socioeconomic activities and environmental issues 1 . Urban areas contribute 80% of the world’s gross domestic product 2 , but also account for around 75% of global resource consumption, 65% of energy use and over 70% of carbon emissions 3 . The ecological footprint of urban environments, which measures the resources required to sustain socioeconomic activities, has been increasing 4 , 5 , and the global urban extent is projected to double by 2030 6 . Similarly, the global urban population is projected to reach 68% by 2050 7 , which could surpass the capacity of most urban areas 8 . Africa and Asia will host most of the future urban populations despite housing and infrastructure inadequacies 7 . Rapid urbanization, poverty and climate change (CC) further intensify the vulnerability of urban dwellers 9 .

Sustainable urban development aims to balance economic production, environmental protection and social inclusiveness. It emerged as a response to the critique of modernist views that prioritized physical appearance and order in cities over context, equity and inclusion 6 . Due to the limited progress in achieving the Millennium Development Goals, the Sustainable Development Goals (SDGs) were established in 2015 to ensure that no country is left behind in achieving sustainable development by 2030 10 . Many of the SDGs are closely related to urban settings, where sustainability challenges are complex and interwoven 11 . SDG11 specifically focuses on urban challenges and aims to make ‘cities and human settlements inclusive, safe resilient and sustainable’ by reducing the negative effects of urban development while improving socioeconomic development 10 .

The importance of SDG11 stems from the principles of inclusive, safe and resilient city. An inclusive city is characterized by the idea that all individuals, irrespective of their economic status, gender, race, ethnicity or religion, have the ability and empowerment to actively engage in the social, economic and political opportunities available within urban environments 6 . It seeks to address environmental racism and promote inclusive and fair urban development through social justice and equitable distribution of environmental benefits and burdens. In such a city, everyone is afforded equal access and participation in the diverse aspects that cities provide. On the other hand, a safe city refers to a city that possesses the capacity to provide protection and security against potential dangers, harm or risks, while a resilient city denotes a city’s ability to recover and restore its fundamental functions and structures following natural disasters and crises caused by human activities 6 , 8 . SDG11 is significant because it aims to ensure that cities develop sustainably.

However, SDG11 has been criticized for its limited emphasis on urban inequalities, decentralization and funding for local authorities 6 . Other challenges include localizing the universal indicators 12 , governance issues 13 , data accessibility and comparability 14 and smart city development 12 , 15 . Nevertheless, SDG11 serves as a platform for directing and monitoring urban development, fostering socioeconomic development and ensuring equity, inclusion and environmental protection 16 . Therefore, it is crucial to assess the literature on progress toward SDG11 targets 10 , especially at the halfway point to the target year, to inform interventions necessary for their achievement 17 .

While SDG11 has attracted significant global research attention 18 , comprehensive reviews of SDG11 literature are limited. Existing studies have primarily focused on assessing all the SDGs 19 , 20 , which obscures specific challenges and makes it difficult to track progress or design targeted interventions for individual goals. Recent work has highlighted the insufficient achievement of the SDGs and the need for transformative governance and participatory approaches 21 . Other studies have underscored the gap between research and policies, the underutilization of cities as pivotal arenas for achieving SDGs 22 and the lack of indicators to measure progress toward implementing SDGs 15 . Some studies have assessed SDGs’ implementation in specific region 17 , their impacts on addressing risks 23 and crises 1 , and their implications for health and well-being 24 , environmental research 25 and private sector involvement 26 . Most of the SDG research emanates from developed countries, showing a gap in the coverage of developing countries 27 . The few SDG11 studies in the Global South have narrow focus. While one paper investigated the impact of SDG11 on forest-based livelihoods 28 , another study researched the challenges of SDG11 implementation using a single-country experience 6 . Therefore, an in-depth and broad review of SDG11 literature is necessary to bridge this knowledge gap and identify key challenges and opportunities as well as potential pathways for achieving the targets set in SDG11.

Therefore, this research aims to assess the SDG11 research trends and themes using a bibliometric technique. It is the first global and comprehensive scientometric study on the SDG11 domain. By focusing on research conducted since the formulation of the SDGs, the study addresses the following research questions: (1) what are the global trends in SDG11 research? (2) How has the thematic focus of SDG11 research evolved over time? (3) What are the challenges and priority areas for SDG11 research? The contributions of the study to theory and practice are to:

Identify significant thematic areas and trends in SDG11 research since the promulgation of the SDGs, which can inform researchers, policymakers and practitioners about the current state of knowledge within the field and highlight priority areas for SDG11 research.

Map research clusters, knowledge sharing and collaboration patterns, thereby providing insights into the dynamics of research networks and facilitating the formulation of strategies to foster research excellence, interdisciplinary and international collaborations and the effective allocating of research resources.

Underscore the knowledge gaps, emerging topics and challenges within SDG11 research, offering evidence-based insights to align urban development initiatives with SDG11 research frontiers, enhance the efficacy of interventions and contribute to the development of inclusive, safe, resilient and sustainable cities.

SDG11 research trends

Research on SDG11 has significantly grown in terms of annual publications and citations since 2016, indicating a rising interest in this field (Fig. 1 ). The number of publications has increased by 1.3-fold, and this upward trajectory is expected to continue. Notable emerging research areas include the institutionalization of SDGs within local and global settings 18 and the impact of smart cities on advancing the SDGs 12 , 15 . Previously, studies on the epistemology and challenges of urban population growth were prevalent 29 . However, SDG11 research has now evolved into multidisciplinary fields, driven by heightened attention to urban challenges such as CC, urbanization and population growth.

A total of 21,153 articles were published, receiving 229,182 citations. The number of publications rose from 9,238 in period 1 (2016–2019) to 11,915 in period 2 (2020–2022).

Source data

The increasing trend in SDG11 publications can be attributed to several factors, including the desire to improve institutional rankings, a supportive research environment, investments and endowments, faculty promotion requirements and advancements in information and communication technology. There are also socioeconomic factors, such as increasing urbanization rates and gross domestic product, urban expansion and transformation, a deeper understanding of urban dynamics and challenges. Additionally, the policy environments in different countries can influence academic interests and research in urban studies, shaping research priorities and collaborations. Other contributing factors include research challenges faced by low-income countries and research support by governments, the private sector, international development agencies and scholars, all focusing on sustainable urban development.

SDG11 research is further propelled by recent international summits and collaborations that highlight the urgency of protecting the ecosystem and ensuring human safety 1 . Since 2015, CC issues have received greater attention due to key factors. The adoption of the Paris Agreement raised awareness and urgency for action on CC, resulting in a greater focus on related issues in various sectors, including urban planning and policy 13 . Scientific consensus on CC impacts and the role of human activities has also strengthened over the years, with Intergovernmental Panel on Climate Change assessments emphasizing the significance of cities in addressing CC 23 . As a result, CC considerations are increasingly integrated into research, policy and planning processes.

Urban planning and development strategies have prioritized climate mitigation and adaptation measures, such as reducing greenhouse gas emissions, promoting renewable energy, enhancing resilience to extreme weather events and incorporating green infrastructure. The focus on CC has accelerated the transition toward low-carbon and resilient cities, with efforts directed toward sustainable transportation, energy-efficient buildings, green spaces and climate-responsive infrastructure 6 . Collaboration and international cooperation are essential in addressing climate change, with cities and countries sharing best practices, knowledge and resources to develop and implement climate action plans 24 . Initiatives such as the C40 Cities Climate Leadership Group facilitate knowledge exchange and collective action among cities 30 . The increased attention to CC signifies a shift toward more sustainable and resilient urban development, emphasizing the need for proactive measures to mitigate greenhouse gas emissions, adapt to climate risks and promote equitable and sustainable urban environments.

Thematic focus of SDG11 research

There is an imbalance in the attention given to research themes within SDG11 as revealed by co-occurrence map (Supplementary Fig. 1 ). The dominant themes are affordable housing (SDG11.1), urban transport (SDG11.2), policy and governance (SDG11.3) and access to public spaces (SDG11.7). Housing affordability issues have consistently remained a focal point in SDG11 research, with urban studies, policy development and community-driven efforts for finding solutions to these complex challenges 30 , 31 . These issues were highlighted in Habitat I (Vancouver, 1976), which emphasized the importance of shifting governance and planning paradigms to develop policies and strategies to address rapid urbanization challenges, including shelter shortages and urban inequalities, and promote affordable housing options 30 , 32 . Habitat I has laid the foundation for subsequent global efforts and policy frameworks, such as Habitat II (Istanbul, 1996) and the New Urban Agenda, which continue to prioritize housing as a pivotal component of sustainable urban development. The persistent focus on affordable housing shows that cities still face many challenges in providing adequate housing for all 30 .

Urban policy and governance are other significant terms, indicating scholarly focus on strategies for promoting inclusive and sustainable urban development, enhancing participatory, integrated and sustainable urban planning and management. However, many cities lack the capacity to address urban inequalities, provide adequate housing 31 , public spaces and other urban services, which disproportionately affect women and racial minorities 30 . Moreover, urban redevelopment practices that lead to gentrification exacerbate existing inequalities 32 . Governance-based approaches seek to improve collaboration between public agencies and civil society to prioritize the implementation of urban planning strategies that enhance livability standards while addressing challenges such as CC and sustainability 30 .

Urban transport, which is related to SDG11.2 aiming to ensure safe, affordable, accessible and sustainable transport systems for all, has emerged as a key research theme. Important issues related to mobility, transportation and urban form include increased automobile dependence amid growing urbanization and suburbanization, challenges faced by public transit systems, growing awareness of environmental concerns, shift toward sustainable and multimodal transportation, transit-oriented development, integration of technology in transportation systems and the relationship between transportation and urban densification, compact development, CC adaptation and resilience, equity and social inclusion, and shifts in policy and governance approaches 1 , 6 , 11 . This theme also emphasizes the importance of walkability, public transit infrastructure and their role in enhancing transportation accessibility and influencing mode choice 33 . The transportation cluster also suggests that improving accessibility through urban form and built environment interventions can impact the travel behavior of urban residents and offer cobenefits for human health and environmental sustainability 24 . Incorporating such cobenefits in SDG11.2 could provide more incentives for access to safe efficient, equitable and sustainable transport infrastructure and systems in cities.

The implications of urbanization and land-use changes for sustainability, resilience and CC adaptation and mitigation in cities are also major themes. SDG11.6 aims to reduce the environmental impacts of cities, particularly in relation to air pollution and waste. The literature suggests that regulating urban growth 6 , controlling land-use changes, conserving biodiversity 27 and promoting green infrastructure are essential for achieving this target 34 . These actions, when implemented within integrated planning frameworks, can also reduce vulnerability, enhance resilience and contribute to progress in CC adaptation and mitigation, as emphasized in SDG11.5 (ref. 6 ). Such integrated frameworks should recognize the interconnections between various urban systems, including water, food, energy, waste and transportation, to promote sustainable and resilient urban development 35 . Cities are adopting strategies to reduce their carbon footprint, enhance energy efficiency and prepare for climate risks.

Smart cities and innovation enabled by information and communication technologies have increasingly been utilized to tackle urban development challenges and facilitate innovative and transformative urban governance mechanisms that contribute to the SDGs 15 . The rapid development and integration of digital technologies, such as the Internet of Things, artificial intelligence, big data analytics and sensor networks, have opened new possibilities for improving urban services, infrastructure and quality of life 33 . Smart cities leverage these technologies to enhance efficiency, connectivity and sustainability. The interest in smart cities stems from the recognition that technology can play a transformative role in addressing urban challenges, improving quality of life, promoting sustainability and fostering economic growth 12 , 36 . However, it is important to ensure that smart city initiatives are inclusive, equitable and responsive to the needs and aspirations of all residents.

Comparing the co-occurrence maps of period 1 and period 2 reveals limited changes in key thematic areas, despite the emergence of the coronavirus disease 2019 (COVID-19) pandemic during period 2 (Fig. 2 ). The key thematic areas in period 2, including urban governance and policy, transportation, urban sustainability and resilience, and urbanization and urban growth, remain consistent with period 1, indicating the continued relevance of these topics in research, albeit with potential expansions. However, a closer analysis of the clusters reveals that COVID-19 has emerged as a new area of SDG11 research in period 2, as attention has shifted toward adapting to the pandemic’s detrimental effects on cities. The pandemic has triggered paradigm shifts in various SDG11 domains, including public health, remote work, digitalization, vulnerabilities, inequalities, resilience, sustainability, urban spaces, proximity-based planning approaches such as the 15-minute city and global cooperation 9 . These shifts have influenced work, health, social equity, environmental stewardship 2 and urban planning, shaping innovative approaches and priorities in the postpandemic world. Urban inequality terms, such as slums and informality, inadequate housing and poverty, are brought to the forefront by the pandemic. Controlling the pandemic and addressing the citizen demand in slums and informal settlements has received significant attention 37 , 38 , 39 , 40 . Mobility restrictions and lockdowns to curb the virus’s transmission have presented challenges for service accessibility, particularly in disadvantaged neighborhoods where vulnerable groups reside. Lastly, the connection between sustainability and resilience has strengthened in the postpandemic period. The pandemic has offered new insights into the susceptibility of cities to various stressors and highlighted the inseparable connections between urban resilience and SDG11 (ref. 28 ).

a , b , The key thematic areas in period 1 (2016–2019) ( a ) are urban governance and policy (red), transportation (blue), urban sustainability and resilience (green), and urbanization and urban growth (yellow), while period 2 (2020–2022) ( b ) primarily focuses on urban governance and policies (red), urban studies (red), transportation (blue) and urbanization (green), particularly after the pandemic.

However, three SDG11 targets are not well-represented in both periods. One such target is SDG11.4, which aims to enhance efforts in preserving and conserving natural heritage, vital for improving urban sustainability 41 . Another target, SDG11.a, which focuses on strengthening urban–rural linkages, is also not adequately reflected in Fig. 2 . The intrinsic connection between cities and their surrounding rural areas necessitates the incorporation and strengthening of ties between urban and rural regions to achieve SDG11 (ref. 6 ). Gaps related to rural–urban linkages include limited understanding of interdependencies, inadequate infrastructure and services in rural areas, weak governance and coordination mechanisms, and social and cultural disconnect 13 . These gaps hinder the development of integrated strategies, contribute to economic disparities, limit access to services, impact agricultural productivity and food security, and create environmental and social challenges. Lastly, there is a lack of research on SDG11.c, which aims to support least-developed nations in developing safe and resilient urban areas, which is not surprising as these countries are often underrepresented in urban studies research 30 .

Major contributors to SDG11 research

Various countries, institutions, journals and authors have contributed to SDG11 research between 2016 and 2022. China leads in terms of the number of publications and citations generated, followed by the United States and the United Kingdom (Supplementary Fig. 2 and Supplementary Table 1 ). Among the top 20 productive countries, 14 are from the Global North countries, with South Africa and Brazil as the sole representative of Africa and Latin America and the Caribbean, respectively (Supplementary Fig. 3 and Supplementary Table 2 ). Increasing research collaboration among the top countries (Fig. 3 ), research infrastructure and facilities, manpower and financial support significantly contribute to their high SDG11 research output.

China followed by the United States and the United Kingdom dominates SDG11 research collaborations. There are significant connections among European, North American and Asian institutions, while Africa is less connected with Asia and Latin America and the Caribbean. Freq, frequently.

A co-citation analysis (Supplementary Table 3 ) reveals that Chinese institutions, such as the Chinese Academy of Sciences, have the highest number of articles and citation counts, followed by University College London and the University of Melbourne. The leading affiliations have changed over time, highlighting the strengthening of research institutes and the correlation between research collaboration and societal impacts (Supplementary Table 4 ). In terms of influential journals for SDG11 research, ‘land’ followed by ‘cities and land use’ policy tops the list (Supplementary Tables 5 and 6 ), with a growing interest in fields related to smart and sustainable cities, transport policies, regional planning and environmentally conscious building practices (Supplementary Fig. 4 ). These journals also address multiple issues related to environmental concerns, technological advancements, economic benefits, quality of life, justice and public awareness, driving the development of smart and sustainable cities.

The 15 most published authors in both periods focused on urbanization and urban growth, and the implementation, challenges and achievements of SDG11 (Supplementary Fig 5 ). This indicates an increased recognition of the SDG11 targets and their implementation over time, with the contributions of these authors significantly increasing from 2002 to 2016. Supplementary Table 7 shows that Chinese authors dominate the SDG11 publications, which correlates with China’s lead in institutions, affiliations and collaborations related to SDG11 research. The most cited SDG11 articles are revealed in Supplementary Table 8 , while the prominent authors that influenced SDG11 research are reported in Supplementary Table 9 . The top cited papers by SDG11 research are presented in Supplementary Tables 10 and 11 .

Key facts from the bibliometric analysis

The research on SDG11 has gained significant prominence across various fields, including urban studies, environmental sciences, geography, transportation and urban governance (Supplementary Table 12 ). The increasing environmental concerns, urbanization and global economic growth have spurred academic interest in SDG11 research from disciplines such as human geography, transportation, forestry, CC and sustainability science (Supplementary Table 13 ). Key thematic areas within SDG11 research encompass urban governance, affordable housing, transportation, urban sustainability and resilience, smart cities, urbanization and urban growth, which align closely with SDG11 targets 18 , 20 , 42 , 43 . However, research focus on SDG11 has remained relatively stable, with limited attention given to urban inequalities, safeguarding cultural and natural heritage 41 and specific impacts of the COVID-19 pandemic on urban sustainability.

This study reveals a notable increase in the total SDG11 research output from 2016 to 2022, reflecting the growing emphasis on SDG11 research in recent years compared with earlier periods. China emerges as the leaders in terms of research outputs, citations, authors, institutions and collaborations, closely followed by the United States and the United Kingdom. These three countries contribute 47.71% of SDG11 research productivity within this period, which is higher than 31% reported in a previous similar study 28 .

The dominance of Global North countries in the top 20 countries with the highest number of publications and citations related to SDG11 research is expected given their strong institutional capacity, research funding, highly ranked universities and collaborations. China’s surge in publications on SDG11 can be attributed to rapid urbanization, economic growth, government support and active international collaborations 2 , 11 . Generally, the landscape of research on SDG11 demonstrates an Anglo–American hegemony, which may reinforce power asymmetries and have significant implications for sustainability and resilience 30 . It is concerning that while projections indicate that 90% of future urban population growth will occur in cities of the Global South, particularly Africa and Asia, there is limited research on urban development challenges in these regions 7 .

The debate about the politics of knowledge production in SDG11 research often revolves around the controls of knowledge production processes. Large, well-funded institutions in developed countries tend to dominate research agendas, focusing on themes and solutions relevant to their own contexts, overlooking the unique needs and challenges of the Global South, which perpetuate existing inequalities and privileging certain types of knowledge. Also, knowledge production involves recognizing and integrating diverse ways of knowing. While Western scientific paradigms have traditionally dominated SDG11 research, there is an increasing recognition of the importance of indigenous and non-Western knowledge systems. Integrating these diverse epistemologies enriches understanding and leads to more effective and culturally relevant solutions.

Additionally, SDG11 research is inherently interdisciplinary, involving fields such as urban planning, sociology, environmental science and public policy. However, interdisciplinary collaboration can be challenging due to differing terminologies, methodologies and research priorities. Navigating these differences becomes crucial in the politics of knowledge production to create cohesive and comprehensive research outputs. Finally, bridging the gap between knowledge production and its implementation faces political, economic and social barriers. Researchers and practitioners are increasingly considering how knowledge on urban sustainability can effectively influence policymaking and practice in diverse urban contexts. Mobilizing knowledge to address these barriers becomes a key consideration in the politics of knowledge production.

Challenges to achieving SDG11

There are several challenges to achieving SDG11 targets, including inadequate provision of affordable housing 31 , essential services 24 , green spaces 2 , 34 , efficient transportation 33 and conservation of cultural and natural assets 25 . Rapid urbanization 1 , 7 , CC impacts 44 , insufficient investment in public infrastructure 30 , poor governance 13 and widening livelihood, land and resources inequalities 43 further exacerbate these challenges. For example, rapid urbanization puts immense pressure on housing, infrastructure, services and resources, making it challenging to effectively manage urban growth and ensure sustainable urban development 11 . Inadequate urban planning and land-use policies lead to inefficient land utilization, urban sprawl and inadequate provision of basic services 7 , 21 . The existence of slums and informal settlements where a large portion of the urban dwellers live in substandard housing conditions without tenure security 14 and limited access to electricity, water, sanitation, education, healthcare and employment opportunities 23 , 37 , and marginalized and vulnerable populations facing social exclusion, add to the complexity.

Moreover, competing priorities and trade-offs, lack of integration among various urban sectors and agencies 35 , inadequate human, technical and material resources at local government levels 45 , and insufficient local indicators and methods for implementation and monitoring 46 often hamper the implementation of SDG11 targets. Additionally, limited awareness of SDG-related challenges for policy formulation and implementation hinders context-depended decision-making and targeted interventions 21 , 27 . Addressing social inequalities, ensuring inclusivity in urban development and synergy among multiple fields, including social, technical, environmental, policy and management are crucial for achieving SDG11 (refs. 14 , 26 , 46 ). A valuable lesson can be learned from the success of the framework for assessing the implementation of SDG11 targets at the local level in Japan 42 .

Conclusions

This study aims to enhance our understanding of urban sustainability and provide insights for future research, policies and actions needed to achieve SDG11 targets. By conducting a comprehensive bibliometric assessment of over 21,000 publications from 2016 to 2022, it significantly contributes to the existing body of knowledge, highlighting trends, thematic areas and knowledge gaps related to SDG11 research across countries, institutions, authors and journals. SDG11 research has evolved into a multidisciplinary field, encompassing diverse themes, such as transportation, housing, urban sustainability, smart cities, urbanization and urban governance and policy. However, there is a need to address the gaps in research on urban safety and inclusion, which are critical dimensions often overlooked in favor of environmental and economic aspects of sustainability. This imbalance in research thematic areas risks perpetuation of already existing disparities within SDG11 research and its goals.

China, the United States and the United Kingdom emerge as the top contributors to SDG11 research and collaboration. To foster more SDG11 research in low-income economies, it is essential to provide increased funding support, capacity building and training for scholars, promote collaboration and knowledge exchange, and improve research infrastructure and data collection. Despite global challenges such as armed conflicts, CC and the COVID-19 pandemic, progress toward achieving the SDGs will become apparent by 2030. However, there are still opportunities for further research, knowledge dissemination and international collaboration toward developing safe, sustainable and inclusive urban development. The following are priority areas for SDG11 research:

Urban policy and governance: reforms should focus on providing equitable access to basic services such as water, sanitation, electricity, healthcare and education; upgrading and formalizing informal settlements; and improving living conditions of over one billion people residing in slums 37 . Participatory governance, community engagement and empowerment can enhance social inclusion by considering the voices and needs of marginalized groups 13 , 23 . Urban policy should also prioritize preserving historic and natural resources, protecting vulnerable areas and implementing sustainable urban design principles 47 . Future studies can help understand the dynamics, challenges and opportunities and monitor progress toward SDG11 targets 15 .

Localizing SDG11 targets: spatial planning and land-use strategies should consider the needs of diverse urban populations, promote inclusive zoning and engage local communities and stakeholders in decision-making processes, crucial for fostering ownership, empowerment and social cohesion, leading to more sustainable and inclusive urban development 3 . However, enhancing the capacity for localizing SDG11 targets requires building the knowledge and skills of local governments, policymakers and practitioners. Capacity-building initiatives, such as training programs, workshops and knowledge exchange, can promote interdisciplinary understanding and sharing of best practices.

Concerted and collaborative efforts: the international community, academics, policymakers and stakeholders can work together to create inclusive, safe, resilient and sustainable communities. Collaborative efforts can facilitate a comprehensive understanding of urban challenges and potential solutions by integrating diverse perspectives, data and methodologies. Disseminating research findings contributes to evidence-based policy development and informed decision-making, enabling the learning of lessons and replication of successful interventions.

Breaking down silos: integrated and cross-sectoral approaches help narrow the gaps between sectors, local governments, policymakers and stakeholders, leveraging local resources and capacities while fostering communication, knowledge sharing and collaboration 31 . Cross-sectoral working groups, joint planning processes and integrated policy frameworks promote holistic and coordinated decision-making among various sectors, including urban planning, housing, transportation, health, education, environment and social welfare 47 .

Digitalization and smart city development: maximizing the benefits of digitalization and smart city solutions requires addressing challenges such as bridging digital divides and ensuring data access, privacy and security. Prioritizing citizen-centric approaches and public accessibility to technology 36 are essential for leveraging expertise and resources 15 . Interoperability, scalability, data-driven decision-making and inclusivity contribute to evidence-based planning and equitable access to smart city technologies 12 , 48 , 49 , 50 , 51 .

This study comprehensively assessed SDG11 research, emphasizing significant thematic areas, trends, challenges and suggestions for prioritizing SDG11, including effective urban policy and governance, localizing SDG11 targets, concerted and collaborative efforts, and digitalization and smart city development. To broaden the scope of SDG11 research, future bibliometric reviews should encompass non-Web of Science databases and gray literature, including publications from government and nongovernmental agencies. Despite its limitations, this study’s findings provide valuable references for further research on SDG11.

The present study utilized a bibliometric technique to analyze academic publication on SDG11, tracing the research trend, the evolving key themes and identifying contributing authors, institutions and countries. Bibliometrics is a quantitative technique that allows for the analysis of trends in scholarly publications, such as research articles, conference papers and books, and visualizes scholarly publication patterns 52 . This technique is instrumental in analyzing extensive literature sets by relying on statistical observations and text-mining capabilities, which qualitative review methods such as systematic reviews cannot accomplish 53 . Additionally, it presents a scientific landscape of authors, countries, organizations and collaborations that contribute to worldwide scientific literature.

Bibliometric analysis requires interpretation, introducing an element of subjectivity 54 . Therefore, a sensemaking approach was adopted to transition from describing the bibliometric results to interpreting them. Sensemaking helps derive insightful information from bibliometric analysis and can be integrated into systematic literature reviews 55 , 56 . It applies to various international indexing, abstracting and citation databases, such as Scopus, Web of Science, Dimensions, PubMed and Education Resources Information Center, which cover journals, books, reviews and conference proceedings from around the world and different regions. For this study, Web of Science was chosen as the database to obtain bibliographic data due to its wide range of topics in various fields of study such as natural sciences, health sciences, engineering, social science, computer science and materials sciences. It is one of the world’s largest peer-reviewed scientific literature databases, with 87 million indexed items.

Specialized bibliometrics software were employed, including VOSviewer (version 1.6.19) 52 , Biblioshiny (version 4.1.3) 55 and BibExcel (version 2017) 57 . VOSviewer, known for its user-friendly interface, was used to understand the thematic focus and evolution of research on SDG11. It generates networks of nodes and links, with node size representing the frequency of the studied item, and link width indicating the strength of connections between items. Clusters of intricately linked nodes are shown in distinct colors. The thematic focus was examined for two periods: period 1 (2016–2019) and period 2 (2020–2022), considering the time since the SDGs were introduced to the time of data collection in this study. Another reason for this categorization is that evidence shows that the pandemic has significantly affected progress toward achieving SDGs 58 . VOSviewer allows for various types of analysis, including term co-occurrence, co-citation, citation and bibliographic coupling 53 . A term co-occurrence analysis was used in this study to highlight key thematic areas. To ensure accuracy and avoid separate counting of synonyms, a thesaurus file was developed and added to the software before the analysis. A summary of the data, including the number of authors and journals, used in the analysis is presented in Table 1 and will be further explained below.

A comprehensive search query was formulated to retrieve relevant data on SDG11, and it was executed in the title, abstract and keywords fields (TS) in Web of Science on 5 July 2023. The initial query shown the following box resulted in a total of 334,224 documents. Co-citation analysis was employed to identify the most influential journals contributing to SDG11 research. Two works are considered co-cited when they are both mentioned in the works cited section of a subsequent publication 59 (Zhao, 2006).

TS = ((‘city’ OR ‘cities’ OR ‘human settlement*’ OR ‘urban’ OR ‘metropoli*’ OR ‘town*’ OR ‘municipal*’ OR ‘peri-urban*’ OR ‘urban-rural’ OR ‘rural-urban’) AND (‘gentrification’ OR ‘congestion’ OR ‘transport*’ OR ‘housing’ OR ‘slum*’ OR ‘informal settlement*’ OR ‘sendai framework’ OR ‘Disaster Risk Reduction’ OR ‘disaster’ OR ‘DRR’ OR ‘smart cit*’ OR ‘resilient building*’ OR ‘sustainable building*’ OR ‘building design’ OR ‘buildings design’ OR ‘urbani?ation’ OR ‘zero energy’ OR ‘zero-energy’ OR ‘basic service*’ OR ‘governance’ OR ‘citizen participation’ OR ‘collaborative planning’ OR ‘participatory planning’ OR ‘inclusiveness’ OR ‘cultural heritage’ OR ‘natural heritage’ OR ‘UNESCO’ OR ‘ecological footprint’ OR ‘environmental footprint’ OR ‘waste’ OR ‘pollution’ OR ‘pollutant*’ OR ‘waste water’ OR wastewater* OR waste-water* OR ‘recycling’ OR ‘circular economy’ OR ‘air quality’ OR ‘green space’ OR ‘green spaces’ OR ‘nature inclusive’ OR ‘nature inclusive building’ OR ‘nature inclusive buildings’ OR ‘resilient’ OR ‘resilience’ OR ‘healthy cit*’ OR ‘sustainable’ OR ‘sustainability’ OR ‘green’ OR ‘nature*’ OR ‘Green infrastructure*’ OR ‘nature-based solution*’ OR ‘nature based solution*’ OR ‘child*’ OR ‘wom?n’ OR ‘elderl*’ OR ‘disabl*’ OR ‘disabilit*’ OR ‘disabled’)) AND PY = (2016–2022) NOT PY = (2023)

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) framework was used to report document search and filtration process. The PRISMA framework is designed to help scholars transparently report why their review study is conducted, what activities are performed and what discoveries are made, ideal for both systematic reviews and bibliometric studies 60 . PRISMA presents the four stages of the above query’s overall searching and filtration process (Fig. 4 ). The identification stage yielded 334,224 records, which were then screened to select only article-type documents ( n  = 277,165). Subsequently, documents were further screened based on language, selecting only English documents ( n  = 257,374). In the final stage, documents were screened based on specific categories closely related to cities and SDG11, resulting in a selection of six major categories: urban studies, environmental studies, geography, urban and regional planning, architecture, transportation and physical geography ( n  = 21,168). Finally, 15 duplicated documents were removed, resulting in a final dataset of 21,153 documents.

A four-phase flow diagram of the data extraction and filtration process of SDG11 literature, adapted from Priyadarshini 57 . WoS, Web of Science.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.

Data availability

The data that support the findings of this study are available as supplementary information. The steps for curating the data from the Web of Science have been provided in the text. If there is a further need, data are available on figshare at https://doi.org/10.6084/m9.figshare.26360125 . Source data are provided with this paper.

Almulhim, A. I. & Cobbinah, P. B. Can rapid urbanization be sustainable? The case of Saudi Arabian cities. Habitat Int. 139 , 102884 (2023).

Article   Google Scholar  

Huang, W. et al. Evaluating green city development in China using an integrated analytical toolbox. J. Clean. Prod. 400 , 136703 (2023).

Feng, T. & Zhou, B. Impact of urban spatial structure elements on carbon emissions efficiency in growing megacities: the case of Chengdu. Sci. Rep. 13 , 9939 (2023).

Javeed, S., Siddique, H. M. A. & Javed, F. Ecological footprint, globalization, and economic growth: evidence from Asia. Environ. Sci. Pollut. Res. 30 , 77006–77021 (2023).

Rees, W. E. Cities, energy, and the uncertain future of urban civilization. Oxf. Dev. Stud. 51 , 11–17 (2023).

Abubakar, I. R. & Aina, Y. A. The prospects and challenges of developing more inclusive, safe, resilient, and sustainable cities in Nigeria. Land Use Policy 87 , 104105 (2019).

UN-Habitat World Cities Report 2022: Envisaging the Future of Cities. (UN-Habitat, 2022).

Almulhim, A. I. & Cobbinah, P. B. Framing resilience in Saudi Arabian cities: on climate change and urban policy. Sustain. Cities Soc. 101 , 105172 (2024).

Barbier, E. B. & Burgess, J. C. Sustainability and development after COVID-19. World Dev. 135 , 105082 (2020).

The Sustainable Development Goals Report 2018 (United Nations, 2018).

Chen, M., Chen, L., Cheng, J. & Yu, J. Identifying interlinkages between urbanization and Sustainable Development Goals. Geogr. Sustain. 3 , 339–346 (2022).

Clement, J., Ruysschaert, B. & Crutzen, N. Smart city strategies—a driver for localizing sustainable development goals? Ecol. Econ. 213 , 107941 (2023).

Hansson, S., Arfvidsson, H. & Simon, D. Governance for sustainable urban development: the double function of SDG indicators. Area Dev. Policy 4 , 217–235 (2019).

Leal Filho, W. et al. Using the sustainable development goals towards a better understanding of sustainability challenges. Int. J. Sustain. Dev. World Ecol. 26 , 179–190 (2019).

Parra-Dominguez, J., Gil-Egido, A. & Rodríguez-González, S. SDGs as one of the drivers of smart city development: the indicator selection process. Smart Cities 5 , 1025–1038 (2022).

Grainger-Brown, J., Malekpour, S., Raven, R. & Taylor, E. Exploring urban transformation to inform the implementation of the Sustainable Development Goals. Cities 131 , 103928 (2022).

Xu, Z. et al. Assessing progress towards sustainable development over space and time. Nature 577 , 74–78 (2020).

Salvia, A. L., Leal Filho, W., Brandli, L. L. & Griebeler, J. S. Assessing research trends related to Sustainable Development Goals: local and global issues. J. Clean. Prod. 208 , 841–849 (2019).

Indana, F. & Pahlevi, R. W. A bibliometric approach to Sustainable Development Goals (SDGs) systematic analysis. Cogent Bus. Manag. 10 , 2224174 (2023).

Yamaguchi, N. U. et al. Sustainable development goals: a bibliometric analysis of literature reviews. Environ. Sci. Pollut. Res. 30 , 5502–5515 (2023).

Pattberg, P. & Bäckstrand, K. Enhancing the achievement of the SDGs: lessons learned at the half-way point of the 2030 Agenda. Int. Environ. Agree. 23 , 107–114 (2023).

Keith, M. et al. A new urban narrative for sustainable development. Nat. Sustain. 6 , 115–117 (2023).

Pedersen, A. B. et al. SDGs at the halfway point: how the 17 global goals address risks and wicked problems. Ambio 52 , 679–682 (2023).

Sweileh, W. M. Bibliometric analysis of scientific publications on ‘sustainable development goals’ with emphasis on ‘good health and well-being’ goal (2015–2019). Global. Health 16 , 68 (2020).

Mihelcic, J. R. et al. Environmental research addressing Sustainable Development Goals. Environ. Sci. Technol. 57 , 3457–3460 (2023).

Palau-Pinyana, E., Llach, J. & Bagur-Femenías, L. Mapping enablers for SDG implementation in the private sector: a systematic literature review and research agenda. Manag. Rev. Q. 26 , 1–30 (2023).

Google Scholar  

Mishra, M. et al. A bibliometric analysis of sustainable development goals (SDGs): a review of progress, challenges, and opportunities. Environ. Dev. Sustain. 26 , 11101–11143 (2023).

Devisscher, T. et al. in Sustainable Development Goals: Their Impacts on Forests and People (eds. Katila, P. et al.) (Cambridge Univ. Press, 2019).

Brenner, N. & Theodore, N. Cities and the geographies of ‘actually existing neoliberalism’. Antipode 34 , 349–379 (2002).

Sharifi, A., Khavarian-Garmsir, A. R., Allam, Z. & Asadzadeh, A. Progress and prospects in planning: a bibliometric review of literature in urban studies and regional and urban planning, 1956–2022. Prog. Plann. 173 , 100740 (2023).

Olanrewaju, A., Tan, S. Y. & Abdul-Aziz, A.-R. Housing providers’ insights on the benefits of sustainable, affordable housing. Sustain. Dev. 26 , 847–858 (2018).

Rice, J. L., Cohen, D. A., Long, J. & Jurjevich, J. R. Contradictions of the climate-friendly city: new perspectives on eco-gentrification and housing justice. Int. J. Urban Reg. Res. 44 , 145–165 (2020).

Park, S., Choi, K. & Lee, J. S. To walk or not to walk: testing the effect of path walkability on transit users’ access mode choices to the station. Int. J. Sustain. Transp. 9 , 529–541 (2015).

Jayasooriya, V. M., Ng, A. W. M., Muthukumaran, S. & Perera, B. J. C. Green infrastructure practices for the improvement of urban air quality. Urban For. Urban Green. 21 , 34–47 (2017).

Hachaichi, M. & Egieya, J. Water–food–energy nexus in global cities: addressing complex urban interdependencies. Water Resour. Manag. 37 , 1811–1825 (2023).

Mora, L., Deakin, M. & Reid, A. Combining co-citation clustering and text-based analysis to reveal the main development paths of smart cities. Technol. Forecast. Soc. Change 142 , 56–69 (2019).

McFarlane, C. Rethinking informality: politics, crisis, and the city. Plan. Theory Pract. 13 , 89–108 (2012).

Harvey, D. From managerialism to entrepreneurialism: the transformation in urban governance in late capitalism. Geogr. Ann. B 71 , 3–17 (1989).

Peck, J. Political economies of scale: fast policy, interscalar relations, and neoliberal workfare. Econ. Geogr. 78 , 331–360 (2002).

Cervero, R. & Landis, J. Twenty years of the Bay Area rapid transit system: land use and development impacts. Transport. Res. A 31 , 309–333 (1997).

Guzman, P. C., Roders, A. R. P. & Colenbrander, B. J. F. Measuring links between cultural heritage management and sustainable urban development: an overview of global monitoring tools. Cities 60 , 192–201 (2017).

Yamasaki, K. & Yamada, T. A framework to assess the local implementation of Sustainable Development Goal 11. Sustain. Cities Soc. 84 , 104002 (2022).

van Zanten, J. A. & van Tulder, R. Towards nexus-based governance: defining interactions between economic activities and Sustainable Development Goals (SDGs). Int. J. Sustain. Dev. World Ecol. 28 , 210–226 (2021).

Londono-Pineda, A. A. & Cano, J. A. Assessments under the United Nations sustainable development goals: a bibliometric analysis. Environ. Clim. Technol. 26 , 166–181 (2022).

Biggeri, M. A. ‘Decade for Action’ on SDG localization. J. Hum. Dev. Capabil. 22 , 706–712 (2021).

Benedek, J., Ivan, K., Török, I., Temerdek, A. & Holobâcă, I. H. Indicator-based assessment of local and regional progress toward the Sustainable Development Goals (SDGs): an integrated approach from Romania. Sustain. Dev. 29 , 860–875 (2021).

Abubakar, I. R. & Alshammari, M. S. Urban planning schemes for developing low-carbon cities in the Gulf Cooperation Council region. Habitat Int. 138 , 102881 (2023).

Batty, M. Smart cities, big data. Environ. Plann. B 39 , 191–193 (2012).

Smith, N. Toward a theory of gentrification, a back to the city movement by capital, not people. J. Am. Plann. Assoc. 45 , 538–548 (1979).

Jacobs, J. The Death and Life of Great American Cities (Random House, 1961).

Smith, N. The New Urban Frontier: Gentrification and the Revanchist City (Routledge, 1996).

Donthu, N., Kumar, S., Mukherjee, D., Pandey, N. & Lim, W. M. How to conduct a bibliometric analysis: an overview and guidelines. J. Bus. Res. 133 , 285–296 (2021).

van Eck, N. J. & Waltman, L. Software survey: VOSviewer, a computer program for bibliometric mapping. Scientometrics 84 , 523–538 (2010).

Hajek, P., Youssef, A. & Hajkova, V. Recent developments in smart city assessment: a bibliometric and content analysis-based literature review. Cities 126 , 103709 (2022).

Lim, W. M. & Kumar, S. Guidelines for interpreting the results of bibliometrics analysis: a sensemaking approach. Glob. Bus. Organ. Excell. 43 , 17–26 (2023).

Aria, M. & Cuccurullo, C. bibliometrix: an R-tool for comprehensive science mapping analysis. J. Informetr. 11 , 959–975 (2017).

Åström, F., Danell, R., Larsen, B. & Schneider, J. (eds) Celebrating Scholarly Communication Studies: A Festschrift for Olle Persson at His 60th Birthday Vol. 05-S (International Society for Scientometrics and Informetrics, 2009).

Priyadarshini, P. The COVID-19 pandemic has derailed the progress of Sustainable Development Goals. Anthr. Sci. 1 , 410–412 (2022).

Zhao, D. Towards all-author co-citation analysis. Inf. Process. Manag. 42 , 1578–1591 (2006).

Shamseer, L. et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ 349 , g7647 (2015).

Download references

Acknowledgements

A.I.A. acknowledges Imam Abdulrahman Bin Faisal University in Dammam, Saudi Arabia, for their support in conducting this study. A.S. acknowledges the support of the Japan Society for the Promotion of Science KAKENHI grant number 22K04493. We appreciate Hiroshima University for supporting the open-access publication of this article.

Author information

Authors and affiliations.

Department of Urban and Regional Planning, College of Architecture and Planning, Imam Abdulrahman Bin Faisal University, Dammam, Saudi Arabia

Abdulaziz I. Almulhim

The IDEC Institute and Network for Education and Research on Peace and Sustainability, Hiroshima University, Hiroshima, Japan

Ayyoob Sharifi

Department of Geomatics Engineering Technology, Yanbu Industrial College, Yanbu, Saudi Arabia

Yusuf A. Aina

Directorate of Library Affairs, Imam Abdulrahman Bin Faisal University, Dammam, Saudi Arabia

Shakil Ahmad

The Business School, Edinburgh Napier University, Edinburgh, UK

Academy of Architecture and Urban Studies, Tallinn University of Technology, Tallinn, Estonia

Department of Natural Sciences, Manchester Metropolitan University, Manchester, UK

Walter Leal Filho

Research and Transfer Centre ‘Sustainable Development and Climate Change Management’, Hamburg University of Applied Sciences, Hamburg, Germany

College of Architecture and Planning, Imam Abdulrahman Bin Faisal University, Dammam, Saudi Arabia

Ismaila Rimi Abubakar

You can also search for this author in PubMed   Google Scholar

Contributions

A.I.A.: conceptualization, methodology, formal analysis, data curation, writing—original draft, writing—review and editing, investigation and project administration. A.S.: methodology, software, formal analysis, visualization and writing—original draft. Y.A.A.: conceptualization, writing—original draft, investigation and validation. S.A.: methodology, software, formal analysis, visualization and data curation. L.M.: writing—review and editing, and investigation. W.L.F.: writing—review and editing, and investigation. I.R.A.: writing—review and editing, investigation, supervision, validation and resources.

Corresponding authors

Correspondence to Abdulaziz I. Almulhim or Ayyoob Sharifi .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Peer review

Peer review information.

Nature Cities thanks Aliyu Salisu Barau, Oluwatobi Owojori, and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary information.

Supplementary Figs. 1–5 and Tables 1–13.

Reporting Summary

Source data fig. 1.

Statistical source data in Excel format.

Source Data Fig. 2

Source data fig. 3, source data fig. 4, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Almulhim, A.I., Sharifi, A., Aina, Y.A. et al. Charting sustainable urban development through a systematic review of SDG11 research. Nat Cities (2024). https://doi.org/10.1038/s44284-024-00117-6

Download citation

Received : 22 September 2023

Accepted : 30 July 2024

Published : 27 August 2024

DOI : https://doi.org/10.1038/s44284-024-00117-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

• Share full article

Advertisement

Supported by

New York City’s Trees Work Hard. Still, They Could Use a Little Help.

A chemical reaction involving emissions from cars and buildings can negate their environmental benefits. New research shows what big cities can do about it.

By Austyn Gaffney

Oak trees with leaves like outstretched hands and thick, stately trunks reduce the effects of extreme heat around New York City’s five boroughs by shading sidewalks and sucking up planet-warming carbon.

These workhorses also naturally emit a chemical called isoprene. Alone, it is harmless. But when isoprene meets nitrogen oxides, a group of gasses emitted when fossil fuels are burned to power buildings and vehicles, the chemical reaction creates ground-level ozone, a harmful pollutant that can cause respiratory issues.

“The trees themselves don’t create any problems, in fact they solve a lot of problems for us in cities,” said Andrew Reinmann, associate professor of environmental science at the City University of New York and an author of a new study examining the isoprene emissions of city trees. “The problem is cars and fossil fuel combustion in cities that can essentially start to negate or take away some of the benefits that trees are providing us.”

The study, published in July in Environmental Science and Technology , looked at citywide data for every six-by-six-inch square of tree canopy across the five boroughs. The researchers analyzed the types of trees and modeled what could happen to ground-level ozone pollution if the city grew enough trees to cover 40 percent of its land.

In one scenario, if the city only planted oaks, ground-level ozone in Manhattan could increase more than 30 percent. In a second scenario, continuing to plant a mixed canopy like the one that exists today could increase peak ground-level ozone by about 10 percent, depending on the borough.

New York City repeatedly violates federal health-based air quality standards for ozone on hot days, according to the study. Every year more than 400 New Yorkers are estimated to die from ground-level ozone pollution , and more than 4,300 adults and children in the city visit emergency rooms for ozone-related asthma.

We are having trouble retrieving the article content.

Please enable JavaScript in your browser settings.

Thank you for your patience while we verify access. If you are in Reader mode please exit and  log into  your Times account, or  subscribe  for all of The Times.

Thank you for your patience while we verify access.

Already a subscriber?  Log in .

Want all of The Times?  Subscribe .

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Dtsch Arztebl Int
• v.106(11); 2009 Mar

Study Design in Medical Research

Bernd röhrig.

1 Institut für Medizinische Biometrie, Epidemiologie und Informatik (IMBEI), Johannes Gutenberg-Universität Mainz

Jean-Baptist du Prel

2 Zentrum Präventive Pädiatrie, Zentrum für Kinder- und Jugendmedizin, Johannes Gutenberg-Universität Mainz

Maria Blettner

The scientific value and informativeness of a medical study are determined to a major extent by the study design. Errors in study design cannot be corrected afterwards. Various aspects of study design are discussed in this article.

Six essential considerations in the planning and evaluation of medical research studies are presented and discussed in the light of selected scientific articles from the international literature as well as the authors’ own scientific expertise with regard to study design.

The six main considerations for study design are the question to be answered, the study population, the unit of analysis, the type of study, the measuring technique, and the calculation of sample size.

Conclusions

This article is intended to give the reader guidance in evaluating the design of studies in medical research. This should enable the reader to categorize medical studies better and to assess their scientific quality more accurately.

Medical research studies can be split into five phases—planning, performance, documentation, analysis, and publication ( 1 , 2 ). Aside from financial, organizational, logistical and personnel questions, scientific study design is the most important aspect of study planning. The significance of study design for subsequent quality, the relability of the conclusions, and the ability to publish a study are often underestimated ( 1 ). Long before the volunteers are recruited, the study design has set the points for fulfilling the study objectives. In contrast to errors in the statistical evaluation, errors in design cannot be corrected after the study has been completed. This is why the study design must be laid down carefully before starting and specified in the study protocol.

The term "study design" is not used consistently in the scientific literature. The term is often restricted to the use of a suitable type of study. However, the term can also mean the overall plan for all procedures involved in the study. If a study is properly planned, the factors which distort or bias the result of a test procedure can be minimized ( 3 , 4 ). We will use the term in a comprehensive sense in the present article. This will deal with the following six aspects of study design: the question to be answered, the study population, the type of study, the unit of analysis, the measuring technique, and the calculation of sample size—, on the basis of selected articles from the international literature and our own expertise. This is intended to help the reader to classify and evaluate the results in publications. Those who plan to perform their own studies must occupy themselves intensively with the issue of study design.

Question to be answered

The question to be answered by the research is of decisive importance for study planning. The research worker must be clear about the objectives. He must think very carefully about the question(s) to be answered by the study. This question must be operationalized, meaning that it must be converted into a measurable and evaluable form. This demands an adequate design and suitable measurement parameters. A distinction must be made between the main questions to be answered and secondary questions. The result of the study should be that open questions are answered and possibly that new hypotheses are generated. The following questions are important: Why? Who? What? How? When? Where? How many? The question to be answered also implies the target group and should therefore be very precisely formulated. For example, the question should not be "What is the quality of life?", but must specify the group of patients (e.g. age), the area (e.g. Germany), the disease (e.g. mammary carcinoma), the condition (e.g. tumor stage 3), perhaps also the intervention (e.g. after surgery), and what endpoint (in this case, quality of life) is to be determined with which method (e.g. the EORTC QLQ-C30 questionnaire) at what point in time. Scientific questions are often not only purely descriptive, but also include comparisons, for example, between two groups, or before and after the intervention. For example, it may be interesting to compare the quality of life of breast cancer patients with women of the same age without cancer.

The research worker specifies the question to be answered, and whether the study is to be evaluated in a descriptive, exploratory or confirmatory manner. Whereas in a descriptive study the units of analysis are to be described by the recorded variables (e.g. blood parameters or diagnosis), the aim in an exploratory analysis is to recognize connections between variables, to evaluate these and to formulate new hypotheses. On the other hand, confirmatory analyses are planned to provide statistical proofs by testing specified study hypotheses.

The question to be answered also determines the type and extent of the data to be recorded. This specifies which data are to be recorded at which point in time. In this case, less is often more. Data irrelevant to the question(s) to be answered should not be collected for the moment. If too many variables are recorded at too many time points, this can lead to low participation rates, high dropout rates, and poor compliance from the volunteers. The experience is then that not all data are evaluated.

The question to be answered and the strategy for evaluation must be specified in the study protocol before the study is started.

Study population

The question to be answered by the study implies that there is a target group for whom this is to be clarified. Nevertheless, the research worker is not primarily interested in the observed study population, but in whether the results can be transferred to the target population. Accordingly, statistical test procedures must be used to generalize the results from the sample for the whole population ( figure 1 ).

Connection between overall population and study population/data

The sample can be highly representative of the study population if it is properly selected. This can be attained with defined and selective inclusion and exclusion criteria, such as sex, age, and tumor stage. Study participants may be selected randomly, for example, by random selection through the residents’ registration office, or consecutively, for example, all patients in a clinical department in the course of one year.

With a selective sample, a statement can only be made about a population corresponding to these selection criteria. The possibility of generalizing the results may, for example, be greatly influenced by whether the patients come from a specialist practice, a specialized hospital department or from several different practices.

The possibility of generalization may also be influenced by the decision to perform the study at a single institution or site, or at several (multicenter study). The advantages of a multicenter study are that the required number of patients can be reached within a shorter period and that the results can more readily be generalized, as they are from different treatment centers. This raises the external validity.

Type of study

Before the study type is specified, the research worker must be clear about the category of research. There is a distinction in principle between research on primary data and research on secondary data.

Research on primary data means performing the actual scientific studies, recording the primary study data. This is intended to answer scientific questions and to gain new knowledge.

In contrast, research on secondary results involves the analysis of studies which have already been performed and published. This may include (renewed) analysis of recorded data, perhaps from a register, from population statistics, or from studies. Another objective may be to win a comprehensive overview of the current state of research and to come to appropriate conclusions. In secondary data research, a distinction is made between narrative reviews, systematic reviews, and meta-analyses.

The underlying question to be answered also influences the selection of the type of study. In primary research, experimental, clinical and epidemiological research are distinguished.

Experimental research includes applied studies, such as animal experiments, cell studies, biochemical and physiological investigations, and studies on material properties, as well as the development of analytical and biometric procedures.

Clinical research includes interventional and noninterventional studies. The objective of interventional clinical studies (clinical trials) is "to study or demonstrate the clinical or pharmacological activities of drugs" and "to provide convincing evidence of the safety or efficacy of drugs" (AMG, German Drugs Act §4) ( 5 ). In clinical studies, patients are randomly assigned to treatment groups. In contrast, noninterventional clinical studies are observational studies, in which patients are given an individually specified treatment ( 6 , 7 ).

Epidemiological research studies the distribution and changes with time of the frequency of diseases and of their causes. Experimental studies are distinguished from observational studies ( 7 , 8 ). Interventional studies (such as vaccination, addition of food additives, fluoride addition to drinking water) are of experimental character. Examples of observational epidemiological studies include cohort studies, case control studies, cross-sectional studies, and ecological studies.

A subsequent article will discuss the different study types in detail.

Unit of analysis

The unit of analysis (investigational unit) must be specified before starting a medical study. In a typical clinical study, the patient is the unit of analysis. However, the unit of analysis may also be a technical model, hereditary information, a cell, a cellular structure, an organ, an organ system, a single test individual (animal or man), or specified subgroup or the population of a region or of a country. In systematic reviews, the unit of analysis is a single study. The sample then includes the total of all units of analysis. The interesting information or data (observations, variables, characteristics) are recorded for the statistical units. For example, if the heart is being investigated in a patient (the unit of analysis), the heart rate may be measured as a characteristic of performance.

The selection of the unit of analysis influences the interpretation of the study results. It is therefore important for statistical reasons to know whether the units of analysis are dependent or independent of each other with respect to the outcome parameter. This distinction is not always easy. For example, if the teeth of test persons are the unit of analysis, it must be clarified whether these are independent with respect to the question to be answered (i.e. from different test persons) or dependent (i.e. from the same test person). Teeth in the mouth of a single test person are generally dependent, as specific factors, such as nutrition and teeth cleaning habits, act on all teeth in the mouth in the same way. On the other hand, extracted teeth are generally independent study objects, as there are no longer any shared factors which influence them. This is particularly the case when the teeth are subject to additional preparation, for example, cutting or grinding. On the other hand, if the observations are on tooth characteristics developed before extraction, these characteristics must be regarded as dependent.

Measuring technique

The term "measuring technique" includes the use of measuring instruments and the method of measurement.

Use of measuring instruments

Measuring instruments include instruments which specifically record measuring data (such as blood pressure or laboratory parameters), as well as data collection with standardized or self-designed questionnaires (for example, quality of life, depression, or satisfaction).

During the validation of a measuring instrument, its quality and practicability are evaluated using statistical parameters. Unfortunately, the nomenclature is not fully standardized and also depends on the special area (for example, chemical analysis, psychological studies with questionnaires, or diagnostic studies). It is always the case that a measuring instrument of high quality should be of high precision and validity.

Precision describes the extent to which a measuring technique consistently provides the same results if the measurement is repeated ( 9 ). The reliability (or precision) provides information on the precision or the occurrence of random errors. If the precision is low, the correlation coefficients are low, measurements are imprecise and a larger sample size is needed ( 9 ). On the other hand, the validity (accuracy of the mean or trueness) of a measuring instrument is high if it measures exactly what it is supposed to measure. Thus the validity provides information on the occurrence of systematic errors ( 10 ). Whereas the precision describes the difference (variance) between repeated measurements, the validity reflects the difference between the measured and true parameter ( 10 ). Figure 2 portrays the terms, using a target as a model.

Portrayal of the terms reliability (precision) and validity (trueness) using a target

Reliability and validity are subsumed in the term accuracy ( 11 , 12 ). The accuracy is only high when both the precision and the validity are high. Table 1 summarizes the important terms to validate a measurement method.

The problem is not only that the measurements may be invalid or false, but also that the measurements may lead to erroneous conclusions. External and internal validity can be distinguished ( 13 ). External validity means the possibility of generalizing the study results for the study population to the target population. The internal validity is the validity of a result for the actual question to be answered. This can be optimized by detailed planning, defined inclusion and exclusion criteria, and reduction of external interfering factors.

Measurement plan

The measurement plan describes the number and time points of the measurements to be performed. To obtain comparable and objective measurements, the measurement conditions must be standardized. For example, clinical study measurements such as blood pressure must always be performed at the same time, in the same room, in the same position, with the same instrument, and by the same person. If there are differences, for example in the investigator, measuring instrument, analytical laboratory or recording time, it must be established that the measurements are in agreement ( 10 , 13 ).

The type of scale used for the recorded parameter is also of decisive importance. Putting it simply, metric scales are superior to ordinal scales, which are superior to nominal scales. The type of scale is so important, as both descriptive statistics and statistical test procedures depend on it. Transformation from a higher to a lower scale type is in principle possible, although the converse is impossible. For example, the hemoglobin content may be determined with a metric scale (e.g. as g/dL). It can then be transformed to an ordinal scale (e.g. low, normal and high hemoglobin status), but not conversely.

Calculation of sample size

Whatever the study design, a calculation must be performed before the start of the study to estimate the necessary number of units of analysis (for example, patients) to answer the main study question ( 14 – 16 ). This requires calculation of sample size, exploiting knowledge of the expected effect (for example, the clinically relevant difference) and its scatter (for example, standard deviation). These may be determined in preliminary studies or from published information. It is generally true that a large sample is required to discover a small difference. The sample must also be large if the scatter of the outcome parameter is large in the study groups. Sample size planning helps to ensure that the study is large enough, but not excessively large. The sample size is often restricted by the available time and/or by the budget. This is not in accordance with good scientific practice. If the sample is small, the power will also be low, bringing the risk that real differences will not be identified ( 16 , 17 ). There are both ethical problems—stress to patients, possibly random allocation of therapy—and economic problems—financial, structural, and with regard to personnel—which make it difficult to justify a study which is either too large or not large enough ( 16 – 19 ). The research worker has to consider whether alternative procedures might be possible, such as increasing the time available, the personnel or the funding, or whether a multicenter study should be performed in collaboration with colleagues.

Planning, performance, documentation, analysis, and publication are the component parts of medical studies ( 1 , 2 ). Study design is of decisive importance in planning. This not only lays down the statistical analysis, but also ultimately the reliability of the conclusions and the significance and implementation of the study results ( 2 ). A six point checklist can be used for the rapid evaluation of the study design ( table 2 ).

According to Sackett, about two thirds of 56 typical errors in studies are connected to errors in design and performance ( 20 ). This cannot be corrected once the data have been collected. This makes the study less convincing. As a consequence, the design must be precisely planned before starting the study and this must be laid down in the study protocol. This requires a great deal of time.

In the final analysis, studies with poor design are unethical. Test persons (or animals) are subjected to unnecessary stress and research capacity is wasted ( 21 , 22 ). Medical studies must consider both individual ethics (protection of the individual) and collective ethics (benefit for society) ( 22 ). The size of medical studies is often too small, so that the power is also too small ( 23 ). For this reason, a real difference—for example, between the activity of two therapies—is either unidentified or only described imprecisely ( 24 ). Low power is the result if the study is too small, the difference between the study groups is too small, or the scatter of the measurements is too great. Sterne demands that the quality of studies should be increased by increasing their size and increasing the precision of measurement ( 25 ). On the other hand, if the study is too large, unnecessarily many test persons (or animals) are exposed to stress and resources (such as personnel or financial resources) are wasted. It is therefore necessary to evaluate the feasibility of a study during the planning phase by calculating the sample size. It may be necessary to take suitable measures to ensure that the power is adequate. The excuse that there is not enough time or money is misplaced. The power may be increased by reducing the heterogeneity, improving measurement precision, or by cooperation in multicenter studies. Much more new knowledge is won from a single accurately performed, well designed study of adequate size than from several inadequate studies.

Only adequately planned studies give results which can be published in high quality journals. Planning errors and inadequacies can no longer be corrected once the study has been completed. It is therefore advisable to consult an experienced biometrician during the planning phase of the study ( 1 , 16 , 17 , 18 ).

Acknowledgments

Translated from the original German by Rodney A. Yeates, M.A., Ph.D.

Conflict of interest statement

The authors declare that no conflict of interest exists according to the guidelines of the International Committee of Medical Journal Editors.